Possession and Familiar are the top offenders for me, but Possession is actually proposed, so I voted for that. Anybody who's been on the recieving end of Possession/Golem deck understands how horrible this card is to play against. Worst of all, I think people are overvaluing because of how horrible it feels when someone plays it, and are playing it more than they should, especially considering it is incredibly hard to get. I see people going for Possession as they only Potion card, and it should please me, because they lose so much time I'm likely to win, but it still annoys me because they're going to play it at some point.
As for Familiar, well, I actually think, now that I analyse it, that it's worse than Possession. The feeling of having 2 and a Potion on second turn of a Familiar game is just that horrible. It's turn 3, and you've basically already lost. At least with Possession, you're only losing on turn 15 or so.
I used to hate Treasure Map, but I don't anymore, because the card is just weak. Sure, you can get lucky, and it's annoying when it happens, but more often than not, seeing my opponent buy Treasure Map when no enablers are available is a pretty good sign.
I'm not sure what's so frustrating with Smugglers though. Sure, it's a variance card that can be frustrating at times, but I don't see it being anywhere the other cards mentionned it, or other not mentioned such as Sea Hag or Ghost Ship.
Edit : reading those answers, I have to agree with Swindler and Tournament being really up there too.