Yes - it can be that bad. It just depends on how your opponent has a deck built to leech on those opportunities you toss at him! For example, one would go and prioritize the purchase of other cards because you are taking care of bishoping away what he would've trashed, basically being employed as his garbageman for the game. And no, I would NEVER pass on an opportunity to trash my own cards while my opponent plays it, regardless of the fact I own a bishop myself... way faster.
Also, most of the times I am disappointed in my 7$ hand I just got dealt when my savior of an opponent plays a council room to nudge me right over the 8$ bar... I can't count how many times this has happened to me. Think of it this way : any card I draw but a green one will help me in that sense, whether it is money, or one of my carefully picked actions I bought. I guess it's particularly dangerous when you pile them up because, chances are, you turned whatever hand your opponent had into a fully enabled, ready to roll hand, and if you did all of that work for 1 province, it is not worth it at all. Don't forget that most hands are just one card away from being perfect, or at least enabled for something greater... Make them draw just one Hunting party and there you go! As an aside, letting them have 1 more card has way less effect if your opponent draws his whole deck every hand anyway, like Alchemists or such...
As for Governor, it has been discussed in other threads, but the Gain a gold option would be the most viable at the start of a game, and yes, can be used as an attack in Colony games. The drawing cards DOES hurt more than you think. It can be THAT bad.