imagine a 4p game with Saboteurs all around, that's not likely to be a fun game.
I've been in some of those; it can be annoying, but as usual, it depends on the board. If there are lots of gainers, or the ability to silver-flood, you might not notice much. With TR/KC, it can be intolerable.
But it's a good point; there are a number of attack cards that become dramatically more powerful/painful with more players: Gainers (Thief, Noble Brigand, and Jester) gain more cards, and Pirate Ship gains coins more reliably. Trash attacks do become more crippling when you get hit by more than one per turn. Junking attacks become much more painful and swingy; since there are more Curses in the supply, anyone who takes more than their share can become hopelessly pinned. If your Moats collide in the early stages of a multiplayer Familiar game, it's generally fatal.
Some off-the-top-of-my-head thoughts on how to mitigate this effect:
Thief, Noble Brigand, Jester: Limit gain to 1 card.
Pirate Ship: Require hitting treasure on at least half of opponents (round up) to gain a coin.
Junkers: Reveal two junk cards to defend against it. Or (big change) add a rule that allows someone to skip their turn to trash up to X curses from their hand (determine X by playtesting).
Trashers: Not sure what to do here...you need some kind of testable state to tell if someone's already been hit, or is already so trashed that they shouldn't be beat down further. Have them reveal their hand to show no card that costs more than X?