We've always had the opposite problem - since it claims to do a large number of people, everyone present starts the game; then another person turns up and joins in, and so on and so on; and the gaps between turns get longer and longer and people's attention drifts, and then people get annoyed that people aren't paying attention. Then someone drops out, and it turns out that we've met the victory conditions for the lower number of people, and so we have a really dissatisfying win and everyone's p***ed off at each other.
It's a fun game with four, but I wouldn't play it with more than 6 at an absolute maximum.