The problem you run into facing Governor isn't the speed. It's the ability to finish the game at will.
A Governor deck can pick up 1-2 Provinces and wait until it can pick up the last 2-3 in one turn, even on boards where no other deck can pull off multi-Prov turns.
I think the last part is the most relevant. In the absence of any other way to get multi-provinces, using the governor for drawing becomes MUCH stronger. Every card over $8 you add to the other player's hand isn't doing them anygood, so you only need to be able to double-province for this to work out in your favor.