Reason that Silver can be a problem in Colony games is that 5 Silvers don't get you a Colony.
But, with cards that increase hand-size like Wharf, you don't mind having Silvers.
At least that's the logic, I've run no simulations to see what little difference it makes if you buy a Silver before your last reshuffle or not.