I think it would be allright without the passing mechanic. It adds very little and if anything it makes the card less interesting, since you won't have control over whether it is in your deck or not. It also weakens the interesting decision of "should I buy a card that will be useless 4 turns from now?" since you're not buying future junk for your deck, you're rather adding balls to the great table tennis game, to the collective detriment of all players rather than you alone. The exact number needs testing, unfortunately it probably varies quite a bit depending on the number of players.
Maybe trash Estates from the Supply and give the bonus if the pile is not empty yet? This would at least add a bit of scaling between player numbers.
