In a 2 player game, assuming both players run Chapel and trash all starting cards, 20 cards are in the trash. So Vagrant = 3 VP. So either the card multiple or the VP per card needs to be boosted.
But more to the point, without trashers on the board this is completely dead. Something like a Farmland effect or an Action-Victory card that trashes stuff would make the card work better.
Finally, this isn't really a mark against the card, but...where's the challenge? As a comparison:
Gardens forces you to quickly build a large deck.
Duke forces you to buy Duchies without killing your economy
Fairgrounds forces you to buy a variety of cards.
Silk Road is similar to Gardens, but with more emphasis on sustained economy to keep greening. (this might not be true, I'm terrible with Silk Road)
Vagrant forces you to trash cards, which is likely what you were going to do in the first place, so you don't have to go out of your way to do something?
This doesn't matter if you only care about balance, but if you want to keep playing with the card, there's some point where you need to decide whether the card is interesting enough. If it's too easy to pump up Vagrant VP, it'll turn into "Oh look, there's that consolation prize again"