Scout is widely considered to be the weakest card in Dominion, and while this is kind of a boring conclusion, I think that's exactly what he is. I also think that being weak is his only problem - the idea of taking victory cards from your deck into your hand is something I really like - but fact is, a card that's ignored in approximately 98% of the times when it shows up is just a waste of space, good idea or not. We're out of cards that I dislike, so a 9-card Kingdom enabler is the next worst thing.
Considering Transmute handily won the worst card poll, I'm not so sure how wide that conclusion is. If it weren't for the existence of the forum meme, I doubt there would be much consensus. I mean Scout is obviously a weak card, but it's not obviously the weakest. I certainly would take a free Scout over a free Thief on most boards - at minimum it can provide useful ordering ahead of a terminal draw and never helps the opponent.
Adventurer is the same as Scout, really. It costs 6$, even though it could easily cost 3$, making it probably the second weakest card in the game. Digging for Treasure cards as a concept is okay, but again, that doesn't help if the card is too weak to ever get bought.
There is no way Adventurer would be fairly priced at $3. On average, it will be about a terminal Gold, so should probably cost 5. It is actually a pretty good card in a money deck (if no strong terminal draw) even at 6. Most people here (and most boards) favor engine to big money, and Adventurer is weak in an engine, so that is part of the reason for the reputation. The only real problem is the opportunity cost of passing up Gold. When you get a "free" Adventurer for some reason (perhaps a swindled Gold), it is normally helpful, albeit somewhat less so than Gold in most decks. At $3, Adventurer would make big money crazy good, which I doubt many people want.
Talisman is the weakest version on a list of Workshop Variants, that is cards that can gain cards costing 4$ or less. Other than Talisman, this list consists of Workshop, Ironworks, Armory, and you could make a case for Hermit, but let's not.
So, even ignoring the Victory card restriction, when is this card ever better than Ironworks? Well, if you
a) draw it with 3$ and want two 4$'s, or
b) draw it with 4$, want a 5$, and either don't want a silver, or don't have Actions left.
I'm sure both cases have happened plenty of times, but still, there is no way I ever buy this card over Ironworks, because you can't plan ahead for either of those cases. The far more common and relevant case is that you draw it with 5$+ and want both a 5$ and a 4$, and here Ironworks is clearly better.
And then there is the restriction. Why is it there? I honestly have no idea. This card is clearly worse than Ironworks without it. I guess if it was there, a hand of multiple talismans could get one victory card each, which none of the other variants can, so maybe that's the reason. Whatever the case, the fact that it does have it means that it's not usable for garden/silk road rushes, which just cuts about half of its utility. There are still situations where you buy it, and on its own this wouldn't be that low, but the problem is really that it's being overshadowed.
The restriction is almost certainly there to block a multiple Garden gain. Workshop+Gardens is already pretty good; Talisman+Gardens would be crazy good. That is not necessarily a problem (see Beggar-Gardens), but allowing (easy) multiple garden gains could lead to a pretty devastating split in a mirror, which is probably not something you want.
3+Gainer is going to be fairly common in the early game, so if a 4-cost card split is highly important, Talisman will beat the others. It also doesn't use an action, so there is a chance for terminal attack, which Armory and Workshop can't do. There is also this sweet combo:
http://gokosalvager.com/static/logprettifier.html?20140920/log.53ae3072e4b024efbd4224b7.1411268187569.txt#2-6 which Ironworks & Workshop couldn't do.