I buy talisman somewhat often if there is caravan on the board and no power $5.
People have also correctly identified that talisman can be powerful with +buys, because there is a multiplication effect. For this reason, I often snag a talisman in engine games with hamlet, because Oh look, I've got $6, three buys and a talisman, and yes I'll buy the six remaining hamlets. This can be critical, especially if hamlet is the only village. Other similar situations can occur, but hamlet has all the critical pieces in one (cheap,+buy,critical engine piece). Obviously +buy makes talisman good at piling out too.
But yeah, it takes a lot of things to go right together for talisman to be right. Its even narrower than workshop, which still allows you to buy a $5 AND use the workshop to get a 4, not to mention the VP thing. Although occasionally the lack of requiring an action is nice.