Really?
Yep. The Hermit says, "I got rid of your junk and gave you a silver, Boss. Can I go crazy now?" And the player says, "No. I've got a bunch of treasure I want to use this turn; you'll have to wait."
Whereas the Madman says, "Here's a couple actions, and you can double the cards in your hand, and I am outta here!"