Other trashers end up playing a big role in how the game plays out. If you don't have to use Hermit as a trasher to trigger your Market Squares, it means you can make them all into Madmen and have a bigger payload. This usually only comes up in the mirror though, or at least whenever components are conested -- almost all of the time that I play H/MS against someone who doesn't know what's going on, they end up contesting Hermits at the very least for some weird reason. I ask them why and they just say that when they saw me rushing them, they followed me just for the sake of following me.
Chapel and Forager come to mind, since they can be gained by Hermits; along with Develop, Trade Route, Steward, Stonemason, Masquerade, and even Embargo!
Lookout and Doctor aren't so good, though.