Yes. I think that's probably simplest.
This card definitely needs a differently-colored back in order to work. That way, all players can witness the exact moment you put it into your hand. Otherwise there'd be no accountability built into this card. Reactions also usually have a command to "reveal" as part of their reaction mechanic, and the way you have this written, you don't even have to do that to draw the extra card. You need more precise wording.
I suggest the following:
In games using this, when you put a yellow-backed card into your hand, you may draw one card.And then Servants' Dormitory would have yellow-colored backs. This wording would incentivize SD users to clearly but quickly show to everyone else exactly when they draw one into their hand, so that they can receive the benefit. The extra game rules used by cards such as Duchess, Young Witch, Trade Route, etc are sometimes the easiest way to implement certain ideas. Just trying to think of how a similar reaction could be written properly makes my head spin.
Of course, my solution introduces the problem of making shuffles technically not as random anymore. This was a feature for Stash, but might complicate Servants' Dormitory. Perhaps you should be given the option to space them however you'd like during a reshuffle as well?
Now, analyzing the actual play of this card: Even in boards without discard-for-benefit, the extra information and options granted by getting a large hand--before even playing your first action--is incredibly valuable. Drawing a few of these in one turn basically turns it into a pseudo-Tactician turn. Granted, it's more the equivalent of a Tactician turn with a lot of nerfed Villages drawn, but you can use Nobles and Hunting Party to much greater effect. And I don't even have to delve into Vault, Stables, Cellar, etc, do I?
As for this card's interaction with hand-size reducing attacks... A hand swollen-up by SD's drawn during the cleanup phase might perhaps be somewhat vulnerable to such attacks, though it at least nerfs them into Margraves. I find that interesting, thought I don't know if it's good or bad.