I have a further improvement for when there are 4 or more people. It takes 2 full rounds, plus 7 extra people in the worst case.
This solution is similar to hhelibebcnofnena's, but compresses some of the rounds.
1st round: Same as before, everyone passes unless they can deduce that their hand and the face down cards are all the same color.
2nd "round": Same idea that everyone passes unless they see someone with two whites and someone with two blacks. However, only the first 3 people need to do this--after 3 people, it becomes common knowledge that there is at most one person with two cards of the same color.
3rd "round": Similar idea to my combined round earlier--people will signal if there is someone with two whites, or two blacks, or neither. For each of the first 4 people in this round, if they can deduce that they have one white and one black, they guess if a) they are 1st or 3rd in the round, and they see someone with two white cards; b) they are 2nd or 4th in the round, and they see someone with two black cards. (Whenever any of these four people guess, the round ends early, as everyone can now deduce their cards.) If all 4 pass, it means that no one has cards of the same color--otherwise, it's guaranteed someone will signal.
4th round: Everyone left guesses their cards.
This takes n + 3 + 4 + n for all the guesses in the worst case, or 2n+7. It doesn't feel like you can do much better, if at all. (Exception: for n = 3, the old 4-round solution is better).