Another plus for Caravansery (vs. Caravan) is that you increase your handsize during other player's turn which means you are more likely to reveal Reactions (you know, Moat, Bane card, Fool's Gold, Tunnel, Market Square...) and in better position for the opponent's Masquerade, Bishop, Vault, Mountebank or Tournament as well.
But maybe $4 is alright for both Relic and Caravansery.
Another thing to try could be Wharf-stile, i.e. making it terminal and draw another card:
Caravansary (Action) $4
At the end of this turn, after drawing cards, +2 cards.