I'm not quite convinced this should cost $4.
Yes, early on if it costs $3 then a Cavern/Cavern opening will rush a $5 pile very quickly. But as the game goes on, its usefulness drops off much faster than Horse Traders.
I don't think it matters how quickly it gets weak if it's a superstar early. If Cavern/Cavern is so good early on that it's the dominant opening a majority of the time (and I suspect it would be) then it's broken, because it closes the space of viable strategic decisions you can make, rather than broadening it.
Also compare with other cards that are powerful in the opening and drop off sharply afterwards: Trading Post is such a strong ability that it has to cost $5, but it's really only great if you buy it on the first shuffle. It's okay on the second and terrible afterwards. Despite the fact that it's only a superstar in the first few turns, it costs $5, not $3.
Remake is maybe a better example, because it's in the $4 tier. Late game, actually wanting two Remakes in your deck would be a niche situation, and lots of times you'd rather not even have the one anymore. But a double Remake opening would be crazy strong, transforming your deck into Silvers (remade Estates) + $5 card (remade Remake) so quickly that most other strategies probably couldn't keep up. By contrast, double Chapel is okay because two early Chapels is rarely an improvement over one.
Anyway, I'm not really so sure Cavern wouldn't still be a strong card in the late game. When you're choked with green, you can still rush Duchies more reliably than with any other card, because not only is it guaranteed to hit, its mechanics make it resilient to the influx of those Duchies. Compare with other Duchy rushers: IGG isn't a guaranteed hit, and sometimes you have to add Copper to your deck as well, thereby spacing out the IGGs. Horse Traders needs additional Treasure. Secret Chamber needs additional Treasure. Vault provides a benefit to opponents. Additionally, note that certain other cards
would be great Duchy rushers except that they don't work on Victory cards, presumably specifically for that very reason: University, Horn of Plenty, etc.