Right, and you can't specify that only non-Victory cards go on the mat without them revealing cards as they're set aside,

Uh... sure you can. If, at the end of the game, there is a victory card on the mat, then they cheated. There's only an accountability issue if they can also anonymously remove cards from the mat.

Alternatively, if you want to exclude victory cards, just say they are allowed to put any card on the mat, but only non-victory cards are counted.

Still probably too strong with Peddler.

Bishop also trashes cards and scores based on their cost, so I don't think this is any different in that regard.

The bigger issue that I see with any card which scores based on cards it puts onto a mat is that it will run into the original Duke problem. (For those unfamiliar with the original Duke, it said "Worth 1 VP per Duke in your deck"; which turns out to be un-balanceable at any ratio because they are terrible if you don't get enough of them and broken if you get a lot of them (or they are always terrible or always broken).) The way that I see this being a problem with Cave is that, with one Cave in your deck, you'll be getting cards onto your mat at some rate, x, and you'll be scoring once based on that rate. If you have two Caves in your deck, you'll be getting cards onto your mat at a rate of 2x, and then you'll be scoring twice based on that rate, so you'll be getting roughly 4 times as many VP than you would if you had one cave. With three caves, you'll be getting cards onto your mat three times as quickly, and then scoring three times as many points per card on the mat, so that's 9 times as many points. You see where this is going; the number of points each Cave is worth increases quadratically, which is exactly the same problem the original Duke had.

Okay, so that's an approximation and in reality I don't think it's as bad as the original Duke, since there's terminal collision, and maybe you can't always get what you want onto your mat. It's still a pretty big problem though. The original version, "x VP per differently named card on your mat", actually does a reasonable job of dealing with this issue, by making it harder and harder to squeeze more points out of Cave with each subsequent play of it. The cost-based version I think is a lot more likely to be terribroken, since you can always keep feeding it more cards for points. In fact, the cost-based version just seems like a poorly executed version of Bishop; it scores points based on the cost of the card trashed (or pseudo-trashed), but this version has an un-balanceable ratio problem, since it scores quadratically with the number of Caves in your deck.

Edit: I should mention though that I don't think the original version does a good enough job of dealing with the issue. Either the ratio will have to be really low to make sure it isn't broken, in which case the fact that it slows down makes it even weaker; or the ratio will have to be reasonably high, in which case it will probably be a dominant strategy to amass them and try to store away as many cards as possible. It's a little slower than quadratic, but I suspect still faster than linear, so the problem is still present, just to a much lesser degree.