The concept is indeed interesting, just not sure it is balancable.
A potential minor issue is that you can exchange those for themselves, allowing indefinite stalling. Lets assume A and B both have one Construction Site, and the Castles up to Grand Castle as well as the Provinces are gone. It is easy to imagine a situation where neither player would want to get Grand Castle when their opponent can get King's Castle afterwards, but taking something else will free the opponent to get the Castle and win. The best move is to keeping exchanging them for themselves until everyone starves to death. Of course this is avoidable by disallowing this exchange.
A bigger problem are Colony games. Here, since at most one of Provinces/Colonies is going to run out usually, these are worth 6 VP at least.
The are also some probably resolvable timing issues. When is "at the end of the game"? By Donate/Mountain Pass ruling, the end of the game happens when it's still a player's turn, so theoretically this means that Construction Site is affected by Possession. Also Fleet turns happen "after the game ends", adding further confusion.