Another problem you may be having is thinking "This card does X, but not very well. If this other card does X better, why don't I get rid of the first one?" but there are two reasons - firstly, the first card probably does something different: Loan, Mint and Mine all trash treasures, but in different ways; secondly, because not every game will have the "better" card in it, and you will need to decide whether the "worse" card is still worth buying for what it does.
For example, nearly any "Village" is better than the plain Village - Worker's Village gives you +Buy, Farming Village guarantees you'll draw a Treasure or an Action, Mining Village gives you an option to trash it for $2 - but sometimes the only village out there is Village, and if you still want to be able to play two terminal actions in a turn then you're going to want that card.
Again, I will suggest that if a particular card seems useless to you, then see if it has a strategy article. For example, you say of Contraband that people will just stop you from buying Province/Colony. And they will - but maybe what you really wanted to buy was Gold. Or a Village and a Smithy. Or some other neat combination of cards on the table. As the
wiki and
blog article points out, Contraband is not an "always buy", but if there are lots of different cheap cards you want then the opponent can't stop you from buying all of them, and once your deck has what it needs you stop playing the Contrabands so they can't stop you from buying those lovely Provinces.
If you don't want to play with a particular card, then go ahead and don't let us tell you that you should or you shouldn't. But I suggest that every once in a while you let one of those "removed" cards back in the game, and see what happens if you buy one or two of them. Play a game where you buy one Chancellor and up to two copies of another strong card, and any time you know both copies are in the discard then choose to let Chancellor flip the deck. Play a game with Scout, Great Hall, Harem and Nobles and see what happens when the Scout is actually drawing useful cards. It doesn't have to be every game, but once in a while see if you can understand when these cards have their moments.