• What balance of card costs and types do you put in?
Donald has addressed card type balance in the interview thread, I think. Let me find the quote.
These days for ~25 cards by default I want about 2.5-3 villages (counting thrones or expensive ones for less), 2-3 +buys, 2-4 remodels/vaults (trash or discard for benefit), 4 attacks (down from 5), 2-3 non-attack player interaction (up from 1), 1 victory card, 1 treasure, 6-7 cards with +1 card +1 action (up from 5, counting villages but not thrones), and a bunch of $5's.
OK now here's my personal take on that list.
• For better or worse, most Dominion sets average more than 2-3 +Buys. It's more like 3-4. If you have a set of about 25 cards and only 2 have +Buy, hopefully your set has a lot of gainers.
• I agree that one pure Victory card is the perfect number. You could totally have more if the rest of them are also Actions or Treasures. But any new hybrid Victory card should have a pretty good reason to have VP attached to it. Having VP just tacked onto another card was cool once (in Intrigue), but now it's been done.
• It's totally fine to have a bunch of Kingdom Treasure cards. It's important to have mostly Actions, since they can be stronger for their cost, but if your set has a few interesting Treasures, great.
• I recommend shooting for 40% of your set's cards costing $5 (unless your set has a major wonky-cost theme, like Alchemy). Realistically you'll probably end up with fewer $5 cards, but 2 out of every 5 cards costing $5 is a good goal.
• What does each set need? Should there be at least one trasher, one gainer, one sifter etc...
This is sort of related to the previous question. It would be weird to have a set with zero trashers. I think it's important to have some way to deal with junk cards, and that usually means having enough trashers/sifters. Your set can be light on one and heavy on the other. Hinterlands leans toward sifting, whereas Dark Ages leans toward trashing. But they both have some of each.
I wouldn't even try to create a set of interesting cards without a gainer. A lot of cards in Dominion gain other cards, even if it's just "Gain a Silver".
• How many themes should an expansion have? How many cards make up a theme? Are off-theme cards OK?
Some "themes" in official Dominion expansions are just 3 cards. Like Looters, Spoils-gainers, VP token cards, etc. I would hesitate to call something a theme if it was used by fewer than 3 cards. But really it's all just semantics. It's good to have at least one theme, and it's good if at least one of your themes has at least 7-8 cards. It's totally OK to have off-theme cards.
• To what extent is it OK to use mechanics from other expansions? What about completely new mechanics?
Some mechanics you can just put wherever, like choose-one and on-gain abilities. Beyond that, there's no consensus in this community. Lots of people are willing to add e.g. one Potion-cost card or one VP-chip card to an expansion. I think that if you're going to require components like that, you should have enough cards to justify requiring them.
• How do these things change in large vs. small expansions?
Small expansions only exist because the German publisher at the time (Hans im Glück) wanted them. You can still totally make one if you don't have enough card ideas for a full set, though. Mostly you just want to keep the ratios of cards the same. About 10%-15% villages, 35%-40% $5 costers, 15% attacks, etc.