Here's a description of some of the stuff I considered in selecting sets. It turned out to be considerably harder than I thought.
You want to encourage unorthodox play, to reward creativity. Some of these sets do that -- Set #8, for example, has a neat hidden trick that olneyce exploited (though not to its fullest potential). But you can't make every set a gimmick set, and even the "gimmick" sets have to be about more than who spots the gimmick. So Set #3, the KC/Goons/Masq set, has a little rock-paper-scissors going on depending on how you deal with the pin. In Set #1 (and 9), the obvious play is not necessarily the wrong one. It's just not actually the obvious best move like it usually is. And in Set #2, the play general idea is obvious -- but how do you implement it?
You want to try to reward building super complex decks. But the problem with this is, 1) these tend to be a bit swingy, and 2) real Dominion is about more than just engines: there's Big Money too! This was the hardest to balance: I am not really satisfied that any of the final 9 sets allows a money strategy to compete. Some of the ones in the 16 final candidates had money as a viable candidate, but then you run into the danger of both players deciding that money would be faster and seeing a championship game devolve into who drew their Envoy/Big Money better.
You want to try to boost unpopular cards. But we didn't include a Kingdom with Thief, because either you make the Thief play incredibly overpowered (i.e., 9 Treasure Kingdoms + Thief), or Thief still just sits there unused. It's really, really hard to make truly bad cards an integral part of someone's deck. I think Set #4 does a good job of integrating Oracle, though I am told Oracle is not actually that bad. Set #8 puts Secret Chamber to good use: that's probably a better example.
You want to try to accurately represent Dominion as a whole. I touched upon this a little bit in how Big Money doesn't get a fair shake here. Well, neither do many attacks, because Sea Hag doesn't really affect your strategy so much as it sits on it. Of course, we do have attacks, but you have to balance it against ...
You want the sets not to be too swingy -- within reason. It's not an accurate representation of Dominion if you try to make it no-luck: I used to play with !attacks for a while, and boy let me tell you that was boring. The key is to try to work in ways to counter the attacks, such that the game is not just who spams the attack more, but still make the attack worth getting. Most random sets don't allow you that option. Here, Ambassador was a big problem, but eventually we decided to include it. olneyce won Game 8 even though he was behind in Ambassadoring. Torturer in Game 1 was another one that could be managed. But there's no Sea Hag, for instance, since either it shows up with Jack/Masq/etc., or it doesn't, and neither makes for an interesting game.
You want to balance the sets between each other. Maybe Ironworks/Gardens is a really good measure of skill, but man you do not want to see that in every game. I think we went a little overboard with Grand Markets/megaturns, as you all have pointed out. But overall I'm satisfied of the mix between slogfests and explosive games. There's more variety in megaturn games. You can only have so many Set #2's.
You want to balance Colony vs non-Colony. We ended up with much more Colony than you'd expect if you chose randomly, but this community likes Colony games, and at least we kept it around 50%.
So I ranked all the cards as "yes", "maybe", and "no". rrenaud gave them numbers. We tallied together and took the highest combined score sets, as well as a couple others that one of us liked a lot, and assembled a list of 16. Honestly, all 16 of those could have been final sets. But these 9 happened to complement each other well. I then ordered them to try to have some game-to-game diversity (for instance, no Gardens back-to-back, mix up Colonies). And though the games at the end are less likely to be played, you also don't want game 9 to be a swingy set (e.g., Governor/Goons).