The first two sets I bought were the base game and Prosperity. I later added Intrigue, Cornucopia, and Seaside in that order.
Now that I have played all of them, I really appreciate why everyone loves the Duration cards in Seaside. They really add a cool element to the game. It also has some powerful attack cards that are popular.
However, I am very glad that I bought Prosperity early on. I definitely don't recommend Prosperity to someone new to the game, but by your description of what you already have, that is not an issue for you. My wife and I love Prosperity because it adds the Colonies and Platinums. For us, it adds so much more depth to the game, and allows you to really investigate different stategies more because it usually lengthens the game. It has a number of cards that cost $7, which also gives you more choices when you hit the dreaded $7 rather than $8. And it has some cards in it that have become very popular among Dominion players, including Goons, Mountebank, City, Grand Market, Peddler, Watchtower (the best reaction card), and King's Court. We actually use the Colonies and Platinums whenever we possibly can in our games, even if we don't have the suggested 5 Prosperity cards in the game.
Basically, I believe that it boils down to what you like about the game. If your favorite thing about it is how fast each game plays, then you will enjoy Seaside more. But if you like the idea of your games being a little longer and allowing you to explore more options in a game, then I definitely recommend Prosperity as your next expansion. Either way, though, I am sure you will be pleased.