It definitely strayed more, but I didn't find the straying bad. The streamlining at Harrenhal worked relatively well, I thought. Did you get to the scenes between Arya and Tywin Lannister? They don't happen in the book, since Tywin never comes to Harrenhal, but they are quite good in the show.
I also liked the changes at Winterfell--Theon killing Rodrik, for instance--though the castle getting destroyed at the end was extremely confusing (deliberately so, I think) if you hadn't read the book. To that end, I didn't mind reserving Ramsay Bolton for a future season, even though the way it happens in the book is a fairly cool reveal. Roose Bolton was doing different thinks in the second season as well, but Roose never seemed like he would develop into an important villain in the first two books, so it seemed sort of appropriate to have him on screen, just kind of being there without being important. And I didn't mind the changes to Robb's marriage.
To me the boring stuff was Daeneryis and Jon, but I found their book 2 plots boring anyway.