Base only is a fine way to play. Also, I think you should have played witch more. If you give him all 10 curses, that adds 10 cards to his deck. If he gets all 8 gardens, then he gets 8 extra points from the gardens, but loses 10 from the curses. There is the added benefit of making his deck much worse, and ensuring that he can't buy duchies.
Does anyone have the sim data on Witch BM vs WS/Gardens? I think it's something like 65-35 in favor of Witch, but it's been a while since I last checked, so I might be off.