Given how often you were topdecking his Colonies, I wonder if a single Forge buy for him to Forge Colony -> Colony would have saved him. Of course, buying that Duchy after you got a Smugglers was a bad idea. As was not trying to three pile while 40 points ahead.