I actually think it should be okay to do. Say, for instance, that you sneakily (or even not-so-sneakily) end the game on piles, and your opponent(s) want to do their turn over, since they didn't realize. Totally bogus, right? It's their fault for not realizing! Same thing here, in my opinion- if they're not paying attention, they can't really justify calling you out. I understand that others may have a different opinion, though.