The "Game End" seems to be three phases:
(1) Start of Game End.
(2) Take all cards (minus cards on mats if not specified to take).
(3) Scoring.
Since your current version is set aside and will get pulled at phase (2), should you have to specify the condition as being "At the start of Game End" or "At the start of the end of the game"?
I don't think this is correct. I don't think cards are "pulled"; Distant Lands shows us that cards remember where they were when the game ended, and there seems to be a general rule (written or not) that all cards that are yours count as part of your deck for the purposes of Gardens, etc. The fact Island says "returned to your deck" really has no rules meaning at all; given that cards like Haven don't have this same wording, but still work exactly the same.
It's a bit of an unknown rule, but cards on mats do not automatically go into your deck at the end of the game. They are still your cards, but for some reason they stay on the mat. Distant Lands is a good case of this. It's on your mat at the end of the game and checks there for 4VP. If it was automatically pulled it would be worth 0VP. Mostly a moot point though.
Really what I'm arguing is that set aside cards (like Judge) definitely get pulled and when the game ends where will Judge be? Maybe still set aside? Maybe pulled? As there is no precident for this I guess it's up to Kru5h, but I think treating the End Game as a phase like Clean-up would go a long way in clearing up any future misunderstanding.
*EDIT* I see your point though and am in no way disagreeing with it. It's a weird, sometimes seemingly incomplete part of the game. I guess it only ever matters if something is expected to be somewhere (i.e. on your Tavern mat, or set aside) that it matters.
*EDIT_02* Here is the actual text from the manual. "Take all of your cards - from your hand, deck, discard pile, play area, and even set aside cards - and sort them for putting them back in their piles. Count up your points." This is a bit off topic, but if a Victory card somehow got onto your Tavern mat and didn't check on itself to be there, would the points still be counted?