Based on the wording of both the base game rules and Philosopher's Stone, I would say that during play, your "deck" specifically refers to your draw pile. However, at the end of the game, all cards that are "yours" get returned to your deck - including cards in hand, in play, in your discard pile, set aside, and on your tavern mat (but not cards that you never had, nor cards you got rid of during play) - for the purpose of scoring.
However, from both the wording and intent of Distant Lands, if you put it on your tavern mat, then at the end of the game it *does* provide you with 4 VP, and it *also* counts as a card in your deck for the purpose of Gardens, Silk Road, etc.
Given this is apparently a bit of a fluid concept, I might try drafting up a wiki article to cover what it means at various points of the game.