Your post is kinda confusing, but I think you're getting at the self-synergy of a card which counts VP in the trash, when it winds up in the trash itself. This is a really confusing, terrible way to word a VP card. There is no unique solution to the problem. With 1 estate and 4 landfills in the trash, the landfills could be worth -1, 0, or 1 VP. With 1 estate and 3 landfills in the trash, you do get an indeterminate score, but that's because ANYTHING works, not nothing. Landfills can be worth ANY amount when there are exactly 3 in the trash and not more than 2 VP otherwise - 1 million VP landfills, woohoo!
(1 Estate VP + 0 Landfill VP x 3)/3 = 1VP/3 = 0 VP per Landfill
(1 Estate VP + 1 Landfill VP x 3)/3 = 4VP/3 = 1 VP per Landfill
(1 Estate VP + -1 Landfill VP x 3)/3 = -2VP/3 = -1 VP per Landfill
(1 Estate VP + 1 million Landfill VP x 3)/3 = 3,000,001 VP/3 = ONE MILLION VP per Landfill
The way the card is worded, I don't think landfill could be worth negative VP.
Also, gardens in the trash are worth 0 VP - since there's no owner, there's no deck to count!