Dominion Strategy Forum

Please login or register.

Login with username, password and session length
Pages: 1 2 3 [4] 5  All

Author Topic: Leaderboard change  (Read 52804 times)

0 Members and 1 Guest are viewing this topic.

rrenaud

  • Administrator
  • *****
  • Offline Offline
  • Posts: 991
  • Uncivilized Barbarian of Statistics
  • Respect: +1197
    • View Profile
    • CouncilRoom
Re: Leaderboard change
« Reply #75 on: November 27, 2011, 04:09:26 pm »
0

samojedi was crazy good.  I wonder who it was. 
Logged

DStu

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2627
  • Respect: +1490
    • View Profile
Re: Leaderboard change
« Reply #76 on: November 27, 2011, 04:15:36 pm »
0

Also interestingly, the level range is still smaller than isotropic's (not to say that that's inherently bad). I wonder what other unpublished tweaks Doug has made to the rating code.

I thought the isotropic Trueskill-code is on Github. Are there parameters missing?
Logged

rspeer

  • Witch
  • *****
  • Offline Offline
  • Posts: 469
  • Respect: +877
    • View Profile
Re: Leaderboard change
« Reply #77 on: November 27, 2011, 05:29:41 pm »
0

Apparently. Because if you run the code as-is, you get everyone having a mean skill between 15 and 35, putting all the really good players in a narrow range of levels from about 25 to 29. And it clearly doesn't contain other changes that have been made to the isotropic leaderboard, such as the change that adds uncertainty at most once per day.

It doesn't make sense to speak of samojedi in the past tense: he played 74 games yesterday. But he's not on the official leaderboard, which means he plays logged out.

I think this highlights a problem with running a leaderboard using the CouncilRoom data: there are players who play logged-out, either because they don't want to worry about their level, or they don't want others to worry about their level, and that's a signal that's not available to us.
Logged

rrenaud

  • Administrator
  • *****
  • Offline Offline
  • Posts: 991
  • Uncivilized Barbarian of Statistics
  • Respect: +1197
    • View Profile
    • CouncilRoom
Re: Leaderboard change
« Reply #78 on: November 27, 2011, 05:42:04 pm »
0

We could take the official leaderboard and drop names that don't appear on it.

I used past tense with samojedi because I played against him like 6 months ago, but haven't seen him since.
Logged

Fabian

  • 2012 Swedish Champion
  • *
  • Offline Offline
  • Posts: 666
  • Respect: +542
    • View Profile
Re: Leaderboard change
« Reply #79 on: November 27, 2011, 05:49:41 pm »
0

Well, finding myself in second place on a leaderboard was a nice surprise!
Logged

fellowmartian

  • Swindler
  • ***
  • Offline Offline
  • Posts: 19
  • Respect: +5
    • View Profile
Re: Leaderboard change
« Reply #80 on: November 28, 2011, 09:46:58 am »
0

Hmmm. On old iso I was 35, new iso 31, new Councilroom 28...

I'm confused.
Logged

rrenaud

  • Administrator
  • *****
  • Offline Offline
  • Posts: 991
  • Uncivilized Barbarian of Statistics
  • Respect: +1197
    • View Profile
    • CouncilRoom
Re: Leaderboard change
« Reply #81 on: November 28, 2011, 10:01:27 am »
0

The many clocks problem :P.  It's why I didn't want to put up a leaderboard on cr unless it was compellingly different.

Levels are very sensitive to implementation details in the scoring system, even to the degree of picking different constants, leaving aside things like whether or not to increase players variance over time.  Ranking systems are also unconstrained with regards to scale, you could multiple every ones level/variance by 2 and the ranking system would be doing it's just as well.

Ranks (or even rank / #players for compensating for systems that filter out certain players) should be a more consistent measure across different scoring systems.  Note that multiplying everyones level by a (positive) constant doesn't change the ranking.

Further, you might even look at the variance in players ranks across different score boards to see how much players successfully gamed the rankings. 
Logged

fellowmartian

  • Swindler
  • ***
  • Offline Offline
  • Posts: 19
  • Respect: +5
    • View Profile
Re: Leaderboard change
« Reply #82 on: November 29, 2011, 02:45:42 pm »
+1

Okay, I've just seen that yaron is level 24.

That's ridiculous.

That kind of thing completely blows out of the water the idea of getting a reasonable match-up based on level. Yaron will pummel your 'average' level 24.

I'm sure everyone here is massively grateful to dougz for isotropic. This is one small thing compared to all the unbelievably fantastically good things he's done to make isotropic what it is, so I hope all the moaning about this doesn't come across as getting on his back.

But.

Dougz, are you there? Can you see all the people above with very good reasoning behind why this change is so so much for the worse? If you agree, can you change it back, please? If not, can you tell us why?
Logged

theory

  • Administrator
  • *****
  • Offline Offline
  • Posts: 3603
  • Respect: +6121
    • View Profile
    • Dominion Strategy
Re: Leaderboard change
« Reply #83 on: November 30, 2011, 12:53:58 pm »
0

I'm curious about something else:

On Nov 27, I was level 26;
on Nov 28 I was level 25;
and today, Nov 30, I am level 24.

I would have assumed this was because my old games are going away, but my # of eligible games is the same at 129, and I have not logged into Isotropic or played a game since the new leaderboard.  What gives?  Why do people's rank slip even when the number of their eligible games stays the same?
Logged

cherdano

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 166
  • Respect: +54
    • View Profile
Re: Leaderboard change
« Reply #84 on: November 30, 2011, 12:56:30 pm »
0

I really like the new rankings, since I am now 10 levels better than theory. ;D
Seriously,  there is something wrong.
Logged

toaster

  • Young Witch
  • ****
  • Offline Offline
  • Posts: 133
  • Respect: +46
    • View Profile
Re: Leaderboard change
« Reply #85 on: November 30, 2011, 01:21:56 pm »
0

It appears that dougz doesn't post here very often in general...theory or rrenaud, I presume that you've had some contact with him before, would either of you be willing to email him and see if he can provide an update on this issue?
Logged

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4381
    • View Profile
    • WanderingWinder YouTube Page
Re: Leaderboard change
« Reply #86 on: November 30, 2011, 01:34:45 pm »
+1

I'm curious about something else:

On Nov 27, I was level 26;
on Nov 28 I was level 25;
and today, Nov 30, I am level 24.

I would have assumed this was because my old games are going away, but my # of eligible games is the same at 129, and I have not logged into Isotropic or played a game since the new leaderboard.  What gives?  Why do people's rank slip even when the number of their eligible games stays the same?

If your rank is based only on the last 30 days, your 'strength of schedule' has got to be changing. Because if you count your games for the change they bear on you based on how strong your opponents were ranked when you played them, then that carries the information of how they were ranked then into your rating, which is information from before that 30 day period.
And yeah, I know that that doesn't really make sense.

olneyce

  • 2011 DS Champion
  • *
  • Offline Offline
  • Posts: 245
  • Respect: +210
    • View Profile
Re: Leaderboard change
« Reply #87 on: November 30, 2011, 01:43:02 pm »
0

If your rank is based only on the last 30 days, your 'strength of schedule' has got to be changing. Because if you count your games for the change they bear on you based on how strong your opponents were ranked when you played them, then that carries the information of how they were ranked then into your rating, which is information from before that 30 day period.
And yeah, I know that that doesn't really make sense.
I thought the system merely calculated the current ratings of people when they play, and adds or subtracts at that moment.  I didn't think it was retrospective in the way that you're describing.  Is that wrong?

As in: people who beat me like a drum when I first started playing don't get credit for beating a high-ranking player, since I was a level 0 at the time the match actually took place.   Right?
Logged

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4381
    • View Profile
    • WanderingWinder YouTube Page
Re: Leaderboard change
« Reply #88 on: November 30, 2011, 09:44:16 pm »
0

If your rank is based only on the last 30 days, your 'strength of schedule' has got to be changing. Because if you count your games for the change they bear on you based on how strong your opponents were ranked when you played them, then that carries the information of how they were ranked then into your rating, which is information from before that 30 day period.
And yeah, I know that that doesn't really make sense.
I thought the system merely calculated the current ratings of people when they play, and adds or subtracts at that moment.  I didn't think it was retrospective in the way that you're describing.  Is that wrong?

As in: people who beat me like a drum when I first started playing don't get credit for beating a high-ranking player, since I was a level 0 at the time the match actually took place.   Right?
That's definitely the way it WAS. Now... I actually have no clue what it's doing now, since it doesn't make sense to me how you can really do a pool based on 'the last 30 days', where those 30 days are constantly floating.

Kirian

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 7096
  • Shuffle iT Username: Kirian
  • An Unbalanced Equation
  • Respect: +9411
    • View Profile
Re: Leaderboard change
« Reply #89 on: November 30, 2011, 10:34:16 pm »
+4

I'm curious about something else:

On Nov 27, I was level 26;
on Nov 28 I was level 25;
and today, Nov 30, I am level 24.

I would have assumed this was because my old games are going away, but my # of eligible games is the same at 129, and I have not logged into Isotropic or played a game since the new leaderboard.  What gives?  Why do people's rank slip even when the number of their eligible games stays the same?

If your rank is based only on the last 30 days, your 'strength of schedule' has got to be changing. Because if you count your games for the change they bear on you based on how strong your opponents were ranked when you played them, then that carries the information of how they were ranked then into your rating, which is information from before that 30 day period.
And yeah, I know that that doesn't really make sense.

Yeah, that probably makes sense.  Presumably what happened is that theory played no games between 27 October and 30 October, so the number of eligible games wouldn't have changed.  But the ratings of the people against whom he played would now have also changed... and I guess would have changed at the time he had played them?

It seems like the new system is basically setting everyone to 25 +- 25, then running through the past thirty days of logs and parsing forward from there.  I trued to type the following in just a couple of sentences, and realized English doesn't have the verb tenses to deal with it, because it's basically time travel.  Let me try this way:

On 27 Nov, rankings were calculated based on 27 Oct to 26 Nov.  Theory played no games in the first four of those days, but other players did.  At the time theory played his first games, those players had ranks, presumably better than 25+-25, based on games they played between 27 Oct and 30 Oct.

On 30 Nov, rankings were calculated based on 30 Oct to 29 Nov.  The first players theory played now, instead, were all 25+-25, so theory's rank increased less for those first games.

----

In other words, if I play 100 games today and then do nothing for 30 days, during that time my ranking will regress to the mean (25+-25) based entirely on the rankings of other players shifting as the time window used to determine peoples' rankings changes.

In other other words, this is a bad time travel novel plot.  A computer program is erasing things in the past, which is causing changes in the present.  All we need is an intrepid team of time travellers (I'm imaging theory as the scientist, rrenaud as the grizzled space marine, guided as the hardened veteran, myself as the redshirt who dies in a spacetime anomaly, and of course Sam Rockwell as the Plucky Comic Relief) to destroy the computer--at some point in the past, before it starts erasing said past.  Or something like that.
Logged
Kirian's Law of f.DS jokes:  Any sufficiently unexplained joke is indistinguishable from serious conversation.

rspeer

  • Witch
  • *****
  • Offline Offline
  • Posts: 469
  • Respect: +877
    • View Profile
Re: Leaderboard change
« Reply #90 on: December 01, 2011, 06:07:39 am »
0

Kirian, that is an amazing description of what's going on.

I think I might have gotten dougz to read this thread by e-mailing him earlier today. But he sounded unconvinced. I fear I did so too early, because he needs to see your post.
Logged

ackack

  • Explorer
  • *****
  • Offline Offline
  • Posts: 302
  • Respect: +19
    • View Profile
Re: Leaderboard change
« Reply #91 on: December 01, 2011, 09:12:35 am »
+1

I'm unconvinced there's really a problem; at any rate, I'm not convinced it's nearly as bad as everybody thinks it is. A couple of thoughts:

- Everybody seems to be assuming that the leaderboard before was basically correct. Certainly it did a pretty good approximate job - top leaderboard players were legitimately good. But top players now are still legitimately good! There seems to be a substantial status quo bias here where the more recognized names are assumed to have been better, but I don't see a ton of evidence put forward to necessarily support this. It's just an assumption.

- Among other things, something to consider is that the old ratings incorporated a lot of data from a different era of play, where things like only playing Colony games was still allowed. The new 30 day approach presumably does not reflect that data at all, and thus has at least some chance of being more accurate in the given climate.

- The best argument for what's weird is that undoubtedly good players who go inactive for a while can have an unusually low level. But if it's strictly the level, how important is that? Having a leaderboard based on level as opposed to estimated skill (another way in which the initial leaderboard might be a bit misleading, although this adjustment would not have produced a huge shakeup) seems to me to lead to some weird distortions in that way. Playing somebody who has fallen down the leaderboard because their random deviation is now 15 instead of 7 does not necessarily mean that your TrueSkill will be unduly punished.

- Resetting to complete unknown status does seem wrong. Here are a few ideas to fix that main problem: ratings floors; mean skills that persist through the 30 day window, which still determines the random deviation; reverting to the old way of doing things, but only taking data from the post-Paralyzed change where restricted games do not contribute to leaderboard.
Logged

theory

  • Administrator
  • *****
  • Offline Offline
  • Posts: 3603
  • Respect: +6121
    • View Profile
    • Dominion Strategy
Re: Leaderboard change
« Reply #92 on: December 01, 2011, 09:39:01 am »
+1

- The best argument for what's weird is that undoubtedly good players who go inactive for a while can have an unusually low level. But if it's strictly the level, how important is that? Having a leaderboard based on level as opposed to estimated skill (another way in which the initial leaderboard might be a bit misleading, although this adjustment would not have produced a huge shakeup) seems to me to lead to some weird distortions in that way. Playing somebody who has fallen down the leaderboard because their random deviation is now 15 instead of 7 does not necessarily mean that your TrueSkill will be unduly punished.
I'm actually OK with the leaderboard, since it does accomplish the desired purpose of having a more transient "what's hot" board instead of an all-time thing.  (I prefer a leaderboard that's all-time, but I can accept the alternate as well.)  But it's not just level that's changed, it's rank as well.  This means that:

1) Automatching based on level (the original purpose of TrueSkill) is now kind of pointless/misleading;
2) You introduce a lot of inconsistent junk data into the system, since you have people who get underrated because they're playing against underrated players, and that in turn underrates the people that play them.
Logged

DStu

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2627
  • Respect: +1490
    • View Profile
Re: Leaderboard change
« Reply #93 on: December 01, 2011, 10:11:59 am »
0

But it's also that the system thinks it is much more sure about the skill. Here the variances from before and now:

Axis: x:number of games played; y variance in TS.

Note that the new ones are almost determined by the number of games you have played, they are all near c/x^2.
Logged

Reyk

  • Young Witch
  • ****
  • Offline Offline
  • Posts: 146
  • Respect: +24
    • View Profile
Re: Leaderboard change
« Reply #94 on: December 01, 2011, 10:32:52 am »
0

@DStu: Would you mind to explain your post to Joe Public? ;-)
Logged

DStu

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2627
  • Respect: +1490
    • View Profile
Re: Leaderboard change
« Reply #95 on: December 01, 2011, 10:40:45 am »
0

@DStu: Would you mind to explain your post to Joe Public? ;-)

No.
I took the leaderboard from the two dates. Took the number of played games (x-axis) and the Variances in the skill (y-axis) and plotted them.

You see that now, the variance is lower than before by the same number of played games. Which per se is not surprising, as most of the games are cut of now. But even if you consider this, the guys with ~>500 games this month have smaller variances now than everybody on the old leaderboard. So somehow I have the impression that some more parameters in the TS changed, causing it to get confident faster than before.

The second thing is that this confidence does not depend on anything anymore but the number of games. It is more or less on the line 1/(games)^(1/2), so the variance decreases with the squareroot of the games. More or less independent of how you play, if you lose/win most of your games or if you win half of them. Before, that was defintely not the case.

Edit: Just the new variances:


Edit2: And it should not be the case in TS that the variance just decreases by the number of games, because other parameters also enter: What was the skill diference to your opponents, what was their variance, etc...
« Last Edit: December 01, 2011, 10:46:00 am by DStu »
Logged

rod-

  • Conspirator
  • ****
  • Offline Offline
  • Posts: 213
  • Respect: +49
    • View Profile
Re: Leaderboard change
« Reply #96 on: December 01, 2011, 10:44:50 am »
0

Before:  Trueskill variance (the number subtracted from your calculated skill in order to generate your level) was loosely correlated with the number of games you played, particularly if you had played less than 500 games.

Now: Trueskill variance is almost exactly correlated with the number of games you have played, no matter how many games you have played. 

The difference here is of course because before, you could accumulate 1000 games in many different ways: either by playing 100 games on 10 individual days or 1 game on 1000 individual days.  The way the system was set up, the person who played 100*10 would have a much lower variance than the person who played 1*1000, because your variance would not increase if you did not play on any given day.  Now, you have 30 days.  Full-stop.  If you don't play on 1 of those 30 days, your overall variance will still change.

I sort of like the change, because I no longer feel obligated to "only" play when i have time to play 7+ games.

 That does not mean i endorse the new variances being so much "lower" than the old ones.
« Last Edit: December 01, 2011, 10:48:35 am by rod- »
Logged

DStu

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2627
  • Respect: +1490
    • View Profile
Re: Leaderboard change
« Reply #97 on: December 01, 2011, 10:55:23 am »
0

I sort of like the change, because I no longer feel obligated to "only" play when i have time to play 7+ games.  I can't hypothesize any logical reason why the variance should not be a straight function of the number of games played.

The reason is that Trueskill's variance is not only a function of the number of games:
Quote
The uncertainty of both players (regardless of win/loss/draw) is going to decrease by the factor 1-σ2/c2 * w((μ_winner-μ_loser)/c,ε/c). Again, the player with the larger uncertainty gets the bigger decrease.
http://research.microsoft.com/en-us/projects/trueskill/details.aspx  (also for what the functions mean)

If you play win against an opponent that is ranked (much*) lower than you, your uncertainity should not decrease. Same if you lose against somehow that is (much*) higher than you. [* what much means depends on the parameters].
What is happening now I think is the following. If at date now-30days, everybody is 25+-25, than for a long part of the intervall we consider, nobody is "much" higher than anybody else, because everybody is 25. So no matter if you win against someone who is 5 or 45 in the end, at the beginning he is 25, you are 25 and the uncertainity decreases the same, no matter against whom you win or lose.
Logged

Reyk

  • Young Witch
  • ****
  • Offline Offline
  • Posts: 146
  • Respect: +24
    • View Profile
Re: Leaderboard change
« Reply #98 on: December 01, 2011, 11:32:01 am »
0

I sort of like the change, because I no longer feel obligated to "only" play when i have time to play 7+ games.

It ist true that you could lose a level under the old system by playing one game per day and winning it.
But now it's much much worse. You simply can't use automatch +/- rank as theory already stated.
Logged

rod-

  • Conspirator
  • ****
  • Offline Offline
  • Posts: 213
  • Respect: +49
    • View Profile
Re: Leaderboard change
« Reply #99 on: December 01, 2011, 01:01:15 pm »
0

I suppose it's easy for me to gloss over that "loss", as I've never used automatch +/- rank.  I'd rather get a fast match against a level 7 than wait for a level 37.  However, I will also note that under the current system, I can't honestly tell a difference in the level of play between the 7 and the 37.
Logged
Pages: 1 2 3 [4] 5  All
 

Page created in 0.057 seconds with 21 queries.