Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Razzishi]

Okay, but not skill levels aren't fixed, so we have to do something else. The Glicko solution is basically to increase the variance of the prior on each player over time. This naturally decays the impact of older games. This is the "gamma" that rspeer mentioned, as far as I can tell. Now applying this solution, but only on days you play, has a totally counter-intuitive effect on rankings. Consider two guys A and B, who on day 0 have identical mu/sigma rankings. Then A goes to study for the bar exam, while B plays a game a day for the next two months, during which time his results are right in line with his previous ranking. This system will claim, obviously implausibly, that we are MORE uncertain about B's ranking than A's, which doesn't make any sense at all.

Yeah, when I read about the change to the variation and the fact that it's only applied on days that you play I was absolutely shocked; the base TrueSkill system as I had read about it updates a player's variance every rating period based on their activity that rating period wherein not playing a match would cause a certain amount of increase in a player's calculated variance (and playing at their level would cause a decrease, playing about or below their level cause an increase).  This system automatically adjusts to a player's skill change over time (assuming the rating periods aren't too long) by increasing a player's variance when their results conflict with their level so that they change level quickly and after they reach their current skill level their variance goes back down.  Such a system shouldn't remember any of the matches of the past rating periods (instead only remembering players' means and variances) thus making the idea of forgetting a player's matches after 30 days be nonsensical - it was only remembering them very vaguely to begin with.

[Anon79]

In e-mail, DougZ challenged me for "evidence" that the new levels were not predictive, before going silent again. So here's my claim. In future games of the Dominion Strategy tournament, I claim that my leaderboard will outperform the tournament seeds (based on the old leaderboard) in predicting who wins, and that the new Isotropic leaderboard will underperform.

I've saved a copy of the Isotropic leaderboard as of today, and we'll use my leaderboard as of the start of the tournament. I don't have Doug's leaderboard from the start of the tournament, so a fair comparison will probably have to only include the games played from today onward. This still gives a tiny advantage to Doug in the case that recent ratings are really important, because it is more recent by about five days.

All predictions will be based on the skill floor (the level, without rounding) instead of the mean skill. Mean skill is more mathematically predictive, but there are solid gameplay reasons to show the skill floor in the interface and on the leaderboard, and this is meant to be a test of which system really belongs in the interface and on the leaderboard.

The data I would suggest you use:

1) The stuff in the spreadsheet (the skill floors used for seeding) -- this is OLD leaderboard
2) This link, which is the NEW leaderboard the day the tournament began: http://bggdl.square7.ch/leaderboard/leaderboard-2011-11-28.html
3) Whatever data you want to come up with

FWIW, the seeds have been pretty stable so far.

As an indicator for predictive power, wouldn't it be more definitive to use all eligible games played on a single arbitrary but pre-specified day in the near future, as the sample? So using both leaderboards as at 7th Dec say, see which has the better predictive ability for 8th Dec.

Also, the degree of confidence matters as well, not just the eventual winner. For instance, taking all the games for which a particular ranking system predicts a winning probabilty for one side of 70-75%, does the actual win percentage for the favoured side actually fall between 70-75%?

[chwhite]

I just logged in, and it appears that the old leaderboard is back?  The FAQ doesn't seem to indicate it, but the higher levels and # of games eligible sure do.

Huzzah!

[Fabian]

Sweeeet!

Particularly sweet as I'm in first place heh. #brag

[Lekkit]

I just noticed this as well. Not sure if I'm really happy about it.

[olneyce]

Great!

I'm particularly happy to no longer be in the top 10, which was frankly pretty embarrassing consider how routinely I get beaten by the actual best players.

[grep]

Looks like a "Buy a goat" story. Do something definitely wrong, wait for a while, and revert the things back. And people will be happy!

[toaster]

It's definitely not the original formula...at least, not unless I suddenly improved to rank ~100 after staying mostly flat for the past few months.  That's not necessarily a bad thing, of course...whatever the new formula is, it has a much longer history and at least is an improvement over what we've had for the past couple of weeks.

[hobo386]

I was wondering why my rank suddenly dropped from 29 to 21 after being on a winning streak against players my level...
Honestly, I'm not a huge fan of this.  I usually use Automatch to try and find opponents who are about as good as me, and This past week I've had some of the easiest games in a long time.  Apparently the people who just got to rank 17-20 are only about as good as they used to be.  Now I'm at rank 30, which is pretty close to my actual skill level, but with my playing habits, that will probably fall off pretty quick again.

[Jeebus]

Before the change to last 30 days, I had around 1200 games played. Now, after the new change, I have 644. So it's definitely not changed back to the way it was.

[rspeer]

I guess all we can do for now is speculate wildly, so: He might have taken the suggestion to only count games back to the last rule change (where games with card restrictions are unrated).

Whatever it is, I'm really happy to see the return of a leaderboard that makes sense!

[theory]

I think that's likely.  I've lost thousands of games on my rating because I routinely played with !black market and !alchemy before the advent of veto.

[Thisisnotasmile]

I haven't lost a noticable number of games from my record, maybe 100 or so maximum and I've played completely random sets since I started playing. The few games that I have lost probably just account for games that I quickly accepted before noticing my opponent had set it up.

[mith]

Seems like he's changed something with the uncertainty; mine is up about 2 from the pre-change leaderboard, after not playing much in the past week. (Maybe something similar to rspeer's 1% per day method.)

[ackack]

Yeah, it definitely seems a little different.

[painted_cow]

Sweet, I lost a level while not playing and got the like biggest uncertainty around. But better than the last experiment!

[HiveMindEmulator]

I guess all we can do for now is speculate wildly, so: He might have taken the suggestion to only count games back to the last rule change (where games with card restrictions are unrated).

...or include all games to the beginning of time, but exclude games with card restrictions.

[ackack]

Now that it's switched back to a time-based uncertainty growth, the last change I'd like to see made is basing level on skill alone rather than skill - 3*SD. I don't really understand the appeal of the latter system from any perspective. The only possible argument I see is for matchmaking purposes for people who are quite new and might have an unduly inflated skill level. But that seems like a very small fraction of all possible games. Elsewhere, matchmaking would likely proceed better if it were based on skill alone. And in terms of ego boosting on the leaderboard, it's never been clear to me why playing lots of games (or now, playing every day) is a major thing that we want to highlight. (added: if you just want to make sure that there aren't fluky people at the top, give everybody a rating but only have the leaderboard display those with uncertainties beneath a certain threshold. This would be like provisional or estimated ratings.)

All that said, I do think this approach is basically correct.

[painted_cow]

I would second the ideas of ackack! I mean, some players have skill+-6, others have +-13 and are like same level.

[rrenaud]

TL:dr;  variance penalty encourages people to play more and mitigates luck and duck (out) strategies.

Imagine two players are the same high skill, and their goal is to optimize their placement on the leaderboard.

Without variance deduction, you wait til you hit a lucky streak, and then you stop playing with your artificially high level to prevent mean reversion.

With variance deduction, this is less likely to be an effective strategy.  Assuming you are truly level 40,  45 +- 10 = 35, your mirror copy of your self could play a lot to hit 40 +- 4 = 36, and pass you on the leaderboard.

Further, as you selectively pick out the highest mean players, you are likely looking at the results of lucky streaks that temporarily boost players over the top of the others.  Subtracting out the variance counters this effect somewhat between top players with few games (== more likely to be blessed by luck), than top players with many.

[greatexpectations]

rrenaud said a lot of what i wanted to, but i figured id throw in a couple more things.

- with the big caveat that this how i understand how the system works...it currently starts out a new player as ranking lvl 25 +/- 25.  any new players would then start off as being level 25.  this would require you to either A. completely refigure the formulas involved or B.  use the current method when calculating a players level for a players level and then use this adjustment after the fact. otherwise not only could you easily game the system but all pretenses of accuracy would get thrown out the window.
- you still need to play games to grow levels anyway. you need either an absurd win rate or at least a couple hundred games if you want to hit 40+ by your method. (and even then, all levels would be inflated quite a bit) reducing your uncertainty by playing more doesn't do much for you unless you actually win those games.  this system seems to punish those who play a lot just as much as it might help those who play less.
- if anyone is just interested in studying how this would affect the leaderboard for science, you can find out with about 5 minutes of work in excel.  you programmers could probably throw it online fairly easily too. 

and of course, i cant help but notice that painted cow would agree with a system that would jump him to the top spot .

[cherdano]

What I don't understand: Why are we discussing this without any data? For anyone with access to the data, it should be trivial (if a bit of work) to see which leaderboard was better at predicting winners, or winning percentages. In fact, I would suggest that the normal way of doing things would be to run such tests before making a drastic change to the rating.
Btw, the one tweak I would try is playing with the initial ranking of new players. One could try replacing this with a setting based on empirical data (e.g., if new players on average have a ranking of 20 after their first 40 games, then probably an initial ranking of 20 is a better guess than an initial ranking of 25).
Anyway, I also understand DougZ's frustration with tweaks - changes to a rating system often will end up behaving differently than you could have anticipated.

[rrenaud]

http://councilroom.com/static/margin.txt.gz

Now you have the same data that everyone else does ;P.  (I can refresh that so it's got the last n-months if anyone cares).

The means by themselves aren't meaningful.  If you start players at 20 rather than 25, everyone just loses 5 levels, and nothing else changes.

[ackack]

Without variance deduction, you wait til you hit a lucky streak, and then you stop playing with your artificially high level to prevent mean reversion.

With the kinds of variance that we're talking about, I'm pretty sure this is quite unlikely. Especially since levels are based on 3*the SD, the actual spread in ratings for active players is going to be quite small. Have the leaderboard only display accounts with 3*SD < 11 or so with the present parameters and I think things would be just fine.

Further, as you selectively pick out the highest mean players, you are likely looking at the results of lucky streaks that temporarily boost players over the top of the others.

Even if the top 100 all played 10 games a day, there would still be people riding lucky streaks up and down. Given that people would be close in ratings, I don't think the variance would be strongly coupled to their performance. Thus I don't see how this really helps with that.

and of course, i cant help but notice that painted cow would agree with a system that would jump him to the top spot

That seems fair, seeing as how the rating system considers him the strongest player there is (and given my experience, I find that very plausible - I'd back him over any of the rest of the top 10 in a match.)

added: Even a "sort by mean skill" on the leaderboard - keeping the levels and all as what shows up on Iso - would be cool.

[greatexpectations]

added: Even a "sort by mean skill" on the leaderboard - keeping the levels and all as what shows up on Iso - would be cool.

sortable leaderboards is a great idea. that way you could easily sift through players by level, skill, variance, or games played. it would also allow some flexibility in adding something like the 30 day leaderboard doug tried implementing.