Dominion Strategy Forum
Dominion => Dominion Online at Shuffle iT => Dominion General Discussion => Dominion Isotropic => Topic started by: theory on December 01, 2011, 04:10:09 pm
-
To get some quantitative data about people's feelings on the issue. See previous discussion here (http://forum.dominionstrategy.com/index.php?topic=1039.0).
-
I don't like the 30 day look-back. Seems to "Swingy" and confusing on how/why rating change from day to day without playing. And the fact that good players will start at a noob level if they haven't played for 30 days would make the data useless to me.
If it were based on say a 300 game look back or something like that, I'd be fine with it.
-
If it were based on say a 300 game look back or something like that, I'd be fine with it.
i am not agreeing or disagreeing with your thought, i just want to point out that for some players this will not change much.
here are the average games played for various groups over the past month, according to today's leaderboard.
all players - 84.4
top 5000 players (lvl 1+) - 109.8
top 4000 players (lvl 5+) - 128.4
top 3000 players (lvl 9+) - 150.9
top 2000 players (lvl 14+) - 177.2
top 1000 players (lvl 19+) - 213.4
top 500 players (lvl 23+) - 237.9
top 100 players (lvl 31+) - 330.2
top 50 players (lvl 33+) - 374.4
top 10 players (lvl 38+) - 398.2
-
Personally, the best suggestion I've seen that works in the spirit of what Doug wanted with the new board is just to place t = 0 at the time when the system switched so that only random boards counted. Which is basically "put it back" with a minor change if something must be changed.
-
It should PROBABLY be going back like two changes (before the per-day uncertainty adjustment, back to the per-game), and probably levels should be based on skill alone. And probably the exact parameters need some tweaking, but I don't know that much of the specific math here.
-
If the purpose of the leaderboard is to provide a crude tool for figuring out how to play players who are roughly at your skill level, it would be better to just have five levels or something. If the idea is to provide a more serious estimate of playing skill, it seems that all the parameters are more or less misconfigured.
1) Having a step function that is either time-based or duration-based whereby games drop off the ranking is bad; using decay or something Bayesian instead is better.
2) The skill variance calculations from the old board weren't good; they were too close for players of widely different volumes of play.
3) The "level" system which is some linear combination of estimated skill and variance is kind of arbitrary and misleading.
4) The assumed skill for new players is probably wrong; most unranked players are kind of bad. This is primarily a variance issue though, winning or losing against players with hardly any record shouldn't do much to rankings. That's not happening, though.
5) People being able to "game" the system by playing sock puppets is really quite avoidable; just restrict the information that games against any particular opponent can add to the system.
I don't know a lot about the guts TrueSkill, but it can hardly be rocket science; I'd be happy to do some research and help in tweaking parameters to make a system that reflects the goals of the leaderboard, whatever those are. (that last question is really quite important, btw).
-
I don't like it at all.
I wouldn't mind having my oldest games ignored in my rank (just because they wouldn't represent my skill level anymore, and if I truly wanted to I could just start anew with another email address to erase those), but it seems that putting a TIME constraint on it is unreasonable, especially since we all have busy lives and cannot always keep up with playing all the time. It puts a certain aspect of "productivity" in leisure, which defeats the purpose of leisure if you ask me.
Bottom line I HATE it. Wish it was rather a past 500-games condition rather than past 30 days. And while I don't necessarily advocate eradicating the past 30 days leaderboard completely, I strongly suggest it being a supplement of info rather than the main leaderboard, where there could be tabs with different classifications and stats (like leaderboard by last 500 games, last 30 days, all time).
Hope our voices will be heard and something will be done as this is slightly annoying and makes me enjoy isotropic a little less. Though people say the socially desirable "i don't care for ranks I play for fun", I had fun looking at my progression as well as having fun playing.
-
I wouldn't mind having my oldest games ignored in my rank (just because they wouldn't represent my skill level anymore...
But the point is that they represented your old skill level, which is part of your record. Games should never get lopped off nor someone's rating "restarted." Gary Kasparov's FIDE rating includes games from when he was in high school, I'm certain. That's part of what makes this quite maddening; the whole point of a system like this is that, yes, it takes your old games into account. It's supposed to.
-
I wouldn't mind having my oldest games ignored in my rank (just because they wouldn't represent my skill level anymore...
But the point is that they represented your old skill level, which is part of your record. Games should never get lopped off nor someone's rating "restarted." Gary Kasparov's FIDE rating includes games from when he was in high school, I'm certain. That's part of what makes this quite maddening; the whole point of a system like this is that, yes, it takes your old games into account. It's supposed to.
I very much support this. It wouldn't help at all to make a 60 day frame instead of 30 days or a 500 game frame. The Paralyzed issue remains then, but again: sort him out manually.
-
That's part of what makes this quite maddening; the whole point of a system like this is that, yes, it takes your old games into account. It's supposed to.
To what end? How much information about how he plays now is conveyed by those old games?
I liked hgfalling's post a lot. His closing point about deciding what you want the leaderboard to actually do is apropos here. If it's supposed to be a lifetime achievement record, then yeah, including all of the games is definitely important. But I'm skeptical that the first 100 registered games I played on Isotropic - the games where I knew essentially nothing about the game - have any predictive power about how well I'm going to do right now. If we are trying to make a measure of 2p skill at the moment (and I'm of the mind that the multi-player adaptability feature of TrueSkill is actually kind of a bug given how most of us use the leaderboard), then getting rid of probably irrelevant data from 1500 games ago seems good. I'd agree again with hgfalling that some sort of exponential windowing or something like that would make more sense, but the general idea of discounting ancient games makes sense.
-
Gary Kasparov's FIDE rating includes games from when he was in high school, I'm certain.
It does, but I think he came on the list with a >2600 rating, so it doesn't make much difference in his case.
(Not that it would anyway: the effect of a game thousands of games back on your current rating is effectively nil.)
-
... 1) Having a step function...
I didn't expect to read about step functions today. Thanks!
-
This is quite random, but a previous post got me thinking...could we have two numbers next to our name? One for skill and one for games played? Right now, a person with many games played but zero skill is ranked equally as someone who is very good but only has played a limited number of games. Again this is me spitballing, but what if we had skill on one axis and games played on another? If a person's two entries were plotted as a point, we could better auto-match by taking the distance between their respective plotted points than by the current TrueSkill system.
Hopefully this makes some sense at all.
-
Here is an anecdote: I have an all time record of 351-228-8, and was rank 27 before the change. I have not been playing a whole lot lately, mainly due to the Thanksgiving holiday and not wishing to excessively play Hinterlands cards before I got the boxed copy. I logged in yesterday and was shocked to see my rank at 12(has since dropped to 10, despite winning the only game I played in that period). That puts me at about the 2800th best player on the site. I can't tell you what it was before the change, but assuming the leaderboard distribution is representative of the previous distribution, that would have put me at roughly the 220th best player. Considering there are 7000 total players on the leaderboard, this moved me from the 97th percentile to the 60th percentile. That just seems broken.
If your goal is to build a stable database of player skill, this is the wrong way to do it, for so many reasons.
Players who play infrequently are penalized, since they won't ever have enough history for them to build their trueskill estimate and drive down the uncertainty. You will need a certain number of matches to do so, and if you can't hit that number in the 30 day window, you will never able to improve your rating beyond a specific number, which is short of your actual rating.
You also penalize their opponents. If I get automatched against a level 10, and he wins, he only gets credit for beating a level 10, not a level 27. Likewise, if I win, he will lose more points than he should.
It reduces my incentive to play because the rating system is unreliable. Because there will be a lot of uncertainty with overlevelled and underlevelled opponents, it's going to be frustrating to try to raise my rank.
Basically, it increases the amount of games that players need to play to have a reliable rating. But if it's going to be that much work, I'm just going to be playing Skyrim or Starcraft 2 instead :/
If you want to reward or encourage frequent play, I would highly recommend either using a ladder system like Starcraft 2, which is separate from the skill rank, or increase the uncertainty of players who haven't played in a while. You can also only count auto-matched veto mode games to reduce the effect of players cherry picking their opponents.
Edit: I wanted to say this - I appreciate all the work that goes into isotropic, and please take this as an explanation of my thought process about why I would choose to play or not play, rather than as a complaint about an overall awesome service!
-
...the general idea of discounting ancient games makes sense.
You should not discount ancient games because your TrueSkill was calculated using your skill at the time(as well as that of your opponent). You are reducing the amount of information used as input to a system that has been built to take advantage of as much information as possible. Even if you totally bomb your first 100 games, TrueSkill will rapidly shoot up your rating if you outperform your expected rating. Throwing out those first games doesn't make the system better, and will not give you a more accurate representation of your rating. If you are familiar with topcoder check out my rating history(same username) as an example of outperforming a low initial rating :)
-
1. Keep the leaderboard as it is now. Maybe change it so that the last 60 or 90 days are the only ones that matter. I don't see how this is any problem besides messing with peoples levels; which leads me to my second note.
2. Keep the levels separated from the leaderboard. This way you can see that Theory (just an example) is a good player; since he has a high level, although he's not ranked one of the best at the moment on the leaderboard.
These are just my thoughts, and I think that it would solve, what seems to me as, the only problem people see with this change. The levels don't always represent your skill anymore.
-
...the general idea of discounting ancient games makes sense.
You should not discount ancient games because your TrueSkill was calculated using your skill at the time(as well as that of your opponent). You are reducing the amount of information used as input to a system that has been built to take advantage of as much information as possible. Even if you totally bomb your first 100 games, TrueSkill will rapidly shoot up your rating if you outperform your expected rating. Throwing out those first games doesn't make the system better, and will not give you a more accurate representation of your rating. If you are familiar with topcoder check out my rating history(same username) as an example of outperforming a low initial rating :)
Well, that's not quite true. The problem DougZ was trying to fix was that people who start out playing badly and then get better could not dig themselves out of the hole fast enough. TrueSkill makes use of all the evidence, including possibly the earlier evidence that you suck, and the only thing that makes new evidence count more than old evidence is a factor called "gamma" that increases your standard deviation (sigma) between games.
DougZ set gamma to be applied once per day you play. In that system, if you want to erase your past mistakes, the best way to do so is to play one or a few games per day in preference to games all at once. (The same reason someone like Bonogi would want to play in massive binges, to not erase his past success.)
Now he's apparently frustrated with tweaking gamma, and is dealing with it more drastically by erasing the past entirely. Whereas I'd prefer a solution that applied gamma once per day no matter what (which is what I put on the CouncilRoom experimental leaderboard).
In e-mail, DougZ challenged me for "evidence" that the new levels were not predictive, before going silent again. So here's my claim. In future games of the Dominion Strategy tournament, I claim that my leaderboard will outperform the tournament seeds (based on the old leaderboard) in predicting who wins, and that the new Isotropic leaderboard will underperform.
I've saved a copy of the Isotropic leaderboard as of today, and we'll use my leaderboard as of the start of the tournament. I don't have Doug's leaderboard from the start of the tournament, so a fair comparison will probably have to only include the games played from today onward. This still gives a tiny advantage to Doug in the case that recent ratings are really important, because it is more recent by about five days.
All predictions will be based on the skill floor (the level, without rounding) instead of the mean skill. Mean skill is more mathematically predictive, but there are solid gameplay reasons to show the skill floor in the interface and on the leaderboard, and this is meant to be a test of which system really belongs in the interface and on the leaderboard.
If Doug decides to change the leaderboard again in the meantime, that's fine -- and if he improves it enough that it makes this comparison irrelevant, that would be a great thing.
-
If we are trying to make a measure of 2p skill at the moment (and I'm of the mind that the multi-player adaptability feature of TrueSkill is actually kind of a bug given how most of us use he leaderboard), then getting rid of probably irrelevant data from 1500 games ago seems good. I'd agree again with hgfalling that some sort of exponential windowing or something like that would make more sense, but the general idea of discounting ancient games makes sense.
The point is, TS alone is getting rid of the irrelevant data already. I can understand if Dougz is tired of tweaking parameters, and I also see that the old (already modified) implementation has some problems (also the unmodified seem to have more serious problems), namely that you don't want to play only 1 or 2 games per day because of variance issues.
But we should not think that what TS weigthes the games from 1 year ago with the same weight with the games of yesterday. All other thing being equal, if I am level 15 and win a couple of games against someone being level 30 and loose one, this will increase my ranking. If I'm level 45, this might even decrease it. So bad performances in the past vanishes. Also, the updates only depends on my mean and my variance (and the once of my opponent), and not of how many games I have played. And variance was greatly independent of the number of games before.
I could also, with not very many games, drop my rating from 30 in the lower 20 and pushing it up to the mid30s again. Don't know how it's at the real top, but I also saw Geronimoo and WW going up to the top form the 30s in quite a fast time. I don't really see that you are stuck with your rating forever...
-
I wouldn't mind having my oldest games ignored in my rank (just because they wouldn't represent my skill level anymore...
But the point is that they represented your old skill level, which is part of your record. Games should never get lopped off nor someone's rating "restarted." Gary Kasparov's FIDE rating includes games from when he was in high school, I'm certain. That's part of what makes this quite maddening; the whole point of a system like this is that, yes, it takes your old games into account. It's supposed to.
Neither ELO (which FIDE use) nor TrueSkill (which is based on ELO) remember your old games beyond the fact that your rating is based upon them. Your ELO rating is a single number, and TrueSkill's is just a rating and a variance where you're ranked by (rating - variance).
Now, I think it's true that if, say, your skill suddenly increased from 20 to 25, compared to some new player (rated 0) who's also really skill 25, the new player will get to 25 first (and you'd get to 25 faster with TrueSkill than ELO). But you will both get there; nothing permanently penalises you for being worse in the past.
-
In e-mail, DougZ challenged me for "evidence" that the new levels were not predictive, before going silent again. So here's my claim. In future games of the Dominion Strategy tournament, I claim that my leaderboard will outperform the tournament seeds (based on the old leaderboard) in predicting who wins, and that the new Isotropic leaderboard will underperform.
I've saved a copy of the Isotropic leaderboard as of today, and we'll use my leaderboard as of the start of the tournament. I don't have Doug's leaderboard from the start of the tournament, so a fair comparison will probably have to only include the games played from today onward. This still gives a tiny advantage to Doug in the case that recent ratings are really important, because it is more recent by about five days.
All predictions will be based on the skill floor (the level, without rounding) instead of the mean skill. Mean skill is more mathematically predictive, but there are solid gameplay reasons to show the skill floor in the interface and on the leaderboard, and this is meant to be a test of which system really belongs in the interface and on the leaderboard.
The data I would suggest you use:
1) The stuff in the spreadsheet (the skill floors used for seeding) -- this is OLD leaderboard
2) This link, which is the NEW leaderboard the day the tournament began: http://bggdl.square7.ch/leaderboard/leaderboard-2011-11-28.html
3) Whatever data you want to come up with
FWIW, the seeds have been pretty stable so far.
-
Interesting, as I personally find it much harder to raise my level nowadays, in spite that I have been 10 level-ish lower on today's leaderboard. Before the change, I can just suck and see my rankings sank from 42 to 35 and then climb back, just by playing random opponents. In this sense I don't see the "gamma problem" at all. However, recently even if I win more than 70% of the games (which is my life time average against random opponents) my rank hardly changes. And if I am being experimental, it seems my rank will sink. It seems that I have to conclude the way I play is ranked quite differently under the two systems: in the previous leaderboard I am 40-ish and in the present day one I am 30-ish.
-
There was substantial deflation moving between the two boards, so I wouldn't take "30ish" as much of a sign. What I'd look at is your rank. My rank appears to be quite uniform across the three leaderboards.
-
In e-mail, DougZ challenged me for "evidence" that the new levels were not predictive, before going silent again. So here's my claim. In future games of the Dominion Strategy tournament, I claim that my leaderboard will outperform the tournament seeds (based on the old leaderboard) in predicting who wins, and that the new Isotropic leaderboard will underperform.
You can compute the log likelihood of the predictions for both systems. I am pretty sure the old way (without throwing out the data) will outperform.
IIRC, summing up the log of diff_p0p1.ProbabilityPositive() after throwing out ties for 2p games will calculate the log likelihood in the implementation I have checked in.
https://github.com/rrenaud/dominionstats/blob/master/trueskill/trueskill.py#L442
Even simpler, you could also just calculate the number of times the highest rated player wins.
Edit: Of course, this requires access to doug's implementation to be super fair :(. OTOH, I guess you could run two systems with the same parameters, varying only the 'throw away history' behavior, and that will get to the heart of the issue.
-
In e-mail, DougZ challenged me for "evidence" that the new levels were not predictive, before going silent again.
I'm not sure more "evidence" is required than:
1. The leaderboard is used to give a sense of how "balanced" a match is, that is, whether you are playing someone similar in skill or vastly different. This is demonstrated by the fact that the system from which the leaderboard is derived uses it to this effect.
2. There are currently people on the leaderboard whose level is much, much lower than previously, because they took breaks. (Separately, one's level will regress to zero as one spends less time playing.) This has been documented multiple times.
3. The problem in (2) will happen any time a player takes a ~10-day break. This can be demonstrated mathematically.
4. Anyone playing one of the people whose ranking is far lower than it ought to be will have their level/ranking decreased (relatively) by playing one of these people with a deflated rank. This can be demonstrated mathematically.
I'm not sure how anyone can see these pieces of evidence and conclude that the new system is better. I'm talking here only about the matchmaking aspect.
-
Re: predictive value of "level".
Level (defined as meanskill - sigma) isn't supposed to be a predictive measure anyway, so why would it be predictive? It's just notational shorthand. I mean, a guy whose estimated rank is 30+/-20 is a lot different than a guy whose estimated rank is 15+/-5, but "level" treats them the same. Of course it's not going to be as good a predictor as, say, mean skill.
Re: arbitrary cutoffs
There is this notion of "consistency" in estimators: you have some parameter t you want to predict, and you have some set of n observations that you want to predict it from. You generally want the following: as n increases, your estimate gets closer to t. Dropping relatively recent observations from the past from consideration guarantees that your estimators will not be consistent. This is a pretty bad thing, considering that you don't really get any offsetting benefit from it.
Re: decaying the past
There are two things going on here, which it seems some posters are missing. Suppose first that skill levels are fixed, like people never improve or get worse, and our goal is correctly assess everyone's skill level in an asymptotically consistent sense. Then we should not drop anything from the past, and all observations should be equally weighted. Then the probably is basically simple except for what prior beliefs we have about the population of isotropic players.
Okay, but not skill levels aren't fixed, so we have to do something else. The Glicko solution is basically to increase the variance of the prior on each player over time. This naturally decays the impact of older games. This is the "gamma" that rspeer mentioned, as far as I can tell. Now applying this solution, but only on days you play, has a totally counter-intuitive effect on rankings. Consider two guys A and B, who on day 0 have identical mu/sigma rankings. Then A goes to study for the bar exam, while B plays a game a day for the next two months, during which time his results are right in line with his previous ranking. This system will claim, obviously implausibly, that we are MORE uncertain about B's ranking than A's, which doesn't make any sense at all.
Now, rspeer mentioned a problem about players playing badly at first and not being able to dig themselves out of the hole fast enough. To my mind, this isn't a problem: our best estimate of their level is what it is under the parameters of the model, so meh. But I think his comment reflects a prior belief about the distribution of skill levels that is a) not accounted for in the model, b) probably true. That belief is that the rate of change of "true" skill levels is much higher for players with low rankings than it is for players with high rankings. To me this is obviously true; when you suck, it's easy to become marginally competent, just read dominionstrategy.com. When you are mediocre, it is harder but not impossible to become strong. When you are strong, it is difficult to become elite OR to become mediocre. When you are elite, it's hard to move anywhere. The higher your meanskill ranking is, the lower the variance of the drift of your meanskill, regardless of the variance of your meanskill.
So if this is really the problem you are trying to solve with all this tweaking of the system, then just use non-uniform gamma based on meanskill. Problem solved, and in a nice Bayesian way.
-
Okay, but not skill levels aren't fixed, so we have to do something else. The Glicko solution is basically to increase the variance of the prior on each player over time. This naturally decays the impact of older games. This is the "gamma" that rspeer mentioned, as far as I can tell. Now applying this solution, but only on days you play, has a totally counter-intuitive effect on rankings. Consider two guys A and B, who on day 0 have identical mu/sigma rankings. Then A goes to study for the bar exam, while B plays a game a day for the next two months, during which time his results are right in line with his previous ranking. This system will claim, obviously implausibly, that we are MORE uncertain about B's ranking than A's, which doesn't make any sense at all.
Yeah, when I read about the change to the variation and the fact that it's only applied on days that you play I was absolutely shocked; the base TrueSkill system as I had read about it updates a player's variance every rating period based on their activity that rating period wherein not playing a match would cause a certain amount of increase in a player's calculated variance (and playing at their level would cause a decrease, playing about or below their level cause an increase). This system automatically adjusts to a player's skill change over time (assuming the rating periods aren't too long) by increasing a player's variance when their results conflict with their level so that they change level quickly and after they reach their current skill level their variance goes back down. Such a system shouldn't remember any of the matches of the past rating periods (instead only remembering players' means and variances) thus making the idea of forgetting a player's matches after 30 days be nonsensical - it was only remembering them very vaguely to begin with.
-
In e-mail, DougZ challenged me for "evidence" that the new levels were not predictive, before going silent again. So here's my claim. In future games of the Dominion Strategy tournament, I claim that my leaderboard will outperform the tournament seeds (based on the old leaderboard) in predicting who wins, and that the new Isotropic leaderboard will underperform.
I've saved a copy of the Isotropic leaderboard as of today, and we'll use my leaderboard as of the start of the tournament. I don't have Doug's leaderboard from the start of the tournament, so a fair comparison will probably have to only include the games played from today onward. This still gives a tiny advantage to Doug in the case that recent ratings are really important, because it is more recent by about five days.
All predictions will be based on the skill floor (the level, without rounding) instead of the mean skill. Mean skill is more mathematically predictive, but there are solid gameplay reasons to show the skill floor in the interface and on the leaderboard, and this is meant to be a test of which system really belongs in the interface and on the leaderboard.
The data I would suggest you use:
1) The stuff in the spreadsheet (the skill floors used for seeding) -- this is OLD leaderboard
2) This link, which is the NEW leaderboard the day the tournament began: http://bggdl.square7.ch/leaderboard/leaderboard-2011-11-28.html
3) Whatever data you want to come up with
FWIW, the seeds have been pretty stable so far.
As an indicator for predictive power, wouldn't it be more definitive to use all eligible games played on a single arbitrary but pre-specified day in the near future, as the sample? So using both leaderboards as at 7th Dec say, see which has the better predictive ability for 8th Dec.
Also, the degree of confidence matters as well, not just the eventual winner. For instance, taking all the games for which a particular ranking system predicts a winning probabilty for one side of 70-75%, does the actual win percentage for the favoured side actually fall between 70-75%?
-
I just logged in, and it appears that the old leaderboard is back? The FAQ doesn't seem to indicate it, but the higher levels and # of games eligible sure do.
Huzzah!
-
Sweeeet!
Particularly sweet as I'm in first place heh. #brag
-
I just noticed this as well. Not sure if I'm really happy about it.
-
Great!
I'm particularly happy to no longer be in the top 10, which was frankly pretty embarrassing consider how routinely I get beaten by the actual best players.
-
Looks like a "Buy a goat" story. Do something definitely wrong, wait for a while, and revert the things back. And people will be happy!
-
It's definitely not the original formula...at least, not unless I suddenly improved to rank ~100 after staying mostly flat for the past few months. That's not necessarily a bad thing, of course...whatever the new formula is, it has a much longer history and at least is an improvement over what we've had for the past couple of weeks.
-
I was wondering why my rank suddenly dropped from 29 to 21 after being on a winning streak against players my level...
Honestly, I'm not a huge fan of this. I usually use Automatch to try and find opponents who are about as good as me, and This past week I've had some of the easiest games in a long time. Apparently the people who just got to rank 17-20 are only about as good as they used to be. Now I'm at rank 30, which is pretty close to my actual skill level, but with my playing habits, that will probably fall off pretty quick again.
-
Before the change to last 30 days, I had around 1200 games played. Now, after the new change, I have 644. So it's definitely not changed back to the way it was.
-
I guess all we can do for now is speculate wildly, so: He might have taken the suggestion to only count games back to the last rule change (where games with card restrictions are unrated).
Whatever it is, I'm really happy to see the return of a leaderboard that makes sense!
-
I think that's likely. I've lost thousands of games on my rating because I routinely played with !black market and !alchemy before the advent of veto.
-
I haven't lost a noticable number of games from my record, maybe 100 or so maximum and I've played completely random sets since I started playing. The few games that I have lost probably just account for games that I quickly accepted before noticing my opponent had set it up.
-
Seems like he's changed something with the uncertainty; mine is up about 2 from the pre-change leaderboard, after not playing much in the past week. (Maybe something similar to rspeer's 1% per day method.)
-
Yeah, it definitely seems a little different.
-
Sweet, I lost a level while not playing and got the like biggest uncertainty around. But better than the last experiment!
-
I guess all we can do for now is speculate wildly, so: He might have taken the suggestion to only count games back to the last rule change (where games with card restrictions are unrated).
...or include all games to the beginning of time, but exclude games with card restrictions.
-
Now that it's switched back to a time-based uncertainty growth, the last change I'd like to see made is basing level on skill alone rather than skill - 3*SD. I don't really understand the appeal of the latter system from any perspective. The only possible argument I see is for matchmaking purposes for people who are quite new and might have an unduly inflated skill level. But that seems like a very small fraction of all possible games. Elsewhere, matchmaking would likely proceed better if it were based on skill alone. And in terms of ego boosting on the leaderboard, it's never been clear to me why playing lots of games (or now, playing every day) is a major thing that we want to highlight. (added: if you just want to make sure that there aren't fluky people at the top, give everybody a rating but only have the leaderboard display those with uncertainties beneath a certain threshold. This would be like provisional or estimated ratings.)
All that said, I do think this approach is basically correct.
-
I would second the ideas of ackack! I mean, some players have skill+-6, others have +-13 and are like same level.
-
TL:dr; variance penalty encourages people to play more and mitigates luck and duck (out) strategies.
Imagine two players are the same high skill, and their goal is to optimize their placement on the leaderboard.
Without variance deduction, you wait til you hit a lucky streak, and then you stop playing with your artificially high level to prevent mean reversion.
With variance deduction, this is less likely to be an effective strategy. Assuming you are truly level 40, 45 +- 10 = 35, your mirror copy of your self could play a lot to hit 40 +- 4 = 36, and pass you on the leaderboard.
Further, as you selectively pick out the highest mean players, you are likely looking at the results of lucky streaks that temporarily boost players over the top of the others. Subtracting out the variance counters this effect somewhat between top players with few games (== more likely to be blessed by luck), than top players with many.
-
rrenaud said a lot of what i wanted to, but i figured id throw in a couple more things.
- with the big caveat that this how i understand how the system works...it currently starts out a new player as ranking lvl 25 +/- 25. any new players would then start off as being level 25. this would require you to either A. completely refigure the formulas involved or B. use the current method when calculating a players level for a players level and then use this adjustment after the fact. otherwise not only could you easily game the system but all pretenses of accuracy would get thrown out the window.
- you still need to play games to grow levels anyway. you need either an absurd win rate or at least a couple hundred games if you want to hit 40+ by your method. (and even then, all levels would be inflated quite a bit) reducing your uncertainty by playing more doesn't do much for you unless you actually win those games. this system seems to punish those who play a lot just as much as it might help those who play less.
- if anyone is just interested in studying how this would affect the leaderboard for science, you can find out with about 5 minutes of work in excel. you programmers could probably throw it online fairly easily too.
and of course, i cant help but notice that painted cow would agree with a system that would jump him to the top spot :).
-
What I don't understand: Why are we discussing this without any data? For anyone with access to the data, it should be trivial (if a bit of work) to see which leaderboard was better at predicting winners, or winning percentages. In fact, I would suggest that the normal way of doing things would be to run such tests before making a drastic change to the rating.
Btw, the one tweak I would try is playing with the initial ranking of new players. One could try replacing this with a setting based on empirical data (e.g., if new players on average have a ranking of 20 after their first 40 games, then probably an initial ranking of 20 is a better guess than an initial ranking of 25).
Anyway, I also understand DougZ's frustration with tweaks - changes to a rating system often will end up behaving differently than you could have anticipated.
-
http://councilroom.com/static/margin.txt.gz
Now you have the same data that everyone else does ;P. (I can refresh that so it's got the last n-months if anyone cares).
The means by themselves aren't meaningful. If you start players at 20 rather than 25, everyone just loses 5 levels, and nothing else changes.
-
Without variance deduction, you wait til you hit a lucky streak, and then you stop playing with your artificially high level to prevent mean reversion.
With the kinds of variance that we're talking about, I'm pretty sure this is quite unlikely. Especially since levels are based on 3*the SD, the actual spread in ratings for active players is going to be quite small. Have the leaderboard only display accounts with 3*SD < 11 or so with the present parameters and I think things would be just fine.
Further, as you selectively pick out the highest mean players, you are likely looking at the results of lucky streaks that temporarily boost players over the top of the others.
Even if the top 100 all played 10 games a day, there would still be people riding lucky streaks up and down. Given that people would be close in ratings, I don't think the variance would be strongly coupled to their performance. Thus I don't see how this really helps with that.
and of course, i cant help but notice that painted cow would agree with a system that would jump him to the top spot
That seems fair, seeing as how the rating system considers him the strongest player there is (and given my experience, I find that very plausible - I'd back him over any of the rest of the top 10 in a match.)
added: Even a "sort by mean skill" on the leaderboard - keeping the levels and all as what shows up on Iso - would be cool.
-
added: Even a "sort by mean skill" on the leaderboard - keeping the levels and all as what shows up on Iso - would be cool.
sortable leaderboards is a great idea. that way you could easily sift through players by level, skill, variance, or games played. it would also allow some flexibility in adding something like the 30 day leaderboard doug tried implementing.
-
I've seen the get to top and perch behavior on the iso leaderboard already. Luckily the timeout killed him.
If everyone is playing about the same number (and quality) of games, then the variances will be all pretty equal, and it won't change the relative ordering.
-
I've seen the get to top and perch behavior on the iso leaderboard already. Luckily the timeout killed him.
Are you talking about Paralyzed, or somebody else? Paralyzed is obviously a completely different example. I think very few players could get sufficiently lucky to get a new account to the top of the board without themselves being near the top.
If everyone is playing about the same number (and quality) of games, then the variances will be all pretty equal, and it won't change the relative ordering.
It seems like it has less to do with number of games than with making sure you're playing every day. People with similar numbers of total games can have 3*SD varying by 2, 3 or more. (Again, painted_cow is a good example here.) I don't have a problem with not showing people on the leaderboard because they're inactive, but dropping them seems perverse to me.
-
It seems like it has less to do with number of games than with making sure you're playing every day. People with similar numbers of total games can have 3*SD varying by 2, 3 or more.
as i understand it, and again maybe im wrong, your variance is also dependent on the quality of opponent you play. people who have logged more games with automatch +/- n levels will likely be different then people who have most of their games before automatch and faced a wider skill level. i think painted cow might fall into that latter category.
-
What I don't understand: Why are we discussing this without any data? For anyone with access to the data, it should be trivial (if a bit of work) to see which leaderboard was better at predicting winners, or winning percentages. In fact, I would suggest that the normal way of doing things would be to run such tests before making a drastic change to the rating.
Btw, the one tweak I would try is playing with the initial ranking of new players. One could try replacing this with a setting based on empirical data (e.g., if new players on average have a ranking of 20 after their first 40 games, then probably an initial ranking of 20 is a better guess than an initial ranking of 25).
Anyway, I also understand DougZ's frustration with tweaks - changes to a rating system often will end up behaving differently than you could have anticipated.
If you give new players an initial skill of 20 and recalculate everything, you will get the exact same results except that everyone's level will be 5 points lower. This is because everyone on the leaderboard was a new player once, and there is no meaning to the level numbers except for how they relate to the starting values.
They're just positions on a bell curve. And one of the important things about a bell curve is that, no matter how you scale it, you can't change the fact that fully half of the people are above average!
Now, what I think you're really saying is that the curve for Dominion isn't bell-shaped, that the median player is considerably worse than the mean player. And that's certainly true. It's easy to play Dominion slightly worse than average. A few people can play it several SDs better than average. To play several SDs worse than average, you kind of have to be trying to lose.
But the bell curve (Gaussian distribution) is really hard-coded into TrueSkill, because it's easy to compute with. It would require significantly changing the mathematics to use a different distribution.
-
as i understand it, and again maybe im wrong, your variance is also dependent on the quality of opponent you play. people who have logged more games with automatch +/- n levels will likely be different then people who have most of their games before automatch and faced a wider skill level. i think painted cow might fall into that latter category.
It does have some influence, but probably not that much, and pre-automatch I'm nearly certain people played closer to their own skill levels. I haven't played him for a while, but back in the day painted_cow was of the "challenge the highest ranked person in the room" school, as I recall.
-
Now that it's switched back to a time-based uncertainty growth, the last change I'd like to see made is basing level on skill alone rather than skill - 3*SD. I don't really understand the appeal of the latter system from any perspective. The only possible argument I see is for matchmaking purposes for people who are quite new and might have an unduly inflated skill level. But that seems like a very small fraction of all possible games. Elsewhere, matchmaking would likely proceed better if it were based on skill alone. And in terms of ego boosting on the leaderboard, it's never been clear to me why playing lots of games (or now, playing every day) is a major thing that we want to highlight. (added: if you just want to make sure that there aren't fluky people at the top, give everybody a rating but only have the leaderboard display those with uncertainties beneath a certain threshold. This would be like provisional or estimated ratings.)
All that said, I do think this approach is basically correct.
The TrueSkill standard is for the leaderboard to be displayed in the way it is now for the sake of new players. That way you start at the bottom and climb up, rather than starting in the middle and falling, which can be disheartening.
However, matchmaking IS supposed to based on mean skill estimate, rather than displayed level. I'm not sure how it's done on iso right now, but the restricting opponents by level via the drop-down box I guess can do the "wrong" thing if you have ea dramatically different variance than other people at your "skill level".
-
I am VERY saddened by the implementation of certainty-decay. At least, if I play frequently for a short while and up to my old standards I should be able to get back my old certainty...
Any decay system punishes paper players like myself, since I play online only when there is no one to play with in person.
The old system was perfect. Who cares if people camp the top of the leaderboard. They earned it by being excellent players and they should be recognized for that. I really liked how the top of the leaderboard acted like a hall-of-fame on occasion: players that play less now, but were spectacular in their time.
It took hard work and patience to make it to near the top of the old board. That made it so much more exciting to move up.
-
Now that it's switched back to a time-based uncertainty growth, the last change I'd like to see made is basing level on skill alone rather than skill - 3*SD. I don't really understand the appeal of the latter system from any perspective.
Can anyone expalin this a bit more? Is it changend now in comparison to the old leaderboard (that we had in action till 24th of november?).
-
But the bell curve (Gaussian distribution) is really hard-coded into TrueSkill, because it's easy to compute with. It would require significantly changing the mathematics to use a different distribution.
Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.
@matchmaking: I think some kind of variance-reduction to the mean is fine. If it really has to be 3sigma, I don't know, but I don't think it will matter much. For the top, where you say can loose 3-5 levels compared to others with the same mean, it might be the better method, but if that really would lead to problems, I would guess it's because the variance-increase from idleing would be to larger.
On the other hand, consider someone entering the system, and perhaps playing not so much, and not so good. When he wants to use the matchmaking, he has to grind is way down from mean 25, which is about rank 1.800 on todays leaderboard and therefore significantly above the median. Instead of playing level 0 people. If you are good, you now get out there quite quickly, because your mean increases and your sigma decreases. But you get out there winning, which is much more fun than loose you're way to where you "belong".
In WoW, as a also-not-that-good-PvE-player, I tried to do some justForTheFunArena, and it is not possible. The system there is (in the sense of "was 3 years ago") that you start with level 1.500 (the equivalent of mean 25), and they have only the mean skill, no model for the variance. And each say 12 weeks it resets. So if you, at some rainy weekend decide to just do some fights, you start at 1.500, lose some games and get down to 1.300. But even there, you don't find bad players, because of course nobody wants to spend an afternoon losing (just to again only meet PvP-optimized people, because nobody wants to spend ....). And some weeks later, all is gone again. So back to farming heroics...
So this might be a system that works for the guys who are really interested in it, but for just dropping by and playing for the fun it is not really good.
-
Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.
Right, and also in such a way that it isn't confusing. People are often intuitively familiar with margins of error, which can be described with Gaussian distributions.
Now I have an amusing image of an alternate-universe Dominion forum, where there's a leaderboard with, say, four degrees of freedom in how it describes your rating.
"hey guys, I finally got to level 20 but I'm still skewed downwards. What do I have to do to make my rank skewed upwards? Do Treasure Maps help or hurt my skew?"
"So you may have seen an account called 'hypothesis' in the top ten recently. That's me, on an account where I only play against people with high kurtosis."
"Is the Making It To Level 42/+3/-1(0.125) article ever going to be finished?"
-
Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.
Right, and also in such a way that it isn't confusing. People are often intuitively familiar with margins of error, which can be described with Gaussian distributions.
Now I have an amusing image of an alternate-universe Dominion forum, where there's a leaderboard with, say, four degrees of freedom in how it describes your rating.
"hey guys, I finally got to level 20 but I'm still skewed downwards. What do I have to do to make my rank skewed upwards? Do Treasure Maps help or hurt my skew?"
"So you may have seen an account called 'hypothesis' in the top ten recently. That's me, on an account where I only play against people with high kurtosis."
"Is the Making It To Level 42/+3/-1(0.125) article ever going to be finished?"
Thanks for making me literally laugh out loud.
-
Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.
Right, and also in such a way that it isn't confusing. People are often intuitively familiar with margins of error, which can be described with Gaussian distributions.
Now I have an amusing image of an alternate-universe Dominion forum, where there's a leaderboard with, say, four degrees of freedom in how it describes your rating.
"hey guys, I finally got to level 20 but I'm still skewed downwards. What do I have to do to make my rank skewed upwards? Do Treasure Maps help or hurt my skew?"
"So you may have seen an account called 'hypothesis' in the top ten recently. That's me, on an account where I only play against people with high kurtosis."
"Is the Making It To Level 42/+3/-1(0.125) article ever going to be finished?"
So. Much. Win.
Especially the bonus theory-alt.
-
I was thinking about the leaderboard as it stands now and how it compares to the leaderboard before the 11/24 change (if it amuses you the way it does me, I encourage you to think of the pre-leaderboard change as "Coke", the 11/24 leaderboard as "New Coke", and the current one as "Coke Classic").
Pros: current leaderboard does not have Karumah or Paralyzed at the top. tat is level 35 instead of 45. I think this more closely resembles "correctness", however we each choose to define that.
Cons: theory and shark_bait are level 36, with only 700 and 500 games played, respectively. I think most of us agree that they are both excellent players and that the current system underrates them (though obviously not nearly as badly as New Coke did).
I tried this thought experiment. Imagine that you had a player who, for whatever reason, just HATED Mining Village. So he plays all of his games with "!mining village", he only plays level 35+ players, and he wins these games 100% of the time. I submit that we would all agree that this player:
- is really, really, really good.
- should be the favorite in a tournament, even one that allowed Mining Village.
We might differ on whether he "deserves" to be on the leaderboard based on philosophical differences of what we perceive the leaderboard's purpose to be, but I think it's clear our fictional player is really good and is probably the best player in the world.
Now extend the example. Let's say our fictional player still only plays 35+ players, wins 100% of the time, but bans 10 cards instead of 1. How good would we say this player is? I think it's fairly uncontroversial to say that this player is excellent a high percentage of the time (the probability that a random kingdom does not contain any of his 10 undesirable cards), and unknown the rest.
So, this got me thinking: what if the ranking system took all games into account, but weighted each game based on the probability that the specified restrictions would be fulfilled in a random game? That is, if I specify all 10 cards, that game is worth basically nothing; if I ban just one card, that game is worth only slightly less than a game with no restrictions. This seems like it would fix theory, shark_bait, and possibly others' ratings, and has the nice side benefit of putting Paralyzed somewhere near the beginners.
Thoughts? (Assume Doug has the data and the necessary processing power.)
-
Your idea intrigues me, but how would your games against such an opponent be weighted?
Also, I think there's some pretty complex math if you get into some of the weirder restraint possibilities (e.g., 3-6 cards from Intrigue, no Possession, 0-1 cards from Alchemy).
-
what if the ranking system took all games into account, but weighted each game based on the probability that the specified restrictions would be fulfilled in a random game?
Introduces a lot of subjectivity and complexity with little marginal gain in utility.
Edit: also, does this mean that each time a new expansion is released, everyone's played-games-to-date take a nosedive in weight?
-
Your idea intrigues me, but how would your games against such an opponent be weighted?
Given that you can always decline proposed games, accepting a game with restrictions should weight the game in the same way for both players. Otherwise I could set my status as "accepting <restriction> games only" and get full credit for them. I guess you can do that anyway.
Introduces a lot of subjectivity and complexity with little marginal gain in utility.
Complexity, sure. But only on the implementation side; it doesn't seem hard to explain how this system works. Why do you think it introduces subjectivity?
I suppose we can agree to disagree, but getting widely-recognized top players to be ranked appropriately does not seem like a small marginal gain to me.
Edit: also, does this mean that each time a new expansion is released, everyone's played-games-to-date take a nosedive in weight?
Yes. Which is how it should be -- when new expansions are released, everyone's variance goes up as they adjust to the new cards (I think someone proposed this specific change in the other leaderboard thread, and this system gets it for free).
-
Introduces a lot of subjectivity and complexity with little marginal gain in utility.
Complexity, sure. But only on the implementation side; it doesn't seem hard to explain how this system works. Why do you think it introduces subjectivity?
I suppose we can agree to disagree, but getting widely-recognized top players to be ranked appropriately does not seem like a small marginal gain to me.
For one,
We might differ on whether he "deserves" to be on the leaderboard based on philosophical differences of what we perceive the leaderboard's purpose to be
For another, the presence of veto mode makes "probability these (conditions) are fulfilled in a random game" non-trivial.
Finally, as you yourself pointed out, someone who wants to game the system can specify no prior restrictions, but refuse to accept any auto-matches with cards he doesn't want, for example.
-
I don't really like the new ranking system either. The most glaring flaw to me is how a player of any rank would be reduced to level 0 after 30 days of inactivity. This is definitely not a representative level of their skill.
I didn't really notice the issue with the old system. It was more stable, was easier to understand (no jumping levels +/-10 overnight which has happened to me already both ways), and I noticed that the skill level would increase after a string of wins and decrease after a string of losses less drastically than it does now.
As noted previously in this thread, I would also like to see "constraint free" games be the only ones counted for the leaderboard. That way, no one can specialize in a specific set of cards and artificially inflate their rank (i.e. no more KC/Masquerade/Goons shenanigans)
-
I'm guessing you jumped -10 levels overnight when the 30-day leaderboard was activated and +10 overnight when it was removed again. The leaderboard as of a week or so ago is back pretty much to where it used to be. It's slightly different and we don't know in what way, but it seems that it includes all games from all time, excluding any that had constraints set.
-
Based on my observations, it increases variance every day now.
-
Yeah I noticed this morning I just dropped a level from not a playing any games in the last few days, which was a bit disheartening :)