Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[rrenaud]

I've seen the get to top and perch behavior on the iso leaderboard already.  Luckily the timeout killed him.

If everyone is playing about the same number (and quality) of games, then the variances will be all pretty equal, and it won't change the relative ordering.

[ackack]

I've seen the get to top and perch behavior on the iso leaderboard already.  Luckily the timeout killed him.

Are you talking about Paralyzed, or somebody else? Paralyzed is obviously a completely different example. I think very few players could get sufficiently lucky to get a new account to the top of the board without themselves being near the top.

If everyone is playing about the same number (and quality) of games, then the variances will be all pretty equal, and it won't change the relative ordering.

It seems like it has less to do with number of games than with making sure you're playing every day. People with similar numbers of total games can have 3*SD varying by 2, 3 or more. (Again, painted_cow is a good example here.) I don't have a problem with not showing people on the leaderboard because they're inactive, but dropping them seems perverse to me.

[greatexpectations]

It seems like it has less to do with number of games than with making sure you're playing every day. People with similar numbers of total games can have 3*SD varying by 2, 3 or more.

as i understand it, and again maybe im wrong, your variance is also dependent on the quality of opponent you play.  people who have logged more games with automatch +/- n levels will likely be different then people who have most of their games before automatch and faced a wider skill level. i think painted cow might fall into that latter category.

[rspeer]

What I don't understand: Why are we discussing this without any data? For anyone with access to the data, it should be trivial (if a bit of work) to see which leaderboard was better at predicting winners, or winning percentages. In fact, I would suggest that the normal way of doing things would be to run such tests before making a drastic change to the rating.
Btw, the one tweak I would try is playing with the initial ranking of new players. One could try replacing this with a setting based on empirical data (e.g., if new players on average have a ranking of 20 after their first 40 games, then probably an initial ranking of 20 is a better guess than an initial ranking of 25).
Anyway, I also understand DougZ's frustration with tweaks - changes to a rating system often will end up behaving differently than you could have anticipated.

If you give new players an initial skill of 20 and recalculate everything, you will get the exact same results except that everyone's level will be 5 points lower. This is because everyone on the leaderboard was a new player once, and there is no meaning to the level numbers except for how they relate to the starting values.

They're just positions on a bell curve. And one of the important things about a bell curve is that, no matter how you scale it, you can't change the fact that fully half of the people are above average!

Now, what I think you're really saying is that the curve for Dominion isn't bell-shaped, that the median player is considerably worse than the mean player. And that's certainly true. It's easy to play Dominion slightly worse than average. A few people can play it several SDs better than average. To play several SDs worse than average, you kind of have to be trying to lose.

But the bell curve (Gaussian distribution) is really hard-coded into TrueSkill, because it's easy to compute with. It would require significantly changing the mathematics to use a different distribution.

[ackack]

as i understand it, and again maybe im wrong, your variance is also dependent on the quality of opponent you play.  people who have logged more games with automatch +/- n levels will likely be different then people who have most of their games before automatch and faced a wider skill level. i think painted cow might fall into that latter category.

It does have some influence, but probably not that much, and pre-automatch I'm nearly certain people played closer to their own skill levels. I haven't played him for a while, but back in the day painted_cow was of the "challenge the highest ranked person in the room" school, as I recall.

[HiveMindEmulator]

Now that it's switched back to a time-based uncertainty growth, the last change I'd like to see made is basing level on skill alone rather than skill - 3*SD. I don't really understand the appeal of the latter system from any perspective. The only possible argument I see is for matchmaking purposes for people who are quite new and might have an unduly inflated skill level. But that seems like a very small fraction of all possible games. Elsewhere, matchmaking would likely proceed better if it were based on skill alone. And in terms of ego boosting on the leaderboard, it's never been clear to me why playing lots of games (or now, playing every day) is a major thing that we want to highlight. (added: if you just want to make sure that there aren't fluky people at the top, give everybody a rating but only have the leaderboard display those with uncertainties beneath a certain threshold. This would be like provisional or estimated ratings.)

All that said, I do think this approach is basically correct.

The TrueSkill standard is for the leaderboard to be displayed in the way it is now for the sake of new players. That way you start at the bottom and climb up, rather than starting in the middle and falling, which can be disheartening.
However, matchmaking IS supposed to based on mean skill estimate, rather than displayed level. I'm not sure how it's done on iso right now, but the restricting opponents by level via the drop-down box I guess can do the "wrong" thing if you have ea dramatically different variance than other people at your "skill level".

[Rjax36]

I am VERY saddened by the implementation of certainty-decay. At least, if I play frequently for a short while and up to my old standards I should be able to get back my old certainty...

Any decay system punishes paper players like myself, since I play online only when there is no one to play with in person.

The old system was perfect. Who cares if people camp the top of the leaderboard. They earned it by being excellent players and they should be recognized for that. I really liked how the top of the leaderboard acted like a hall-of-fame on occasion: players that play less now, but were spectacular in their time.

It took hard work and patience to make it to near the top of the old board. That made it so much more exciting to move up.

[Reyk]

Now that it's switched back to a time-based uncertainty growth, the last change I'd like to see made is basing level on skill alone rather than skill - 3*SD. I don't really understand the appeal of the latter system from any perspective.

Can anyone expalin this a bit more? Is it changend now in comparison to the old leaderboard (that we had in action till 24th of november?).

[DStu]

But the bell curve (Gaussian distribution) is really hard-coded into TrueSkill, because it's easy to compute with. It would require significantly changing the mathematics to use a different distribution.

Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.

@matchmaking: I think some kind of variance-reduction to the mean is fine. If it really has to be 3sigma, I don't know, but I don't think it will matter much. For the top, where you say can loose 3-5 levels compared to others with the same mean, it might be the better method, but if that really would lead to problems, I would guess it's because the variance-increase from idleing would be to larger.
On the other hand, consider someone entering the system, and perhaps playing not so much, and not so good. When he wants to use the matchmaking, he has to grind is way down from mean 25, which is about rank 1.800 on todays leaderboard and therefore significantly above the median. Instead of playing level 0 people. If you are good, you now get out there quite quickly, because your mean increases and your sigma decreases. But you get out there winning, which is much more fun than loose you're way to where you "belong".

In WoW, as a also-not-that-good-PvE-player, I tried to do some justForTheFunArena, and it is not possible. The system there is (in the sense of "was 3 years ago") that you start with level 1.500 (the equivalent of mean 25), and they have only the mean skill, no model for the variance. And each say 12 weeks it resets. So if you, at some rainy weekend decide to just do some fights, you start at 1.500, lose some games and get down to 1.300. But even there, you don't find bad players, because of course nobody wants to spend an afternoon losing (just to again only meet PvP-optimized people, because nobody wants to spend ....). And some weeks later, all is gone again.  So back to farming heroics...
So this might be a system that works for the guys who are really interested in it, but for just dropping by and playing for the fun it is not really good.

[rspeer]

Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.

Right, and also in such a way that it isn't confusing. People are often intuitively familiar with margins of error, which can be described with Gaussian distributions.

Now I have an amusing image of an alternate-universe Dominion forum, where there's a leaderboard with, say, four degrees of freedom in how it describes your rating.

"hey guys, I finally got to level 20 but I'm still skewed downwards. What do I have to do to make my rank skewed upwards? Do Treasure Maps help or hurt my skew?"

"So you may have seen an account called 'hypothesis' in the top ten recently. That's me, on an account where I only play against people with high kurtosis."

"Is the Making It To Level 42/+3/-1(0.125) article ever going to be finished?"

[WanderingWinder]

Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.

Right, and also in such a way that it isn't confusing. People are often intuitively familiar with margins of error, which can be described with Gaussian distributions.

Now I have an amusing image of an alternate-universe Dominion forum, where there's a leaderboard with, say, four degrees of freedom in how it describes your rating.

"hey guys, I finally got to level 20 but I'm still skewed downwards. What do I have to do to make my rank skewed upwards? Do Treasure Maps help or hurt my skew?"

"So you may have seen an account called 'hypothesis' in the top ten recently. That's me, on an account where I only play against people with high kurtosis."

"Is the Making It To Level 42/+3/-1(0.125) article ever going to be finished?"

Thanks for making me literally laugh out loud.

[Kirian]

Probably the math would not be so much harder, I think the bigger problem is to find a distribution which is arguable better than Gaussian in such a way that it is worth all the trouble.

Right, and also in such a way that it isn't confusing. People are often intuitively familiar with margins of error, which can be described with Gaussian distributions.

Now I have an amusing image of an alternate-universe Dominion forum, where there's a leaderboard with, say, four degrees of freedom in how it describes your rating.

"hey guys, I finally got to level 20 but I'm still skewed downwards. What do I have to do to make my rank skewed upwards? Do Treasure Maps help or hurt my skew?"

"So you may have seen an account called 'hypothesis' in the top ten recently. That's me, on an account where I only play against people with high kurtosis."

"Is the Making It To Level 42/+3/-1(0.125) article ever going to be finished?"

So.  Much.  Win.

Especially the bonus theory-alt.

[mischiefmaker]

I was thinking about the leaderboard as it stands now and how it compares to the leaderboard before the 11/24 change (if it amuses you the way it does me, I encourage you to think of the pre-leaderboard change as "Coke", the 11/24 leaderboard as "New Coke", and the current one as "Coke Classic").

Pros: current leaderboard does not have Karumah or Paralyzed at the top. tat is level 35 instead of 45. I think this more closely resembles "correctness", however we each choose to define that.
Cons: theory and shark_bait are level 36, with only 700 and 500 games played, respectively. I think most of us agree that they are both excellent players and that the current system underrates them (though obviously not nearly as badly as New Coke did).

I tried this thought experiment. Imagine that you had a player who, for whatever reason, just HATED Mining Village. So he plays all of his games with "!mining village", he only plays level 35+ players, and he wins these games 100% of the time. I submit that we would all agree that this player:
 - is really, really, really good.
 - should be the favorite in a tournament, even one that allowed Mining Village.

We might differ on whether he "deserves" to be on the leaderboard based on philosophical differences of what we perceive the leaderboard's purpose to be, but I think it's clear our fictional player is really good and is probably the best player in the world.

Now extend the example. Let's say our fictional player still only plays 35+ players, wins 100% of the time, but bans 10 cards instead of 1. How good would we say this player is? I think it's fairly uncontroversial to say that this player is excellent a high percentage of the time (the probability that a random kingdom does not contain any of his 10 undesirable cards), and unknown the rest.

So, this got me thinking: what if the ranking system took all games into account, but weighted each game based on the probability that the specified restrictions would be fulfilled in a random game? That is, if I specify all 10 cards, that game is worth basically nothing; if I ban just one card, that game is worth only slightly less than a game with no restrictions. This seems like it would fix theory, shark_bait, and possibly others' ratings, and has the nice side benefit of putting Paralyzed somewhere near the beginners.

Thoughts? (Assume Doug has the data and the necessary processing power.)

[theory]

Your idea intrigues me, but how would your games against such an opponent be weighted? 

Also, I think there's some pretty complex math if you get into some of the weirder restraint possibilities (e.g., 3-6 cards from Intrigue, no Possession, 0-1 cards from Alchemy).

[Anon79]

what if the ranking system took all games into account, but weighted each game based on the probability that the specified restrictions would be fulfilled in a random game?

Introduces a lot of subjectivity and complexity with little marginal gain in utility.

Edit: also, does this mean that each time a new expansion is released, everyone's played-games-to-date take a nosedive in weight?

[mischiefmaker]

Your idea intrigues me, but how would your games against such an opponent be weighted? 

Given that you can always decline proposed games, accepting a game with restrictions should weight the game in the same way for both players. Otherwise I could set my status as "accepting <restriction> games only" and get full credit for them. I guess you can do that anyway.

Introduces a lot of subjectivity and complexity with little marginal gain in utility.

Complexity, sure. But only on the implementation side; it doesn't seem hard to explain how this system works. Why do you think it introduces subjectivity?

I suppose we can agree to disagree, but getting widely-recognized top players to be ranked appropriately does not seem like a small marginal gain to me.

Edit: also, does this mean that each time a new expansion is released, everyone's played-games-to-date take a nosedive in weight?

Yes. Which is how it should be -- when new expansions are released, everyone's variance goes up as they adjust to the new cards (I think someone proposed this specific change in the other leaderboard thread, and this system gets it for free).

[Anon79]

Introduces a lot of subjectivity and complexity with little marginal gain in utility.

Complexity, sure. But only on the implementation side; it doesn't seem hard to explain how this system works. Why do you think it introduces subjectivity?

I suppose we can agree to disagree, but getting widely-recognized top players to be ranked appropriately does not seem like a small marginal gain to me.

For one,

We might differ on whether he "deserves" to be on the leaderboard based on philosophical differences of what we perceive the leaderboard's purpose to be

For another, the presence of veto mode makes "probability these (conditions) are fulfilled in a random game" non-trivial.

Finally, as you yourself pointed out, someone who wants to game the system can specify no prior restrictions, but refuse to accept any auto-matches with cards he doesn't want, for example.

[andwilk]

I don't really like the new ranking system either.  The most glaring flaw to me is how a player of any rank would be reduced to level 0 after 30 days of inactivity.  This is definitely not a representative level of their skill.

I didn't really notice the issue with the old system.  It was more stable, was easier to understand (no jumping levels +/-10 overnight which has happened to me already both ways), and I noticed that the skill level would increase after a string of wins and decrease after a string of losses less drastically than it does now.

As noted previously in this thread, I would also like to see "constraint free" games be the only ones counted for the leaderboard.  That way, no one can specialize in a specific set of cards and artificially inflate their rank (i.e. no more KC/Masquerade/Goons shenanigans)

[Thisisnotasmile]

I'm guessing you jumped -10 levels overnight when the 30-day leaderboard was activated and +10 overnight when it was removed again. The leaderboard as of a week or so ago is back pretty much to where it used to be. It's slightly different and we don't know in what way, but it seems that it includes all games from all time, excluding any that had constraints set.

[theory]

Based on my observations, it increases variance every day now.

[Fabian]

Yeah I noticed this morning I just dropped a level from not a playing any games in the last few days, which was a bit disheartening