Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[ragingduckd]

I've been scraping the Goko top-100 leaderboard for a while and I've now gathered enough data to say a little about the infamous "rating drift."

Your rating changes when you play a game, of course, but Goko also updates everybody's rating simultaneously once a day. This happens around midnight EST, when their server runs a script called omgwtfwearemorons.sh

Here's what happened to the top 100 pro ratings yesterday morning:

Code: [Select]

Stef -20
Wandering Winder 0
Andrew Iannaccone -19
LESPEUTERE -20
Mic Qsenoch -24
Rabid -22
Geronimoo -22
PitrPicko -20
eliegel -22
Stealth Tomato -20
Robz888 -24
nomnomnom -48
SheCantSayNo -18
Troninho -19
Davio -71
RTT -18
Gorling -34
Lekkit -20
Fabian -22
Eevee -32
Bulec -60
flyingkuyt -20
qmech -35
GwinnR -29
kenyou2859 -29
Rene Kuroi -29
jsh357 -58
manzi -24
scott pilgrim -24
Watno -49
TrickStaR -27
Boodaloo -41
First -33
Zakharov -27
nnn -28
Twistedarcher -24
lightbulb -18
Jeebus -23
Adam Horton -33
wsc -28
kn1tt3r -47
Lagrangian -23
dudeabides -20
iriho -22
fah -28
attrill -26
majormajor -21
SM.SM -20
Darter -21
zporiri -27
theParty -20
Masschy -22
jhovall_goko -25
Dubdubdubdub -59
Titandrake -22
faw -35
yac -27
ChiChi -22
daniel greif -62
Perna -20
Dumbassador -24
YeahBoooey -25
Slyfox -21
pâté de campagne -31
yed -19
Underdog -24
nidzela -19
David Hunter -24
houroku -21
Blobby -36
loppo -31
Lotoreo -20
kilgoretrout103 -19
Emeric -21
Tombery -48
jaybeez -23
Aron35 -22
BadAMutha -25
Drab Emordnilap -82
Perry Green -24
Pex Golder -55
Tom Collett -45
LosChikitos -20
HF -25
Cruxis -26
A Drowned Kernel -28
sangatsu -21
Rhun -23
heron -27
Lukáš Černoušek -26
gagnerouperdretelleestlaquestion? -28
secret tunnel -22
Indur -26
Kirian -23
Warfreak2 -20
MarkowKette -26
hiroki -20
Alexey Rakhmanov -23
Silverfinger -26
Psyduck 128

Don't ask me why WW's rating didn't adjust that time or why Psyduck's went up... equally strange things happened to other players on other days. Anyway, with few exceptions, the changes are always negative and mostly around -25 points. Here's are the mean changes every day for the past month:

Code: [Select]

2013-05-12 -27.76
2013-05-13 -21.75
2013-05-14 -28.34
2013-05-15 -26.83
2013-05-16  01.71
2013-05-17 -28.13
2013-05-18 -27.58
2013-05-19 -29.07
2013-05-20 -29.10
2013-05-21 -28.29
2013-05-22 -27.60
2013-05-23 -27.57
2013-05-24 -26.66
2013-05-25 -27.60
2013-05-26 -26.74
2013-05-27 -27.55
2013-05-28 -26.16
2013-05-29 -27.28
2013-05-30 -27.82
2013-05-31 -24.80
2013-06-01 -08.69
2013-06-02 -26.72
2013-06-03 -26.70
2013-06-04 -31.97
2013-06-05 -24.71
2013-06-06 -26.11
2013-06-07 -25.36
2013-06-08 -26.45
2013-06-09 -25.03
2013-06-10 -26.57
2013-06-11 -27.30
2013-06-12 -26.54
2013-06-13 -26.70

Ok, sorry for all the raw numbers. Here's what this means:

First, your rating will drop like a stone if you ever take a break from Dominion. If you're currently #50 with a rating of ~6000 and you quit playing today, you'll drop out of the top 100 in about two weeks. Edit: Several responses below report drops faster than this for 1+ week of inactivity. Perhaps the drift accelerates.

Second, anyone who isn't playing a lot is dramatically underrated. This is a crude approximation, but the per-game change that you get from Goko is something like 2x the change you'd get from the de-facto standard rating system for online chess (Elo with K=32). Roughly, if you're really a 6500 player, but you only play 2-3 games a day, you can expect to hover around 6000 indefinitely. Edit: you can expect to oscillate around a mean of 6000 or so.

I can only scrape the top 100, so I don't know whether this is also happening at lower ratings. It's hard to imagine that everyone on Goko be could dropping by 25 pts a day, so maybe players with really low ratings are gaining a comparable amount. There's no relationship between rating and drift in the top 100; ratings between 5700 and 7300 all drift in about the same way.

I'm guessing that at least everyone above 5000 is seeing a daily loss. Edit: See responses below for several confirmations that ratings drop over time all the way down to 2000.

If you're interested in playing with this data, just send me a PM and I'll send you either database access or just the 16M csv file.

[Watno]

My rating drop seems to have been one of the highest. This is probably due to the fact that I didn't play on Goko at all for about a week.
So I assume how much you lose per day increases depending on how long you haven't played.

You rating changes when you play a game, of course, but Goko also updates everybody's rating simultaneously once a day. This happens around midnight EST, when their server runs a script called omgwtfwearemorons.sh

Just to make sure: The name is a joke, right?

[Blueswan]

I'm usually hovering in the 4700-5200 range and my rating also drops overnight.

[werothegreat]

I took a break for a couple months, and I dropped from 5000s to 2000s.

[serakfalcon]

Hey there, I can confirm that, in fact, below 4000 and down to 2000, the ratings still decrease over time. I got up to ~5500 when I was playing constantly (I was around lvl 30 on iso when I played constantly, lvl 26-28 when not) but after a month or so hiatus from dominion (been too busy at work) my rating has dropped. It was about 4600-4800 when I played on the 22nd of May, (I remember playing you ragingduckd, and I noticed I had dropped below your 5000+ specification after the game started) and has dropped to 3683 today, which is an average loss of 40ish points per day. Obviously we need more data but this is definitely an 1000 point loss over 20 days, which is kind of crazy

[Qvist]

+1 just for omgwtfwearemorons.sh 

Thanks for the data and I can confirm that. I was at #20 a few weeks ago, have only played a few games in the last couple of weeks, now I'm out of the Top100.

[Davio]

I only play up to 3 pro games max per evening if I even get around to that much.
Still I made it into the top 15-ish, which was quite surprising for me with so few games.

Quality over quantity, ey.

[Watno]

Don't ask me why WW's rating didn't adjust that time or why Psyduck's went up... equally strange things happened to other players on other days. Anyway, with few exceptions, the changes are always negative and mostly around -25 points.

Is it possible they simple played a game in the timeframe between your 2 checks on the leaderboard?

[Psyduck]

Don't ask me why WW's rating didn't adjust that time or why Psyduck's went up... equally strange things happened to other players on other days. Anyway, with few exceptions, the changes are always negative and mostly around -25 points.

Is it possible they simple played a game in the timeframe between your 2 checks on the leaderboard?

At least not for me. I only played one game yesterday, which must have been around 3pm EST.

[philosophyguy]

I was in the 4-5k range and stopped playing for about a week and a half over frustration with the constant freezes. I logged in today to play with Guilds, and my pro rating had dropped to around 1k. So yes, you get pummeled for not playing constantly.

[ragingduckd]

I only play up to 3 pro games max per evening if I even get around to that much.
Still I made it into the top 15-ish, which was quite surprising for me with so few games.

I think you're exactly the type of underrated infrequent player I'm talking about. You played 3 pro games per day over the last month and your median rating was 5919.

[WanderingWinder]

I didn't play at all yesterday, and I haven't for some time.

I don't think it's a terrible idea to increase rating uncertainties over time (many respectable systems do this). The real issue is that they are somehow subtracting this from your rating, and showing that as your rating. Which means this doesn't actually reflect your rating.

Something else weird is clearly going on though - I would expect some kind of reversion towards a high uncertainty, or a gradual increasing in uncertainty, but it should in some way be constant - either constant decay, or more likely the uncertainty has a constant percentage increase towards a max value or something.

I know I have *gained* 3 points overnight before, so something strange is up. I also sometimes get no rating change from winning/losing against relatively high rated guys, which makes no sense. I wonder if they re-rate things at some point, or something like that? Really strange.

Also, June 1st and especially May 16th are hugely baffling.

[michaeljb]

Too bad the rating process is too complicated to ever put in a formula; we can never know why it does this.

[andwilk]

I haven't played in about 3 weeks and my rating has dropped from right around 6000 to just above 4500.  Rating decay is way too fast in this system

[Qvist]

I haven't played in about 3 weeks and my rating has dropped from right around 6000 to just above 4500.  Rating decay is way too fast in this system

Yeah. And now I played 2 games only and went from 5500 back to around 6100, that's way too fast too. 600 points, come on!

[Tables]

I haven't played pro in a little while, and my rating has dropped down to something insane. It was about 4K after a short losing streak, now it's like 1.5K.

[Beyond Awesome]

I have noticed the rating decline as well. It is pretty big. I was around 6000-6200. But, now I find myself around 5700 these days. I did take a break for about a week, and take the occasional 3 or 4 day break.

[snorkelbike]

Same thing here. I've played just a few pro games in the last 2-3 weeks. Went from >6000 to now 4812. Not as fast a decline as some of you are reporting, but still a little bit discouraging.

[Joseph2302]

I basically didn't play pro games for about 2-3 weeks, and declined from ~4500 to ~2500. Having played about 20 pro games in the last few days, winning ~13, I have somehow got rating up to ~5400.

[Obi Wan Bonogi]

Wow, that is harsh decay. 

[Tables]

Not so much drift, but despite having played a lot of pro games in the past, my ratings now are jumping around incessantly. I won a few games and went up to about 5.5K, and about 50th on the leaderboard. Then I proceeded to lose 3 games against similarish ranked opponents and was down to just under 3000. I only realised I'd dropped so much when I tried to join a 3000+ game someone was hosting and got autokicked.


(incidentally, I feel the need to vent about two of those games... let's leave it as: open terminal/silver. Both miss the first reshuffle. Buy silver/another terminal turns 3/4. Both terminals in T5 hand. Both games. I resigned one when I was down 4-1 on a curse split, having two Sea Hags to his one, BM type boards. What great shuffling skills I have.)

[DStu]

Agree with WW. Their uncertainity seems to change much to fast.  Otherwise, it's absolutely sane to increase uncertainity for non-playing, if you haven't played for some month the system should be more unsecure on your rating.
But I would say it is a safe bet that the system should not be unsecure on your rating as it is on the rating on a new player, so there would be some canonical upper bound to converge to*.  Furthermore, you don't forget this game that fast that you are practically as hard to predict as some new player to the site.


*also of course you might have a higher mean than the average that the new player has, this is not true in the strictly mathematical provable sense, but I don't think one makes a big mistake with that.

[dnkywin]

This explains why my rating went from ~6200 to less than 4000 in May (when I didn't play). Not that I really care all that much about rating, but the most annoying part of this rating drop was that people with "5000+" in their game titles were kicking me out.

Will Goko ever release how they calculate ratings? That would be _really_ helpful.

[WanderingWinder]

Will Goko ever release how they calculate ratings? That would be _really_ helpful.

It's apparently too complicated to possibly write down.

[Kirian]

Will Goko ever release how they calculate ratings? That would be _really_ helpful.

It's apparently too complicated to possibly write down.

I imagine there's a randomization element that involves live gerbils, 23-sided dice, and the phase of the moon.

[ashersky]

Will Goko ever release how they calculate ratings? That would be _really_ helpful.

It's apparently too complicated to possibly write down.

Is there no way you math whizzes can reverse engineer the formula by having two players create new accounts and play a specific number of games a day?

[DStu]

Is there no way you math whizzes can reverse engineer the formula by having two players create new accounts and play a specific number of games a day?

That would imply that it is possible to write it down, and we all know that it isn't...


Seriously, at least it should be possible to fit TrueSkill parameters, and from there seeing how well they fit one could say more.  In principle, there are infinite possibilities of setting up such a model, especially if you want to account for unreasonable choices, which I assume we must.  The more degrees of freedom the parameter space you are looking on is, the more difficult that gets.

But it couldn't hurt to have the data, I would best would to have many (~100) games from one day because that ignores whatever they do overnight, and this would better be fitted seperately once you know what they do usually.  Resign at turn1 is fine (or?). Can one automate this with a browser extension?

[WanderingWinder]

Is there no way you math whizzes can reverse engineer the formula by having two players create new accounts and play a specific number of games a day?

That would imply that it is possible to write it down, and we all know that it isn't...


Seriously, at least it should be possible to fit TrueSkill parameters, and from there seeing how well they fit one could say more.  In principle, there are infinite possibilities of setting up such a model, especially if you want to account for unreasonable choices, which I assume we must.  The more degrees of freedom the parameter space you are looking on is, the more difficult that gets.

But it couldn't hurt to have the data, I would best would to have many (~100) games from one day because that ignores whatever they do overnight, and this would better be fitted seperately once you know what they do usually.  Resign at turn1 is fine (or?). Can one automate this with a browser extension?

The second half of this, are you talking about to set up our own system, or to reverse engineer theirs? I don't think we should be married to TrueSkill so much for a system, but ok whatever. But I really don't see why would try to force Goko's system into a TrueSkill framework.


Exactly reverse engineering their system would be very hard. But getting a good approximation should be possible. The biggest things needed are to work out how much is gained by playing (some kind of uncertainty decrease we can't see) and what the expected win% for a given point differential, which is probably only roughly possible.

[DStu]

I was thinking about taking TS, maybe with some additional degrees of freedom (don't use Gaussian measures but also try some else), and try to fit the parameters to their data.

Edit: But of course you are right, hitting theirs exactly will be hard, as there are tons of degrees of freedom, and it can well be that it's not a simple process but some patches  going on.

But it in the end it's "just" fitting two functions from \R^4->\R^2, namely
ratingUpgradeIfYouWin( yourRating, yourUncertainity, opponentRating, opponentUncertainity) -> (yourNewRating, yourNewUncertainity) and
ratungUpgradeIfYouLose( the same )

Biggest Problem is that you do not observe Rating and Uncertainity, but just the number Level(rating, uncertainity), but this should be handleable given enough data, especially if we assume that Level = Rating - a*Uncertainity for some constant a.

Edit2: And of course assuming that rating and uncertainity are the only parameters that goko uses to model your skill, but I have seen nothing that lets me think otherwise.

[ashersky]

I was thinking about taking TS, maybe with some additional degrees of freedom (don't use Gaussian measures but also try some else), and try to fit the parameters to their data.

Edit: But of course you are right, hitting theirs exactly will be hard, as there are tons of degrees of freedom, and it can well be that it's not a simple process but some patches  going on.

But it in the end it's "just" fitting two functions from \R^4->\R^2, namely
ratingUpgradeIfYouWin( yourRating, yourUncertainity, opponentRating, opponentUncertainity) -> (yourNewRating, yourNewUncertainity) and
ratungUpgradeIfYouLose( the same )

Biggest Problem is that you do not observe Rating and Uncertainity, but just the number Level(rating, uncertainity), but this should be handleable given enough data, especially if we assume that Level = Rating - a*Uncertainity for some constant a.

Edit2: And of course assuming that rating and uncertainity are the only parameters that goko uses to model your skill, but I have seen nothing that lets me think otherwise.

This is what I meant.  Whatever all that up there means.  (English majors ftw!)

I think the easiest way to collect data is two or more dummy accounts and lots of resignations.

[WanderingWinder]

To me, the biggest thing I want to know is the shape of the distribution which is underlying the expected winrates of the players, i.e. how often is someone with a 1 point rating advantage expected to win, with a 10 point edge, a 100 point edge, 1000, etc.

ragingduckd: If you have the programming skills, time, and I data I hope you do, there is a way we could get a good approximation of this. The first thing you would need to do is break down all the games in your sample by the rating difference of the players before the game (i.e. the higher-rated was at a 1 or 3, or 264 point advantage). Because their system takes on so many values, and depending on your data set, you might bucket this, so everyone from 0-4 is in the same group, 5-9 in a group, etc. 5 point buckets or 10 point or 15 or 20 or whatever your data tells you. We want to have big enough buckets to get a reasonably high sample size (I dunno, maybe 50ish games - off the top of my head) in each bucket, but we also want the buckets to be as small as possible, obviously.

Okay, then for each game in your bucket, you split it up based on who won - the higher-rated guy, or the lower-rated guy. Then, for the higher-rated guy, take all the wins, and find how many rating points he gained, on average, from winning. Then do the same in all the losses. Divide these, (average loss change divided by average win change), and that will give you the odds for the higher player to win - do they need to win 3:1 or 2:1 or whatever, in order to maintain their rating. This can be converted into an expected winning percentage.

Oh, and of course, you will want to do the same, but separately, for all the lower-rated players. For any rating difference, it should be pretty close to matching (if the higher-rated needs to win 2:1, then the lower would only need 1:2). The differing uncertainties will make this not be exactly so, but that should at least mostly wash out over a large enough sample. The increasing uncertainties is a bigger problem, but hopefully we will be able to control for that after we have this data, or anyway we would at least have an approximate shape.


Of course, this is a lot of work, so I don't expect you to do it really. But that is how you would do it, if anyone wants to.

[DG]

And remember that the rating system still has errors! I'm pretty sure that the rating change is sometimes half what it should be.

[Watno]

You mean exactly half? I wonder how they managed to program this, but it helps explaining why it can't be put into a formula.

[WanderingWinder]

And remember that the rating system still has errors! I'm pretty sure that the rating change is sometimes half what it should be.

What makes you think that?

The only bugs I can tell for sure are that it sometimes gives 0 change when it obviously ought not.

[DG]

I play a lot of 3 player games against the bots and the ranking is usually +18 to +24 for a win. Sometimes it is +9 and there's no good reason for it other than it being an error. It's been like that ever since beta. If there's one error then there might always be more as well.

[WanderingWinder]

I play a lot of 3 player games against the bots and the ranking is usually +18 to +24 for a win. Sometimes it is +9 and there's no good reason for it other than it being an error. It's been like that ever since beta. If there's one error then there might always be more as well.

Well, the 0 thing seems like a much more reasonable error to me. The +18 to +24 down to +9 there could be a lot of reasons for, depending not only on the others' ratings, but also their uncertainties. Some of this is down to how Goko is showing things. If there is someone who is rated 6000 but has 1500 points subtracted off that for uncertainty vs someone who is rated 5000 but only 500 points off for uncertainty... well, in the former case, you beat a pretty good player, and in the latter, you beat someone we're pretty sure is significantly worse than you. I'm not saying that this is right, but I wouldn't jump to thinking it had to be an error.

[DG]

The +18 to +24 down to +9 there could be a lot of reasons for, depending not only on the others' ratings, but also their uncertainties.

If there are mathematical anomalies in rating system that allow rating changes to be 100% out then that itself is a problem even it is not a bug. It will also significantly handicap any attempt to model the ratings system.

[WanderingWinder]

The +18 to +24 down to +9 there could be a lot of reasons for, depending not only on the others' ratings, but also their uncertainties.

If there are mathematical anomalies in rating system that allow rating changes to be 100% out then that itself is a problem even it is not a bug. It will also significantly handicap any attempt to model the ratings system.

ANY reasonable rating system should have 100% or more differences reasonable. If I beat Stef, I should certainly gain more than twice as many points as if I had beat the lowest ranked player on the server, no?

[DG]

I mean 100% variance between the results you get if you beat the banker bot in games in a row. That's what I'm seeing.

[WanderingWinder]

I mean 100% variance between the results you get if you beat the banker bot in games in a row. That's what I'm seeing.

The same opponent? The only issue I can see there is if, because it's a bot, it was rated much much worse the second game you played it, because it had lost a lot of games to other players in the meantime. But okay, this seems unlikely, and yeah, that is a problem. Maybe you can't write the formula down because they add a somewhat random amount, and... yeah?

[heron]

Perhaps in three player games it normally gives you +9 for each player/bot you beat, but sometimes one of the bots gets the +0 error but the other one doesn't.

[Warfreak2]

It may be weighted according to what cards are in the kingdom; some cards are higher variance than others.

[Kirian]

It may be weighted according to what cards are in the kingdom; some cards are higher variance than others.

Goko is not that smart.  Nor is that a good reason to change the rating system.

[Warfreak2]

They did say it was too complicated to write down, and it would explain why you get different rating increases from beating the same bots successively... where did I suggest changing anything?

[Kirian]

Let me rephrase: taking the board into account would be a terrible ranking system.  I guess I was implying that doing so would be a change from sanity.

[Warfreak2]

I don't see why; a rating system is supposed to predict not only which player will win, but with what probability. That probability depends on the board, since some cards create more opportunity for the weaker player to get lucky than others do (looking at you, Treasure Map and Tournament).

[WanderingWinder]

I don't think that taking the board into account would prima facie be a theoretical problem. But trying to determine what the probability for each board is - there are astronomically far too many boards out there to do an accurate job. You could do it card-by-card, influencing an overall number, but this will be only very roughly accurate, and I really don't know that it will be a help rather than a hindrance (in comparison to just finding an overall average). I mean, treasure map doesn't actually increase variance much at all I think, tournament maybe, sometimes. But how about village? On a board with no terminals, it probably increases variance. But on a board with a big-time engine, it probably decreases it. So overall?

tl;dr hypothetically fine, but no rating system is nearly this good yet, focus on getting the basics first.

[DStu]

I don't think that taking the board into account would prima facie be a theoretical problem. But trying to determine what the probability for each board is - there are astronomically far too many boards out there to do an accurate job. You could do it card-by-card, influencing an overall number, but this will be only very roughly accurate, and I really don't know that it will be a help rather than a hindrance (in comparison to just finding an overall average). I mean, treasure map doesn't actually increase variance much at all I think, tournament maybe, sometimes. But how about village? On a board with no terminals, it probably increases variance. But on a board with a big-time engine, it probably decreases it. So overall?

tl;dr hypothetically fine, but no rating system is nearly this good yet, focus on getting the basics first.

Exactly this.  You also get >200 more parameters for your model just for accounting for single-cards, which themselves are only an average over every board that features this cards.  And an average over all skill levels.

[Warfreak2]

I didn't say it would be easy to do correctly, or even sensible to try! However, it's possible Goko is trying anyway, which would explain why games between the same players, with the same result, could produce different ratings swings.

[ragingduckd]

I don't think that taking the board into account would prima facie be a theoretical problem. But trying to determine what the probability for each board is - there are astronomically far too many boards out there to do an accurate job. You could do it card-by-card, influencing an overall number, but this will be only very roughly accurate, and I really don't know that it will be a help rather than a hindrance (in comparison to just finding an overall average). I mean, treasure map doesn't actually increase variance much at all I think, tournament maybe, sometimes. But how about village? On a board with no terminals, it probably increases variance. But on a board with a big-time engine, it probably decreases it. So overall?

I started trying this, and it did indeed deliver some modest improvement in the prediction accuracy. I also got relative card variances that seemed plausible to me: Minion was noisy, Chancellor was irrelevant, etc.

I was working with Elo and adjusting the logistic curve exponent based on the kingdom variance. I tried some rather primitive ways of calculating kingdom variance from card variance: kvar = max(cvar), kvar = sqrt(mean(cvar^2)), etc -- measures that were designed to overweight the noisiest cards in the kingdom. I was measuring accuracy using binomial deviance.

The big problem and the reason I abandoned the effort was that it was awfully computationally intensive. I could only work with really small samples. There's probably some much better algorithm than my first attempt, but as you say, this is second-order stuff.

PS: I am indeed interested in the shape of the win probability curve, as per your earlier post. I haven't gotten around to doing the sort of analysis you suggest for Goko ratings, but I plan to eventually. I'll also post the data I've collected in some useable form here so others can try too. Unfortunately, I seem to have broken my database so everything else is on hold. That's also why the log search is down.