Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[ragingduckd]

I've been scraping the Goko top-100 leaderboard for a while and I've now gathered enough data to say a little about the infamous "rating drift."

Your rating changes when you play a game, of course, but Goko also updates everybody's rating simultaneously once a day. This happens around midnight EST, when their server runs a script called omgwtfwearemorons.sh

Here's what happened to the top 100 pro ratings yesterday morning:

Code: [Select]

Stef -20
Wandering Winder 0
Andrew Iannaccone -19
LESPEUTERE -20
Mic Qsenoch -24
Rabid -22
Geronimoo -22
PitrPicko -20
eliegel -22
Stealth Tomato -20
Robz888 -24
nomnomnom -48
SheCantSayNo -18
Troninho -19
Davio -71
RTT -18
Gorling -34
Lekkit -20
Fabian -22
Eevee -32
Bulec -60
flyingkuyt -20
qmech -35
GwinnR -29
kenyou2859 -29
Rene Kuroi -29
jsh357 -58
manzi -24
scott pilgrim -24
Watno -49
TrickStaR -27
Boodaloo -41
First -33
Zakharov -27
nnn -28
Twistedarcher -24
lightbulb -18
Jeebus -23
Adam Horton -33
wsc -28
kn1tt3r -47
Lagrangian -23
dudeabides -20
iriho -22
fah -28
attrill -26
majormajor -21
SM.SM -20
Darter -21
zporiri -27
theParty -20
Masschy -22
jhovall_goko -25
Dubdubdubdub -59
Titandrake -22
faw -35
yac -27
ChiChi -22
daniel greif -62
Perna -20
Dumbassador -24
YeahBoooey -25
Slyfox -21
pâté de campagne -31
yed -19
Underdog -24
nidzela -19
David Hunter -24
houroku -21
Blobby -36
loppo -31
Lotoreo -20
kilgoretrout103 -19
Emeric -21
Tombery -48
jaybeez -23
Aron35 -22
BadAMutha -25
Drab Emordnilap -82
Perry Green -24
Pex Golder -55
Tom Collett -45
LosChikitos -20
HF -25
Cruxis -26
A Drowned Kernel -28
sangatsu -21
Rhun -23
heron -27
Lukáš Černoušek -26
gagnerouperdretelleestlaquestion? -28
secret tunnel -22
Indur -26
Kirian -23
Warfreak2 -20
MarkowKette -26
hiroki -20
Alexey Rakhmanov -23
Silverfinger -26
Psyduck 128

Don't ask me why WW's rating didn't adjust that time or why Psyduck's went up... equally strange things happened to other players on other days. Anyway, with few exceptions, the changes are always negative and mostly around -25 points. Here's are the mean changes every day for the past month:

Code: [Select]

2013-05-12 -27.76
2013-05-13 -21.75
2013-05-14 -28.34
2013-05-15 -26.83
2013-05-16  01.71
2013-05-17 -28.13
2013-05-18 -27.58
2013-05-19 -29.07
2013-05-20 -29.10
2013-05-21 -28.29
2013-05-22 -27.60
2013-05-23 -27.57
2013-05-24 -26.66
2013-05-25 -27.60
2013-05-26 -26.74
2013-05-27 -27.55
2013-05-28 -26.16
2013-05-29 -27.28
2013-05-30 -27.82
2013-05-31 -24.80
2013-06-01 -08.69
2013-06-02 -26.72
2013-06-03 -26.70
2013-06-04 -31.97
2013-06-05 -24.71
2013-06-06 -26.11
2013-06-07 -25.36
2013-06-08 -26.45
2013-06-09 -25.03
2013-06-10 -26.57
2013-06-11 -27.30
2013-06-12 -26.54
2013-06-13 -26.70

Ok, sorry for all the raw numbers. Here's what this means:

First, your rating will drop like a stone if you ever take a break from Dominion. If you're currently #50 with a rating of ~6000 and you quit playing today, you'll drop out of the top 100 in about two weeks. Edit: Several responses below report drops faster than this for 1+ week of inactivity. Perhaps the drift accelerates.

Second, anyone who isn't playing a lot is dramatically underrated. This is a crude approximation, but the per-game change that you get from Goko is something like 2x the change you'd get from the de-facto standard rating system for online chess (Elo with K=32). Roughly, if you're really a 6500 player, but you only play 2-3 games a day, you can expect to hover around 6000 indefinitely. Edit: you can expect to oscillate around a mean of 6000 or so.

I can only scrape the top 100, so I don't know whether this is also happening at lower ratings. It's hard to imagine that everyone on Goko be could dropping by 25 pts a day, so maybe players with really low ratings are gaining a comparable amount. There's no relationship between rating and drift in the top 100; ratings between 5700 and 7300 all drift in about the same way.

I'm guessing that at least everyone above 5000 is seeing a daily loss. Edit: See responses below for several confirmations that ratings drop over time all the way down to 2000.

If you're interested in playing with this data, just send me a PM and I'll send you either database access or just the 16M csv file.

[Watno]

My rating drop seems to have been one of the highest. This is probably due to the fact that I didn't play on Goko at all for about a week.
So I assume how much you lose per day increases depending on how long you haven't played.

You rating changes when you play a game, of course, but Goko also updates everybody's rating simultaneously once a day. This happens around midnight EST, when their server runs a script called omgwtfwearemorons.sh

Just to make sure: The name is a joke, right?

[Blueswan]

I'm usually hovering in the 4700-5200 range and my rating also drops overnight.

[werothegreat]

I took a break for a couple months, and I dropped from 5000s to 2000s.

[serakfalcon]

Hey there, I can confirm that, in fact, below 4000 and down to 2000, the ratings still decrease over time. I got up to ~5500 when I was playing constantly (I was around lvl 30 on iso when I played constantly, lvl 26-28 when not) but after a month or so hiatus from dominion (been too busy at work) my rating has dropped. It was about 4600-4800 when I played on the 22nd of May, (I remember playing you ragingduckd, and I noticed I had dropped below your 5000+ specification after the game started) and has dropped to 3683 today, which is an average loss of 40ish points per day. Obviously we need more data but this is definitely an 1000 point loss over 20 days, which is kind of crazy

[Qvist]

+1 just for omgwtfwearemorons.sh 

Thanks for the data and I can confirm that. I was at #20 a few weeks ago, have only played a few games in the last couple of weeks, now I'm out of the Top100.

[Davio]

I only play up to 3 pro games max per evening if I even get around to that much.
Still I made it into the top 15-ish, which was quite surprising for me with so few games.

Quality over quantity, ey.

[Watno]

Don't ask me why WW's rating didn't adjust that time or why Psyduck's went up... equally strange things happened to other players on other days. Anyway, with few exceptions, the changes are always negative and mostly around -25 points.

Is it possible they simple played a game in the timeframe between your 2 checks on the leaderboard?

[Psyduck]

Don't ask me why WW's rating didn't adjust that time or why Psyduck's went up... equally strange things happened to other players on other days. Anyway, with few exceptions, the changes are always negative and mostly around -25 points.

Is it possible they simple played a game in the timeframe between your 2 checks on the leaderboard?

At least not for me. I only played one game yesterday, which must have been around 3pm EST.

[philosophyguy]

I was in the 4-5k range and stopped playing for about a week and a half over frustration with the constant freezes. I logged in today to play with Guilds, and my pro rating had dropped to around 1k. So yes, you get pummeled for not playing constantly.

[ragingduckd]

I only play up to 3 pro games max per evening if I even get around to that much.
Still I made it into the top 15-ish, which was quite surprising for me with so few games.

I think you're exactly the type of underrated infrequent player I'm talking about. You played 3 pro games per day over the last month and your median rating was 5919.

[WanderingWinder]

I didn't play at all yesterday, and I haven't for some time.

I don't think it's a terrible idea to increase rating uncertainties over time (many respectable systems do this). The real issue is that they are somehow subtracting this from your rating, and showing that as your rating. Which means this doesn't actually reflect your rating.

Something else weird is clearly going on though - I would expect some kind of reversion towards a high uncertainty, or a gradual increasing in uncertainty, but it should in some way be constant - either constant decay, or more likely the uncertainty has a constant percentage increase towards a max value or something.

I know I have *gained* 3 points overnight before, so something strange is up. I also sometimes get no rating change from winning/losing against relatively high rated guys, which makes no sense. I wonder if they re-rate things at some point, or something like that? Really strange.

Also, June 1st and especially May 16th are hugely baffling.

[michaeljb]

Too bad the rating process is too complicated to ever put in a formula; we can never know why it does this.

[andwilk]

I haven't played in about 3 weeks and my rating has dropped from right around 6000 to just above 4500.  Rating decay is way too fast in this system

[Qvist]

I haven't played in about 3 weeks and my rating has dropped from right around 6000 to just above 4500.  Rating decay is way too fast in this system

Yeah. And now I played 2 games only and went from 5500 back to around 6100, that's way too fast too. 600 points, come on!

[Tables]

I haven't played pro in a little while, and my rating has dropped down to something insane. It was about 4K after a short losing streak, now it's like 1.5K.

[Beyond Awesome]

I have noticed the rating decline as well. It is pretty big. I was around 6000-6200. But, now I find myself around 5700 these days. I did take a break for about a week, and take the occasional 3 or 4 day break.

[snorkelbike]

Same thing here. I've played just a few pro games in the last 2-3 weeks. Went from >6000 to now 4812. Not as fast a decline as some of you are reporting, but still a little bit discouraging.

[Joseph2302]

I basically didn't play pro games for about 2-3 weeks, and declined from ~4500 to ~2500. Having played about 20 pro games in the last few days, winning ~13, I have somehow got rating up to ~5400.

[Obi Wan Bonogi]

Wow, that is harsh decay. 

[Tables]

Not so much drift, but despite having played a lot of pro games in the past, my ratings now are jumping around incessantly. I won a few games and went up to about 5.5K, and about 50th on the leaderboard. Then I proceeded to lose 3 games against similarish ranked opponents and was down to just under 3000. I only realised I'd dropped so much when I tried to join a 3000+ game someone was hosting and got autokicked.


(incidentally, I feel the need to vent about two of those games... let's leave it as: open terminal/silver. Both miss the first reshuffle. Buy silver/another terminal turns 3/4. Both terminals in T5 hand. Both games. I resigned one when I was down 4-1 on a curse split, having two Sea Hags to his one, BM type boards. What great shuffling skills I have.)

[DStu]

Agree with WW. Their uncertainity seems to change much to fast.  Otherwise, it's absolutely sane to increase uncertainity for non-playing, if you haven't played for some month the system should be more unsecure on your rating.
But I would say it is a safe bet that the system should not be unsecure on your rating as it is on the rating on a new player, so there would be some canonical upper bound to converge to*.  Furthermore, you don't forget this game that fast that you are practically as hard to predict as some new player to the site.


*also of course you might have a higher mean than the average that the new player has, this is not true in the strictly mathematical provable sense, but I don't think one makes a big mistake with that.

[dnkywin]

This explains why my rating went from ~6200 to less than 4000 in May (when I didn't play). Not that I really care all that much about rating, but the most annoying part of this rating drop was that people with "5000+" in their game titles were kicking me out.

Will Goko ever release how they calculate ratings? That would be _really_ helpful.

[WanderingWinder]

Will Goko ever release how they calculate ratings? That would be _really_ helpful.

It's apparently too complicated to possibly write down.

[Kirian]

Will Goko ever release how they calculate ratings? That would be _really_ helpful.

It's apparently too complicated to possibly write down.

I imagine there's a randomization element that involves live gerbils, 23-sided dice, and the phase of the moon.