Dominion Strategy Forum

Please login or register.

Login with username, password and session length
Pages: [1] 2  All

Author Topic: There IS Something Off About the Ratings  (Read 7643 times)

0 Members and 1 Guest are viewing this topic.

Robz888

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2644
  • Shuffle iT Username: Robz888
  • Respect: +3391
    • View Profile
There IS Something Off About the Ratings
« on: May 30, 2013, 12:16:55 pm »
0

I am currently hovering around 6200.

When I play mostly people who are 5500 and up, I steadily gain points, and have risen as high as number 9 on the leaderboard, around the 6500 mark.

When I play mostly people who are between 4500 and 5000 (which is easier, since there are way more of them), I steadily lose points. I can NEVER pull ahead, net-points wise, by playing people in this range, but I can if I play much higher level people.

Under a truly accurate rating system, this shouldn't be the case, right? I guess it could be that I am for some reason anomalously bad at beating lower ranked opponents, but I don't see why that would be true...
Logged
I have been forced to accept that lackluster play is a town tell for you.

Robz888

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2644
  • Shuffle iT Username: Robz888
  • Respect: +3391
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #1 on: May 30, 2013, 12:18:18 pm »
0

I mention this because it's happened several times now, where I was on at the same time as high level opponents, and I rapidly rose up the leaderboard. And then the next day I could finally only mostly 4500ish people, and my level plummets.
Logged
I have been forced to accept that lackluster play is a town tell for you.

greatexpectations

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1097
  • Respect: +1067
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #2 on: May 30, 2013, 12:29:33 pm »
+1

did you log any data to back this up? far too easy to misremember anecdotes.
Logged
momomoto: ...I looked at the tableau and went "Mountebank? That's for jerks."
rrenaud: Jerks win.

Robz888

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2644
  • Shuffle iT Username: Robz888
  • Respect: +3391
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #3 on: May 30, 2013, 12:41:26 pm »
0

did you log any data to back this up? far too easy to misremember anecdotes.

Well no.
Logged
I have been forced to accept that lackluster play is a town tell for you.

greatexpectations

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1097
  • Respect: +1067
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #4 on: May 30, 2013, 12:51:12 pm »
+1

Well no.

not trying to be a jerk. mostly anyway. i can easily believe that their system is flawed but without legit data to prove it we aren't likely to see any change or improvement. especially when it isn't likely to be a priority item. what we really would need is some kind of CR type system where we could monitor skill levels and results to pseudo reverse engineer their formulas.
Logged
momomoto: ...I looked at the tableau and went "Mountebank? That's for jerks."
rrenaud: Jerks win.

Robz888

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2644
  • Shuffle iT Username: Robz888
  • Respect: +3391
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #5 on: May 30, 2013, 12:54:51 pm »
0

Oh, I know you weren't being a jerk. I wish I had documented evidence of it. All I can say is that I've noticed it three times, the rise from playing higher ranking people, and the stark drop from playing lower ranked people.
Logged
I have been forced to accept that lackluster play is a town tell for you.

Watno

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2745
  • Shuffle iT Username: Watno
  • Respect: +2983
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #6 on: May 30, 2013, 03:57:21 pm »
0

Well no.

not trying to be a jerk. mostly anyway. i can easily believe that their system is flawed but without legit data to prove it we aren't likely to see any change or improvement. especially when it isn't likely to be a priority item. what we really would need is some kind of CR type system where we could monitor skill levels and results to pseudo reverse engineer their formulas.

We'd never be able to do that. They're formulae are too complicated for us to understand, didn't you know?
Logged

florrat

  • Minion
  • *****
  • Offline Offline
  • Posts: 542
  • Shuffle iT Username: florrat
  • Respect: +748
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #7 on: May 30, 2013, 05:39:38 pm »
+1

We'd never be able to do that. They're formulae are too complicated for us to understand, didn't you know?
Indeed. And didn't you know that the rating system is too complicated to put in a formula at all?
Logged

Watno

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2745
  • Shuffle iT Username: Watno
  • Respect: +2983
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #8 on: May 30, 2013, 05:41:58 pm »
0

That's what I meant.
Logged

shMerker

  • Duke
  • *****
  • Offline Offline
  • Posts: 357
  • Respect: +389
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #9 on: May 30, 2013, 05:45:29 pm »
0

The Goko UI encourages you to look at your rating as points you are gaining and losing by displaying a big +/-X next to your rating after every game. This is a visual cue that you're supposed to care how much you're going up or down by.
Logged
"I take no responsibility whatsoever for those who get dizzy and pass out from running around this post."

Stealth Tomato

  • Minion
  • *****
  • Offline Offline
  • Posts: 507
  • Dorkneel
  • Respect: +480
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #10 on: May 31, 2013, 03:21:16 pm »
0

I am currently hovering around 6200.

When I play mostly people who are 5500 and up, I steadily gain points, and have risen as high as number 9 on the leaderboard, around the 6500 mark.

When I play mostly people who are between 4500 and 5000 (which is easier, since there are way more of them), I steadily lose points. I can NEVER pull ahead, net-points wise, by playing people in this range, but I can if I play much higher level people.

Under a truly accurate rating system, this shouldn't be the case, right? I guess it could be that I am for some reason anomalously bad at beating lower ranked opponents, but I don't see why that would be true...

Of note, this is perfectly possible in a reasonable system. I would wager that my ranking tends to increase against 4500-5500 level guys, and decrease against 6000+. I'm good at turning my incremental advantages over these players into victories, whereas better players tend to win against me because they're quite simply better.

Perhaps you just need to work on a different set of skills in order to defeat these players more consistently.
Logged

dudeabides

  • Young Witch
  • ****
  • Offline Offline
  • Posts: 138
  • Shuffle iT Username: dudeabides
  • Respect: +136
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #11 on: May 31, 2013, 06:06:02 pm »
+1

Under a truly accurate rating system, this shouldn't be the case, right? I guess it could be that I am for some reason anomalously bad at beating lower ranked opponents, but I don't see why that would be true...

I noticed the exact same thing, when I went from 6,200 to 5,600 in no time by playing players in lower ranges.

I personally feel like I play much better Dominion against high-ranked players.  It's totally psychological; I know that I have to play better against Stef, for example, if I want to win.  So I think about my play more than when I play players in the 4,500-5,500 range.  I am more likely to watch new episodes of Arrested Development while playing or get creative with my play against a lower ranked player.

I also think that players in the 4,500-5,500 range are more likely to play suboptimal strategies with lower expected outcomes but higher variances, so the chance that you lose against them is higher than it should be according to the distribution assumed by the ranking algorithm.  Of course, this assumes the ranking algorithm assumes a distribution--not an innocuous assumption.
« Last Edit: May 31, 2013, 06:08:10 pm by dudeabides »
Logged
jsh357: the dude doesn't need to explain himself

Titandrake

  • Mountebank
  • *****
  • Offline Offline
  • Posts: 2210
  • Respect: +2856
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #12 on: May 31, 2013, 06:09:18 pm »
0

Under a truly accurate rating system, this shouldn't be the case, right? I guess it could be that I am for some reason anomalously bad at beating lower ranked opponents, but I don't see why that would be true...

I noticed the exact same thing, when I went from 6,200 to 5,600 in no time by playing players in lower ranges.

I personally feel like I play much better Dominion against high-ranked players.  It's totally psychological; I know that I have to play better against Stef, for example, if I want to win.  So I think about my play more than when I play players in the 4,500-5,500 range.  I am more likely to watch new episodes of Arrested Development while playing or get creative with my play against a lower ranked player.

I also think that players in the 4,500-5,500 range are more likely to play suboptimal strategies with lower expected outcomes but higher variances, so the chance that you lose against them is higher than it should be according to the distribution assumed by the ranking algorithm.  Of course, this assumes the ranking algorithm assumes a distribution--not an innocuous assumption.

I found this to be true on Iso as well. I played the best when going against someone considerably above what my Iso ranking said, because it felt like I needed to do that to have a shot at winning.
Logged
I have a blog! It's called Sorta Insightful. Check it out?

dondon151

  • 2012 US Champion
  • *
  • Offline Offline
  • Posts: 2522
  • Respect: +1856
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #13 on: May 31, 2013, 06:16:21 pm »
+2

I guess I am different from all of you then, because I mercilessly try to crush newbies who wander into the mouth of my automatch.
Logged

eHalcyon

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 8689
  • Respect: +9187
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #14 on: May 31, 2013, 06:42:42 pm »
+3

I guess I am different from all of you then, because I mercilessly try to crush newbies who wander into the mouth of my automatch.
"Oh, you thought this was A GAME?"
Logged

dondon151

  • 2012 US Champion
  • *
  • Offline Offline
  • Posts: 2522
  • Respect: +1856
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #15 on: May 31, 2013, 07:19:40 pm »
0

"Oh, you thought this was A GAME?"

You owe me 10 ccs of blood per VP differential at game end.
Logged

ragingduckd

  • Board Moderator
  • *
  • Offline Offline
  • Posts: 1059
  • Respect: +3527
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #16 on: May 31, 2013, 08:57:43 pm »
0

"Oh, you thought this was A GAME?"

You owe me 10 ccs of blood per VP differential at game end.

...and we're playing with KC, Goons, and Masquerade.
Logged
Salvager Extension | Isotropish Leaderboard | Game Data | Log Search & other toys | Salvager Bug Reports

Salvager not working for me at all today. ... Please help! I can't go back to playing without it like an animal!

Awaclus

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 11817
  • Shuffle iT Username: Awaclus
  • (´。• ω •。`)
  • Respect: +12870
    • View Profile
    • Birds of Necama
Re: There IS Something Off About the Ratings
« Reply #17 on: June 01, 2013, 08:18:34 am »
0

I personally feel like I play much better Dominion against high-ranked players.  It's totally psychological; I know that I have to play better against Stef, for example, if I want to win.  So I think about my play more than when I play players in the 4,500-5,500 range.  I am more likely to watch new episodes of Arrested Development while playing or get creative with my play against a lower ranked player.
It's probably not totally psychological, because Stef plays faster and makes better decisions than players in the 4,5k-5,5k range (at least usually - surely there are 4,5ks who play faster than Stef and misclicks happen for Stef, too). At least for me, it's easier to focus on the game when my opponent isn't sleeping on his turn and it's difficult to predict bad players while it is easy to predict good players (not always possible, but at least you can expect them to know about the PPR and other things) and play accordingly. Okay, 4,5k-5,5ks aren't bad bad, I'm hardly better than 5,5k myself, but you get the point.
« Last Edit: June 01, 2013, 08:21:23 am by Awaclus »
Logged
Bomb, Cannon, and many of the Gunpowder cards can strongly effect gameplay, particularly in a destructive way

The YouTube channel where I make musicDownload my band's Creative Commons albums for free

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4386
    • View Profile
    • WanderingWinder YouTube Page
Re: There IS Something Off About the Ratings
« Reply #18 on: June 03, 2013, 03:06:33 pm »
0

Stealth Tomato correctly points out that this is possible in a good system - your skillset can be particularly good at crushing bunnies or maybe it's better at hanging with very strong players. I tend to be more impressed with the latter than the former, but well, that's not really the point.

The point is that if the system is good, this phenomenon shouldn't be the same for all players - some are going to do better by playing against strong, others against weak, and on average, i.e. for the average player, this should balance out. It is my experience that goko has this wrong, and that most everyone would have the same experience as Robz888, but on the other hand, I felt the same way about iso.

Stealth Tomato

  • Minion
  • *****
  • Offline Offline
  • Posts: 507
  • Dorkneel
  • Respect: +480
    • View Profile
Re: There IS Something Off About the Ratings
« Reply #19 on: June 03, 2013, 05:19:51 pm »
0

Stealth Tomato correctly points out that this is possible in a good system - your skillset can be particularly good at crushing bunnies or maybe it's better at hanging with very strong players. I tend to be more impressed with the latter than the former, but well, that's not really the point.

So am I, which is why I'm fairly certain I'm no longer truly an elite player.
Logged

Warfreak2

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1149
  • KC->KC->[Scavenger, Scavenger, Lookout]
  • Respect: +1324
    • View Profile
    • Music what I do
Re: There IS Something Off About the Ratings
« Reply #20 on: June 03, 2013, 06:27:23 pm »
0

This is probably because of those games where someone 2000 points weaker opens Treasure Map and activates them on turn 5, and you didn't get to $5 until turn 8. &c. That's extreme, but luck is always a factor, whether it's something extreme like your Sea Hag getting Sea Hagged twice, or something else like not hitting $5 on the first reshuffle.

It's hard to account for that baseline of randomness where certain boards can let a much weaker player win just because of luck and first turn advantage, when other boards are very complex and the stronger player has lots of opportunity to take the lead. Theoretically, the rating differences could be weighted according to how swingy the chosen cards were, or what opening splits each player had... that sounds like it would be an inadequate mess, though.
« Last Edit: June 03, 2013, 06:28:54 pm by Warfreak2 »
Logged
If the only engine on the board is Procession->Conspirator, I will play it.

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4386
    • View Profile
    • WanderingWinder YouTube Page
Re: There IS Something Off About the Ratings
« Reply #21 on: June 03, 2013, 06:30:08 pm »
0

It's hard to account for that baseline of randomness where certain boards can let a much weaker player win just because of luck and first turn advantage,

Actually, you just need a different win expectancy curve - it's not that hard at all. Well, it's relative - the more accurate you need it to be, the harder it is. But it's certainly doable - this isn't some extra-special-hard thing about Dominion or anything.

Warfreak2

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1149
  • KC->KC->[Scavenger, Scavenger, Lookout]
  • Respect: +1324
    • View Profile
    • Music what I do
Re: There IS Something Off About the Ratings
« Reply #22 on: June 03, 2013, 06:50:36 pm »
0

My point was that said baseline Depends On The Board, so your rating system needs to know what boards are swingy and how much.
Logged
If the only engine on the board is Procession->Conspirator, I will play it.

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4386
    • View Profile
    • WanderingWinder YouTube Page
Re: There IS Something Off About the Ratings
« Reply #23 on: June 03, 2013, 06:52:28 pm »
+2

My point was that said baseline Depends On The Board, so your rating system needs to know what boards are swingy and how much.
Well, yes and no. Really, it's just that part of the luck is what board you get. You can still find what it is in general, how much spread there typically is, etc. etc.

Warfreak2

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1149
  • KC->KC->[Scavenger, Scavenger, Lookout]
  • Respect: +1324
    • View Profile
    • Music what I do
Re: There IS Something Off About the Ratings
« Reply #24 on: June 03, 2013, 07:04:17 pm »
0

Also it may be that you can adjust the curve to account for that, but that they just didn't. We may never know!
Logged
If the only engine on the board is Procession->Conspirator, I will play it.
Pages: [1] 2  All
 

Page created in 2.419 seconds with 20 queries.