Dominion Strategy Forum

Please login or register.

Login with username, password and session length
Pages: 1 2 [3]  All

Author Topic: Bot "Pro" Games Effect Leaderboard  (Read 27107 times)

0 Members and 1 Guest are viewing this topic.

Ratsia

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 168
  • Respect: +113
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #50 on: March 02, 2015, 03:31:26 am »
0

Is that really a large issue though? I mean, if your skill really is above most rating caps, you should be able to quickly jump up in ratings by playing normal players anyways. And it is only likely to affect very new players, a lot of whom aren't skilled enough to start with to jump into games with rating caps.
The people who are clearly above most rating caps are not the ones for whom this matters. Instead, it matters for those who are probably just a bit above them but have no chance of regularly playing against humans to prove it. More generally, this is a question that should not be primarily looked at from the perspective of the absolute top players, but from the perspective of the average-quality players who are probably the ones more likely to play against the bots in significant numbers. Think about people who cap at level 20 or so in isotropish, who actually represent probably the largest mass of experienced Dominion players.

While a level ~45 player could indeed climb up to high enough rating in a few matches, it takes quite a while for a level 20-30 player who consistently plays at that level. I remember trying with another account while Isotropic was still around, and it took me roughly 50 matches to reach within 5 levels of my actual strength (25 vs 30 or so). That's quite a lot of matches for less active players, often meaning several weeks of time. Right now I almost exclusive play against bots since I cannot play matches without occasional excessive delays. Since this maintains a reasonable (though indeed a bit over-inflated, in my case probably by roughly 5 levels) rating, I can occasionally play against a human opponent and get immediately matched with roughly the right people. If it didn't, the threshold of switching to the human-opponent mode would be so high that I would never bother -- would anyway join if I hosted a game with a rating threshold of 5000+ if my rating is 1000?
« Last Edit: March 02, 2015, 03:33:43 am by Ratsia »
Logged

ragingduckd

  • Board Moderator
  • *
  • Offline Offline
  • Posts: 1059
  • Respect: +3527
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #51 on: March 03, 2015, 02:32:13 pm »
+1

I'm messing with the leaderboard... please temporarily ignore anything that looks weird.
Logged
Salvager Extension | Isotropish Leaderboard | Game Data | Log Search & other toys | Salvager Bug Reports

Salvager not working for me at all today. ... Please help! I can't go back to playing without it like an animal!

ragingduckd

  • Board Moderator
  • *
  • Offline Offline
  • Posts: 1059
  • Respect: +3527
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #52 on: March 03, 2015, 04:19:55 pm »
+5

I'm messing with the leaderboard... please temporarily ignore anything that looks weird.

All is restored.

I'm considering excluding bot games from Isotropish.  After reading this thread and comparing the two boards, I'm leaning heavily towards making the switch.  The modified leaderboard can be viewed at https://gokosalvager.com/leaderboard_nobots
Logged
Salvager Extension | Isotropish Leaderboard | Game Data | Log Search & other toys | Salvager Bug Reports

Salvager not working for me at all today. ... Please help! I can't go back to playing without it like an animal!

Mic Qsenoch

  • 2015 DS Champion
  • *
  • Offline Offline
  • Posts: 1709
  • Respect: +4329
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #53 on: March 03, 2015, 04:45:11 pm »
+1

I knew I was terrible.
Logged

jaybeez

  • Explorer
  • *****
  • Offline Offline
  • Posts: 335
  • Shuffle iT Username: jaybeez
  • Respect: +395
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #54 on: March 03, 2015, 05:28:54 pm »
0

Well the current leaderboard has me at Level 37, but the no-bots leaderboard has me at Level 38, so you know which has my vote.

Thanks for looking into this, Andrew.
Logged

qmech

  • Torturer
  • *****
  • Offline Offline
  • Posts: 1918
  • Shuffle iT Username: qmech
  • What year is it?
  • Respect: +2320
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #55 on: March 03, 2015, 05:58:49 pm »
+2

Playing nothing but bots for the last week has boosted me from level 42 to level 45.  I must be stopped.
Logged

Throwaway_bicycling

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 151
  • Respect: +140
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #56 on: March 03, 2015, 06:21:36 pm »
0

I knew I was terrible.
I knew I was...way, way, way more terrible than Mic Qsenoch. :-)

But more seriously, the new leaderboard removes almost 1400 players (!) who were above Level Zero when bot games were included, but not above that (or possibly not even eligible) with bot games removed. So, although it's possible the bots included board would be a better predictor of how you would do against bots, the human game is certainly different, and if that's what we're after here, yeah, go with that. Some day the AI might be worthy of the name, and including games against them would have some meaning, but that empirically that doesn't seem to be the case now.

Disclaimer: I lose 2.something levels but gain almost 100 places on the no bots board (and my 3sigma goes up). In some cases, that helps give the answer to "how did I beat the crap out of that Level 28 person?" when it turns out that they are, well, not as good against humans as that rating would indicate.
Logged

GeoLib

  • Jester
  • *****
  • Offline Offline
  • Posts: 965
  • Respect: +1265
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #57 on: March 03, 2015, 07:41:44 pm »
0

So I go up a level, my mu goes up by 1.5, and I jump 100 spots on the full leaderboard. You have my vote :P

In all seriousness, I'm somewhat concerned about how this will effect people's ability to break into the human game scene, but I do think it is likely to be a better predictor of skill.
Logged
"All advice is awful"
 —Count Grishnakh

Mic Qsenoch

  • 2015 DS Champion
  • *
  • Offline Offline
  • Posts: 1709
  • Respect: +4329
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #58 on: March 03, 2015, 08:23:17 pm »
+2

In all seriousness, I'm somewhat concerned about how this will effect people's ability to break into the human game scene, but I do think it is likely to be a better predictor of skill.

I don't see a lot of people putting isotropish level restrictions in their tables names (though there's some). So I wouldn't expect it to be a problem, any Pro rating restrictions are exactly the same as they were before.
Logged

Titandrake

  • Mountebank
  • *****
  • Offline Offline
  • Posts: 2210
  • Respect: +2854
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #59 on: March 04, 2015, 12:21:25 am »
0

I'm concerned about how people will manage to break into the 20s/30s as well, but if Goko continues to keep bot games for the Pro rating it'll probably be fine. When I restarted Dominion I played ~20 games against bots to get up to a high enough rating for human players. If people collectively switch to Isotropish level restrictions it could be a problem.
Logged
I have a blog! It's called Sorta Insightful. Check it out?

Beyond Awesome

  • Global Moderator
  • *****
  • Offline Offline
  • Posts: 2941
  • Shuffle iT Username: Beyond Awesome
  • Respect: +2466
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #60 on: March 04, 2015, 01:37:36 am »
0

I lose a couple of levels, but I think the switch is a good idea. After reading some of these threads, I played some bot games to see how easy it was to gain levels, and well, it's easy. Too easy. Bot ratings need to go.
Logged

Burning Skull

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1150
  • Shuffle iT Username: Burning Skull
  • See you in the Outpost
  • Respect: +1843
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #61 on: March 04, 2015, 04:12:02 am »
0

It strikes me as peculiar, I'm one level worse in no-bots version and I don't think I ever played a single game against a bot.
Or does the old version also count adventures?

qmech

  • Torturer
  • *****
  • Offline Offline
  • Posts: 1918
  • Shuffle iT Username: qmech
  • What year is it?
  • Respect: +2320
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #62 on: March 04, 2015, 05:06:44 am »
+1

If you've played opponents who have played against bots, or who have played opponents who have played against bots, or ... then you'll still feel some influence from bot games on your rating.
Logged

Throwaway_bicycling

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 151
  • Respect: +140
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #63 on: March 04, 2015, 06:20:38 pm »
0

If you've played opponents who have played against bots, or who have played opponents who have played against bots, or ... then you'll still feel some influence from bot games on your rating.
Right; this is the "How did I beat that  person 10 levels above me?" issue I noted above. The answer is that their ratings were inflated by bots. Also, I think I figured out another reason why the bot ratings were "wrong" at least with respect to the upper levels of the leaderboard despite playing a ton of games against non-bot opponents: there was a lot less connectivity between players who played bots and players who did not. Almost 1400 people dropped off the leaderboard either because they played only bots or played so poorly against people they were below level zero...that's huge.
Logged

JW

  • Jester
  • *****
  • Offline Offline
  • Posts: 968
  • Shuffle iT Username: JW
  • Respect: +1781
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #64 on: March 04, 2015, 06:32:53 pm »
0

Right; this is the "How did I beat that  person 10 levels above me?" issue I noted above. The answer is that their ratings were inflated by bots. Also, I think I figured out another reason why the bot ratings were "wrong" at least with respect to the upper levels of the leaderboard despite playing a ton of games against non-bot opponents: there was a lot less connectivity between players who played bots and players who did not. Almost 1400 people dropped off the leaderboard either because they played only bots or played so poorly against people they were below level zero...that's huge.

The main leaderboard page only shows players with games played in the past month and 0+ ranking, but the full leaderboard includes all players. Player with level less than 0 or without games in the past month are counted in the rankings, and your games against them count for the rankings, they just don't show up on the main leaderboard page. https://gokosalvager.com/leaderboard/?full=False&sortkey=level
Logged

Titandrake

  • Mountebank
  • *****
  • Offline Offline
  • Posts: 2210
  • Respect: +2854
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #65 on: March 04, 2015, 11:52:30 pm »
0

I think the biggest disconnect happened when Goko changed the front page play bots button to be Pro instead of Casual. Before, the bot ratings were determined based on those people who were willing to enter the multiplayer lobby, create a pro game, add a bot opponent, then start a game. Now, they're determined by anyone who tries out Goko for the first time, so there's a much larger pool of newbs that the bots farm pro rating from.

Or, maybe I'm remembering things wrong. But I don't remember people having this issue before the change.
Logged
I have a blog! It's called Sorta Insightful. Check it out?

Ratsia

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 168
  • Respect: +113
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #66 on: March 05, 2015, 02:07:38 am »
0

I think the biggest disconnect happened when Goko changed the front page play bots button to be Pro instead of Casual. Before, the bot ratings were determined based on those people who were willing to enter the multiplayer lobby, create a pro game, add a bot opponent, then start a game. Now, they're determined by anyone who tries out Goko for the first time, so there's a much larger pool of newbs that the bots farm pro rating from.
This sounds intuitively reasonable. However, an interesting observation is that the bot ratings are roughly the same today as they were before the change. Right now the bots are between levels 10 and 16. On August 2013, well before Play Bots switched to Pro ratings, I wrote a post stating they were between 9 and 16. Based on occasional checks every now and then, I can confirm they have remained roughly at those levels as long as the Isotropish leaderboard has existed. To conclude, the bot ratings do not look any more inflated today than they were in the past, even though they in all likelihood do play more against very bad opponents.


As someone for whom removal of bot matches from Isotropish matters more (I am not eligible for the non-bot version at all), I would vote in favor of the non-bot version since it seems to be somewhat more accurate. I still think it is crucially important Goko itself ranks matches against bots in Pro mode since having high enough rating is important when switching to play against humans, but Isotropish can well ignore the bots since those ratings are not typically used for matchmaking. Now the only effect is that if I start again playing against humans, the first 50-100 opponents I play against will get their ratings adapted somewhat unfairly as the system things they are playing against a Lvl 0 player instead of Lvl 30. This could naturally happen otherwise too if I was a truly new player and the effect is small anyways, so this is not a big deal.
Logged

Ratsia

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 168
  • Respect: +113
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #67 on: March 05, 2015, 02:13:38 am »
0

Almost 1400 people dropped off the leaderboard either because they played only bots or played so poorly against people they were below level zero...that's huge.
I would guess the main reason is the former, that majority of those really played (almost) only against bots. That's indeed a huge effect as such, but an interesting question is how many of those were ever checking the leaderboard. I think there are three possible scenarios:
(1) Majority of them never looked at Isotropish, so having those 1400 people removed from the list does not matter at all -- they won't notice
(2) Many of them did use that as their primary source of ranking information, and they will now be irrirated of being no longer ranked
(3) Many of them did use that as their primary source of ranking information, and now that they see bot games are no longer included they will switch to playing against humans instead

Switching to the non-bot leaderboard is a problem only if (2) holds, since then some subgroup of players will not anymore get a service they were enjoying. I (3) holds then it actually has a positive impact for the community as a whole.

I would be surprised if (2) holds.
Logged

qmech

  • Torturer
  • *****
  • Offline Offline
  • Posts: 1918
  • Shuffle iT Username: qmech
  • What year is it?
  • Respect: +2320
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #68 on: March 05, 2015, 04:01:09 am »
+1

A recent convert to Dominion has been practicing using the Play Bots button and was matched against Serf Bot 30 times in a row.  They eventually decided that this was because they had lost a few Casual games against friends, and that, despite counting for the Pro rankings, bot opponents are chosen using the Casual rating.  This appears to be true, as they fixed the problem by playing a few manually set up Casual games against Lord Bottington.
Logged

Throwaway_bicycling

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 151
  • Respect: +140
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #69 on: March 05, 2015, 09:05:03 am »
0

The main leaderboard page only shows players with games played in the past month and 0+ ranking, but the full leaderboard includes all players. Player with level less than 0 or without games in the past month are counted in the rankings, and your games against them count for the rankings, they just don't show up on the main leaderboard page. https://gokosalvager.com/leaderboard/?full=False&sortkey=level
To be clear...I knew that. And, needless to say, you don't get much ratings bump from playing somebody who now has a rating of, oh, -3, and heaven help you if you lose...

I think there's no question that the leaderboard needs to change this way; it leads to a much more accurate picture of relative skill, thus potentially better matches, and possibly (in the long run) more rapid improvement in play.

One adjustment to behavior this move might suggest is for people to specify ratings limits for games in terms of Isotropish levels. Right now, you do confuse people if you specify (say) "4000+" but then leave or kick them out when they turn out to be Iso Level 8 or something (I don't do that and think it's a bad idea). If you say "ISO 20+", that will confuse some people in the short-run, too, but my guess is that there really are only like 3000 or so people who play a lot on line, and a smaller number who play really a lot, and the number of people you would have to chat with to describe ISO isn't that large and would get smaller.
Logged

Throwaway_bicycling

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 151
  • Respect: +140
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #70 on: March 05, 2015, 09:23:36 am »
0

I would be surprised if (2) holds.
I agree. Most of the 1400 who disappeared won't care. Some who played more games with those 1400 will be annoyed for a minute, but won't care for very long. I think the board shift is mostly a "concern" (in terms of ratings loss) for players with Iso ratings between about 15 and 25, for reasons I will get to below.

I think the real reasons why there was a lot of bot ply by people who played Pro basically all the time and humans most of the time were:

1) Nobody seems to be joining their game and they were bored.
2) They weren't trying to juice their Isotrophish ratings, but they *were* trying to juice their Goko Pro ratings to get above (say) 4000 or 5000 or some popular ratings limit.

I suspect people not playing bots much or at all but who are between Iso 15 and 25 are possibly more likely to have been in ratings limit games with people who "bots-ed up" their ratings. And, yes, gained more points from those matches than the true skill (heh) differential would have indicated. So let's fix that.
Logged

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4381
    • View Profile
    • WanderingWinder YouTube Page
Re: Bot "Pro" Games Effect Leaderboard
« Reply #71 on: March 05, 2015, 09:27:58 am »
0

That playing bot games can inflate your rating means there is a fundamental problem with the rating system, it's not doing its job correctly. (Note, however, that just because Goko ratings don't have this problem does not necessarily mean that they ARE accurate, either). On the other hand, it's not something so simple to truly and correctly fix. Removing the bots from the pro rankings is what I would suggest, as it will at least remove one of the symptoms, and the main ones we can see, though people should be aware that it hasn't fixed the underlying problem.

DG

  • Governor
  • *****
  • Offline Offline
  • Posts: 4074
  • Respect: +2624
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #72 on: March 05, 2015, 10:02:27 am »
0

Remember that some very human factors can be involved. For example, some of the ten bot games that were removed from my pro record were played by friends who were at novice level and just trying Goko out.
Logged

Throwaway_bicycling

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 151
  • Respect: +140
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #73 on: March 08, 2015, 05:18:27 pm »
+2

That playing bot games can inflate your rating means there is a fundamental problem with the rating system, it's not doing its job correctly.
Okay, so I have gone back and forth, up and down on this, because I agree this is a troubling result. I also peeked into some of the recent stuff on chess ratings, which have had a surprisingly checkered and complicated history (I thought things were weird enough decades back when I was still playing "competitive" chess). At the very end of the day, I think the root of the issue is ultimately the one raised by Polk5440:
Quote
Also, Trueskill is designed to be very good when players of similar rankings play each other (which is why microsoft paired it with an automatch system for its games). Trueskill will be bad if players don't sufficiently mix. There aren't "enough" comparisons made and comparisons between sets of players who don't play each other become much less reliable.
I think what was really going on here was that we had was a mixture of two subpopulations of "Pro" Dominion players. One subpopulation played a really low proportion of games against bots. Another subpopulation, rich in novices, played a pretty high proportion of games against bots, and humans who played a lot of bots. Most of the novices who play bots are, not to be too judgmental, really poor players. And there are a lot of them, and, in aggregate, their role in this story is to "feed" points to the bots.

A relatively small number of players playing mostly bots and players who played bots were of somewhat higher skill, so were in turn "fed" some of the points the bots ate. Back before the "Play Bots" rating system switch, this wasn't a major problem in Pro land. But after the "Play Bots" button defaulted to Pro, there really was a big inflow of points to bots from novices now suddenly playing in "Pro" land, which were in turn "fed" to the mediocre players (including me).

Problem: this population of middling players don't mix nearly as much as you might think with stronger human players, so "The points stopped here". Without the kind of mixing Polk5440 mentions, two players that have (say) a 5000 rating could be very different in skill if one player's rating was padded by bots and the other player's wasn't. The good news, is that I think we can (and just have) "drained" the bots points out of the system, but there is still a fair amount of disconnection or lack of mixing in the system, and my hunch is that when/if there is some correction for that, there may be some downward ratings pressure on at least a subset of the (say) Top 500 or so as they lose the same proportion of games as previously to the bot-inflated players, who now have lower ratings.

In any case, it's an empirical question.
Logged
Pages: 1 2 [3]  All
 

Page created in 0.091 seconds with 23 queries.