Dominion Strategy Forum

Please login or register.

Login with username, password and session length
Pages: [1] 2 3  All

Author Topic: Bot "Pro" Games Effect Leaderboard  (Read 27105 times)

0 Members and 1 Guest are viewing this topic.

shark_bait

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1103
  • Shuffle iT Username: shark_bait
  • Luckyfin and Land of Hinter for iso aliases
  • Respect: +1868
    • View Profile
Bot "Pro" Games Effect Leaderboard
« on: September 22, 2014, 03:29:25 pm »
+4

This is a request for Salvager to automatically not count "Pro" games containing a bot opponent from the isotropish leaderboard.  Our wonderful Canton Pilots Association has been climbing due to this.  Whether intentional or not it is clear that playing bots is not an accurate way to measure skill.  Also, I want to play with the "Play Bots" button and I no longer care about my Goko rating.  I do however care about isotropish and sometimes I just want to quit vs. bots or play a 4-player bot game for fun and I don't want to go through hassle of joining a lobby when I could just click a button.
Logged
Hello.  Name's Bruce.  It's all right.  I understand.  Why trust a shark, right?

Is quite curious - Who is the mystical "Celestial Chameleon"?

Mic Qsenoch

  • 2015 DS Champion
  • *
  • Offline Offline
  • Posts: 1709
  • Respect: +4329
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #1 on: September 22, 2014, 03:39:42 pm »
+1

I'm not sure what the issue is. Unless you think the Bots isotropish ratings are majorly out of whack, this shouldn't matter. . I wouldn't call what Canton Pilots is doing unfair or anything. The Play Bots button thing I get though.

Ok, it's problematic because you can control the kingdom with the Play Bots button with Salvager, and could just play all Rats games or something. So this is probably a good idea.
Logged

blueblimp

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2849
  • Respect: +1559
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #2 on: September 22, 2014, 04:24:20 pm »
0

I'm not sure what the issue is.
Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.
Logged

Mic Qsenoch

  • 2015 DS Champion
  • *
  • Offline Offline
  • Posts: 1709
  • Respect: +4329
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #3 on: September 22, 2014, 04:33:22 pm »
0

I'm not sure what the issue is.
Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.

As I said, I understand (and agree with) the request in the context of the new "Play Bots" button, which is the convenience aspect (wanting to quit out of a quick bot game). I don't think I'll agree if people are accused of gaming the system by playing bots if they are just playing random kingdoms against the bots. As long as it's easy to make a quick Casual game against the bots (i.e. the old "Play Bots" button), I don't see any reason to not include Pro bot games in the leaderboard.
« Last Edit: September 22, 2014, 04:34:59 pm by Mic Qsenoch »
Logged

amalloy

  • Witch
  • *****
  • Offline Offline
  • Posts: 453
  • Respect: +620
    • View Profile
    • Twitch stream
Re: Bot "Pro" Games Effect Leaderboard
« Reply #4 on: September 22, 2014, 04:44:08 pm »
0

I'm not sure what the issue is. Unless you think the Bots isotropish ratings are majorly out of whack, this shouldn't matter. . I wouldn't call what Canton Pilots is doing unfair or anything. The Play Bots button thing I get though.

Ok, it's problematic because you can control the kingdom with the Play Bots button with Salvager, and could just play all Rats games or something. So this is probably a good idea.

The bots' iso levels seem pretty reasonable to me, ranging from 5 to 15, which is all in the neighborhood of "pretty bad". It looks like CPA has won 95 games, lost 10, and tied 2. Does winning 90% of games against level 15 players mean you're a level 44 player? I wouldn't think so, but I guess I don't know how trueskill works.
Logged

shark_bait

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1103
  • Shuffle iT Username: shark_bait
  • Luckyfin and Land of Hinter for iso aliases
  • Respect: +1868
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #5 on: September 22, 2014, 04:49:22 pm »
+1

I'm not sure what the issue is.
Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.

As I said, I understand (and agree with) the request in the context of the new "Play Bots" button, which is the convenience aspect (wanting to quit out of a quick bot game). I don't think I'll agree if people are accused of gaming the system by playing bots if they are just playing random kingdoms against the bots. As long as it's easy to make a quick Casual game against the bots (i.e. the old "Play Bots" button), I don't see any reason to not include Pro bot games in the leaderboard.

It is dis-proportionally easy to win bot games compared to human games.  I haven't played vs. our bot-playing CPA goko user so I don't know exactly his skill level.  However, that being said, if he were to play another 107 games against the current upper echelon of dominion players I would put down a very heavy wager that he could not maintain his current level 44.  That fact that someone can get that high by solely playing bots that are sitting around Lv. 13-14 is rather absurd.  Also, I think it there is a bit of credence to them being artificially inflated due to having an increase in "quits" against them.
Logged
Hello.  Name's Bruce.  It's all right.  I understand.  Why trust a shark, right?

Is quite curious - Who is the mystical "Celestial Chameleon"?

Mic Qsenoch

  • 2015 DS Champion
  • *
  • Offline Offline
  • Posts: 1709
  • Respect: +4329
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #6 on: September 22, 2014, 05:08:52 pm »
+4

It is dis-proportionally easy to win bot games compared to human games.  I haven't played vs. our bot-playing CPA goko user so I don't know exactly his skill level.  However, that being said, if he were to play another 107 games against the current upper echelon of dominion players I would put down a very heavy wager that he could not maintain his current level 44.  That fact that someone can get that high by solely playing bots that are sitting around Lv. 13-14 is rather absurd.  Also, I think it there is a bit of credence to them being artificially inflated due to having an increase in "quits" against them.
You could do the exact same thing by restricting games to only human players at that iso level. You would still have to maintain a ridiculously high win % to achieve that rank. As far as I can tell, Canton Pilots is demonstrating high-levels of Dominion skill. I don't find it easy at all to maintain my ranking while playing against bots, you lose a game here or there and boom your ranking gets shot for a -100 points. All the while you're gaining 7-8 points for a win. I've played a lot of bot games, I don't think they're nearly as bad as people describe them to be. They just do outrageously stupid things with certain cards.
Logged

Beyond Awesome

  • Global Moderator
  • *****
  • Offline Offline
  • Posts: 2941
  • Shuffle iT Username: Beyond Awesome
  • Respect: +2466
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #7 on: September 22, 2014, 05:30:24 pm »
0

I could play lots and lots of games against L15 players, and I am sure my rating would go up as well. Is it gaming the system by playing opponents weaker than you? I don't know the answer to that.
Logged

amalloy

  • Witch
  • *****
  • Offline Offline
  • Posts: 453
  • Respect: +620
    • View Profile
    • Twitch stream
Re: Bot "Pro" Games Effect Leaderboard
« Reply #8 on: September 22, 2014, 05:34:28 pm »
+2

I could play lots and lots of games against L15 players, and I am sure my rating would go up as well. Is it gaming the system by playing opponents weaker than you? I don't know the answer to that.

If this is really true (and I suspect it's not as true as you think), it's evidence that the rating system isn't right in general, or that it doesn't have enough games to judge ratings accurately (or some other weird thing has happened, like two divergent pockets of players with little overlap). Imagine if every active player could play a million games against every other player. Once that's done, the ratings will have converged to a point where, if you chose to play against a bunch of low-ranked players, your rating would stay exactly the same: you'd get a bunch of small boosts for most of your wins, and lose a few big chunks when you get unlucky.

So if you really think you can improve your rating by beating up on some low-level players, go for it! You'll be helping the rating system learn that the low-level players deserve to be ranked even lower than they currently are, and that you deserve to be ranked higher.
Logged

Awaclus

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 11808
  • Shuffle iT Username: Awaclus
  • (´。• ω •。`)
  • Respect: +12846
    • View Profile
    • Birds of Necama
Re: Bot "Pro" Games Effect Leaderboard
« Reply #9 on: September 22, 2014, 05:56:54 pm »
0

Well, the bots are more predictable. If you play against the bots a lot, you'll learn how to beat the bots. Beating different human players around the same skill level is much more difficult, because they all make different kinds of mistakes instead of the same mistakes over and over, and even if you manage to find another human player around that skill level who's willing to play with you hundreds of times, he is going to learn at least as fast as you are.

See, for example this game here: http://gokologs.drunkensailor.org/static/logprettifier.html?20140921/log.5086d8330cf22cec0dd1cfe9.1411361496802.txt

I think this is clear Butcher/BM with one CSM (maybe two if you hit $2 after you've already bought one). Now, I'm not the best player and my strategy could be wrong, but it would most certainly beat what Canton Pilots Association did here. Since there are no attacks, judging from how long it took him to buy the points he had by the end of the game, it looks like even a simple BMU would be competitive against Canton Pilots Association's strategy. There were other kingdoms where I think a simple BM strategy could have beat Canton Pilots Association, too. He also failed to see clear engine boards in some of those games. In other games, he played pretty well, too, so he's not the worst player ever or anything, but if he played like this against human opponents, there's just no way he could be level 44.
Logged
Bomb, Cannon, and many of the Gunpowder cards can strongly effect gameplay, particularly in a destructive way

The YouTube channel where I make musicDownload my band's Creative Commons albums for free

DG

  • Governor
  • *****
  • Offline Offline
  • Posts: 4074
  • Respect: +2624
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #10 on: September 22, 2014, 06:00:26 pm »
+1

I play against the bots multiplayer and it quite to lose by
- experimenting
- not concentrating
- misclicks
- quitting and lost connection
- falling foul of multiplayer player dynamics
- bot group think
- lack of end game control in multiplayer
- dull kingdoms, bad luck, bad turn order

Once you push your ranking up far enough you do find yourself dropping over 100 points for a loss and only gaining about 10 points for a win and it's difficult to maintain a 90% win rate in multiplayer.
« Last Edit: September 22, 2014, 06:02:28 pm by DG »
Logged

blueblimp

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2849
  • Respect: +1559
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #11 on: September 22, 2014, 06:22:12 pm »
+1

If this is really true (and I suspect it's not as true as you think), it's evidence that the rating system isn't right in general
Rating systems are never exactly right. They use simplified models of skill and performance in order to develop a system that's mathematically and computationally tractable. As far as I know, they are typically tweaked to perform best when a variety of players of roughly similar skill play each other, because those are the games that good matchmaking systems try to set up. Playing repeated games against a particular player of hugely different skill is almost certain to give a rating that isn't very good for doing ordinary matchmaking, although whether it's too high or too low depends on the rating system and the chosen parameters.
Logged

GeoLib

  • Jester
  • *****
  • Offline Offline
  • Posts: 965
  • Respect: +1265
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #12 on: September 22, 2014, 06:38:14 pm »
+10

If this is really true (and I suspect it's not as true as you think), it's evidence that the rating system isn't right in general
Rating systems are never exactly right. They use simplified models of skill and performance in order to develop a system that's mathematically and computationally tractable. As far as I know, they are typically tweaked to perform best when a variety of players of roughly similar skill play each other, because those are the games that good matchmaking systems try to set up. Playing repeated games against a particular player of hugely different skill is almost certain to give a rating that isn't very good for doing ordinary matchmaking, although whether it's too high or too low depends on the rating system and the chosen parameters.

God, if only someone were to do a comprehensive analysis of the rating system. Maybe they could do a three-part series explaining first how goko does their ratings, then maybe explain how they reverse-engineered goko's system, and in the final part do a comparison to Isotropish and a determination of the optimal trueskill parameters for isotropish to use. I don't know though. There probably wouldn't be that much interest in the last part, so maybe it's not worth ever posting it.
Logged
"All advice is awful"
 —Count Grishnakh

Holger

  • Minion
  • *****
  • Offline Offline
  • Posts: 736
  • Respect: +458
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #13 on: September 23, 2014, 11:12:31 am »
0

I'm not sure what the issue is.
Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.

As I said, I understand (and agree with) the request in the context of the new "Play Bots" button, which is the convenience aspect (wanting to quit out of a quick bot game). I don't think I'll agree if people are accused of gaming the system by playing bots if they are just playing random kingdoms against the bots. As long as it's easy to make a quick Casual game against the bots (i.e. the old "Play Bots" button), I don't see any reason to not include Pro bot games in the leaderboard.

Agreed. The problem is the possibility of choosing kingdoms for Pro games, which for whatever reason has only existed since the "Play bots" button was changed to Pro games recently. (But when I used it, the game didn't seem to count for Isotropish, only for Goko's rating.)
Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?
Logged

Awaclus

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 11808
  • Shuffle iT Username: Awaclus
  • (´。• ω •。`)
  • Respect: +12846
    • View Profile
    • Birds of Necama
Re: Bot "Pro" Games Effect Leaderboard
« Reply #14 on: September 23, 2014, 12:12:23 pm »
0

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?
No, it's very unlikely that he did, because he misplayed most of his games.
Logged
Bomb, Cannon, and many of the Gunpowder cards can strongly effect gameplay, particularly in a destructive way

The YouTube channel where I make musicDownload my band's Creative Commons albums for free

ThaddeusB

  • Young Witch
  • ****
  • Offline Offline
  • Posts: 133
  • Respect: +140
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #15 on: September 23, 2014, 01:06:21 pm »
+4

I actually find the bots are harder to beat than a human of similar ranking. That is I usually beat either, but the bot more often chooses a strategy that has some chance of success (e.g. big money) whereas the human does something resembling the correct strategy, but does it poorly. If so-so big money play beats the best strategy 20%, then the bot wins 20%. The human pursuing a poor version of the proper strategy has almost no chance, however.

I think people only tend to remember the incredibly dumb thing a bot does occasionally (that no human ever would) and forget the many, many times the bot made a decent play. (At least the boys will give themselves a chance  by buying sufficient money, which most weak humans do not do.). Incidentally, I'm pretty sure the infamous rats  bug was fixed.
Logged

shraeye

  • Minion
  • *****
  • Offline Offline
  • Posts: 690
  • Shuffle iT Username: shraeye
  • More Graph Theory please
  • Respect: +299
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #16 on: September 23, 2014, 01:17:59 pm »
+2

affect
Logged

shark_bait

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1103
  • Shuffle iT Username: shark_bait
  • Luckyfin and Land of Hinter for iso aliases
  • Respect: +1868
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #17 on: September 23, 2014, 02:02:38 pm »
0

affect

My dominion skills do not translate to grammar skills.
Logged
Hello.  Name's Bruce.  It's all right.  I understand.  Why trust a shark, right?

Is quite curious - Who is the mystical "Celestial Chameleon"?

soulnet

  • Mountebank
  • *****
  • Offline Offline
  • Posts: 2142
  • Respect: +1751
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #18 on: September 23, 2014, 07:20:26 pm »
+1

affect

My dominion skills do not translate to grammar skills.

You mean they do not effect them?
Logged

serakfalcon

  • Apprentice
  • *****
  • Offline Offline
  • Posts: 278
  • Shuffle iT Username: serakfalcon
  • Respect: +230
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #19 on: September 24, 2014, 05:28:31 am »
+1

affect

My dominion skills do not translate to grammar skills.

You mean they do not effect them?

Did you do that on porpoise?
Logged

Holger

  • Minion
  • *****
  • Offline Offline
  • Posts: 736
  • Respect: +458
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #20 on: September 24, 2014, 06:00:09 am »
0

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?
No, it's very unlikely that he did, because he misplayed most of his games.

I don't think he'd get to level 47 (and a μ slightly higher than Stef's) by misplaying most of his games?!
That said, I also didn't see a pattern of specific kingdoms in his last bot games.
Logged

Awaclus

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 11808
  • Shuffle iT Username: Awaclus
  • (´。• ω •。`)
  • Respect: +12846
    • View Profile
    • Birds of Necama
Re: Bot "Pro" Games Effect Leaderboard
« Reply #21 on: September 24, 2014, 06:17:18 am »
0

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?
No, it's very unlikely that he did, because he misplayed most of his games.

I don't think he'd get to level 47 (and a μ slightly higher than Stef's) by misplaying most of his games?!
That said, I also didn't see a pattern of specific kingdoms in his last bot games.
Not against real players, but against bots, you can misplay and win inspite of it (or even because of it). Take a look at the game I linked, it's one of the worst but not the only one. Seriously, an opponent buying nothing but Silvers, Golds and green cards could beat him in that game with some decent luck, and it is not a small oversight even a top player could do sometimes, it's going for Philosopher's Stone when there's a fast big money strategy with support in the kingdom.
Logged
Bomb, Cannon, and many of the Gunpowder cards can strongly effect gameplay, particularly in a destructive way

The YouTube channel where I make musicDownload my band's Creative Commons albums for free

Holger

  • Minion
  • *****
  • Offline Offline
  • Posts: 736
  • Respect: +458
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #22 on: September 24, 2014, 07:15:49 am »
0

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?
No, it's very unlikely that he did, because he misplayed most of his games.

I don't think he'd get to level 47 (and a μ slightly higher than Stef's) by misplaying most of his games?!
That said, I also didn't see a pattern of specific kingdoms in his last bot games.
Not against real players, but against bots, you can misplay and win inspite of it (or even because of it). Take a look at the game I linked, it's one of the worst but not the only one. Seriously, an opponent buying nothing but Silvers, Golds and green cards could beat him in that game with some decent luck, and it is not a small oversight even a top player could do sometimes, it's going for Philosopher's Stone when there's a fast big money strategy with support in the kingdom.

Agreed about this game; but usually (in the absence of PS ;-)) the (non-serf) bots play better than pure BM. Banker Bot would probably have played Butcher BM on that very board. And CPA must have had something like a 99% win rate against bots to get this high a rating. You don't get that by regularly playing as bad as BM.
However, CPA is only at #132 on Goko's Pro leaderboard; so maybe there's something wrong with Isotropish's rating here?

Edit: From the log search, it seems CPA only won ~90% of his games against bots (which is still the win rate of Rebuild BM against pure BM, so you can't expect much better against decent opponents). Still, you shouldn't be able to reach the top 5 with that performance, I think...
« Last Edit: September 24, 2014, 07:27:09 am by Holger »
Logged

SCSN

  • Mountebank
  • *****
  • Offline Offline
  • Posts: 2227
  • Respect: +7140
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #23 on: September 24, 2014, 08:11:47 am »
+5

This is just selection bias: you won't notice the hundreds of guys who play a bunch against bots and achieve nothing spectacular, but you will notice the one guy who runs extremely hot. I predict he'll be nowhere near his current level after 110 more games.

Trusting the description of his play in this thread is accurate and taking into account the very low prior, it's extremely unlikely he's actually very good. Moreover, I've actually proofread the 3rd Rating Analysis article many months ago, and one of its conclusions was that in the long run it's impossible to maintain your rating when playing much lower ranked opposition. This happens because both Goko and to a lesser extent Iso underappreciate the amount of randomness in Dominion and thus give disproportional weights to upsets.

I see there's still interest in this matter; AI is extremely busy right now, but I'll ask him if he's willing to post it when he has a little more time (I don't think he's reading this thread).
Logged

liopoil

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2587
  • Respect: +2479
    • View Profile
Re: Bot "Pro" Games Effect Leaderboard
« Reply #24 on: September 24, 2014, 08:14:58 am »
0

Yeah, I feel like I lose to people much lower rated than me and beat people much higher rated than me much more often then I should.
Logged
Pages: [1] 2 3  All
 

Page created in 0.06 seconds with 21 queries.