Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[shark_bait]

This is a request for Salvager to automatically not count "Pro" games containing a bot opponent from the isotropish leaderboard.  Our wonderful Canton Pilots Association has been climbing due to this.  Whether intentional or not it is clear that playing bots is not an accurate way to measure skill.  Also, I want to play with the "Play Bots" button and I no longer care about my Goko rating.  I do however care about isotropish and sometimes I just want to quit vs. bots or play a 4-player bot game for fun and I don't want to go through hassle of joining a lobby when I could just click a button.

[Mic Qsenoch]

I'm not sure what the issue is. Unless you think the Bots isotropish ratings are majorly out of whack, this shouldn't matter. . I wouldn't call what Canton Pilots is doing unfair or anything. The Play Bots button thing I get though.

Ok, it's problematic because you can control the kingdom with the Play Bots button with Salvager, and could just play all Rats games or something. So this is probably a good idea.

[blueblimp]

I'm not sure what the issue is.

Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.

[Mic Qsenoch]

I'm not sure what the issue is.

Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.

As I said, I understand (and agree with) the request in the context of the new "Play Bots" button, which is the convenience aspect (wanting to quit out of a quick bot game). I don't think I'll agree if people are accused of gaming the system by playing bots if they are just playing random kingdoms against the bots. As long as it's easy to make a quick Casual game against the bots (i.e. the old "Play Bots" button), I don't see any reason to not include Pro bot games in the leaderboard.

[amalloy]

I'm not sure what the issue is. Unless you think the Bots isotropish ratings are majorly out of whack, this shouldn't matter. . I wouldn't call what Canton Pilots is doing unfair or anything. The Play Bots button thing I get though.

Ok, it's problematic because you can control the kingdom with the Play Bots button with Salvager, and could just play all Rats games or something. So this is probably a good idea.

The bots' iso levels seem pretty reasonable to me, ranging from 5 to 15, which is all in the neighborhood of "pretty bad". It looks like CPA has won 95 games, lost 10, and tied 2. Does winning 90% of games against level 15 players mean you're a level 44 player? I wouldn't think so, but I guess I don't know how trueskill works.

[shark_bait]

I'm not sure what the issue is.

Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.

As I said, I understand (and agree with) the request in the context of the new "Play Bots" button, which is the convenience aspect (wanting to quit out of a quick bot game). I don't think I'll agree if people are accused of gaming the system by playing bots if they are just playing random kingdoms against the bots. As long as it's easy to make a quick Casual game against the bots (i.e. the old "Play Bots" button), I don't see any reason to not include Pro bot games in the leaderboard.

It is dis-proportionally easy to win bot games compared to human games.  I haven't played vs. our bot-playing CPA goko user so I don't know exactly his skill level.  However, that being said, if he were to play another 107 games against the current upper echelon of dominion players I would put down a very heavy wager that he could not maintain his current level 44.  That fact that someone can get that high by solely playing bots that are sitting around Lv. 13-14 is rather absurd.  Also, I think it there is a bit of credence to them being artificially inflated due to having an increase in "quits" against them.

[Mic Qsenoch]

It is dis-proportionally easy to win bot games compared to human games.  I haven't played vs. our bot-playing CPA goko user so I don't know exactly his skill level.  However, that being said, if he were to play another 107 games against the current upper echelon of dominion players I would put down a very heavy wager that he could not maintain his current level 44.  That fact that someone can get that high by solely playing bots that are sitting around Lv. 13-14 is rather absurd.  Also, I think it there is a bit of credence to them being artificially inflated due to having an increase in "quits" against them.

You could do the exact same thing by restricting games to only human players at that iso level. You would still have to maintain a ridiculously high win % to achieve that rank. As far as I can tell, Canton Pilots is demonstrating high-levels of Dominion skill. I don't find it easy at all to maintain my ranking while playing against bots, you lose a game here or there and boom your ranking gets shot for a -100 points. All the while you're gaining 7-8 points for a win. I've played a lot of bot games, I don't think they're nearly as bad as people describe them to be. They just do outrageously stupid things with certain cards.

[Beyond Awesome]

I could play lots and lots of games against L15 players, and I am sure my rating would go up as well. Is it gaming the system by playing opponents weaker than you? I don't know the answer to that.

[amalloy]

I could play lots and lots of games against L15 players, and I am sure my rating would go up as well. Is it gaming the system by playing opponents weaker than you? I don't know the answer to that.

If this is really true (and I suspect it's not as true as you think), it's evidence that the rating system isn't right in general, or that it doesn't have enough games to judge ratings accurately (or some other weird thing has happened, like two divergent pockets of players with little overlap). Imagine if every active player could play a million games against every other player. Once that's done, the ratings will have converged to a point where, if you chose to play against a bunch of low-ranked players, your rating would stay exactly the same: you'd get a bunch of small boosts for most of your wins, and lose a few big chunks when you get unlucky.

So if you really think you can improve your rating by beating up on some low-level players, go for it! You'll be helping the rating system learn that the low-level players deserve to be ranked even lower than they currently are, and that you deserve to be ranked higher.

[Awaclus]

Well, the bots are more predictable. If you play against the bots a lot, you'll learn how to beat the bots. Beating different human players around the same skill level is much more difficult, because they all make different kinds of mistakes instead of the same mistakes over and over, and even if you manage to find another human player around that skill level who's willing to play with you hundreds of times, he is going to learn at least as fast as you are.

See, for example this game here: http://gokologs.drunkensailor.org/static/logprettifier.html?20140921/log.5086d8330cf22cec0dd1cfe9.1411361496802.txt

I think this is clear Butcher/BM with one CSM (maybe two if you hit $2 after you've already bought one). Now, I'm not the best player and my strategy could be wrong, but it would most certainly beat what Canton Pilots Association did here. Since there are no attacks, judging from how long it took him to buy the points he had by the end of the game, it looks like even a simple BMU would be competitive against Canton Pilots Association's strategy. There were other kingdoms where I think a simple BM strategy could have beat Canton Pilots Association, too. He also failed to see clear engine boards in some of those games. In other games, he played pretty well, too, so he's not the worst player ever or anything, but if he played like this against human opponents, there's just no way he could be level 44.

[DG]

I play against the bots multiplayer and it quite to lose by
- experimenting
- not concentrating
- misclicks
- quitting and lost connection
- falling foul of multiplayer player dynamics
- bot group think
- lack of end game control in multiplayer
- dull kingdoms, bad luck, bad turn order

Once you push your ranking up far enough you do find yourself dropping over 100 points for a loss and only gaining about 10 points for a win and it's difficult to maintain a 90% win rate in multiplayer.

[blueblimp]

If this is really true (and I suspect it's not as true as you think), it's evidence that the rating system isn't right in general

Rating systems are never exactly right. They use simplified models of skill and performance in order to develop a system that's mathematically and computationally tractable. As far as I know, they are typically tweaked to perform best when a variety of players of roughly similar skill play each other, because those are the games that good matchmaking systems try to set up. Playing repeated games against a particular player of hugely different skill is almost certain to give a rating that isn't very good for doing ordinary matchmaking, although whether it's too high or too low depends on the rating system and the chosen parameters.

[GeoLib]

If this is really true (and I suspect it's not as true as you think), it's evidence that the rating system isn't right in general

Rating systems are never exactly right. They use simplified models of skill and performance in order to develop a system that's mathematically and computationally tractable. As far as I know, they are typically tweaked to perform best when a variety of players of roughly similar skill play each other, because those are the games that good matchmaking systems try to set up. Playing repeated games against a particular player of hugely different skill is almost certain to give a rating that isn't very good for doing ordinary matchmaking, although whether it's too high or too low depends on the rating system and the chosen parameters.

God, if only someone were to do a comprehensive analysis of the rating system. Maybe they could do a three-part series explaining first how goko does their ratings, then maybe explain how they reverse-engineered goko's system, and in the final part do a comparison to Isotropish and a determination of the optimal trueskill parameters for isotropish to use. I don't know though. There probably wouldn't be that much interest in the last part, so maybe it's not worth ever posting it.

[Holger]

I'm not sure what the issue is.

Well IMO the purpose of the Goko bots is not to provide a competitive experience but rather to provide a quick and easy way to play a game with no commitment. If quitting vs bots distorts the quality of matchmaking with human players, that's a problem.

I don't know of _any_ game where the built-in convenience AIs are counted on the leaderboard. I think there are some games where AIs play for competitive rating (Go?), but in those cases, the AIs are being developed as competitive AIs, which is a totally different situation.

As I said, I understand (and agree with) the request in the context of the new "Play Bots" button, which is the convenience aspect (wanting to quit out of a quick bot game). I don't think I'll agree if people are accused of gaming the system by playing bots if they are just playing random kingdoms against the bots. As long as it's easy to make a quick Casual game against the bots (i.e. the old "Play Bots" button), I don't see any reason to not include Pro bot games in the leaderboard.

Agreed. The problem is the possibility of choosing kingdoms for Pro games, which for whatever reason has only existed since the "Play bots" button was changed to Pro games recently. (But when I used it, the game didn't seem to count for Isotropish, only for Goko's rating.)
Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?

[Awaclus]

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?

No, it's very unlikely that he did, because he misplayed most of his games.

[ThaddeusB]

I actually find the bots are harder to beat than a human of similar ranking. That is I usually beat either, but the bot more often chooses a strategy that has some chance of success (e.g. big money) whereas the human does something resembling the correct strategy, but does it poorly. If so-so big money play beats the best strategy 20%, then the bot wins 20%. The human pursuing a poor version of the proper strategy has almost no chance, however.

I think people only tend to remember the incredibly dumb thing a bot does occasionally (that no human ever would) and forget the many, many times the bot made a decent play. (At least the boys will give themselves a chance  by buying sufficient money, which most weak humans do not do.). Incidentally, I'm pretty sure the infamous rats  bug was fixed.

[shraeye]

affect

[shark_bait]

affect

My dominion skills do not translate to grammar skills.

[soulnet]

affect

My dominion skills do not translate to grammar skills.

You mean they do not effect them?

[serakfalcon]

affect

My dominion skills do not translate to grammar skills.

You mean they do not effect them?

Did you do that on porpoise?

[Holger]

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?

No, it's very unlikely that he did, because he misplayed most of his games.

I don't think he'd get to level 47 (and a μ slightly higher than Stef's) by misplaying most of his games?!
That said, I also didn't see a pattern of specific kingdoms in his last bot games.

[Awaclus]

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?

No, it's very unlikely that he did, because he misplayed most of his games.

I don't think he'd get to level 47 (and a μ slightly higher than Stef's) by misplaying most of his games?!
That said, I also didn't see a pattern of specific kingdoms in his last bot games.

Not against real players, but against bots, you can misplay and win inspite of it (or even because of it). Take a look at the game I linked, it's one of the worst but not the only one. Seriously, an opponent buying nothing but Silvers, Golds and green cards could beat him in that game with some decent luck, and it is not a small oversight even a top player could do sometimes, it's going for Philosopher's Stone when there's a fast big money strategy with support in the kingdom.

[Holger]

Is there evidence that this Canton Pilots Association did select specific kingdoms regularly?

No, it's very unlikely that he did, because he misplayed most of his games.

I don't think he'd get to level 47 (and a μ slightly higher than Stef's) by misplaying most of his games?!
That said, I also didn't see a pattern of specific kingdoms in his last bot games.

Not against real players, but against bots, you can misplay and win inspite of it (or even because of it). Take a look at the game I linked, it's one of the worst but not the only one. Seriously, an opponent buying nothing but Silvers, Golds and green cards could beat him in that game with some decent luck, and it is not a small oversight even a top player could do sometimes, it's going for Philosopher's Stone when there's a fast big money strategy with support in the kingdom.

Agreed about this game; but usually (in the absence of PS ;-)) the (non-serf) bots play better than pure BM. Banker Bot would probably have played Butcher BM on that very board. And CPA must have had something like a 99% win rate against bots to get this high a rating. You don't get that by regularly playing as bad as BM.
However, CPA is only at #132 on Goko's Pro leaderboard; so maybe there's something wrong with Isotropish's rating here?

Edit: From the log search, it seems CPA only won ~90% of his games against bots (which is still the win rate of Rebuild BM against pure BM, so you can't expect much better against decent opponents). Still, you shouldn't be able to reach the top 5 with that performance, I think...

[SCSN]

This is just selection bias: you won't notice the hundreds of guys who play a bunch against bots and achieve nothing spectacular, but you will notice the one guy who runs extremely hot. I predict he'll be nowhere near his current level after 110 more games.

Trusting the description of his play in this thread is accurate and taking into account the very low prior, it's extremely unlikely he's actually very good. Moreover, I've actually proofread the 3rd Rating Analysis article many months ago, and one of its conclusions was that in the long run it's impossible to maintain your rating when playing much lower ranked opposition. This happens because both Goko and to a lesser extent Iso underappreciate the amount of randomness in Dominion and thus give disproportional weights to upsets.

I see there's still interest in this matter; AI is extremely busy right now, but I'll ask him if he's willing to post it when he has a little more time (I don't think he's reading this thread).

[liopoil]

Yeah, I feel like I lose to people much lower rated than me and beat people much higher rated than me much more often then I should.

[Polk5440]

This is just selection bias: you won't notice the hundreds of guys who play a bunch against bots and achieve nothing spectacular, but you will notice the one guy who runs extremely hot. I predict he'll be nowhere near his current level after 110 more games.

Trusting the description of his play in this thread is accurate and taking into account the very low prior, it's extremely unlikely he's actually very good. Moreover, I've actually proofread the 3rd Rating Analysis article many months ago, and one of its conclusions was that in the long run it's impossible to maintain your rating when playing much lower ranked opposition. This happens because both Goko and to a lesser extent Iso underappreciate the amount of randomness in Dominion and thus give disproportional weights to upsets.

I see there's still interest in this matter; AI is extremely busy right now, but I'll ask him if he's willing to post it when he has a little more time (I don't think he's reading this thread).

Also, Trueskill is designed to be very good when players of similar rankings play eachother (which is why microsoft paired it with an automatch system for its games). Trueskill will be bad if players don't sufficiently mix. There aren't "enough" comparisons made and comparisons between sets of players who don't play each other become much less reliable.

[Holger]

This is just selection bias: you won't notice the hundreds of guys who play a bunch against bots and achieve nothing spectacular, but you will notice the one guy who runs extremely hot. I predict he'll be nowhere near his current level after 110 more games.

That's possible, but substracting 3σ as in Isotropish's level should mean that any player is above his nominal level at 95% 99,7% certainty. And CPA has been improving his level further since this thread was started.
If Iso underestimates upsets, the skill of a strong player mainly playing bots should actually be underestimated, shouldn't it?

Many people have been asking AI to post his article in his "Part 2" thread; but maybe he'll listen to you since you've proofread it   
Does the article give a hint why CPA is ranked so much lower at Goko in spite of them substracting only 2σ, and CPA playing dozens of games per day (so Goko's daily uncertainty increase can only help him)?

Edit: fixed percentage of 3σ level.

[Mic Qsenoch]

Not against real players, but against bots, you can misplay and win inspite of it (or even because of it). Take a look at the game I linked, it's one of the worst but not the only one. Seriously, an opponent buying nothing but Silvers, Golds and green cards could beat him in that game with some decent luck, and it is not a small oversight even a top player could do sometimes, it's going for Philosopher's Stone when there's a fast big money strategy with support in the kingdom.

This is totally wrong. This same thing happens all the time with human players. I win games despite major strategic misplays, and I try silly strategies sometimes that look like something stupid a bot would do. I play against humans, even top players, who choose nonsensical strategies. You can watch streams of top players playing really stupid mirrors against each other, it's not that uncommon.

Certainly if you were playing only humans of iso level 5-15 you would be playing against a lot of stupid strategies. It would still take a decent amount of skill to maintain a high ranking in that pool of players.

You are ignoring the fact that the goal isn't to play "well" by some measure where Awaclus looks at your games and decides whether you chose a good strategy, it's just to win the game. So you are allowed wiggle room to try stupid stuff if your opponent is playing something dumb, you just need to show enough skill to beat whatever they are doing. And this is how it works at every level of play! High or low.

[ThaddeusB]

This is just selection bias: you won't notice the hundreds of guys who play a bunch against bots and achieve nothing spectacular, but you will notice the one guy who runs extremely hot. I predict he'll be nowhere near his current level after 110 more games.

Trusting the description of his play in this thread is accurate and taking into account the very low prior, it's extremely unlikely he's actually very good. Moreover, I've actually proofread the 3rd Rating Analysis article many months ago, and one of its conclusions was that in the long run it's impossible to maintain your rating when playing much lower ranked opposition. This happens because both Goko and to a lesser extent Iso underappreciate the amount of randomness in Dominion and thus give disproportional weights to upsets.

I see there's still interest in this matter; AI is extremely busy right now, but I'll ask him if he's willing to post it when he has a little more time (I don't think he's reading this thread).

This. It's not at all unlikely that someone with a true expected win rate of 75-80% over ISO 10 players could hit 90% over a 125 sample. I also think the second part is true - almost always when someone seems too  highly rayed to me it turns out they play almost exclusively against much higher ranked opponents, getting too high of a boost from the occasional upset.

[Awaclus]

Not against real players, but against bots, you can misplay and win inspite of it (or even because of it). Take a look at the game I linked, it's one of the worst but not the only one. Seriously, an opponent buying nothing but Silvers, Golds and green cards could beat him in that game with some decent luck, and it is not a small oversight even a top player could do sometimes, it's going for Philosopher's Stone when there's a fast big money strategy with support in the kingdom.

This is totally wrong. This same thing happens all the time with human players. I win games despite major strategic misplays, and I try silly strategies sometimes that look like something stupid a bot would do. I play against humans, even top players, who choose nonsensical strategies. You can watch streams of top players playing really stupid mirrors against each other, it's not that uncommon.

Certainly if you were playing only humans of iso level 5-15 you would be playing against a lot of stupid strategies. It would still take a decent amount of skill to maintain a high ranking in that pool of players.

You are ignoring the fact that the goal isn't to play "well" by some measure where Awaclus looks at your games and decides whether you chose a good strategy, it's just to win the game. So you are allowed wiggle room to try stupid stuff if your opponent is playing something dumb, you just need to show enough skill to beat whatever they are doing. And this is how it works at every level of play! High or low.

It's not the same thing. People play bad strategies too, but you can't predict what low level players do, while you can oftentimes predict what a bot is going to do. In order to beat random L5-15 strangers 90% of the time, you need to be very good at playing Dominion. In order to beat bots, you need to be very good at playing against the bots.

[Mic Qsenoch]

Not against real players, but against bots, you can misplay and win inspite of it (or even because of it). Take a look at the game I linked, it's one of the worst but not the only one. Seriously, an opponent buying nothing but Silvers, Golds and green cards could beat him in that game with some decent luck, and it is not a small oversight even a top player could do sometimes, it's going for Philosopher's Stone when there's a fast big money strategy with support in the kingdom.

This is totally wrong. This same thing happens all the time with human players. I win games despite major strategic misplays, and I try silly strategies sometimes that look like something stupid a bot would do. I play against humans, even top players, who choose nonsensical strategies. You can watch streams of top players playing really stupid mirrors against each other, it's not that uncommon.

Certainly if you were playing only humans of iso level 5-15 you would be playing against a lot of stupid strategies. It would still take a decent amount of skill to maintain a high ranking in that pool of players.

You are ignoring the fact that the goal isn't to play "well" by some measure where Awaclus looks at your games and decides whether you chose a good strategy, it's just to win the game. So you are allowed wiggle room to try stupid stuff if your opponent is playing something dumb, you just need to show enough skill to beat whatever they are doing. And this is how it works at every level of play! High or low.

It's not the same thing. People play bad strategies too, but you can't predict what low level players do, while you can oftentimes predict what a bot is going to do. In order to beat random L5-15 strangers 90% of the time, you need to be very good at playing Dominion. In order to beat bots, you need to be very good at playing against the bots.

I don't think predictability is all that valuable in upping your winrate. I mean probably the most glaring issue with the bots is easier wins on boards with Potion cards. And presumably, any harm to the bots from being predictable is reflected in them having lower rankings and thus wins against them are less valuable (I understand that there's asymmetry here if a player is only playing bots, I just don't believe it's a huge deal, I could be wrong). I have played hundreds (thousands?) of bots games and I see a lot of variety in the strategies they pursue, even if they lack finesse. I have been downright shocked by some of the stuff they've cobbled together.

[Holger]

Does the article give a hint why CPA is ranked so much lower at Goko in spite of them substracting only 2σ, and CPA playing dozens of games per day (so Goko's daily uncertainty increase can only help him)?

I suppose I can answer this myself: Goko's rating still contains lots of overrated people due to the recent "Adventures count as Pro games" bug.

This. It's not at all unlikely that someone with a true expected win rate of 75-80% over ISO 10 players could hit 90% over a 125 sample. I also think the second part is true - almost always when someone seems too  highly rayed to me it turns out they play almost exclusively against much higher ranked opponents, getting too high of a boost from the occasional upset.

I've done the math with a true expected win rate of 80% and the current 145 bot games CPA has now played: The probability for a coincidental 90% win rate in this case is about 0.3% - it's very close to the upper 3 sigma level. That's not at all likely. 100+ games are quite a representative sample.

(And even an 80% win rate against bots would still be quite good IMO - I'm not sure I've reached that rate against bots without choosing the kingdom.)

[Ratsia]

This is just selection bias: you won't notice the hundreds of guys who play a bunch against bots and achieve nothing spectacular, but you will notice the one guy who runs extremely hot.

Indeed.

I play almost exclusively against bots on Goko, for the simple reason that I mostly play in situations where I cannot play a full game in a row but can guarantee that I (usually) manage to do my next turn before the time out. In other words, I could not play against humans but bots don't mind my slow-playing.

After quite a bit of games, my isotropish rating is at 28 and has been fluctuating between roughly 25 and 30 all the time. When Isotropic was still around, my rating hovered between 27 and 31. The final rating was 29, after roughly as many games that I have now played on Goko.

Even though I'm just another isolated example, for me the rating system seems to work very accurately despite playing only against the bots, giving me a rating that is practically indistinguishable from what I had on Isotropic playing only against humans (and in general the two ratings seem to be quite comparable). I guess I could improve my rating on Goko a bit by trying to game the bots, but I have zero interested in doing that.


Edit: ...and I would hate to have games against bots removed from the ranking. If I have an opportunity to play against humans it's way better that my rating is roughly at the right level so that I can get right opponents via automatch.

[flies]

i play against bots when I don't really want to pay attention to what I'm doing or when I want to experiment with long-shot strategies.  I do not want "play bots" games included in the ranking.  I do not want "play bots" counted on the Pro leaderboard at all.  I suppose it makes some sense to have the option to play bots on the pro leaderboard, but for now, if I want to play casual games against bots it is veryrelatively laborious (must do it thru the multiplayer interface).

[Ratsia]

i I suppose it makes some sense to have the option to play bots on the pro leaderboard, but for now, if I want to play casual games against bots it is veryrelatively laborious (must do it thru the multiplayer interface).

It indeed should be possible to do both, with equal easy.

For a long time it was laborious to play with the pro ranking, using the same multiplayer interface trick. in fact, I had not even realized I don't anymore need to do this but can use the Play bots -link instead.

Edit: Apparently one still needs to use the multiplayer interface. I just tried with Play bots and it put me against Serf bot, which is not really a bot at all. The other bots are not great either, but at least they attempt to play and hence a match against them can be justified as reasonable use of time. Playing against Serf bot does not qualify for that.

[Holger]

i I suppose it makes some sense to have the option to play bots on the pro leaderboard, but for now, if I want to play casual games against bots it is veryrelatively laborious (must do it thru the multiplayer interface).

It indeed should be possible to do both, with equal easy.

Yes, it should be (just ask for the rating in addition to the player number in the pop-up window). But for some reason MF seem to prefer bot games only in Pro, because they consider it "fairer". It was (is) indeed easy to rise to the top of the Casual leaderboard by choosing a kingdom which bots play badly (e.g. a Rebuild board); but at the very least they should give a simple way to play Unrated bot games.

For a long time it was laborious to play with the pro ranking, using the same multiplayer interface trick. in fact, I had not even realized I don't anymore need to do this but can use the Play bots -link instead.

Edit: Apparently one still needs to use the multiplayer interface. I just tried with Play bots and it put me against Serf bot, which is not really a bot at all. The other bots are not great either, but at least they attempt to play and hence a match against them can be justified as reasonable use of time. Playing against Serf bot does not qualify for that.

I've also been using the Play Bots button lately and have never been matched against Serf Bot. I think they automatically match you against the bot who's closest to your rating, so you would only play Serf Bot if you have a Goko rating <2000 or something like that. Otherwise I'd consider this a bug...

[Ratsia]

I've also been using the Play Bots button lately and have never been matched against Serf Bot. I think they automatically match you against the bot who's closest to your rating, so you would only play Serf Bot if you have a Goko rating <2000 or something like that. Otherwise I'd consider this a bug...

I tried twice, both time getting Serf bot. My Pro rating is around 4700, clearly above all bots, so apparently it is not choosing the best match at least according to that rating. It could use the Casual rating for making the choice, despite the result influencing the Pro rating; it wouldn't make sense, but this sounds like a rather easy mistake to make. My Casual rating is as low as possible, since I haven't played any games in that mode in ages.

[Holger]

I've also been using the Play Bots button lately and have never been matched against Serf Bot. I think they automatically match you against the bot who's closest to your rating, so you would only play Serf Bot if you have a Goko rating <2000 or something like that. Otherwise I'd consider this a bug...

I tried twice, both time getting Serf bot. My Pro rating is around 4700, clearly above all bots, so apparently it is not choosing the best match at least according to that rating. It could use the Casual rating for making the choice, despite the result influencing the Pro rating; it wouldn't make sense, but this sounds like a rather easy mistake to make. My Casual rating is as low as possible, since I haven't played any games in that mode in ages.

That's strange. Your explanation might be true; I have a Casual rating of about 2000, which would be enough to not get Serf Bot, at least if the bots are also judged by their Casual ratings. Since MF already mistakenly allows the Kingdom Generator to be used with that button, maybe they also forgot to switch the "bot matchmaking" from Casual to Pro rating.

Another stupid thing that they should change ASAP is timing out in bot games - it's very annoying to lose a bot game just because you're doing something else for 5 or 10 minutes.

[flies]

plus having your quit % affected when playing against bots is wrong.

the simplest solution, suggested above, is to have a rating-mode switch when hitting 'play bots'.

[jaybeez]

Sorry for the thread necro, but I was unaware of this until I used the Play Bots option last night for the first time probably a year or more.  I just wanted to try out a couple of things, actually lost one of the games, and quit another, and noticed my Pro rating drop by a bunch.

Can I suggest some kind of pop-up that warns people of this when clicking the Play Bots button?  Just a little dialog window like, "Warning: games played using Play Bots are counted as Pro games!"  Is that even possible with the extension?

[SCSN]

Sorry for the thread necro

One of the many nice things about Goko is that you don't have to worry about things like dates and stuff, for if something was an issue two years ago it's almost certainly still an issue today!

[WanderingWinder]

To be fair, this is also a thing on isotropish.

[-Stef-]

To be fair, this is also a thing on isotropish.

Yes but it's only an Isotopish thing because Andrew is under the impression that we (the f.ds community) think it's better this way.
I certainly don't.

[jaybeez]

To be fair, this is also a thing on isotropish.

Yes, that's precisely what the problem is.

[WanderingWinder]

To be fair, this is also a thing on isotropish.

Yes but it's only an Isotopish thing because Andrew is under the impression that we (the f.ds community) think it's better this way.
I certainly don't.

Yes, and it's only a Goko thing because they were under the impression that their customer base would like it better that way. They may or may not be wrong (I, like you, think they shouldn't be included), but on this issue at least, there's not much difference. The real difference is that this is one of few issues for Salvager, and many for Goko.

[qmech]

I've played quite a lot of bots recently to see what effect it has on my rating, and the answer is that it has certainly increased. 

Unless you think the Bots isotropish ratings are majorly out of whack, this shouldn't matter.

If you believe in the rating system, then I think you have to take this line.  But it's easy to imagine reasons why the bots' ratings might be inflated: for example, they're likely to win a reasonable number of games against opponents who hit the Play Bots button then quit when they run out of time to play.

Whatever the truth there, I've found playing the bots a much easier way to gain rating than playing human opposition, for two related reasons.  The games are usually easier, so if you're in a reasonable position then you can play much more quickly.  And that means it's easy to play a lot of games in a short time span: I might have gone up by the same amount if I'd played the same number of games against high level humans, but it would have taken a lot longer and required a lot more mental effort.

I can also back up MicQ's observation in whichever thread that the Goko ratings do not respond as positively to repeated crushing of the bots.

[polot38]

I don't really understand the opposition to having bots removed from the rankings. I mean, either:
a) bots are easy to beat if you know how to play against specifically bots (or they are somehow systematically underrated), in which case bots should clearly be removed, or
b) bots are roughly equal in skill to players around their level, in which case you don't really lose much by kicking them out anyways.

In either case, you still have the issue that some people would like to play bot-games when they don't really like playing serious dominion, and you could also make the case that, even with a 100% win-rate, you shouldn't be able to get to one of the highest ranking in dominion by only playing lower level players.

[GeoLib]

The problem with removing bots is that they give people a fast way to practice and prove their skill so they don't have to jump into multiplayer at the bottom. When so many games have rating caps it's kind of nice to be able to quickly climb the ladder (because bot games are quick).

[polot38]

The problem with removing bots is that they give people a fast way to practice and prove their skill so they don't have to jump into multiplayer at the bottom. When so many games have rating caps it's kind of nice to be able to quickly climb the ladder (because bot games are quick).

Is that really a large issue though? I mean, if your skill really is above most rating caps, you should be able to quickly jump up in ratings by playing normal players anyways. And it is only likely to affect very new players, a lot of whom aren't skilled enough to start with to jump into games with rating caps.

[Throwaway_bicycling]

I've played quite a lot of bots recently to see what effect it has on my rating, and the answer is that it has certainly increased. 

Skipping the boring background, I can now confess that I never actually made a conscious decision to play Pro games on Dominion Online, but started to do so when I found out that I was actually on the Isotropish leaderboard because bot games had started to count as Pro. In any case, I had a ranking of about 200 and Isotropish thought I was about Level 30 at the end of October, 2014. That gave me (over)confidence, and I have played about 300 real games after that...and at first watched my rank and rating sink like a stone. In my case, I would estimate the "Bot Effect" as being 10-12 Isotropish levels.

So that *sounds* like a serious distortion, but I am actually not sure the problem is completely with the Bots' ratings in the broad sense (they have all played about 250K games apiece in two-player, against all levels of opponents) as it is that when you are playing Bots, you are playing only one of 6 opponents, and opponents who do not learn or change, rather than the dozens or hundreds of different opponents you could in real life. I don't know for sure (it's an empirical question, though), but my guess is that stronger players would have some "ratings inflation" generally against hapless weaker opponents they played relatively often. In general, there may be meaningful effects of the diversity of opponents you play. My intuition is that if you had two players equal in mu and sigma and games played, the one who had played a significantly larger number of opponents would be the favorite in a match played between them.

[Ratsia]

Is that really a large issue though? I mean, if your skill really is above most rating caps, you should be able to quickly jump up in ratings by playing normal players anyways. And it is only likely to affect very new players, a lot of whom aren't skilled enough to start with to jump into games with rating caps.

The people who are clearly above most rating caps are not the ones for whom this matters. Instead, it matters for those who are probably just a bit above them but have no chance of regularly playing against humans to prove it. More generally, this is a question that should not be primarily looked at from the perspective of the absolute top players, but from the perspective of the average-quality players who are probably the ones more likely to play against the bots in significant numbers. Think about people who cap at level 20 or so in isotropish, who actually represent probably the largest mass of experienced Dominion players.

While a level ~45 player could indeed climb up to high enough rating in a few matches, it takes quite a while for a level 20-30 player who consistently plays at that level. I remember trying with another account while Isotropic was still around, and it took me roughly 50 matches to reach within 5 levels of my actual strength (25 vs 30 or so). That's quite a lot of matches for less active players, often meaning several weeks of time. Right now I almost exclusive play against bots since I cannot play matches without occasional excessive delays. Since this maintains a reasonable (though indeed a bit over-inflated, in my case probably by roughly 5 levels) rating, I can occasionally play against a human opponent and get immediately matched with roughly the right people. If it didn't, the threshold of switching to the human-opponent mode would be so high that I would never bother -- would anyway join if I hosted a game with a rating threshold of 5000+ if my rating is 1000?

[ragingduckd]

I'm messing with the leaderboard... please temporarily ignore anything that looks weird.

[ragingduckd]

I'm messing with the leaderboard... please temporarily ignore anything that looks weird.

All is restored.

I'm considering excluding bot games from Isotropish.  After reading this thread and comparing the two boards, I'm leaning heavily towards making the switch.  The modified leaderboard can be viewed at https://gokosalvager.com/leaderboard_nobots

[Mic Qsenoch]

I knew I was terrible.

[jaybeez]

Well the current leaderboard has me at Level 37, but the no-bots leaderboard has me at Level 38, so you know which has my vote.

Thanks for looking into this, Andrew.

[qmech]

Playing nothing but bots for the last week has boosted me from level 42 to level 45.  I must be stopped.

[Throwaway_bicycling]

I knew I was terrible.

I knew I was...way, way, way more terrible than Mic Qsenoch. :-)

But more seriously, the new leaderboard removes almost 1400 players (!) who were above Level Zero when bot games were included, but not above that (or possibly not even eligible) with bot games removed. So, although it's possible the bots included board would be a better predictor of how you would do against bots, the human game is certainly different, and if that's what we're after here, yeah, go with that. Some day the AI might be worthy of the name, and including games against them would have some meaning, but that empirically that doesn't seem to be the case now.

Disclaimer: I lose 2.something levels but gain almost 100 places on the no bots board (and my 3sigma goes up). In some cases, that helps give the answer to "how did I beat the crap out of that Level 28 person?" when it turns out that they are, well, not as good against humans as that rating would indicate.

[GeoLib]

So I go up a level, my mu goes up by 1.5, and I jump 100 spots on the full leaderboard. You have my vote

In all seriousness, I'm somewhat concerned about how this will effect people's ability to break into the human game scene, but I do think it is likely to be a better predictor of skill.

[Mic Qsenoch]

In all seriousness, I'm somewhat concerned about how this will effect people's ability to break into the human game scene, but I do think it is likely to be a better predictor of skill.

I don't see a lot of people putting isotropish level restrictions in their tables names (though there's some). So I wouldn't expect it to be a problem, any Pro rating restrictions are exactly the same as they were before.

[Titandrake]

I'm concerned about how people will manage to break into the 20s/30s as well, but if Goko continues to keep bot games for the Pro rating it'll probably be fine. When I restarted Dominion I played ~20 games against bots to get up to a high enough rating for human players. If people collectively switch to Isotropish level restrictions it could be a problem.

[Beyond Awesome]

I lose a couple of levels, but I think the switch is a good idea. After reading some of these threads, I played some bot games to see how easy it was to gain levels, and well, it's easy. Too easy. Bot ratings need to go.

[Burning Skull]

It strikes me as peculiar, I'm one level worse in no-bots version and I don't think I ever played a single game against a bot.
Or does the old version also count adventures?

[qmech]

If you've played opponents who have played against bots, or who have played opponents who have played against bots, or ... then you'll still feel some influence from bot games on your rating.

[Throwaway_bicycling]

If you've played opponents who have played against bots, or who have played opponents who have played against bots, or ... then you'll still feel some influence from bot games on your rating.

Right; this is the "How did I beat that  person 10 levels above me?" issue I noted above. The answer is that their ratings were inflated by bots. Also, I think I figured out another reason why the bot ratings were "wrong" at least with respect to the upper levels of the leaderboard despite playing a ton of games against non-bot opponents: there was a lot less connectivity between players who played bots and players who did not. Almost 1400 people dropped off the leaderboard either because they played only bots or played so poorly against people they were below level zero...that's huge.

[JW]

Right; this is the "How did I beat that  person 10 levels above me?" issue I noted above. The answer is that their ratings were inflated by bots. Also, I think I figured out another reason why the bot ratings were "wrong" at least with respect to the upper levels of the leaderboard despite playing a ton of games against non-bot opponents: there was a lot less connectivity between players who played bots and players who did not. Almost 1400 people dropped off the leaderboard either because they played only bots or played so poorly against people they were below level zero...that's huge.

The main leaderboard page only shows players with games played in the past month and 0+ ranking, but the full leaderboard includes all players. Player with level less than 0 or without games in the past month are counted in the rankings, and your games against them count for the rankings, they just don't show up on the main leaderboard page. https://gokosalvager.com/leaderboard/?full=False&sortkey=level

[Titandrake]

I think the biggest disconnect happened when Goko changed the front page play bots button to be Pro instead of Casual. Before, the bot ratings were determined based on those people who were willing to enter the multiplayer lobby, create a pro game, add a bot opponent, then start a game. Now, they're determined by anyone who tries out Goko for the first time, so there's a much larger pool of newbs that the bots farm pro rating from.

Or, maybe I'm remembering things wrong. But I don't remember people having this issue before the change.

[Ratsia]

I think the biggest disconnect happened when Goko changed the front page play bots button to be Pro instead of Casual. Before, the bot ratings were determined based on those people who were willing to enter the multiplayer lobby, create a pro game, add a bot opponent, then start a game. Now, they're determined by anyone who tries out Goko for the first time, so there's a much larger pool of newbs that the bots farm pro rating from.

This sounds intuitively reasonable. However, an interesting observation is that the bot ratings are roughly the same today as they were before the change. Right now the bots are between levels 10 and 16. On August 2013, well before Play Bots switched to Pro ratings, I wrote a post stating they were between 9 and 16. Based on occasional checks every now and then, I can confirm they have remained roughly at those levels as long as the Isotropish leaderboard has existed. To conclude, the bot ratings do not look any more inflated today than they were in the past, even though they in all likelihood do play more against very bad opponents.


As someone for whom removal of bot matches from Isotropish matters more (I am not eligible for the non-bot version at all), I would vote in favor of the non-bot version since it seems to be somewhat more accurate. I still think it is crucially important Goko itself ranks matches against bots in Pro mode since having high enough rating is important when switching to play against humans, but Isotropish can well ignore the bots since those ratings are not typically used for matchmaking. Now the only effect is that if I start again playing against humans, the first 50-100 opponents I play against will get their ratings adapted somewhat unfairly as the system things they are playing against a Lvl 0 player instead of Lvl 30. This could naturally happen otherwise too if I was a truly new player and the effect is small anyways, so this is not a big deal.

[Ratsia]

Almost 1400 people dropped off the leaderboard either because they played only bots or played so poorly against people they were below level zero...that's huge.

I would guess the main reason is the former, that majority of those really played (almost) only against bots. That's indeed a huge effect as such, but an interesting question is how many of those were ever checking the leaderboard. I think there are three possible scenarios:
(1) Majority of them never looked at Isotropish, so having those 1400 people removed from the list does not matter at all -- they won't notice
(2) Many of them did use that as their primary source of ranking information, and they will now be irrirated of being no longer ranked
(3) Many of them did use that as their primary source of ranking information, and now that they see bot games are no longer included they will switch to playing against humans instead

Switching to the non-bot leaderboard is a problem only if (2) holds, since then some subgroup of players will not anymore get a service they were enjoying. I (3) holds then it actually has a positive impact for the community as a whole.

I would be surprised if (2) holds.

[qmech]

A recent convert to Dominion has been practicing using the Play Bots button and was matched against Serf Bot 30 times in a row.  They eventually decided that this was because they had lost a few Casual games against friends, and that, despite counting for the Pro rankings, bot opponents are chosen using the Casual rating.  This appears to be true, as they fixed the problem by playing a few manually set up Casual games against Lord Bottington.

[Throwaway_bicycling]

The main leaderboard page only shows players with games played in the past month and 0+ ranking, but the full leaderboard includes all players. Player with level less than 0 or without games in the past month are counted in the rankings, and your games against them count for the rankings, they just don't show up on the main leaderboard page. https://gokosalvager.com/leaderboard/?full=False&sortkey=level

To be clear...I knew that. And, needless to say, you don't get much ratings bump from playing somebody who now has a rating of, oh, -3, and heaven help you if you lose...

I think there's no question that the leaderboard needs to change this way; it leads to a much more accurate picture of relative skill, thus potentially better matches, and possibly (in the long run) more rapid improvement in play.

One adjustment to behavior this move might suggest is for people to specify ratings limits for games in terms of Isotropish levels. Right now, you do confuse people if you specify (say) "4000+" but then leave or kick them out when they turn out to be Iso Level 8 or something (I don't do that and think it's a bad idea). If you say "ISO 20+", that will confuse some people in the short-run, too, but my guess is that there really are only like 3000 or so people who play a lot on line, and a smaller number who play really a lot, and the number of people you would have to chat with to describe ISO isn't that large and would get smaller.

[Throwaway_bicycling]

I would be surprised if (2) holds.

I agree. Most of the 1400 who disappeared won't care. Some who played more games with those 1400 will be annoyed for a minute, but won't care for very long. I think the board shift is mostly a "concern" (in terms of ratings loss) for players with Iso ratings between about 15 and 25, for reasons I will get to below.

I think the real reasons why there was a lot of bot ply by people who played Pro basically all the time and humans most of the time were:

1) Nobody seems to be joining their game and they were bored.
2) They weren't trying to juice their Isotrophish ratings, but they *were* trying to juice their Goko Pro ratings to get above (say) 4000 or 5000 or some popular ratings limit.

I suspect people not playing bots much or at all but who are between Iso 15 and 25 are possibly more likely to have been in ratings limit games with people who "bots-ed up" their ratings. And, yes, gained more points from those matches than the true skill (heh) differential would have indicated. So let's fix that.

[WanderingWinder]

That playing bot games can inflate your rating means there is a fundamental problem with the rating system, it's not doing its job correctly. (Note, however, that just because Goko ratings don't have this problem does not necessarily mean that they ARE accurate, either). On the other hand, it's not something so simple to truly and correctly fix. Removing the bots from the pro rankings is what I would suggest, as it will at least remove one of the symptoms, and the main ones we can see, though people should be aware that it hasn't fixed the underlying problem.

[DG]

Remember that some very human factors can be involved. For example, some of the ten bot games that were removed from my pro record were played by friends who were at novice level and just trying Goko out.

[Throwaway_bicycling]

That playing bot games can inflate your rating means there is a fundamental problem with the rating system, it's not doing its job correctly.

Okay, so I have gone back and forth, up and down on this, because I agree this is a troubling result. I also peeked into some of the recent stuff on chess ratings, which have had a surprisingly checkered and complicated history (I thought things were weird enough decades back when I was still playing "competitive" chess). At the very end of the day, I think the root of the issue is ultimately the one raised by Polk5440:

Also, Trueskill is designed to be very good when players of similar rankings play each other (which is why microsoft paired it with an automatch system for its games). Trueskill will be bad if players don't sufficiently mix. There aren't "enough" comparisons made and comparisons between sets of players who don't play each other become much less reliable.

I think what was really going on here was that we had was a mixture of two subpopulations of "Pro" Dominion players. One subpopulation played a really low proportion of games against bots. Another subpopulation, rich in novices, played a pretty high proportion of games against bots, and humans who played a lot of bots. Most of the novices who play bots are, not to be too judgmental, really poor players. And there are a lot of them, and, in aggregate, their role in this story is to "feed" points to the bots.

A relatively small number of players playing mostly bots and players who played bots were of somewhat higher skill, so were in turn "fed" some of the points the bots ate. Back before the "Play Bots" rating system switch, this wasn't a major problem in Pro land. But after the "Play Bots" button defaulted to Pro, there really was a big inflow of points to bots from novices now suddenly playing in "Pro" land, which were in turn "fed" to the mediocre players (including me).

Problem: this population of middling players don't mix nearly as much as you might think with stronger human players, so "The points stopped here". Without the kind of mixing Polk5440 mentions, two players that have (say) a 5000 rating could be very different in skill if one player's rating was padded by bots and the other player's wasn't. The good news, is that I think we can (and just have) "drained" the bots points out of the system, but there is still a fair amount of disconnection or lack of mixing in the system, and my hunch is that when/if there is some correction for that, there may be some downward ratings pressure on at least a subset of the (say) Top 500 or so as they lose the same proportion of games as previously to the bot-inflated players, who now have lower ratings.

In any case, it's an empirical question.