Dominion Strategy Forum
Dominion => Dominion Online at Shuffle iT => Dominion General Discussion => Goko Dominion Online => Topic started by: ragingduckd on July 18, 2013, 08:00:06 am
-
http://isotropish.com is an implementation of Microsoft's TrueSkill rating system (http://research.microsoft.com/en-us/projects/trueskill/). It is meant to be similar to the rating system that was used by Isotropic, but it isn't quite faithful (see below).
- TrueSkill parameters:
- Initial rating: μ=25, σ=25
- Adjustment: β=25, τ=25/100, draw rate = 5%
- Uses the Python package from trueskill.org (http://www.trueskill.org)
- Sorted by "Level" defined as μ - 3σ. Click column names for alternative sorting.
- Games with bots count, games with guests don't
- Only 2-player "Pro" games count
- No daily increase in σ
I originally meant for these parameters to replicate those of the isotropic leaderboard, but there was some confusion at the time about just what those parameters were:
From Isotropic's FAQ (no longer available):
New players are assigned a skill of "25 ± 25", which is to say, we don't really have any idea what that person's skill is.
...
I've set β = 25, γ = σ0 / 100 (applied daily), and the draw probability at 5%
From a PM with dougz:
No, I start with μ=25, σ=25/3 like everyone else. But since I display "μ ± 3σ" on the leaderboard it shows up as 25 ± 25.
I may not have ever actually had the clamping on the upper end.
Choice of 5%: pulled out of thin air.
Yes, there was a small daily increase in σ. (Moved 1% of the way back to 25/3, I think.) I didn't want people to be able to camp out on the leaderboard by getting to a good position and then not playing.
Based on all this, it looks like dougz was probably using μ0=25, σ0=25/3, β = 25, draw_prob=5%, τ=?, and a daily increase in uncertainty of either (0.01)σ or (0.01)(σ0-σ).
β = 25 is much larger than the TrueSkill default of σ0/2 = 25/6. The default for τ is σ0/100 = 25/300, but I don't have anything definite about what value was used for Isotropic. The differences between Isotropic and Isotropish are likely due to the values used for β and τ, and to the fact that Isotropish does not have the daily increase in σ. The different σ0 values have some effect too, but can't explain the differences in σ for players with thousands of games.
No need to post expressing your personal preferences for these parameters. I'm doing some testing now and will switch to the best-performing values shortly.
-
I'd prefer that the Trueskill rating for each person be shown as mu +- (3*sigma), not mu +- sigma. But that's just me.
-
When is it updated? daily?
-
When is it updated? daily?
Every 10 min
-
Good work! Thanks!
-
I assume that you are using the default values for Beta and Tau and all that jazz?
Okay, so if you have Beta=25/6, there are a couple things: first, the values of sigma for the different players are quickly falling to the point that they matter almost nothing. If we take even the highest guy from the top 200 (daniel greif, currently #69, sigma = 2.34), against someone with the seemingly typical sigma of .8, the overall difference in the Sigma of the distribution is from 4.32 to 4.85, which at most is going to have a 2.79% change in the expected winrate, which only happens if the opponent is 4.57 points of mu) away from this fellow - corresponding to a jump from 82.7% to 85.5%.
Furthermore, it should be very easy to whip up the expected winrate in a match between any two players. Maybe another page where you can enter the mu and sigma for both of you, and it spits out the result? Of course, it would be most nice to do it on name vs name, but I don't know how much work that is - the thing I have suggested should be very very simple.
Okay, lastly, my pet peeve, the mu-3*sigma thing. Basically, it is systematically underrating players with high uncertainties. Especially against strong players. If you look at it, the system expects #204 to do better against Stef than it does #6.
Oh, also, you are rating guests, just not games against them. I don't know if you want to clean that up or not.
-
If we take even the highest guy from the top 200 (daniel greif, currently #69, sigma = 2.34), against someone with the seemingly typical sigma of .8, the overall difference in the Sigma of the distribution is from 4.32 to 4.85, which at most is going to have a 2.79% change in the expected winrate, which only happens if the opponent is 4.57 points of mu) away from this fellow - corresponding to a jump from 82.7% to 85.5%.
My intuition is failing me here. Is this obviously undesirable? Is fixing it a matter of calibrating sigma or of using a different family of curves?
Furthermore, it should be very easy to whip up the expected winrate in a match between any two players. Maybe another page where you can enter the mu and sigma for both of you, and it spits out the result? Of course, it would be most nice to do it on name vs name, but I don't know how much work that is - the thing I have suggested should be very very simple.
Good idea. I have a mapping from player names to current ratings, so neither approach is difficult.
Okay, lastly, my pet peeve, the mu-3*sigma thing. Basically, it is systematically underrating players with high uncertainties. Especially against strong players. If you look at it, the system expects #204 to do better against Stef than it does #6.
That's true. But do you really want the ordering to reflect the probability of beating Stef? For top-whatever players that's probably a good metric, but for the median player it might not be so great.
In any case, here's the mu-sorted top-25:
games_played | pname | mu | sigma
--------------+-------------------+---------+--------
7 | lettukastike | 36.7890 | 3.7491
7 | Tydude | 36.6604 | 3.8999
7 | Guest_213286 | 36.5075 | 4.8567
5 | ShivaBowl | 36.3048 | 4.8451
7 | jonts | 36.0121 | 3.9443
6 | Tommy | 35.9250 | 4.0632
8 | Meteora | 35.9202 | 4.0425
2 | Tomas Ramanauskas | 35.6200 | 5.6838
4 | Lilly Roettgen | 35.2591 | 5.2667
4 | w455up | 35.0957 | 4.7752
5 | Guest_860460 | 35.0173 | 4.5824
1 | gianthsiao | 34.7435 | 6.0963
4 | Jason Orne | 34.7036 | 4.4389
11 | Robert Birks | 34.5059 | 4.9000
5 | Saphy89 | 34.4803 | 5.4340
7 | Schub | 34.4702 | 3.9738
3 | Bahguerra | 34.4432 | 5.3661
8 | rinshan | 34.3814 | 3.6950
3 | JEL64 | 34.3160 | 5.5604
2 | Nightwing | 34.2906 | 5.3101
2 | RobinRaven | 34.2896 | 5.7889
3 | GamingNoise | 34.2733 | 4.8472
13 | daniel greif | 34.2320 | 2.3373
7 | QuidProBro | 34.1935 | 4.4836
3 | Focastars | 34.1898 | 5.0783
Limiting it to players with 30+ eligible games makes it more palatable:
games_played | pname | mu | sigma
--------------+-------------------+---------+--------
86 | Tao Chen | 32.3903 | 1.1416
111 | nomnomnom | 32.3275 | 0.9555
748 | Stef | 32.1135 | 0.7955
134 | Boodaloo | 32.0649 | 0.8679
657 | Mic Qsenoch | 31.6395 | 0.7953
490 | Stealth Tomato | 31.5151 | 0.8173
212 | Rene Kuroi | 31.4289 | 0.8053
675 | Wandering Winder | 31.3437 | 0.7959
531 | Obi Wan Bonogi | 31.2981 | 0.8044
681 | hiroki | 31.2741 | 0.7907
1580 | SheCantSayNo | 31.2577 | 0.7875
138 | HiveMindEmulator | 31.2219 | 0.8641
439 | Rabid | 31.2037 | 0.8033
53 | Holger | 31.1264 | 1.2410
581 | Geronimoo | 31.1062 | 0.8065
690 | yudai214 | 31.0915 | 0.7872
51 | First | 31.0688 | 1.1839
41 | yuuna_tu | 30.9809 | 1.2839
48 | awall | 30.8608 | 1.4834
718 | LESPEUTERE | 30.8312 | 0.8034
485 | iriho | 30.8213 | 0.7930
419 | jaybeez | 30.7857 | 0.7980
1113 | Andrew Iannaccone | 30.7585 | 0.7888
46 | faw | 30.7349 | 1.2166
33 | ooo | 30.7265 | 1.4631
Oh, also, you are rating guests, just not games against them. I don't know if you want to clean that up or not.
Thanks. It looks like the problem is that some of the guest players are slipping past my filtering. Goko seems to have used several different guest-naming conventions. That'll be easy to fix for future games, but it might be a while before I clean up the existing data.
-
If we take even the highest guy from the top 200 (daniel greif, currently #69, sigma = 2.34), against someone with the seemingly typical sigma of .8, the overall difference in the Sigma of the distribution is from 4.32 to 4.85, which at most is going to have a 2.79% change in the expected winrate, which only happens if the opponent is 4.57 points of mu) away from this fellow - corresponding to a jump from 82.7% to 85.5%.
My intuition is failing me here. Is this obviously undesirable? Is fixing it a matter of calibrating sigma or of using a different family of curves?
Oh, this isn't meant to be a criticism at all, just something to note.
It does play a little into my mu-3*sigma point, insofar as the difference between 1.0 and 0.75 is basically nothing to the system, whereas the .75 difference in mu that it is equating to is pretty significant.
Your first listing of straight mu sort is obviously not the most desirable thing ever, but I actually don't see how the second is only 'a little more palatable': to me, this IS the leaderboard. I don't know about cutting off at 30 games - I would probably cut off at sigma of 1.5 or 1 or something. But basically, yeah, that's what I would go with.
-
Your first listing of straight mu sort is obviously not the most desirable thing ever, but I actually don't see how the second is only 'a little more palatable': to me, this IS the leaderboard
Ok, enourmously more palatable. ;)
I don't know about cutting off at 30 games - I would probably cut off at sigma of 1.5 or 1 or something. But basically, yeah, that's what I would go with.
I like N games better than a sigma cutoff. The only thing worse than slipping down the leaderboard after winning a game would be falling off of it entirely. I do wish there were a cleaner solution.
-
Your first listing of straight mu sort is obviously not the most desirable thing ever
lol, I love these kind of understatements.
I don't know about cutting off at 30 games - I would probably cut off at sigma of 1.5 or 1 or something.
If you're going with this I have a strong preference for cutting of at 1, as I've never seen any of those guys with sigma > 1, while I've seen all the players with sigma < 1 quite alot, and played all of them except HME.
-
Your first listing of straight mu sort is obviously not the most desirable thing ever, but I actually don't see how the second is only 'a little more palatable': to me, this IS the leaderboard
Ok, enourmously more palatable. ;)
I don't know about cutting off at 30 games - I would probably cut off at sigma of 1.5 or 1 or something. But basically, yeah, that's what I would go with.
I like N games better than a sigma cutoff. The only thing worse than slipping down the leaderboard after winning a game would be falling off of it entirely. I do wish there were a cleaner solution.
For some reason I was thinking you were still using a time cutoff. While surely it's not really going to make much difference, as whatever sigma cutoff you would pick would be something that an established player isn't going to fall off of without a serious amount of game fixing, I agree with you here - my thought was that you could fall off based on inactivity, which seems fine, but that is not the case, so yeah, games makes more sense.
-
Any chance you could add the "games played" to the leaderboard?
Very nice overall.
-
FWIW it looks a lot better for me.
I just clawed my way up to #10 on the Goko board, but I did it without really challenging myself. I post a new game and basically accept anyone with a rating above 4500 or so. Which means a lot of games with people below 5000 rating.
I'm learning that that level on goko corresponds to about 15 on iso?
It's low enough that I rarely lose, even when making pretty significant mistakes. (One of the few I lost, I messed up the interface and kept trashing cards with my Jack that I was trying to play. Including a King's Court. And it was still a close game!)
At the same time the few games I've played against the top players have not gone great. I've only won a handful.
Whatever system goko uses you can move up the ladder to the very top by playing people far below your level and not losing very often (I think I gain about 20 points on a win and lose about 70 on a loss).
On the Trueskill board I'm in #20 which seems a lot more reasonable given how I've been playing.
Nice work.
Ed
-
For what it's worth, if you want to emulate the isotropic leaderboard exactly, then the web archive still has the settings for it:
http://web.archive.org/web/20130116154350/http://dominion.isotropic.org/faq/ (http://web.archive.org/web/20130116154350/http://dominion.isotropic.org/faq/)
n a nutshell: skill is measured on a scale that goes from roughly 0 to 50 points. (Actually skill can be any number, but 99.8% of players should fall in the 0–50 range.) The skill range column is a 99.8% confidence interval — the system is 99.8% sure your true skill lies somewhere in that range. New players are assigned a skill of "25 ± 25", which is to say, we don't really have any idea what that person's skill is. As you play more, your mean skill moves up or down and the range gets smaller as the system believes it has a better estimate of your skill.
The level column is the low end of your range, rounded down to an integer and clamped to the range [0, 50]. If we ignore the clamping, it is a conservative skill estimate in the sense that we are 99.9% confident that you are at least that skillful.
Because Dominion has a lot of randomness (it's not uncommon for a low-skill player to beat a high-skill player, through fortunate shuffling), it takes a relatively long time to change your skill — the system needs to see a lot of examples of you winning before it accepts that it's not just due to luck. (For those interested in the details, I've set β = 25, γ = σ0 / 100 (applied daily), and the draw probability at 5%.)
-
It does play a little into my mu-3*sigma point, insofar as the difference between 1.0 and 0.75 is basically nothing to the system, whereas the .75 difference in mu that it is equating to is pretty significant.
Your first listing of straight mu sort is obviously not the most desirable thing ever, but I actually don't see how the second is only 'a little more palatable': to me, this IS the leaderboard. I don't know about cutting off at 30 games - I would probably cut off at sigma of 1.5 or 1 or something. But basically, yeah, that's what I would go with.
The problem is that the leaderboard means different things to different people. Using the ATP as an example, Andy Murray has been playing very well over the past year, but he didn't play in the French Open this year, so he can't be #1. This doesn't mean you shouldn't expect him to be able to beat Djokovic in the US Open, it just means he hasn't done enough to deserve to be #1, based on the meaning of the ATP rankings. The rankings are not predictive, they are accomplishment-based.
Some people like it this way, and some people don't. Psychologically, sorting by mu makes it kind of crazy. People are already upset at how much one game affects your Goko rating, so I shudder to think how they'll feel about a leaderboard sorted by mu. However, the people who play the most often don't really want to be rewarded for simply playing more games, but only for playing better, so these people may tend to prefer to sort by mu.
Maybe a good compromise would be to provide both? Have the leaderboard be the standard Trueskill thing, but make it sortable by mu, so people can see that if they want?
-
Feature request:
Add "grey level numbers" when a lot of people are in one level. Like isotropic:
http://dominion.isotropic.org/leaderboard/
-
Why are there a bunch of players with ".0000" after there username?
Lightning edit: After some more perusal, it looks like it's for duplicated usernames. It seems strange that Goko would allow those!
-
For what it's worth, if you want to emulate the isotropic leaderboard exactly, then the web archive still has the settings for it:
http://web.archive.org/web/20130116154350/http://dominion.isotropic.org/faq/ (http://web.archive.org/web/20130116154350/http://dominion.isotropic.org/faq/)
Thanks. I'm considering it. It would be nice for consistency's sake, if nothing else.
Maybe a good compromise would be to provide both? Have the leaderboard be the standard Trueskill thing, but make it sortable by mu, so people can see that if they want?
I like the idea of providing variations, but I also fear that this way lies madness. One unofficial leaderboard is bad enough...
Why are there a bunch of players with ".0000" after there username?
Lightning edit: After some more perusal, it looks like it's for duplicated usernames. It seems strange that Goko would allow those!
I was wondering about that. Are you sure? I haven't looked at their hashes.
-
Why are there a bunch of players with ".0000" after there username?
Lightning edit: After some more perusal, it looks like it's for duplicated usernames. It seems strange that Goko would allow those!
I was wondering about that. Are you sure? I haven't looked at their hashes.
I'm not sure. It just so happened that the first one I searched for (John Wyatt) had a duplicate. But, a bunch of others don't. So, now I'm much less sure, but it's a decent hypothesis if others with the same name just use a custom username.
-
If you're going with this I have a strong preference for cutting of at 1, as I've never seen any of those guys with sigma > 1, while I've seen all the players with sigma < 1 quite alot, and played all of them except HME.
If you're getting unknown players at the top of the leaderboard, setting a hard cutoff isn't the answer. It means that you think these people actually have much more rating uncertainty than what you calculated.
If you haven't switched from (mu - sigma) to (mu - 3*sigma) for the ranking, certainly do that. If you have, there might be other parameters you need to change.
-
If you're going with this I have a strong preference for cutting of at 1, as I've never seen any of those guys with sigma > 1, while I've seen all the players with sigma < 1 quite alot, and played all of them except HME.
If you're getting unknown players at the top of the leaderboard, setting a hard cutoff isn't the answer. It means that you think these people actually have much more rating uncertainty than what you calculated.
The issue with this is, he isn't saying they should be ranked lower because they aren't as good, but because he specifically hasn't played against or seen them. This could even just be time zone issues. It's really not a basis for anything.
If you haven't switched from (mu - sigma) to (mu - 3*sigma) for the ranking, certainly do that.
Why? Why why why why why?
-
If you haven't switched from (mu - sigma) to (mu - 3*sigma) for the ranking, certainly do that.
Why? Why why why why why?
My justification for leaderboards being conservative: people in general prefer to be conservative when proclaiming who's the best at anything. Otherwise, any newcomer on a hot streak would be proclaimed the best, only to be immediately dethroned as it turns out it was mostly luck. Requiring good performance over a period of many games is more stable.
Subtracting N*sigma from mu is just a way of formalizing that conservatism.
-
If you haven't switched from (mu - sigma) to (mu - 3*sigma) for the ranking, certainly do that.
Why? Why why why why why?
My justification for leaderboards being conservative: people in general prefer to be conservative when proclaiming who's the best at anything. Otherwise, any newcomer on a hot streak would be proclaimed the best, only to be immediately dethroned as it turns out it was mostly luck. Requiring good performance over a period of many games is more stable.
Subtracting N*sigma from mu is just a way of formalizing that conservatism.
I think we should go for mu - 50*sigma personally.
-
WW, your plaintive cries of "why" have straightforward answers. (edit: brain fart on who I was addressing)
You subtract some multiple of sigma because you need a confidence interval. You don't want false positives at the top of the leaderboard. If you have lots of false positives, it's not a leaderboard, it's a luckyboard, and everyone can recognize that and suggests blunt fixes by filtering out people who don't meet other criteria.
That multiple is 3*sigma because you want a 99% confidence interval, which handwavily means that 1 player in the top 100 will be there by a lucky fluke. You could choose a different number, sure. 2*sigma would probably be acceptable. 1*sigma gets into the silly range. The 50*sigma that you trollishly suggest is deep, deep into the silly range and you know it -- such a leaderboard would not function at all.
-
I hereby declare a ceasefire. I can anticipate the next three or four posts and I doubt that any of them will be particularly helpful. Let's keep this thread focused.
There seem to be three reasonable options:
- Sort by mu with a cutoff based on variance or number of games
- Sort by mu-k*sigma for some k between 1 and 3
- Implement the isotropic leaderboard's algorithm and be done with it
I'm pretty sure that nobody wants the fourth option, sorting by mu with no cutoff.
If I understand correctly, the purpose of sorting by mu-k*sigma is the same as that of implementing a cutoff. In both cases, the goal is to keep the top of the leaderboard from being filled up by mediocre players who have been lucky in a small number of games. Either option deviates from a rating system's one truly objective goal: estimating the probability that any given player beats another.
Microsoft Research appears to advocate the mu-k*sigma approach, but they don't take a strong stance on what k should be used. Using any k>0 means sorting players by a deliberate underestimate of their actual skill, but the degree of that underestimate varies with k. With k=3, a player's rank derives from the skill level that we're 99% confident is below their actual skill. To me, that seems a little excessive and possibly unfair to new players. This is what the leaderboard on drunkensailor.org is doing now, and it seems to be what Goko does as well.
Isotropic used k=1. In other words, a player's rank derived from the skill level that was 84%<?> certain to be below their actual skill level. This is still conservative, but not nearly as brutal to new/lucky players as mu-3*sigma. Iso also used an unusually high starting uncertainty: sigma=mu instead of sigma=mu/3. I'm not sure what the motivation for this was, but it explains why Iso had "levels" as high as 53 and as low as -35, while mine runs from 29 to -3.
Finally, note that sigma appears to converge to 0.80 in with my standard Trueskill implementation. A great many of the experience players have ratings between 0.79 and 0.82, and the lowest sigma in all of Goko is 0.78. On Iso, uncertainties never seem to have gotten below 6.5, and they didn't converge nearly as uniformly.
None of this makes sense to me. Intuitively, I would have expected uncertainties to converge asymptotically to zero. I also wouldn't have expected my uncertainties to converge any more uniformly than Iso's did. Are these anomalies evidence of a failure in TrueSkill, in my parameters, in my code, or in my intuition?
-
Isotropic uses mu - 3*sigma. The "uncertainty" number the leaderboard shows is 3*sigma.
-
Why are there a bunch of players with ".0000" after there username?
Lightning edit: After some more perusal, it looks like it's for duplicated usernames. It seems strange that Goko would allow those!
I suspect .0000 username is created after Facebook/Google+ login with duplicate name.
-
I vote for
3. Implement the isotropic leaderboard's algorithm and be done with it
-
While I certainly haven't put as much thought into this as some of you other guys have, you know, Awaclus suggestion doesn't seem half bad to me.
-
On Iso, uncertainties never seem to have gotten below 6.5, and they didn't converge nearly as uniformly.
None of this makes sense to me. Intuitively, I would have expected uncertainties to converge asymptotically to zero. I also wouldn't have expected my uncertainties to converge any more uniformly than Iso's did. Are these anomalies evidence of a failure in TrueSkill, in my parameters, in my code, or in my intuition?
Maybe it has something to do with GAMMA parameter in iso rating:
https://github.com/dougz/trueskill/blob/master/trueskill.py#L285
EDIT: Added quote from iso trueskill source linked above.
gamma is a small amount by which a player's uncertainty (sigma) is
increased prior to the start of each game. This allows us to
account for skills that vary over time; the effect of old games
on the estimate will slowly disappear unless reinforced by evidence
from new games.
-
Kirian, your plaintive cries of "why" have straightforward answers.
I believe I am the one crying 'why'. But whilst these answers may be straightforward, that doesn't mean they are good.
You subtract some multiple of sigma because you need a confidence interval.
Uh, why? You really don't. Beyond this, you could use 50% confidence, which would get you to 0*sigma. But subtracting some multiple of sigma does *not* give you a confidence level anyway. It would be the bottom value of a CI - why don't you use the top? Except that this doesn't even work anyway, because there really isn't any evidence that the thing you are measuring is normally distributed - even if you are assuming this to be the case with no justification (as TrueSkill implicitly does.... this doesn't really make sense though).
You don't want false positives at the top of the leaderboard.
You've now switched from using the language of CIs to the language of hypothesis tests. But it doesn't really make sense to have a hypothesis test in this case. What is the hypothesis you are testing? The big issue with using a hypothesis test here is that you have thousands of players which each need measurement, which leads to thousands of different hypotheses, which are all related, and all the hypotheses you could purport to be testing off the top of my head are going to have *every* player fail the test... But what does 'false positive' even mean here? A player who shows up as good but isn't? In setting up the system this way, though, aren't you creating lots of false negatives? I mean, by cutting down alpha so low, you are way increasing beta? Or to look at it another way, you are saying people who haven't played are really bad players. As I look at this, there are 6905 or 6906 (can't tell on one because of rounding) players who are rated as being better than a new player, and only 7301 players total. This simply doesn't make sense.
If you have lots of false positives, it's not a leaderboard, it's a luckyboard, and everyone can recognize that and suggests blunt fixes by filtering out people who don't meet other criteria.
If this is a problem, then it is because your underlying system is a bad one. After all, it is your system which thinks that these players actually have whatever strength you are assigning them. Indeed, the system you have now isn't a leaderboard either so much as a 'be-sorta-good-but-most-of-all-play-a-lot board'.
That multiple is 3*sigma because you want a 99% confidence interval, which handwavily means that 1 player in the top 100 will be there by a lucky fluke.
This isn't correct at all. First, again, it is assuming that each player's rating is normally distributed, which is wrong, but more than this, it's actually a 99.73% confidence interval even in that case, and okay, it looks like that's not a big difference, but it swings things from being 1 in 100 to 1 in 370. But okay, the whole normal distribution on a rating thing is just so wrong... if that were the case, we would have based on the players all starting with mu = 25 sigma = 25/3, one in 370 players should have mu over 50 and one in 370 should have mu less than zero. In actuality, you don't have anyone over 37 or below 11, with several thousand players.
You could choose a different number, sure. 2*sigma would probably be acceptable. 1*sigma gets into the silly range. The 50*sigma that you trollishly suggest is deep, deep into the silly range and you know it -- such a leaderboard would not function at all.
But you're being excessively handwaving here. 2*sigma is fine why? 1 is silly how? In fact, with just 1 sigma, you are already getting to where these people you haven't heard of aren't at the very tip top, it doesn't seem to make all that much difference from 3 to me... And how is 50 deeply deeply silly? I don't understand why it's silly to have it at a low number, silly to have a really high number, but good to have it be 2-3 only. What is so special about those values? I truly don't understand. My best guess is that this is just the way it looks to you because you are used to it - you do 95% confidence intervals or 98% or 99% all the time, so you don't like using less than 2 sigma, and 50 sigmas looks weird to you because it's magically too high a standard for you. Well, I agree it's too high a standard, but you have to look at what you are measuring - it's not something where we want to give anything but our best guess.
The real issue is that subtracting or adding any level of sigma from the mean distorts what it is you are measuring. You are no longer measuring the strength of the results alone and reporting it, but rather a combination of the strength and the number of games played*. This means that to maximize your level, it is no longer so much about playing well when you play, but about playing a lot and playing pretty well. The higher number of sigma you subtract essentially shifts this function toward the playing more side (while adding would shift it to the playing less side). I am strongly of the opinion that what number you are reporting as your measurement should match inasmuch as possible the actual objective of what it is you are trying to measure, and as players play to win, we should be trying to measure their ability to win and not their ability to play a lot.
*technically sigma isn't a straight-up measurement of how many games get played, but there is a very high correlation there, such that by far the easiest way to affect your sigma is to play a lot.
-
It does play a little into my mu-3*sigma point, insofar as the difference between 1.0 and 0.75 is basically nothing to the system, whereas the .75 difference in mu that it is equating to is pretty significant.
Your first listing of straight mu sort is obviously not the most desirable thing ever, but I actually don't see how the second is only 'a little more palatable': to me, this IS the leaderboard. I don't know about cutting off at 30 games - I would probably cut off at sigma of 1.5 or 1 or something. But basically, yeah, that's what I would go with.
The problem is that the leaderboard means different things to different people. Using the ATP as an example, Andy Murray has been playing very well over the past year, but he didn't play in the French Open this year, so he can't be #1. This doesn't mean you shouldn't expect him to be able to beat Djokovic in the US Open, it just means he hasn't done enough to deserve to be #1, based on the meaning of the ATP rankings. The rankings are not predictive, they are accomplishment-based.
Some people like it this way, and some people don't. Psychologically, sorting by mu makes it kind of crazy. People are already upset at how much one game affects your Goko rating, so I shudder to think how they'll feel about a leaderboard sorted by mu. However, the people who play the most often don't really want to be rewarded for simply playing more games, but only for playing better, so these people may tend to prefer to sort by mu.
Maybe a good compromise would be to provide both? Have the leaderboard be the standard Trueskill thing, but make it sortable by mu, so people can see that if they want?
The ATP is not a good comparison. Here's why:
For one thing, their ranking system is really terrible - it treats all events within the last year absolutely equally (well, okay, no, it gives more weight to more important tournaments; my point is that *when* the events happened doesn't matter - yesterday is given the same weight as 50 weeks ago), and then throws out everything before that. So, if Andy Murray loses every match he plays from now until next year's Wimbledon, and then he gets to the finals of Wimbledon next year but loses to Djokovic or Nadal (or whatever the best player is at that point), his ranking will DECREASE on the basis of that result, because it was worse than he did this year. It also takes only round into account, and not opponents. So the guy who beat Nadal in the early rounds this year got no more credit for that than someone beating random world number 58. (There are other issues with the system that aren't really relevant here, such as that it makes no differentiation based on surface).
But the big point of your post here is this: "The rankings are not predictive, they are accomplishment-based." Well, the problem with this is, what is an accomplishment? For Tennis, it sort of makes sense to do things as you suggest, because while they are 'open' tournaments, they don't really just let anyone play. But the biggest thing is, you measure the goal. Well, in tennis, it's not just about winning the highest percentage of matches against the best players you can. Oh sure, that's somewhat in there, but really you are trying to win Grand Slams, which is much more important, and then you are trying to win whatever other tournament, that's less important, etc. The point is that not all events are equally important, and so you want to take that into account when doing the rankings. That's not really true of Dominion, and moreover, not in a way which matters: In tennis, playing 20 events a year may well make you more tired, which can inhibit your ability to do as well in them. In Dominion, the fatigue factor... well, maybe it's not non-existent, but it's much less.
But most importantly along these lines, what is accomplishment? I would say that it's having good winning rates much moreso than winning a lot. The thing about accomplishment systems is that everything is positive, you don't lose rating. But I think losing 50 games and winning 2 should be rated worse than just winning any one of those same games, no?
Oh, it's also a misunderstanding if you think that sorting by mu will make the ratings swingier - it really won't.
-
There seem to be three reasonable options:
- Sort by mu with a cutoff based on variance or number of games
- Sort by mu-k*sigma for some k between 1 and 3
- Implement the isotropic leaderboard's algorithm and be done with it
I'm pretty sure that nobody wants the fourth option, sorting by mu with no cutoff.
I actually do want option number four. The problem with it is only that the system is probably quite wrong. Playing one game, no matter how good you do, shouldn't get you to high enough of a rating that you would be rated like number one on the leaderboard. I mean, that's actually an empirical question - does this make for a better rating system than the alternative or not? You all react against the mu sort because you think it should probably be worse. Well, I tend to agree with that line of thought, but it's an empirical question, and if it is one which we are right on, it actually just means the entire rating system is bad. Well, I suppose I would probably prefer an 'active' leaderboard, such that you fall off after a certain period of inactivity, but still I definitely want a mu sort.
If I understand correctly, the purpose of sorting by mu-k*sigma is the same as that of implementing a cutoff. In both cases, the goal is to keep the top of the leaderboard from being filled up by mediocre players who have been lucky in a small number of games. Either option deviates from a rating system's one truly objective goal: estimating the probability that any given player beats another.
Well, I don't actually think this is the goal of either, particularly of the mu-k*sigma sort, where I think the goal is to spur more playing. But you're quite right on the goal of a rating system, and this is really why I would want a mu sort - you are expected to be better than every player below you and worse than every player above you. This isn't the case with the current system.
Microsoft Research appears to advocate the mu-k*sigma approach,
If they have any research saying this, it's market research. Seriously, they put this in their general information about the system, but you don't see it in the scholarly papers, and there's really no statistical backing for it.
but they don't take a strong stance on what k should be used. Using any k>0 means sorting players by a deliberate underestimate of their actual skill, but the degree of that underestimate varies with k. With k=3, a player's rank derives from the skill level that we're 99% confident is below their actual skill. To me, that seems a little excessive and possibly unfair to new players. This is what the leaderboard on drunkensailor.org is doing now, and it seems to be what Goko does as well.
We actually still don't really know what Goko does. For sure they have some uncertainty thing such that playing more helps your rating, but for all I know it actually gets folded into a single rating number and not separated out as a mu and sigma kind of thing.
Isotropic used k=1.
Actually, iso used k = 3. You might be confused because the numbers they showed were mu+/-3*sigma, so it looks like they just subtracted the two numbers. But the second number displayed was 3*sigma, not just sigma.
In other words, a player's rank derived from the skill level that was 84%<?> certain to be below their actual skill level.
Except that this assumes that players' skills are normally distributed, which isn't true. But I've covered this.
This is still conservative, but not nearly as brutal to new/lucky players as mu-3*sigma. Iso also used an unusually high starting uncertainty: sigma=mu instead of sigma=mu/3. I'm not sure what the motivation for this was, but it explains why Iso had "levels" as high as 53 and as low as -35, while mine runs from 29 to -3.
As I've explained above, you have this wrong, because they displayed 3sigma and not sigma. But actually yours running from -3 to 29 is not something in your favor - if the ratings were actually normally distributed, you would have, based on your number of players, a much bigger range (of mu!) than you do.
Finally, note that sigma appears to converge to 0.80 in with my standard Trueskill implementation. A great many of the experience players have ratings between 0.79 and 0.82, and the lowest sigma in all of Goko is 0.78. On Iso, uncertainties never seem to have gotten below 6.5, and they didn't converge nearly as uniformly.
They don't actually converge to .80. It's just that it's very hard to get lower than that by playing the way people actually do. For that, I would have to look, but you would either need higher draw rates, or you'd need to do something like play very weak players a lot and win a lot. But it has to do with how their updating equations work, and basically there's enough uncertainty in the game that you can't get lower than this. I don't' think they should ever go down to 0 though, because you really can't ever get totally sure of what someone's skill is with no uncertainty. Anyway, iso's were higher because they incremented upward a little bit with every day that passed (and with every game? I can't recall exactly), which meant that to get them very low, you not only needed to do what you need to do for your system, but you needed to play a heckuva lot, all the time.
Anyway, the real thing to me is, the proof is in the pudding. You go with the system that best measures things, and the only way we have of telling this is based on the predictions, so you go with the thing which best predicts things. Since you are actually only making predictions centred on the value of mu, that is what you should be sorting by.
Edit: Incidentally, it has been suggested at points in the past that I have made such comments in a way which is self-serving. This is pretty clearly not the case here. Relative to other players (if I have counted correctly), this change would help me relative to 8 players, no change relative to 139 besides myself and hurt me relative to 7153.
-
While I certainly haven't put as much thought into this as some of you other guys have, you know, Awaclus suggestion doesn't seem half bad to me.
It was AI's suggestion, not mine. I just voted for it.
-
If what WW is true, that skill is not normally distributed, then there is little that can be done to accurately model win rates without doing a complete overhaul of the rating system. I don't know much about statistical analysis, but this question has me thinking.
The thing about Dominion is that there is so much inherent randomness, and that randomness is assumed to fit into a normal distribution, but then the skill is also assumed to be normally distributed. Is that correct? Think of games that are almost all pure skill or strength, like Starcraft or arm-wrestling. In those games, a player flat out wins against a much weaker player, barring exceptional circumstances (sickness maybe?). The players of games like that have to be very close in skill for there to even be a contest. So for those games, mu is much more informative about who will win that sigma. Things get complicated because skill/strength clearly decreases without continuous practice/training, so some sort of rating drift seems appropriate.
The normal distribution just seems more convenient than anything else. Both randomness and rating drift can be lumped into sigma, and that seems less controversial that the alternatives like decreasing mu or whatever. Applying another model would just be guesswork without empirical data, and so far the normal distribution doesn't seem to be THAT off. I think the normal distribution mainly has problems with predicting the win rate when the rating gap gets too large.
-
All models are wrong, some models are useful.
-
So long as a model helps predict outcomes given inputs and makes sense, it's doing what it's supposed to.
-
ITT: whining
-
The normal distribution just seems more convenient than anything else.
Well, and you can try to hide behind central limit theorems, even if they don't necessarily apply....
-
I forgot to say this: It's great that you went out of your way to make this. Awesome work, Andrew Iannaccone.
-
So, you're saying I'm only 4 levels behind Stef?
Guys I finally broke the equivalent of Iso level 40!
-
So, you're saying I'm only 4 levels behind Stef?
Guys I finally broke the equivalent of Iso level 40!
I'm still 4 steps behind my iso glory days as level 33 :(
-
I vote for
3. Implement the isotropic leaderboard's algorithm and be done with it
+1. That algorithm was good enough and people are used to it - reimplementing means I can compare how I play to how I played on Iso. Currently I have no idea how the level on Goko official leaderboard or the unofficial one compares to low 20ish level that was my Iso peak.
-
Ok, I've reached a conclusion. I'm wimping out.
I'm still interested in what sort of rating system is best for Dominion, but I don't feel qualified to answer that question. For now, what I can definitely do is give the peoples something they want and something that's better than Goko.
So please vote on which system you prefer. If there are two or three options you like equally, you can vote for all of them. I'm not going to implement multiple views, as I think that's just inviting chaos.
Incidentally, I found a way to see the full Goko rating data, or at least your mu and sigma together. It turns out that the rating they show is actually mu - 2 * sigma.
-
Incidentally, I found a way to see the full Goko rating data, or at least your mu and sigma together. It turns out that the rating they show is actually mu - 2 * sigma.
I'm very curious as to which part of that is too complicated to plop into a formula. Is it the subtraction, or perhaps the multiplication by two?
-
Incidentally, I found a way to see the full Goko rating data, or at least your mu and sigma together. It turns out that the rating they show is actually mu - 2 * sigma.
I'm very curious as to which part of that is too complicated to plop into a formula. Is it the subtraction, or perhaps the multiplication by two?
Goko already told us this directly. (http://forum.dominionstrategy.com/index.php?topic=6707.msg185148#msg185148)
It's the updating of mu and sigma that they are not explicitly revealing.
Edit: Since this post they added the drift downward for inactivity.
-
I would vote for Iso's rating system.
-
If what WW is true, that skill is not normally distributed, then there is little that can be done to accurately model win rates without doing a complete overhaul of the rating system. I don't know much about statistical analysis, but this question has me thinking.
The thing about Dominion is that there is so much inherent randomness, and that randomness is assumed to fit into a normal distribution, but then the skill is also assumed to be normally distributed. Is that correct? Think of games that are almost all pure skill or strength, like Starcraft or arm-wrestling. In those games, a player flat out wins against a much weaker player, barring exceptional circumstances (sickness maybe?). The players of games like that have to be very close in skill for there to even be a contest. So for those games, mu is much more informative about who will win that sigma. Things get complicated because skill/strength clearly decreases without continuous practice/training, so some sort of rating drift seems appropriate.
The normal distribution just seems more convenient than anything else. Both randomness and rating drift can be lumped into sigma, and that seems less controversial that the alternatives like decreasing mu or whatever. Applying another model would just be guesswork without empirical data, and so far the normal distribution doesn't seem to be THAT off. I think the normal distribution mainly has problems with predicting the win rate when the rating gap gets too large.
Not exactly. I mean, whatever distribution you're going to use is going to calibrate, so that essentially (this is a bit of a simplification) it will be accurate for your average matchup. Then, the curves aren't going to be *that* far off in the region of your matchup. I could spout to you a whole bunch of plausible curves that for their mid-sections are within half a percent of each other. And you're not going to really notice that - you probably won't even notice the difference between 55% and 58% very much, or at least certainly not very quickly. By far the biggest differences you are going to get though are in the further out matchups, where one player is heavily favored.
Anyway, long story short, the curve can be reasonably bad overall and still perform pretty well for the majority of matchups you're in, but the CIs break down very quickly if you aren't normal.
Also, @AI/ragingduckd or everyone really: I strongly think that the way to sort is just by mu, but for all that... it's not really that important, and I am anyway pretty happy with it so long as you continue to show it broken up, so that everyone can get out of it what they want, either way you end up sorting.
Incidentally, I found a way to see the full Goko rating data, or at least your mu and sigma together. It turns out that the rating they show is actually mu - 2 * sigma.
Interesting. How does it compare to what you have up?
-
How do i find me? is there a search engine for names? that would be awesome.
-
The other thing to consider is that if you are dumping data an HTML table, it's really easy to provide options to sort by various columns. I used this sorttable Javascript library (http://kryogenix.org/code/browser/sorttable/) a lot in councilroom.
Maybe the best answer to "which way to sort" is "who cares? Just click whichever column you like".
-
Maybe the best answer to "which way to sort" is "who cares? Just click whichever column you like".
Good thing -Stef- dropped his hyphens, now I can at least outrank him via alphabetical sort.
-
The other thing to consider is that if you are dumping data an HTML table, it's really easy to provide options to sort by various columns. I used this sorttable Javascript library (http://kryogenix.org/code/browser/sorttable/) a lot in councilroom.
Maybe the best answer to "which way to sort" is "who cares? Just click whichever column you like".
This.
-
How do i find me? is there a search engine for names? that would be awesome.
Ctrl-F doesn't work for you?
-
How do i find me? is there a search engine for names? that would be awesome.
If you're in windows, ctrl+f is your friend.
-
Woah, how am I nine levels behind Stef?
-
How do i find me? is there a search engine for names? that would be awesome.
Ctrl-F doesn't work for you?
{facepalm}
-
How do i find me? is there a search engine for names? that would be awesome.
Ctrl-F doesn't work for you?
{facepalm}
It's all right, man, we all have those days.
-
Woah, how am I nine levels behind Stef?
You're saying "only 9" or "9, so many"?
-
Feature request: "Last updated on Tue, Jul 23 at 05:19 AM PDT (5 minutes ago)"
-
Woah, how am I nine levels behind Stef?
You're saying "only 9" or "9, so many"?
"only 9". On iso I was around level 13, and you know where he was...
-
I was about level 1 on iso.
Now I'm 22. I feel so proud.
-
I was 3. Now I'm 24.
-
I was 3. Now I'm 24.
I'm 18... wait, this wasn't the age thread?
-
(http://www.myconfinedspace.com/wp-content/uploads/2009/07/at-first-I-was-like.jpg)
-
I wrote to dougz asking for clarification on his TS implementation, but I'll ask the same questions here in case any of you know the answers. The quotes are from http://web.archive.org/web/20130116154350/http://dominion.isotropic.org/faq/
In a nutshell: skill is measured on a scale that goes from roughly 0 to 50 points. (Actually skill can be any number, but 99.8% of players should fall in the 0–50 range.) The skill range column is a 99.8% confidence interval — the system is 99.8% sure your true skill lies somewhere in that range. New players are assigned a skill of "25 ± 25", which is to say, we don't really have any idea what that person's skill is.
Does 25 +/- 25 mean that you're starting with mu=25, σ=25? If so, then 0-50 would be more like a 68% CI, right? The default for the other implementations I've seen is 25 +/- 8.66. That is, σ=25/3.
The level column is the low end of your range, rounded down to an integer and clamped to the range [0, 50]. If we ignore the clamping, it is a conservative skill estimate in the sense that we are 99.9% confident that you are at least that skillful.
Did you eventually remove this on the upper end? I see a lot of people on your leaderboard who would be at large negative levels but are listed at 0. But I also see lespeutere and -Stef- over 50.
I've set β = 25, γ = σ0 / 100 (applied daily), and the draw probability at 5%.)
Empirically, I find more like a 1.75% draw rate. Was there any special reason for chooing 5%?
@Obi Wan Bonogi: You voted for isotropic's TS algorithm "without the decay," but dougz doesn't mention a decay in mu or a daily increase in σ. Can you clarify?
-
I'm pretty sure for iso initial sigma = 25/3 so the leaderboard value for a new player mu - 3*sigma is 25 ± 25. And the decay affects just the uncertainty parameter which comes into play with
γ = σ0 / 100 (applied daily)
-
I'm pretty sure for iso initial sigma = 25/3 so the leaderboard value for a new player mu - 3*sigma is 25 ± 25. And the decay affects just the uncertainty parameter which comes into play with γ = σ0 / 100 (applied daily)
The thing is, I'm using initial sigma=25/3 too but my final mu's and sigma's are both way smaller than isotropic's. I'm not sure what else I'm doing differently that could have such a large effect.
-
INITIAL_MU = 25.0
INITIAL_SIGMA = INITIAL_MU / 3.0
https://github.com/dougz/trueskill/blob/master/trueskill.py#L261
What do you use as BETA, EPSILON, GAMMA?
https://github.com/dougz/trueskill/blob/master/trueskill.py#L267
-
INITIAL_MU = 25.0
INITIAL_SIGMA = INITIAL_MU / 3.0
https://github.com/dougz/trueskill/blob/master/trueskill.py#L261
What do you use as BETA, EPSILON, GAMMA?
https://github.com/dougz/trueskill/blob/master/trueskill.py#L267
Aha! Thanks. I think I'm screwing up my betas. I'm using sublee's default of beta = INITIAL_SIGMA/2, while dougz used beta = 25.
Edit: I'm doing the epsilons and gammas differently too, but I think it's the betas that are causing real trouble.
-
I'm pretty sure for iso initial sigma = 25/3 so the leaderboard value for a new player mu - 3*sigma is 25 ± 25. And the decay affects just the uncertainty parameter which comes into play with γ = σ0 / 100 (applied daily)
Well, actually I don't understand this either. If σ0 is the starting sigma, a constant, then how can γ = σ0 / 100 be applied daily? Further, gamma is global, not player-specific.
-
I'm pretty sure for iso initial sigma = 25/3 so the leaderboard value for a new player mu - 3*sigma is 25 ± 25. And the decay affects just the uncertainty parameter which comes into play with γ = σ0 / 100 (applied daily)
Well, actually I don't understand this either. If σ0 is the starting sigma, a constant, then how can γ = σ0 / 100 be applied daily? Further, gamma is global, not player-specific.
If I understand it correctly for each player new sigma is daily calculated like this:
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
(I think pl.skill[1] is old sigma.)
https://github.com/dougz/trueskill/blob/master/trueskill.py#L345
-
I'm pretty sure for iso initial sigma = 25/3 so the leaderboard value for a new player mu - 3*sigma is 25 ± 25. And the decay affects just the uncertainty parameter which comes into play with γ = σ0 / 100 (applied daily)
Well, actually I don't understand this either. If σ0 is the starting sigma, a constant, then how can γ = σ0 / 100 be applied daily? Further, gamma is global, not player-specific.
If I understand it correctly for each player new sigma is daily calculated like this:
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
(I think pl.skill[1] is old sigma.)
https://github.com/dougz/trueskill/blob/master/trueskill.py#L345
Yes, that I follow. But the GAMMA in that equation is the same for all players in the game.
Actually, I think it's just constant over all games. It's possible that dougz was updating it for each matchup using isotropic code that isn't included in his trueskill package, but even then each player in that matchup would be using the same GAMMA regardless of their individual sigmas, so I still don't understand what γ = σ0 / 100 (applied daily) means.
-
I undersand it like
γ = σ0 / 100
corresponds to this code:
GAMMA = INITIAL_SIGMA / 100.0
https://github.com/dougz/trueskill/blob/master/trueskill.py#L307
And (applied daily)
means apply it as here:
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
https://github.com/dougz/trueskill/blob/master/trueskill.py#L345
I don't understand it completely. And probably because of that I don't get why using same GAMMA regardless of their individual sigmas is a problem.
-
so I still don't understand what γ = σ0 / 100 (applied daily) means.
In the early days Iso would increase the variance after each game. Since that put a hard floor on how low the variance could go, it was later changed to happen once a day. So "applied daily" means that you only fudge the variance once a day, rather than recalculating the (constant) gamma daily.
-
Yes, that I follow. But the GAMMA in that equation is the same for all players in the game.
Actually, I think it's just constant over all games. It's possible that dougz was updating it for each matchup using isotropic code that isn't included in his trueskill package, but even then each player in that matchup would be using the same GAMMA regardless of their individual sigmas, so I still don't understand what γ = σ0 / 100 (applied daily) means.
Yed covered most of it but just to clarify, at the end of every day, the sigma parameter for each player is adjusted by the gamma parameter. Gamma is the same for every player, every day and is equal to σ0 / 100 or 0.08333...
I had thought that it was just added to the end of the day sigma, but it seems it's figured in using the equation sigma_new = sqrt(sigma_old^2 + gamma^2). Actually, that way makes more sense as it provides a diminishing returns on how fast the variance increases.
-
I had thought that it was just added to the end of the day sigma, but it seems it's figured in using the equation sigma_new = sqrt(sigma_old^2 + gamma^2). Actually, that way makes more sense as it provides a diminishing returns on how fast the variance increases.
sigma^2 is the variance. ;)
(I know you know this!)
-
I had thought that it was just added to the end of the day sigma, but it seems it's figured in using the equation sigma_new = sqrt(sigma_old^2 + gamma^2). Actually, that way makes more sense as it provides a diminishing returns on how fast the variance increases.
sigma^2 is the variance. ;)
(I know you know this!)
Accurate terminology is for chumps.
-
Thanks all for your help. I think I've got the iso TS algorithm now, or at least a close and reasonable variant.
Yed covered most of it but just to clarify, at the end of every day, the sigma parameter for each player is adjusted by the gamma parameter. Gamma is the same for every player, every day and is equal to σ0 / 100 or 0.08333...
This makes sense mathematically, it's consistent with other TS descriptions, and it appears to be what dougz's code does.
I had thought that it was just added to the end of the day sigma, but it seems it's figured in using the equation sigma_new = sqrt(sigma_old^2 + gamma^2). Actually, that way makes more sense as it provides a diminishing returns on how fast the variance increases.
This is a plausible algorithm, but I don't see it in dougz's code (https://github.com/dougz/trueskill/blob/master/trueskill.py) or in his description (http://web.archive.org/web/20130116154350/http://dominion.isotropic.org/faq/), nor in others' TS descriptions or implementations. The term (σ02 + γ2) does make an appearance in the ordinary TS algorithm (what Dangauthier et al call "Vanilla" TrueSkill), but it's only in the per-game prior for each player at the top of the factor graph. That affects the post-game values of σ, but only after passing through the whole factor graph and the Bayesian update from the game result.
It's possible that dougz was doing this after each day's results as a modification of TS, though again I don't think that's what the description says. In any case, I just don't like it all that much. I suppose I'll include it if dougz says that it's what isotropic was using and/or if it gives a big boost to predictive performance.
-------
I ran sublee's TS implementation on the Goko data with with μ0=σ0=25, β=σ0/2, τ=σ0/100, and ε such that the draw probability is 5%. The parameters that generate my current leaderboard are μ0=25, σ0=μ0/3, β=σ0/2, τ=σ0/100, and ε such that the draw probability is 1.75%. Using μ0=σ0 is a little unusual, and 1.75% is the draw probability in my dataset, but these differences don't seem to matter much for either the ordering or the predictive performance (see below).
Note that what dougz calls τ isn't the "precision" τ:=1/σ2 but the σ-adjusting term we've been talking about. For reasons I cannot fathom, the TS papers and sublee's code use the symbol τ for both of these values. In dougz's code, he sensible renames the σ-adjustment term to γ (as in the equation at the top of this post).
The mean binomial deviance I calculated from the Goko sample was basically the same regardless of which parameters I used, 0.379 and 0.373 respectively. So that's not strong evidence that either set of parameters is superior. I don't know what mbd Glicko or Elo would generate on this data set, but I expect it's worse. I also expect that there's plenty of room for improvement here.
If assume that isotropic was displaying μ +/- 3σ as the skill range and also using that number for the level, then the leaderboard resulting from my might-be-iso parameters looks a fair amount like the iso leaderboard (http://dominion.isotropic.org/leaderboard/), and it's ordered a lot like my current implementation:
Stef - Level 57: 67.64 +/- 10.10 ( 893 games)
Mic Qsenoch - Level 53: 63.87 +/- 10.19 ( 837 games)
LESPEUTERE - Level 51: 62.05 +/- 10.24 ( 960 games)
nomnomnom - Level 50: 64.44 +/- 14.42 ( 118 games)
Rene Kuroi - Level 49: 59.74 +/- 10.65 ( 274 games)
Geronimoo - Level 48: 58.51 +/- 10.18 ( 717 games)
Andrew Iannaccone - Level 48: 58.10 +/- 9.99 (1154 games)
Wandering Winder - Level 48: 58.07 +/- 10.06 ( 854 games)
jog - Level 47: 58.04 +/- 10.09 ( 862 games)
Obi Wan Bonogi - Level 46: 57.10 +/- 10.12 ( 587 games)
Fabian - Level 45: 56.12 +/- 10.27 ( 366 games)
Boodaloo - Level 45: 58.44 +/- 12.94 ( 134 games)
Tao Chen - Level 45: 62.69 +/- 17.44 ( 86 games)
kenyou2859 - Level 45: 55.59 +/- 10.45 ( 394 games)
Rabid - Level 44: 54.98 +/- 10.11 ( 508 games)
SheCantSayNo - Level 44: 54.81 +/- 9.95 (1781 games)
blueblimp - Level 44: 55.34 +/- 10.89 ( 261 games)
HiveMindEmulator - Level 44: 56.90 +/- 12.53 ( 149 games)
eliegel - Level 44: 54.33 +/- 10.30 ( 398 games)
Stealth Tomato - Level 43: 54.07 +/- 10.24 ( 573 games)
shark_bait - Level 43: 54.48 +/- 10.78 ( 243 games)
Jeebus - Level 42: 53.45 +/- 10.54 ( 300 games)
manzi - Level 42: 53.03 +/- 10.18 ( 356 games)
Mike Harris.0001 - Level 42: 53.08 +/- 10.24 ( 423 games)
The major remaining difference is that the 3σ values are higher and more tightly clustered than on the iso board, but that's consistent with the smaller number and variation in number of games played.
Summary:
I find it pretty plausible that isotropic was using "Vanilla" TrueSkill with the parameters above. The difference between those parameters and the ones I would choose doesn't seem large, and I see no real justification for any major deviation from the standard algorithm. Until I hear otherwise from dougz or unless someone else has a compelling argument to the contrary, I'm going to change my leaderboard to use those parameters.
-
Ha ha, this leaderboard is so schizophrenic right now, I'm guessing it's catching up and parsing data?
-
Ha ha, this leaderboard is so schizophrenic right now, I'm guessing it's catching up and parsing data?
Yes, it can't make up it's mind. :D
Edit: All caught up. Now using TS with the parameters described two posts up, also implemented sortable rows, # of games, and those gray "Level" entries.
-
Aw, now I got demoted even more than I expected. :(
Oh well, now my main question is: "DID STEF GO SUPERHUMAN?!?"
-
Although I've wasted the last who knows how long minutes reading all the gammas and alphas and mus without a clue, this leader-board is brilliant work. Kudos to Andrew.
-
Aw, now I got demoted even more than I expected. :(
Oh well, now my main question is: "DID STEF GO SUPERHUMAN?!?"
Stef has always been superhuman. My main question is: "DID STEF GO SUPERSTEF?!?"
-
Is anyone else having the problem that this site is almost without fail causing crashes/extraordinarily high lag?
-
Is anyone else having the problem that this site is almost without fail causing crashes/extraordinarily high lag?
I don't know if anyone else is having the problem, but it has been working quite well for me.
-
Is anyone else having the problem that this site is almost without fail causing crashes/extraordinarily high lag?
I don't know if anyone else is having the problem, but it has been working quite well for me.
The table-sorting plugin is a bit of a memory hog... probably isn't really designed to deal with 8k rows. I guess it might be a problem with mobile devices, or maybe if you're running a JS debugger or certain browser add-ons.
In any case, I just disabled the sortable columns. See if it's still a problem.
-
Is anyone else having the problem that this site is almost without fail causing crashes/extraordinarily high lag?
I don't know if anyone else is having the problem, but it has been working quite well for me.
The table-sorting plugin is a bit of a memory hog... probably isn't really designed to deal with 8k rows. I guess it might be a problem with mobile devices, or maybe if you're running a JS debugger or certain browser add-ons.
In any case, I just disabled the sortable columns. See if it's still a problem.
It's been a problem almost every time I refreshed it since before the sorter, though it was usually just that it would be 15ish seconds of I can't do anything. The sorting made it WAY worse and totally froze me out to where I had to stop the browser's process. Just tried it again and first time it took 10 seconds or so, second time it was loading for a minute and I gave up on it. FWIW, it doesn't seem to be a memory thing - it's making my processor usage go up to the max I am letting my browser use (25% of my total processing power). And I am on a laptop.
Edit: Okay, seems to not really be an issue if I bring the page up in chrome (well, the sorting thing would still squawk and take forever to actually sort, but it wasn't locking me up). But the problems I am experiencing are in firefox (22.0), which is weird because iirc that's what you use to test....
-
Is anyone else having the problem that this site is almost without fail causing crashes/extraordinarily high lag?
I don't know if anyone else is having the problem, but it has been working quite well for me.
The table-sorting plugin is a bit of a memory hog... probably isn't really designed to deal with 8k rows. I guess it might be a problem with mobile devices, or maybe if you're running a JS debugger or certain browser add-ons.
In any case, I just disabled the sortable columns. See if it's still a problem.
It's been a problem almost every time I refreshed it since before the sorter, though it was usually just that it would be 15ish seconds of I can't do anything. The sorting made it WAY worse and totally froze me out to where I had to stop the browser's process. Just tried it again and first time it took 10 seconds or so, second time it was loading for a minute and I gave up on it. FWIW, it doesn't seem to be a memory thing - it's making my processor usage go up to the max I am letting my browser use (25% of my total processing power). And I am on a laptop.
Edit: Okay, seems to not really be an issue if I bring the page up in chrome (well, the sorting thing would still squawk and take forever to actually sort, but it wasn't locking me up). But the problems I am experiencing are in firefox (22.0), which is weird because iirc that's what you use to test....
Yes, I have no problems in FF. There's no client-side code anymore, so it shouldn't be taking any CPU. It's an unusually large page though (~2MB), which isn't a problem in itself, but I suppose it's possible that some FF Add-on is causing trouble, if it processes pages in a particularly CPU-intensive way. I have an appreciable delay when opening up Firebug, for example. Even if that's the problem though, the real solution would be for me to paginate the leaderboard so it's not such a massive page.
-
Yes, I have no problems in FF. There's no client-side code anymore, so it shouldn't be taking any CPU. It's an unusually large page though (~2MB), which isn't a problem in itself, but I suppose it's possible that some FF Add-on is causing trouble, if it processes pages in a particularly CPU-intensive way. I have an appreciable delay when opening up Firebug, for example. Even if that's the problem though, the real solution would be for me to paginate the leaderboard so it's not such a massive page.
Unless is causing trouble to the server, please don't do that, or at least keep the option to get the entire table. Being able to ctrl+f through it and browse up and down to look who you know or just played is close is great.
-
Maybe a dumb question, but why is the number of games so different from the number of games listed in the goko stats (when you click on avatars in the lobby)? Does the latter include casual or adventures or are some pro games excluded from these rankings?
-
Maybe a dumb question, but why is the number of games so different from the number of games listed in the goko stats (when you click on avatars in the lobby)? Does the latter include casual or adventures or are some pro games excluded from these rankings?
The latter includes casual and adventures, plus games against bots.
-
Maybe a dumb question, but why is the number of games so different from the number of games listed in the goko stats (when you click on avatars in the lobby)? Does the latter include casual or adventures or are some pro games excluded from these rankings?
The latter includes casual and adventures, plus games against bots.
Yes it even includes adventure games and bot games. At first I though that was kind of silly, but then I figured that every game adds to a player's experience, whether the opponent was a bot or not. It would be more deceptive to see someone who played 100+ bot games appeared as someone with less than 10 games played on their profile.
-
Yes, I have no problems in FF. There's no client-side code anymore, so it shouldn't be taking any CPU. It's an unusually large page though (~2MB), which isn't a problem in itself, but I suppose it's possible that some FF Add-on is causing trouble, if it processes pages in a particularly CPU-intensive way. I have an appreciable delay when opening up Firebug, for example. Even if that's the problem though, the real solution would be for me to paginate the leaderboard so it's not such a massive page.
Unless is causing trouble to the server, please don't do that, or at least keep the option to get the entire table. Being able to ctrl+f through it and browse up and down to look who you know or just played is close is great
I won't paginate unless I can find a graceful solution. Sorting a 8k row table in JS isn't very graceful though. I'll keep the option to display the whole table regardless. No server load problems so far.
-
Maybe a dumb question, but why is the number of games so different from the number of games listed in the goko stats (when you click on avatars in the lobby)? Does the latter include casual or adventures or are some pro games excluded from these rankings?
The latter includes casual and adventures, plus games against bots.
Yes it even includes adventure games and bot games. At first I though that was kind of silly, but then I figured that every game adds to a player's experience, whether the opponent was a bot or not. It would be more deceptive to see someone who played 100+ bot games appeared as someone with less than 10 games played on their profile.
Adventure games is a little weird though, since they're not really like real games. But that's not a subject for this thread I guess...
-
I won't paginate unless I can find a graceful solution. Sorting a 8k row table in JS isn't very graceful though. I'll keep the option to display the whole table regardless. No server load problems so far.
Maybe exclude all players with less than 10 (or so) played games? And/or maybe exclude all players who haven't played for over a month?
That should make the table significantly smaller, while keeping almost all interesting data there.
-
Aw, now I got demoted even more than I expected. :(
Oh well, now my main question is: "DID STEF GO SUPERHUMAN?!?"
Stef has always been superhuman. My main question is: "DID STEF GO SUPERSTEF?!?"
(http://cdn.meme.li/instances/600x/40123502.jpg)
-
The variance values in the new leaderboard are interesting. There seems to be a pretty hard floor right around 10/3 which is reached at about 600ish games played. Is there some sort of theoretical limit to how low sigma can get based on the model and is 10/3 a good value for that? My intuition doesn't tell me much here.
-
I was about level 1 on iso.
Now I'm 22. I feel so proud.
never mind 10. :-\
-
I was about level 1 on iso.
Now I'm 22. I feel so proud.
never mind 10. :-\
Wait, now I'm seven. I was 23 earlier, and on iso I was 13...
-
Is anyone else having the problem that this site is almost without fail causing crashes/extraordinarily high lag?
Yes, I am having 1/5 games crash right now.
Edit: My 99th post is reply 99 to this thread; neat :)
-
Is anyone else having the problem that this site is almost without fail causing crashes/extraordinarily high lag?
Yes, I am having 1/5 games crash right now.
Edit: My 99th post is reply 99 to this thread; neat :)
Unfortunately:
(http://i.qkme.me/3vbunt.jpg)
This is Andrew's leaderboard thread. However! Whining about Goko is encouraged—any thread over here (http://forum.dominionstrategy.com/index.php?board=28.0) should do.
-
This is a plausible algorithm, but I don't see it in dougz's code (https://github.com/dougz/trueskill/blob/master/trueskill.py)
So this line
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
https://github.com/dougz/trueskill/blob/master/trueskill.py#L345
does something else?
-
This is a plausible algorithm, but I don't see it in dougz's code (https://github.com/dougz/trueskill/blob/master/trueskill.py)
So this line
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
https://github.com/dougz/trueskill/blob/master/trueskill.py#L345
does something else?
Yeah, that updating is a normal part of the Vanilla TS algorithm. It happens once per game, not once per day. But going back to the original quote:
(For those interested in the details, I've set β = 25, γ = σ0 / 100 (applied daily), and the draw probability at 5%.)
... I understand your original statement now. I agree that dougz may have meant that he's doing the once-a-day updating in addition to or instead of the once-per-game updating. I think I'll wait until he chimes in to make any such changes though.
-
Yeah, that updating is a normal part of the Vanilla TS algorithm. It happens once per game, not once per day. But going back to the original quote:
Ok, my bad. Thx for explaining.
... I understand your original statement now. I agree that dougz may have meant that he's doing the once-a-day updating in addition to or instead of the once-per-game updating. I think I'll wait until he chimes in to make any such changes though.
Sure, sounds reasonable.
-
How impossible would it be to add a feature to the chrome extension where, instead of displaying the Goko pro rating, you just display the
isotropic drunkensailor leaderboard level, or something similar? Now that we have a real rating system, the Goko ratings somehow seem even more like a joke than usual.
-
How long till Stef gets to level 60?
-
How long till Stef gets to level 60?
Whenever I feel like letting him. ;)
-
How long till Stef gets to level 60?
Stef doesn't get to level 60. Stef makes level 60 come to him.
-
How impossible would it be to add a feature to the chrome extension where, instead of displaying the Goko pro rating, you just display the isotropic drunkensailor leaderboard level, or something similar? Now that we have a real rating system, the Goko ratings somehow seem even more like a joke than usual.
Well, they look pretty similar to me. This TS thing has increased my faith in Goko's leaderboard rather than decreased it.
-
How impossible would it be to add a feature to the chrome extension where, instead of displaying the Goko pro rating, you just display the isotropic drunkensailor leaderboard level, or something similar? Now that we have a real rating system, the Goko ratings somehow seem even more like a joke than usual.
Well, they look pretty similar to me. This TS thing has increased my faith in Goko's leaderboard rather than decreased it.
Well, losing ~5 games in a row won't shoot me from 5th to 30th place on the isotropic leaderboard, that's for sure.
-
How long till Stef gets to level 60?
Stef doesn't get to level 60. Stef makes level 60 come to him.
Wait, Stef is Chuck Norris and lives in Soviet Russia?
-
I don't like this. I'm rank 22 in both Casual and Pro on GOKO, but only rank 39 here :'(
-
I'm level 32 and ranked #159. Amazing. That is WAY higher than expected.
-
How impossible would it be to add a feature to the chrome extension where, instead of displaying the Goko pro rating, you just display the isotropic drunkensailor leaderboard level, or something similar? Now that we have a real rating system, the Goko ratings somehow seem even more like a joke than usual.
Well, they look pretty similar to me. This TS thing has increased my faith in Goko's leaderboard rather than decreased it.
Well, losing ~5 games in a row won't shoot me from 5th to 30th place on the isotropic leaderboard, that's for sure.
Have you actually tested this?
-
How impossible would it be to add a feature to the chrome extension where, instead of displaying the Goko pro rating, you just display the isotropic drunkensailor leaderboard level, or something similar? Now that we have a real rating system, the Goko ratings somehow seem even more like a joke than usual.
Well, they look pretty similar to me. This TS thing has increased my faith in Goko's leaderboard rather than decreased it.
Well, losing ~5 games in a row won't shoot me from 5th to 30th place on the isotropic leaderboard, that's for sure.
Have you actually tested this?
Let's just say the other night when I posted it, I was #30 on the goko leaderboard and 12th or 13th on the isotropish.
The goko ratings are way way too swingy and hugely depend on your latest results. Of course, the best players tend to have had good results in the recent past too, but, you know.
-
How impossible would it be to add a feature to the chrome extension where, instead of displaying the Goko pro rating, you just display the isotropic drunkensailor leaderboard level, or something similar? Now that we have a real rating system, the Goko ratings somehow seem even more like a joke than usual.
Well, they look pretty similar to me. This TS thing has increased my faith in Goko's leaderboard rather than decreased it.
Well, losing ~5 games in a row won't shoot me from 5th to 30th place on the isotropic leaderboard, that's for sure.
Have you actually tested this?
Yesterday I was 50 places higher on AI's leaderboard than on Goko's. Losing games drops me 1 or 2 places at most. Seems a lot less swingy to me.
-
The goko ratings are way way too swingy and hugely depend on your latest results. Of course, the best players tend to have had good results in the recent past too, but, you know.
In all fairness, the goko ratings have been made so swingy to get rid of the cheaters that "quitted" their way up the leaderboard. They were less swingy before, and eventually they will revert this to normal soon.
If i look at the goko leaderboard, if find names in the top spot that eventually belong there, and all of the cheaters are gone. For me thats a good sign: swingy or not, the right peolpe are at the top.
quote from getsatisaction on this very topic:
You are right: the subtraction is too tough now. This is on purpose because we had a situation where some people were cheating their way up the leaderboard by quitting (before quits were penalized) and setting it tough means that you have to actively play -- and play at your rating level -- in order to maintain your rating. It's a little more like "king of the hill". Remember, it's happening to everybody, not just you, so it's fair. But now that the overrated quit-cheaters have had time to drop, it's getting time to revert these setting to be more moderate.
-
Alright, now my level is more how it used to be. And up a couple levels too, yay!
-
I don't like this. I'm rank 22 in both Casual and Pro on GOKO, but only rank 39 here :'(
My difference is even greater. I am #22 on Goko Pro and #63 on Isotropish.
I tend to move around a lot on both leaderboards because I play inconsistently (both in frequency of playing pro games and quality of play -- TV can be so distracting sometimes). For instance, I've moved about 150 places on Goko's leaderboard after coming back from vacation and a week of playing casual games with friends. But in June, I was in the top 50 before dropping out of the top 100 because of poor play.
I am motivated to keep my pro rating above now 6000 because then I can legitimately title those games "Polk, 5440+".
-
Judging between this, the Goko leaderboard, and the leaderboard from back in the Isotropic days, I've been about the 20th-best player on the internet for more than a year.
This despite about half of the 19 above me having changed in that time.
-
From dougz in response to my questions (above):
No, I start with μ=25, σ=25/3 like everyone else. But since I display "μ ± 3σ" on the leaderboard it shows up as 25 ± 25.
I may not have ever actually had the clamping on the upper end.
Choice of 5%: pulled out of thin air.
Yes, there was a small daily increase in σ. (Moved 1% of the way back to 25/3, I think.) I didn't want people to be able to camp out on the leaderboard by getting to a good position and then not playing.
So the current parameters aren't quite Isotropic's. The "Isotropish" board is tougher on new players and less tough on inactive players. I'm not sure what influence the different draw percentage has.
Edit: The current parameters probably also lead to a greater overall spread in ratings, so top ratings are higher and bottom ratings are lower. So until/unless I change this, don't try to make much of direct comparisons between Isotropic and Isotropish.
-
Now loads partial leaderboard by default. Enabled column sorting on the server side. Both of these should help with performance/crash issues.
-
I suspect Goko's algorithm is more favourable to playing lower-level players than this leaderboard. I label my games 4000++ so I play a lot of players in the 4000-5000 range and tend to win a lot as a result. On the Goko leaderboard, this tends to keep me around the top 10 (right now at #4 due to a lucky streak) whereas on the isotropic-style leaderboard I'm sitting around #16.
Also there must be a lot less competition on Goko compared to isotropic, because there I would hang around the top 50-ish.
-
Also there must be a lot less competition on Goko compared to isotropic, because there I would hang around the top 50-ish.
I think there's probably more competition. It's just that the people competing on Goko aren't as competitive as the ones on Isotropic were.
-
Also there must be a lot less competition on Goko compared to isotropic, because there I would hang around the top 50-ish.
I think there's probably more competition. It's just that the people competing on Goko aren't as competitive as the ones on Isotropic were.
If you define competition as the number of people around your skill level, then Goko is less competitive at mid-high levels. The very top is probably around the same as Iso.
-
Now loads partial leaderboard by default. Enabled column sorting on the server side. Both of these should help with performance/crash issues.
Apparently, the leaderboard hasn't been updated since that change.
-
Now loads partial leaderboard by default. Enabled column sorting on the server side. Both of these should help with performance/crash issues.
Apparently, the leaderboard hasn't been updated since that change.
Whoops. I thought I'd squashed that bug. Fixed now. Thanks.
-
My write-in suggestion (I'm not very invested in this so don't take it too seriously): Remi Coulom's Whole History Rating.
It's similar to TrueSkill, but instead of incremental updates to a skill and variance parameter, it calculates "the exact maximum a posteriori over the whole rating history of all players". It's more computing intensive than TrueSkill, but still fairly fast.
-
My number of games doesn't seem to be going up. I only started keeping track quantitatively today, but I don't think it's moved in a while. It's certainly been stuck on 59 all day today (despite me having played 5ish games). My rating on the other hand, has changed. ???
-
My number of games doesn't seem to be going up. I only started keeping track quantitatively today, but I don't think it's moved in a while. It's certainly been stuck on 59 all day today (despite me having played 5ish games). My rating on the other hand, has changed. ???
Looks like the game count isn't updating properly... you have 127 80* rated games, not 59.
* 80 games of the sort that the leaderboard counts (two players, pro, not guest)
-
OK. It says 80 now. Not sure when I played 47 non-pro games though... Would that number include adventures (the games that aren't counted)?
-
OK. It says 80 now. Not sure when I played 47 non-pro games though... Would that number include adventures (the games that aren't counted)?
Here's what I have for you in my database:
rating | count
-----------+-------
unknown | 14
adventure | 27
casual | 4
pro | 83
"unknown" is games before 5/12/2013, which aren't included in the ratings or counted on the leaderboard. That leaves 3 pro games unaccounted for. One was excluded b/c it had 3+ players, but I'm not sure about the other two. Probably they were games where you beat 6500+ players. ;)
-
Haha. Probably not. Until recently I couldn't play any because of the rating caps. My leaderboard ranking is changing, but the number of games isn't again (it's stuck at 80). I just played at least 3 more games that should be counted (2 against bama and one against faust).
-
And now it's up to 84, which seems right. So I guess there's just some delay for updating number of games played but there isn't for rating shifts. That seems weird to me, but whatever.
-
Last recorded game finished 293 min, 41 seconds ago
Something is wrong...
-
Last recorded game finished 293 min, 41 seconds ago
Something is wrong...
Thanks. Fixed the bug. It wasn't creating the new day's log directory at midnight. Updating now.
-
So, has my skill increased since the good ol' days on iso, or is that the "ish" of isotropish?
-
So, has my skill increased since the good ol' days on iso, or is that the "ish" of isotropish?
It's hard to say. The starting variance is 3x higher and it doesn't include the daily increase in variance. So it's not as sensitive to how often you play and it's quicker to adjust at first and slower to adjust afterwards. There's probably more value in comparing how you stack up relative to others than comparing the levels directly.
None of this was really intentional, btw. I set it up some days before dougz told me his actual implementation details and then I was hesitatant to keep switching the leaderboard around. Maybe I should just switch to Isotropic's exact algorithm and be done with it.
-
Maybe I should just switch to Isotropic's exact algorithm and be done with it.
I vote for this.
-
I did what I never managed on Iso: Level 40! Yay.
-
Maybe I should just switch to Isotropic's exact algorithm and be done with it.
Yes, please.
-
So, has my skill increased since the good ol' days on iso, or is that the "ish" of isotropish?
It's hard to say. The starting variance is 3x higher and it doesn't include the daily increase in variance. So it's not as sensitive to how often you play and it's quicker to adjust at first and slower to adjust afterwards. There's probably more value in comparing how you stack up relative to others than comparing the levels directly.
None of this was really intentional, btw. I set it up some days before dougz told me his actual implementation details and then I was hesitatant to keep switching the leaderboard around. Maybe I should just switch to Isotropic's exact algorithm and be done with it.
Ok, it seems best to me too. So if anyone has a strong objection, then now's the time to say so. Otherwise I'm definitely going to implement the Isotropic algorithm just as soon as I get around to it.
-
The earlier observations seem to suggest that the Goko ranking is fairly reasonable (that is, close to the trueskill leaderboard) near the top of the ranking. Has anyone looked at how they compare for the lower ranks? Or is it too hard to retrieve a full Goko ranking?
I'm asking because I noticed by chance that the bots seem to rank higher in the Trueskill list.
Bottington has now level 14 in Trueskill and rating 3576 in Goko (and three other bots are, btw, between levels 12-13 so their levels are quite stable). I checked three human players who also have level 14 in Trueskill (and enough games, at least 500 for each) and each of those had rating around 4500 in Goko, full 1000 points higher. The same holds for ranks; Bottington is ranked 886 in Trueskill and around 1500 in Goko, whereas the humans were around 900 in Trueskill and around 700 in Goko. The opposite check shows similar signs. Three arbitrary managers ranked near Bottington in Goko had levels 4, 8 and 11 in Isotropic. These are again lower than Bottington's rank in Trueskill.
Overall, it looks like the bots have at least some hundreds of points too low rating in Goko or, equivalently, perhaps 5-10 levels too high rating in Trueskill. If we believe Trueskill to be a better estimate, then this implies that pushing one's rating up in Goko by beating the bots is actually a bit harder than it should be. In fact, this also matches my own experience. I've played primarily against bots (because I very often need to pause during playing) and my rank in Trueskill is 156 compared to 673 in Goko.
Disclaimer: The above observation might simply be because of very small and potentially biased sample.
-
I think the bots just haven't played enough games for their rating to converge.
-
Has anyone looked at how they compare for the lower ranks? Or is it too hard to retrieve a full Goko ranking?
It's a nuisance. I don't have the data anyway.
I'm asking because I noticed by chance that the bots seem to rank higher in the Trueskill list.
TS doesn't count games with guests or 3+ players. It's hard to know what effect those have on the Bot ratings.
-
How difficult is it to create a system that includes 3P and 4P the way Isotropic did? (1st records a win over all other players; everyone else records a loss against 1st; I forget whether 3rd vs. 2nd was a draw or a non-game.)
Edit: I accidentally hit the Edit button instead of reply. Sorry. --ragingduckd
-
How difficult is it to create a system that includes 3P and 4P the way Isotropic did? (1st records a win over all other players; everyone else records a loss against 1st; I forget whether 3rd vs. 2nd was a draw or a non-game.)
Trueskill is actually well designed for this. I've excluded 3+ player games because I feel that they represent a different skill set.
That said, anyone who's interested in this is certainly welcome to use my code and database to implement it. It's all open source, after all.
-
Edit: I accidentally hit the Edit button instead of reply. Sorry. --ragingduckd
Oh, I've done this...
-
I think the bots just haven't played enough games for their rating to converge.
I guess this was a joke, but you are kind of right anyway. Today Lord Bottington has only level 9 and is down 300 positions, while Conqueror has reached level 16. Despite 5000+ games for some of the bots their levels are fluctuating heavily, and I probably simply happened to look at the ratings when Bottington was unusually high.
Anyway, high fluctuation in the mean skill is interesting in itself. Trueskill often has the tendency of the uncertainty to drop artificially low, which also prevents large changes in skill, but perhaps the bots truly vary so much in actual skill that their rating never stabilizes? I guess that would be caused by their AI being so much worse on some kingdoms than it is on others.
-
I think bot's rating is affected by series of games with same result. If newbies lose to them a lot they go up, if someone good is climbing to 5000, they go down...
-
Who really cares about the trueskill from bots.??
-
Who really cares about the trueskill from bots.??
I only looked at them for the reason that they might reveal something about the Goko rating system (and possibly about Trueskill in the context of Dominion as well, even though the underlying mechanisms are known), out of academic interest as someone who has (very briefly) worked with designing such rating systems.
Potential under/overestimation of bot skill in Goko is also relevant from a meta game design perspective. While hardcore Dominion players might not care about the internal ranking in Goko, the leaderboard is an integral part of the product Goko is offering and non-negligible part of their customers play against bots frequently enough to be affected by such potential bias in their quest of playing the meta game of leaderboard climbing.
Edit: Also, if the bots would, for some reason, have biased estimates in Trueskill, playing against them could be used to game the system. You probably couldn't reach the very top that way, since they have so low skill, but maybe the average players could get a few extra levels that could matter for a contest amongst a group of friends?
-
Who really cares about the trueskill from bots.??
I only looked at them for the reason that they might reveal something about the Goko rating system (and possibly about Trueskill in the context of Dominion as well, even though the underlying mechanisms are known), out of academic interest as someone who has (very briefly) worked with designing such rating systems.
Potential under/overestimation of bot skill in Goko is also relevant from a meta game design perspective. While hardcore Dominion players might not care about the internal ranking in Goko, the leaderboard is an integral part of the product Goko is offering and non-negligible part of their customers play against bots frequently enough to be affected by such potential bias in their quest of playing the meta game of leaderboard climbing.
Edit: Also, if the bots would, for some reason, have biased estimates in Trueskill, playing against them could be used to game the system. You probably couldn't reach the very top that way, since they have so low skill, but maybe the average players could get a few extra levels that could matter for a contest amongst a group of friends?
I am sure in casual you could reach the top by playing the bots by pre-selecting an engine kingdom they have no chance of winning
-
Who really cares about the trueskill from bots.??
I only looked at them for the reason that they might reveal something about the Goko rating system (and possibly about Trueskill in the context of Dominion as well, even though the underlying mechanisms are known), out of academic interest as someone who has (very briefly) worked with designing such rating systems.
Potential under/overestimation of bot skill in Goko is also relevant from a meta game design perspective. While hardcore Dominion players might not care about the internal ranking in Goko, the leaderboard is an integral part of the product Goko is offering and non-negligible part of their customers play against bots frequently enough to be affected by such potential bias in their quest of playing the meta game of leaderboard climbing.
Edit: Also, if the bots would, for some reason, have biased estimates in Trueskill, playing against them could be used to game the system. You probably couldn't reach the very top that way, since they have so low skill, but maybe the average players could get a few extra levels that could matter for a contest amongst a group of friends?
I am sure in casual you could reach the top by playing the bots by pre-selecting an engine kingdom they have no chance of winning
Rats and Ambassador.
-
Just got this error message when trying to go to the site:
Unhandled exception (probably a programming error). Please report in the forums if you can reproduce it.
-
I just got the same message.
-
Looks like I've joined the ranks of the bots:
(http://img841.imageshack.us/img841/3352/mtu5.jpg)
-
Looks like I've joined the ranks of the bots:
(http://img841.imageshack.us/img841/3352/mtu5.jpg)
Well, maybe Villager Bot will be as strong as you are once he's played a couple hundred more games...
-
I'm not sure if this is a bug or not, but the uncertainty value should decrease as your number of recent games increases right? Because just played ~10 games and at the start of the day my 3sigma was 10.18 (I think) and now it's 10.23. I could be wrong on the exact numbers, but I know it increased as I played games.
-
If your recent results have been anomalous relative to your previous ones (either winning or losing more often than expected), your sigma should increase.
-
If your recent results have been anomalous relative to your previous ones (either winning or losing more often than expected), your sigma should increase.
Yes, I thought about that, I just didn't know if the Trueskill algorithm actually worked that way. Because on iso your uncertainty would always decrease with the number of recent games, I never noticed any dependency on the outcomes.
-
If your recent results have been anomalous relative to your previous ones (either winning or losing more often than expected), your sigma should increase.
Yes, I thought about that, I just didn't know if the Trueskill algorithm actually worked that way. Because on iso your uncertainty would always decrease with the number of recent games, I never noticed any dependency on the outcomes.
I am pretty sure TrueSkill does work this way. The reason you didn't see it on iso was because it had the upward time increment, which prevented you from ever (or at least, almost ever) getting to the point where the general downward trend from more games -> more certainty to be thus counterbalanced.
I am sure this isn't a bug, though whether it's more accurate or not is less clear.
-
Feature request: Would it be easy to include goko's official rating scores in the table? I would like to see a comparison of both.
-
Feature request: Would it be easy to include goko's official rating scores in the table? I would like to see a comparison of both.
AI has said that it's a real pain to scrape the entire Goko leaderboard.
-
AI has said that it's a real pain to scrape the entire Goko leaderboard.
Ok, nevermind then. Thanks for the quick response.
-
I realize this might not be relevant for this particular leaderboard, as AI has indicated that he just wants to implement the isotropic formula and be down with it, but perhaps this will be useful for those who expressed an interest in creating a better rating system for Dominion (like WW).
I was thinking that perhaps it would make sense to implement something that takes into account first player advantage. Like the expectation for a win by first player is increased. Since turn order is random the effect of on rating would probably be minimal, but it still seems like something that makes sense to include and could be one solution to the oft-repeated question of how to mitigate the P1 edge. Additionally, if this algorithm were interested in multi-player games, it could increase the uncertainty on the outcome with more players (and decrease the expectation that p4 wins). Anyway, just a thought I had. I wouldn't be surprised if someone had already considered it, but I thought I'd offer it here.
-
... as AI has indicated that he just wants to implement the isotropic formula and be down with it, ...
Oh yeah...
-
... as AI has indicated that he just wants to implement the isotropic formula and be down with it, ...
Oh yeah...
Sorry I haven't gotten around to this. My development schedule is as erratic as my personal schedule.
I just banned ottocar from the Isotropish leaderboard for cheating way too blatantly (http://gokologs.drunkensailor.org/logsearch?p1name=ottocar&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=10%2F07%2F2013&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&submitted=true&offset=0). No public witch hunts please, but if anyone else out there is cheating in some equally obvious way, feel free to PM me about it.
-
... as AI has indicated that he just wants to implement the isotropic formula and be down with it, ...
Oh yeah...
Sorry I haven't gotten around to this. My development schedule is as erratic as my personal schedule.
I just banned ottocar from the Isotropish leaderboard for cheating way too blatantly (http://gokologs.drunkensailor.org/logsearch?p1name=ottocar&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=10%2F07%2F2013&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&submitted=true&offset=0). No public witch hunts please, but if anyone else out there is cheating in some equally obvious way, feel free to PM me about it.
I need to play against this 11111 guy, he seems to suck.
-
I need to play against this 11111 guy, he seems to suck.
There is a great thing to do with someone that sucks that is not playing Dominion.
-
Right behind you as second human player in the list SCSN ! I'm not an addict, it's cool ...
-
Big thanks AI for doing this. This is great.
Also:
I did what I never managed on Iso: Level 40! Yay.
Ditto. Yeah. Now I only need to play again to appear on the full leaderboard.
-
Hi,
I have just seen a new way to lose points while supposedly winning
http://dom.retrobox.eu/?/20131114/log.50d804c7e4b0b34255505855.1384490982851.txt
This game ended on my opponents turn 12. I assumed that while he was ahead, he saw that I was most likely going to win and ended it. If you look at the bottom of the log there are two game over boxes, one with my opponent winning and one with me winning. I noticed something was wrong when I went to the leaderboard post game and saw that while there was a +21 next to my name, I had lost about 20 points. I would have lost about 40 if he had won because of rating differences. Its like we played two games in that one and split them.
Has anyone seen this before or is it new?
AI how do you parse this log?
-
Hi,
I have just seen a new way to lose points while supposedly winning
http://dom.retrobox.eu/?/20131114/log.50d804c7e4b0b34255505855.1384490982851.txt
This game ended on my opponents turn 12. I assumed that while he was ahead, he saw that I was most likely going to win and ended it. If you look at the bottom of the log there are two game over boxes, one with my opponent winning and one with me winning. I noticed something was wrong when I went to the leaderboard post game and saw that while there was a +21 next to my name, I had lost about 20 points. I would have lost about 40 if he had won because of rating differences. Its like we played two games in that one and split them.
Has anyone seen this before or is it new?
AI how do you parse this log?
My parser reads the final number of VPs from the log and counts how many turns each player took. It then draws its own conclusions about who won.
It checks those conclusions against what Goko prints at the bottom of the log, but it ignores any discrepancy in the known cases where Goko screws up the rankings and/or number of turns. See the conditions at the end of the parser's code (https://github.com/aiannacc/goko-dominion-tools/blob/master/gdt/logparse/gokoparse.py).
This is one of those known situations, so the parser ignores the ranks listed in the log (both versions) and records its own conclusions. From the log search engine:
2013/11/14 20:49
Player: Rank: VPs: Turns: Quit?
kylar 2nd 5 12
untiedshoe 1st 12 12
Provinces / Estates - Caravan, Copper, Coppersmith, Curse, Duchy, Estate, Forager, Fortress, Gold, JackOfAllTrades, Mine, Nobles, Potion, Province, Silver, Steward, Talisman, Vineyard
Log Viewer Kingdom
-
See the conditions at the end of the parser's code (https://github.com/aiannacc/goko-dominion-tools/blob/master/gdt/logparse/gokoparse.py).
Wow, it is possible to screw it this many ways...
-
I think the leaderboard stopped updating. If you have time, it would be great if you have a look at it.
"Last recorded game finished 132 min, 43 seconds ago"
-
I think the leaderboard stopped updating. If you have time, it would be great if you have a look at it.
"Last recorded game finished 132 min, 43 seconds ago"
I restarted it. Dunno what went wrong.
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
Honestly, I have no idea. It seems suspicious to me too.
The code is here (https://github.com/aiannacc/goko-dominion-tools/blob/master/gdt/ratings/gdt_trueskill.py) if anyone wants to look at it. Basically I'm just outsourcing the job to this TrueSkill package (http://trueskill.org/).
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
This could be a natural property of the game. I mean sometimes you do everything possible and still lose.
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
This could be a natural property of the game. I mean sometimes you do everything possible and still lose.
Possibly, but the isotropic leaderboard (http://dominion.isotropic.org/leaderboard/) didn't look quite like that.
-
Maybe there's a lower signal-to-noise ratio on Goko, with more games being lost because of bad connections, abandoned due to slowplaying, &c.?
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
This could be a natural property of the game. I mean sometimes you do everything possible and still lose.
Possibly, but the isotropic leaderboard (http://dominion.isotropic.org/leaderboard/) didn't look quite like that.
Could it possibly have something to do with the games played that count toward your rank? The isotropic leaderboard includes all eligible games, so lespeutere, for example, has over 10,000 games counting toward his ranking. Meanwhile since the isotropish includes only games played in the last month, lespeutere has just under 3000 games counting toward the isotropish leaderboard. I haven't followed all of the implementation talk and such, so this is just a bystander's casual look at the rankings, but that seems to be the biggest difference in the two leaderboards.
-
Could it possibly have something to do with the games played that count toward your rank? The isotropic leaderboard includes all eligible games, so lespeutere, for example, has over 10,000 games counting toward his ranking. Meanwhile since the isotropish includes only games played in the last month, lespeutere has just under 3000 games counting toward the isotropish leaderboard. I haven't followed all of the implementation talk and such, so this is just a bystander's casual look at the rankings, but that seems to be the biggest difference in the two leaderboards.
I think the isotropish leaderboard includes all eligible games in its ranking, it just doesn't display your name on the leaderboard if you haven't played a game in the last month.
-
AI: please restart it again: Last recorded game finished 40 min, 21 seconds ago
-
AI: please restart it again: Last recorded game finished 40 min, 21 seconds ago
Ok, and I've turned on massive debugging too. Until now I haven't even been able to find an error in the log. The loop seems to just stop silently.
-
Is it possible that log gathering is not working? I have two games played about 10 minutes ago that do not appear in the search and the leaderboard was updated more than 30 minutes ago according to its own displayed info (every time I remember before was less than a minute ago).
-
Is it possible that log gathering is not working? I have two games played about 10 minutes ago that do not appear in the search and the leaderboard was updated more than 30 minutes ago according to its own displayed info (every time I remember before was less than a minute ago).
Yes... it shut off again. I restarted it.
Even with my in-depth logging, I can't figure out why this is happening. I don't mind restarting it whenever people notice it's busted tho.
-
Sorry to bother you AI. But there is something wrong for sure. The # of games isn't updating anymore.
I don't know if this is only a display bug or if the rating is also calculated wrong, but the # games doesn't rise anymore.
-
Sorry to bother you AI. But there is something wrong for sure. The # of games isn't updating anymore.
I don't know if this is only a display bug or if the rating is also calculated wrong, but the # games doesn't rise anymore.
Restarted it again.
This is probably the same bug that's stopped the system the last several times. I've looked for it to no avail and I don't expect to have time for a serious bug hunt any time soon.
Just PM me if/when this happens again, rather than posting here. No need to clutter up the thread.
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
This could be a natural property of the game. I mean sometimes you do everything possible and still lose.
Possibly, but the isotropic leaderboard (http://dominion.isotropic.org/leaderboard/) didn't look quite like that.
Could it possibly have something to do with the games played that count toward your rank? The isotropic leaderboard includes all eligible games, so lespeutere, for example, has over 10,000 games counting toward his ranking. Meanwhile since the isotropish includes only games played in the last month, lespeutere has just under 3000 games counting toward the isotropish leaderboard. I haven't followed all of the implementation talk and such, so this is just a bystander's casual look at the rankings, but that seems to be the biggest difference in the two leaderboards.
It's pretty much a natural property of TrueSkill. There is a point at which you can't decrease your uncertainty any more, because the uncertainty increase per game balances the uncertainty decrease you get by playing.
This happened on Isotropic too. It had two different sources of uncertainty, though, one that was per-game and one that was per-day. The players with games >> days could get their uncertainty down to 6-ish.
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
This could be a natural property of the game. I mean sometimes you do everything possible and still lose.
Possibly, but the isotropic leaderboard (http://dominion.isotropic.org/leaderboard/) didn't look quite like that.
Could it possibly have something to do with the games played that count toward your rank? The isotropic leaderboard includes all eligible games, so lespeutere, for example, has over 10,000 games counting toward his ranking. Meanwhile since the isotropish includes only games played in the last month, lespeutere has just under 3000 games counting toward the isotropish leaderboard. I haven't followed all of the implementation talk and such, so this is just a bystander's casual look at the rankings, but that seems to be the biggest difference in the two leaderboards.
It's pretty much a natural property of TrueSkill. There is a point at which you can't decrease your uncertainty any more, because the uncertainty increase per game balances the uncertainty decrease you get by playing.
This happened on Isotropic too. It had two different sources of uncertainty, though, one that was per-game and one that was per-day. The players with games >> days could get their uncertainty down to 6-ish.
But the per-day source only ever increased the uncertainty on Isotropic, according to Doug; so how could people get uncertainties substantially below 9.9 on Isotropic, but not on Isotropish?
(I don't think this can only be due to the number of games played either - e.g. jog now has more games on oko, but a much lower uncertainty (8.6) on Isotropic.)
-
I noticed that as of today, the Isotropish Leaderboard only displays the top 100 ranked players. It only displays the top 100 now regardless of whether you select partial or full displays. As a newer player, I have enjoyed being able to reference the Isotropish Leaderboard to review my stats as I try to work my way up the leaderboard, but I'm still down around the 400 rank or so, and now I can no longer see my stats or those of my other family and friends who are also not in the top 100. Anybody know if this is a known issue, or if there is someplace else I should report it. I don't really do any coding myself, so do not have a sign-on on Github. Thanks.
-
I noticed that as of today, the Isotropish Leaderboard only displays the top 100 ranked players. It only displays the top 100 now regardless of whether you select partial or full displays. As a newer player, I have enjoyed being able to reference the Isotropish Leaderboard to review my stats as I try to work my way up the leaderboard, but I'm still down around the 400 rank or so, and now I can no longer see my stats or those of my other family and friends who are also not in the top 100. Anybody know if this is a known issue, or if there is someplace else I should report it. I don't really do any coding myself, so do not have a sign-on on Github. Thanks.
Whoops. Sorry about that. It's back to normal now.
Thanks for letting me know.
-
I noticed that as of today, the Isotropish Leaderboard only displays the top 100 ranked players. It only displays the top 100 now regardless of whether you select partial or full displays. As a newer player, I have enjoyed being able to reference the Isotropish Leaderboard to review my stats as I try to work my way up the leaderboard, but I'm still down around the 400 rank or so, and now I can no longer see my stats or those of my other family and friends who are also not in the top 100. Anybody know if this is a known issue, or if there is someplace else I should report it. I don't really do any coding myself, so do not have a sign-on on Github. Thanks.
Whoops. Sorry about that. It's back to normal now.
Thanks for letting me know.
Wow...fast response! Thanks much!!...I rarely got that fast of service even when I was an IT manager. ; )
-
Here are a couple of videos from one of the three guys who developed TrueSkill. He talks about some other stuff, too.
(The first link has three videos - second video there is where most of the TS stuff is for that link, though he sets up mathematical principlas in the first part).
http://videolectures.net/acml2013_herbrich_real_time_bayesian_learning/
http://videolectures.net/acml2013_herbrich_technology_transfer/
Anyway, might post some more thoughts on this later, but thought people here might be interested.
-
Here are a couple of videos from one of the three guys who developed TrueSkill. He talks about some other stuff, too.
(The first link has three videos - second video there is where most of the TS stuff is for that link, though he sets up mathematical principlas in the first part).
http://videolectures.net/acml2013_herbrich_real_time_bayesian_learning/
http://videolectures.net/acml2013_herbrich_technology_transfer/
Anyway, might post some more thoughts on this later, but thought people here might be interested.
Related:
Microsoft Research has a lot of good material on TrueSkill. For instance, here is an overview (http://research.microsoft.com/en-us/projects/trueskill/), a detailed description (http://research.microsoft.com/en-us/projects/trueskill/details.aspx), a lengthy article (http://www.moserware.com/2010/03/computing-your-skill.html), and an academic-style article (http://research.microsoft.com/apps/pubs/default.aspx?id=67956) on the system.
-
A crazy question about the calculations: is there something intrinsic to the system that creates a floor for sigma? There's literally no one--no matter how many games played--with a sigma under 9.92. What's up with that?
This could be a natural property of the game. I mean sometimes you do everything possible and still lose.
Possibly, but the isotropic leaderboard (http://dominion.isotropic.org/leaderboard/) didn't look quite like that.
Could it possibly have something to do with the games played that count toward your rank? The isotropic leaderboard includes all eligible games, so lespeutere, for example, has over 10,000 games counting toward his ranking. Meanwhile since the isotropish includes only games played in the last month, lespeutere has just under 3000 games counting toward the isotropish leaderboard. I haven't followed all of the implementation talk and such, so this is just a bystander's casual look at the rankings, but that seems to be the biggest difference in the two leaderboards.
It's pretty much a natural property of TrueSkill. There is a point at which you can't decrease your uncertainty any more, because the uncertainty increase per game balances the uncertainty decrease you get by playing.
This happened on Isotropic too. It had two different sources of uncertainty, though, one that was per-game and one that was per-day. The players with games >> days could get their uncertainty down to 6-ish.
But the per-day source only ever increased the uncertainty on Isotropic, according to Doug; so how could people get uncertainties substantially below 9.9 on Isotropic, but not on Isotropish?
(I don't think this can only be due to the number of games played either - e.g. jog now has more games on Goko, but a much lower uncertainty (8.6) on Isotropic.)
I think I've found the explanation (right in this thread ;)):so I still don't understand what γ = σ0 / 100 (applied daily) means.
In the early days Iso would increase the variance after each game. Since that put a hard floor on how low the variance could go, it was later changed to happen once a day. So "applied daily" means that you only fudge the variance once a day, rather than recalculating the (constant) gamma daily.
So apparently Isotropic only increased the variance per day, not also per game. Therefore its variances varied much more, depending on the number of games played per day. On Isotropish, the increase per game leads to an absolute floor, as we currently observe. Players with >1 game/day usually had lower variances on Isotropic, players with <1 game/day usually have lower variances now.
-
I think I've found the explanation (right in this thread ;)):so I still don't understand what γ = σ0 / 100 (applied daily) means.
In the early days Iso would increase the variance after each game. Since that put a hard floor on how low the variance could go, it was later changed to happen once a day. So "applied daily" means that you only fudge the variance once a day, rather than recalculating the (constant) gamma daily.
So apparently Isotropic only increased the variance per day, not also per game. Therefore its variances varied much more, depending on the number of games played per day. On Isotropish, the increase per game leads to an absolute floor, as we currently observe. Players with >1 game/day usually had lower variances on Isotropic, players with <1 game/day usually have lower variances now.
I don't think I understand. Isotropish doesn't increase variance after each game either. I would expect that the more games you play per day under dougz's system, the closer you get to the floor your variance hits on Isotropish. I don't get how your variance could actually get lower.
-
I don't think I understand. Isotropish doesn't increase variance after each game either. I would expect that the more games you play per day under dougz's system, the closer you get to the floor your variance hits on Isotropish. I don't get how your variance could actually get lower.
I haven't looked at your code, but you said previously that you just used the vanilla TrueSkill algorithm, which does increase the variance (by GAMMA^2) once for each game:
So, what is going on here? Between any two games of a gamer, the TrueSkill ranking system assumes that the true skill of a gamer, that is, μ, can have changed slightly either up or down; this property is what allows the ranking system to adapt to a change in the skill of a gamer. Technically, this is achieved by a small increase in the σ of each participating gamer before the game outcome is incorporated.
gamma is a small amount by which a player's uncertainty (sigma) is
increased prior to the start of each game. This allows us to
account for skills that vary over time; the effect of old games
on the estimate will slowly disappear unless reinforced by evidence
from new games.
This is a plausible algorithm, but I don't see it in dougz's code (https://github.com/dougz/trueskill/blob/master/trueskill.py)
So this line
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
https://github.com/dougz/trueskill/blob/master/trueskill.py#L345
does something else?
Yeah, that updating is a normal part of the Vanilla TS algorithm. It happens once per game, not once per day. But going back to the original quote:
(For those interested in the details, I've set β = 25, γ = σ0 / 100 (applied daily), and the draw probability at 5%.)
... I understand your original statement now. I agree that dougz may have meant that he's doing the once-a-day updating in addition to or instead of the once-per-game updating. I think I'll wait until he chimes in to make any such changes though.
(Bold by me.) So you apply GAMMA once per game, while Doug (according to qmech's quote above) ended up applying it only once per day, not per game.
-
TL;DR: Isotropish and Isotropic use an identical updating algorithm, but with different parameters values. I think it's these different values that give Isotropic lower uncertainties.
I haven't looked at your code, but you said previously that you just used the vanilla TrueSkill algorithm, which does increase the variance (by GAMMA^2) once for each game:
So, what is going on here? Between any two games of a gamer, the TrueSkill ranking system assumes that the true skill of a gamer, that is, μ, can have changed slightly either up or down; this property is what allows the ranking system to adapt to a change in the skill of a gamer. Technically, this is achieved by a small increase in the σ of each participating gamer before the game outcome is incorporated.
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
...
(Bold by me.) So you apply GAMMA once per game, while Doug (according to qmech's quote above) ended up applying it only once per day, not per game.
I understand now. I think our confusion is Microsoft's fault. ;)
(http://research.microsoft.com/en-us/projects/trueskill/equations.jpg)
This is Vanilla TrueSkill updating as described on Microsoft's TrueSkill detail page (http://research.microsoft.com/en-us/projects/trueskill/details.aspx). So I think what Microsoft means by the rather misleading "... small increase in the σ of each participating gamer before the game outcome is incorporated" is that the scaling factor c is not just the sum of the players' variances, but also includes 2β². That's the sense in which the variance is increased. The variable itself isn't increased by β², but its effect on the updating is.
Okay, I'm not sure I'd believe me either if I read that, so here's evidence that dougz and I are doing the exact same updating. Attached are scripts that do an example ratings update in two different ways: Isotropish's and dougz's. Note that the 3rd party package I use in isotropish.py has the same name as dougz's ("trueskill") even though it's entirely different code. You can run these yourself if you download those packages: for isotropish (https://github.com/sublee/trueskill) and for dougz (https://github.com/dougz/trueskill).
Output (identical for both scripts):
Before the game:
Alice: 25.0000 +/- 8.3333
Bob: 33.3333 +/- 8.3333
After Alice wins:
Alice: 26.8396 +/- 8.1911
Bob: 31.4937 +/- 8.1911
So we really are running the same algorithm for updating after individual games. I've discovered, however, that there are some differences between my parameters and those that dougz used. See updated version of the OP (http://forum.dominionstrategy.com/index.php?topic=8900.msg269929#msg269929) for details. I suspect that it's actually these that account for the difference in our variance floors.
-
I understand now. I think our confusion is Microsoft's fault. ;)
Fucking Microsoft. Figures.
:)
-
(http://research.microsoft.com/en-us/projects/trueskill/equations.jpg)
I've always been a bit skeptical about this whole idea that colors make things easier to understand, but this is some convincing shit.
-
Seeing c^2 as a denominator in a big equation makes me think they probably took this from some physics model.
-
TL;DR: Isotropish and Isotropic use an identical updating algorithm, but with different parameters values. I think it's these different values that give Isotropic lower uncertainties.
I haven't looked at your code, but you said previously that you just used the vanilla TrueSkill algorithm, which does increase the variance (by GAMMA^2) once for each game:
So, what is going on here? Between any two games of a gamer, the TrueSkill ranking system assumes that the true skill of a gamer, that is, μ, can have changed slightly either up or down; this property is what allows the ranking system to adapt to a change in the skill of a gamer. Technically, this is achieved by a small increase in the σ of each participating gamer before the game outcome is incorporated.
sigma=sqrt(pl.skill[1] ** 2 + GAMMA ** 2)
...
(Bold by me.) So you apply GAMMA once per game, while Doug (according to qmech's quote above) ended up applying it only once per day, not per game.
I understand now. I think our confusion is Microsoft's fault. ;)
I agree, but in another way, I think. :P
(http://research.microsoft.com/en-us/projects/trueskill/equations.jpg)
This is Vanilla TrueSkill updating as described on Microsoft's TrueSkill detail page (http://research.microsoft.com/en-us/projects/trueskill/details.aspx). So I think what Microsoft means by the rather misleading "... small increase in the σ of each participating gamer before the game outcome is incorporated" is that the scaling factor c is not just the sum of the players' variances, but also includes 2β². That's the sense in which the variance is increased. The variable itself isn't increased by β², but its effect on the updating is.
I don't think this interpretation of β is correct; according to DougZ's code documentation, β is a measure of how random the game is, and is clearly distinct from gamma:
beta is a measure of how random the game is. You can think of it as
the difference in skill (mean) needed for the better player to have
an ~80% chance of winning. A high value means the game is more
random (I need to be *much* better than you to consistently overcome
the randomness of the game and beat you 80% of the time); a low
value is less random (a slight edge in skill is enough to win
consistently). The default value of beta is half of INITIAL_SIGMA
(the value suggested by the Herbrich et al. paper).
[...]
gamma is a small amount by which a player's uncertainty (sigma) is
increased prior to the start of each game. This allows us to
account for skills that vary over time; the effect of old games
on the estimate will slowly disappear unless reinforced by evidence
from new games.
Now the Microsoft formulas quoted don't even mention gamma. On the Details page, they seem to imply that the increase in uncertainty happens before these formulas are used: "Before starting to determine the new skill beliefs of all participating players for a new game outcome, the TrueSkill ranking system assumes that the skill of each player may have changed slightly between the current and the last game played by each player."
So I think they just neglected to give the corresponding equations for the uncertainty increase, and only give the more "interesting" change based on the game's outcome.
I also notice that DougZ does talk about an uncertainty increase per game here; so if qmech was right, this would be DougZ' old TrueSkill implementation, before he switched to an uncertainty increase per day. If this is the case, it's not so surprising that it leads to the same output as your script, because both implementations do it once per game...
-
I don't think this interpretation of β is correct; according to DougZ's code documentation, β is a measure of how random the game is, and is clearly distinct from gamma:
...
Now the Microsoft formulas quoted don't even mention gamma.
Ah, okay. I follow now. You're quite right about beta vs gamma. And gamma isn't in Microsoft's docs because they call it tau. I don't know why they didn't just conform to dougz's notation. Nope. It looks like most of the Microsoft researchers also call it gamma. Some of their researchers and sublee (http://trueskill.org) call it tau though.
I also notice that DougZ does talk about an uncertainty increase per game here; so if qmech was right, this would be DougZ' old TrueSkill implementation, before he switched to an uncertainty increase per day. If this is the case, it's not so surprising that it leads to the same output as your script, because both implementations do it once per game...
And now I understand what you're saying here too. Yeah, I suppose it's possible that dougz set tau=0 and compensated with a daily increase in sigma. Using tau=0 would certainly drive the uncertainty through the floor... I just tried a random sample of 5000 games, and Banker Bot was down to sigma=0.9 by the end. So maybe with the daily increase it could work out to sane-looking but small values sigmas.
-
I don't think this interpretation of β is correct; according to DougZ's code documentation, β is a measure of how random the game is, and is clearly distinct from gamma:
...
Now the Microsoft formulas quoted don't even mention gamma.
Ah, okay. I follow now. You're quite right about beta vs gamma. And gamma isn't in Microsoft's docs because they call it tau. I don't know why they didn't just conform to dougz's notation.
I also notice that DougZ does talk about an uncertainty increase per game here; so if qmech was right, this would be DougZ' old TrueSkill implementation, before he switched to an uncertainty increase per day. If this is the case, it's not so surprising that it leads to the same output as your script, because both implementations do it once per game...
And now I understand what you're saying here too. Yeah, I suppose it's possible that dougz set tau=0 and compensated with a daily increase in sigma. Using tau=0 would certainly drive the uncertainty through the floor... I just tried a random sample of 5000 games, and Banker Bot was down to sigma=0.9 by the end. So maybe with the daily increase it could work out to sane-looking but small values sigmas.
Holger seems to get this stuff. I don't think you are (I don't mean this as an insult - it's an incredibly daunting, technical wall of text). If you look here (http://research.microsoft.com/pubs/74419/TR-2006-80.pdf), they do talk about both Gamma and Tau. Gamma is about halfway down page 2. They don't explicitly give an equation for its effect - it's in words "an additive variance component" (of gamma squared) "in the subsequent prior" - i.e. you are literally adding gamma squared to the player's stored uncertainty every "so often". They say at the bottom that in what they did for Halo on Xbox live is use a Gamma squared = (Sigma_initial/100)^2. It's not clear how often this update is accruing though.
Tau is not the same thing - they are using Tau for the Precision-adjusted mean, Mu*Pi = Mu/(sigma^2), as they say in the first few lines after the box of equations on page 5. This is because they're working with inverse space a lot, as it's convenient for a lot of things they want to do.
I don't know what parameter values DougZ used, or if they'd be correct.
Oh, a few other notes - you are saying you are trying to figure out the best parameter values, and I applaud you for this. My question is how - assuming you can figure out how to actually measure what their effect is on past data, what error/accuracy metric are you using? I would suggest maximum likelihood - effectively this means log likelihood. Also, you have in your OP right now that the Initial Ratings have sigma = 25, but that should be sigma^2 = 25/3. He's also using Beta^2 = 25/6, not just 25, if I am reading the code right (this also jives with what I remember) - this is also the exact same as the value you say it is much different than - I mean, I think Doug just remembered a little wrong when he sent that message.
I am sure the biggest, maybe the only differences between your system and DougZ's are in the updating - i.e. the Gamma updating which could occur per-game or per-time, and you clearly aren't doing the same. I think it's a bit misleading to say that what you're doing is exactly the same, because while it's technically right to say that "it's the same with parameter values", lots of systems are going to be true by that - the parameter values are REALLY important.
Anyway, I hope it goes well for you. I might look at cracking into some of this stuff later on (maybe even today) if I have some time, but I don't know how long it's going to take me, and I probably wouldn't proceed under a TS framework.
-
Holger seems to get this stuff. I don't think you are (I don't mean this as an insult - it's an incredibly daunting, technical wall of text). If you look here (http://research.microsoft.com/pubs/74419/TR-2006-80.pdf), they do talk about both Gamma and Tau. Gamma is about halfway down page 2. They don't explicitly give an equation for its effect - it's in words "an additive variance component" (of gamma squared) "in the subsequent prior" - i.e. you are literally adding gamma squared to the player's stored uncertainty every "so often". They say at the bottom that in what they did for Halo on Xbox live is use a Gamma squared = (Sigma_initial/100)^2. It's not clear how often this update is accruing though.
Tau is not the same thing - they are using Tau for the Precision-adjusted mean, Mu*Pi = Mu/(sigma^2), as they say in the first few lines after the box of equations on page 5. This is because they're working with inverse space a lot, as it's convenient for a lot of things they want to do.
Microsoft doesn't mention either gamma or tau on their details page, which I believe is what Holger was referring to. I believe dougz is following Herbrich in using gamma as the dynamics factor. Moser and sublee use tau in place of gamma, possibly following Dangauthier et al, 2008 (http://research.microsoft.com/apps/pubs/default.aspx?id=74417).
We're all talking about the same thing though:
Herbrich & Graepel:
(http://i.imgur.com/BZ5ZMlV.png)
(http://i.imgur.com/RoIjhzN.png)
Moser (http://www.moserware.com/2010/03/computing-your-skill.html):
(http://i.imgur.com/tKiABq7.png)
dougz:
https://github.com/dougz/trueskill/blob/master/trueskill.py#L347
sublee:
https://github.com/sublee/trueskill/blob/master/trueskill/__init__.py#L316
https://github.com/sublee/trueskill/blob/master/trueskill/factorgraph.py#L99
Oh, a few other notes - you are saying you are trying to figure out the best parameter values, and I applaud you for this. My question is how - assuming you can figure out how to actually measure what their effect is on past data, what error/accuracy metric are you using? I would suggest maximum likelihood - effectively this means log likelihood.
Yes, I find max likelihood compelling. Herbrich and Graepel use "prediction error (fraction of teams that were predicted in the wrong order before the game)," which also seems reasonable and maybe easier to understand.
Also, you have in your OP right now that the Initial Ratings have sigma = 25, but that should be sigma^2 = 25/3. He's also using Beta^2 = 25/6, not just 25, if I am reading the code right (this also jives with what I remember) - this is also the exact same as the value you say it is much different than - I mean, I think Doug just remembered a little wrong when he sent that message.
The OP now correctly shows what Isotropish is actually doing and what I now believe that dougz was doing. Isotropish really is still using σ0=25, though that will change soon. I'm assuming those "^2"s are just typos, but where are you reading dougz's leaderboard code or seeing beta=25/6?
-
Stuff
Okay, all this makes sense now. I think we're on the same page.
The reason I like maximum likelihood over the 'fraction of teams that were predicted in the wrong order before the game' is that scale is really important. Under prediction error, predicting that X has a 50.1% chance of winning against Y is the same as predicting that X has a 93% chance of winning against Y, and I really think you want to scale there. Basically, I want rating more than just ranking.
-
(https://i.chzbgr.com/maxW500/325645056/hAF3C4F4A/)
-
(https://i.chzbgr.com/maxW500/325645056/hAF3C4F4A/)
... yelled the online Dominion tournament organizer.
-
(https://i.chzbgr.com/maxW500/325645056/hAF3C4F4A/)
... yelled the online Dominion tournament organizer.
You know it's serious when that's the guy calling you a nerd.
-
Just noticed that the Isotropish Leaderboard has not recorded any new games now for over 4 hours, so it probably needs some attention.
Just curious - when this happens, will the games that finished during this period will eventually be posted to the leaderboard, or is some of the data lost?
Also - Is there somewhere else where I should report issues like this rather than on this forum topic? I don't have a GitHub ID, but I could certainly get one if that is the better place to report issues with the non-Goko software.
Thanks.
-
Just noticed that the Isotropish Leaderboard has not recorded any new games now for over 4 hours, so it probably needs some attention.
Thanks. I restarted it. Someday I will definitely figure out why that happens.
Just curious - when this happens, will the games that finished during this period will eventually be posted to the leaderboard, or is some of the data lost?
Everything gets posted... eventually.
Almost everything gets posted when I restart the downloader, but I've encountered some rare bugs that cause a game to get skipped (recorded but not rated). In such cases, the game doesn't show up until the next time I regenerate the leaderboard from scratch (i.e. starting from game one in May 2013). I'd like to claim that that can't happen anymore, but you probably shouldn't trust a programmer who 100% guarantees his code to be bug-free.
There's also a MF/Goko bug that can result in a game log not getting created at all. Or there was in recent months anyway.
Also - Is there somewhere else where I should report issues like this rather than on this forum topic? I don't have a GitHub ID, but I could certainly get one if that is the better place to report issues with the non-Goko software.
Probably just sending me an e-mail or PM is best.
-
Any idea why one would be massively higher rated on Isotropish than on Goko? I'm currently #587 on Goko, but #229 on Isotopish. Infact, I always seem to be higher rated on Isotropish.
-
Any idea why one would be massively higher rated on Isotropish than on Goko? I'm currently #587 on Goko, but #229 on Isotopish. Infact, I always seem to be higher rated on Isotropish.
Perhaps because your uncertainity is high, which means that Goko drops your rating more than Isotropish does.
-
Any idea why one would be massively higher rated on Isotropish than on Goko? I'm currently #587 on Goko, but #229 on Isotopish. Infact, I always seem to be higher rated on Isotropish.
Perhaps because your uncertainity is high, which means that Goko drops your rating more than Isotropish does.
No, Blueswan has Isotropish uncertainty of only 10.2, close to the lowest uncertainty of 9.x. The 1 sigma difference between the two ratings is unlikely to ever be the cause of such a difference, since Isotropish gives you a massively higher starting uncertainty - you start at 1000 points in Goko, but at level -50 (!) in Isotropish.
The most probable reason is that Blueswan plays very few games per day/week, so Goko's daily increase in uncertainty dominates the rating. It's the same for me.
Edit: Some bots (and one player) now have uncertainties <9. Was the algorithm changed in the meantime, or is this a rare coincidence?
-
Any idea why one would be massively higher rated on Isotropish than on Goko? I'm currently #587 on Goko, but #229 on Isotopish. Infact, I always seem to be higher rated on Isotropish.
Perhaps because your uncertainity is high, which means that Goko drops your rating more than Isotropish does.
No, Blueswan has Isotropish uncertainty of only 10.2, close to the lowest uncertainty of 9.x. The 1 sigma difference between the two ratings is unlikely to ever be the cause of such a difference, since Isotropish gives you a massively higher starting uncertainty - you start at 1000 points in Goko, but at level -50 (!) in Isotropish.
The most probable reason is that Blueswan plays very few games per day/week, so Goko's daily increase in uncertainty dominates the rating. It's the same for me.
Edit: Some bots (and one player) now have uncertainties <9. Was the algorithm changed in the meantime, or is this a rare coincidence?
The last few days I've been playing quite a bit, but up until a couple of weeks ago I pretty much hadn't played in 6 months or so. Still trying to find my groove. Getting better today even if my Goko rating is still fairly mediocre.
I don't know if it could have anything to do with the fact that the vast majority of my games are against Conqueror Bot.
-
The last few days I've been playing quite a bit, but up until a couple of weeks ago I pretty much hadn't played in 6 months or so. Still trying to find my groove. Getting better today even if my Goko rating is still fairly mediocre.
That explains it then. Isotropish froze your rating from 6 months ago, while Goko probably increased your uncertainty by a factor >4 as "punishment" for not playing that long; it will take lots of games to reduce it to its previous value. That's why I prefer Isotropish ratings...
I don't know if it could have anything to do with the fact that the vast majority of my games are against Conqueror Bot.
You'd need a few more games against weaker opponents to increase your rating/reduce your uncertainty, but it shouldn't matter too much.
-
Before you ask, yes, something is definitely wrong. I'm looking into it.
Edit: fixed.
-
My leaderboard is stuck at 830 minutes ago. Don't know if that was what you were trying to fix, but I don't think it is fixed...
-
My leaderboard is stuck at 830 minutes ago. Don't know if that was what you were trying to fix, but I don't think it is fixed...
You're right. I broke something else with my previous fix.
Now it's running again, but it skipped all the games from the last 7 hours. I'm fixing that too, but it'll take an hour or two. If you played games in the last 7 hours, your rating will "jump" when I switch to the fixed version that incorporates those games.
-
Okay, obviously there is something wrong with the new logparsing/leaderboard code I wrote last week. The weird part is that whenever I regenerate the leaderboard from scratch, this nonsense goes away:
(http://i.imgur.com/Ftg3Zly.jpg)
-
Okay, obviously there is something wrong with the new logparsing/leaderboard code I wrote last week. The weird part is that whenever I regenerate the leaderboard from scratch, this nonsense goes away:
(http://i.imgur.com/Ftg3Zly.jpg)
There's me thinking I had lost most of my games recently....
-
I've been having a number of problems with online games the last few days. I just posted the following in a Tech Support thread at the Making Fun site where I described the issues. Note my last concern #5 as it relates directly to the Isotropish Leaderboard. It just seems out of line that 3 new user id's would pop up at the top of the list of number of games in the last few days, and that all 3 of them would have over 50,000 games, and that Banker Bot would have shot up so much in games. If the numbers are correct, then either a) Goko/Makingfun is using these ID's to test the system; b) Someone is gaming the Goko/Makingfun system with automated bot programs and that could be why the servers have been slow; or c) Someone is gaming the Isotropish leaderboard. If there is another possible answer, I can't think of it at the moment... I don't usually expand the full Iso list, but I have in the past, and these names did not roll to the top of # of games list, so I do not think it is just because they have not played for a while. 2 of the new id's show in the partial list, but elhobb only shows in the full list. It is also odd that the numbers for the Goko Bots have changed with Banker Bot shooting up so significantly, and if my memory is correct, Conqueror Bot dropping down in # of games, as Conqueror Bot always had notably the most games.
******** Copy of my post on Makingfun site
(http://forum.makingfun.com/showthread.php?4653-Lots-of-problems-with-games-last-couple-days)
Lots of problems with games last couple days
The last couple days I have had lot of problems with games:
1) Very slow response time, with frequent long lags.
2) Games not starting sometimes after hitting the play button. They just never start and I have to hit the Create button to start a new game.
3) One time where the game started but the whole screen was displayed with dark gray overlay so that I had to hit refresh to go back to home screen.
4) One game where it hung up during the game and never came back. I checked the game log, and it was logged as a Quit against a Bot, but it was not counted as a loss on my loss record. While it doesn't seem quite fair to get a loss when the system hangs up, I think it is important for any game logged as a quit to count as a loss. Otherwise the door is still open for people to game the system and simply quit anytime they are behind to preserve their record without a loss. Especially if this can be done against bots without a quit counting as a loss it will cause the potential for players to be fixing even the pro ratings by just playing bots and quitting any time they are behind to achieve a perfect record.
5) Finally, I noticed in the Isotropish rankings taken from Goko games (http://gokologs.drunkensailor.org/le...rtkey=numgames) that if you sort them by number of games played there has been some radical changes at the top. I could usually sort by # of games to quickly see all the Bots at the top and see what their respective ISO ratings were. Now I see that Banker Bot has gone up by at least 20,000 or 30,000 games in the last week or so. I also see 3 new user names which have never been at the top (frankdom, LuciferousPeridot, and elhobb) - all 3 of these users suddenly show as having over 50,000 games. Now I understand this is a 3rd party site not maintained by Goko/Makingfun, but if the numbers are true, it might indicate that the servers are slow because they are being swamped by some sort of automated programmed bots (who perhaps have been playing Banker Bot?). Perhaps these are Makingfun tests?? Something you probably want to look at.
-
So I see now that LuciferousPeridot is a user in this forum. I have just never noticed that ID having over 50,000 games before and being up there with the number of games from the bots. Is this all still just issues with the programming for the Isotropish list?
-
So I see now that LuciferousPeridot is a user in this forum. I have just never noticed that ID having over 50,000 games before and being up there with the number of games from the bots. Is this all still just issues with the programming for the Isotropish list?
Yes, same bug. I'm pretty sure it's only affecting those few players who have an absurd number of games and/or unreasonably high sigma. Everyone else's ratings still appear to be getting calculated correctly.
If the numbers are correct, then either a) Goko/Makingfun is using these ID's to test the system; b) Someone is gaming the Goko/Makingfun system with automated bot programs and that could be why the servers have been slow; or c) Someone is gaming the Isotropish leaderboard. If there is another possible answer, I can't think of it at the moment...
d) AI broke the leaderboard code.
I guess I should be flattered that this wasn't #1 on your list of possibilities. ;)
-
I posted the following P.S. at Makingfun site:
P.S. Regarding #5 - in the meantime, going to the Dominion Strategy Forum (forum.dominionstrategy.com), I see that at least LuciferousPeridot is a true user, and also that they have been having issues with the Isotropish Leaderboard counts, so it may be just an issue with the Isotropish Leaderboard at this time. I would probably hold on investigating #5 until the folks who maintain the Isotropish Leaderboard have had a chance to look at their system.
-
So I'm not an automated script. Phew!
-
So I'm not an automated script. Phew!
Well, you could be trying to pass the Turing test here.
-
So I'm not an automated script. Phew!
Well, you could be trying to pass the Turing test here.
I'm calling AmISelfAware() but it gets stuck in a strange loop.
-
To be honest, I worry about the reverse Turing test, where it turns out I'm actually a
cylon machine.
-
To be honest, I worry about the reverse Turing test, where it turns out I'm actually a cylon machine.
Voight-Kampff test?
-
Voight-Kampff test?
It's too bad she won't live, but then again, who does!
-
To be honest, I worry about the reverse Turing test, where it turns out I'm actually a cylon machine.
Tanya had me fooled for a while, but she just posted, "Unfortunately online services needs maintenance every noun, and we have to do it."
-
To be honest, I worry about the reverse Turing test, where it turns out I'm actually a cylon machine.
Voight-Kampff test?
FDS: "What do you mean you're not helping?"
Goko: "I mean we're not helping, guys."
-
This guy seems to be a hard core player, that just wins against the good players and loses against the Newbies :) Over 50000 games and with a 3sigma of 23.98 :o.
Maybe you can take a look Andrew, it seems something could be wrong with the Leaderboard
Level 11
22.35 ± 10.85 1248 241 doyouknow
21.72 ± 10.23 1249 338 Metatemujin
35.47 ± 23.98 1250 50705 erossite
21.46 ± 9.99 1251 703 Lexii
21.55 ± 10.10 1252 480 Estinox
-
This guy seems to be a hard core player, that just wins against the good players and loses against the Newbies :) Over 50000 games and with a 3sigma of 23.98 :o.
Maybe you can take a look Andrew, it seems something could be wrong with the Leaderboard
Level 11
22.35 ± 10.85 1248 241 doyouknow
21.72 ± 10.23 1249 338 Metatemujin
35.47 ± 23.98 1250 50705 erossite
21.46 ± 9.99 1251 703 Lexii
21.55 ± 10.10 1252 480 Estinox
This is the same bug I didn't solve last week. I'll continue planning to get to it eventually. ;)
-
Also, we have changed a policy (which had the potential for abuse and caused some problems with customer service locating the right person) and no longer allow two accounts to have a username that is identical except for the case of the letters. In cases where accounts already had such a duplicate name, we have renamed one of the accounts, giving preferential treatment to active accounts with recent logins. If your account happened to be one that was renamed and you don't like the new name, you are free to use the Profile link to pick another name.
http://forum.makingfun.com/showthread.php?4692-Release-Notes-5-9-2014
Isotropish is still using name to identify players right? If yes, some playes may have lost their ranking because of this. To be precise, they will be there more than once.
EDIT: probably nothing to do about this... there is only the name in the log.
-
Level 54 65.01 ± 10.17 1 1847 Mic Qsenoch
Level 52 62.50 ± 10.03 2 2447 Stef
72.54 ± 20.50 3 1826 Petra Körkkö
Level 51 72.26 ± 21.04 4 2790 jakey
My god! Caution Stef and Mic Qsenoch, a new generation of dominion players is coming! ;)
-
Also, we have changed a policy (which had the potential for abuse and caused some problems with customer service locating the right person) and no longer allow two accounts to have a username that is identical except for the case of the letters. In cases where accounts already had such a duplicate name, we have renamed one of the accounts, giving preferential treatment to active accounts with recent logins. If your account happened to be one that was renamed and you don't like the new name, you are free to use the Profile link to pick another name.
http://forum.makingfun.com/showthread.php?4692-Release-Notes-5-9-2014
Isotropish is still using name to identify players right? If yes, some playes may have lost their ranking because of this. To be precise, they will be there more than once.
EDIT: probably nothing to do about this... there is only the name in the log.
There is a solution to this problem, but it's a fair amount of work.
Goko gives only the player names in the log, but the host's player id appears in the log filename. So when you play against bots, in adventure mode, or against players whose ids are known, we can determine your id too. Any multi-player games you play before you play such a game can be back-filled. So one can at least theoretically deal with basically all the cases except where someone is being deliberately perverse (e.g. create two accounts, play them only against each other, change their names, etc.).
On the other hand, the duplicate name issue affects only 0.5% of players and an even smaller fraction of Pro games. Obviously there are more pressing matters for Isotropish, including 1. the bug that Monsiuer X keeps reminding us about and 2. the Goko vs Isotropish analysis I still haven't posted yet because it'll lead immediately to 3. the debate over optimized TrueSkill parameters for Isotropish.
When I'm finished not dealing with those, I'll try to find some time to not deal with this. ;)
-
oh! that's bug. I thougth these guys were cheaters! (didnt look at the last messages!)
-
Does the leaderboard bug affect Iso rating in the lobbys as well? I just saw the top two people in one lobby: 'Rightiswrong' had a Goko rating of ~7K and Iso level 17. The other (don't remember the name) had a Goko rating of ~1500 and Iso level 30-something. I didn't think they would be that far off.
-
Does the leaderboard bug affect Iso rating in the lobbys as well? I just saw the top two people in one lobby: 'Rightiswrong' had a Goko rating of ~7K and Iso level 17. The other (don't remember the name) had a Goko rating of ~1500 and Iso level 30-something. I didn't think they would be that far off.
No. And the bug is only affecting a very small number of players. Just those who are showing up with 50k games and a few of their opponents.
There are ways that the sort of discrepancy you observe can legitimately arise. That's because of the different TrueSkill parameters the two systems use. Most strikingly:
1. A new player who performs very well can top the Goko charts long before he reaches the top on Isotropish
2. A player who takes a long break effectively gets his Goko rating reset, because of Goko's daily increase in player uncertainty. If his next few games are particularly good or bad, his new Goko rating can end up way different from his Isotropish rating.
In this case, however, there's a much simpler explanation: you're looking at Goko "Casual" ratings. Isotropish only cares about Pro games.
You can click the player's avatar to see his Pro rating. If you use the Salvager extension, there's also an option to display Pro ratings instead of Casual ratings in the list of players in your lobby.
-
2. the Goko vs Isotropish analysis I still haven't posted yet because it'll lead immediately to 3. the debate over optimized TrueSkill parameters for Isotropish.
When I'm finished not dealing with those, I'll try to find some time to not deal with this. ;)
I spent some time thinking about this last week. There are objective measures for judging the quality of predictions--the Brier score (http://en.wikipedia.org/wiki/Brier_score) is one that seemed pretty logical to me after I gave it some thought. The Trueskill Python module will provide a prediction, so in theory it seems pretty straightforward to throw the big dataset into two Trueskill systems set up with different parameters. Depending on how long it takes to run, you could even run several different parameter sets to see what gives the best predictions.
If it would be of use, I'm a programmer and could probably work on that myself if you make the data available. I managed to connect to your DB with the guest login you have on another forum, but it looked like the tables needed weren't available to guests.
-
2. the Goko vs Isotropish analysis I still haven't posted yet because it'll lead immediately to 3. the debate over optimized TrueSkill parameters for Isotropish.
When I'm finished not dealing with those, I'll try to find some time to not deal with this. ;)
I spent some time thinking about this last week. There are objective measures for judging the quality of predictions--the Brier score (http://en.wikipedia.org/wiki/Brier_score) is one that seemed pretty logical to me after I gave it some thought. The Trueskill Python module will provide a prediction, so in theory it seems pretty straightforward to throw the big dataset into two Trueskill systems set up with different parameters. Depending on how long it takes to run, you could even run several different parameter sets to see what gives the best predictions.
If it would be of use, I'm a programmer and could probably work on that myself if you make the data available. I managed to connect to your DB with the guest login you have on another forum, but it looked like the tables needed weren't available to guests.
Great! Yes, what I've done so far has been based on a similar metric: Statistical Deviance (http://en.wikipedia.org/wiki/Deviance_(statistics)), which is the same basic measure but with a bunch of logs involved. I'm no expert, but I understand that Deviance is well-accepted in stats/econometric circles, and variations on it are used in many different contexts. It's also (almost) the metric that FIDE used for its chess rating system competition (http://www.kaggle.com/c/ChessRatings2) back in 2011.
I'd be happy to add you as a collaborator on the not-yet-public project I've been using for this stuff: https://github.com/aiannacc/Trueskill-Analysis. I've wanted to keep it private until I'm confident that its bug-free and easy to download and run. Since this issue is certain to spark a lot of debate, I want that debate to at least be well-informed.
As for running time, comparing two different TrueSkill systems on the ~1 million Pro games played on Goko to date takes a couple hours on an Intel i7 920 @ 2.67 GHz. That time goes, almost exclusively, into doing the TrueSkill rating updates: specifically into doing the error function integral approximations that TS requires. Of course, that's only the speed when running in Python (which is slow), but it's using the Python scipy module, and that module outsources most of the heavy lifting to Fortran. So there's probably not a lot of time to be saved by working in another language. However, there's a lot that can be gained by multi-threading, depending on the number of physical cores in your CPU.
Contact me if you'd like access to my github project and current analysis. I'll make you a project collaborator. Same for anyone else who's interested in this.
-
Is it just me, or is the leaderboard stuck? 'Last recorded game finished 194 min, 44 seconds ago'
-
Is it just me, or is the leaderboard stuck? 'Last recorded game finished 194 min, 44 seconds ago'
Same here, its now 201 minutes since last update- it's usually only a couple of minutes.
-
Is it just me, or is the leaderboard stuck? 'Last recorded game finished 194 min, 44 seconds ago'
Yes, it's stuck. It'll be back online shortly.
-
In this case, however, there's a much simpler explanation: you're looking at Goko "Casual" ratings. Isotropish only cares about Pro games.
<facepalm> oops, yeah I just forgot that I turned off that option in Salvager to make it show pro ratings...
-
I just changed my username (hugovj->Hugovj), but the leaderboard shows both. I don't know how that can be fixed? (Can it?) Weirdly, in the lobby I see my level as 14, while on the leaderboard it says -7 (for the Hugovj with capital letter).
-
I just changed my username (hugovj->Hugovj), but the leaderboard shows both. I don't know how that can be fixed? (Can it?) Weirdly, in the lobby I see my level as 14, while on the leaderboard it says -7 (for the Hugovj with capital letter).
This ugly problem arose from Goko's decision to not put unique player-identifying information in their logs. There are ways for us to deal with it, as briefly discussed here (http://forum.dominionstrategy.com/index.php?topic=8900.msg377287#msg377287), but honestly I don't think anyone's likely to bother.
I'll implement a manual workaround for your username, so that the games you played as "hugovj" will be considered part of your record as "Hugovj." This will take effect next time I do a full rebuild of the leaderboard.
FYI, your "level" isn't the same as Isotropish's best guess of your ability. That's what "mu" is. There are legitimate (though debatable) reasons for using level instead of mu for experienced players, but it heavily penalizes brand new players.
Your performance as "Hugovj" has actually been considerably stronger than your performance as "hugovj." If you keep up your recent quality of play as Hugovj, your level will quickly surpass hugovj's level 14.
-
Thanks a lot :D
-
I just changed my username (hugovj->Hugovj), but the leaderboard shows both. I don't know how that can be fixed? (Can it?) Weirdly, in the lobby I see my level as 14, while on the leaderboard it says -7 (for the Hugovj with capital letter).
This ugly problem arose from Goko's decision to not put unique player-identifying information in their logs. There are ways for us to deal with it, as briefly discussed here (http://forum.dominionstrategy.com/index.php?topic=8900.msg377287#msg377287), but honestly I don't think anyone's likely to bother.
I'm working on it! Eventually!
-
It seems that both log search and Isotropish have stopped working. Last recorded game according to Isotropish (at the moment) is more than 2 hours ago.
-
It seems that both log search and Isotropish have stopped working. Last recorded game according to Isotropish (at the moment) is more than 2 hours ago.
Dealing with this and some other issues by moving to Linode (http://forum.dominionstrategy.com/index.php?topic=11242.new#new) (finally!).
-
I've fixed the gokosalvager.com bug that was making games go missing. Both the log search and the leaderboard appear to be correct now.
Unfortunately, the MF/Goko that causes some logs not to be generated at all still remains, and they seem to be dropping quite a lot of games. If you can find the original log file, but it doesn't show up in search, please post here, as it means that something is wrong on my side. If the log file is missing too, it's just Goko being Goko.
FYI, Goko lists each day's logs at an address like http://dominionlogs.goko.com/20140617/. Games that you host will be prefixed by "log.[your player id]" (e.g. "log.5101a6c4e4b02b7235c3860f.1402894381446.txt").
-
A few anomalies have occurred. These are some ratings I'm currently seeing. Tuff and Marin have 50k+ games played. While I'm sure they both like Dominion, not sure they really played that many.
Ballsy -108
Tuff_Gong 56
Martin Dambachmair 99
Shihao Koh -179
-
A few anomalies have occurred. These are some ratings I'm currently seeing. Tuff and Marin have 50k+ games played. While I'm sure they both like Dominion, not sure they really played that many.
Ballsy -108
Tuff_Gong 56
Martin Dambachmair 99
Shihao Koh -179
This is a known bug. I've put some logging in place that may help me catch it next time it happens. FWIW, the overall effect on the leaderboard is pretty minimal, except for the lucky few who get massively overrated (temporarily).
-
The Isotropish leaderboard won't be updating until further notice. As of yesterday, my Linode instance can no longer access the MF/Goko log server.
From home, I'm able to talk to their server just fine:
iron> ping -q -c 10 google.com
PING google.com (74.125.224.200) 56(84) bytes of data.
--- google.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9014ms
rtt min/avg/max/mdev = 12.021/15.808/44.671/9.665 ms
But from Linode, my requests just time out:
li566-22> ping logs.prod.dominion.makingfun.com
PING ec2-54-213-198-64.us-west-2.compute.amazonaws.com (54.213.198.64) 56(84) bytes of data.
^C
--- ec2-54-213-198-64.us-west-2.compute.amazonaws.com ping statistics ---
435 packets transmitted, 0 received, 100% packet loss, time 433999ms
The same happens when I request specific log files using 'curl' or 'wget'.
Anyone have any insight on why this might happen? Is the most likely thing that they've specifically cut off service to my Linode instance?
Assuming that's the case, can anyone guess who I'd have to contact to resolve that? There's a decent chance that I'm just doing my log requests inefficiently and in a way that demands more of their server than is really necessary. It may be that I could fix that to everyone's satisfaction if I could make contact with someone tech-savvy on their end.
-
Anyone have any insight on why this might happen? Is the most likely thing that they've specifically cut off service to my Linode instance?
That'd be my guess. Which linode datacenter are you in? I can ping that server just fine from a linode in Fremont, so it's not a linode-wide issue (which would be pretty weird, but plausible).
Assuming that's the case, can anyone guess who I'd have to contact to resolve that? There's a decent chance that I'm just doing my log requests inefficiently and in a way that demands more of their server than is really necessary. It may be that I could fix that to everyone's satisfaction if I could make contact with someone tech-savvy on their end.
https://github.com/aiannacc/goko-dominion-tools/blob/f0ae8522af6151e4a459eb49dc820c00931c4729/start_logwatcher.py#L81-L97 does look pretty rude: hitting them with 40 new connections all at once. Isn't there a python HTTP client that supports Connection: Keep-Alive and/or pipelining? It looks like urlgrabber, urllib2, httplib2, and pycurl may all support this sort of thing; but I don't have the python experience to judge which of them are actually any good.
Of course I don't know who on their end you'd have to talk to, but if they really are blocking you I'm kinda amazed they decided to do so without even contacting you. If you end up getting your networking code a little more polite but can't convince makingfun to let you back in, you could try connecting over tor.
-
Anyone have any insight on why this might happen? Is the most likely thing that they've specifically cut off service to my Linode instance?
Try
traceroute logs.prod.dominion.makingfun.com
to determine if it is not a network issue.
-
Also can you connect to archive-dominionlogs.goko.com ?
That is a different server.
-
https://github.com/aiannacc/goko-dominion-tools/blob/f0ae8522af6151e4a459eb49dc820c00931c4729/start_logwatcher.py#L81-L97 does look pretty rude: hitting them with 40 new connections all at once. Isn't there a python HTTP client that supports Connection: Keep-Alive and/or pipelining? It looks like urlgrabber, urllib2, httplib2, and pycurl may all support this sort of thing; but I don't have the python experience to judge which of them are actually any good.
Of course I don't know who on their end you'd have to talk to, but if they really are blocking you I'm kinda amazed they decided to do so without even contacting you. If you end up getting your networking code a little more polite but can't convince makingfun to let you back in, you could try connecting over tor.
Whether that's the problem or not, I would definitely appreciate some help in making my connections more "polite." I'm sure that the Python Requests module supports all the basic protocols and header options, but honestly I haven't a clue what I'm doing with this stuff. Like I have it doing 40 requests at once because that was the only way I could think to deal with the long delays (often 30 seconds or more) when I request individual log files.
Could you point me towards a useful reference for understanding how to do polite and efficient http requests in high(-ish) volume? It doesn't have to be Python-specific.
I realize that Tor is an option, but I'd like to try to civil route first. I'm also surprised to be cut off without being contacted. But that's assuming that a human made the decision. Maybe their server defends itself from attacks by automatically cutting off abusive requesters?
Also can you connect to archive-dominionlogs.goko.com ?
That is a different server.
Yes, I can still access that server from Linode. But it runs about 2 days behind on the game logs. The only source I know that provides logs in real time is logs.prod.dominion.makingfun.com (the same server as dominionlogs.goko.com).
Try
traceroute logs.prod.dominion.makingfun.com
to determine if it is not a network issue.
Ok. I haven't used traceroute before though. Does this output mean anything to you?
li566-22> traceroute logs.prod.dominion.makingfun.com
traceroute to logs.prod.dominion.makingfun.com (54.213.198.64), 30 hops max, 60 byte packets
1 23.92.24.3 (23.92.24.3) 0.377 ms 0.522 ms 0.638 ms
2 10ge8-3.core3.fmt2.he.net (64.71.132.137) 0.153 ms 0.143 ms 0.146 ms
3 10ge10-1.core1.sjc2.he.net (184.105.222.14) 12.244 ms 12.202 ms 12.149 ms
4 216.218.193.42 (216.218.193.42) 0.678 ms 0.691 ms 0.656 ms
5 205.251.229.155 (205.251.229.155) 0.678 ms 205.251.229.157 (205.251.229.157) 0.682 ms 205.251.229.155 (205.251.229.155) 0.703 ms
6 205.251.232.68 (205.251.232.68) 30.317 ms 205.251.232.112 (205.251.232.112) 22.153 ms 205.251.232.68 (205.251.232.68) 29.517 ms
7 205.251.232.147 (205.251.232.147) 22.359 ms 205.251.232.153 (205.251.232.153) 22.272 ms 205.251.232.141 (205.251.232.141) 22.262 ms
8 205.251.232.165 (205.251.232.165) 22.997 ms 22.948 ms 205.251.232.63 (205.251.232.63) 22.938 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
-
Whether that's the problem or not, I would definitely appreciate some help in making my connections more "polite." I'm sure that the Python Requests module supports all the basic protocols and header options, but honestly I haven't a clue what I'm doing with this stuff. Like I have it doing 40 requests at once because that was the only way I could think to deal with the long delays (often 30 seconds or more) when I request individual log files.
Could you point me towards a useful reference for understanding how to do polite and efficient http requests in high(-ish) volume? It doesn't have to be Python-specific.
Keepalive and pipelining are designed to cut down on the number of TCP round-trips between client and server. Keepalive (http://en.wikipedia.org/wiki/HTTP_persistent_connection) is supported by all HTTP 1.1 servers (including goko's), and works by requesting multiple files without having to repeat the TCP handshake for each (because it reuses the same connection). Pipelining (http://en.wikipedia.org/wiki/HTTP_pipelining) is built on top of this, and works by pre-sending a bunch of HTTP requests over the same connection before receiving any response, letting the server queue them all up and process them rapid-fire rather than waiting for your client to make the next request.
These should help if the time it takes to download a log is based mainly on round-trip time, but if transferring a single file takes a long time this won't help much. Some poking around with curl suggests to me that with one TCP connection per file, it takes 0.2s to download a log; with keepalive over 100 log files I can get 0.1s. curl doesn't support pipelining, but if it did I think we'd make it down to 0.05s by halving the number of round trips again. That might be fast enough that being singlethreaded is fine, I would think, but you could do it on a small handful of threads if necessary.
Ok. I haven't used traceroute before though. Does this output mean anything to you?
li566-22> traceroute logs.prod.dominion.makingfun.com
traceroute to logs.prod.dominion.makingfun.com (54.213.198.64), 30 hops max, 60 byte packets
1 23.92.24.3 (23.92.24.3) 0.377 ms 0.522 ms 0.638 ms
2 10ge8-3.core3.fmt2.he.net (64.71.132.137) 0.153 ms 0.143 ms 0.146 ms
3 10ge10-1.core1.sjc2.he.net (184.105.222.14) 12.244 ms 12.202 ms 12.149 ms
4 216.218.193.42 (216.218.193.42) 0.678 ms 0.691 ms 0.656 ms
5 205.251.229.155 (205.251.229.155) 0.678 ms 205.251.229.157 (205.251.229.157) 0.682 ms 205.251.229.155 (205.251.229.155) 0.703 ms
6 205.251.232.68 (205.251.232.68) 30.317 ms 205.251.232.112 (205.251.232.112) 22.153 ms 205.251.232.68 (205.251.232.68) 29.517 ms
7 205.251.232.147 (205.251.232.147) 22.359 ms 205.251.232.153 (205.251.232.153) 22.272 ms 205.251.232.141 (205.251.232.141) 22.262 ms
8 205.251.232.165 (205.251.232.165) 22.997 ms 22.948 ms 205.251.232.63 (205.251.232.63) 22.938 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Your ping is getting as far as 205.251.232.165, which is an Amazon server in Seattle (according to http://www.tcpiputils.com/browse/ip-address/205.251.232.165), and then being dropped. Since apparently the server you're requesting is an ec2 server and you made it to amazon before being ignored, I'd say it's the goko server itself rejecting you.
-
...
Thank you. That's all very helpful.
-
(Cross-posted in the Log Search thread (http://forum.dominionstrategy.com/index.php?topic=8266.msg394737#msg394737))
On Sunday (June 29), MakingFun's log server stopped responding to requests from gokosalvager.com. Their log server itself still works fine, but it's specifically ignoring requests from my Linode instance.
It's possible that this is just their server automatically defending itself from what it thinks is an attack or just an overly-rapacious user. I haven't received any notice about this from MF, nor have I received any complaints in the past. I've asked about it on their forum (http://forum.makingfun.com/showthread.php?4896-gokosalvager-com-can-no-longer-access-MakingFun-s-Dominion-log-server) and will update you as I learn more.
I don't plan to anonymize or otherwise disguise my log server requests, not even as a temporary workaround. If MF ever decides to thwart the log search, Isotropish, or the Salvager extension, they'd have a pretty easy time doing it, so there's not much sense in trying to defy them. And I'd really like to have some sort of ongoing dialogue with MF anyway.
Until this is resolved, new games will not show up in the log search, and the Isotropish leaderboard will not update. The "Iso" ratings displayed by the Salvager extension will not update either, but it's other features will continue working (including automatch). The log prettifier and kingdom-image generator on still work, but you have to find the logs yourself.
-
Keepalive and pipelining are designed to cut down on the number of TCP round-trips between client and server. Keepalive (http://en.wikipedia.org/wiki/HTTP_persistent_connection) is supported by all HTTP 1.1 servers (including goko's), and works by requesting multiple files without having to repeat the TCP handshake for each (because it reuses the same connection). Pipelining (http://en.wikipedia.org/wiki/HTTP_pipelining) is built on top of this, and works by pre-sending a bunch of HTTP requests over the same connection before receiving any response, letting the server queue them all up and process them rapid-fire rather than waiting for your client to make the next request.
This is great! Keepalive more than doubles the rate at which I download individual log files. More importantly, none of my requests time out anymore, even from my home machine. That was what was making my log downloader spaz out and stop fetching new games so that you guys would have to bug me about it and I'd have to go restart the script.
If I understand correctly, pipelining is the better way to do what I've been doing by sending simultaneous requests in multiple threads. I haven't yet figured out how to do it with the Python Requests module though. Instead, I reduced the number of threads I'm using from 40 to 10. I could drop it further without much loss if even 10 simultaneous requests sounds like it might be considered "rude." With a single thread, it can download about a day's worth of logs in an hour, which is plenty fast enough as long as I'm not catching up from being months behind or anything.
I've also reduced the rate at which I check for new logs to once every every 60 seconds. I think that's actually what their server was unhappy with me about, since it has to send me several megs (or several-hundred kB compressed) of redundant data every time I requested the list of today's games. And given that it takes them ~7s to send that list, they either have the machine throttled down to 1990s internet speeds or it's doing something stupidly resource-intensive to generate it.
So now there will be up to a 1 min delay between when you finish your game and when gokosalvager.com adds it to the search database and updates your Isotropish rating (sorry). Given what a nuisance this has turned out to be, I really don't want to risk it happening again.
-
So now there will be up to a 1 min delay between when you finish your game and when gokosalvager.com adds it to the search database and updates your Isotropish rating (sorry). Given what a nuisance this has turned out to be, I really don't want to risk it happening again.
What?! Wait up to one whole minute for my game log and for my rating to update?! I don't know if I can handle this.
-
http://forum.dominionstrategy.com/index.php?topic=1204.msg340225;topicseen#msg340225
In the golden days you woke up to the update. ;)
-
The Isotropish leaderboard isn't currently working, hasn't been updated for 5 hours, and the Iso ratings aren't being displayed.
-
The Isotropish leaderboard isn't currently working, hasn't been updated for 5 hours, and the Iso ratings aren't being displayed.
Fixed... probably and eventually. It could take quite a while to catch up. Now that I have to anonymize my log file requests, downloading logs is a lot slower.
MakingFun still hasn't responded to my post on their tech support forum (http://forum.makingfun.com/showthread.php?4896-gokosalvager-com-can-no-longer-access-MakingFun-s-Dominion-log-server), but on the other hand, they did send me a birthday email! :D
From: Making Fun Forums <help@makingfun.com>
Date: Wed, Jul 9, 2014 at 5:12 PM
Subject: Happy Birthday from Making Fun Forums
To: andrewiemail@gmail.com
Hello ragingduckd,
We at Making Fun Forums would like to wish you a happy birthday today!
-
The Isotropish leaderboard isn't currently working, hasn't been updated for 5 hours, and the Iso ratings aren't being displayed.
Fixed... probably and eventually. It could take quite a while to catch up. Now that I have to anonymize my log file requests, downloading logs is a lot slower.
Not that time... I had a bug that didn't show up until it had downloaded quite a few logs. Might be fixed this time, but don't trust the "last game recorded" stat right now. I'm actually further behind than it says.
Edit: It really is all caught up now.
-
Leaderboard marks me as level 42, but I'm exactly level 43.
I can has >= check?
-
Leaderboard marks me as level 42, but I'm exactly level 43.
I can has >= check?
In the database you're mu=52.9957 and sigma=3.3339, so I'm sure it's some subtlety about where the rounding and truncation happens.
Nice job hitting the range where the website and the extension disagree! It can't be very wide. :)
-
Monsieur X is a good friend and I don't think he played 50.000 games in the last few days :
-
I wouldn't be so sure about that, based on the limited amount I've played him he seems quite the beast.
-
Monsieur X is a good friend and I don't think he played 50.000 games in the last few days :
Thanks. This is the same old leaderboard bug that's been happening on and off for months. It's been really hard to replicate and track down, so I've just been fixing the damage by regenerating the leaderboard from scratch when it happens.
I've done that again (so there are no 50k+ players now), and I think I've solved the underlying problem too this time. It was happening when the log parser encountered two pro games with identical timestamps (down to the microsecond). I had some faulty code that resulted in parsing the two games over and over until the parser hit it's 100k max-games-per-query. So the winning player's rating and variance would go through the roof, as would his number of games played.
-
The log search is updating (i.e. new games appear when searched for), but the ratings aren't updating currently (and for the last 16 hours or so).
-
The log search is updating (i.e. new games appear when searched for), but the ratings aren't updating currently (and for the last 16 hours or so).
Thanks. I broke something else while fixing the 50k game issue. Ratings are updating again.
-
Conclusive proof that Goko Pro has it all wrong compared to Isotropish :)
(http://i.imgur.com/C7V5R32.png)
-
Alright I'm a bit confused. I just installed Salvager a few days ago and I'm enjoying it a lot but I don't understand the Iso ratings. I saw someone who had a pretty high pro rating and a good win ratio but his Iso level was in the -20 range. Meanwhile, I've leveled up twice today - once by losing a game and once from my opponent quitting. Isn't that the kind of stuff that's not supposed to happen? I think I'm going to need a more detailed explanation of the variables so I know what's going on.
-
Alright I'm a bit confused. I just installed Salvager a few days ago and I'm enjoying it a lot but I don't understand the Iso ratings. I saw someone who had a pretty high pro rating and a good win ratio but his Iso level was in the -20 range. Meanwhile, I've leveled up twice today - once by losing a game and once from my opponent quitting. Isn't that the kind of stuff that's not supposed to happen? I think I'm going to need a more detailed explanation of the variables so I know what's going on.
First, pro mode and Iso aren't the same, that's why we have Iso ratings. I'm at level 25 in Iso ratings and just checked my pro rating recently and it was 2700 (I haven't played in a while) (I started playing again, pushing it up to 4500). I don't know why you got rating for losing a game, and when your opponent quit you should definitely get rating for that.
-
Alright I'm a bit confused. I just installed Salvager a few days ago and I'm enjoying it a lot but I don't understand the Iso ratings. I saw someone who had a pretty high pro rating and a good win ratio but his Iso level was in the -20 range. Meanwhile, I've leveled up twice today - once by losing a game and once from my opponent quitting. Isn't that the kind of stuff that's not supposed to happen? I think I'm going to need a more detailed explanation of the variables so I know what's going on.
I can't give a massively detailed explanation, however:
(1). Goko Pro includes games against bots and games with 3+players, Iso only counts 2 players, no bots . Therefore, if someone has a good Pro rating and bad Iso rating it probably means they play most of their games against bots, or 3+players.
(2). If your opponent quits/resigns, you still gain Iso rating, unless the game is really short (i.e. it doesn't load for someone).
(3). Iso rating is calculated by taking a skill part and an uncertainty part, and subtracts the uncertainty from the skill, and rounds down to get your level i.e. according to the Iso leaderboard, I'm currently 31.26±9.94, so level 21, as 31.26-9.94 = 21.32, which become 21. So, I guess (not 100% sure about this) that your uncertainty must have increased decreased more than the skill bit decreased, so get higher despite losing a game. Especially if you don't play much, your uncertainty can be high, and then gets lower quite quickly if you play lots of games.
Hope this was helpful :)
-
Note that Iso doesn't update instantly, so the changes you observed might have been caused by games different from those that you guessed.
-
your uncertainty must have increased decreased more than the skill bit
I hate to nitpick but this has a potential to make things super confusing, so FTFY.
Note that Iso doesn't update instantly, so the changes you observed might have been caused by games different from those that you guessed.
Doesn't it update almost instantly anyway?
-
Note that Iso doesn't update instantly, so the changes you observed might have been caused by games different from those that you guessed.
Doesn't it update almost instantly anyway?
If it's working properly, then it updates every 1-2 minutes I believe.
-
Yes, it's pretty quick, but as it's not living on the MF servers it has to download all of the logs, which it doesn't do constantly so as not to overload their server. (Let's face it, they need all the help they can get.)
-
(1). Goko Pro includes games against bots and games with 3+players, Iso only counts 2 players, no bots . Therefore, if someone has a good Pro rating and bad Iso rating it probably means they play most of their games against bots, or 3+players.
Isotropish includes bot games for sure.
-
(1). Goko Pro includes games against bots and games with 3+players, Iso only counts 2 players, no bots . Therefore, if someone has a good Pro rating and bad Iso rating it probably means they play most of their games against bots, or 3+players.
Isotropish includes bot games for sure.
I should really stop talking, seems everything here is wrong.
-
Both isotropish and goko pro are implementations of Microsoft TrueSkill (http://research.microsoft.com/en-us/projects/trueskill/) (though goko has not admitted it, AI reverse-engineered the system). They use different parameters however.
-
Isotropish has me down for 78 games. Those are all my league games and it is not counting any of my bot games (which is good).
-
Isotropish has me down for 78 games. Those are all my league games and it is not counting any of my bot games (which is good).
Do you play Pro or Casual against the bots? I am pretty sure it counts bots, as there was a thread about Canton Pilots Association (who just plays bots, and is highly rated), also Tao Chen (hdu88 on the forums I think?) has been almost exclusively playing bots recently while reaching his current high level.
-
I select the 'play bots' option rather than going into the kingdom creation screen. It adjusts the Goko score for both casual and pro (I think). To be honest I'd rather nobody investigated this thoroughly enough to make any changes!
-
I select the 'play bots' option rather than going into the kingdom creation screen. It adjusts the Goko score for both casual and pro (I think). To be honest I'd rather nobody investigated this thoroughly enough to make any changes!
I think that the "play bots" option just goes to your casual score (which is weird because it doesn't give you the option of picking the kingdom). That's what it was last time I checked. There's an option to play a pro game with a bot in the multiplayer server and that's probably what Iso counts.
Weird thing though - when I played casual games with bots on the multiplayer server it didn't do anything to my rating. I wonder if it's the same thing for pro because that would kind of defeat the purpose of pro vs. bots.
-
Sometimes when I'm bored, I play bots through the main screen. I can assure you that it counts for pro rating.
-
Sometimes when I'm bored, I play bots through the main screen. I can assure you that it counts for pro rating.
Okay then, I stand corrected.
-
Sometimes when I'm bored, I play bots through the main screen. I can assure you that it counts for pro rating.
Okay then, I stand corrected.
It used to count for casual. They recently changed it.
-
I have a request: please no longer count bot games for Isotropish.
I'm starting to get the feeling the bot games seriously mess with the ratings now.
My theory is this: It's just too easy to beat them once you get the basic concepts of the game. Their ratings won't be really low because with their reasonably good executions of mediocre strategies they beat up a lot of new players.
Maybe some players that actually play bots can weigh in on this one.
-
I have a request: please no longer count bot games for Isotropish.
I'm starting to get the feeling the bot games seriously mess with the ratings now.
My theory is this: It's just too easy to beat them once you get the basic concepts of the game. Their ratings won't be really low because with their reasonably good executions of mediocre strategies they beat up a lot of new players.
Maybe some players that actually play bots can weigh in on this one.
My theory is that the rating system is actually improperly calibrated, because it forces a normal distribution. If Tommy is good enough against Suzy that he wins 70% of the time, and Suzy is good enough that she beats Pat 70% of the time, then it expects Suzy would beat Pat 85.3% of the time. (Well, it tries to correct for this somewhat via the sigma parameters, but this is limited). The thing is, in Dominion, after a point, it's very very hard to achieve such a high threshhold. But it will try, via the Sigma, and over a wide enough scale, it screws things up.
tl;dr there are ranges of rating differences I expect people to outperform expectations for, and bots are probably the most egregious problem here.
-
There are at least some choices for the Trueskill parameters that (in my experience) actually make you lose points if you play a bunch of bot games because my Goko Pro rating tanks by about ~400-600 when I play bots a bunch. Contrast with isotropish where I think playing a bunch of bots gives me a level inflation of somewhere around ~1.5-3, but I'm just guessing. At the top end you can't really exploit it for too many levels but I wouldn't be surprised if a level 40 player could get 10 or so levels out of it. I don't know how much you'd get with "good" play, I know my play against the bots it quite a bit sloppier than against humans.
-
Multiplayer games against the bots are a different matter as well. It's much more difficult to impose a better strategy.
-
Multiplayer games against the bots are a different matter as well. It's much more difficult to impose a better strategy.
Multiplayer games aren't included in isotropish.
-
I've never really played bot games so for, er, non-rating related reasons I thought I'd give it a go.
Defender Bot proceeds to open 5/2 for Witch.
-
Isotropish no longer counts bot games. This includes both the "Play Bots" mode and "pro" games played against bots in the multiplayer lobby. They still affect your Goko Pro rating, of course, but Isotropish now ignores them. This change means your Isotropish rating will jump (or just has).
See this thread (http://forum.dominionstrategy.com/index.php?topic=11810.0) for copious discussion of this issue.
-
I played a game where the player just stopped playing, probably on purpose, as s/he was losing. So the game timed out and I won. This game doesn't show up in the search in the gokosalvager log search.
Then I played a game where Goko just kicked me out of the game, so I lost. This game doesn't shop up either.
I assume noen of them count for the rating.
Are both of these on purpose?
-
I played a game where the player just stopped playing, probably on purpose, as s/he was losing. So the game timed out and I won. This game doesn't show up in the search in the gokosalvager log search.
Then I played a game where Goko just kicked me out of the game, so I lost. This game doesn't shop up either.
I assume noen of them count for the rating.
Are both of these on purpose?
Gokosalvager's default search setting excludes games where a player quit. You need to adjust the search settings to find these games. Your first example seems to be this game: http://www.gokosalvager.com/static/logprettifier.html?20150729/log.50fd569ae4b0f82020d83fac.1438200691106.txt.
Edit: So games where a player quits do count for ratings. However, I believe that if a player is prevented from joining the game by a server error (and therefore quits), the game is not counted.
-
Gokosalvager's default search setting excludes games where a player quit. You need to adjust the search settings to find these games. Your first example seems to be this game: http://www.gokosalvager.com/static/logprettifier.html?20150729/log.50fd569ae4b0f82020d83fac.1438200691106.txt.
Edit: So games where a player quits do count for ratings. However, I believe that if a player is prevented from joining the game by a server error (and therefore quits), the game is not counted.
Ah, I see. Thanks. Hadn't noticed that. So both games count then, even the one where the server kicked me out, because it looks like I quit.
-
Are Isotropish levels still active? All players have Iso ratings of null in the playdominion.com game rooms.
Sadly, Isotropish levels are still down which sucks because some opponents seem to have artificially inflated Goko ratings from playing bots and whatnot.
-
The first question would be whether game logs are still being produced.
-
What I really want to know is if the games we've been playing while Iso is down still count for our rankings.
-
It's fixed today
Sorry folks, my bad. Obviously I'm not paying much attention to this.
Please feel free to email or PM me when the system goes down. I hadn't realized.
-
http://gokosalvager.com/ is down atm :/
-
http://gokosalvager.com/ is down atm :/
Sorry folks, I accidentally let the domain name expire. I tried to renew it this evening, but the name registration service I use keeps encountering an "unknown error." I'll try contacting their support people tomorrow.
For what it's worth, the server itself is running fine and you can still access it directly at http://192.155.80.22
-
The gokosalvager.com site is working pretty well for me. But the extension deleted my avatar (and won't reupload it) and isn't showing Iso rankings. Any word?
-
Hello, I just joined dominion league season 12. My making fun username is XD9. I noticed the isotropish leaderboard has me listed as XD9-2-. This had something to do with a problem with GOKO and my username a while ago. Also, Isotropish doesn't track my games since then. Was wondering if you could eradicate this XS9-2- thing from the board and get it tracking my XD9 again?
-
Hello, I just joined dominion league season 12. My making fun username is XD9. I noticed the isotropish leaderboard has me listed as XD9-2-. This had something to do with a problem with GOKO and my username a while ago. Also, Isotropish doesn't track my games since then. Was wondering if you could eradicate this XS9-2- thing from the board and get it tracking my XD9 again?
I have removed the manual name change. You are listed as XD9.
But I think your rating was correct.
-
Hello, I just joined dominion league season 12. My making fun username is XD9. I noticed the isotropish leaderboard has me listed as XD9-2-. This had something to do with a problem with GOKO and my username a while ago. Also, Isotropish doesn't track my games since then. Was wondering if you could eradicate this XS9-2- thing from the board and get it tracking my XD9 again?
I have removed the manual name change. You are listed as XD9.
But I think your rating was correct.
Thank you!
-
Doesn't look as though the Iso leaderboard is updating.
-
Seems to be working again. Whoever fixed that, thanks.
-
How often does it update these days?
-
How often does it update these days?
Every day except when it's down
-
Looks like it hasn't updated in a few days.
-
Looks like it hasn't updated in a few days.
It was updated 0.5434838 days ago.
-
It appears that the leaderboard is down.