Dominion Strategy Forum

Please login or register.

Login with username, password and session length
Pages: 1 [2]  All

Author Topic: Dominion Data Mining: Cards that correlate with skill  (Read 15454 times)

0 Members and 1 Guest are viewing this topic.

ben_king

  • Golem
  • ****
  • Offline Offline
  • Posts: 190
  • Shuffle iT Username: ben.king
  • formerly grsbmd
  • Respect: +612
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #25 on: January 10, 2015, 11:44:27 pm »
+1

I have a suggestion for another similar analysis.  (this one is definitely cool)

If only one player in a game buys a card, and the other player does not, what % of the time is the player who bought the card the one with a high skill rating? (You could weight it by the skill gap, but I don't think that would be that helpful)

Definitely.  That's one of the things I'm planning to do next with this data.

Can someone estimate the likelyhood of a card getting +/-0.100 correlation just by chance? Or do we need further assumptions? We have more than 200 cards and I'm wondering whether some outliers could be explained by this.

A correlation of +/- 0.100 would be about 10 standard deviations from the mean, so it's pretty safe to say that none of the results in the first post are due to chance.

I must've missed the previous thread... has someone put up that whole DB to be downloaded?  I have an experiment or two I'd like to try myself. 

It's actually quite easy to get this data.  The data is from gokosalvager.com.  All I did was download all the game logs for each of the top 100 players.
Logged

Davio

  • 2012 Dutch Champion
  • *
  • Offline Offline
  • Posts: 4787
  • Respect: +3412
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #26 on: January 11, 2015, 04:48:10 am »
0

Another idea for analysis would be to look at the impact certain cards have on a kingdom.

Which cards cause other cards to be bought more or less?
Which cards are always useful (low standard deviation in how often they're bought) and which cards require a more specific kingdom?

We sort of already know the answers to these questions, but maybe the math can reveal something.
Logged

BSG: Cagprezimal Adama
Mage Knight: Arythea

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4381
    • View Profile
    • WanderingWinder YouTube Page
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #27 on: January 11, 2015, 07:31:10 am »
0

Can someone estimate the likelyhood of a card getting +/-0.100 correlation just by chance? Or do we need further assumptions? We have more than 200 cards and I'm wondering whether some outliers could be explained by this.

A correlation of +/- 0.100 would be about 10 standard deviations from the mean, so it's pretty safe to say that none of the results in the first post are due to chance.

How do you come up with that?

Ratsia

  • Moneylender
  • ****
  • Offline Offline
  • Posts: 168
  • Respect: +113
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #28 on: January 11, 2015, 08:12:37 am »
0

A correlation of +/- 0.100 would be about 10 standard deviations from the mean, so it's pretty safe to say that none of the results in the first post are due to chance.
How do you come up with that?
Can't say how he did that, but generally permutation tests are very good for questions like that. Simply randomly permute the player IDs so that no information about the true good/bad player dichotomy remains and re-compute the correlation. Repeat 1000 times or so. This gives readily both the mean and the variance for the null-distribution and (if one is into such things) enables trivial statistical significance testing by counting how many of the 1000 replicates are above the observed value.

It rarely pays off to do any other kinds of tests, and the only thing one has to think about is what exactly to permute (for example, here it probably makes a difference whether the permutation is for each player-rating pair of for each individual match).
Logged

ben_king

  • Golem
  • ****
  • Offline Offline
  • Posts: 190
  • Shuffle iT Username: ben.king
  • formerly grsbmd
  • Respect: +612
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #29 on: January 11, 2015, 02:08:27 pm »
0

How do you come up with that?

I used bootstrap sampling.  http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

The highest standard deviations are for Promo cards (which have the least data), so you should be slightly more wary of Promo cards in the original list, but even Prince only has a standard deviation of 0.03.
Logged

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4381
    • View Profile
    • WanderingWinder YouTube Page
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #30 on: January 11, 2015, 03:34:48 pm »
+1

How do you come up with that?

I used bootstrap sampling.  http://en.wikipedia.org/wiki/Bootstrapping_(statistics)

The highest standard deviations are for Promo cards (which have the least data), so you should be slightly more wary of Promo cards in the original list, but even Prince only has a standard deviation of 0.03.

This still doesn't tell me what you did. Yes, I understand bootstrapping - statistics is my job - but I don't know, for instance, what .03 is the standard deviation of. So you took a bootstrap - of what? Games? Players? How big was the bootstrap? How big is the entire thing? You realize that these things aren't uncorrelated, right?

The reason I'm so curious is that 10 standard deviations means actually nothing. Seriously. You make the claim that it means these are really big effects you wouldn't see by random chance. The problem is, if you're implying there's a normal distribution, the chance of getting a result like you claim is so small, I can't get a computer to give me that calculation (at least, within a few minutes on my home machine; point is, it's REALLY small, small enough you need to use a non-standard data type). Even if it's a wacky distribution, Chebyshev's theorem tells us we shouldn't really be getting these kinds of results. So, if I were doing this calculation and getting this result, my natural assumption would be that I had done something wrong in my calculation.

ben_king

  • Golem
  • ****
  • Offline Offline
  • Posts: 190
  • Shuffle iT Username: ben.king
  • formerly grsbmd
  • Respect: +612
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #31 on: January 11, 2015, 03:57:32 pm »
+2

This still doesn't tell me what you did. Yes, I understand bootstrapping - statistics is my job - but I don't know, for instance, what .03 is the standard deviation of. So you took a bootstrap - of what? Games? Players? How big was the bootstrap? How big is the entire thing? You realize that these things aren't uncorrelated, right?

The reason I'm so curious is that 10 standard deviations means actually nothing. Seriously. You make the claim that it means these are really big effects you wouldn't see by random chance. The problem is, if you're implying there's a normal distribution, the chance of getting a result like you claim is so small, I can't get a computer to give me that calculation (at least, within a few minutes on my home machine; point is, it's REALLY small, small enough you need to use a non-standard data type). Even if it's a wacky distribution, Chebyshev's theorem tells us we shouldn't really be getting these kinds of results. So, if I were doing this calculation and getting this result, my natural assumption would be that I had done something wrong in my calculation.

I feel like I get more scrutiny here than with peer review.  The reason I post these is so that people can learn from them if they find them useful.  I have my own thesis I need to finish, so I don't have time to write another one about Dominion.

Normality is a reasonable assumption in the absence of evidence to the contrary, which is why I make that assumption.  I haven't had time yet to see if the distribution of correlation coefficients seems to fit that well.  Your post might be that evidence to the contrary.  The standard deviation of the coefficient averaged over all cards when I run 100 bootstraps on games is 0.01.  The range is 0.007 to 0.032.  I've also done a second experiment where I take each game and ignore which players buy which cards and instead assign these randomly.  This also produces a standard deviation on the correlation coefficient of ~0.01.  In 20+ runs times 206 cards per run, I was not able to get a chance correlation of greater than 0.04.

So it seems very unlikely to me that a card could achieve a correlation of +/- 0.100 simply by chance.
Logged

rrenaud

  • Administrator
  • *****
  • Offline Offline
  • Posts: 991
  • Uncivilized Barbarian of Statistics
  • Respect: +1197
    • View Profile
    • CouncilRoom
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #32 on: January 13, 2015, 03:33:50 pm »
+7

Maybe you should get better peer reviewers? :) 

Great job with this! 

FWIW, I wouldn't take criticisms as offenses, but more like challenges.  Can I make the explanation better/clearer?  Is there another analysis that can demonstrate the same point without the noted problems?

But of course, only do it so much as you enjoy it.
Logged

Polk5440

  • Torturer
  • *****
  • Offline Offline
  • Posts: 1708
  • Respect: +1788
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #33 on: January 13, 2015, 05:10:37 pm »
+1

I feel like I get more scrutiny here than with peer review. 

You would be correct.
Logged

Davio

  • 2012 Dutch Champion
  • *
  • Offline Offline
  • Posts: 4787
  • Respect: +3412
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #34 on: January 14, 2015, 03:48:16 am »
0

I love watching experts go head to head on a topic I know very little about. ;D
Logged

BSG: Cagprezimal Adama
Mage Knight: Arythea

SwitchedFromStarcraft

  • Saboteur
  • *****
  • Offline Offline
  • Posts: 1088
  • Respect: +856
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #35 on: January 14, 2015, 08:10:12 am »
+1

Want some popcorn?
Logged
Quote from: Donald X.
Posting begets posting.

Quote from: Asper
Donald X made me a design snob.

There is a sucker born every minute.

WanderingWinder

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 5275
  • ...doesn't really matter to me
  • Respect: +4381
    • View Profile
    • WanderingWinder YouTube Page
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #36 on: January 30, 2015, 08:33:05 am »
+2

So I wanted to say some more things here, because this has been bugging me a bit.

First of all, it wasn't my intention to try to insult or attack grbsmd or his work here. It seems some people have gotten that impression, and this was not my intention. Most specifically to grbsmd himself, if you feel this way, I really am sorry for that.


I also feel like while my gut told me that something was wrong of the analysis that 'these are significant findings', I didn't present a terribly good explanation of why that isn't true. I'm going to try to do that now.

That the details of the bootstrap procedure isn't really important (it would be in a peer-reviewed scholarly paper, but whatever, this isn't, we can just take your word here that you did it 'a lot'). You don't need to give standard deviations on this (I'm presuming what you quoted are actually standard errors, but again, whatever, no big deal), though, because when you're bootstrapping, you can just give the exact percentage of bootstraps you crossed your threshold. Also, Standard Deviations of a metric like r don't really make sense, as it is a summary statistic; it would be like asking what the standard deviation of mean player skill is, or the standard deviation of the maximum. But anyway, that's not the point - even if I don't believe what 10 SE implies, these numbers probably are quite assuredly different from zero in a 'statistical significant' sense. But basically what that means is, we're really sure they're different from zero. It doesn't tell us how far from zero they are.



The biggest point I have to make is my original one: these numbers show that players' abilities to correctly rate the general strength of cards is not a very big chunk of how strong they are. I'm going to quote the rebut of my original statement here:


So, the biggest thing to note here is that all of the numbers are tiny. You're not finding anything significant. Well, maybe statistically significant, but not practically so.

I'd argue that finding correlations this large is actually fairly substantial.  If we assume that how often you buy a card is independent of other cards (which is a fairly reasonable assumption as far as independence assumptions go, since in full random the chance of getting any two specific cards in a kingdom is ~0.002%), then the r^2 values range from 0.7% to 4%.  This means that statistically, I can explain 4% of the variation in skill among players simply by looking at how often the player buys Governor.  If you sum up the top 20 cards on the weighted list, that explains 29% of the variance in the skill.

That's huge.  This doesn't even include things like how cards are played once they're bought, when to start greening, etc.  So the fact that we can explain so much of the variance in skill simply by a how often a few cards are bought is a really big deal.

First of all, as has been pointed out, some of this is down to cause vs effect. I usually win when I get more provinces. Is that because I value province more? No - in fact, I'm pretty sure I value province less than most players. But when I get more, I am just more likely to win. It's like the old John Madden quote "the team that scores more points - well, they usually win the game".

Moreover, I don't think you're looking at these numbers correctly. I assume that you are taking gain rate when available in the kingdom, rather than cards bought per game (in which case, you'd only get information about set ownership really drowning out most everything else). Which means you can't really combine all these different values together. Moreover, you can't add the 'variance explained' at all. If you wanted to do that, you would want to multiply, 96% of the variance remains unexplained from the first card, 96% from the second leads us to a bit more than 92% remaining unexplained after two. The difference is pretty small between two cards, but once you're compounding 200 times, it will add up.

Most importantly, though, you really can't combine these together at all. You ran a whole bunch of correlations between the single card's gain rate and player skill. This gives you a bunch of different things. However, what you WANT to do is run one multiple correlation. You really should only get one combined r. And the independence assumption breaks down, HARD. If I don't buy an A, that means I probably bought a B. The things are absolutely related to each other, though again, some of what you'll see on the right is that better players buy more stuff overall, but that is because they are better, not why they are better. On the left, you're going to end up seeing that overall, it's going to be something like 5% (or less) of the variance in skill is explained by knowing if a card is good or bad. On the right, it will be somewhat higher, but again, I think this difference is mostly down to the cause/effect imbalance.

I mean, quick back-of-the-napkin shows that, because you only ever have 10 kingdom cards (yes, there are edge cases), if you multiply across a kingdom, even if it's close to the highest-scoring one, you're going to get 1-.99^10, ~= 9.6% of the variation in the skill between the players comes from knowing which cards are more rawly powerful than the others. Once you correctly take independence into consideration, and/or take an average set of 10 cards, I expect that will come down a lot from this even.



The thing is, yes, some cards are better than others, but by far, there is a lot more skill in knowing what is good or bad on a particular board. And then more skill yet in knowing how to sequence things, adjust to the gamestate and opponents' plans, etc. Knowing raw card skill is just a really small thing, and one that's pretty easy to pick up on.

werothegreat

  • Adventurer
  • ******
  • Offline Offline
  • Posts: 8172
  • Shuffle iT Username: werothegreat
  • Let me tell you a secret...
  • Respect: +9625
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #37 on: January 30, 2015, 09:30:18 am »
0

I find it interesting how, even unweighted, Masterpiece is bought more often by experienced players, especially given the other list where Masterpiece was the least-gained $3 card.
Logged
Contrary to popular belief, I do not run the wiki all on my own.  There are plenty of other people who are actively editing.  Go bother them!

Check out this fantasy epic adventure novel I wrote, the Broken Globe!  http://www.amazon.com/Broken-Globe-Tyr-Chronicles-Book-ebook/dp/B00LR1SZAS/

DStu

  • Margrave
  • *****
  • Offline Offline
  • Posts: 2627
  • Respect: +1490
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #38 on: January 30, 2015, 02:03:00 pm »
0

For me, 0.1 is not tiny, it's a buisness case...
Logged

TheExpressicist

  • Conspirator
  • ****
  • Offline Offline
  • Posts: 200
  • Respect: +203
    • View Profile
Re: Dominion Data Mining: Cards that correlate with skill
« Reply #39 on: January 30, 2015, 02:50:42 pm »
+1

Well done. This is interesting and useful data.

I am not sure why we are even entertaining the notion that these numbers might explain what makes up a player's skill. The data is already very useful, it doesn't need to be a gauge of people's skill. But yeah, it doesn't matter how strong, weak, valid or invalid the correlation is: there's nothing in the numbers that suggests Gain%  impacts Skill, rather than the other way around. Common sense would dictate that a player's skill is what impacts a card's gain rate, not the other way around.

As I mentioned, the data is useful enough as it is. It shows us which cards are purchased more often by good players and more often by bad players. If you had asked me beforehand what impacts a card's gain %, I would say it's a combination of player skill, other kingdom cards, and isolated card strength. So it's nice to have at least one of those variables knocked out.




« Last Edit: January 30, 2015, 02:59:22 pm by TheExpressicist »
Logged
Pages: 1 [2]  All
 

Page created in 0.109 seconds with 21 queries.