Maybe you should restrict it to games where the top 50 person does the particular opening? It's possible with these smaller numbers that a particular opening just happened to have the weaker player as P1 (or P2) in an above average number of games. Maybe it makes no difference.
Assuming it's random whether the better player is first or second player, then these sample sizes are big enough to be meaningful, I think. Like, if you flipped 100 coins, it would be very rare to get more than 66 heads. In particular, the first row really stands out to me:
Top openings that are much better for P2 than P1
Opening | Type | Number of occurences | P1 win % | P2 win % |
Rebuild-Silver | 5/3 | 90 | 36% | 67% |
Spice Merchant-Lookout | 4/3 | 51 | 33% | 60% |
Caravan-Workshop | 4/3 | 59 | 39% | 64% |
Quarry-Trade Route | 4/3 | 64 | 35% | 57% |
Quarry-Lookout | 4/3 | 77 | 35% | 57% |
Putting it in a binomial distribution calculator (http://stattrek.com/online-calculator/binomial.aspx), you can see that if you assume it's random who wins (50% chance for each player), the probability of getting 60 or more wins out of 90 is 0.103%, about 1/1000. So obviously you want to say that these results are meaningful (99.9% chance it's not coincidence), but man, I have no idea why Rebuild/Silver would favor P2.
This is how stats works, right? Someone who knows this stuff better than me should check, but this feels right to me.
I believe you're missing something. Certainly, if we were to select an opening and then run 100 trials with it and we got 66 heads, that would be some indication that we have an unfair coin. This is different, however. We have flipped 100 different coins each 100 times and then selected the ones that were far from 50/50. Here we would expect some of the sets of 100 coins to have 66 heads even if we thought all the coins were fair. For more info on this check out the Texas sharpshooter fallacy.
Yeah, I misunderstood what the percentages meant. I thought all of the games represented were ones in which both players went for the same opening. If we had 90 games in which both players went for a particular opening, and in 60 of those games, P2 had won, that would be strong evidence that the opening is advantageous for P2, right?
Anyway, I now realize that that's not what these statistics are representing, and I think JW's post correctly addresses the actual situation.
Wait, you're talking about a different problem in my reasoning. Yeah, that's true, but still, if the probability is low enough it can be statistically significant, right? If you flipped 1000 different coins 1000 times each, and one of those sets got 600+ heads, you would still conclude that that coin is unfair, because that's just way more likely than that you got it by coincidence, even though it was just 1 set out of 1000.
Anyway all of this is a moot point since it was based on my incorrect understanding of the stats in the OP.
(BTW, the probability of getting 66 or more heads out of 100 flips is 0.0895%, so even if we flipped 100 coins 100 times, there would only be an 8.56% chance to see at least one set with 66 or more heads.)