Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[AdamH]

There's been some discussion on continuing the Dominion Cards List in Qvist's absence. Most of the data for the 2013 Ed. 1 list was released, with the exception of the Potion-cost cards and the Favorite cards. It would be nice to have a list that includes the Guilds cards (which means the list would be "complete!")

Out of respect for Qvist, given that he put a lot of work into it, I think we should try and keep it as close to what he did as possible. I'm willing to contribute to the cause, and I'm willing to take a leadership/coordinating role if nobody else wants to do it, but I know it can be a lot better if we involve more people to help out.

The purpose of this thread is to:

1. Determine if the community would like to continue making new version(s) of this list in Qvist's absence.

If it's decided we want to do this, then also...

2. Determine the pieces of work that need to be done
3. "Crowd source" volunteers who may be able to help out with the pieces of the job and exchange any relevant contact information
4. Organize timelines and specifics for each task


If you're interested in helping put it together, even if you only have (constructive) comments or (helpful) information, please post here.

If you're interested in contributing your personal rankings to the pool should this go forward, please post here.

If you're interested in reading about the results of this project, please post here.

[AdamH]

In this post, I'll do my best do keep up-to-date what tasks need done, the status of those tasks, and who has volunteered to work on them.

VOLUNTEERS FOR HELP ON ANY OF THESE TASKS IS GREATLY APPRECIATED

Things that might be useful to have done:

-- Data Collection --

1. Attempt to find the rest of the data from the 2013 Ed. 1 Cards list

I can try reaching out to Qvist through my channels again, but if this doesn't work I don't know what else we can do. We might have to move forward without this data.


2. Figure out exactly how each of the statistics was calculated for previous cards lists.

My guess to what stats were used is below, but without raw data to verify, maybe someone more mathematically inclined could do a sanity check for me?

To convert from rank to percentage, do  1 - (rank / sample size)
To calculate a change in a value, take (new - old) / old  <-- this can be done with either percentages or ranks, just be consistent (do percentages)

Average (%): arithmetic mean
Weighted Average (%): Weights are calculated using isotropish leaderboard level as the only input; the exact function used is not clear yet.
Change in Weighted Average (pp)

Median (%)
Change in Median (pp)

Standard Deviation (%): any funny stuff going on here? Or just stdef = sqrt((sum(x-avg)^2)/N)
Change in Standard Deviation (pp)

Max and min values (%)


3. Did we ever come to a consensus about this discussion on overpay cards? We should probably figure out what we're doing there that will make the most people happy, right?

If anyone disagrees with putting each overpay card in the category corresponding to its base cost only, speak now or forever hold your peace.


4. Make shiny web-based portal for data entry

GendoIkari?

- Not required, but if this is done, it would result in a standard format for all results, which would be nice to have even if we don't do this.

- Whatever format for results we use should be OK with me, but it would be really nice if we used something that could easily be imported into Excel, though, such as csv.

- We need a way to tie the results acquired to a username on the Isotropish Leaderboard. In the absence of this (or even maybe not) we'd like some way to protect against the same person casting multiple votes. Should we require a F.DS username as well? This is up to the designer.

- Using any O(n*log(n)) sorting method and user input for comparisons seems to be the consensus for the algorithm we should use. Depending on how well this works we may just be able to force people to use this interface to input their lists, even if they choose to sort them some other way before entering them in. I think this would be ideal, but maybe another feature could be a way to just input your list and have the web portal format it for you?


-- once results are collected --

5. Collect relevant data from previous editions of the Cards Lists for comparison purposes, and draw appropriate conclusions


6. Write up a paragraph about each card

I don't think I'm the best person for this job


7. Update the wiki page with the new results

I don't think I'm the best person for this job


8. Make videos for each of the lists

I should be able to do this one, though contributions from others are welcome (you can even head this part up if you want! Less work for me! )


9+. Are there other tasks that need done?

[AdamH]

I wonder if some moderator magic would be useful here to copy relevant posts from the previous discussion (link in OP) to here?

[Schneau]

Did he reveal the favorite cards list? I was looking forward to those

In any case, if there isn't an outcry of people telling me this is a bad idea by later today I'll start a new thread with an outline for what needs done, volunteers to help, etc.

...or if someone else wants to head it up that's fine with me too.

I'm all for this. We got 55 submissions last time, and it would be great to get more. I think one thing that would really increase the number of lists submitted is to have some sort of website that helps you rank the lists. Qvist did this last time using SurveyMonkey, but the interface was clunky and didn't make it much easier. A site that compares 2 cards at a time and uses those comparisons for sorting is much more user-friendly, and would probably result in more submissions. I wrote a Python script to do this for the programmers in the crowd, but for everyone else a website would be great. Maybe we can cajole ragingduckd (or someone else with web skillz) into doing this.

I can probably throw together such a page pretty quickly. But explain in more detail how you see it working? It just keeps showing 2 cards and you pick which of the 2 is better?

Basically, you just use a sorting algorithm (hopefully a O(nlogn) one) and use the user input as the comparator function. So, the user first picks which list they want to sort, and then you show them the two cards (or card names) that the sorting algorithm wants them to compare. Do this until the sorting algorithm finishes, and output the list. I'd recommend outputting the list as text, so that the user can rearrange a few cards afterward if they realize part way through that they messed up somewhere. Feel free to look at or use my Python code linked above - I'm basically recommending that, except with a web interface.

[GendoIkari]

Did he reveal the favorite cards list? I was looking forward to those

In any case, if there isn't an outcry of people telling me this is a bad idea by later today I'll start a new thread with an outline for what needs done, volunteers to help, etc.

...or if someone else wants to head it up that's fine with me too.

I'm all for this. We got 55 submissions last time, and it would be great to get more. I think one thing that would really increase the number of lists submitted is to have some sort of website that helps you rank the lists. Qvist did this last time using SurveyMonkey, but the interface was clunky and didn't make it much easier. A site that compares 2 cards at a time and uses those comparisons for sorting is much more user-friendly, and would probably result in more submissions. I wrote a Python script to do this for the programmers in the crowd, but for everyone else a website would be great. Maybe we can cajole ragingduckd (or someone else with web skillz) into doing this.

I can probably throw together such a page pretty quickly. But explain in more detail how you see it working? It just keeps showing 2 cards and you pick which of the 2 is better?

Basically, you just use a sorting algorithm (hopefully a O(nlogn) one) and use the user input as the comparator function. So, the user first picks which list they want to sort, and then you show them the two cards (or card names) that the sorting algorithm wants them to compare. Do this until the sorting algorithm finishes, and output the list. I'd recommend outputting the list as text, so that the user can rearrange a few cards afterward if they realize part way through that they messed up somewhere. Feel free to look at or use my Python code linked above - I'm basically recommending that, except with a web interface.

Got it. Yeah, if people agree that it's helpful, I'd love to help!

One possible issue with that method though... nothing really stopping people from having Mountebank > Witch > Rebuild > Mountebank. (Rock-paper-scissors issue). I guess in theory a good sort algorithm won't ask the user to compare all 3 of those... but which 2 of the 3 happen to be chosen could influence the final list. Though of course they can re-sort anything they don't like. Is that about right?

[Schneau]

Did he reveal the favorite cards list? I was looking forward to those

In any case, if there isn't an outcry of people telling me this is a bad idea by later today I'll start a new thread with an outline for what needs done, volunteers to help, etc.

...or if someone else wants to head it up that's fine with me too.

I'm all for this. We got 55 submissions last time, and it would be great to get more. I think one thing that would really increase the number of lists submitted is to have some sort of website that helps you rank the lists. Qvist did this last time using SurveyMonkey, but the interface was clunky and didn't make it much easier. A site that compares 2 cards at a time and uses those comparisons for sorting is much more user-friendly, and would probably result in more submissions. I wrote a Python script to do this for the programmers in the crowd, but for everyone else a website would be great. Maybe we can cajole ragingduckd (or someone else with web skillz) into doing this.

I can probably throw together such a page pretty quickly. But explain in more detail how you see it working? It just keeps showing 2 cards and you pick which of the 2 is better?

Basically, you just use a sorting algorithm (hopefully a O(nlogn) one) and use the user input as the comparator function. So, the user first picks which list they want to sort, and then you show them the two cards (or card names) that the sorting algorithm wants them to compare. Do this until the sorting algorithm finishes, and output the list. I'd recommend outputting the list as text, so that the user can rearrange a few cards afterward if they realize part way through that they messed up somewhere. Feel free to look at or use my Python code linked above - I'm basically recommending that, except with a web interface.

Got it. Yeah, if people agree that it's helpful, I'd love to help!

One possible issue with that method though... nothing really stopping people from having Mountebank > Witch > Rebuild > Mountebank. (Rock-paper-scissors issue). I guess in theory a good sort algorithm won't ask the user to compare all 3 of those... but which 2 of the 3 happen to be chosen could influence the final list. Though of course they can re-sort anything they don't like. Is that about right?

Yeah, it should be fine. Most sorting algorithms assume transitivity, so they won't ask for situations like that. I'd recommend just using some out-of-the-box sorting algorithm -- I just used Python's sort() myself.

[jsh357]

I think any format should be fine as long as it is consistent.  A rating system would be susceptible to the problem that everyone interprets scores differently, so I kind of like the sorting (X is better than Y) idea myself.

As for going ahead and doing the lists, I say do it. Qvist was basically done, and while it's sad not to have the Potions it's not the end of the world.  People know the Dark Ages cards better now anyway (Rebuild is crazy low on the last list).

[werothegreat]

Re: Overpay - I think it would be best, and simplest, to just group each overpay card at their nominal cost.  There are only 4 of them, and we can't really have them in every list.  Sure, Masterpiece at $3 is probably the worst $3 purchase you can make, but I think it's overkill to have it on every list.  Judge the overpay cards by their full function, not just what they're worth at their nominal cost, but still group them by their nominal cost.

[GendoIkari]

Did he reveal the favorite cards list? I was looking forward to those

In any case, if there isn't an outcry of people telling me this is a bad idea by later today I'll start a new thread with an outline for what needs done, volunteers to help, etc.

...or if someone else wants to head it up that's fine with me too.

I'm all for this. We got 55 submissions last time, and it would be great to get more. I think one thing that would really increase the number of lists submitted is to have some sort of website that helps you rank the lists. Qvist did this last time using SurveyMonkey, but the interface was clunky and didn't make it much easier. A site that compares 2 cards at a time and uses those comparisons for sorting is much more user-friendly, and would probably result in more submissions. I wrote a Python script to do this for the programmers in the crowd, but for everyone else a website would be great. Maybe we can cajole ragingduckd (or someone else with web skillz) into doing this.

I can probably throw together such a page pretty quickly. But explain in more detail how you see it working? It just keeps showing 2 cards and you pick which of the 2 is better?

Basically, you just use a sorting algorithm (hopefully a O(nlogn) one) and use the user input as the comparator function. So, the user first picks which list they want to sort, and then you show them the two cards (or card names) that the sorting algorithm wants them to compare. Do this until the sorting algorithm finishes, and output the list. I'd recommend outputting the list as text, so that the user can rearrange a few cards afterward if they realize part way through that they messed up somewhere. Feel free to look at or use my Python code linked above - I'm basically recommending that, except with a web interface.

Got it. Yeah, if people agree that it's helpful, I'd love to help!

One possible issue with that method though... nothing really stopping people from having Mountebank > Witch > Rebuild > Mountebank. (Rock-paper-scissors issue). I guess in theory a good sort algorithm won't ask the user to compare all 3 of those... but which 2 of the 3 happen to be chosen could influence the final list. Though of course they can re-sort anything they don't like. Is that about right?

Yeah, it should be fine. Most sorting algorithms assume transitivity, so they won't ask for situations like that. I'd recommend just using some out-of-the-box sorting algorithm -- I just used Python's sort() myself.

Being a .Net guy, I'm assuming that the Sort() method it provides is going to be the most efficient possible. But, I don't think I can use that in a web application. Sure I can pass in a custom Comparer function, but I don't think there's a way to have that function ask the user for input... this is all happening on the server side, and I can't tell the server to go to the client for that sort of input.. HTTP and ASP just don't work that way. So I'd have to do the sort algorithm manually; keeping track of which cards have or have not been compared on each trip back to the server. Shouldn't be a problem though.

[eHalcyon]

Re: Overpay - I think it would be best, and simplest, to just group each overpay card at their nominal cost.  There are only 4 of them, and we can't really have them in every list.  Sure, Masterpiece at $3 is probably the worst $3 purchase you can make, but I think it's overkill to have it on every list.  Judge the overpay cards by their full function, not just what they're worth at their nominal cost, but still group them by their nominal cost.

Should we really have this giant conversation again?  If you're going to do that, might as well just do a full comparison between all cards (like WW did a while back) instead of grouping by cost at all.  This is a serious suggestion.

[AdamH]

All right, I've made our default position to put each overpay card with its base price's list only and barring further discussion, we'll just go with that. The second post in the thread reflects that.

[GendoIkari]

All right, I've made our default position to put each overpay card with its base price's list only and barring further discussion, we'll just go with that. The second post in the thread reflects that.

If it helps, I was the one who originally started that whole debate with my suggestion that there be a separate list for the overpay cards... and having read various opinions, and spent the last 4 months thinking about it non-stop (ok, not really), I now feel like just putting them in the list with their base costs is probably just fine.

[Watno]

Weighted Average (%): arithmetic mean?
Change in Weighted Average (pp)

I think these were the ratings that were weighted according to isotropic levels. Not sure if the weights were the levels themselves or scaled differently.

[Beyond Awesome]

I was actually thinking about this today. lol I love Qvists lists, and I think a new one is needed to better reflect DA rankings also Guilds cards now that enough time has elapsed for us to have reasonable opinions of those cards.

Also, I really want overpay to just be at the base costs.

Edit: Just noticed in your sig Adam the posts to the Power Grid forum. Didn't know about that. I love that game.

[AdamH]

Weighted Average (%): arithmetic mean?
Change in Weighted Average (pp)

I think these were the ratings that were weighted according to isotropic levels. Not sure if the weights were the levels themselves or scaled differently.

Ah, yes. how much were the levels weighted? I would think that without raw data present we wouldn't be able to know, right? Unless someone out there can work some number magic for me...

I remember now Qvist saying that certain cards were higher or lower on the unweighted list and drawing conclusions from that. Is there a standard way to calculate something like this?

...and I imagine we'll just take a snapshot of the Isotropish leaderboard for this, right? I've sort of assumed that's the most legitimate ranking we have, or will there be problems there with people who haven't played in a while not showing up there?

[flies]

Is there a standard way to calculate something like this?

weighted average of card X = sum( rank of X by player Y * level of Y) / sum(all levels)

[WanderingWinder]

Is there a standard way to calculate something like this?

weighted average of card X = sum( rank of X by player Y * level of Y) / sum(all levels)

The problem is, this is painfully wrong - it equates level 2 to be twice as good as level 1, 40 to be twice as good as 20, etc, and don't even get me started on 0 or negative levels, when (say it with me)

The Ratings are on an Interval Scale.

So this is not a correct thing to do, mathematically, for this case. If you had count data, that's how you'd do it....

[PSGarak]

Aggregating individual rankings into a community ranking is actually a well-studied problem, called Ranked Voting in the Voting Theory branch of mathematics. The good news is, there's methods already written and thought about, that have their behaviors and corner-cases documented. The bad news is, there's rather a lot of them.

http://en.wikipedia.org/wiki/Ranked_voting_system

Usually the purpose of these systems is to choose a single winner, not an entire list of rankings, so most algorithms only pick a single winner. That's a problem that can be easily, if brutely, solved: Take the data, find that, say, Wharf is the best card. Go back to the data, remove Wharf from everyone's rankings, re-run to find the second-best card. Repeat until all the cards are done.

I would recommend using a voting system that meets the Condercet criteria. This criteria says, basically, that if every individual voter ranks Mountebank higher than Witch, then the voting system is guaranteed to output that Mountebank ranks higher than Witch. Surprisingly, most voting systems have corner cases where this does not always occur. People tend not to trust voting systems where that doesn't happen.

http://en.wikipedia.org/wiki/Condorcet_method

The above article includes a list of counting methods. Kemeny-Young sounds promising, since it naturally results in a full ranking, not just an individual winner, which is what we want anyways.

As far as player level? I dunno, I would just say that players at level greater-than-N will count as casting two identical ballots, or three, etc. Voting Theory does not usually account for some people being more important than others.

[Schneau]

Aggregating individual rankings into a community ranking is actually a well-studied problem, called Ranked Voting in the Voting Theory branch of mathematics. The good news is, there's methods already written and thought about, that have their behaviors and corner-cases documented. The bad news is, there's rather a lot of them.

http://en.wikipedia.org/wiki/Ranked_voting_system

Usually the purpose of these systems is to choose a single winner, not an entire list of rankings, so most algorithms only pick a single winner. That's a problem that can be easily, if brutely, solved: Take the data, find that, say, Wharf is the best card. Go back to the data, remove Wharf from everyone's rankings, re-run to find the second-best card. Repeat until all the cards are done.

I would recommend using a voting system that meets the Condercet criteria. This criteria says, basically, that if every individual voter ranks Mountebank higher than Witch, then the voting system is guaranteed to output that Mountebank ranks higher than Witch. Surprisingly, most voting systems have corner cases where this does not always occur. People tend not to trust voting systems where that doesn't happen.

http://en.wikipedia.org/wiki/Condorcet_method

The above article includes a list of counting methods. Kemeny-Young sounds promising, since it naturally results in a full ranking, not just an individual winner, which is what we want anyways.

As far as player level? I dunno, I would just say that players at level greater-than-N will count as casting two identical ballots, or three, etc. Voting Theory does not usually account for some people being more important than others.

I agree that Kemeny-Young looks like a good system. But, it is a NP-hard problem, and probably requires exponential time in the number of candidates. The Wikipedia article says that a good algorithm can reasonably solve up to about 40 candidates. So, if we can find a good algorithm, we might be able to do $4 (53 cards) and $5 (61 cards), but I doubt we could use it for the Favorite Cards List (205 cards).

Of course, if we change the ranking method now, I don't know if it will make sense to compare the lists to the old ones. So, I think I'd actually suggest we stick with whatever Qvist was doing, which is some sort of (weighted) average rank of all the votes.

[AdamH]

So after reading some of these wikipedia links my brain hurts. It seems that they can only provide a ranking and not the percentage values that were present in Qvist's lists. I've scoured old posts on the topic and I couldn't find anything, but for some reason I thought I remembered that more priority was given to people whose iso level was 30 or higher and that was it.

...I would really like to figure out exactly how Qvist computed this, though, because we can't make comparisons to previous results without that. In all of his posts mentioning it, he said that weighing the average didn't end up affecting things that much.

[hsiale]

...I would really like to figure out exactly how Qvist computed this (...) the percentage values that were present in Qvist's lists

The standard, unweighted score is easy. Every user inputs a list. Now you give the first card on the list a result of 100%, the last one 0%. And the remaining ones are distributed on the 0-100 interval one after another, with constant distance from card to next one. Let's say we rank a group of 21 cards (this leads to nice numbers so is good as an example). Card ranked 1st gets score of 100%. Card ranked 2nd gets score of 95%, card ranked 3rd gets score of 90%. And so on, for example the middle card (ranked 11th) gets score of 50%, card ranked 14th gets score of 35%. Card ranked dead last gets 0%.

Now it is possible, that a player ranks only some of the expansions. So, let's say, someone ranked only 11 cards out of those 21. Now you need to give 1st card a result of 100%, but 2nd gets 90%, 3rd 80% and so on.

After you do this with every list submitted to you, for each card you have some results (number of results may differ). Simply take average of those. Rank cards from top to bottom. You get the unweighted list.

Now for weighted average, you just make some votes count more. I don't know how Qvist weighted players. Definitely we should use the Isotropish leaderboard to make weights, but how exactly should they be calculated, I have no idea.

[Jack Rudd]

My preferred voting method is, funnily enough, not a ranked method. It's called "Range Voting" or "Score Voting": everybody gives each card a number on a scale (say 0-10), and the card with the highest total score (or highest average score, depending on which you wish to use - if everyone's ranking everything, it doesn't matter which you use) wins.

[AdamH]

I'm not saying there aren't other ways to do this, and I'm not saying they aren't better. Maybe they are.

On the other hand, it might take some convincing (at least for me) to make me believe that changing the way we rate/calculate stats significantly (significantly = enough that it means we can't make valid comparisons to past cards lists) benefits us enough to offset what we lose by making those comparisons.

I mean, Qvist's thing worked, we got tons of good results from it, and that's the reason I want to continue do the thing with the same name. If we want to change it, then sure we can talk about it (and now is the time to do so) but I think that the "default" position should be to get as close to the way that Qvist did it, similar to what we're doing for the overpay cards. If a clear benefit can be shown by changing it, I'm certainly willing to listen but I have a feeling I'll be focusing on other parts of this project enough that I don't want to lead that discussion.

---------------------

...as far as this weighted average goes, I think the only input to the function is the Isotropish leaderboard level of each entrant, and the tricky part is just figuring out how to assign weight so that it's closest to whatever Qvist did (or "best," maybe depending on other things).

The good news, though, is that we don't have to have the answer to that before we start collecting data. All we need to know for right now is that we need to collect the Isotropish level of each person surveyed at some point.

I think this means we have enough information to start development of a web portal for this? Schneau/GendoIkari, is it OK if I make one of you the "leader" of this task? I'll do my best to update the brainstorming/status post at the top with all the specs we need.

[Schneau]

I know nothing about web development. So, I'm ok if you make GendoIkari the leader of this task 

I'm willing to help with algorithmic concerns.

[flies]

The problem is, this is painfully wrong - it equates level 2 to be twice as good as level 1, 40 to be twice as good as 20, etc, and don't even get me started on 0 or negative levels, when (say it with me)

The Ratings are on an Interval Scale.

So this is not a correct thing to do, mathematically, for this case. If you had count data, that's how you'd do it....

Okay, it's not mathematically perfect.  I took it that AdamH wasn't familiar with weighted averages and provided a simple example.  But honestly, what's the fuss?

Whatever the scaling, there ought to be some straightforward f(trueskill=s) such that

I suggested f(s) = s, but this has obvious problems, the worst of which are from negative skill.  I see no problem simply tossing votes from players with s <= 0, so, again, this f(s)=s isn't ideal but in the absence of a definitive solution it should be just fine.

Trueskill is based on Glicko, which is based on Elo.  In Elo, if I have x skill more than you, I have y:1 odds in favor of beating you (Elo sets x=100 and y=16/9).  In this case, f(s) = y^(s/x).  In traditional Elo, a difference of 100 skill points corresponds to 16:9 odds, so f(s) = 1.78^(s/100). 

[edit: i had y for Elo set to 2]