Dominion Strategy Forum

Dominion => Dominion Online at Shuffle iT => Dominion General Discussion => Goko Dominion Online => Topic started by: ragingduckd on May 29, 2013, 09:42:35 pm

Title: Log Search Engine
Post by: ragingduckd on May 29, 2013, 09:42:35 pm
Search engine for Goko game logs: http://gokologs.drunkensailor.org (http://gokologs.drunkensailor.org)

Please post feature requests and bug reports in this thread.

------------------
Status (AFAIK): Online
- Downloading new logs every 10 minutes

Known limitations:
- Some 2012 games can't be unambiguously parsed
- Can't distinguish unrated/casual/pro before May 15, 2013

Outstanding feature requests:
- display number of copies of each card gained
- save default search options
- more elegant UI generally
- bring back the win-loss results summary
Title: Re: Log Search Engine
Post by: StrongRhino on May 29, 2013, 09:57:04 pm
This is great!
I just learned I do badly with potions (4-9 record not counting the solitaire game I played)
I'm also good with looters (5-0)
And average with colonies (3-2)
Title: Re: Log Search Engine
Post by: heatthespurs on May 30, 2013, 04:23:45 am
This is EXCELLENT! Thank you very much!

We keep doing things that is supposed to be done by Goko staff (in-game log, log finder, etc.)... anyone could write a auto-match system?  :-\
Title: Re: Log Search Engine
Post by: loppo on May 30, 2013, 05:02:11 am
that is superb. And besides the useful information i found out that i have an Doppelgänger on goko. There is a "loppo" and a "Loppo".

Fortunately the wrong one has only losses, so its quite easy to distinguish.
Title: Re: Log Search Engine
Post by: Davio on May 30, 2013, 05:22:18 am
that is superb. And besides the useful information i found out that i have an Doppelgänger on goko. There is a "loppo" and a "Loppo".

Fortunately the wrong one has only losses, so its quite easy to distinguish.
Oh my, case-sensitive usernames... :'(
Or is that a capital i?  ;)
Title: Re: Log Search Engine
Post by: ragingduckd on May 30, 2013, 05:32:52 am
that is superb. And besides the useful information i found out that i have an Doppelgänger on goko. There is a "loppo" and a "Loppo".

Fortunately the wrong one has only losses, so its quite easy to distinguish.
Oh my, case-sensitive usernames... :'(
Or is that a capital i?  ;)

And here I thought I was doing people a favor by making it case insensitive.  I'll add a toggle.

You can't imagine how many places I had to add code to deal with goko bugs.  They're all commented with "F U Goko."
Title: Re: Log Search Engine
Post by: yed on May 30, 2013, 05:35:49 am
You can't imagine how many places I had to add code to deal with goko bugs.  They're all commented with "F U Goko."
How about a few examples? Humor us :)
Title: Re: Log Search Engine
Post by: Davio on May 30, 2013, 06:00:49 am
Yes, as a fellow programmer, I love seeing someone else's misery code. :)
Title: Re: Log Search Engine
Post by: ragingduckd on May 30, 2013, 07:04:36 am
You can't imagine how many places I had to add code to deal with goko bugs.  They're all commented with "F U Goko."
How about a few examples? Humor us :)

Here's me parsing a log where Outpost makes Goko counts the turns wrong.  In tied games, the wrong player would win because of it.

Code: [Select]
Now handling log.5021d3b637e80b087f124420.1346557284089.txt
Success!
Now handling log.503e653a4e423bf9f6fe53c2.1346546032904.txt
Success!
Now handling log.503fc49fde403bf99ee37a2b.1346534477235.txt
TurnCountException in log.503fc49fde403bf99ee37a2b.1346534477235.txt.  Fuck you, Goko.
Title: Re: Log Search Engine
Post by: ragingduckd on May 30, 2013, 07:29:02 am
This was a fun one too.  Here's are two excerpts from the same game log:

Code: [Select]
------------ Game Over ------------
guest2836 - cards: 2 Remodel, 1 Rabble, 1 Shanty Town, 7 Copper, 2 Silver, 1 Hoard, 1 Gold, 1 Bank, 2 Estate
guest2836 - total victory points: 2
guest2836 - turns: 8
 
Rotkäppchen - cards: 2 Farming Village, 1 Upgrade, 4 Copper, 1 Silver
Rotkäppchen - total victory points: 0
Rotkäppchen - turns: 7
 
1st place: guest2836
2nd place: Rotkäppchen

Code: [Select]
------------ Game Over ------------
Rotkäppchen - cards: 2 Farming Village, 1 Upgrade, 4 Copper, 1 Silver
Rotkäppchen - total victory points: 0
Rotkäppchen - turns: 7
 
guest2836 - quit
guest2836 - cards: 2 Remodel, 1 Rabble, 1 Shanty Town, 7 Copper, 2 Silver, 1 Hoard, 1 Gold, 1 Bank, 2 Estate
guest2836 - total victory points: 2
guest2836 - turns: 8
 
1st place: Rotkäppchen
2nd place: guest2836
Title: Re: Log Search Engine
Post by: Watno on May 30, 2013, 07:50:17 am
Those are old logs, aren't they?
Can't wait to have them all.
Title: Re: Log Search Engine
Post by: ragingduckd on May 30, 2013, 02:20:28 pm
Those are old logs, aren't they?
Can't wait to have them all.

The second one is from March.  I had to deal with that before I could parse those logs.  My parser is none too zippy, but it'll probably get through the March files sometime today.
Title: Re: Log Search Engine
Post by: ragingduckd on June 01, 2013, 02:01:48 pm
Update:
- All logs back to March 1 are now included.  February is in progress.
- Can now search by number of players
- Can now search by Colonies vs. Provinces
- Can now exclude games played with bots/guests

Support for shelters/estates and rating system is coming.

I'm not happy with the speed, but I think I can improve it with some database re-jiggering.  FYI, the second search you run on your username will generally be a lot faster than the first.
Title: Re: Log Search Engine
Post by: AdamH on June 06, 2013, 08:40:10 am
So this thing is amazing.

Does it take a day or so for logs to show up on here? For example if I played a game yesterday and want to get the log using this, I wasn't able to find any of them yet; does that just take a couple of days?
Title: Re: Log Search Engine
Post by: loppo on June 06, 2013, 08:42:16 am
this thread should get stickied, just like nutkis online extension.
Title: Re: Log Search Engine
Post by: ragingduckd on June 06, 2013, 12:32:47 pm
Does it take a day or so for logs to show up on here? For example if I played a game yesterday and want to get the log using this, I wasn't able to find any of them yet; does that just take a couple of days?

It's even worse than that.  It takes until I get around to manually running a script to download the logs and parse them.  I had a script that parsed them in real time, but it was badly written and it sucked up all my disk I/O and that interfered with my downloading illegal media.

Anyway, sorry about that.  Logs for the last few days are going in now, and I promise I'll fix the automated updating as soon as I feel like it.
Title: Re: Log Search Engine
Post by: AdamH on June 06, 2013, 12:34:52 pm
Hey no problem. You don't owe us anything for doing this. I was just curious if I was doing something wrong.
Title: Re: Log Search Engine
Post by: ragingduckd on June 09, 2013, 02:46:06 am
Hey no problem. You don't owe us anything for doing this. I was just curious if I was doing something wrong.

No worries.  I was just feeling a little frustrated with my code.  I very much enjoy working on this stuff (it's a welcome break from my real work) and I appreciate getting feature requests, even when the request is basically "make it work right." :)

I'm actually parsing the logs automatically every hour now, but for reasons I don't understand, the MySQL database doesn't seem to refresh the data that it sends to my Python web server without manual intervention.  I'm really not sure why, though I know it's not a query cache issue and I think it may have something to do with the InnoDB buffer.  If anyone out there has some clue, I'd be grateful for any assistance.
Title: Re: Log Search Engine
Post by: ragingduckd on June 09, 2013, 12:20:48 pm
Sweet!  Now updating every 15 minutes. :D
Title: Re: Log Search Engine
Post by: ragingduckd on June 09, 2013, 07:31:35 pm
I've been thinking -- perhaps a little too late -- about the privacy issues with the log search.  Yes, I know the logs are already public information b/c Goko is posting them, but there's a big difference between having a window in your bedroom and having someone set up a webcam outside it.

I'm considering adding a feature that would allow players to block searches for their name specifically.  If player X were blocked but player Y wasn't, it would work like this:

Search for player X: get nothing
Search for player X vs Y: get games between X and Y (like usual)
Search for player Y: get all of Y's games, including games with X (like usual)

Please let me know how you feel about this and/or vote in the poll above.
Title: Re: Log Search Engine
Post by: StrongRhino on June 09, 2013, 08:58:15 pm
It says its a discrete poll- but I can see the results  ???
Title: Re: Log Search Engine
Post by: sudgy on June 09, 2013, 08:59:40 pm
It says its a discrete poll- but I can see the results  ???

You can't see who voted what things.
Title: Re: Log Search Engine
Post by: heron on June 09, 2013, 09:05:26 pm
It says its a discrete poll- but I can see the results  ???

Why would a poll's discreteness affect its discreetness?  :P

Although I seem to remember it saying that it was a discrete poll, so you may not have made an error there. But how could a poll not be discrete  ???
Title: Re: Log Search Engine
Post by: StrongRhino on June 09, 2013, 09:11:20 pm
It says its a discrete poll- but I can see the results  ???

You can't see who voted what things.
Oh. Duh.
I thought it meant only the creator could see the results, which I translated as what was voted for.
Title: Re: Log Search Engine
Post by: DStu on June 10, 2013, 02:07:28 am
I've been thinking -- perhaps a little too late -- about the privacy issues with the log search.  Yes, I know the logs are already public information b/c Goko is posting them, but there's a big difference between having a window in your bedroom and having someone set up a webcam outside it.

I'm considering adding a feature that would allow players to block searches for their name specifically.  If player X were blocked but player Y wasn't, it would work like this:

Search for player X: get nothing
Search for player X vs Y: get games between X and Y (like usual)
Search for player Y: get all of Y's games, including games with X (like usual)

Please let me know how you feel about this and/or vote in the poll above.
I'm a bit undecided.

On the one hand it's a simple feature which does not effect anyone who does not want to use it, and it cares for the ones who care.

On the other hand, it's giving a false sense of security and takes away knowledge from the public, while preserving it for anyone wit a little bit of skill.  I mean it does not protect against any assumed real threats, anyone who really wants to find useful information finds them in the goko logs. It just 'protect' against the public, and I've seen that too many times in the last years: Everyone who could really harm you (financially or however) will get the data anyhow, but the whole public rants on how their neighbour can now see their houses on the internet.  That's what left us Germans with this (https://dl.dropboxusercontent.com/u/23087358/Dominion/sv.png) crippled version of Streetview, while at the same time nobody cares that you can get every relevant data on your houses from the state for a minimal fee.

On the third hand, I don't play on goko, so ...
Title: Re: Log Search Engine
Post by: Kirian on June 10, 2013, 02:22:49 am
That's what left us Germans with this (https://dl.dropboxusercontent.com/u/23087358/Dominion/sv.png) crippled version of Streetview

Huh.  Seeing that map (and looking at the rest of the world using the same view) certainly makes Geoguessr easier.
Title: Re: Log Search Engine
Post by: ragingduckd on June 10, 2013, 02:28:39 am
On the other hand, it's giving a false sense of security and takes away knowledge from the public, while preserving it for anyone wit a little bit of skill.  I mean it does not protect against any assumed real threats, anyone who really wants to find useful information finds them in the goko logs.

Finding games in the Goko log archive is possible but not easy.  If you know a player's hash code, you can find the games they hosted on any given day, but there's no easy way to find the games they joined or to search over multiple days.  You'd have to download all the logs and grep them, and the sheer volume of games makes rather slow and at least a little technically challenging.

All that said, you're right... I'm certainly not offering any real security.
Title: Re: Log Search Engine
Post by: Warfreak2 on June 10, 2013, 07:39:55 am
I would call it a courtesy rather than a security measure.
Title: Re: Log Search Engine
Post by: soulnet on June 10, 2013, 09:14:41 am
I would not block it, but I think its reasonable to do it, just like Google give sites the possibility to not show up in searches, even though they are publicly available on the internet. In any case, it does seem like a feature that will go mostly unused, so I would call that an unnecesary effort from someone who's work can clearly be put to better use, as the very existence of this application shows.
Title: Re: Log Search Engine
Post by: StrongRhino on June 10, 2013, 07:53:07 pm
Would it be possible to add the bots in adventures in the "yes, no, allow" for the bots when searching? Thanks.
Title: Re: Log Search Engine
Post by: soulnet on June 10, 2013, 08:21:01 pm
Would it be possible for the parameters to show up in the url (i.e., use get instead of post as the form method) so that we can get links to searches, like we used to do in CouncilRoom.com? In particular, I want my bookmark pointing to this to default to search logs including myself, which is what I mostly do. It will also be a simple way for people to save "default" values for all options.
Title: Re: Log Search Engine
Post by: ragingduckd on June 10, 2013, 08:39:33 pm
Would it be possible to add the bots in adventures in the "yes, no, allow" for the bots when searching? Thanks.

Implemented, but with some errors.  My database isn't entirely certain who's an adventure bot and who isn't. :D

Would it be possible for the parameters to show up in the url (i.e., use get instead of post as the form method) so that we can get links to searches, like we used to do in CouncilRoom.com? In particular, I want my bookmark pointing to this to default to search logs including myself, which is what I mostly do. It will also be a simple way for people to save "default" values for all options.

Great idea! I'll implement it as soon as I figure out how to. In the meantime and if it's worth the bother, you can construct URLs manually.  You'll have to look at the page source to get all the details, but it's basically like this:

http://gokologs.drunkensailor.org/?p1=Andrew%20Iannaccone&p2=&kreq=Minion,Tournament&bots=no&adventures=no&colonies=either&pcount=two&submit=Search


Implemented, as per soulnet's instructions. Thanks!
Title: Re: Log Search Engine
Post by: soulnet on June 10, 2013, 08:44:33 pm
Great idea! I'll implement it as soon as I figure out how to. In the meantime and if it's worth the bother, you can construct URLs manually.  You'll have to look at the page source to get all the details, but it's basically like this:

http://gokologs.drunkensailor.org/?p1=Andrew%20Iannaccone&p2=&kreq=Minion,Tournament&bots=no&adventures=no&colonies=either&pcount=two&submit=Search

Using chrome's inspect element function I changed "post" to "get" in the form "method" parameter and it worked and generated an URL with all parameters like the one you posted and gives the result.

EDIT: Of course, after clicking search again, or newer/older buttons it returns to the page with "post" as it is stored in your server. My point was that just making that change in the server should suffice.

BTW, another really simple thing that would also be nice to have is the "current page / total number of pages" close to the newer/older buttons, and the total number of games (this last one probably right next to the win/tie/loss report).
Title: Re: Log Search Engine
Post by: StrongRhino on June 10, 2013, 08:50:13 pm
I'll report Adventure bugs here: Bonifacius is in Adventures.
Also, this might not be possible, but can you tell of someone quit through a log?
Title: Re: Log Search Engine
Post by: ragingduckd on June 10, 2013, 09:00:11 pm
I'll report Adventure bugs here: Bonifacius is in Adventures.
Also, this might not be possible, but can you tell of someone quit through a log?

Yes, but I haven't been parsing it out.  I'll include it in my next major update, for which I'm re-parsing all the logs to include Shelters and some other info anyway.

On that same topic, if there are other things people would like to search by, this would be a good time to tell me.
Title: Re: Log Search Engine
Post by: StrongRhino on June 10, 2013, 09:15:33 pm
I'll report Adventure bugs here: Bonifacius is in Adventures.
Also, this might not be possible, but can you tell of someone quit through a log?

Yes, but I haven't been parsing it out.  I'll include it in my next major update, for which I'm re-parsing all the logs to include Shelters and some other info anyway.

On that same topic, if there are other things people would like to search by, this would be a good time to tell me.
Thanks. It would also be nice if it could rule out games a specific person quit (read: you or your opponent) but all these things being added are great. Thanks do much for your time and effort.
Title: Re: Log Search Engine
Post by: SCSN on June 11, 2013, 03:24:10 am
On that same topic, if there are other things people would like to search by, this would be a good time to tell me.

The option to exclude games where either of the players quit from the results.

And of course differentiating between pro/casual/unrated games, but I think you've that one on your list already.
Title: Re: Log Search Engine
Post by: ragingduckd on June 11, 2013, 12:16:14 pm
I've been thinking -- perhaps a little too late -- about the privacy issues with the log search.  Yes, I know the logs are already public information b/c Goko is posting them, but there's a big difference between having a window in your bedroom and having someone set up a webcam outside it.

I'm considering adding a feature that would allow players to block searches for their name specifically.

The votes are in (32 of them anyway). I'm calling it. Voyeurs win! ;)
Title: Re: Log Search Engine
Post by: Watno on June 14, 2013, 09:39:10 am
Can you include 1 as a possible player count (you can play solo games in the secret chamber lobby) just for completeness' sake?
Title: Re: Log Search Engine
Post by: treebeardtheent on June 14, 2013, 07:54:51 pm
This is great!

Can you add a search feature for number of turns? I have had a lot crashing issues an  I want to see what the real ratio is vs what I feel it is.

Edit: Or better yet can you add a standing search? 1st, 2nd, etc. I think that would be helpful to more people because you learn more from your losses.
Title: Re: Log Search Engine
Post by: ragingduckd on June 18, 2013, 08:07:36 pm
My automation broke and failed to download a bunch of logs this last week. It's caught up now and back to updating every 15 minutes.

All logs since the Aug 2012 launch are now included. Things are a little slower as a result, but I should be able to fix that with some slightly smarter queries.

I'm also working on a smarter parser to implement some of the features that have been requested, but ftl has been making progress with log parsing for CouncilRoom. I'd certainly rather add the advanced search features to CR rather than maintain a separate server, so we'll see what actually happens.
Title: Re: Log Search Engine
Post by: WanderingWinder on June 18, 2013, 08:11:35 pm
Add Guilds cards to the kingdom card search?
Title: Re: Log Search Engine
Post by: ragingduckd on June 18, 2013, 08:46:53 pm
Now with Guilds!  ::)
Title: Re: Log Search Engine
Post by: soulnet on June 20, 2013, 12:48:34 pm
Would it be too difficult to perform a rating calculation using the retrieved logs? It can be WW's rating or any other reasonable rating that we know the formula. Having a pseudo-official ranking automated would be great, just a leaderboard page with the ranking would suffice. After that, integration with the extension to display it directly into goko and use it for autokicking should fix one of the main issues of online dominion today.
Title: Re: Log Search Engine
Post by: yed on June 20, 2013, 05:01:56 pm
Would it be too difficult to perform a rating calculation using the retrieved logs? It can be WW's rating or any other reasonable rating that we know the formula. Having a pseudo-official ranking automated would be great, just a leaderboard page with the ranking would suffice. After that, integration with the extension to display it directly into goko and use it for autokicking should fix one of the main issues of online dominion today.
It would be possible only for games after half of April. Before that there were no indication of Pro game in log.
EDIT: And few days are missing in mid May.
Title: Re: Log Search Engine
Post by: ragingduckd on June 20, 2013, 08:09:17 pm
Would it be too difficult to perform a rating calculation using the retrieved logs? It can be WW's rating or any other reasonable rating that we know the formula. Having a pseudo-official ranking automated would be great, just a leaderboard page with the ranking would suffice. After that, integration with the extension to display it directly into goko and use it for autokicking should fix one of the main issues of online dominion today.

Yes, I can do this. Actually, I've already written most of the code. I've hesitated for various reasons and I've prioritized other stuff: writing Dominion strategy, playing Dominion, real life. But I'll finish it if it's something that people are eager to have. There isn't as much Goko ratings discontent as there used to be.

@nutki/michaeljb: Is this something you guys would be interested in doing? I could provide the results in rss or whatever's easy to integrate.
Title: Re: Log Search Engine
Post by: rrwoods on June 21, 2013, 02:21:33 am
Would it be too difficult to perform a rating calculation using the retrieved logs? It can be WW's rating or any other reasonable rating that we know the formula. Having a pseudo-official ranking automated would be great, just a leaderboard page with the ranking would suffice. After that, integration with the extension to display it directly into goko and use it for autokicking should fix one of the main issues of online dominion today.

Yes, I can do this. Actually, I've already written most of the code. I've hesitated for various reasons and I've prioritized other stuff: writing Dominion strategy, playing Dominion, real life. But I'll finish it if it's something that people are eager to have. There isn't as much Goko ratings discontent as there used to be.

@nutki/michaeljb: Is this something you guys would be interested in doing? I could provide the results in rss or whatever's easy to integrate.
What would be super-cool is if there was some way to get the Dominion Log Viewer extension to support this rating system; possible if these ratings are stored in a database somewhere?

<- knows nothing about how either of these technologies work
Title: Re: Log Search Engine
Post by: ragingduckd on June 23, 2013, 09:12:10 pm
We're currently offline due to server errors due to administrator incompetence. I'm taking the "opportunity" to do some upgrades and fixes. I'm really not sure when I'll finish, but it's not likely to be before tomorrow. Sorry.
Title: Re: Log Search Engine
Post by: dominator 123 on June 24, 2013, 09:46:56 am
Downvoting it until it's back online.

Edit: OK it's back.
Title: Re: Log Search Engine
Post by: ragingduckd on June 24, 2013, 10:26:19 am
Downvoting it until it's back online.

Well shit. Guess I'd better get cracking.
Title: Re: Log Search Engine
Post by: michaeljb on June 25, 2013, 01:12:11 am
Would it be too difficult to perform a rating calculation using the retrieved logs? It can be WW's rating or any other reasonable rating that we know the formula. Having a pseudo-official ranking automated would be great, just a leaderboard page with the ranking would suffice. After that, integration with the extension to display it directly into goko and use it for autokicking should fix one of the main issues of online dominion today.

Yes, I can do this. Actually, I've already written most of the code. I've hesitated for various reasons and I've prioritized other stuff: writing Dominion strategy, playing Dominion, real life. But I'll finish it if it's something that people are eager to have. There isn't as much Goko ratings discontent as there used to be.

@nutki/michaeljb: Is this something you guys would be interested in doing? I could provide the results in rss or whatever's easy to integrate.

Sounds like it'd be cool. Definitely interested!
Title: Re: Log Search Engine
Post by: ragingduckd on June 25, 2013, 12:01:40 pm
Back online! Sorry for the downtime and thanks for your patience.

The log search should be noticeably faster now, at least once it finishes re-parsing all the logs. I upgraded some hardware, switched database software, and cleaned up some of the uglier query code.

Recent games are available now, and it should finish parsing all the old ones in a few hours.

New features:
- filtering by Shelters/Estates
- filtering by rating system (for games after May 15)
- filtering by number of turns
- filtering by whether any player quits
- display quits in game summary
- direct link to kingdom visualizer (http://forum.dominionstrategy.com/index.php?topic=8629.0)

Note:
- Usernames are now case sensitive.
- Searching by rating system will exclude all games before May 15, when Goko starting logging it

There's a bunch of new code in there, so some new errors may crop up. If anything looks awry, please report it here.
Title: Re: Log Search Engine
Post by: dominator 123 on June 26, 2013, 01:04:44 am
Quote
Error: GabehcoudException while searching for player "dominator 123"

What is the problem?
Title: Re: Log Search Engine
Post by: Watno on June 26, 2013, 11:40:37 am
I think it's still missing some games., is that possible?
Title: Re: Log Search Engine
Post by: ragingduckd on June 26, 2013, 11:52:39 am
I think it's still missing some games., is that possible?

Yes, I ran out of disk space in the middle of the re-parsing. A bunch of pre-March logs seem to have gotten skipped. Was it old ones that you didn't find?

Edit: I will get the ones I missed. I just have to make a second (much faster) pass through the logs.
Title: Re: Log Search Engine
Post by: Watno on June 26, 2013, 12:05:42 pm
I just noticed the number was lower than it should be.
Title: Re: Log Search Engine
Post by: florrat on June 26, 2013, 03:00:28 pm
If I search for my games with any rating type, I get more results than the sum of the number of games with the individual rating types.
I have played 1347 in total, but if I search for the four different rating types, I get 10+731+17+131 < 1347. So something is missing somewhere (or are the missing games the games where the rating type was not mentioned in the game log yet?)
Title: Re: Log Search Engine
Post by: Watno on June 26, 2013, 03:32:40 pm
(or are the missing games the games where the rating type was not mentioned in the game log yet?)
They are I think.
Title: Re: Log Search Engine
Post by: ragingduckd on June 26, 2013, 04:09:21 pm
If I search for my games with any rating type, I get more results than the sum of the number of games with the individual rating types.
I have played 1347 in total, but if I search for the four different rating types, I get 10+731+17+131 < 1347. So something is missing somewhere (or are the missing games the games where the rating type was not mentioned in the game log yet?)

(or are the missing games the games where the rating type was not mentioned in the game log yet?)
They are I think.

Yes, that's exactly the problem. The games I have for you (florrat) in the database look like this:

  rating   | count
-----------+-------
           |   548
 adventure |   176
 casual    |   758
 unrated   |    10
 pro       |    17


The 548 games with no known rating system are the unrated/casual/pro games you played before May 15th.

I suppose the interface could be clearer about this. Maybe a fifth "unknown" category?
Title: Re: Log Search Engine
Post by: ragingduckd on June 26, 2013, 04:20:14 pm
My new policy to automatically upvote all reasonable bug report or feature request. I keep finding bugs on my own that I just know other people must have run into first. I want to encourage people to post bug reports here whenever something looks wrong.

I know it might feel rude to bug me about broken features, but it's actually really helpful. You'd be surprised how many things can break on my end without my noticing. Any post that gives me information is valuable.

Thanks!
Title: Re: Log Search Engine
Post by: florrat on June 26, 2013, 08:02:01 pm
I suppose the interface could be clearer about this. Maybe a fifth "unknown" category?
Yeah, I think that would make things more intuitive for the user.

Also, I can't find any games of me which are not 2 player games, though I do sometimes play adventures or against bots with 3 or 4 players (also recently).
Title: Re: Log Search Engine
Post by: Schneau on June 26, 2013, 08:37:35 pm
My new policy to automatically upvote all reasonable bug report or feature request. I keep finding bugs on my own that I just know other people must have run into first. I want to encourage people to post bug reports here whenever something looks wrong.

I know it might feel rude to bug me about broken features, but it's actually really helpful. You'd be surprised how many things can break on my end without my noticing. Any post that gives me information is valuable.

Thanks!

And my new policy is to upvote any bug fixes or new features. You guys (nutki and michaeljb included) are making Goko and online Dominion analysis not only usable, but downright enjoyable. Hats off to you gentlemen.
Title: Re: Log Search Engine
Post by: ragingduckd on June 26, 2013, 09:04:33 pm
I suppose the interface could be clearer about this. Maybe a fifth "unknown" category?
Yeah, I think that would make things more intuitive for the user.

Good call. Added. It won't actually work until a rather slow DB update finishes though.

Also, I can't find any games of me which are not 2 player games, though I do sometimes play adventures or against bots with 3 or 4 players (also recently).

Er... yeah. I made a silly mistake. Should work now.
Title: Re: Log Search Engine
Post by: soulnet on June 26, 2013, 11:48:17 pm
Good call. Added. It won't actually work until a rather slow DB update finishes though.

Minor detail: If unknown is equivalent to pre May 15th, I would put that (pre May 15th or some equivalent text) instead, its more informative than just "unknown".
Title: Re: Log Search Engine
Post by: ragingduckd on June 27, 2013, 04:17:01 pm
And my new policy is to upvote any bug fixes or new features. You guys (nutki and michaeljb included) are making Goko and online Dominion analysis not only usable, but downright enjoyable. Hats off to you gentlemen.

Thanks for the kind words. I heartily approve of your policy. :)

I'd suggest doing the same for Qvist's card lists and the other serial projects around here. People often upvote only the first post of a labor-intensive series. That's perfectly legitimate, of course, but I'd like to see more such projects and want to encourage them.
Title: Re: Log Search Engine
Post by: florrat on June 27, 2013, 08:16:05 pm
And my new policy is to upvote any bug fixes or new features. You guys (nutki and michaeljb included) are making Goko and online Dominion analysis not only usable, but downright enjoyable. Hats off to you gentlemen.
That's a good point. It's silly that a meme in the meme thread is typically worth more respect than taking multiple hours coding some cool and useful stuff.
Title: Re: Log Search Engine
Post by: Qvist on June 28, 2013, 11:40:15 am
And my new policy is to upvote any bug fixes or new features. You guys (nutki and michaeljb included) are making Goko and online Dominion analysis not only usable, but downright enjoyable. Hats off to you gentlemen.

Thanks for the kind words. I heartily approve of your policy. :)

I'd suggest doing the same for Qvist's card lists and the other serial projects around here. People often upvote only the first post of a labor-intensive series. That's perfectly legitimate, of course, but I'd like to see more such projects and want to encourage them.

Here's an upvote for you. Now please fix the log search, it seems to have stopped working.  :P
Title: Re: Log Search Engine
Post by: ragingduckd on June 28, 2013, 12:10:07 pm
Here's an upvote for you. Now please fix the log search, it seems to have stopped working.  :P

Fixed. I introduced a bug while enabling case-insensitive player-name searching. I really need to add some regression tests. :P
Title: Re: Log Search Engine
Post by: Qvist on June 28, 2013, 03:08:23 pm
Here's an upvote for you. Now please fix the log search, it seems to have stopped working.  :P

Fixed. I introduced a bug while enabling case-insensitive player-name searching. I really need to add some regression tests. :P

Cool, thanks a lot.
Title: Re: Log Search Engine
Post by: Qvist on June 28, 2013, 07:07:58 pm
So, I have tested it out now, it works now, but it doesn't find games from the last 5 days.
Title: Re: Log Search Engine
Post by: ragingduckd on June 28, 2013, 07:51:25 pm
So, I have tested it out now, it works now, but it doesn't find games from the last 5 days.

Oy, yes. I seem to have missed a ton of stuff during the server transition. Importing those now.

I'm going to add some redundancy to make sure I notice when logs get missed.

Edit: Found the problem. I normally really like archlinux, but in this case I just imagine them going "Bwahaha! We've left the cron scheduling service disabled by default. Noone will even think to check that."
Title: Re: Log Search Engine
Post by: ragingduckd on June 29, 2013, 10:18:02 am
New features:
- Added a date range to the search. Dates must be formatted like "YYYY-MM-DD"
- Default rating mode is now "Pro/Unknown." It returns pro games since May 15 and all games before that
- Clear button does exactly what you'd expect

I switched to an SSD drive and I'm finally happy with the speed. None of my test searches took longer than 10 seconds, and most were a whole lot faster. If you're experiencing long delays, let me know. It probably means I've broken something.

I laid out the UI in Firefox, so it looks a bit odd in Chrome. If anyone would be willing to help out with the appearance or can point me to the right sort of resources/tutorials, let me know. I don't want to be a full-fledged CSS/JS developer. I just want to make my UIs a little less hideous.
Title: Re: Log Search Engine
Post by: soulnet on June 29, 2013, 10:34:00 am
-For some reason, JackOfAllTrades appears written like that, without spaces, while all other cards appear with their names spelled nicely, even with special characters like Worker's Village's '.
-Check this out for improving the interface on dates: http://jqueryui.com/datepicker/
-In the search results, I would highlight (like, boldface) the things that were searched for (player names especially, when you search for only one player, but also maybe kingdom cards searched for).
-Its strange to be able to search for wins and losses but not ties. I don't specifically want or need it, but my obsession for completeness does.
Title: Re: Log Search Engine
Post by: Kirian on June 30, 2013, 08:02:10 am
-For some reason, JackOfAllTrades appears written like that, without spaces, while all other cards appear with their names spelled nicely, even with special characters like Worker's Village's '.
-Check this out for improving the interface on dates: http://jqueryui.com/datepicker/
-In the search results, I would highlight (like, boldface) the things that were searched for (player names especially, when you search for only one player, but also maybe kingdom cards searched for).
-Its strange to be able to search for wins and losses but not ties. I don't specifically want or need it, but my obsession for completeness does.

The lack of spaces in JoaT is Goko's fault.
Title: Re: Log Search Engine
Post by: Watno on July 01, 2013, 10:29:36 am
It would be awesome if key cards in the kingdom could be highlighted and searched for.
I just tried Searching for games where Rats were awesome, but had to open all the logs of games where they didn't get bought.

Maybe do something like putting the number of times a card was gained in brackets after it (or the number of times it was in player's decks at the end, which would be easier but less useful), and allow searching for that.
So this game (http://dom.retrobox.eu/?/20130602/log.50b20dc3e4b0c9ce0cf27eb3.1370190267817.txt) would show
Fishing Village (0), Market Square (10), Oasis (2), Bridge (1), Caravan (5), Rats (10), Remodel (1), Pillage (0), Harem (4), Peddler (10)

and could be found by entering stuff like the following in the kingdom field:
"Rats 1+", "Peddler 10", or "Fishing Village 0"

Also, I'd prefer if clicking the link to the log of a game would open it in a new tab instead of the one the search is in.
Title: Re: Log Search Engine
Post by: StrongRhino on July 01, 2013, 06:13:32 pm
It would be awesome if key cards in the kingdom could be highlighted and searched for.
I just tried Searching for games where Rats were awesome, but had to open all the logs of games where they didn't get bought.

Maybe do something like putting the number of times a card was gained in brackets after it (or the number of times it was in player's decks at the end, which would be easier but less useful), and allow searching for that.
So this game (http://dom.retrobox.eu/?/20130602/log.50b20dc3e4b0c9ce0cf27eb3.1370190267817.txt) would show
Fishing Village (0), Market Square (10), Oasis (2), Bridge (1), Caravan (5), Rats (10), Remodel (1), Pillage (0), Harem (4), Peddler (10)

and could be found by entering stuff like the following in the kingdom field:
"Rats 1+", "Peddler 10", or "Fishing Village 0"

Also, I'd prefer if clicking the link to the log of a game would open it in a new tab instead of the one the search is in.
Similar to this, maybe you could be able to search for what cards aren't in a kingdom (like golem with no- *name all villages* to narrow down games to find possible golem engines). Thanks.
Title: Re: Log Search Engine
Post by: GwinnR on July 04, 2013, 10:05:36 am
Hey, thanks for that!

I have a (little) wishe. It would be cool to find games without naming a player. So I could find for example all games with Rebuild to learn from it.
Title: Re: Log Search Engine
Post by: ragingduckd on July 04, 2013, 10:37:25 am
Hey, thanks for that!

I have a (little) wishe. It would be cool to find games without naming a player. So I could find for example all games with Rebuild to learn from it.

Unfortunately, this is prohibitively slow, at least with how my database query code is written now. I may be able to improve it.

Do you really want all Rebuild games? How about all Rebuild games with top-25 players? Or all games between two top-100 players?
Title: Re: Log Search Engine
Post by: serakfalcon on July 06, 2013, 04:45:59 pm
Quote
Unfortunately, this is prohibitively slow, at least with how my database query code is written now. I may be able to improve it.


with MySQL u could use LIMIT ($pagenum * $resultsperpage,($pagenum +1)*$resultsperpage) and query per page, instead of running the entire query all at once
Title: Re: Log Search Engine
Post by: ragingduckd on July 06, 2013, 04:58:52 pm
Quote
Unfortunately, this is prohibitively slow, at least with how my database query code is written now. I may be able to improve it.


with MySQL u could use LIMIT ($pagenum * $resultsperpage,($pagenum +1)*$resultsperpage) and query per page, instead of running the entire query all at once

Okay, okay. Yeah. What I really meant was "I will definitely be able to improve it, but I'm out of town and can't get to it right away." ;)
Title: Re: Log Search Engine
Post by: fprefect on July 08, 2013, 08:23:44 am
Dear RagingDuckd,

Would it be possible to extract a history of all game results (no logs, but information about gamelength and maybe kingdom) into a single text file in any way (if it is not too much work). I want to have a look at the issues concerning rating systems.

regards,

fprefect
Title: Re: Log Search Engine
Post by: ragingduckd on July 10, 2013, 03:15:26 am
Dear RagingDuckd,

Would it be possible to extract a history of all game results (no logs, but information about gamelength and maybe kingdom) into a single text file in any way (if it is not too much work). I want to have a look at the issues concerning rating systems.

regards,

fprefect

Yes. I can do that.

What data format would you like? Either csv or postgresql dump is easiest for me.

Which variables do you want? I have game date/time; kingdom; whether the game has colonies/shelters; player names; whether players are bots/guests; rating system; each player's vps, # of turns, and whether he quit/resigned. I don't have Goko ratings for many of the games.
Title: Re: Log Search Engine
Post by: fprefect on July 10, 2013, 07:41:33 am
Hello,

thank you for the offer. I would be most pleased with csv (one result : one line). I wouldnt mind to have all the information in you are suggesting but the most important for me would be:

date/time; player names; result; bots/guests; #turns; quit/resign; vps;

I am not interested in goko ratings much. You could upload it to somewhere and leave the link here, because I might not be the only one interested.

fprefect
Title: Re: Log Search Engine
Post by: WanderingWinder on July 10, 2013, 07:48:23 am
Hello,

thank you for the offer. I would be most pleased with csv (one result : one line). I wouldnt mind to have all the information in you are suggesting but the most important for me would be:

date/time; player names; result; bots/guests; #turns; quit/resign; vps;

I am not interested in goko ratings much. You could upload it to somewhere and leave the link here, because I might not be the only one interested.

fprefect
You aren't.

Lightning Edit: You aren't the only one interested, that is.
Title: Re: Log Search Engine
Post by: ragingduckd on July 10, 2013, 10:45:27 am
Here's all the 2-player pro games since goko started distinguishing pro from casual and unrated. Let me know if you want 3+ player games or the other rating modes too.

http://www.sendspace.com/file/21v5ym

Note that "bot" is true if either player is a bot. Same for "guest."
Title: Re: Log Search Engine
Post by: greatexpectations on July 10, 2013, 11:19:18 am
this probably isn't the right thread for this, but i wasn't sure it was worth starting a new one. what would the difficulty be of creating an archive of the goko leaderboards? or does such a thing exist already? i'm thinking of something along the lines of this (http://bggdl.square7.ch/leaderboard/) for the old iso leaderboards. it wouldn't necessarily have to include game totals or even be pulled once a day. but i think it would be interesting to track progress/ratings over time as well as having a simple way of scanning the leaderboard for names of players.
Title: Re: Log Search Engine
Post by: fprefect on July 11, 2013, 10:45:51 am
Thanks a million times,

I will tell you when I had a close look.

fprefect
Title: Re: Log Search Engine
Post by: Watno on July 11, 2013, 05:37:08 pm
Not a huge deal, but I would prefer if the default timerange included all games and not just those of the last month.
Title: Re: Log Search Engine
Post by: ragingduckd on July 12, 2013, 07:51:48 pm
For those of you who are interested in large piles of data, you can now access my game database directly. Install a PostgreSQL client and connect to:

host: gokosalvager.com
port: 5432
user: forum
pass: fds

This is the same server that's supporting the search engine and kingdom visualizer, so please be considerate.
Title: Re: Log Search Engine
Post by: Watno on July 13, 2013, 07:48:12 pm
I can't fid games with both players specified anymore. The games still show up if you specify just one player.
Title: Re: Log Search Engine
Post by: yed on July 13, 2013, 08:02:14 pm
I keep getting this error:
500: Internal Server Error
Title: Re: Log Search Engine
Post by: ragingduckd on July 13, 2013, 08:48:19 pm
Both of those errors are fixed now worse than when I started. I'm puzzled.

Fixed for real this time.
Title: Re: Log Search Engine
Post by: ragingduckd on July 16, 2013, 09:28:59 pm
Sorry if the log search has been down for you recently. My internet was down for most of the day. Please bear in mind that home internet access is a new and challenging technology and we all need to be patient for the next 10-20 years it'll take for the United States' government-enforced telecom monoplies to get a handle on it.

Seriously, it should be okay now. I'll switch to an offsite server if the problem keeps up and if I can find hosting that will give me fast access to the ~100GB of game data.
Title: Re: Log Search Engine
Post by: ashersky on July 16, 2013, 11:27:10 pm
AI, can you and DougZ have a baby that will grow up to create the Virtual Reality Live Action Dominion-torium for us all to battle in Yu-Gi-Oh style?



What I mean is, thank you for all your incredible work.  It's awesome.
Title: Re: Log Search Engine
Post by: ragingduckd on July 17, 2013, 12:37:19 am
AI, can you and DougZ have a baby that will grow up to create the Virtual Reality Live Action Dominion-torium for us all to battle in Yu-Gi-Oh style?

That's a remarkably patient feature request!
Title: Re: Log Search Engine
Post by: ragingduckd on July 18, 2013, 08:03:53 am
Updated.

Removed features:
- Case-sensitive username search
- Overall win-loss record

I'll reimplement the win-loss record elsewhere. It's a nuisance when I'm trying to code for reasonable search speed.

Added features:
- Open log in new tab/window
- Date range defaults to Goko's entire history
- correct Goko's "JackOfAllTrades" nonsense
- allow searching without specifying a player (it's rather slow though)
- filter by draws
- filter by resignations
- filter by cards not in kingdom
- more elegant date picker

If you bookmarked a search with your preferred search paramters, you'll need to replace it. I changed some of the parameter names and formatting.

Can anyone confirm that this is working on Chrome or Chromium? I developed in Firefox and I think I may have loused up the JS or something.
Title: Re: Log Search Engine
Post by: florrat on July 19, 2013, 08:02:09 pm
I tested all new features in Chrome. The features work good.

Some comments:
-When putting a non-existing card in the "In Kingdom"/"Not in Kingdom"-fields, I get a blank screen except for the message "500: Internal Server Error". Maybe give a little more subtle error message?  :)
-If I specify no player but a lot of cards in "In Kingdom", why don't you want to search it? Checking for an additional card in the kingdom should only slightly increase the search time, shouldn't it? Because you only have to check the hits for the new card.
Title: Re: Log Search Engine
Post by: ragingduckd on July 19, 2013, 08:19:58 pm
I tested all new features in Chrome. The features work good.

Thanks! I'm not facile with regression testing on web apps, so this is very good to know.

Some comments:
-When putting a non-existing card in the "In Kingdom"/"Not in Kingdom"-fields, I get a blank screen except for the message "500: Internal Server Error". Maybe give a little more subtle error message?  :)

The nicely formatted error messages are things that I'm explicitly checking for. The "500" errors happen when there's something wrong with my code or a situation I didn't think of. So I can give you a clearer generic error like "AI's code farted" or a stack trace, but that's about it. Thanks for the bug report tho.

-If I specify no player but a lot of cards in "In Kingdom", why don't you want to search it? Checking for an additional card in the kingdom should only slightly increase the search time, shouldn't it? Because you only have to check the hits for the new card.

Actually each additional card is an enormous increase in search time.

My database is indexed by player names, but not by supply cards. That means that I actually have to look through every individual game and check the supply against the search parameters, which is very slow. It keeps looking until it finds N matches, so the more cards you specify, the longer it has to look. Asking for 20 games with any particular set of 4+ cards requires looking through the entire database.

I'll eventually add indexing by supply cards, so those searches will be faster, but I don't see a good way to index by cards that aren't in the supply. If someone has a good suggestion, I'll be happy to implement it, but I also suspect this isn't at the top of many people's feature request lists.
Title: Re: Log Search Engine
Post by: florrat on July 19, 2013, 08:30:22 pm
The nicely formatted error messages are things that I'm explicitly checking for. The "500" errors happen when there's something wrong with my code or a situation I didn't think of. So I can give you a clearer generic error like "AI's code farted" or a stack trace, but that's about it. Thanks for the bug report tho.
I meant adding a nicely formatted error message for this particular error. I understand that if something happens you didn't think of cannot give a very meaningful error message to a casual user.



Thanks for the explanation about the search time. I've never programmed a search function myself, so I naively thought you would generate all results and then display the first 20. Now that I actually think about it, I understand that this is not the way things work :-X
Title: Re: Log Search Engine
Post by: serakfalcon on July 19, 2013, 09:42:32 pm
Quote
I'll eventually add indexing by supply cards, so those searches will be faster, but I don't see a good way to index by cards that aren't in the supply. If someone has a good suggestion, I'll be happy to implement it, but I also suspect this isn't at the top of many people's feature request lists.
Create a join table of the game ID and card IDs
for example, if game is 1, then the two columns will be
1,cardID(smuggler)
1,cardID(scout)
etc.
then do an inner join of the game table to the cards per game table. (inner join cards per game table on game id, add a where clause restricting the results where the card id is present) It should be faster since both columns will be indexed integers as opposed to a text search.
Its a little tricky to set up, after thinking about it a bit more but that should be the general right way to get started
Title: Re: Log Search Engine
Post by: serakfalcon on July 19, 2013, 09:45:35 pm
I can devote some time to this in a couple weeks
Title: Re: Log Search Engine
Post by: ragingduckd on July 20, 2013, 05:11:02 am
I can devote some time to this in a couple weeks

That would be great. Thanks!
Title: Re: Log Search Engine
Post by: AdamH on July 26, 2013, 12:12:35 pm
Some easy things that might be nice:

The pull-down menus for Rating, Colonies, etc. might be better served by using checkboxes instead of pull-down menus. Rating for sure would be better, but for something like Colonies, instead of "Allow/Require/No" you could have a checkbox for finding colony games and another one for finding no-colony games. Just a thought.

Also, it might be nice to display some stats with the results: total number of games found would be nice. If one or more users were specified in "Player 1" or "Player 2" it might be cool to display the record of each of those users in all games found (this used to be there, didn't it?)

Thanks for all you do, in any case :-D
Title: Re: Log Search Engine
Post by: sudgy on July 28, 2013, 07:03:52 pm
Yay, now I know that my winning streak just now was eight games!

edit: And Stealth Tomato, you broke it...
Title: Re: Log Search Engine
Post by: Kirian on July 28, 2013, 07:16:40 pm
Yay, now I know that my winning streak just now was eight games!

edit: And Stealth Tomato, you broke it...

I bet you didn't see him coming.
Title: Re: Log Search Engine
Post by: sudgy on July 28, 2013, 07:27:30 pm
Yay, now I know that my winning streak just now was eight games!

edit: And Stealth Tomato, you broke it...

I bet you didn't see him coming.

Well, my eighth game was beating him...
Title: Re: Log Search Engine
Post by: ragingduckd on August 01, 2013, 07:05:07 pm
Going offline for server maintenance. Back in 2 hours or less.

Edit: Back. Three minutes ahead of schedule even!
Title: Re: Log Search Engine
Post by: sudgy on August 01, 2013, 07:06:10 pm
Aaaah!  This thread won't go off of my "unread posts" list!
Title: Re: Log Search Engine
Post by: mail-mi on August 07, 2013, 01:19:34 pm
Hm... I'm trying to find a log to a game that I just played, and it says no matches found.
Title: Re: Log Search Engine
Post by: sudgy on August 07, 2013, 01:23:14 pm
Hm... I'm trying to find a log to a game that I just played, and it says no matches found.

It might not have added it yet, it updates every ten minutes.
Title: Re: Log Search Engine
Post by: Kirian on August 07, 2013, 09:56:24 pm
Hm... I'm trying to find a log to a game that I just played, and it says no matches found.

It might not have added it yet, it updates every ten minutes.

Which is, notably, faster than CR.com was!  Hopefully the new (eventually?) CR.com will use the new script.
Title: Re: Log Search Engine
Post by: ragingduckd on August 07, 2013, 10:07:16 pm
Hm... I'm trying to find a log to a game that I just played, and it says no matches found.

It might not have added it yet, it updates every ten minutes.

Which is, notably, faster than CR.com was!  Hopefully the new (eventually?) CR.com will use the new script.

Hopefully not. That script is probably the ugliest thing in my whole project. And a 10-minute delay is still pretty excessive, really.
Title: Re: Log Search Engine
Post by: Kirian on August 07, 2013, 10:40:47 pm
Hm... I'm trying to find a log to a game that I just played, and it says no matches found.

It might not have added it yet, it updates every ten minutes.

Which is, notably, faster than CR.com was!  Hopefully the new (eventually?) CR.com will use the new script.

Hopefully not. That script is probably the ugliest thing in my whole project. And a 10-minute delay is still pretty excessive, really.

Better than 24 hours though.  Although that was because Dougz made the logs available as gz files at the end of the day.
Title: Re: Log Search Engine
Post by: ragingduckd on August 08, 2013, 01:51:07 am
Hm... I'm trying to find a log to a game that I just played, and it says no matches found.

It might not have added it yet, it updates every ten minutes.

Which is, notably, faster than CR.com was!  Hopefully the new (eventually?) CR.com will use the new script.

Hopefully not. That script is probably the ugliest thing in my whole project. And a 10-minute delay is still pretty excessive, really.

Better than 24 hours though.  Although that was because Dougz made the logs available as gz files at the end of the day.

True. But you motivated me to finally fix my crappy code. The logsearch and Isotropish leaderboard are now updating continuously. There's a brief delay b/c it takes a moment for me to download the list of available logs and find and parse the additions, but otherwise they're getting adding as soon as Goko posts them..

Edit: And by that I mean either updating immediately or not updating at all, depending on whether or not my Python code feels like screwing with me.
Title: Re: Log Search Engine
Post by: Watno on August 09, 2013, 09:53:10 am
Minor inconsistency:
For colony games, the output is like this: Colonies, Estates - Bridge, Contraband, Develop, Fortress, Masquerade, Mint, Swindler, Trade Route, Treasure Map, Wharf
For province games it is: Provinces / Estates - Adventurer, Altar, Beggar, Counterfeit, Hunting Grounds, Mystic, Nobles, Saboteur, Smithy, Wishing Well
(, vs /)
Title: Re: Log Search Engine
Post by: Watno on August 12, 2013, 01:08:05 pm
Goko staff is now actually refering people to this page: https://getsatisfaction.com/goko/topics/bureaucrat_island_bug
Title: Re: Log Search Engine
Post by: Kirian on August 12, 2013, 03:35:30 pm
Goko staff is now actually refering people to this page: https://getsatisfaction.com/goko/topics/bureaucrat_island_bug

So much win.
Title: Re: Log Search Engine
Post by: Voltaire on August 12, 2013, 04:32:57 pm
Goko staff is now actually refering people to this page: https://getsatisfaction.com/goko/topics/bureaucrat_island_bug

So much win.

I hope Goko is compensating them appropriately.
Title: Re: Log Search Engine
Post by: ragingduckd on August 12, 2013, 05:38:52 pm
Goko staff is now actually refering people to this page: https://getsatisfaction.com/goko/topics/bureaucrat_island_bug

Hilarious! Someone who isn't me should post to getsatisfaction.com:

"Dear Lord Humanton,

Thank you for providing your users with this wonderful service. Your log viewer and automatch are also big steps forward. It's nice to see that Goko is really listening to its userbase and giving them the features they need."
Title: Re: Log Search Engine
Post by: StrongRhino on August 12, 2013, 07:04:52 pm
Goko staff is now actually refering people to this page: https://getsatisfaction.com/goko/topics/bureaucrat_island_bug

Hilarious! Someone who isn't me should post to getsatisfaction.com:

"Dear Lord Humanton,

Thank you for providing your users with this wonderful service. Your log viewer and automatch are also big steps forward. It's nice to see that Goko is really listening to its userbase and giving them the features they need."
Done.
Title: Re: Log Search Engine
Post by: ftl on August 13, 2013, 03:44:24 am
http://gokologs.drunkensailor.org/ (http://gokologs.drunkensailor.org/) gives me "Unhandled exception (probably a programming error). Please report in the forums if you can reproduce it."

Reporting on forums as requested.

[edit] Wait it started working later. It was reproducible for a few minutes which is why I posted it but then it started working. Nevermind then. [/edit]
Title: Re: Log Search Engine
Post by: ragingduckd on August 13, 2013, 03:55:02 am
http://gokologs.drunkensailor.org/ (http://gokologs.drunkensailor.org/) gives me "Unhandled exception (probably a programming error). Please report in the forums if you can reproduce it."

Reporting on forums as requested.

[edit] Wait it started working later. It was reproducible for a few minutes which is why I posted it but then it started working. Nevermind then. [/edit]

I guess I really meant "please report what you were doing and how to reproduce the error." Were you just going to the site?
Title: Re: Log Search Engine
Post by: ftl on August 13, 2013, 04:03:27 am
yep. Loaded http://gokologs.drunkensailor.org/ (http://gokologs.drunkensailor.org/) to look for a log, got the error. Refreshed a few times, kept getting it so I posted it here. A few minutes later tried again and it worked, so dunno.
Title: Re: Log Search Engine
Post by: ragingduckd on August 13, 2013, 04:23:49 am
yep. Loaded http://gokologs.drunkensailor.org/ (http://gokologs.drunkensailor.org/) to look for a log, got the error. Refreshed a few times, kept getting it so I posted it here. A few minutes later tried again and it worked, so dunno.

Interesting... I found it in my logs and it's "too many open files" for my OS. I have no clue how that could happen.

I rarely have more than 2 or 3 simultaneous database connections. The only major change I've made recently is the extra websocket connections from automatch, but I don't think those shouldn't be talking to the disk at all. I'll have to think about it some more. If anyone else experiences the same problem, please post so.
Title: Re: Log Search Engine
Post by: SCSN on August 13, 2013, 05:19:39 am
http://gokologs.drunkensailor.org/ keeps trying to load for a while before giving up, leaving "Unhandled exception (probably a programming error). Please report in the forums if you can reproduce it." as its resignation letter.
Title: Re: Log Search Engine
Post by: yed on August 13, 2013, 05:30:45 am
It is probably caused by websockets.

I'm not sure what server you use, but someone had similar problem:
http://stackoverflow.com/questions/5695107/socket-io-how-many-concurrent-connections-can-websockets-handle/5695547
Title: Re: Log Search Engine
Post by: Awaclus on August 13, 2013, 05:35:04 am
Goko staff is now actually refering people to this page: https://getsatisfaction.com/goko/topics/bureaucrat_island_bug
This is... so much, so much win. Someone else has to implement their features for them, and then they can't even type the URL correctly.  ;D
Title: Re: Log Search Engine
Post by: ragingduckd on August 13, 2013, 09:29:31 am
It is probably caused by websockets.

I'm not sure what server you use, but someone had similar problem:
http://stackoverflow.com/questions/5695107/socket-io-how-many-concurrent-connections-can-websockets-handle/5695547

Good call. I ran an lsof and confirmed. Very surprising, both because I wouldn't have expected websockets to use the disk at all and second because my max file limit is already absurdly high.

I suspect it's a file descriptor leak. It seems to be a semi-common problem with the Tornado (http://www.tornadoweb.org/en/stable/), the server I'm using. I may have to try a different server package.
Title: Re: Log Search Engine
Post by: GeoLib on August 15, 2013, 02:48:36 am
I was playing a game today and afterward my opponent commented that he wished he could see the trash, and I said "you can" and then realized that you actually can't.

Feature request: Would it be possible to show the trash in the log prettifier. Maybe with the deck contents at the bottom. Not a huge priority, but sometimes it can be interesting.
Title: Re: Log Search Engine
Post by: sudgy on August 15, 2013, 02:52:00 am
I was playing a game today and afterward my opponent commented that he wished he could see the trash, and I said "you can" and then realized that you actually can't.

Feature request: Would it be possible to show the trash in the log prettifier. Maybe with the deck contents at the bottom. Not a huge priority, but sometimes it can be interesting.

During the game you can.  Just click on the trash pile at the bottom right.
Title: Re: Log Search Engine
Post by: ragingduckd on August 15, 2013, 02:54:35 am
I was playing a game today and afterward my opponent commented that he wished he could see the trash, and I said "you can" and then realized that you actually can't.

Feature request: Would it be possible to show the trash in the log prettifier. Maybe with the deck contents at the bottom. Not a huge priority, but sometimes it can be interesting.

During the game you can.  Just click on the trash pile at the bottom right.

But it would be nice to have it while reviewing the game too. In nutki's log prettifier on retrobox. "Hey nutki..."
Title: Re: Log Search Engine
Post by: michaeljb on August 15, 2013, 03:07:12 am
I was playing a game today and afterward my opponent commented that he wished he could see the trash, and I said "you can" and then realized that you actually can't.

Feature request: Would it be possible to show the trash in the log prettifier. Maybe with the deck contents at the bottom. Not a huge priority, but sometimes it can be interesting.

During the game you can.  Just click on the trash pile at the bottom right.

But it would be nice to have it while reviewing the game too. In nutki's log prettifier on retrobox. "Hey nutki..."

I think the prettifier is this (http://forum.dominionstrategy.com/index.php?topic=6788.10), so we might be able to do something with it.
Title: Re: Log Search Engine
Post by: sudgy on August 28, 2013, 02:23:59 am
Is the site being down related to automatch being down?
Title: Re: Log Search Engine
Post by: ragingduckd on August 28, 2013, 02:33:41 am
Is the site being down related to automatch being down?

Yes, there's a serious bug that makes the automatch server go crazy maybe once a day or so, and it crashes everything else too. So far I've just been restarting the server process when it happens, which is a stupid solution and is why we're having these brief outages. I haven't had time to track down and fix the actual problem.

Thanks for letting me know. I've restarted it again.
Title: Re: Log Search Engine
Post by: ragingduckd on August 31, 2013, 06:43:29 pm
Finally fixed the bug that makes searching for losses break.

I may also have fixed the "too many open files" bug that has been making the whole server crash. We'll have to wait and see if it worked. Nope :(
Title: Re: Log Search Engine
Post by: GeoLib on September 10, 2013, 11:52:47 pm
Currently "http://gokologs.drunkensailor.org/" and "http://gokologs.drunkensailor.org/logsearch" are giving me blank pages, while the leaderboard and prettifier appear to be working as normal.
Title: Re: Log Search Engine
Post by: flies on September 11, 2013, 02:32:36 pm
it's working for me...
Title: Re: Log Search Engine
Post by: ragingduckd on September 11, 2013, 02:47:45 pm
Currently "http://gokologs.drunkensailor.org/" and "http://gokologs.drunkensailor.org/logsearch" are giving me blank pages, while the leaderboard and prettifier appear to be working as normal.

it's working for me...

I restarted it. It was throwing an I/O error while trying to print to the console (??).
Title: Re: Log Search Engine
Post by: GeoLib on September 11, 2013, 04:31:08 pm
Currently "http://gokologs.drunkensailor.org/" and "http://gokologs.drunkensailor.org/logsearch" are giving me blank pages, while the leaderboard and prettifier appear to be working as normal.

it's working for me...

I restarted it. It was throwing an I/O error while trying to print to the console (??).

It's working now. Thanks! :-)
Title: Re: Log Search Engine
Post by: amalloy on September 21, 2013, 05:43:27 am
I just played this game (http://dom.retrobox.eu/?/20130921/log.516d31dbe4b082c74d7b3391.1379752819309.txt) - note the Asian characters in my opponent's name. When I see it in the log-search page (http://gokologs.drunkensailor.org/logsearch?p1name=alan+malloy&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=09%2F21%2F2013&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&submitted=true&offset=0), it comes out garbled. I haven't dug into the details, but the log viewer sets charset=UTF-8, and the log search doesn't set a charset at all, so I assume my browser is interpreting it as Latin-1 or some similarly silly thing.
Title: Re: Log Search Engine
Post by: GwinnR on October 08, 2013, 05:26:10 am
Is there any possibility to search for an exact card-name? If I search for Count, it finds Count, Counting House, Counterfeit and Count. Searching for "Count" like I would do in google doesn't work.
Title: Re: Log Search Engine
Post by: Emeric on October 10, 2013, 10:06:34 am
Is it possible to add an option, the setting of my timezone, and like that to display the time of the game in my local time ?
Title: Re: Log Search Engine
Post by: flies on October 10, 2013, 12:09:19 pm
Yesterday i was searching logs and found that http://gokologs.drunkensailor.org/ would only show games prior to mid april whereas http://www.andrewiannaccone.com/ was up to date.  Can anyone explain this?  What's the relationship between these two addresses anyway?  (sorry if the answer to the last question is explained above.)
Title: Re: Log Search Engine
Post by: ragingduckd on October 10, 2013, 01:21:43 pm
Yesterday i was searching logs and found that http://gokologs.drunkensailor.org/ would only show games prior to mid april whereas http://www.andrewiannaccone.com/ was up to date.  Can anyone explain this?  What's the relationship between these two addresses anyway?  (sorry if the answer to the last question is explained above.)

I have no idea why this should be the case. I was certain they linked to the same page. Can you post or PM me two search links that show different results?
Title: Re: Log Search Engine
Post by: flies on October 10, 2013, 02:35:56 pm
Yesterday i was searching logs and found that http://gokologs.drunkensailor.org/ would only show games prior to mid april whereas http://www.andrewiannaccone.com/ was up to date.  Can anyone explain this?  What's the relationship between these two addresses anyway?  (sorry if the answer to the last question is explained above.)

I have no idea why this should be the case. I was certain they linked to the same page. Can you post or PM me two search links that show different results?
I can't duplicate the bug.  I would have thought they were just pointing to the same page, but I can describe how I produced the error.

I just put in a player name and made damn sure I wasn't just setting the dates wrong, and it would stop in mid-April.
Title: Re: Log Search Engine
Post by: SirPeebles on October 10, 2013, 02:45:48 pm
Am I the only person who thinks that this is a topic about O(log n) search algorithms every time?
Title: Re: Log Search Engine
Post by: flies on October 10, 2013, 02:55:17 pm
Am I the only person who thinks that this is a topic about O(log n) search algorithms every time?
those exist?  or did you mean O(n log n)?
Title: Re: Log Search Engine
Post by: soulnet on October 10, 2013, 03:00:11 pm
those exist?  or did you mean O(n log n)?

You are confusing searching with sorting. Most sorted structures, including sorted arrays, search in O(log n).
Title: Re: Log Search Engine
Post by: SirPeebles on October 10, 2013, 03:03:37 pm
Am I the only person who thinks that this is a topic about O(log n) search algorithms every time?
those exist?  or did you mean O(n log n)?

Well, it depends on what you are searching and whether there is known structure.  If you an ordered list of names and you are searching for which position Karen is in, then it would be a O(log n) search, right?  You'd query the name in the middle, and determine whether Karen is before or after it, and continue in this manner.
Title: Re: Log Search Engine
Post by: GeoLib on October 10, 2013, 03:56:34 pm
Am I the only person who thinks that this is a topic about O(log n) search algorithms every time?
those exist?  or did you mean O(n log n)?

Well, it depends on what you are searching and whether there is known structure.  If you an ordered list of names and you are searching for which position Karen is in, then it would be a O(log n) search, right?  You'd query the name in the middle, and determine whether Karen is before or after it, and continue in this manner.

Even if it's unsorted you can get at worst O(n) by just traversing the entire array. Sounds like he's thinking of sorting
Title: Re: Log Search Engine
Post by: flies on October 10, 2013, 04:33:10 pm
right.  sorting != searching.
Title: Re: Log Search Engine
Post by: Warfreak2 on October 10, 2013, 04:50:12 pm
If someone really did unwittingly write an O(n log n) search algorithm, I will buy them a beer.
Title: Re: Log Search Engine
Post by: Schneau on October 10, 2013, 04:57:16 pm
If someone really did unwittingly write an O(n log n) search algorithm, I will buy them a beer.

I'm sure someone has: 1. Sort list. 2. Binary search.
Title: Re: Log Search Engine
Post by: Schneau on October 10, 2013, 04:58:19 pm
Am I the only person who thinks that this is a topic about O(log n) search algorithms every time?

It's funny you mention it, since I saw the topic and thought it was about an engine deck featuring Woodcutter and Scout.
Title: Re: Log Search Engine
Post by: SCSN on October 10, 2013, 05:03:52 pm
If someone really did unwittingly write an O(n log n) search algorithm, I will buy them a beer.

Forwarding this to info@goko.com...
Title: Re: Log Search Engine
Post by: flies on October 11, 2013, 12:09:58 am
Yesterday i was searching logs and found that http://gokologs.drunkensailor.org/ would only show games prior to mid april whereas http://www.andrewiannaccone.com/ was up to date.  Can anyone explain this?  What's the relationship between these two addresses anyway?  (sorry if the answer to the last question is explained above.)

I have no idea why this should be the case. I was certain they linked to the same page. Can you post or PM me two search links that show different results?
I found a way to reproduce a very similar bug.  Search for a player's games with an invalid date, e.g. '10/7/2013' (should be '07'), producing an error message, then put in a valid date and watch the wackiness ensue.  Same effect on http://www.andrewiannaccone.com/ and drunkensailor.

correct: http://gokologs.drunkensailor.org/logsearch?p1name=stef&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=10%2F07%2F2013&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0

incorrect: http://gokologs.drunkensailor.org/logsearch?p1name=stef&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=10%2F07%2F2013&supply=&nonsupply=&rating=unrated&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=any&resign=any&limit=20&submitted=true&offset=0

So the difference wasn't the domain but some hysteresis.

I can see that correct says quit=false while incorrect says quit=any.  I see no other differences, but manually setting the quit=false doesn't solve the problem.
Title: Re: Log Search Engine
Post by: yed on December 10, 2013, 10:44:52 am
http://gokologs.drunkensailor.org/ is down
Title: Re: Log Search Engine
Post by: ragingduckd on December 10, 2013, 01:17:23 pm
http://gokologs.drunkensailor.org/ is down

Restarted it.  Sorry about that.

Sorry also that I can't promise much better anytime soon.  I'm traveling and will have only very infrequent internet access for the next couple weeks.  Here's hoping that my server starts behaving itself a little better in my absence.   :-\
Title: Re: Log Search Engine
Post by: GeoLib on December 10, 2013, 03:28:23 pm
http://gokologs.drunkensailor.org/ is down

Restarted it.  Sorry about that.

Sorry also that I can't promise much better anytime soon.  I'm traveling and will have only very infrequent internet access for the next couple weeks.  Here's hoping that my server starts behaving itself a little better in my absence.   :-\

I heard from the people over at Goko that servers actually perform best when left alone entirely without any maintenance, so you should be fine :P
Title: Re: Log Search Engine
Post by: soulnet on February 04, 2014, 06:02:40 pm
It seems like something is not working properly. I am sure I played two games in a row against SheCantSayNo today, and only one appears in:

http://gokologs.drunkensailor.org/logsearch?p1name=soulnet+goko&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=07%2F27%2F2050&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0

and also in

http://gokologs.drunkensailor.org/logsearch?p1name=shecantsayno&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=07%2F27%2F2050&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0

Since it is the same, I will conjecture that some logs are not being parsed.
Title: Re: Log Search Engine
Post by: WanderingWinder on February 04, 2014, 06:14:59 pm
It seems like something is not working properly. I am sure I played two games in a row against SheCantSayNo today, and only one appears in:

http://gokologs.drunkensailor.org/logsearch?p1name=soulnet+goko&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=07%2F27%2F2050&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0

and also in

http://gokologs.drunkensailor.org/logsearch?p1name=shecantsayno&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=07%2F27%2F2050&supply=&nonsupply=&rating=pro%2B&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0

Since it is the same, I will conjecture that some logs are not being parsed.

http://forum.dominionstrategy.com/index.php?topic=10215.25
Title: Re: Log Search Engine
Post by: ragingduckd on April 24, 2014, 12:29:57 pm
Hooray for amalloy, who fixed the special characters issues in the log prettifier!  Now your username can end in "$" or "\" or whatever sort of regex-abusing tomfoolery you want: http://gokosalvager.com/static/logprettifier.html?20140424/log.5101a6c4e4b02b7235c3860f.1398355996250.txt

I suggest upvoting this random post of his (http://forum.dominionstrategy.com/index.php?topic=9217.msg284256#msg284256).
Title: Re: Log Search Engine
Post by: Awaclus on April 24, 2014, 03:35:40 pm
Hooray for amalloy, who fixed the special characters issues in the log prettifier!  Now your username can end in "$" or "\" or whatever sort of regex-abusing tomfoolery you want: http://gokosalvager.com/static/logprettifier.html?20140424/log.5101a6c4e4b02b7235c3860f.1398355996250.txt

I suggest upvoting this random post of his (http://forum.dominionstrategy.com/index.php?topic=9217.msg284256#msg284256).
How do you do the randomization for the posts?
Title: Re: Log Search Engine
Post by: ragingduckd on April 24, 2014, 05:10:36 pm
Hooray for amalloy, who fixed the special characters issues in the log prettifier!  Now your username can end in "$" or "\" or whatever sort of regex-abusing tomfoolery you want: http://gokosalvager.com/static/logprettifier.html?20140424/log.5101a6c4e4b02b7235c3860f.1398355996250.txt

I suggest upvoting this random post of his (http://forum.dominionstrategy.com/index.php?topic=9217.msg284256#msg284256).
How do you do the randomization for the posts?

I search through old posts and choose the one that maximizes:
   f(p) = p_irrelevance * p_irony / log(p_currentupvotes) + (AI_boredoflookingthroughposts)^500
Title: Re: Log Search Engine
Post by: qmech on April 24, 2014, 05:14:19 pm
I search through old posts and choose the one that maximizes:
   f(p) = p_irrelevance * p_irony / log(p_currentupvotes) + (AI_boredoflookingthroughposts)^500

I'd have thought that would be eventually increasing.
Title: Re: Log Search Engine
Post by: DStu on April 25, 2014, 02:55:15 am
Hooray for amalloy, who fixed the special characters issues in the log prettifier!  Now your username can end in "$" or "\" or whatever sort of regex-abusing tomfoolery you want: http://gokosalvager.com/static/logprettifier.html?20140424/log.5101a6c4e4b02b7235c3860f.1398355996250.txt

I suggest upvoting this random post of his (http://forum.dominionstrategy.com/index.php?topic=9217.msg284256#msg284256).
How do you do the randomization for the posts?

I search through old posts and choose the one that maximizes:
   f(p) = p_irrelevance * p_irony / log(p_currentupvotes) + (AI_boredoflookingthroughposts)^500
That does not look very random...
Title: Re: Log Search Engine
Post by: Qvist on June 12, 2014, 08:08:12 pm
I justed wanted to write the game summary of my game against qmech, but in the logsearch only 3 of the 6 games show up, weirdly all the games I won.
Any idea what is causing this?

http://gokosalvager.com/logsearch?p1name=Qvist&p1score=any&p2name=qmech&startdate=06%2F12%2F2014&enddate=06%2F12%2F2014&supply=&nonsupply=&rating=any&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0
Title: Re: Log Search Engine
Post by: ragingduckd on June 12, 2014, 08:12:18 pm
I justed wanted to write the game summary of my game against qmech, but in the logsearch only 3 of the 6 games show up, weirdly all the games I won.
Any idea what is causing this?

http://gokosalvager.com/logsearch?p1name=Qvist&p1score=any&p2name=qmech&startdate=06%2F12%2F2014&enddate=06%2F12%2F2014&supply=&nonsupply=&rating=any&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0

I'm not sure... another wrinkle from the database transition.  It's a separate issue from the one I identified in the "moving to Linode" thread.
Title: Re: Log Search Engine
Post by: ragingduckd on July 02, 2014, 03:55:54 am
(Cross-posted in the Isotropish thread (http://forum.dominionstrategy.com/index.php?topic=8900.msg394736#msg394736))

On Sunday (June 29), MakingFun's log server stopped responding to requests from gokosalvager.com.  Their log server itself still works fine, but it's specifically ignoring requests from my Linode instance.

It's possible that this is just their server automatically defending itself from what it thinks is an attack or just an overly-rapacious user.  I haven't received any notice about this from MF, nor have I received any complaints in the past.  I've asked about it on their forum (http://forum.makingfun.com/showthread.php?4896-gokosalvager-com-can-no-longer-access-MakingFun-s-Dominion-log-server) and will update you as I learn more.

I don't plan to anonymize or otherwise disguise my log server requests, not even as a temporary workaround.  If MF ever decides to thwart the log search, Isotropish, or the Salvager extension, they'd have a pretty easy time doing it, so there's not much sense in trying to defy them.  And I'd really like to have some sort of ongoing dialogue with MF anyway.

Until this is resolved, new games will not show up in the log search, and the Isotropish leaderboard will not update.  The "Iso" ratings displayed by the Salvager extension will not update either, but it's other features will continue working (including automatch).  The log prettifier and kingdom-image generator on still work, but you have to find the logs yourself.
Title: Re: Log Search Engine
Post by: ragingduckd on July 03, 2014, 02:14:56 am
On Sunday (June 29), MakingFun's log server stopped responding to requests from gokosalvager.com.  Their log server itself still works fine, but it's specifically ignoring requests from my Linode instance.

I got a brief reply from an MF admin that suggests that this was an accident.  They're using AWS, and it looks like Amazon cut my IP off because it was demanding so much data that they considered it abuse/attack.

I've modified my code to query for new games much less frequently, which I think will prevent this from happening again.  I've also asked MF if they can provide a less horrible way for me to query for new game logs.  Right now the only way is to get the whole 10MB list of all games that have been played today.

I've also asked MF to contact Amazon to remove the block on my IP, and I asked Linode how to get a different IP.  If/when either of those things happens, my database should start updating again.  If neither happens soon, I'll find a way to anonymize or redirect my log server requests instead.
Title: Re: Log Search Engine
Post by: amalloy on July 05, 2014, 03:38:34 pm
On Sunday (June 29), MakingFun's log server stopped responding to requests from gokosalvager.com.  Their log server itself still works fine, but it's specifically ignoring requests from my Linode instance.

I got a brief reply from an MF admin that suggests that this was an accident.  They're using AWS, and it looks like Amazon cut my IP off because it was demanding so much data that they considered it abuse/attack.

I've modified my code to query for new games much less frequently, which I think will prevent this from happening again.  I've also asked MF if they can provide a less horrible way for me to query for new game logs.  Right now the only way is to get the whole 10MB list of all games that have been played today.

I've also asked MF to contact Amazon to remove the block on my IP, and I asked Linode how to get a different IP.  If/when either of those things happens, my database should start updating again.  If neither happens soon, I'll find a way to anonymize or redirect my log server requests instead.

It looks like you've got the log-search engine working again. If so, you probably want to remove the header/disclaimer at http://www.gokosalvager.com/, no?
Title: Re: Log Search Engine
Post by: ragingduckd on July 05, 2014, 04:05:46 pm
It looks like you've got the log-search engine working again. If so, you probably want to remove the header/disclaimer at http://www.gokosalvager.com/, no?

Yup.  The leaderboard too.
Title: Re: Log Search Engine
Post by: yed on July 05, 2014, 04:08:10 pm
Maybe add this over logsearch too:

Last recorded game finished 5 min, 7 sec ago

Title: Re: Log Search Engine
Post by: ragingduckd on July 05, 2014, 04:38:16 pm
Maybe add this over logsearch too:

Last recorded game finished 5 min, 7 sec ago

Good call.  It runs a lot further behind now because I'm anonymizing my requests and because of the other changes I made to reduce my impact on their log server.

Obviously I reversed my earlier decision and am now anonymizing my requests.  I didn't want to do that when I thought that a) I might actually be serious messing with their server, and b) they might actually be willing to talk to me.  But I'm now pulling ~5 kB/s from them, while their total responses to me on their forum add up to about 200B, none of it substantive.  ::)
Title: Re: Log Search Engine
Post by: LibraryAdventurer on July 26, 2014, 10:42:03 pm
Whenever I try to use the log search on gokosalvager, the screen just goes blank. Is this just me?

EDIT: It only does this when I clear the date range. Even when it works, it isn't finding anything newer than 7/14
Title: Re: Log Search Engine
Post by: ragingduckd on July 27, 2014, 04:59:19 pm
Whenever I try to use the log search on gokosalvager, the screen just goes blank. Is this just me?

EDIT: It only does this when I clear the date range. Even when it works, it isn't finding anything newer than 7/14

The games I have for you in the database after 7/14 are all "casual" mode.  Change the log search's "Game: Rating" field to "casual" or "any" to see those.  Example search (http://gokosalvager.com/logsearch?p1name=LibraryAdventurer&p1score=any&p2name=&startdate=07%2F14%2F2014&enddate=07%2F27%2F2014&supply=&nonsupply=&rating=any&pcount=2&colony=any&bot=false&shelters=any&guest=false&minturns=&maxturns=&quit=false&resign=any&limit=20&submitted=true&offset=0)

If you've been playing pro games since 7/14 but they're not in the database, then something else is going on.  It may be on MF/Goko's side, as sometimes they just fail to create a log and there's nothing gokosalvager.com can do about it.  But if you:
1. play a pro game
2. and the log exists
3. and the game doesn't show up in the log search

... then please post links to the log and the failed search here and I'll try to track down the bug.
Title: Re: Log Search Engine
Post by: LibraryAdventurer on July 28, 2014, 11:56:58 pm
Oh. I didn't realize the search engine defaults to find only pro games. oops. I was thinking it would default to 'any', but for some reason I didn't bother actually looking at the setting...
I almost never play pro games now that I've bought the cards.
Title: Re: Log Search Engine
Post by: Holger on September 24, 2014, 02:13:19 pm
If you've been playing pro games since 7/14 but they're not in the database, then something else is going on.  It may be on MF/Goko's side, as sometimes they just fail to create a log and there's nothing gokosalvager.com can do about it.  But if you:
1. play a pro game
2. and the log exists
3. and the game doesn't show up in the log search

... then please post links to the log and the failed search here and I'll try to track down the bug.

The engine doesn't find any games that I played since (or before) New Year's Day (http://gokologs.drunkensailor.org/logsearch?p1name=Holger&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=09%2F24%2F2014&supply=&nonsupply=&rating=any&pcount=any&colony=any&bot=any&shelters=any&guest=any&minturns=&maxturns=&quit=any&resign=any&limit=20&submitted=true&offset=0); actually I'm not quite sure if I played those 5 games at all, but I played plenty since then.
I played mainly pro games and adventure games against bots, but not exclusively. E.g. this game (http://archive-dominionlogs.goko.com/20140911/log.509ed425e4b0e72a0b03f118.1410432883389.txt) can be found by searching for my opponent, but not under my name. The games also usually caused my Isotropish rating to update.
In case it matters, I never changed my username, only my computer. The few games shown give my name as "holger", while Isotropish uses "Holger" for whatever reason.

Edit: I recall the search engine working fine for me several months ago; unfortunately I don't remember if this was before or after January.
Title: Re: Log Search Engine
Post by: ragingduckd on September 28, 2014, 02:41:00 am
If you've been playing pro games since 7/14 but they're not in the database, then something else is going on.  It may be on MF/Goko's side, as sometimes they just fail to create a log and there's nothing gokosalvager.com can do about it.  But if you:
1. play a pro game
2. and the log exists
3. and the game doesn't show up in the log search

... then please post links to the log and the failed search here and I'll try to track down the bug.

The engine doesn't find any games that I played since (or before) New Year's Day (http://gokologs.drunkensailor.org/logsearch?p1name=Holger&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=09%2F24%2F2014&supply=&nonsupply=&rating=any&pcount=any&colony=any&bot=any&shelters=any&guest=any&minturns=&maxturns=&quit=any&resign=any&limit=20&submitted=true&offset=0); actually I'm not quite sure if I played those 5 games at all, but I played plenty since then.
I played mainly pro games and adventure games against bots, but not exclusively. E.g. this game (http://archive-dominionlogs.goko.com/20140911/log.509ed425e4b0e72a0b03f118.1410432883389.txt) can be found by searching for my opponent, but not under my name. The games also usually caused my Isotropish rating to update.
In case it matters, I never changed my username, only my computer. The few games shown give my name as "holger", while Isotropish uses "Holger" for whatever reason.

Edit: I recall the search engine working fine for me several months ago; unfortunately I don't remember if this was before or after January.

You, sir, are a sneaky one.  But I figured out your game:

Code: [Select]

 2012-12-08 10:34:20    "Holger "
 ...
 2013-12-18 05:02:42    "Holger "
 2013-12-31 01:17:26    "holger"
 ...
 2014-01-01 13:19:21    "holger"
 2014-01-02 11:46:15    "Holger "
 ...
 2014-09-26 11:18:48    "Holger "


So does your name end in a space, or is there something wrong with my log parsing?  And is "holger" a different player, or did you change your username for the 2014 new year?
Title: Re: Log Search Engine
Post by: Holger on September 28, 2014, 01:05:30 pm
If you've been playing pro games since 7/14 but they're not in the database, then something else is going on.  It may be on MF/Goko's side, as sometimes they just fail to create a log and there's nothing gokosalvager.com can do about it.  But if you:
1. play a pro game
2. and the log exists
3. and the game doesn't show up in the log search

... then please post links to the log and the failed search here and I'll try to track down the bug.

The engine doesn't find any games that I played since (or before) New Year's Day (http://gokologs.drunkensailor.org/logsearch?p1name=Holger&p1score=any&p2name=&startdate=08%2F05%2F2012&enddate=09%2F24%2F2014&supply=&nonsupply=&rating=any&pcount=any&colony=any&bot=any&shelters=any&guest=any&minturns=&maxturns=&quit=any&resign=any&limit=20&submitted=true&offset=0); actually I'm not quite sure if I played those 5 games at all, but I played plenty since then.
I played mainly pro games and adventure games against bots, but not exclusively. E.g. this game (http://archive-dominionlogs.goko.com/20140911/log.509ed425e4b0e72a0b03f118.1410432883389.txt) can be found by searching for my opponent, but not under my name. The games also usually caused my Isotropish rating to update.
In case it matters, I never changed my username, only my computer. The few games shown give my name as "holger", while Isotropish uses "Holger" for whatever reason.

Edit: I recall the search engine working fine for me several months ago; unfortunately I don't remember if this was before or after January.

You, sir, are a sneaky one.  But I figured out your game:

Code: [Select]

 2012-12-08 10:34:20    "Holger "
 ...
 2013-12-18 05:02:42    "Holger "
 2013-12-31 01:17:26    "holger"
 ...
 2014-01-01 13:19:21    "holger"
 2014-01-02 11:46:15    "Holger "
 ...
 2014-09-26 11:18:48    "Holger "


So does your name end in a space, or is there something wrong with my log parsing?  And is "holger" a different player, or did you change your username for the 2014 new year?

I never changed my username, AFAIR. There shouldn't be a space at the end of my username, but how could I check if there is? Goko wouldn't show a space, and it also wouldn't show a non-capital "H" since the usernames are given only in capital letters. The "H" does seems to be a pixel taller than the other letters of the name, but it's hard to tell...
I don't know if there's another "holger", either; I don't remember if I actually played those New Year's Day games or not, unfortunately.

On Isotropish, my games are given correctly under "Holger" (around level 39 currently) - but I wouldn't see a space at the end of the username either, obviously. I suppose you would?
Title: Re: Log Search Engine
Post by: ragingduckd on September 28, 2014, 02:59:50 pm
I never changed my username, AFAIR. There shouldn't be a space at the end of my username, but how could I check if there is? Goko wouldn't show a space, and it also wouldn't show a non-capital "H" since the usernames are given only in capital letters. The "H" does seems to be a pixel taller than the other letters of the name, but it's hard to tell...
I don't know if there's another "holger", either; I don't remember if I actually played those New Year's Day games or not, unfortunately.

On Isotropish, my games are given correctly under "Holger" (around level 39 currently) - but I wouldn't see a space at the end of the username either, obviously. I suppose you would?

You're listed as "Holger " with a space all throughout my database (games, rating, playerinfo).  So Goko definitely thinks there's a space there.

The search for ought to be able to handle a space at the end of the name, but something appears to be eating it.  I can't figure out why, but I'll try again later.

Anyone else with HTML and/or Python skills wants to look into it?
https://github.com/aiannacc/goko-dominion-tools/blob/master/gdt/logsearch/logsearch_handler.py
https://github.com/aiannacc/goko-dominion-tools/blob/master/web/logsearch.html
Title: Re: Log Search Engine
Post by: Holger on September 28, 2014, 03:34:45 pm
You're listed as "Holger " with a space all throughout my database (games, rating, playerinfo).  So Goko definitely thinks there's a space there.

That's strange. I can try and remove the space from my username, if that helps.
Title: Re: Log Search Engine
Post by: ragingduckd on September 28, 2014, 03:40:10 pm
You're listed as "Holger " with a space all throughout my database (games, rating, playerinfo).  So Goko definitely thinks there's a space there.

That's strange. I can try and remove the space from my username, if that helps.

Yes, that would work.
Title: Re: Log Search Engine
Post by: mpsprs on September 28, 2014, 03:48:21 pm
Anyone else with HTML and/or Python skills wants to look into it?
https://github.com/aiannacc/goko-dominion-tools/blob/master/gdt/logsearch/logsearch_handler.py
https://github.com/aiannacc/goko-dominion-tools/blob/master/web/logsearch.html

I don't have either skill, really, so feel free to ignore this.  But it looks to me like the self.get_argument in line 53 defaults to stripping white space from the beginning and and of the string.  Certainly this behavior is what I see.  I can add whitespace to the front and rear of my name and still find all my games.
Title: Re: Log Search Engine
Post by: Holger on September 28, 2014, 04:00:40 pm
You're listed as "Holger " with a space all throughout my database (games, rating, playerinfo).  So Goko definitely thinks there's a space there.

That's strange. I can try and remove the space from my username, if that helps.

Yes, that would work.

You're right about the space, it also appears when I open the "Edit Profile" box. Unfortunately I can't remove the space: "Player name already exists". So apparently there's another player with my name - can it be that "holger" whose logs are shown, or does Goko distinguish different capitalization and there's even another "Holger" around?
Title: Re: Log Search Engine
Post by: Holger on September 28, 2014, 04:06:01 pm
Anyone else with HTML and/or Python skills wants to look into it?
https://github.com/aiannacc/goko-dominion-tools/blob/master/gdt/logsearch/logsearch_handler.py
https://github.com/aiannacc/goko-dominion-tools/blob/master/web/logsearch.html

I don't have either skill, really, so feel free to ignore this.  But it looks to me like the self.get_argument in line 53 defaults to stripping white space from the beginning and and of the string.  Certainly this behavior is what I see.  I can add whitespace to the front and rear of my name and still find all my games.

It's the same for me, form what I see in the user interface. I can add spaces to the name (or in any other entry), but they're automatically removed even from the entry when I press "Search".
Title: Re: Log Search Engine
Post by: ragingduckd on September 29, 2014, 08:14:07 am
Anyone else with HTML and/or Python skills wants to look into it?
https://github.com/aiannacc/goko-dominion-tools/blob/master/gdt/logsearch/logsearch_handler.py
https://github.com/aiannacc/goko-dominion-tools/blob/master/web/logsearch.html

I don't have either skill, really, so feel free to ignore this.  But it looks to me like the self.get_argument in line 53 defaults to stripping white space from the beginning and and of the string.  Certainly this behavior is what I see.  I can add whitespace to the front and rear of my name and still find all my games.

Line 53:

Code: [Select]
arg_str[p] = self.get_argument(p, default=self.search_params[p])
This doesn't actually strip any whitespace directly, though the array name arg_stp sort of suggests that it might.

However, the self object is a RequestHandler from the third-party tornadoweb package I use.  And if you look at the package's srouce code, its get_argument() method delegates to an internal method:

Code: [Select]
    def _get_arguments(self, name, source, strip=True):
        values = []
        for v in source.get(name, []):
            v = self.decode_argument(v, name=name)
            if isinstance(v, unicode_type):
                # Get rid of any weird control chars (unless decoding gave
                # us bytes, in which case leave it alone)
                v = RequestHandler._remove_control_chars_regex.sub(" ", v)
            if strip:
                v = v.strip()
            values.append(v)
        return values

... and the v = v.strip() part does in fact strip whitespace!  So I don't know whether you misread my code or went the extra mile to find the problem, but either way thanks! :D

I implemented the fix, which was just a one-liner once I knew the problem.  Searching for "Holger " works properly now.
Title: Re: Log Search Engine
Post by: mpsprs on September 29, 2014, 05:17:12 pm
So I don't know whether you misread my code or went the extra mile to find the problem, but either way thanks! :D

I did dig into tornadoweb.  I never found the full def you found (where it deals with the strip=True flag), but I did see the fact that there was a strip=True default parameter, and made an intelligent guess. 

I still don't have any real skills at actually coding.  Debugging I'm probably better at.  Maybe if I find lots of time I'll learn the coding side.
Title: Re: Log Search Engine
Post by: yed on September 30, 2014, 11:47:27 am
does Goko distinguish different capitalization and there's even another "Holger" around?
Goko saves the capitalization but does not allow two usernames, which are different only in the capitalization.
Title: Re: Log Search Engine
Post by: Holger on October 01, 2014, 06:46:18 am
I implemented the fix, which was just a one-liner once I knew the problem.  Searching for "Holger " works properly now.

Thank you very much, it does indeed.

does Goko distinguish different capitalization and there's even another "Holger" around?
Goko saves the capitalization but does not allow two usernames, which are different only in the capitalization.

How stupid is that? Especially if they allow two usernames that vary only in an invisible space at the end, which are totally indistinguishable in the lobby.
Title: Re: Log Search Engine
Post by: ThaddeusB on October 01, 2014, 12:40:23 pm
does Goko distinguish different capitalization and there's even another "Holger" around?
Goko saves the capitalization but does not allow two usernames, which are different only in the capitalization.

How stupid is that? Especially if they allow two usernames that vary only in an invisible space at the end, which are totally indistinguishable in the lobby.

Being case insensitive is smart. Failing to ignore whitespace ("user name" and "username" should be treated the same) is stupid.  Allowing meaningful whitespace at the end of a name is exceptionally stupid. The reason is the same in all cases - to avoid confusion.
Title: Re: Log Search Engine
Post by: LastFootnote on March 09, 2015, 01:54:23 pm
So is the log search permanently down, or just temporarily down?
Title: Re: Log Search Engine
Post by: amalloy on March 09, 2015, 01:58:38 pm
So is the log search permanently down, or just temporarily down?

Working fine for me at this very moment. Are you going to some old URL instead of to gokosalvager.com?
Title: Re: Log Search Engine
Post by: LastFootnote on March 09, 2015, 02:04:13 pm
So is the log search permanently down, or just temporarily down?

Working fine for me at this very moment. Are you going to some old URL instead of to gokosalvager.com?

Yep, thanks.