Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Hugovj]

I just changed my username (hugovj->Hugovj), but the leaderboard shows both. I don't know how that can be fixed? (Can it?) Weirdly, in the lobby I see my level as 14, while on the leaderboard it says -7 (for the Hugovj with capital letter).

[ragingduckd]

I just changed my username (hugovj->Hugovj), but the leaderboard shows both. I don't know how that can be fixed? (Can it?) Weirdly, in the lobby I see my level as 14, while on the leaderboard it says -7 (for the Hugovj with capital letter).

This ugly problem arose from Goko's decision to not put unique player-identifying information in their logs.  There are ways for us to deal with it, as briefly discussed here, but honestly I don't think anyone's likely to bother.

I'll implement a manual workaround for your username, so that the games you played as "hugovj" will be considered part of your record as "Hugovj."  This will take effect next time I do a full rebuild of the leaderboard.

FYI, your "level" isn't the same as Isotropish's best guess of your ability.  That's what "mu" is.  There are legitimate (though debatable) reasons for using level instead of mu for experienced players, but it heavily penalizes brand new players.

Your performance as "Hugovj" has actually been considerably stronger than your performance as "hugovj."  If you keep up your recent quality of play as Hugovj, your level will quickly surpass hugovj's level 14.

[Hugovj]

Thanks a lot 

[serakfalcon]

I just changed my username (hugovj->Hugovj), but the leaderboard shows both. I don't know how that can be fixed? (Can it?) Weirdly, in the lobby I see my level as 14, while on the leaderboard it says -7 (for the Hugovj with capital letter).

This ugly problem arose from Goko's decision to not put unique player-identifying information in their logs.  There are ways for us to deal with it, as briefly discussed here, but honestly I don't think anyone's likely to bother.

I'm working on it! Eventually!

[soulnet]

It seems that both log search and Isotropish have stopped working. Last recorded game according to Isotropish (at the moment) is more than 2 hours ago.

[ragingduckd]

It seems that both log search and Isotropish have stopped working. Last recorded game according to Isotropish (at the moment) is more than 2 hours ago.

Dealing with this and some other issues by moving to Linode (finally!).

[ragingduckd]

I've fixed the gokosalvager.com bug that was making games go missing.  Both the log search and the leaderboard appear to be correct now.

Unfortunately, the MF/Goko that causes some logs not to be generated at all still remains, and they seem to be dropping quite a lot of games.  If you can find the original log file, but it doesn't show up in search, please post here, as it means that something is wrong on my side.  If the log file is missing too, it's just Goko being Goko.

FYI, Goko lists each day's logs at an address like http://dominionlogs.goko.com/20140617/.  Games that you host will be prefixed by "log.[your player id]" (e.g. "log.5101a6c4e4b02b7235c3860f.1402894381446.txt").

[warlordprias]

A few anomalies have occurred. These are some ratings I'm currently seeing. Tuff and Marin have 50k+ games played. While I'm sure they both like Dominion, not sure they really played that many.
Ballsy -108
Tuff_Gong 56
Martin Dambachmair 99
Shihao Koh -179

[ragingduckd]

A few anomalies have occurred. These are some ratings I'm currently seeing. Tuff and Marin have 50k+ games played. While I'm sure they both like Dominion, not sure they really played that many.
Ballsy -108
Tuff_Gong 56
Martin Dambachmair 99
Shihao Koh -179

This is a known bug.  I've put some logging in place that may help me catch it next time it happens.  FWIW, the overall effect on the leaderboard is pretty minimal, except for the lucky few who get massively overrated (temporarily).

[ragingduckd]

The Isotropish leaderboard won't be updating until further notice.  As of yesterday, my Linode instance can no longer access the MF/Goko log server.

From home, I'm able to talk to their server just fine:

Code: [Select]

iron> ping -q -c 10 google.com
PING google.com (74.125.224.200) 56(84) bytes of data.

--- google.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9014ms
rtt min/avg/max/mdev = 12.021/15.808/44.671/9.665 ms

But from Linode, my requests just time out:

Code: [Select]

li566-22> ping logs.prod.dominion.makingfun.com
PING ec2-54-213-198-64.us-west-2.compute.amazonaws.com (54.213.198.64) 56(84) bytes of data.
^C
--- ec2-54-213-198-64.us-west-2.compute.amazonaws.com ping statistics ---
435 packets transmitted, 0 received, 100% packet loss, time 433999ms

The same happens when I request specific log files using 'curl' or 'wget'.

Anyone have any insight on why this might happen?  Is the most likely thing that they've specifically cut off service to my Linode instance?

Assuming that's the case, can anyone guess who I'd have to contact to resolve that?  There's a decent chance that I'm just doing my log requests inefficiently and in a way that demands more of their server than is really necessary.  It may be that I could fix that to everyone's satisfaction if I could make contact with someone tech-savvy on their end.

[amalloy]

Anyone have any insight on why this might happen?  Is the most likely thing that they've specifically cut off service to my Linode instance?

That'd be my guess. Which linode datacenter are you in? I can ping that server just fine from a linode in Fremont, so it's not a linode-wide issue (which would be pretty weird, but plausible).

Assuming that's the case, can anyone guess who I'd have to contact to resolve that?  There's a decent chance that I'm just doing my log requests inefficiently and in a way that demands more of their server than is really necessary.  It may be that I could fix that to everyone's satisfaction if I could make contact with someone tech-savvy on their end.

https://github.com/aiannacc/goko-dominion-tools/blob/f0ae8522af6151e4a459eb49dc820c00931c4729/start_logwatcher.py#L81-L97 does look pretty rude: hitting them with 40 new connections all at once. Isn't there a python HTTP client that supports Connection: Keep-Alive and/or pipelining? It looks like urlgrabber, urllib2, httplib2, and pycurl may all support this sort of thing; but I don't have the python experience to judge which of them are actually any good.

Of course I don't know who on their end you'd have to talk to, but if they really are blocking you I'm kinda amazed they decided to do so without even contacting you. If you end up getting your networking code a little more polite but can't convince makingfun to let you back in, you could try connecting over tor.

[yed]

Anyone have any insight on why this might happen?  Is the most likely thing that they've specifically cut off service to my Linode instance?

Try

Code: [Select]

traceroute logs.prod.dominion.makingfun.com
to determine if it is not a network issue.

[yed]

Also can you connect to archive-dominionlogs.goko.com ?

That is a different server.

[ragingduckd]

https://github.com/aiannacc/goko-dominion-tools/blob/f0ae8522af6151e4a459eb49dc820c00931c4729/start_logwatcher.py#L81-L97 does look pretty rude: hitting them with 40 new connections all at once. Isn't there a python HTTP client that supports Connection: Keep-Alive and/or pipelining? It looks like urlgrabber, urllib2, httplib2, and pycurl may all support this sort of thing; but I don't have the python experience to judge which of them are actually any good.

Of course I don't know who on their end you'd have to talk to, but if they really are blocking you I'm kinda amazed they decided to do so without even contacting you. If you end up getting your networking code a little more polite but can't convince makingfun to let you back in, you could try connecting over tor.

Whether that's the problem or not, I would definitely appreciate some help in making my connections more "polite."  I'm sure that the Python Requests module supports all the basic protocols and header options, but honestly I haven't a clue what I'm doing with this stuff.  Like I have it doing 40 requests at once because that was the only way I could think to deal with the long delays (often 30 seconds or more) when I request individual log files.

Could you point me towards a useful reference for understanding how to do polite and efficient http requests in high(-ish) volume?  It doesn't have to be Python-specific.

I realize that Tor is an option, but I'd like to try to civil route first.  I'm also surprised to be cut off without being contacted.  But that's assuming that a human made the decision.  Maybe their server defends itself from attacks by automatically cutting off abusive requesters?

Also can you connect to archive-dominionlogs.goko.com ?

That is a different server.

Yes, I can still access that server from Linode.  But it runs about 2 days behind on the game logs.  The only source I know that provides logs in real time is logs.prod.dominion.makingfun.com (the same server as dominionlogs.goko.com).

Try

Code: [Select]

traceroute logs.prod.dominion.makingfun.com
to determine if it is not a network issue.

Ok.  I haven't used traceroute before though.  Does this output mean anything to you?

Code: [Select]

li566-22> traceroute logs.prod.dominion.makingfun.com
traceroute to logs.prod.dominion.makingfun.com (54.213.198.64), 30 hops max, 60 byte packets
 1  23.92.24.3 (23.92.24.3)  0.377 ms  0.522 ms  0.638 ms
 2  10ge8-3.core3.fmt2.he.net (64.71.132.137)  0.153 ms  0.143 ms  0.146 ms
 3  10ge10-1.core1.sjc2.he.net (184.105.222.14)  12.244 ms  12.202 ms  12.149 ms
 4  216.218.193.42 (216.218.193.42)  0.678 ms  0.691 ms  0.656 ms
 5  205.251.229.155 (205.251.229.155)  0.678 ms 205.251.229.157 (205.251.229.157)  0.682 ms 205.251.229.155 (205.251.229.155)  0.703 ms
 6  205.251.232.68 (205.251.232.68)  30.317 ms 205.251.232.112 (205.251.232.112)  22.153 ms 205.251.232.68 (205.251.232.68)  29.517 ms
 7  205.251.232.147 (205.251.232.147)  22.359 ms 205.251.232.153 (205.251.232.153)  22.272 ms 205.251.232.141 (205.251.232.141)  22.262 ms
 8  205.251.232.165 (205.251.232.165)  22.997 ms  22.948 ms 205.251.232.63 (205.251.232.63)  22.938 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *


[amalloy]

Whether that's the problem or not, I would definitely appreciate some help in making my connections more "polite."  I'm sure that the Python Requests module supports all the basic protocols and header options, but honestly I haven't a clue what I'm doing with this stuff.  Like I have it doing 40 requests at once because that was the only way I could think to deal with the long delays (often 30 seconds or more) when I request individual log files.

Could you point me towards a useful reference for understanding how to do polite and efficient http requests in high(-ish) volume?  It doesn't have to be Python-specific.

Keepalive and pipelining are designed to cut down on the number of TCP round-trips between client and server. Keepalive is supported by all HTTP 1.1 servers (including goko's), and works by requesting multiple files without having to repeat the TCP handshake for each (because it reuses the same connection). Pipelining is built on top of this, and works by pre-sending a bunch of HTTP requests over the same connection before receiving any response, letting the server queue them all up and process them rapid-fire rather than waiting for your client to make the next request.

These should help if the time it takes to download a log is based mainly on round-trip time, but if transferring a single file takes a long time this won't help much. Some poking around with curl suggests to me that with one TCP connection per file, it takes 0.2s to download a log; with keepalive over 100 log files I can get 0.1s. curl doesn't support pipelining, but if it did I think we'd make it down to 0.05s by halving the number of round trips again. That might be fast enough that being singlethreaded is fine, I would think, but you could do it on a small handful of threads if necessary.

Ok.  I haven't used traceroute before though.  Does this output mean anything to you?

Code: [Select]

li566-22> traceroute logs.prod.dominion.makingfun.com
traceroute to logs.prod.dominion.makingfun.com (54.213.198.64), 30 hops max, 60 byte packets
 1  23.92.24.3 (23.92.24.3)  0.377 ms  0.522 ms  0.638 ms
 2  10ge8-3.core3.fmt2.he.net (64.71.132.137)  0.153 ms  0.143 ms  0.146 ms
 3  10ge10-1.core1.sjc2.he.net (184.105.222.14)  12.244 ms  12.202 ms  12.149 ms
 4  216.218.193.42 (216.218.193.42)  0.678 ms  0.691 ms  0.656 ms
 5  205.251.229.155 (205.251.229.155)  0.678 ms 205.251.229.157 (205.251.229.157)  0.682 ms 205.251.229.155 (205.251.229.155)  0.703 ms
 6  205.251.232.68 (205.251.232.68)  30.317 ms 205.251.232.112 (205.251.232.112)  22.153 ms 205.251.232.68 (205.251.232.68)  29.517 ms
 7  205.251.232.147 (205.251.232.147)  22.359 ms 205.251.232.153 (205.251.232.153)  22.272 ms 205.251.232.141 (205.251.232.141)  22.262 ms
 8  205.251.232.165 (205.251.232.165)  22.997 ms  22.948 ms 205.251.232.63 (205.251.232.63)  22.938 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *


Your ping is getting as far as 205.251.232.165, which is an Amazon server in Seattle (according to http://www.tcpiputils.com/browse/ip-address/205.251.232.165), and then being dropped. Since apparently the server you're requesting is an ec2 server and you made it to amazon before being ignored, I'd say it's the goko server itself rejecting you.

[ragingduckd]

...

Thank you.  That's all very helpful.

[ragingduckd]

(Cross-posted in the Log Search thread)

On Sunday (June 29), MakingFun's log server stopped responding to requests from gokosalvager.com.  Their log server itself still works fine, but it's specifically ignoring requests from my Linode instance.

It's possible that this is just their server automatically defending itself from what it thinks is an attack or just an overly-rapacious user.  I haven't received any notice about this from MF, nor have I received any complaints in the past.  I've asked about it on their forum and will update you as I learn more.

I don't plan to anonymize or otherwise disguise my log server requests, not even as a temporary workaround.  If MF ever decides to thwart the log search, Isotropish, or the Salvager extension, they'd have a pretty easy time doing it, so there's not much sense in trying to defy them.  And I'd really like to have some sort of ongoing dialogue with MF anyway.

Until this is resolved, new games will not show up in the log search, and the Isotropish leaderboard will not update.  The "Iso" ratings displayed by the Salvager extension will not update either, but it's other features will continue working (including automatch).  The log prettifier and kingdom-image generator on still work, but you have to find the logs yourself.

[ragingduckd]

Keepalive and pipelining are designed to cut down on the number of TCP round-trips between client and server. Keepalive is supported by all HTTP 1.1 servers (including goko's), and works by requesting multiple files without having to repeat the TCP handshake for each (because it reuses the same connection). Pipelining is built on top of this, and works by pre-sending a bunch of HTTP requests over the same connection before receiving any response, letting the server queue them all up and process them rapid-fire rather than waiting for your client to make the next request.

This is great!  Keepalive more than doubles the rate at which I download individual log files.  More importantly, none of my requests time out anymore, even from my home machine.  That was what was making my log downloader spaz out and stop fetching new games so that you guys would have to bug me about it and I'd have to go restart the script.

If I understand correctly, pipelining is the better way to do what I've been doing by sending simultaneous requests in multiple threads.  I haven't yet figured out how to do it with the Python Requests module though.  Instead, I reduced the number of threads I'm using from 40 to 10.  I could drop it further without much loss if even 10 simultaneous requests sounds like it might be considered "rude."  With a single thread, it can download about a day's worth of logs in an hour, which is plenty fast enough as long as I'm not catching up from being months behind or anything.

I've also reduced the rate at which I check for new logs to once every every 60 seconds.  I think that's actually what their server was unhappy with me about, since it has to send me several megs (or several-hundred kB compressed) of redundant data every time I requested the list of today's games.  And given that it takes them ~7s to send that list, they either have the machine throttled down to 1990s internet speeds or it's doing something stupidly resource-intensive to generate it.

So now there will be up to a 1 min delay between when you finish your game and when gokosalvager.com adds it to the search database and updates your Isotropish rating (sorry).  Given what a nuisance this has turned out to be, I really don't want to risk it happening again.

[2.71828.....]

So now there will be up to a 1 min delay between when you finish your game and when gokosalvager.com adds it to the search database and updates your Isotropish rating (sorry).  Given what a nuisance this has turned out to be, I really don't want to risk it happening again.

What?! Wait up to one whole minute for my game log and for my rating to update?!  I don't know if I can handle this. 

[shark_bait]

http://forum.dominionstrategy.com/index.php?topic=1204.msg340225;topicseen#msg340225

In the golden days you woke up to the update. 

[Joseph2302]

The Isotropish leaderboard isn't currently working, hasn't been updated for 5 hours, and the Iso ratings aren't being displayed.

[ragingduckd]

The Isotropish leaderboard isn't currently working, hasn't been updated for 5 hours, and the Iso ratings aren't being displayed.

Fixed... probably and eventually.  It could take quite a while to catch up.  Now that I have to anonymize my log file requests, downloading logs is a lot slower.

MakingFun still hasn't responded to my post on their tech support forum, but on the other hand, they did send me a birthday email!

From: Making Fun Forums <help@makingfun.com>
Date: Wed, Jul 9, 2014 at 5:12 PM
Subject: Happy Birthday from Making Fun Forums
To: andrewiemail@gmail.com


Hello ragingduckd,

We at Making Fun Forums would like to wish you a happy birthday today!

[ragingduckd]

The Isotropish leaderboard isn't currently working, hasn't been updated for 5 hours, and the Iso ratings aren't being displayed.

Fixed... probably and eventually.  It could take quite a while to catch up.  Now that I have to anonymize my log file requests, downloading logs is a lot slower.

Not that time... I had a bug that didn't show up until it had downloaded quite a few logs.  Might be fixed this time, but don't trust the "last game recorded" stat right now.  I'm actually further behind than it says.

Edit: It really is all caught up now.

[Titandrake]

Leaderboard marks me as level 42, but I'm exactly level 43.

I can has >= check?

[ragingduckd]

Leaderboard marks me as level 42, but I'm exactly level 43.

I can has >= check?

In the database you're mu=52.9957 and sigma=3.3339, so I'm sure it's some subtlety about where the rounding and truncation happens.

Nice job hitting the range where the website and the extension disagree!  It can't be very wide.