Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[DStu]

Does anybody has (more or less) completely downloaded the gamelogs from goko, at least the last month or so and want to forward them to me in zipped from?  I want to do some statistics, but it takes ages to scrape the unzipped logs from their server...

[DStu]

Alternatively, is there a faster way than piping all the filenames into wget?
I get something like 50kb/s download with this, getting 800MB files is no fun that way.

[MrHaas]

I made a script which downloaded them in multiple threads which did about 150 logs/s. Didn't finish b/c my VPS ran out of space. This week they archived the old dates in zipped tarballs and I started to download them, but when I reached 18 feb. (about 15 minutes into the process) they removed the old files and now only the logs from april are there as you can see. So I'm missing half of february and march .

[ragingduckd]

I made a script which downloaded them in multiple threads which did about 150 logs/s.

How did you manage this?  I can't seem to do better than 10/s.

[ragingduckd]

I zipped up the currently-available Goko log files and put them here: http://www.sendspace.com/folder/u5w9ig

MrHaas, can you send me the logs from February and before?

[nutki]

Here is a script to grab the logs quite fast. With 20 threads I got to about 200logs/s. The logs will be saved gzipped, so you have to unpack them or (to save space) use 'zcat' when needed.

Code: [Select]

#!/bin/bash
BASE=http://dominionlogs.goko.com/20130513/
THREADS=20
wget --header='Accept-Encoding: gzip' $BASE -O- |
zcat |
perl -lne 'print "$1" if /href="(.*?txt)"/' |
split -l 100 - _index.
ls _index.* | xargs -n 1 -P $THREADS wget --base=$BASE --header='Accept-Encoding: gzip' -i
rm _index.*


[Watno]

All the logs from before April are available in archives at http://dominionlogs.goko.com/

[yed]

They changed that:
New logs: http://dominionlogs.goko.com/
Old logs: http://archive-dominionlogs.goko.com/

Missing old logs redirects to archive. Example:
http://dominionlogs.goko.com/20120805/log.4fe8b232785d0b08dee907fa.1344196906203.txt
redirects to
http://archive-dominionlogs.goko.com/20120805/log.4fe8b232785d0b08dee907fa.1344196906203.txt