| Author |
Topic |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 04:25:00 - [ 1]
I got really tired of EVE making a billion log files in it's log directories, so I wanted to condense them so windows could open those directories without choking on the sheer number of files. So I wrote a couple of bash scripts that will aggregate all those files into a much smaller number of them, completely dynamically. The Gamelogs are simply appended to one aggregate file in order by date modified. So they should end up in chronological order as long as you have not manually edited any of them. The chat logs are aggregated the same way, except individually by channel name. If you're running Windows you can install Cygwin and then run them. If you're running *nix, you probably don't need instructions. To "install" the scripts save the contents of each file as designated by the "author" to a scripts directory in your home directory in cygwin. Then add that directory to your $PATH. To use them, start up your shell and cd to your C:\Documents and Settings\<username>\My Documents\EVE\logs\<logdir> directory and run the appropriate script. For Chatlogs run aggregatechatlogs.sh. For Gamelogs run aggregategamelogs.sh. They may take several minutes to run if you have several thousand log files like I did. Be patient. I make no warranties as to the function of these scripts. And I take no responsibilities if you somehow hose your system with them. Use at your own risk.  Originally by: aggregategamelogs.sh
#!/bin/bash
for logfile in $(ls -lt -r *_*.txt | sed -e 's/ \+/ /g' | cut -f 9 -d " "); do
nice -20 cat $logfile >> Aggregate_Gamelogs.txt
nice -20 rm $logfile
done
Originally by: aggregatechatlogs.sh
#!/bin/bash
aggregatechattype() {
type=$1
for logfile in $(ls -lt -r "$type"_* | sed -e 's/ \+/ /g' | cut -f 9 -d " "); do
nice -20 cat $logfile >> Aggregate_"$type"_Log.txt
nice -20 rm $logfile
done
}
for type in $(ls *.txt -1 | grep -v Aggregate | sed -r -e 's/(.+)_[0-9]{8,8}_[0-9]{6,6}\.txt/\1/' | sort | uniq); do
echo "Aggegating $type logs"
aggregatechattype $type
done
|
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 04:26:00 - [ 2]
Reserved for the inevitable questions. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 04:27:00 - [ 3]
Reserved for the inevitable questions. |
Adunh Slavy
Ammatar Trade Syndicate
|
Posted - 2008.02.24 06:12:00 - [ 4]
Could try a batch too,
for /f %%a in ('dir /b') do type %%a >>logs.new
del *.txt
ren logs.new logs.%random%.txt
Don't need cygwin and all that. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 06:23:00 - [ 5]
Originally by: Adunh Slavy
Could try a batch too,
for /f %%a in ('dir /b') do type %%a >>logs.new
del *.txt
ren logs.new logs.%random%.txt
Don't need cygwin and all that.
That does not accomplish the same thing. You get all your log files in one, new randomly named file. Subsequent executions append that new file at random to the next random one. In other words, it makes a mess of your log files. Mine creates better organiztion. i.e. you get something like this: Aggregate_Corp_Log.txt Aggregate_Fleet_Log.txt Aggregate_Gang_Log.txt Aggregate_Local_Log.txt Aggregate_Private_Chat_Log.txt Aggregate_Squad_Log.txt Without ever having to specify names, or get random log file names. Subsequent executions append new files to the existing aggregate ones in the order they were created (so logs end up in correct, chronological order). |
Wet Ferret |
Posted - 2008.02.24 07:55:00 - [ 6]
wtf with your sig, Xaen? |
Adunh Slavy
Ammatar Trade Syndicate
|
Posted - 2008.02.24 07:59:00 - [ 7]
True, it is not the same thing, so much as illustrating that you don't need bash. Not saying it is bad, just not necessary. The DIR argument of FOR could take globs, could use nested FOR to iterate a list of prefixes, etc. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 08:17:00 - [ 8]
Originally by: Wet Ferret
wtf with your sig, Xaen?
Tired of waiting for them to hire a hire a UI person. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 08:19:00 - [ 9]
Originally by: Adunh Slavy
True, it is not the same thing, so much as illustrating that you don't need bash. Not saying it is bad, just not necessary. The DIR argument of FOR could take globs, could use nested FOR to iterate a list of prefixes, etc.
You do need bash. Or some other more powerful language than batch/dos commands to accomplish what my script does. If you want a huge mess, yes batch files will work. |
Adunh Slavy
Ammatar Trade Syndicate
|
Posted - 2008.02.24 10:07:00 - [ 10]
Originally by: Xaen
If you want a huge mess, yes batch files will work.
With out cheating by using resource kit toys and sed compiled for windows, you win :) |
flashfreaking
LFC
Executive Outcomes |
Posted - 2008.02.24 10:58:00 - [ 11]
your script nukes the forum, please, make linkies, or make the font smaller.
Thank you |
Erotic Irony
0bsession
|
Posted - 2008.02.24 14:12:00 - [ 12]
Edited by: Erotic Irony on 24/02/2008 14:13:07 Originally by: Wet Ferret
wtf with your sig, Xaen?
Xaen, have you thought about making your sig a jpg of one of eve's modal windows, "Support UI Repair"[Ok]edit: I've been logged out twice while trying to post this, surprise surprise. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 17:35:00 - [ 13]
Originally by: flashfreaking
your script nukes the forum, please, make linkies, or make the font smaller.
Thank you
I can't make the font smaller. That's what the code tag gives you. Links over complicate things. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.24 17:36:00 - [ 14]
Originally by: Erotic Irony
Edited by: Erotic Irony on 24/02/2008 14:13:07
Originally by: Wet Ferret
wtf with your sig, Xaen?
Xaen, have you thought about making your sig a jpg of one of eve's modal windows,
"Support UI Repair"
[Ok]
edit: I've been logged out twice while trying to post this, surprise surprise.
Sadly, I don't think anyone would get it. |
Vanora Kime
Caldari
Reapage Inc.
|
Posted - 2008.02.25 15:37:00 - [ 15]
I like your idea, very useful. I actually didn't even dare to open one of the log directories, as it would have taken hours to just show me the files ... I put both your scripts in a single one and made it a bit shorter, the result is the same: Quote:
#!/bin/bash
# Gamelogs:
GAMELOG_DIR="/mnt/sata2/Eigene Dateien/eve/logs/Gamelogs"
cd "$GAMELOG_DIR" && find . -name "*.txt" -exec cat "{}" >> ./gamelogs.log ";" -exec rm "{}" ";"
# Chatlogs, one file for each chat window:
CHATLOG_DIR="/mnt/sata2/Eigene Dateien/eve/logs/Chatlogs"
cd "$CHATLOG_DIR" && for type in $(ls *.txt | sed -r -e 's/(.+)_[0-9]{8,8}_[0-9]{6,6}\.txt/\1/' $
find . -name "$type*.txt" -exec cat "{}" >> "$type.log" ";" -exec rm "{}" ";"
done
I've put this into a cronjob that runs every night, no more billions of logfiles for me, yay! Thanks for this one :-) |
Garat Mant
Moons of Pluto
Space Exploration and Logistic Services |
Posted - 2008.02.25 15:51:00 - [ 16]
At work, so no access to logfiles - do chatlogs have your play start and end timestamps in them? i.e. with some careful parsing, could you get aggregate hours played by week/month? |
An Anarchyyt
Gallente
Battlestars
GoonSwarm |
Posted - 2008.02.25 15:53:00 - [ 17]
Horizontal scroll is much worse than any logfiles.
Also, I hope this counts as modding Eve and you get banned. |
Lil'Red Ridin'Hood
FinFleet
KenZoku |
Posted - 2008.02.25 16:05:00 - [ 18]
When I get home, I'll build a small executable doing the same.
I hope you don't mind if I copy your idea for that.
No need for cygwin then. Select log-folder, click run, cross your fingers, done. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.25 17:02:00 - [ 19]
Originally by: Vanora Kime
I like your idea, very useful. I actually didn't even dare to open one of the log directories, as it would have taken hours to just show me the files ... I put both your scripts in a single one and made it a bit shorter, the result is the same:
I've put this into a cronjob that runs every night, no more billions of logfiles for me, yay! Thanks for this one :-)
You're welcome. But careful running find. The scripts I wrote sort by modification date so that the files are appended in chronological order. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.25 17:15:00 - [ 20]
Originally by: Garat Mant
At work, so no access to logfiles - do chatlogs have your play start and end timestamps in them? i.e. with some careful parsing, could you get aggregate hours played by week/month?
Yes they do. But I hear using the logserver is a more effective way to get time played. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.25 17:15:00 - [ 21]
Originally by: An Anarchyyt
Horizontal scroll is much worse than any logfiles.
Also, I hope this counts as modding Eve and you get banned.
You've got to be kidding me. |
Kazuo Ishiguro
House of Marbles
|
Posted - 2008.02.25 17:19:00 - [ 22]
If you leave the log window open ingame, gamelogs get lumped into one file until you close it; you don't get a fresh log after each session change. I've occasionally found this useful.
|
Lil'Red Ridin'Hood
FinFleet
KenZoku |
Posted - 2008.02.27 22:26:00 - [ 23]
Originally by: Lil'Red Ridin'Hood
When I get home, I'll build a small executable doing the same.
I hope you don't mind if I copy your idea for that.
No need for cygwin then. Select log-folder, click run, cross your fingers, done.
I haven't forgotten about this, only had RL get in the way a bit. I'll do my best to have a version ready by the end of the weekend. I'd hope to have one ready before that, actually. |
AndrewRyan
Amarr
Imperial Academy
|
Posted - 2008.02.27 22:28:00 - [ 24]
Bishy Bashi boshi! |
Scilent Enigma
Minmatar
Total Mayhem.
Cry Havoc. |
Posted - 2008.02.27 22:41:00 - [ 25]
I dunno.. I've only found one batch script useful for the log files as to date, and that is: Quote:
del /Q "logs\Chatlogs\*.*"
del /Q "logs\Gamelogs\*.*"
del /Q "logs\Marketlogs\*.*"
Placed in a bat file in the EVE directory in My Documents... Though might give cygwin a try, sounds fun being able to scrip bash in windows... |
Maglorre |
Posted - 2008.02.28 04:59:00 - [ 26]
Edited by: Maglorre on 28/02/2008 05:00:40
Nice idea... couple of minor nit-picky things with the scripts people have posted.
[stupid bit removed, I misread the initial script]
None of the suggested scripts provide proper error testing before deleting the original files. If your append command fails for some reason then the original file is still trashed.
|
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.28 22:21:00 - [ 27]
Originally by: Maglorre
None of the suggested scripts provide proper error testing before deleting the original files.
The list of files to delete is generated based on files that exist. The only way this could be a problem is if you start the script the delete one of the log files while it's aggregating that type, but before it gets to that file. And then it will just throw an error and do the rest of them. Originally by: Maglorre
If your append command fails for some reason then the original file is still trashed.
True, but you'd have to run out of disk space or some other catastrophic problem. And it's not very likely since it deletes them as it goes. The resulted aggregated files should take up the same amount of space on disk. So for in order for it to fail you'd have to have one enormous file be the first one aggregated and very little disk space left. |
Xaen
Caldari
Sniggerdly
Pandemic Legion |
Posted - 2008.02.28 22:25:00 - [ 28]
Originally by: Scilent Enigma
I dunno.. I've only found one batch script useful for the log files as to date, and that is:
Quote:
del /Q "logs\Chatlogs\*.*"
del /Q "logs\Gamelogs\*.*"
del /Q "logs\Marketlogs\*.*"
Placed in a bat file in the EVE directory in My Documents...
Though might give cygwin a try, sounds fun being able to scrip bash in windows...
After working in *nix so long I can't live in windows without Cygwin. I use it almost daily. Some asshat uploads 50 video files in 50 individual .rar files? (mp3s/vidoes rarely compress appreciably) nice -20 for x in $(ls *.rar); do rar -x "$x"; done & And walk off to the living room to stream the first one to XBMC. ;) |
Scilent Enigma
Minmatar
Total Mayhem.
Cry Havoc. |
Posted - 2008.02.29 00:11:00 - [ 29]
Originally by: Xaen
After working in *nix so long I can't live in windows without Cygwin. I use it almost daily.
Some asshat uploads 50 video files in 50 individual .rar files? (mp3s/vidoes rarely compress appreciably) nice -20 for x in $(ls *.rar); do rar -x "$x"; done & And walk off to the living room to stream the first one to XBMC. ;)
Ahh.. the joys of *nix.. I really should learn it better, just wished I wasn't so stuck with all the programs in windows. Maybe in my new computer I will be able to install it and VMware Vista when needed.. |
Maglorre |
Posted - 2008.02.29 01:45:00 - [ 30]
Originally by: Xaen
The list of files to delete is generated based on files that exist. The only way this could be a problem is if you start the script the delete one of the log files while it's aggregating that type, but before it gets to that file. And then it will just throw an error and do the rest of them.
Yeah, that's not an issue. Originally by: Xaen
Originally by: Maglorre
If your append command fails for some reason then the original file is still trashed.
True, but you'd have to run out of disk space or some other catastrophic problem. And it's not very likely since it deletes them as it goes. The resulted aggregated files should take up the same amount of space on disk. So for in order for it to fail you'd have to have one enormous file be the first one aggregated and very little disk space left.
Well, you are correct that it's not likely but that's no excuse to not implement correct error handling  . Just because you can't think of any likely modes of failure doesn't mean there is none. I try to keep this in mind whenever I'm writing code and it has saved my bacon on more than one occasion. Take this as an example. Modify your script so that the aggregated logs are stored on a remote machine. Start the script then turn off your router. Cheers |
All Channels
Xaen's EVE logfile aggregating bash scripts
Screenshots & Art
