| Author |
Topic |
Ryysa
Firefly Inc.
|
Posted - 2007.04.07 03:16:00 - [ 1]
Edited by: Ryysa on 01/11/2007 08:34:58Since I don't know a better forum to post this, I will do so here. I have coded it for my own use, since I often use multiple killboards (corp, alliance, personal) for my kills. Also sometimes I have people in my gang who are not in the alliance and hunting down killmails is a pain in the butt. So I made a small tool. It is written in TCL and thus is fully crossplatform. Most linux distributions come with TCL pre-installed, if you need the interpreter for any other OS (windows/mac/etc), take a look here - http://www.activestate.com/Products/activetcl/ (it's free). The tool allows you to read in all killmails matching a certain criteria from one killboard and post them on another one. I've tested it with most templates of griefwatch and EvE-Dev killboards, and haven't encountered any problems yet. It is a console-only application, can be easily stuck into a crontab with some tweaks. Usage: tclsh leechkm.tcl <source kb> <destination kb> <destination kb pass> <constraints> <source kb> - source killboard url, for example http://www.mykillboard.com/ <destination kb> - destination killboard url, for example http://www.myotherkb.com/ <password> - password for posting mails on the destination kb. <constraints> - are defined as -constraint "option" (if your OS uses different signs for quoting, supply them instead.) -corp "My corporation" - Only takes killmails with involvements by My Corporation. -pilot "pilot" - Only takes killmails with involvements by pilot. -alliance "My alliance" - Only takes killmails with involvements by My Alliance. NOTE: Only one of the above constraints can be passed at a time, however one is required. -time "timestring" - Only posts killmails made after certain timestring. Timestring should be in MM/DD/YYYY format. For example -time "04/05/2007" would only fetch killmails posted after the 5th of April 2007. This is not required, if you don't specify one, it will get all mails since the beginning. Extra: -proxy "host:port" - Uses given HTTP proxy for all outgoing HTTP requests. Example: tclsh leechkm.tcl http://myboard.griefwatch.net/ http://www.myboard.com/ secret -pilot "Ryysa" -time "04/05/2007" TODO: Add HTTP error catching and retrying. Add Battleclinic support. Possibly beautify code. Keep up with changes? Download: http://www.rzr5.com/rise/leechkm-RC1.25-wy01.tclAny feedback and bugreports are welcome... Please use constraints properly - don't overload killboards, this script can easily leech 10+ killmail pages per second (!). You will get some feedback in the console based on status, etc. If running from cron pipe to /dev/null. Changelog (+ = new feature, ! = bugfix): Version RC1.25 ! Once and for all solved problems with corp2id/alliance2id/nick2id functions returning wrong values. Thanks to Wyehr for testing :) Version RC1.24 ! Made alliance2id function work a bit better :) Version RC1.23 ! Modified constraint input slightly, should be no issues with different shells and ' marks in character, corporation or alliance names. Version RC1.22 + Added alliance constraint. (by Wyehr) Version RC1.21 ! Improved compatibility with modded EvE-Dev killboards. Version RC1.2 + Added HTTP Proxy support. Version RC1.1 + Added more informative error messages when posting mails. + Added some comments to code. ! Made time constraint non-mandatory. ! Bugfix on griefwatch killmail count when none found. ! Exiting really means exiting ;) Version RC1.0 + Initial version. |
Mannington Skank
Gallente
Capitalism Amuck
|
Posted - 2007.04.08 04:29:00 - [ 2]
This is freakin nice. tyvm |
Ryysa
Firefly Inc.
|
Posted - 2007.04.08 05:47:00 - [ 3]
Hehe, yw, let me know if it works properly for you and if there are some bugs/errors.
I run this atm in a crontab to copy all mails made by our corp to our corp killboard (griefwatch) from our alliance killboard (eve-dev). |
Sidster
FATAL REVELATIONS
FATAL Alliance |
Posted - 2007.04.13 13:19:00 - [ 4]
Great stuff, but it doesn't seem to be working for me.
#1:
tclsh leechkm.tcl sourcekb destkb password -pilot='Sidster'
EvE-Dev killboard detected at sourcekb...
Insufficient constraints specified, exiting...
Fetching kills... none found.
Insufficient constraints specified, exiting...
Fetching losses... none found.
All Done!
Now, I know full well that there are mails matching those constraints.
#2:
A similar command, using a griefwatch killboard instead of an EDK. Instead of finding none, however, it's currently fetching number 1370 - and it's still going. I know that there aren't that many that match the constraints on there :P
#2 could be explained if that script fetches ALL mails and then parses them locally, but I don't know TCL nearly well enough to work out if that's happening.
As for #1 .. no idea. A bug in the constraints coding I'd guess. |
Sidster
FATAL REVELATIONS
FATAL Alliance |
Posted - 2007.04.13 14:25:00 - [ 5]
Played with the code a tiny bit - #1 seems definitely to be caused by the "pilot" var not being caught properly. If you mess things up a bit by causing the pilot var to be filled by the third argument on the command line, it works fine. |
Ryysa
Firefly Inc.
|
Posted - 2007.04.14 08:38:00 - [ 6]
Edited by: Ryysa on 14/04/2007 09:33:30uhm, you needed to specify date. I need to work on the description of errors better. If you want all kills, just make date "01/01/00" for example. I need to look into eve-dev kb and griefwatch. Last I checked, contraints worked properly. And no, it doesn't fetch all mails and process them locally, that would be a terrible waste of bandwidth. I tested with griefwatch board - constraints work fine. The only parameters that need to be in a certain order are the source kb, destination kb and password. If you look on how the parsing of the contraints is done, you will see that there is no difference where they are and in which order. Example: tclsh leechkm.tcl <source> <dest> <pass> -pilot='pilot' -time='mm/dd/yyyy' Is the same as: tclsh leechkm.tcl <source> <dest> <pass> -time='mm/dd/yyyy' -pilot='pilot' The only problem that could be happening with griefwatch is if there are some really weird templates, but so far it should handle every template, because it just fetches a page and searches for all links inside it that point towards kill details, also makes sure it doesn't follow the same link multiple times. Update: Version RC1.1 + Added more informative error messages when posting mails. + Added some comments to code. ! Made time constraint non-mandatory. ! Bugfix on griefwatch killmail count when none found. ! Exiting really means exiting ;) Link here: http://fails.rainsite.net/leechkm.tcl (same as before). |
Sidster
FATAL REVELATIONS
FATAL Alliance |
Posted - 2007.04.15 13:59:00 - [ 7]
Still no joy here.
Using v1.1, the commmand:
tclsh leechkm.tcl sourcekb destkb password -pilot='Sidster'
Still gives "Eve-Dev kill board detected" then "Insufficient constraints specified, exiting ..."
I get the same if I include the -time arg.
If I hardcode the pilot var in the script, it works fine. |
Ryysa
Firefly Inc.
|
Posted - 2007.04.15 19:05:00 - [ 8]
well, that's weird, you /must/ be mistyping something. Otherwise, please mail me your killboard addresses ingame along with the EXACT string you are executing and I can test it. It's fairly simple how it works. It looks through the command line arguments after the third argument and runs a regex on them, after which it stores the pairs... regexp -inline -all {\-(.+?)\='(.+?)'} [join [lrange [split $argv] 3 end]]Returns an element list in which every 3 elements are <matched string part> <constraint> <value> In fact, you could specify any random arbitary constraints and it would parse them... |
DanMck
Amarr
Rionnag Alba
Against ALL Authorities |
Posted - 2007.05.03 13:40:00 - [ 9]
download link not working for me ?  really interested to test this out  cheers |
Splagada
Minmatar
Tides of Silence
|
Posted - 2007.05.06 11:25:00 - [ 10]
Edited by: Splagada on 06/05/2007 13:02:39
thanks a ton, using it and i love it
also had the problem with constraints and it was because i was starting the wrong exe, the good one is tclsh
edit : i got timeout errors when too many kills are transferred, any trick to go around that? |
Niedar
MASS |
Posted - 2007.05.08 06:35:00 - [ 11]
Just used it for the first time and it worked very good, even though I dont know tcl I was able to understand it enough to modify it. |
Ryysa
Firefly Inc.
|
Posted - 2007.05.10 09:55:00 - [ 12]
Edited by: Ryysa on 10/05/2007 09:51:52 Originally by: DanMck
download link not working for me ?
really interested to test this out
cheers
Let me know if it still doesn't work and I'll mirror it. Originally by: Splagada
edit : i got timeout errors when too many kills are transferred, any trick to go around that?
Uhm, i did transfer about 1000 kills from one killboard to the other and it worked just fine. I guess it depends on your connection quality. I didn't implement any timeout handling whatsoever. However, it is possible to do, and if you are still often getting timeout errors, I can try and implement something that will retry 2-3 times and then ask you if you want to continue retrying... Originally by: Niedar
Just used it for the first time and it worked very good, even though I dont know tcl I was able to understand it enough to modify it.
Yes except you messed it up a wee bit :x Oh also, if any of you want to run this from a crontab or some windows scheduling script, you'll most likely need a wrapper script. The simplest way is to fake the argument line. With something like this (first line for linux only). #!/path/to/tclsh
# Get the date and substract one day from it.
set mytime [clock format [expr [clock seconds] - 60*60*24]]
# Fake the command line
set argv "http://fromkn.com http://tokb.com password -corp='mycorp' -time='$mytime'"
# Execute the script, use SLASHES not backslashes
source "path/to/leechkm.tcl" Now you can obviously repeat the last two commands as much as you like - if you want for example to keep two killboards in sync, you can run it again with the killboard address positions swapped and the password of the first killboard. This can simply be stuck in a cron and executed daily. Hope that makes sense and is useful in some way. |
Niedar
MASS |
Posted - 2007.05.11 20:56:00 - [ 13]
Lets not mention that Ryysa ;0 |
darkseed
Omega Fleet Enterprises
Executive Outcomes |
Posted - 2007.05.21 11:40:00 - [ 14]
Quote:
# Get the date and substract one day from it.
set mytime [clock format [expr [clock seconds] - 60*60*24]]
You got a little bug in this, the timeformat which will be returned is in full data + time + timezone format, you need to add a second formatting option to 'clock format' in tcl. The line should be:
# Get the date and substract one day from it.
set mytime [clock format [expr [clock seconds] - 60*60*24] -format %D]
This will return the date as 'mm/dd/yyyy' |
Ryysa
Firefly Inc.
|
Posted - 2007.05.22 07:13:00 - [ 15]
Edited by: Ryysa on 22/05/2007 07:20:43The format doesn't really need to be MM/DD/YYYY, it can be anything clock scan recognizes. Clock scan recognizes the format of the usual clock format, so it doesn't make any difference whatsoever. Try this: clock scan [clock format [clock seconds]] EDIT: Just tried this with a newer distro of TCL, and it seems broken, will test some more. OK. basically, on my linux tclsh install from 8.4 this works, on my windows one it does not. I tested on Linux. After checking some things, the entire code should be rewritten as: #!/path/to/tclsh
# Get the date and substract one day from it.
set mytime [clock format [clock scan yesterday] -format %D]]
# Fake the command line
set argv "http://fromkb.com http://tokb.com password -corp='mycorp' -time='$mytime'"
# Execute the script, use SLASHES not backslashes
source "path/to/leechkm.tcl" Thanks for pointing out btw, never knew that clock format [clock scan [clock seconds]] is broken on windows. |
Splagada
Minmatar
Tides of Silence
|
Posted - 2007.05.22 09:33:00 - [ 16]
Edited by: Splagada on 22/05/2007 09:31:04
C:\Temp\Temp\tcl\demos\leechkb>tclsh leechkm.tcl http://triumvirate.griefwatch.net/ http://hydraalliance.org/kb/ xxxx -corp='Ganja Labs' -time='01/01/2007'
----
Ryysa's killboard parser RC1.1
couldn't open socket: connection timed out
while executing
"http::geturl $url"
(procedure "kbtype" line 2)
invoked from within
"kbtype $furl"
(procedure "process_kb" line 2)
invoked from within
"process_kb $furl $turl"
invoked from within
"if {[info exist input]} {
if {[file exist $input]} {
set mails [read [set fs [open $input r]]]
close $fs
post_mails $mails $turl
} {
puts "$..."
(file "leechkm.tcl" line 251) |
Ryysa
Firefly Inc.
|
Posted - 2007.05.23 07:45:00 - [ 17]
Edited by: Ryysa on 23/05/2007 07:43:39
Well, that means that it can't connect to the server you're giving it.
Are you using a proxy to access the internet?
It fails at the very first connection where it determines the killboard type... So give some more info and perhaps I can help you out.
Perhaps I should work more on this thing and give it slightly more informative error messages :) |
Splagada
Minmatar
Tides of Silence
|
Posted - 2007.05.23 10:59:00 - [ 18]
Originally by: Ryysa
Edited by: Ryysa on 23/05/2007 07:43:39
Well, that means that it can't connect to the server you're giving it.
Are you using a proxy to access the internet?
It fails at the very first connection where it determines the killboard type... So give some more info and perhaps I can help you out.
Perhaps I should work more on this thing and give it slightly more informative error messages :)
damn yes there is a proxy, but rest of internet apps work fine i tried to play with tcl commands like autoproxy but no succes. i got all the proxy passes and logins etc but cant implement that in the script is this doable? |
Ryysa
Firefly Inc.
|
Posted - 2007.05.24 20:09:00 - [ 19]
Edited by: Ryysa on 24/05/2007 20:17:40
Yes, it is...
I'll see what I can do.
I assume you are using a http proxy?
If it's socks5 you'll probably have to use something like sockscap...
Also, if it's passworded, I am not quite sure what I can do about it.
I don't have any experience whatsoever with passworded http proxies, since usually they're just limited to a certain subnet or set of subnets.
I'll implement basic proxy functionality. |
Ryysa
Firefly Inc.
|
Posted - 2007.05.24 20:54:00 - [ 20]
Edited by: Ryysa on 24/05/2007 20:58:40
OK, added basic http proxy functionality.
I tested it with the proxy from my isp and it seems to work.
Command is same as other constraints -proxy='host:port'.
The script tests whether it can connect to the killboard it's supposed to be fetching kills from via the proxy, if it fails, it informs what proxy it used and where it tried to connect to.
If it succeeds, it proceeds as normal.
Going to update the main post also. |
Hux Ley |
Posted - 2007.05.31 20:23:00 - [ 21]
Quote:
Testing proxy... Proxy OK!
Unsupported killboard at http://killboard.eve-d2.net/, exiting...
That's due to the fact the killboard got modified or did the proxy not work? |
Ryysa
Firefly Inc.
|
Posted - 2007.06.03 01:56:00 - [ 22]
Edited by: Ryysa on 03/06/2007 02:20:03
That's due to it not being able to detect the killboard type.
If it says proxy OK then it can at least connect to the address of the killboard you gave it through the proxy.
Edit:
That killboard seems to be messed up pretty bad.
Even if I hack around a bit to get past the initial detection, I can't get my nick2id / corp2id functions to work.
They just don't seem to submit properly and I have nfi why...
I'll keep checking...
Edit2:
Well, I made the thing more robust in handling eve-dev killboards now. It should work with d2's now, and also with any other boards that aren't modded too hard.
Just download the leechkm.tcl file again from the same URL and overwrite your old one. |
Hux Ley |
Posted - 2007.06.03 17:26:00 - [ 23]
thanks! |
Ryysa
Firefly Inc.
|
Posted - 2007.06.03 17:59:00 - [ 24]
You're welcome, do let me know if there are any other issues. |
Manatheren
The Aduro Protocol
The Fifth Race |
Posted - 2007.07.28 21:21:00 - [ 25]
Edited by: Manatheren on 28/07/2007 21:22:30
I had the same problem as Sidster mentioned in this thread - no matter what I did, the script insisted that there were insufficient constraints specified. After much wrangling, I figured out why the script thought so and a workaround, but not really why it happens.
"Something" (most likely my shell - Bash 3.1.17) was stripping the quotation marks in between me hitting enter and the script taking over. As a result, the regexp found no matches (as it relies on seeing those 'xxx' strings) - no constraints.
Once I noticed that, it was just a case of using -pilot=\'Manatheren\' (i.e. escaping the apostrophes), and it all works fine.
No idea why this only happened to Sidster and myself (can't just be us two who have used it on Bash, surely?), but this appears to be the solution.
Thanks for a great script Ryysa, makes life wonderfully easier. |
Ryysa
Firefly Inc.
|
Posted - 2007.07.30 13:14:00 - [ 26]
hehe good thing you figured out.
instead of escaping you /could/ try " instead of '. |
civari
0utbreak
KrautbreaK |
Posted - 2007.08.09 11:45:00 - [ 27]
Edited by: civari on 09/08/2007 11:50:46
I came across a problem where 2 pilots had similar names. So the script just copies the mails from one of them. Is there any way to maybe to instead of using name, that i could use the plt_id?
Thanks for a nice tool btw ;)
edit: I did hardcode it in, so got it to work, but bit boring have to to that all the time :D |
Ryysa
Firefly Inc.
|
Posted - 2007.08.16 04:54:00 - [ 28]
AFAIK, "ID" is different for eve-dev/griefwatch killboards.
Regarding people with a similar name, I'll need an exact way to reproduce it, otherwise feel free to extend the scripts functionality yourself and post it here if you like.
With my current busy schedule I have no idea when I will have time to do anything remotely EvE related :( |
Marina Charnatie
Celestial Janissaries
Curatores Veritatis Alliance |
Posted - 2007.08.24 20:43:00 - [ 29]
Is there any suggestion on how to handle a KB that does not use a pwd? |
Xealot
hirr
Morsus Mihi |
Posted - 2007.08.26 13:09:00 - [ 30]
Edited by: Xealot on 26/08/2007 13:35:59Edited by: Xealot on 26/08/2007 13:33:21Edited by: Xealot on 26/08/2007 13:33:01Hi. Our alliance killboard ( link) shows all the most recent kills as Classified, Im not sure wether this is a mod or new feature in the Eve-Dev killboard, regardless. The script with version 1.21 will break because it expects proper killmails to parse but instead it receives an error message about the mail not being available I have patched up your version, The changes are the following: - Script will no longer break on classified mails, instead, it will ignore them Edit: The colors I added was removed, the windows command prompt didnt like 'em :( DownloadPS. I have never touched TCL before so excuse me for the ****ty solution, but at least it works.. |
Screenshots & Art
All Channels
[Utility] Killboard fetcher/mirrorer, supports EvE-Dev and Griefwatch.