| Author |
Topic |
Kaladr
Amarr |
Posted - 2009.08.05 01:46:00 - [ 1] - Quote
Edited by: Kaladr on 19/10/2009 21:56:26Edited by: Kaladr on 16/09/2009 06:25:36Edited by: Kaladr on 16/09/2009 06:24:35Edited by: Kaladr on 07/09/2009 05:31:58I present you with libevecache, a library for reading and digesting EVE-Online cache files Code: http://gitorious.org/libevecache More information, including how to build and binaries: http://dev.eve-central.com/libevecache/Its portable (ISO C++) and Open Source (GPL v2.0+). Things it does so far: * Structure dumping of the Market Orders cache file, and many other CacheMethodCalls files (though not all). * RLE decompression of the actual market data rows (the zeros! we have zeros!) * Understands nested constructs, such as a second cache file stream embedded in it. * Produces a real Market Order list from the cache file, and even comes with a utility to spit out a CSV file in the same structure as the EVE client. * Basic SWIG bindings (use libevecache in Python, Ruby, .NET, Tcl, and others) Development in progress on: * Figuring out bulkdata compression (zlib?) * Contribtastic integration * SWIG .i file * More documentation (Its fairly simple internally, just think AST) Things it will never do: * Reserializing of the cache file. Have fun! |
wxEswat
Amarr |
Posted - 2009.08.06 09:03:00 - [ 2] - Quote
Interesting.....haven't had time to look too much at the code yet though. Was happy to see GPL and from my initial look appears to be decently cross-platform? I'll definitely take a longer look at this later this week for possible inclusion into my own Eve project. wxEswat |
Kaladr
Amarr |
Posted - 2009.08.06 16:26:00 - [ 3] - Quote
Edited by: Kaladr on 10/08/2009 06:12:07Edited by: Kaladr on 10/08/2009 06:08:36Yes, the library is GPL v2.0 So far, it is entirely (mostly) ISO C++. It builds and runs correctly on Linux (32 and 64bit), though I haven't yet tried building it on Win32/Win64. The right DLL export bits are in place however. You can see some of the dumper output here. The make system is qmake (from Qt), mainly designed to support a future product using that framework (the EVE-Central uploader 2.0). If someone would want to contribute waf/scons/cmake rules, I'd be willing to accept them. No pre-canned VS solution files or autoconf-based builders, please. |
Kaladr
Amarr |
Posted - 2009.08.09 06:45:00 - [ 4] - Quote
Edited by: Kaladr on 09/08/2009 06:45:26
There are some small fixes available now which allow the library to correctly(?) understand the cache file structure nesting, with only some small hacking going on at the end of the file. If you know better, feel free to fix some of the logic :-)
Next task is the market decoder engine, which is not difficult. |
darius mclever |
Posted - 2009.08.09 14:21:00 - [ 5] - Quote
gplv3 is one of the worst licenses to pick for a library. the adaption rate will be close to 0. |
Xaroth Brook
Minmatar
BIG
Majesta Empire |
Posted - 2009.08.09 15:04:00 - [ 6] - Quote
I have to agree on the GPLv3 thing.. it'd be a lot more useful if it were changed to GPLv2 (or 2+ for that matter) .. 3 is just .. wrong. |
darius mclever |
Posted - 2009.08.09 16:39:00 - [ 7] - Quote
actually LGPL would be better. |
Kaladr
Amarr |
Posted - 2009.08.10 06:09:00 - [ 8] - Quote
Edited by: Kaladr on 10/08/2009 06:12:26
Edited by: Kaladr on 10/08/2009 06:09:48
License adjusted to GPL v2.0+. I didn't go as far as doing what I normally do (BSD), but figure dropping 3.0 is a good step in the right direction. |
darius mclever |
Posted - 2009.08.10 11:40:00 - [ 9] - Quote
 Originally by: Kaladr
Edited by: Kaladr on 10/08/2009 06:12:26
Edited by: Kaladr on 10/08/2009 06:09:48
License adjusted to GPL v2.0+. I didn't go as far as doing what I normally do (BSD), but figure dropping 3.0 is a good step in the right direction.
why not lgpl? this would allow people to link it also in non gpl programs. |
Xenofur
Aliastra
|
Posted - 2009.08.11 13:45:00 - [ 10] - Quote
Why not go the whole mile and call it PD? Do you really care about restricting how others use it? |
Muscaat
ASCENTIS CORPORATION
Free Worlds Alliance |
Posted - 2009.08.11 13:50:00 - [ 11] - Quote
Kaladr, this is brilliant. If I had any C++ knowledge I'd offer to help in any way I could - but my experience is in Java.
I'd very much like to use what you've learned here to help get my Java cache reader working properly. |
Dragonaire
Caldari
Corax.
Circle-Of-Two |
Posted - 2009.08.11 13:59:00 - [ 12] - Quote
Problem with PD is laws about it varies depending on country etc. but all the open source licenses avoid that problem by clearly stating how it can be used.
I'd also suggest using LGPL as it is a 'library' and that is what it was made for but GPL2 should be workable for most people as well. |
Xenofur
Aliastra
|
Posted - 2009.08.11 14:56:00 - [ 13] - Quote
Edited by: Xenofur on 11/08/2009 14:58:07 Originally by: Dragonaire
Problem with PD is laws about it varies depending on country etc. but all the open source licenses avoid that problem by clearly stating how it can be used.
Not to get too far off-topic here, but: What the hell? How can laws have any effect on a license that consists of only the sentiment "Do whatever you want."? Got any links to back that up? And hell, if you really do need an explicit license, there's still this: http://creativecommons.org/publicdomain/zero/1.0/ |
Dragonaire
Caldari
Corax.
Circle-Of-Two |
Posted - 2009.08.11 15:18:00 - [ 14] - Quote
Let me wikipedia that for you http://en.wikipedia.org/wiki/Public_domain Quote:
Furthermore, the laws of various countries define the scope of the public domain differently, making it necessary to specify which jurisdiction's public domain is being discussed.
BTW that article was the first hit on Google also so I Googled it for you too  Quote:
And hell, if you really do need an explicit license, there's still this: http://creativecommons.org/publicdomain/zero/1.0/
Which is a great license if it was a work of art or book etc but creative commons themselves say for software there are better ones to use but I'll leave that for those interested in doing their own research to find it.  |
Kaladr
Amarr |
Posted - 2009.08.11 16:21:00 - [ 15] - Quote
Originally by: Xenofur
Edited by: Xenofur on 11/08/2009 14:58:07
Originally by: Dragonaire
Problem with PD is laws about it varies depending on country etc. but all the open source licenses avoid that problem by clearly stating how it can be used.
Not to get too far off-topic here, but: What the hell? How can laws have any effect on a license that consists of only the sentiment "Do whatever you want."? Got any links to back that up?
And hell, if you really do need an explicit license, there's still this: http://creativecommons.org/publicdomain/zero/1.0/
The problem with public domain is that your are giving up all ownership of the code. It is not a license in which you retain ownership through copyright. Some countries aren't set up to deal with property in the commons. And to answer questions, not using LGPL is an intended consequence. Call it my rebellion against closed source applications and hidden secrets. If you're developing an open source product not covered by the GPL (such as the BSD, Apache-derivative, etc) and would like to use the library, contact me. I will not allow any closed source applications to use the library however. |
Entity
X-Factor Industries
Synthetic Existence |
Posted - 2009.08.11 18:19:00 - [ 16] - Quote
Keep in mind that this is a program that handles a proprietary dataformat that has been reverse engineered, so I'm not entirely sure where this stands, legally, in terms of licencing.
On a side note, did this project get approved by CCP? As last time I checked they were a bit tight on stuff pertaining their cache format :P |
Xenofur
Aliastra
|
Posted - 2009.08.11 19:06:00 - [ 17] - Quote
Edited by: Xenofur on 11/08/2009 19:07:28 Originally by: Kaladr
The problem with public domain is that your are giving up all ownership of the code.
Well yeah, but why would you care abo... Originally by: Kaladr
my rebellion against closed source applications
Oh. :v Good to see that you're honest and straight-forward about it and don't try to call it "free". :) Any idea if and when the API is stabilized? I'm considering writing a Perl module to access the library via that. |
Muscaat
ASCENTIS CORPORATION
Free Worlds Alliance |
Posted - 2009.08.11 20:44:00 - [ 18] - Quote
|
Entity
X-Factor Industries
Synthetic Existence |
Posted - 2009.08.11 21:00:00 - [ 19] - Quote
|
Kaladr
Amarr |
Posted - 2009.08.12 01:01:00 - [ 20] - Quote
Originally by: Xenofur
Any idea if and when the API is stabilized? I'm considering writing a Perl module to access the library via that.
Not really stable, and the direct access to market data methods isn't implemented yet. You can always attempt a SWIG based wrapper for simplicity. |
Kaladr
Amarr |
Posted - 2009.08.12 01:06:00 - [ 21] - Quote
Edited by: Kaladr on 12/08/2009 01:14:10 Originally by: Muscaat
Originally by: Entity
[D]id this project get approved by CCP? As last time I checked they were a bit tight on stuff pertaining their cache format :P
Canonical source of CCP's attitude towards cache readers
This is the official direction I am taking with this in regards to CCP's attitude regarding cache reading, and talk about the format thereof. If they didn't want user contributed documentation regarding the cache, its very easy for them to make threads go *poof*. If this thread goes *poof*, or the next patch contains a radical cache format change (rot13!), then we know what their stance is  libevecache was written without any inspection, debugging, disassembly, or other "bad stuff" of EVE binaries or DLLs. It only used intuition (spread over about a month part time), comparison with .CSV exports of the market data, and third party information, such as that above mentioned forum thread. |
Xenofur
Aliastra
|
Posted - 2009.08.12 22:24:00 - [ 22] - Quote
Originally by: Kaladr
Not really stable, and the direct access to market data methods isn't implemented yet.
Alright, thanks. :) Originally by: Kaladr
You can always attempt a SWIG based wrapper for simplicity.
I don't really understand this however. Wrapping your library in perl pretty much only means to write a .xs file which defines all the functions that your library exports and doing some munging so they're a bit nicer to use data-wise. How would SWIG help there? |
Kaladr
Amarr |
Posted - 2009.08.13 16:11:00 - [ 23] - Quote
SWIG would allow for the same thing, but is language agnostic, and understands how to deal with C++ STL types -> native type conversion as well. If Perl XS is what you want, I see no problem in writing a wrapper for Perl, and if you would want to contribute it, would be willing to accept it |
Kaladr
Amarr |
Posted - 2009.08.19 05:24:00 - [ 24] - Quote
Made some updates regarding my current knowledge of each entry in the DBRow. There is no direct parser yet, but it can be hand decoded. Parser for DB row next. |
Herio Mortis
Dark Nebula Academy
O X I D E |
Posted - 2009.08.19 13:56:00 - [ 25] - Quote
Edited by: Herio Mortis on 19/08/2009 13:56:37
In the type field in the parser, the one you do:
check = iter.readChar() & 0x3f; // magic
Also check the 0x40 bit in that one. If it is set, store a reference to the decoded object starting the count from 0. The 0x1b (EStreamIdent) opcode refers to this, so when you encounter that one replace with the stored object matching the 0x1b data byte.
At least that is my understanding of how that works, makes sense in my own playing around at least. |
Anders1
Volatile Instability |
Posted - 2009.08.23 09:36:00 - [ 26] - Quote
I tried to compile this, you seem to use qmake or so, but I don't have Qt installed on my Linux box. I don't want to install it either because it pulls in lots of other dependencies like X, gstreamer, etc.
for x in *.cpp; do
g++ -fPIC -c $x -o $x.o -I.
done
g++ -o libevecache.so *.o -shared
then in util, g++ -o dumper dumper.cpp -I../lib -L../lib -levecache
../lib/libevecache.so: undefined reference to `EveCache::SDBRow::setLast(bool)'
../lib/libevecache.so: undefined reference to `EveCache::SDBRow::isLast() const'
../lib/libevecache.so: undefined reference to `EveCache::ColumnLookup::isInit'
git a6a652b88ada9a9e1e4cacf16bbd5b651d30b423 I guess ? |
Muscaat
ASCENTIS CORPORATION
Free Worlds Alliance |
Posted - 2009.08.24 08:59:00 - [ 27] - Quote
FWIW I also get that compilation error (running KDE, I have much of Qt installed). |
Herio Mortis
Dark Nebula Academy
O X I D E |
Posted - 2009.08.24 13:06:00 - [ 28] - Quote
Yep, current build is broken.
I used a quick scons to build it myself:
---
env = Environment(CPPPATH = ['lib/'])
lib = env.SharedLibrary('evecache', Glob('lib/*cpp'))
env.Program('util/dumper.cpp', LIBS=lib)
---
|
Kaladr
Amarr |
Posted - 2009.08.25 03:00:00 - [ 29] - Quote
Edited by: Kaladr on 25/08/2009 05:49:58
Edited by: Kaladr on 25/08/2009 05:47:28
Edited by: Kaladr on 25/08/2009 05:33:20
Silly build errors fixed, and new SConstruct file added (for easier top level building)
Also, a bunch of CachedMethodCalls cache files can be decoded now, beyond the GetOrders call.
Windows .dll builds work now. |
Skyvyr
Caldari
Ascendent.
On the Rocks |
Posted - 2009.08.25 13:17:00 - [ 30] - Quote
Originally by: Kaladr
Also, a bunch of CachedMethodCalls cache files can be decoded now, beyond the GetOrders call.
I haven't looked close enough, but are the S&I windows cached in there? I would love to be able to export a list of blueprints with their me/pe and on copies, run levels. Keep up the good work! |
All Channels
Screenshots & Art
