| Author |
Topic |
 Dr Slurm General Commodities
|
Posted - 2006.12.28 19:13:00 - [ 1]
Edited by: Dr Slurm on 19/04/2007 01:15:26UPDATE: All data collected may be purchased for 200m. These profits will be equally distributed to all individuals who contributed. Contact me ingame with questions or purchase requests. The Market Data that I speak of is not the data that comes from the market logs. It comes from the table in "Price History" tab in the market window. By default this displays a graph, there's a button at the bottom that says show table. This table gives you the data of what actually happened on a specific day. This is much more useful then the market logs that EVE generates. The market logs only present the orders that people have created. Although this does tell you a lot, just not all the history of the market. Why I bring this up: I've made a couple of nice graphs that present the history of the market. These graphs are similar to those in game. The advantage of collecting this data is it can be combined with the market logs for interesting results. It can also be analyzed in other ways. These graphs require a good sampling of data. So far I've hand entered this data into a spreadsheet and converted it into a mysql table. This was very slow and tedious. So I found a software package that allows you to reliably pull data with only minor corrections needed. What I need from other people: Screen shots of the above described table in consecutive order for at least 1 year time span for each mineral in each region. Screenshot Requirements:Size: 1024x768 Font: Choose "Expanded" Font in the settings menu. Quality: Choose "High Quality Screenshots" in the settings menu. UPDATE: Background: Change the transparency of the background to solid black.Dates must not overlap. Window Position: The market window should be positioned and sized as similar to this placement as possible. Images should be placed in a folder similar to their nature i.e. JITA-TRIT. All folders containing images should be archived (zip, rar, gz I don't care) Email all files to drslurm@gmail.com How you can help: Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below). What I will do: I will run the screen shots through this OCR software and weed out the bad data. And then compile it into a number of formats. (.CSV, EXCEL, MYSQL, or anything else reasonable) What you get for helping: Everyone who helps gets a copy in their choice of formats of all data collected and access to daily updates. The public who did not contribute will pay for the data at rates yet to be determined. I'm interested in all regions and all minerals for regions. So far from my testing it takes about 5 minutes to take screen shots of an entire year for one mineral. NOTE: Low sec regions aren't as important as Empire for this task but I would like the data if it's available. Sign up has commenced please post here if you would like to be responsible for a region or whatever. There's not really a deadline for screen shots, but I don't think the data will be useable without the majority of Empire being recorded. It will probably take me another month to put it all together, conservatively. |
 Dr Slurm General Commodities
|
Posted - 2006.12.28 19:14:00 - [ 2]
Edited by: Dr Slurm on 05/02/2007 13:30:13 Priority Regions Not Received: 0 Received: 23 Regions w/o Volunteers: 40 In Progress: 4 Fully Processed: 15
Priority Regions Aridia In Progress -Dak Biet Derelik Fully Processed -Feya Kortimna Devoid Fully Processed -Ki Shodan Domain Fully Processed -Dr Slurm Essence Fully Processed -Dr Slurm Everyshore In Progress -Dak Biet Genesis In Progress -Pang Grohl Heimatar Fully Processed -Dr Slurm Kador Fully Processed -Ki Shodan Khanid Fully Processed -Block Ukx Kor-Azor Fully Processed -Menkaure Lonetrek Fully Processed -Menkaure Metropolis Fully Processed -Pang Grohl Molden Heath Fully Processed -Block Ukx Sinq Laison Fully Processed -Jade Vendetta Tash-Murkon Received -Sidurii The Bleak Lands Fully Processed -Ki Shodan The Citadel Fully Processed -Menkaure The Forge Fully Processed -Dr Slurm Verge Vendor Fully Processed -Technovar
Non-Priority Regions 6HL8-L 7-KXBJ 87-1CW Branch Cache Catch Received -Menkaure Cloud Ring Curse Deklein Delve Detorid Esoteria Fade Feythabolis Fountain G5KW-3 Geminate Great Wildlands H3J8-U Immensea Impass Insmother LQ-0QN Omist Outer Ring Paragon Soul Period Basis Placid -Sidurii Providence Received -Menkaure Pure Blind Querious S-I6VU Scalding Pass Solitude Stain Received -Menkaure Syndicate Tenal Tenerifis Tribute Vale of the Silent Venal VU-WU2 Wicked Creek |
 SencneS Rebellion Against Big Irreversible Dinks
|
Posted - 2006.12.28 19:53:00 - [ 3]
It's a great idea, but I can't help but think there may be an easier way.
http://eve-central.com/home/marketstat_xml.html?evemon=1
It grabs all the minerals, buy orders across all reporting regions across Eve.
These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space. |
 Dr Slurm General Commodities
|
Posted - 2006.12.28 20:11:00 - [ 4]
Edited by: Dr Slurm on 07/01/2007 14:33:25 Originally by: SencneS It's a great idea, but I can't help but think there may be an easier way.
http://eve-central.com/home/marketstat_xml.html?evemon=1
It grabs all the minerals, buy orders across all reporting regions across Eve.
These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space.
That data is orders that ARE on the market currently. The data I want is what actually happened in the market for the day. The data is really only good for the previous date, but having a years worth on hand for each mineral in each region would be nice. /edit: Here's a screenshot of what I mean. |
 Pang Grohl Gallente Sudo Corp
|
Posted - 2006.12.28 20:31:00 - [ 5]
Any and every marketeer should be drooling over the opportunity this represents for OOG analysis of market trends in EVE.
I'll gladly volunteer my services to help you, Dr. Slurm. Sign me up for the Metropolis region for all minerals.
Also, what kind of deadline do you have for getting the data? And, could you let us in on the OCR software you're using? I need this kind of functionality for transaction logging from the wallet.
|
 Dr Slurm General Commodities
|
Posted - 2006.12.28 20:39:00 - [ 6]
I didn't mean to sound secretive, the software is ABBYY FineReader Pro 8.0 (FR). It's rather expensive and thus I didn't expect anyone else to do the OCR work.
So far this is my refining process: Take Screen shots move to individual folder like DOMAIN-ISO Open a batch of screenshots in FR Apply a table block format to all of them Adjust the table on each screen shot so it lines up Read all the pages Check each page for obvious errors like letters Remove all of " ; : | , . and really anything that isn't a number Save all the data to a .CSV file Open the .CSV file in a text editor Look over the data again to catch anymore mistakes and remove any "'s that are generated by FR Save any changes Insert .CSV into Open Office to check for more errors
This pretty much cleans the data up. I'll admit there's a few errors but that is in part thanks to the eve client. This does leave the ISK values for LOW HIGH and AVG without proper decimnal places but this can be fixed by dividing all data by 100.
|
 Ramblin Man Empyreum |
Posted - 2006.12.28 20:52:00 - [ 7]
Edited by: Ramblin Man on 28/12/2006 20:52:16
I'll help out. (Will check later when I've got time and see what region/data is unclaimed)
Where's the "window" by default in the OCR software? Or do you have to manually draw it? It seems like a bit of time could be saved by having people properly position their market windows ingame before taking screenshots. |
 Dr Slurm General Commodities
|
Posted - 2006.12.28 20:55:00 - [ 8]
You can see an example screenshot in one of my above posts.
I doubt everyone will be able to get it exactly the same.
I will update the first post with screen shot requirements. |
 Technovar Gallente Eve University Ivy League |
Posted - 2006.12.29 22:59:00 - [ 9]
Originally by: Dr Slurm
Images should be placed in a folder similar to their nature i.e. JITA-TRIT. All folders containing images should be archived (zip, rar, gz I don't care) Email all files to eve_market_ocr@hotmail.com
How you can help: Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below).
*snip* Originally by: Dr Slurm
I'd like feedback on this idea before I start an actual sign up and start collecting data.
I know you're looking for feedback, but it does look like you've gotten good feedback so far, so I took action.  Since I'm reasonably new, my region choice is somewhat limited if I want to stay alive. So, I've got all minerals in Verge Vendor screenshotted from 2005-Nov-01 to 2006-Dec-28, rar'ed into a 75MB file, and ready to go when you do take signups. If you have available space in that account, I can go ahead and mail it. I'm certainly interested in the SQL once you get this done, as well. |
 Dr Slurm General Commodities
|
Posted - 2006.12.29 23:15:00 - [ 10]
Edited by: Dr Slurm on 30/12/2006 16:24:41 I guess the full sign up is on. I can't think of anymore kinks to work out.
I've already finished the OCR work for Domain and Essence. I have Heimatar already screen shotted. You are welcome to send what you have drslurm@gmail.com . The email box is 1gig so I should have plenty of room.
Please I encourage more people to be a part of this. It doesn't take more then 2-3 mins to grab the screen shots of one mineral. It might take 20 mins to grab all of them for one region. This isn't hard and you reap large amount of otherwise unattainable data from it. |
 Dr Slurm General Commodities
|
Posted - 2006.12.30 16:43:00 - [ 11]
I had to change the email to drslurm@gmail.com hopefully there won't be virus scanning issues like with hotmail.
I also updated the first post to say that screenshots need to be taken on a 0% transparency background preferably solid black. |
 Ray McCormack Nordar Innovations. |
Posted - 2006.12.30 18:39:00 - [ 12]
I was pretty sure The Forge wasn't marked when I started doing it... Oh well.
|
 Dr Slurm General Commodities
|
Posted - 2006.12.30 18:45:00 - [ 13]
I'm sorry, I hate to waste your time, I just finished it. Please let me know if you'd be willing to do one of the other priority regions before you start.
|
 Ki Shodan Gallente deep blue
|
Posted - 2006.12.30 20:58:00 - [ 14]
I am taking shots of all minerals in "Devoid", "The Bleak Lands" and "Kador" regions, if still available. (Can do regular updates on Domain later, too.) Is Morphite also needed or just the normal minerals? |
 Dr Slurm General Commodities
|
Posted - 2006.12.30 21:13:00 - [ 15]
I will sign you up.
Just the normal minerals Trit, Pye, Mex, Iso, Noc, Zyd, and Mega. |
 Menkaure Amarr LEM0N
|
Posted - 2006.12.30 23:45:00 - [ 16]
Edited by: Menkaure on 31/12/2006 00:31:25 Guess I can take lonetrek. :)
Be Very interesting to see the total quantity of minerals sold throughout eve every day, etc.
Great Idea.
|
 Dr Slurm General Commodities
|
Posted - 2006.12.31 01:22:00 - [ 17]
I'm making good head way processing the screen shots I've taken myself and those sent to me by Technovar. I've refined the process and am able to check data much quicker now.
If I had all the screen shots tonight I could come close to finishing. I can take them, but I'd rather spend my time processing. |
 Menkaure Amarr LEM0N
|
Posted - 2006.12.31 03:57:00 - [ 18]
Got them for you but Hotmail/Gmail both seem to be crapping out when I try to upload em so I'll try again tommorow morning at some point.
|
 Dr Slurm General Commodities
|
Posted - 2006.12.31 04:13:00 - [ 19]
Technovar was able to send them through google in about 9mb packages. One archive for each mineral. |
 Menkaure Amarr LEM0N
|
Posted - 2006.12.31 16:14:00 - [ 20]
k you should have mine now, ill hop over to the citadel and do them aswell |
 Dr Slurm General Commodities
|
Posted - 2007.01.01 07:17:00 - [ 21]
Edited by: Dr Slurm on 01/01/2007 07:25:05 I currently have all of The Forge, Domain, Essence, Heimatar, and Verge Vendor OCR'ed and purified of erroneous data.
I currently have 2 regions and some change to work on.
Just an update.
/I put my name on The Citadel by accident. I hope that didn't confuse.
Tip to contributors: If you expand the size of the column in the eve market window it gives me more wiggle room to place the template the OCR software uses over it. |
 Pang Grohl Gallente Sudo Corp
|
Posted - 2007.01.01 09:11:00 - [ 22]
Dr. did you get the screenies I sent you for Metropolis? |
 Dr Slurm General Commodities
|
Posted - 2007.01.01 12:07:00 - [ 23]
Originally by: Pang Grohl Dr. did you get the screenies I sent you for Metropolis?
Did you send it to drslurm@gmail.com ? the hotmail address wouldn't accept the large attachments. Either way I haven't found an email saying it was from you. |
 Pang Grohl Gallente Sudo Corp
|
Posted - 2007.01.02 16:30:00 - [ 24]
Originally by: Dr Slurm
Originally by: Pang Grohl Dr. did you get the screenies I sent you for Metropolis?
Did you send it to drslurm@gmail.com ? the hotmail address wouldn't accept the large attachments.
Either way I haven't found an email saying it was from you.
I did send it to the gmail address. I had some contradictory messages from my e-mail client about the success of the send & wanted to make sure you didn't get it before I resent. I will try again now. |
 Dr Slurm General Commodities
|
Posted - 2007.01.03 13:18:00 - [ 25]
I still haven't received an email from Pang.
I really appreciate everyone's help so far, this is going to be a great project.
|
 Jade Vendetta Gallente The Babylon5 Consortuim
|
Posted - 2007.01.03 15:48:00 - [ 26]
Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with. |
 Dr Slurm General Commodities
|
Posted - 2007.01.03 23:53:00 - [ 27]
Originally by: Jade Vendetta Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with.
You have been signed up. Only 9 real regions left for the priority group, 10 if you count the unpronounceable one. |
 Menkaure Amarr LEM0N
|
Posted - 2007.01.04 01:47:00 - [ 28]
Are we going to have to wait till those 10 regions are in before you show the data?
|
 Icy One Shadows of the Dead The Dawn of Darkness |
Posted - 2007.01.04 09:57:00 - [ 29]
I can take on doing all minerals in the Tash-Murkon region. I will get some SS to you after the extended DT.
|
 Dr Slurm General Commodities
|
Posted - 2007.01.05 12:47:00 - [ 30]
Originally by: Menkaure Are we going to have to wait till those 10 regions are in before you show the data?
I'd rather not redistribute a bunch of different copies. I've setup an SVN ( subversion) on my local computer. I'm not sure if it's accessible from the outside. Once I get that setup all that contributed will be welcome to check out and possibly commit changes. |