Search

 
 Join the Community, Create an EVE-ONLINE account or log in.

open All Channels
sepopen Market Discussions
blankseplocked Market Data OCR Project - Core Regions Finished and On Sale
 
This thread is older than 90 days and has been locked due to inactivity.

New Topic   
 
Pages: [1] 2 3 4

Author Topic

Dr Slurm
General Commodities
Posted - 2006.12.28 19:13:00 - [1]
 

Edited by: Dr Slurm on 19/04/2007 01:15:26
UPDATE: All data collected may be purchased for 200m. These profits will be equally distributed to all individuals who contributed. Contact me ingame with questions or purchase requests.

The Market Data that I speak of is not the data that comes from the market logs. It comes from the table in "Price History" tab in the market window. By default this displays a graph, there's a button at the bottom that says show table.

This table gives you the data of what actually happened on a specific day. This is much more useful then the market logs that EVE generates. The market logs only present the orders that people have created. Although this does tell you a lot, just not all the history of the market.

Why I bring this up:
I've made a couple of nice graphs that present the history of the market. These graphs are similar to those in game. The advantage of collecting this data is it can be combined with the market logs for interesting results. It can also be analyzed in other ways. These graphs require a good sampling of data. So far I've hand entered this data into a spreadsheet and converted it into a mysql table. This was very slow and tedious. So I found a software package that allows you to reliably pull data with only minor corrections needed.

What I need from other people:
Screen shots of the above described table in consecutive order for at least 1 year time span for each mineral in each region.

Screenshot Requirements:
Size: 1024x768
Font: Choose "Expanded" Font in the settings menu.
Quality: Choose "High Quality Screenshots" in the settings menu.
UPDATE: Background: Change the transparency of the background to solid black.
Dates must not overlap.
Window Position: The market window should be positioned and sized as similar to this placement as possible.
Images should be placed in a folder similar to their nature i.e. JITA-TRIT.
All folders containing images should be archived (zip, rar, gz I don't care)
Email all files to drslurm@gmail.com

How you can help:
Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below).

What I will do:
I will run the screen shots through this OCR software and weed out the bad data. And then compile it into a number of formats. (.CSV, EXCEL, MYSQL, or anything else reasonable)

What you get for helping:
Everyone who helps gets a copy in their choice of formats of all data collected and access to daily updates. The public who did not contribute will pay for the data at rates yet to be determined.

I'm interested in all regions and all minerals for regions. So far from my testing it takes about 5 minutes to take screen shots of an entire year for one mineral.
NOTE: Low sec regions aren't as important as Empire for this task but I would like the data if it's available.

Sign up has commenced please post here if you would like to be responsible for a region or whatever. There's not really a deadline for screen shots, but I don't think the data will be useable without the majority of Empire being recorded. It will probably take me another month to put it all together, conservatively.

Dr Slurm
General Commodities
Posted - 2006.12.28 19:14:00 - [2]
 

Edited by: Dr Slurm on 05/02/2007 13:30:13
Priority Regions Not Received: 0
Received: 23
Regions w/o Volunteers: 40
In Progress: 4
Fully Processed: 15

Priority Regions
Aridia In Progress
-Dak Biet
Derelik Fully Processed
-Feya Kortimna
Devoid Fully Processed
-Ki Shodan
Domain Fully Processed
-Dr Slurm
Essence Fully Processed
-Dr Slurm
Everyshore In Progress
-Dak Biet
Genesis In Progress
-Pang Grohl
Heimatar Fully Processed
-Dr Slurm
Kador Fully Processed
-Ki Shodan
Khanid Fully Processed
-Block Ukx
Kor-Azor Fully Processed
-Menkaure
Lonetrek Fully Processed
-Menkaure
Metropolis Fully Processed
-Pang Grohl
Molden Heath Fully Processed
-Block Ukx
Sinq Laison Fully Processed
-Jade Vendetta
Tash-Murkon Received
-Sidurii
The Bleak Lands Fully Processed
-Ki Shodan
The Citadel Fully Processed
-Menkaure
The Forge Fully Processed
-Dr Slurm
Verge Vendor Fully Processed
-Technovar

Non-Priority Regions
6HL8-L
7-KXBJ
87-1CW
Branch
Cache

Catch Received
-Menkaure
Cloud Ring
Curse
Deklein
Delve
Detorid
Esoteria
Fade
Feythabolis
Fountain
G5KW-3
Geminate
Great Wildlands
H3J8-U
Immensea
Impass
Insmother
LQ-0QN
Omist
Outer Ring
Paragon Soul
Period Basis

Placid
-Sidurii
Providence Received
-Menkaure
Pure Blind
Querious
S-I6VU
Scalding Pass
Solitude

Stain Received
-Menkaure
Syndicate
Tenal
Tenerifis
Tribute
Vale of the Silent
Venal
VU-WU2
Wicked Creek

SencneS
Rebellion Against Big Irreversible Dinks
Posted - 2006.12.28 19:53:00 - [3]
 

It's a great idea, but I can't help but think there may be an easier way.

http://eve-central.com/home/marketstat_xml.html?evemon=1

It grabs all the minerals, buy orders across all reporting regions across Eve.

These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space.

Dr Slurm
General Commodities
Posted - 2006.12.28 20:11:00 - [4]
 

Edited by: Dr Slurm on 07/01/2007 14:33:25
Originally by: SencneS
It's a great idea, but I can't help but think there may be an easier way.

http://eve-central.com/home/marketstat_xml.html?evemon=1

It grabs all the minerals, buy orders across all reporting regions across Eve.

These prices are probably a good representation of what you could sell Minerals for anywhere in Empire space.


That data is orders that ARE on the market currently. The data I want is what actually happened in the market for the day. The data is really only good for the previous date, but having a years worth on hand for each mineral in each region would be nice.

/edit: Here's a screenshot of what I mean.

Pang Grohl
Gallente
Sudo Corp
Posted - 2006.12.28 20:31:00 - [5]
 

Any and every marketeer should be drooling over the opportunity this represents for OOG analysis of market trends in EVE.

I'll gladly volunteer my services to help you, Dr. Slurm. Sign me up for the Metropolis region for all minerals.

Also, what kind of deadline do you have for getting the data? And, could you let us in on the OCR software you're using? I need this kind of functionality for transaction logging from the wallet.

Dr Slurm
General Commodities
Posted - 2006.12.28 20:39:00 - [6]
 

I didn't mean to sound secretive, the software is ABBYY FineReader Pro 8.0 (FR). It's rather expensive and thus I didn't expect anyone else to do the OCR work.

So far this is my refining process:
Take Screen shots move to individual folder like DOMAIN-ISO
Open a batch of screenshots in FR
Apply a table block format to all of them
Adjust the table on each screen shot so it lines up
Read all the pages
Check each page for obvious errors like letters
Remove all of " ; : | , . and really anything that isn't a number
Save all the data to a .CSV file
Open the .CSV file in a text editor
Look over the data again to catch anymore mistakes and remove any "'s that are generated by FR
Save any changes
Insert .CSV into Open Office to check for more errors

This pretty much cleans the data up. I'll admit there's a few errors but that is in part thanks to the eve client. This does leave the ISK values for LOW HIGH and AVG without proper decimnal places but this can be fixed by dividing all data by 100.

Ramblin Man
Empyreum
Posted - 2006.12.28 20:52:00 - [7]
 

Edited by: Ramblin Man on 28/12/2006 20:52:16

I'll help out. (Will check later when I've got time and see what region/data is unclaimed)

Where's the "window" by default in the OCR software? Or do you have to manually draw it? It seems like a bit of time could be saved by having people properly position their market windows ingame before taking screenshots.

Dr Slurm
General Commodities
Posted - 2006.12.28 20:55:00 - [8]
 

You can see an example screenshot in one of my above posts.

I doubt everyone will be able to get it exactly the same.

I will update the first post with screen shot requirements.

Technovar
Gallente
Eve University
Ivy League
Posted - 2006.12.29 22:59:00 - [9]
 

Originally by: Dr Slurm

Images should be placed in a folder similar to their nature i.e. JITA-TRIT.
All folders containing images should be archived (zip, rar, gz I don't care)
Email all files to eve_market_ocr@hotmail.com

How you can help:
Sign up for a whole region or just one mineral in a region or anywhere in between. Take the required screen shots and transfer them to me (described below).



*snip*

Originally by: Dr Slurm

I'd like feedback on this idea before I start an actual sign up and start collecting data.


I know you're looking for feedback, but it does look like you've gotten good feedback so far, so I took action. Twisted Evil

Since I'm reasonably new, my region choice is somewhat limited if I want to stay alive. So, I've got all minerals in Verge Vendor screenshotted from 2005-Nov-01 to 2006-Dec-28, rar'ed into a 75MB file, and ready to go when you do take signups. If you have available space in that account, I can go ahead and mail it. I'm certainly interested in the SQL once you get this done, as well.

Dr Slurm
General Commodities
Posted - 2006.12.29 23:15:00 - [10]
 

Edited by: Dr Slurm on 30/12/2006 16:24:41
I guess the full sign up is on. I can't think of anymore kinks to work out.

I've already finished the OCR work for Domain and Essence. I have Heimatar already screen shotted. You are welcome to send what you have drslurm@gmail.com . The email box is 1gig so I should have plenty of room.

Please I encourage more people to be a part of this. It doesn't take more then 2-3 mins to grab the screen shots of one mineral. It might take 20 mins to grab all of them for one region. This isn't hard and you reap large amount of otherwise unattainable data from it.

Dr Slurm
General Commodities
Posted - 2006.12.30 16:43:00 - [11]
 

I had to change the email to drslurm@gmail.com hopefully there won't be virus scanning issues like with hotmail.

I also updated the first post to say that screenshots need to be taken on a 0% transparency background preferably solid black.

Ray McCormack
Nordar Innovations.
Posted - 2006.12.30 18:39:00 - [12]
 

I was pretty sure The Forge wasn't marked when I started doing it... Oh well.

Dr Slurm
General Commodities
Posted - 2006.12.30 18:45:00 - [13]
 

I'm sorry, I hate to waste your time, I just finished it. Please let me know if you'd be willing to do one of the other priority regions before you start.


Ki Shodan
Gallente
deep blue
Posted - 2006.12.30 20:58:00 - [14]
 

I am taking shots of all minerals in "Devoid", "The Bleak Lands" and "Kador" regions, if still available.
(Can do regular updates on Domain later, too.)
Is Morphite also needed or just the normal minerals?

Dr Slurm
General Commodities
Posted - 2006.12.30 21:13:00 - [15]
 

I will sign you up.

Just the normal minerals Trit, Pye, Mex, Iso, Noc, Zyd, and Mega.

Menkaure
Amarr
LEM0N
Posted - 2006.12.30 23:45:00 - [16]
 

Edited by: Menkaure on 31/12/2006 00:31:25
Guess I can take lonetrek. :)

Be Very interesting to see the total quantity of minerals sold throughout eve every day, etc.

Great Idea.


Dr Slurm
General Commodities
Posted - 2006.12.31 01:22:00 - [17]
 

I'm making good head way processing the screen shots I've taken myself and those sent to me by Technovar. I've refined the process and am able to check data much quicker now.

If I had all the screen shots tonight I could come close to finishing. I can take them, but I'd rather spend my time processing.

Menkaure
Amarr
LEM0N
Posted - 2006.12.31 03:57:00 - [18]
 

Got them for you but Hotmail/Gmail both seem to be crapping out when I try to upload em so I'll try again tommorow morning at some point.




Dr Slurm
General Commodities
Posted - 2006.12.31 04:13:00 - [19]
 

Technovar was able to send them through google in about 9mb packages. One archive for each mineral.

Menkaure
Amarr
LEM0N
Posted - 2006.12.31 16:14:00 - [20]
 

k you should have mine now, ill hop over to the citadel and do them aswell

Dr Slurm
General Commodities
Posted - 2007.01.01 07:17:00 - [21]
 

Edited by: Dr Slurm on 01/01/2007 07:25:05
I currently have all of The Forge, Domain, Essence, Heimatar, and Verge Vendor OCR'ed and purified of erroneous data.

I currently have 2 regions and some change to work on.

Just an update.

/I put my name on The Citadel by accident. I hope that didn't confuse.

Tip to contributors: If you expand the size of the column in the eve market window it gives me more wiggle room to place the template the OCR software uses over it.

Pang Grohl
Gallente
Sudo Corp
Posted - 2007.01.01 09:11:00 - [22]
 

Dr. did you get the screenies I sent you for Metropolis?

Dr Slurm
General Commodities
Posted - 2007.01.01 12:07:00 - [23]
 

Originally by: Pang Grohl
Dr. did you get the screenies I sent you for Metropolis?


Did you send it to drslurm@gmail.com ? the hotmail address wouldn't accept the large attachments.

Either way I haven't found an email saying it was from you.

Pang Grohl
Gallente
Sudo Corp
Posted - 2007.01.02 16:30:00 - [24]
 

Originally by: Dr Slurm
Originally by: Pang Grohl
Dr. did you get the screenies I sent you for Metropolis?


Did you send it to drslurm@gmail.com ? the hotmail address wouldn't accept the large attachments.

Either way I haven't found an email saying it was from you.


I did send it to the gmail address. I had some contradictory messages from my e-mail client about the success of the send & wanted to make sure you didn't get it before I resent. I will try again now.


Dr Slurm
General Commodities
Posted - 2007.01.03 13:18:00 - [25]
 

I still haven't received an email from Pang.

I really appreciate everyone's help so far, this is going to be a great project.

Jade Vendetta
Gallente
The Babylon5 Consortuim
Posted - 2007.01.03 15:48:00 - [26]
 

Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with.

Dr Slurm
General Commodities
Posted - 2007.01.03 23:53:00 - [27]
 

Originally by: Jade Vendetta
Excellent idea Dr Slurm, and I'll be glad to help. Sign me up for Sinq Laison (if it hasn't been taken yet) to start with.


You have been signed up.

Only 9 real regions left for the priority group, 10 if you count the unpronounceable one.

Menkaure
Amarr
LEM0N
Posted - 2007.01.04 01:47:00 - [28]
 

Are we going to have to wait till those 10 regions are in before you show the data?


Icy One
Shadows of the Dead
The Dawn of Darkness
Posted - 2007.01.04 09:57:00 - [29]
 

I can take on doing all minerals in the Tash-Murkon region. I will get some SS to you after the extended DT.


Dr Slurm
General Commodities
Posted - 2007.01.05 12:47:00 - [30]
 

Originally by: Menkaure
Are we going to have to wait till those 10 regions are in before you show the data?




I'd rather not redistribute a bunch of different copies. I've setup an SVN (subversion) on my local computer. I'm not sure if it's accessible from the outside. Once I get that setup all that contributed will be welcome to check out and possibly commit changes.


Pages: [1] 2 3 4

This thread is older than 90 days and has been locked due to inactivity.

New Topic