| Author |
Topic |
Miss Dominatrix
Evolution
Band of Brothers |
Posted - 2008.01.06 00:25:00 - [ 1]
Do you have a timetable for these changes ? How good will the up and out scaling be ? We could really use some more oomph  |
ElfeGER
Versatech Co.
Aeternus. |
Posted - 2008.01.06 11:30:00 - [ 2]
afaik they said 1+ year as that clustering technology doesn't exist for ms atm
if the players can wait for that long is an other question as well as if it scales one grid as well or just reduces the size of the shared area to system/grid size
also just compare the signal runtime to a different host with the runtime to the same machine but different core
|
Cyron Tout |
Posted - 2008.01.06 12:24:00 - [ 3]
Hiya,
I'm actually very interested in learning exactly how the EVE cluster works. What software its using, hardware (how much?), etc.
I've heard snippets here and there and people seem to know hows it built. Is there a thread or blog around that a dev has gone in length to about explaining it all?
One more thing, i've heard a new cluster is planning on being built. Is there an active blog of this happening? I'd love to see a bit of a dev journal on it if and when it does finally happen. |
Ix Forres
Caldari
Vanguard Frontiers
Antaeus Combine |
Posted - 2008.01.06 14:32:00 - [ 4]
 Originally by: Cyron Tout
Hiya,
I'm actually very interested in learning exactly how the EVE cluster works. What software its using, hardware (how much?), etc.
I've heard snippets here and there and people seem to know hows it built. Is there a thread or blog around that a dev has gone in length to about explaining it all?
One more thing, i've heard a new cluster is planning on being built. Is there an active blog of this happening? I'd love to see a bit of a dev journal on it if and when it does finally happen.
It'd be nice to get some transparency on hardware/software- there's not really that much these days. Currently TQ has a load of hardware running Microsoft Windows Server and Microsoft SQL Server. No idea on versions. One has to wonder why CCP is still hanging onto Micro$oft given the absolutely dismal state of Windows Server OS and it's total inability to do distributed computing properly. *nix/Solaris makes a much better platform for distributed computing- which is, really, what EVE needs. The problems we see on a day-to-day basis are the inability of the current architecture to fluidly react to load. In a fleet battle the node should be able to accomodate several CPUs, and other systems with less going on should have to share more for the duration of that battle- or you have spare compute clusters for fleet battles to spread onto. Either way, the system needs a way to fluidly react to the hugely dynamic loading on systems that fleet warfare creates, while having the ability to have a trade hub use several compute processes rather than one. The trouble is that there is always a central component- the database. However, I suspect this is not a bottleneck unless CCP makes a lot of use of pessimistic locking. The way the EVE system works at present as I understand is a three-tier system- the outer tier is the node servers, one to a system, which are handling actual communication to the clients, next is application servers, handling all the work, and then the database servers. The problem is that right now this one-to-a-system bottleneck on the node server and application servers exists down to the CPU level and they are just processes running on a CPU. Because of this they must be killed to be moved. I think a system where a _set_ of nodeservers/app servers exist for a given system, but only one is active until the others are needed, would be a huge improvement requiring very little extra code- this is of course assuming system servers are thread-safe, but one assumes they would be... But the real benefit would be an entirely flexible system where new node/app servers could come up to react to loading and clients could be evenly distributed over these. Obvious bottlenecks would be connectivity- where infiniband would come in. |
Cyron Tout |
Posted - 2008.01.07 03:44:00 - [ 5]
Very interesting! YOu sound like you have had a bit of a career in such an environment.
I've recently become very interested in cluster networks and performance computing due to the fact that my big project this year will be to research and build one.
I wonder if we can't get into contact with CCP to see if they can get some of their engineers to blog their daily tasks or progress. |
Dr Slaughter
Minmatar
Rabies Inc.
|
Posted - 2008.01.08 14:49:00 - [ 6]
Originally by: Ix Forres
It'd be nice to get some transparency on hardware/software- there's not really that much these days.
Currently TQ has a load of hardware running Microsoft Windows Server and Microsoft SQL Server. No idea on versions.
One has to wonder why CCP is still hanging onto Micro$oft given the absolutely dismal state of Windows Server OS and it's total inability to do distributed computing properly. *nix/Solaris makes a much better platform for distributed computing- which is, really, what EVE needs.
Database isn't the problem (anymore), Network isn't the problem, Microsoft, really, isn't the problem,  (how many times must we go through this?) CPU load and IO on the blades IS one part of the problem, SOL services only being able to run on a single core at a time.. IS THE biggest part of the problem. CCP say they're going to upgrade the cluster in early 08. CCP say they're re-writing the code so that they can have 'grids' run on seperate CPUs. This will help stop entire nodes in the cluster crashing but in all likely hood will not stop grids effectively crashing. Infiniband, etc., helps CCP as they can use the bandwidth to shunt tons of state data between nodes. Anyone saying CCP should throw their current system away and re-implment it on a different OS with different hardware, with a different database because it will be better is being disingenuous. From reading and posting on this topic more actively in the last couple of days (click my picture if you want to find the threads) it's becoming clear to me that: a. CCP have put tons of effort and thought into trying to find a solution for us, b. They're spending a lot of money already on implementing the solutions, c. They're not ready to give us fine detail on the proposed solution yet, d. Without the fine details we're ****ing in the wind.  |
Ix Forres
Caldari
Vanguard Frontiers
Antaeus Combine |
Posted - 2008.01.08 18:16:00 - [ 7]
Originally by: Dr Slaughter
Originally by: Ix Forres
It'd be nice to get some transparency on hardware/software- there's not really that much these days.
Currently TQ has a load of hardware running Microsoft Windows Server and Microsoft SQL Server. No idea on versions.
One has to wonder why CCP is still hanging onto Micro$oft given the absolutely dismal state of Windows Server OS and it's total inability to do distributed computing properly. *nix/Solaris makes a much better platform for distributed computing- which is, really, what EVE needs.
Database isn't the problem (anymore),
Network isn't the problem,
Microsoft, really, isn't the problem, (how many times must we go through this?)
CPU load and IO on the blades IS one part of the problem,
SOL services only being able to run on a single core at a time.. IS THE biggest part of the problem.
CCP say they're going to upgrade the cluster in early 08.
CCP say they're re-writing the code so that they can have 'grids' run on seperate CPUs.
This will help stop entire nodes in the cluster crashing but in all likely hood will not stop grids effectively crashing.
Infiniband, etc., helps CCP as they can use the bandwidth to shunt tons of state data between nodes.
Anyone saying CCP should throw their current system away and re-implment it on a different OS with different hardware, with a different database because it will be better is being disingenuous.
From reading and posting on this topic more actively in the last couple of days (click my picture if you want to find the threads) it's becoming clear to me that:
a. CCP have put tons of effort and thought into trying to find a solution for us,
b. They're spending a lot of money already on implementing the solutions,
c. They're not ready to give us fine detail on the proposed solution yet,
d. Without the fine details we're ****ing in the wind.
Totally agree on the points below, though there's no real evidence of A/B. Grid splitting really won't cut it- what's needed is real true concurrency (Have several servers handle data for the same grid/node). SQL Server has been at the root of many problems with the cluster in the past. I stand by my comments on distributed computing with Microsoft Server OSes- compared to systems like Linux and Solaris it's entirely geared away from true concurrency on objects, and writing distributed code for *nix environments is a hell of a lot easier than writing for w32... |
CCP Lingorm
C C P
|
Posted - 2008.01.09 08:48:00 - [ 8]
Couple of thing.
We have 3 parts to the cluster - proxies, Sol's and the DB.
You are assigned to a single proxy when you connect. It does not change. It is a broker between your client and the Sol nodes you talk to.
Sol's handle the game logic, a Sol can handle 1 (or many) solar systems, regional markets, chat channels etc etc. They can by mixed round to handle a mixture of these. Some handle just 1 thing (f.ex Jita).
The DB is the DB and is not the 'Limiting' factor for performance at the moment.
The limit is that a Sol process is limited to 1 CPU, and this is in our software, not an OS limitation so this would not change if we switched OS's.
Part of the complexity is that we basically wrote our own Cluster management software to run EVE. What HPC and OS Clustering would do for us is allow us to use industry standard clustering techniques and software and allow us to focus more of our efforts on other parts of the code base. We would then not have to write our own versions of any new technology that we wanted to use (f.ex infinband interfaces) but could rely on the OS HPC implementation.
I have no details on the time frames involved here so can not comment on that.
He have a team of people that are really working hard to implement the HPC technology and make all the changes to our code, but it is a Major major job, we basically have to rewrite ALL the in house cluster interface layer to work on HPC style ... not a small task but well worth it in our opinion and already underway.
Can't give you any more details as I am not involved in this project at this time but have kept and eye on it. When it is closer to stable design then expect a Large Dev blog with more details.
|
Mashie Saldana
Red Federation |
Posted - 2008.01.09 14:27:00 - [ 9]
I wonder if Infiniband + RDMA ever will be efficient enough to allow per user loadbalancing. That would mean identical number of users per node regardless of how many are on each grid/system.
Ah well we can at least dream about it. |
Dr Slaughter
Minmatar
Rabies Inc.
|
Posted - 2008.01.09 15:33:00 - [ 10]
Edited by: Dr Slaughter on 09/01/2008 16:27:49 Originally by: CCP Lingorm
Couple of thing.
We have 3 parts to the cluster - proxies, Sol's and the DB.
You are assigned to a single proxy when you connect. It does not change. It is a broker between your client and the Sol nodes you talk to.
Sol's handle the game logic, a Sol can handle 1 (or many) solar systems, regional markets, chat channels etc etc. They can by mixed round to handle a mixture of these. Some handle just 1 thing (f.ex Jita).
The DB is the DB and is not the 'Limiting' factor for performance at the moment.
The limit is that a Sol process is limited to 1 CPU, and this is in our software, not an OS limitation so this would not change if we switched OS's.
Part of the complexity is that we basically wrote our own Cluster management software to run EVE. What HPC and OS Clustering would do for us is allow us to use industry standard clustering techniques and software and allow us to focus more of our efforts on other parts of the code base. We would then not have to write our own versions of any new technology that we wanted to use (f.ex infinband interfaces) but could rely on the OS HPC implementation.
I have no details on the time frames involved here so can not comment on that.
He have a team of people that are really working hard to implement the HPC technology and make all the changes to our code, but it is a Major major job, we basically have to rewrite ALL the in house cluster interface layer to work on HPC style ... not a small task but well worth it in our opinion and already underway.
Can't give you any more details as I am not involved in this project at this time but have kept and eye on it. When it is closer to stable design then expect a Large Dev blog with more details.
Thanks for the reply. I had forgotten about hte Proxy part. There was a nice animated graphic done showing the whole thing a while (over a year) ago. _edit_ found it hereI take it the SOL process is monolithic in nature, i.e. it runs grids, market, station services, missions, etc. and part of the impending re-write is to break those individual functions up so they can be distributed onto separate cores, or potentially, onto different physical machines (using Infiniband to shunt the state data over)? Anyway, thanks for posting! :) |
Lazuran
Gallente
Aliastra
|
Posted - 2008.01.10 12:31:00 - [ 11]
Originally by: CCP Lingorm
The limit is that a Sol process is limited to 1 CPU, and this is in our software, not an OS limitation so this would not change if we switched OS's.
Sounds like this is easy to fix then. Get a very good C programmer (or whatever it is) to make this multithreaded. Even if you don't get it to scale well, you'll be much better off with current hardware with 4-8 and more cores per system. |
CCP Lingorm
C C P
|
Posted - 2008.01.10 12:36:00 - [ 12]
Originally by: Lazuran
Originally by: CCP Lingorm
The limit is that a Sol process is limited to 1 CPU, and this is in our software, not an OS limitation so this would not change if we switched OS's.
Sounds like this is easy to fix then. Get a very good C programmer (or whatever it is) to make this multithreaded. Even if you don't get it to scale well, you'll be much better off with current hardware with 4-8 and more cores per system.
Not written in C++. It's in Python like most of EVE and it is very hard to Thread. Do you want a threaded combat engine ... I don't. It has to be handled sequentially or it will cause to many errors. If one thread completed processing before another thread that started sooner and this was then applied (say damage from my guns) but then the thread from you damage came back and said ... my ship was destroyed we have a problem. But moving other parts of the 'solar system' to their own cpu (like station services, etc) and then grouping those services will make a vast improvement. |
Dr Slaughter
Minmatar
Rabies Inc.
|
Posted - 2008.01.10 14:06:00 - [ 13]
Originally by: CCP Lingorm
It has to be handled sequentially or it will cause to many errors.
 Any chance of a turn based curses interface to combat?   More seriously... when you rip the grid management/combat code out and re-write it, is the plan to support having multiple instances of that per solar system right from the start, or is the plan to initially just split the grid management/combat code from the other services? i.e. smaller boost initially, working towards bigger boost, or just big boost right from the start? |
Yon89
Triumvirate. |
Posted - 2008.01.10 15:23:00 - [ 14]
after the rewriten code for grids and things i take it that when you load grid there will be no session change? |
Dr Slaughter
Minmatar
Rabies Inc.
|
Posted - 2008.01.10 16:02:00 - [ 15]
Originally by: Yon89
after the rewriten code for grids and things i take it that when you load grid there will be no session change?
If they cache the session state in memory on the same physical host that's running your grid that would probably be true (no session change) as the core running the grid could easily read the data from the hosts cache (direct access to the memory). If the grid was being run elsewhere they could use infiniband to pull the state data over (rather than accessing it remotely) which would mean there was a session change but it would be pretty quick. Session changes would still take place between nodes too, but again, if Infiniband is being used to shunt the data around, it should speed up the process. Will Infiniband be used to replace the node to proxy communication channel too? |
Slaine Valdorian
Perkone
|
Posted - 2008.01.10 22:41:00 - [ 16]
Originally by: Lazuran
Sounds like this is easy to fix then. Get a very good C programmer (or whatever it is) to make this multithreaded. Even if you don't get it to scale well, you'll be much better off with current hardware with 4-8 and more cores per system.
Please show me the C programmer, who easily changes sequential code to multithreaded. I want to get rich ;-) Originally by: CCP Lingorm
Not written in C++. It's in Python like most of EVE and it is very hard to Thread.
Praise the GIL ! |
CCP Lingorm
C C P
|
Posted - 2008.01.11 13:16:00 - [ 17]
I don't know how/why/when the Infini band will be used in the cluster.
I do know that there are some very smart people looking at how it will be implemented and that they will do some very good work with it.
|
Zarch AlDain
Hematite Rose
|
Posted - 2008.01.11 13:45:00 - [ 18]
Originally by: Slaine Valdorian
Originally by: Lazuran
Sounds like this is easy to fix then. Get a very good C programmer (or whatever it is) to make this multithreaded. Even if you don't get it to scale well, you'll be much better off with current hardware with 4-8 and more cores per system.
Please show me the C programmer, who easily changes sequential code to multithreaded. I want to get rich ;-)
You probably can't afford me ;) The performance gains are often small though unless the task lends itself to multi-threading. You can end up spending as much time synchronizing the threads as you gain in spreading the load! |
Haulin'Hauling |
Posted - 2008.01.16 14:26:00 - [ 19]
Originally by: CCP Lingorm
Originally by: Lazuran
Originally by: CCP Lingorm
The limit is that a Sol process is limited to 1 CPU, and this is in our software, not an OS limitation so this would not change if we switched OS's.
Sounds like this is easy to fix then. Get a very good C programmer (or whatever it is) to make this multithreaded. Even if you don't get it to scale well, you'll be much better off with current hardware with 4-8 and more cores per system.
Not written in C++. It's in Python like most of EVE and it is very hard to Thread. Do you want a threaded combat engine ... I don't. It has to be handled sequentially or it will cause to many errors.
If one thread completed processing before another thread that started sooner and this was then applied (say damage from my guns) but then the thread from you damage came back and said ... my ship was destroyed we have a problem.
But moving other parts of the 'solar system' to their own cpu (like station services, etc) and then grouping those services will make a vast improvement.
I was thinking about this threaded combat system. You are right that it's not good to have a combat system threaded when you have a 1vs1 fight, but this is not true anymore if you have a many vs many fight. You only need a sequential system between ships that are actually shooting at each other (shooting ammo or repper). Ships that are not involved in the fight do not need to receive data before their thread end so they can be moved to another process. In the end you end up with a bunch of thread processing sequencial fights between "chain" of ships (few ship shooting at another ship which is being repaired by someone else for instance). The rest of the ships in the grid not involved can receive the update a bit later (maybe offloading that task to yet another process/machine). If you end up with a blob vs blob of 500 vs 500 ships there will be several fights going on at the same time so you can split up the blob in smaller pieces dynamicaly between all the cpu of the same machine (maybe between several machine?). That should improve things if you manage to get a grid to sit alone on it's server. If you manage to split each thread fight in a blob vs blob to another machine you can probably scale the system so it can handle any number vs any number of ships. Does that make any sense? :) |
shadime
Solitude Empires
OWN Alliance |
Posted - 2008.01.24 02:58:00 - [ 20]
multithreading fleet combat into their "granular blobs" will be very tricky. Essentially you need to have all ships which are in a "locked" network as they can affect each other. So say you have 1 network (in terms of having a lock on each other and thus being able to affect each others state) of ships that are interlocked (i.e. there is a path of "locks" between any of the ships within the network. Now imagine another of these networks fighting close by. I guess you could calculate these separately until one dude decides to remote rep his friend in the "other network". Then suddenly, the networks affect each other and need to quickly merge...sounds nightmarish to me.... |
Ashoka TG
Wreckless Abandon
G00DFELLAS |
Posted - 2008.01.24 06:44:00 - [ 21]
|
CCP Lingorm
C C P
|
Posted - 2008.01.24 09:28:00 - [ 22]
Originally by: Haulin'Hauling
Originally by: CCP Lingorm
Not written in C++. It's in Python like most of EVE and it is very hard to Thread. Do you want a threaded combat engine ... I don't. It has to be handled sequentially or it will cause to many errors.
If one thread completed processing before another thread that started sooner and this was then applied (say damage from my guns) but then the thread from you damage came back and said ... my ship was destroyed we have a problem.
But moving other parts of the 'solar system' to their own cpu (like station services, etc) and then grouping those services will make a vast improvement.
I was thinking about this threaded combat system.
You are right that it's not good to have a combat system threaded when you have a 1vs1 fight, but this is not true anymore if you have a many vs many fight.
You only need a sequential system between ships that are actually shooting at each other (shooting ammo or repper).
Ships that are not involved in the fight do not need to receive data before their thread end so they can be moved to another process.
In the end you end up with a bunch of thread processing sequencial fights between "chain" of ships (few ship shooting at another ship which is being repaired by someone else for instance). The rest of the ships in the grid not involved can receive the update a bit later (maybe offloading that task to yet another process/machine).
If you end up with a blob vs blob of 500 vs 500 ships there will be several fights going on at the same time so you can split up the blob in smaller pieces dynamicaly between all the cpu of the same machine (maybe between several machine?).
That should improve things if you manage to get a grid to sit alone on it's server.
If you manage to split each thread fight in a blob vs blob to another machine you can probably scale the system so it can handle any number vs any number of ships.
Does that make any sense? :)
Good in theory until you consider a few things. What happens if these different threads get out of step? If the Thread that is handling 1 group is small it will continue to process in real time but say there is another thread that has 200 people (100 per side) all shooting at each other, this thread could become slower than realtime and now the 2 threads are out of step. |
Gnulpie
Minmatar
Miner Tech |
Posted - 2008.02.13 16:13:00 - [ 23]
Originally by: Haulin'Hauling
I was thinking about this threaded combat system.
You are right that it's not good to have a combat system threaded when you have a 1vs1 fight, but this is not true anymore if you have a many vs many fight.
You only need a sequential system between ships that are actually shooting at each other (shooting ammo or repper).
Ships that are not involved in the fight do not need to receive data before their thread end so they can be moved to another process.
...
Does that make any sense? :)
Yes, it makes sense to a certain degree. The only question is if you have enough 'serial' mini-battles going on or if the whole battlefield is connected which each other. If you have the later case, then you wouldn't gain anything from your approach. Is suspect that there are too many ships available which breaks the 'serial' mini-battles: every ship which affects more than one ship like: jammers, webbers, smartbombers, interdictors, drone boats etc. |
All Channels
Question for the devs -> Infiniband and MPI
Screenshots & Art
