Encore: EVE Online 'Core'

Engineering updates to EVE

We recently released significant engineering updates to EVE Online in Apocrypha 1.2 and Apocrypha 1.3. This work is actually not mentioned at all in the Apocrypha 1.2 release notes, and the release notes for Apocrypha 1.3 only briefly describe the exact part that was released in that particular update. There is a longer history anyway and more work was done in earlier releases, hence this dev blog to detail the story and the results.

Let's start with the results

Before we released StacklessIO and EVE64 last year, Jita would reach a maximum of 800-900 pilots, often with a fair amount of lag. After those releases, Jita would reach 800 pilots with no lag and was able to manage about 1,000 pilots with little or moderate lag. It even spiked to 1,400 pilots but with a lot of lag.

The Sunday after the release of Apocrypha 1.3, Jita climbed to 1,150 pilots with no lag. In fact there was plenty of CPU and memory to spare and you noticed the increased performance. We estimate that Jita can now handle 1,400-1,500 pilots with little or moderate lag.

More importantly, there was no lag on Monday morning after this record lag-free Sunday. Jita and other major trade and mission hubs have long suffered from a "hangover" issue that we have referred to internally in EVE Development as "The Jita Problem", where performance has decreased over time.

Here is a graph of the average CPU usage on the Jita node for a month. I'll let you, the reader, guess when Apocrypha 1.3 was released. It's pretty obvious.

Here is a more detailed CPU comparison for one day, the same weekday before and after the release with Jita topping out at 1,000 pilots in both cases.

So how did we do this?

CCP has a technical vision of sharing game-agnostic code between different projects. We have a Core Technology team that works specifically on such code, e.g., the Trinity graphics engine, and then the game teams, such as the EVE Development team, integrate that code and build game specific functionality leveraging as much of the CCP Core Technology Platform as possible. When writing new functionality, the game teams are encouraged to separate the game-specific code from the game-agnostic code and submit as additions and enhancements to the CCP Core Technology Platform. That way all the different development teams within CCP further all our projects. (True leveraging of global synergies.)

Of course the CCP Core Technology Platform, as stands today, didn't come into existence this way. It was originally written simply as a part of EVE and then later the game-agnostic parts were extracted from it. This process has come to be known internally at CCP as "corification", and the act of releasing Core Technology Platform enhancements to EVE Online is called "battletesting".

The history

We started this process a while back. The Trinity expansion was, in a certain sense, the first release containing "corified" code, where we overhauled the Trinity graphics engine. We then made various improvements to the setup of our code branches and the results were deployed in Quantum Rise. Then in Apocrypha we corified more code using the new setup, such as the audio engine.

Apocrypha 1.2

We had, however, only corified the C++ code (about 250,000 lines), and all the Python code (about 600,000 lines) was still left. It was not for lack of trying, as we had attempted to corify the Python code twice but reverted the changes and not deployed them for various reasons. Those attempts were not failures since they provided a valuable blueprint for the EVE Engineering team when we started 14 April on the third attempt, which simplified the work a lot in this final attempt. In addition to splitting the Python code into core code and EVE code, we also refactored a few systems and fixed a lot of defects (as you can read from the release notes). We then deployed Apocrypha 1.2 a month later, on 14 May.

Apocrypha 1.3

But we were not done. When we started this final corification push we also decided to refactor one of the most important parts of EVE - the Inventory System.

The Inventory System keeps track of the ownership and location of everything in the game. We refactored it to use a new database service backend to connect and communicate with the Tranquility database. This new database layer has actually been in use in EVE for two and half years but the biggest migration task, modifying the Inventory System to use it, was still left. We worked on that task from 14 April until we deployed Apocrypha 1.3 on 29 June. Replacing the database layer would yield noticeably less network I/O and memory usage than the previous system. The performance of most inventory related operations would improve, but there might be cases where performance would degrade. We expected a good performance gain on the whole though.

Whereas the corification effort involved everyone in CCP Engineering at some point in the process, then the Inventory System changes were almost exclusively done by CCP GingerDude and testing organised by CCP Oneiromancer.

But wait, there is more! If you call in the next 20 minutes then ... ahem, sorry, the lines got crossed there.

Replacing the database layer in the Inventory System allowed us to fix an issue that had affected major trade and mission hubs, especially Jita, without resorting to extreme hacking. We added a new index to the in-memory inventory storage to address "The Jita Problem", where performance had been degrading over time as more items were primed in local storage. Maintaining an extra index induces some overhead, but frees the server from having to search through very large sets in highly populated systems. Degradation of searches over time is now a function of the player population at a given location and not a function of the number of items in a location as before. This optimisation was released in Apocrypha 1.3 on 29 June along with the database layer changes.

As detailed above, the results were significant, obvious and noticeable. In particular, CPU usage in major trade hubs, such as Jita, was significantly reduced.

It's worth emphasising that these changes affect all of EVE, not just Jita, since the Inventory System is used everywhere in EVE, but the effects are most noticeable in major trade and mission hubs. We use Jita as a reference in this dev blog since it's easiest to measure and compare a known and stable quantity such as Jita.

The future

So, what's next, you may ask? Well, for the near term we are focusing on the Apocrypha 1.5 release scheduled for this August and our upcoming winter expansion. But fear not! We will continue to work on engineering tasks such as these in upcoming releases. As EVE grows we will continue to refactor the code and optimise it to allow more and more players to enjoy the one universe.