Software Development Practices at CCP | EVE Online

Software Development Practices at CCP

2007-02-01 - CCP Redundancy

WARNING: Highly technical geek-speak follows! Reading this may cause: itching, crazy hair days, an urge to wear a labcoat, or carry cutting edge technology products and as many pencils and pens as can fit in your pockets!

Cells

The software division at CCP currently stands at around 20 people, currently organized into 5 software cells:

  • Core Technology Cell
  • Graphics Cell
  • Game Cell
  • UI Cell
  • Web Cell

Core Tecnology: These are the people working on underlying systems in Eve. Our Python integration, support for multilingual clients, database stuff.

Graphics Cell: Works on the graphics engine and tools technology. Works closely with the Core Cell, because the Graphics Cell is the "largest consumer" of things like python integration at a fundamental level.

UI Cell: Works on the Eve UI. There is actually approximately as much code for handling the Eve UI, as there is underlying C++ code for the whole framework.

Game Cell: Responsible for "Game Systems", such as the combat engine, corporations, alliances, dungeons, agents, missions, deadspace...

Web Cell: Responsible for all of our web technologies, which is far, far more than just the forums (includes billing systems, GM support systems, web store, ...) They've been doing a lot of under-the-hood stuff recently.

Different software cells have slightly different ways of working, depending on the context of the work. The graphics cell tends to do the most peer-review, because we tend to be working in C++ on new features in very closely related code - it's very important that we're in-sync with each other. Peer-review means that we sit down with each other and go through the code that we've been writing, increasing our familiarity with different areas, and trying to spot issues before the code is submitted.

We're starting to do much more peer review everywhere else too, and improving our coding standards further. Coding standards help to keep everyone on the team writing code that is more readable by other members, and along with good commenting and documenting procedures makes the code much easier to maintain.

Code Versioning

We use Perforce to maintain the current and past versions of our code and content - this is incredibly important in large scale software projects. We need to enable people to work synchronously, see who made changes to code, what the reason for them was, and exactly what they were. This allows us to quickly 'revert' changes that break something, as well as have multiple people working on the same files at once, by allowing them to perform 3 way code merges to combine the changes of another person with their own before they submit them.

Each change that someone submits is a standalone unit that can contain multiple files and is called a 'Changelist' - the idea is that you submit a change that comprises a full logical change to the entire code in one go, so that you can revert it all in one go, and also not run into problems with changes that are dependent on other changes not being submitted at the same time.

We also make heavy use of a concept called "branches". You can imagine the code as a whole as a line that goes through time, and changes internally as people submit changes to it. But what happens if you want to split off from the main code for a while, and work on a whole sequence of things without being disturbed by other people checking things in? How would we keep a stable set of code for release to TQ, while doing huge redevelopment of the graphics engine with multiple people collaborating? The answer is branching the code.

As you branch off the code, you effectively take a snapshot of the code at that point in time, and create a copy. Anyone working on that branch, is only making changes within that branch, not to the code that it was branched from. When you want to bring those changes back into the main branch, you do an integrate back from the new branch.

You can take a number of different approaches to branching - off the top of my head, you can do it by release, by project and by user. We do a bit of everything - The Kali and Vista projects have their own branches, and there's a "Staging" branch from Kali, which itself branches into a "live" branch. I have my own "sandbox" branch, just to try random things out in.

Why so many branches? This way, we know exactly which changes are going in, and can manage which fixes can make it into any hotfix or patch for the release code. We can impose different levels of required code stability on different branches (Vista even has a VistaExtreme branch where nothing is guaranteed to work at all) and focus our testing on the code that's required to be solid.

the best way to think about the colour coding is probably this way:

Of course, the threat level is less to do with the chances of buggy changes, and more about the alertness level that is required, or the threat that they represent to that branch.

Defects

Dealing with defects is naturally a large part of the job description and work performed by the software division, but I'll leave it to Mephysto to cover this at some point, since he can give you a better high-level picture of this than I can.

Jobs!

CCP is hiring. We don't care if you don't have "Industry" experience, we just want great programmers who can work well in teams. Although we're not currently advertising for the positions, we are almost always interested in applications. My personal advice is to keep your CV short (1 page, all relevant!) and try to make sure you have a piece of sample code available for any further interviews, where you can explain what you were doing and show good coding style. Beware of claiming competency in a language you're only slightly familiar with!