Testing, Automation, Monitoring and Benchmarking OH MY!

Following on from my mention in Tanis' blog I thought I would introduce myself and give you some more details on what I am doing here are CCP.

I am the Team Leader for the newly established QA Engineering Team.

So what does that mean?

Well that is a very expansive question. For some time we have had an automated test that logs a client in, runs a set series of tests and records the results to a database. We now have an internal tool that allows us to graph the results and compare them to other builds, examine trends and make sure that we are getting more out of the client.

This is expanding and we are getting a dedicated group of machines to perform these tests. The machines are not top of the line but are a selected set of hardware that matches our 'Minimum Specification' and our 'Recommended Specification' and will have a number of different Operating Systems installed on them to cover the different OS's are supported by EVE.

Once these are in place we will be adding additional machines to simulate a cross-section of our player hardware, so we can get performance stats for the commonly used hardware combinations.

Finally, we want to use all the PCs in the office after hours for even more testing goodness. This will allow us to reach the 300-500 user target that Tanis mentioned, which is also significant for a number of other reasons. It has to do with Statistics and sample significance and other such cool things.

Monitoring 'Agents' on Tranquility

You will soon get to see this on Tranquility. We have a new set of 'agents' that will be flying round EVE recording the performance of EVE. This data will then be used to compare the new development builds against to ensure performance gains. All of these 'characters' are members of the 'CCP Space Monitoring Service' so that they can not be impersonated and will be flying Polaris frigates to prevent them from being targeted and from targeting players.

But that is not all!

We will also be looking to monitoring, on a more fine grained level, serverside performance. Initially this will be limited to test servers as we do not want to adversely impact performance on the Live Servers. We are currently looking into software packages that provide this functionality as it will allow us to get this up and running quicker than a custom built solution.

Wow, that's great but what does it mean for me?

What these changes mean is that we will have even greater confidence that the 'performance' fixes we deploy will improve performance, as we will have the ability to compare the performance of the old code with the new code and look at the hard stats. I can see you about to ask "Aren't you doing that already?" Well yes and no. Yes we compare the old code performance with the new code, but that information is not persisted in any readily accessible system (think LOTS of Excel files open at the some time and flicking between them to compare, not exactly the most user friendly).

Is there still more?

Yes, actually there is. As Tanis mentioned we are planning upgrades to the bug submission system with more Dev interaction into the Game Development forum in an effort to keep the information there more up to date and relevant. Extensions to our testing script system to allow us to automate the testing of a greater variety of in game actions, allowing us to automate the testing of some bugs and the confirmation of the fix, then monitor future changes as we deploy patches, thus reducing the chance of old Bugs reoccurring.

Automated Regression Tests

One of the side effects of extending the test script system is it will allow us to automate some of the regression tests that get run against all major builds. This will free valuable tester time allowing them to concentrate on more complex tests. In addition these tests can be run over night, every night to validate the latest code changes and allow testers and programmers to see any failures first thing in the morning.

I can see some of you asking why we are not doing this already. Well complexity mainly. We are doing some unit testing on individual code components but it is very hard to develop a true unit testing environment for EVE because of the software structure. We are using them where we can and the work my team will be doing will allow us use them even more.

Load Client

We are planning a trimmed down client that can have many (think 20-50) clients loaded per machine that will allow us to better simulate player load on the test servers, thus allowing better testing of code under load. Think Jita simulations and fleet battle simulations.

Sound good?

I hope so. We have put a significant amount of effort into the planning of these changes, but what comes next is the really exciting part, actually doing it and here is where you come in. I need more people, particularly good programmers so if you have not already sent your CV to CCP please do so. In the near future we will also be asking you to submit information on the configuration of your systems so that we can see what are the common hardware configurations in use.

The Future, or what's after that?

At this point things are still settling down in my mind as to what will be happening after all that, but I do have some crazy ideas like:

• Publishing hardware performance to allow you to make more informed hardware purchases
• Further extensions to the Automation Script language to allow more tests to be automated

Lingorm signing off.