MCV/DEVELOP News, events, research and jobs from the games industry
It’s the largest single-server MMO in existence. It has over 300,000 subscribers, each inhabiting their own corner of the 5,000 solar systems that make up the giant game world. It gets two massive expansion packs added per year, in addition to regular patches.
EVE Online’s vast scale means that maintaining and extending that universe is a huge challenge for developer CCP. With offices in Iceland, America and China, the volume of activity means that the code base and assets are constantly being added to, while existing data is being polished.
“We use the staging streaming model: main, staging, release,” says Berglind Rós Guðmundsdóttir, software engineer/configuration manager at CCP. “Similar to how websites are developed and deployed, this is a good fit for MMOs, where we need to regularly deploy patches and at the same time develop the next expansion to the software.”
The team uses Perforce to ensure the smooth-running of this model, and to track art assets, marketing materials, website code and content.
“Having an SCM system is crucial, because it allows us to selectively release updates from our development codeline and keep track of how the code has evolved over the last five years. It also gives us the option to work on projects in a separate codeline until they are stable enough to be brought into the main development codeline,” explains Berglind.
With almost 200 developers spread between Reykjavik, Atlanta and Shanghai, SCM facilitates collaboration and flexibility. “The different teams work on various projects, sometimes co-operatively, sometimes independently. Projects that are ready for release are then brought together for rigorous testing before release. SCM is important because it enables dispersed teams to work in a single code base, as well as being able to branch out when appropriate.”
APOCRYPHALIPSE NOW
To give us an example of how SCM plays an integral role in developing EVE Online, Berglind explains how the recent Apocrypha expansion saw the Shanghai and Atlanta teams join together to achieve ambitious aims.
“By using Perforce proxies to cache files and reduce the network traffic across the globe, we were able to have all the teams working in the same codeline and on the same server, without too much latency. It helps us deal with remote worker collaboration.”
Perforce proxy is a self-maintaining server that caches versioned files for re-use on any local network with remote access to the Perforce server. Any number of proxies can be quickly deployed without requiring additional hardware or software at the main site. CCP has one in each of its three development locations.
“In the case of the Apocrypha project, this meant all teams were working with the latest code. Continuous testing of the expansion was practical and this helped detect any issues as early as possible,” added Berglind.
“We had a team working on improving fundamental parts of the game and it needed to be able to check-in intermediate steps that could break the game client. This could have significantly disrupted the development work in other teams, so this team worked in a separate codeline that was integrated at defined delivery points.”
PAST TIMES
While future development is naturally an important area of focus, Berglind also stresses the need to maintain accurate information about code and other assets that are no longer being developed.
“With CCP’s mature codebase, it’s very important to track the change history of files and pinpoint when and why changes were made.”
As far as development work is concerned, CCP has seven active codelines in use daily. The SCM repository also includes data for the websites, marketing materials, utilities and various sandboxes. Test branches that allow developers to modify code without affecting the rest of the team, allowing the developer to try out scenarios and create experimental designs. The total size of the SCM repository is 300GB and the metadata is about 10GB.
Perforce has been installed within CCP since 2004 and is used by all the developers, technical artists, deployment teams, and individuals within the test and marketing teams, currently covering 220 user licenses.
“There are many reasons why Perforce was chosen, but one of the main ones is that it’s really fast, even with large binary files. In games development, speed is important,” says Berglind.
Ensuring consistent work practices across multiple users is a challenge for any developer and CCP uses triggers within the SCM to enforce rules about how code is treated. Berglind says: “We have used the trigger feature to enforce the rule that there shouldn’t be any tabs in the code, only spaces. We also use triggers for check-in rules, for instance when certain files are changed, the changelist description must contain a reference to a work item.”
Changelists are another feature that CCP uses. “The concept of changelists is where changes to multiple files are grouped together. It helps us track files and change history.”
Although Perforce is designed to provide users with comprehensive tools, its range of APIs also enable companies, such as CCP, to tweak Perforce to suit its own requirements. “The flexible API means we’re able to create scripts that perform various actions against the Perforce server, for example in production cycle.”
Clearly, SCM has helped CCP to become one of the most innovative developers of games in the world, and as EVE Online continues to go from strength to strength. However fast users and content grows, SCM will help to ensure that development and release cycles are kept on course.
