MCV/DEVELOP News, events, research and jobs from the games industry
Version management is pretty much a ‘given’ in most games studios today. How it is carried out though varies across tools and processes. Whilst there is not a single ‘one size fits all’ mantra, there are definitely some universal approaches we’ve come across from the developers we speak to. To dig a bit deeper, we asked some of the more innovative games studios we’re currently working with – from large, small, long-established and new – to share some of their experiences and advice.
We started with a fairly basic question: Why do we need version management anyway? Can’t you get the job done in other ways, without needing version management tools, especially if you’re a small studio?
There are other ways of managing files, of course, such as zip files and shared drives. But it is only viable for small teams who are very strict around file naming and who do not need to roll back to a previous version of their project. As soon as the game assets repository starts to build – let’s face it, in games development that can happen very fast, even with a very small team – then it can quickly become impossible to keep track of the latest version. That really matters if something goes wrong and the team need to go back to a previous version to find out what went wrong.
Stuart Yarham is Lead Programmer at The Chinese Room, a 2016 Develop Awards winner and the studio behind Dear Esther, Amnesia: A Machine For Pigs and Everybody’s Gone to The Rapture. He describes an earlier time in his career when he was working without version management. “When I think back to the past,” he says, “we used to zip and share files, which could make it a nightmare keeping everything in sync.”
Jakub Kutrzuba of CD Projekt Red – another Develop Award winner – adds, “Version management makes a big difference. If something is outside the version management system, then it can create multiple issues.”
Having argued the case for ‘proper’ version management, what else do studios need to be doing? One common thread that keeps cropping up is the idea of a ‘single source of truth’. An increasing number of studios are keeping source code and art files in one place, to make it easier to keep everything in sync and, if people make changes, to bring everything back together in one build.
This single view also helps everyone – coders and artists – to work as a cohesive team and The Chinese Room is a case in point. “In the early days of Rapture,” Yarham says, “all staff were working remotely, so we used Skype and Perforce to sync files and work together. Also, if we needed to go back a version to find out where something had gone wrong so that we could fix it, we could. That’s particularly important with binary assets, which you can’t just open up and debug.”
Kutrzuba of CD Projekt Red adds, “Everything should be versioned, that’s how you get transparency and a history of the game being built.”
Version management also supports a more flexible and creative approach to games development. “There’s a lot of talk in the industry about iterating and experimenting and that’s really important to us,” Yarham continues, “because creativity is such a big focus for us. People in the team can work on different ideas then come back together again. Version management is useful because you can try something out, then if it doesn’t work, you can try something else.”
Branching strategies
Branching strategy varies a lot between organisations. There’s a balance between having branches and accepting the risks that brings, compared to a single codeline with no branching which is very slow. One technique we advocate is to think in terms of components and break actions down into smaller chunks. Modify them, test them then bring them back together. This makes development and check-ins more manageable, even for companies who have a single main trunk of code. It also fits in with being agile, developing in increments and having tighter feedback loops.
Nick Penwarden, UE4 Development Manager with Epic Games, explains how his company manages branching. “Programmers are human and even great programmers make mistakes. Let’s assume that each programmer makes a mistake and checks it into the development branch once per month. If there are thirty programmers working on the team, on average the build will be broken once a day. If there are sixty programmers on the team, then on average the build will be broken twice a day, and so on.
"Worse, once a programmer has identified why the build is broken and fixed it, chances are good that another mistake has been checked in that has broken the build again. This leads to a situation where the development branch is broken more often than not once your team reaches a certain size.
“Our solution is to have sub-teams that are responsible for different areas of the engine development features and fix bugs in their own branches. Each of these team branches go through a QA process before the new code is pushed to the mainline development branch. Ensuring it is always stable for content development. The team branches also remain more stable for the programmers working in them because any change not made by that team has gone through a QA process before being merged in to it. With this process in place our content creators remain productive in the mainline development branch and programmers remain productive in their team branches.
“The primary trade-off to our approach is that the latency increases between when a feature is checked into a team branch and when the rest of the development team can use it. The benefit is that the feature, as well as any unintended consequences of the code change, has gone through QA before being rolled out. On balance I think it is the right trade-off to make and enables development teams to scale to much larger sizes."
A couple of further points to stress is the need to choose a system that is suited to all users, both now and in the future. Artists may not be as technical as coders, so the version management tool needs to be intuitive and easy for them to use. Third parties may also need to use the system so, again, usability for different job titles is often important.
Another piece of advice from CD Projekt Red’s Kutrzuba is that “Scale can be everything. If you think that the game and the team is going to scale, then it makes sense to version everything – art and code – from the very beginning and to choose a system that will allow you to do that, as well as have the ability to scale, however fast you are growing.”
“Don’t skimp on the set-up,” adds The Chinese Room’s Yarham. “Do the research, don’t wing it – get the foundations right from the very start. It’s much harder to change the setup when you’ve got a live development environment.”
Stuart’s point is a good one. You should plan the layout of your files in your version management system as well as on disk in the workspaces of individual users, also thinking of how these two locations are related to each other. Determine if and where you will store any build artefacts. Versioned artefacts can improve build performance through reuse and simplify tracking of issues from the finished product back to its source. Finally, versioning every single file that goes into your product will consume considerable hardware resources. Make sure to provide adequate CPU power, memory and disk space as well as a fast network.
I think that if I personally had to leave you with one piece of advice, it is this – have a ‘single source of truth’ for all your game assets. In other words have one single place for all code, art and other files. Because in today’s fast paced, increasingly complex development environments, that is the only way you can be sure to keep all versions in sync.
Sven Erik Knop is Technical Marketing Manager and Evangelist at Perforce Software
