Former Rockstar Vienna dev Jurie Horneman responds to Criterion’s front-page Develop feature
The cover article of Develop issue 120, about how Criterion handles crunch, can be reduced to "You can deal with crunch by investing in your build infrastructure".
Before I argue why I think this is very, very wrong, some caveats: I am, in fact, a huge fan of automated build systems, continuous integration, automated testing, and general automation and metrics-based analysis of the game development process.
I first worked on an automated build process during the porting of GTA3 and Vice City to the Xbox, at Rockstar Games in 2003. At Mi'pu'mi Games, the company I co-founded in 2009, we use automated build processes, continuous integration and unit testing as a matter of course.
Having worked on these processes, I know how hard they are to set up and to keep them running, especially on big console projects. It's tough to even find the people who want to work on them: it's unglamorous yet crucial.
So I am aware of the effort it takes to do what Criterion has done, and I agree with the direct advantages of the build infrastructure that Paul Ross mentions in his article.
However, I do not agree with the connection between build infrastructure and crunch at all. Crunch is not a technical problem. It's an economic problem, and a people problem.
(While I'm going to be picking on AAA console development, much of what I'm saying also applies to AAA PC development.)
Fundamentally, crunch is very straightforward. You have a team, a deadline, and a game, meaning a certain amount of features and content executed to a certain level of quality, that you want to make.
(This can be represented by the scope-time-cost triangle from project management - see here. It's a simplification, but a useful one.)
Only it turns out that your team cannot finish the game before the given deadline. So you make the team work harder: you crunch.
I assume it's clear why crunch is bad. There's a number of good reasons in Paul Ross's article. I'm not even going to talk about how you can manage crunch, even though we developed some good policies at Rockstar Vienna.
No, what *causes* crunch? Crunch is a bad solution to a problem: Why do we keep picking it? Aren't there are other solutions?
There are. Let's look at that triangle again. If you can't achieve your desired scope with your given time and cost, well, can't you just change one of those three factors?
Increasing cost is hard for two reasons. First of all: people like money, and they like to keep their money. So when you go to the person with the money and say: "Look, I know I said I needed X million a year ago, but gee, it turns out I need Y more," you can understand if they're not very willing to give it to you. If you need more money, the odds of them getting it back go down.
The second reason is: even if you *had* money, it probably wouldn't solve your problem. You can't just add people and expect things to go faster. This is one of the oldest principles of software development (it's called Brook's Law, after Fred Brooks - see here).
I won't go into why it doesn't work or how you can sometimes make it work. In general, adding money is not the solution people pick.
If adding money doesn't work, can't you move the deadline, or reduce the scope? Yes, you can. In fact, those are excellent solutions. So why don't we use them?
Well, moving the deadline has a couple of downsides. First of all, your team needs to get paid every month, so if you work longer on the project, your costs go up. So you're back to having to ask someone for more money.
The other downside is that in AAA console development, hitting your deadline is very, very important. There are an enormous amount of things that need to be orchestrated for a AAA console game release.
The consumer press and the public need to be whipped into a frenzy so that they're ready to buy your game the day it comes out. DVDs need to be pressed onto discs in factories and put into boxes and transported to stores. All of this is expensive, and changing when it happens costs tons of money.
What about reducing scope? Well, that would work, except you've told the public which features and how much content will be in your game, and you don't want to disappoint them. They'll be paying $60, after all. Plus, some other AAA console developer is making a similar game, and if *they* have more features and content, players will flock to them. So you have to deliver.
If you choose to play by those rules.
Rockstar Games, my former employer, famously doesn't. They make games that are different from others', they take as long as it takes to develop them, and they don't announce a game until they're very sure they can release it when they say they will. They value quality over time and cost. The same goes for Valve and Blizzard. They release games when they're done.
I don't know if people crunch or not at Valve or Blizzard. I know we had projects at Rockstar Vienna that were low on crunch, but that doesn't mean they were low on stress. But in any case these companies show the triangle can be bent.
Of course few companies are Valve, Blizzard or Rockstar Games, and you can't eliminate crunch tomorrow by emulating them.
Let's look instead at online games. These games, once they are playable by the public in some way, are not, in fact, projects. They are operations. They are services, not products. This is a very hard thing to understand for people from the project-based development world. You don't have hard deadlines, you just manage priorities, trying to improve the game as best as you can, every single day. It's stressful, and sometimes you have to scramble to fix a server problem, but the causes of crunch are not programmed in, the way they are in AAA console development.
What about indie games? If you are going to distribute the game yourself over the internet, you are no longer dependent on factories and trucks, so it's a lot easier to move that deadline, if you can afford it.
(But wait, won't moving the deadline screw up your marketing plans? Only if you use launch marketing, which is what AAA console publishers use. Read this blog post by Nicholas Lovell about other options you have as an indie)
At Mi'pu'mi Games, we know we can't predict exactly what we need to build or how long it will take, so we try to deal with that uncertainty.
Despite having five certified scrum masters in the company, we don't use any particular methodology, although our approach is definitely agile. That agility allows us to adapt as we learn more about the game we're building.
It's not always easy to combine this with milestones in work-for-hire projects. We've had stressful periods, we occasionally work late (but not very late) and sometimes someone even works on a weekend.
But on the other hand we developed 5 Nintendo DSi games in 6 weeks, and that involved exactly half a Sunday's worth of crunch, for three people.
On a side note: if we wanted to crunch more, Austrian labour law would make it more expensive than in the UK, so that gives us an extra incentive to avoid it.
But this article is not an advertisement for Mi'pu'mi Games. Why can indie and online developers avoid crunch when AAA console developers can't?
Because AAA console development is like Formula One racing: incredibly expensive, fantastically complex, and insanely competitive. And it has been getting more expensive, complex and competitive for over 15 years. Which is why companies have gone bust, developers have left to start new, smaller companies, and the survivors have very little margin of error left.
At the beginning I said crunch is an economic problem and a people problem.
It's an economic problem because AAA console developers have to crunch to be able to play where they're playing. (In a red ocean.) The economic incentives of the people who decide that the team must crunch (and who often crunch themselves) are simply set up that way.
It's a people problem because of our psychological limitations. You want to make the best game possible. You want to think things will work out. Crunching worked last time, right? (If you survived it.) It wasn't that bad. We've made better plans. This time is different.
Better build management is like shaving a tenth of a second off changing tyres during a pit stop. It doesn't prevent crunching: it prevents wasting time. The deadline won't move. The budget won't grow. The scope will, as soon as your competitors also improve their build management.
I am convinced that as long as the business model of AAA console development is set up the way it is, AAA console developers will crunch. But it's not inherent in game development.
Jurie, you're awesome. Later I'll post a more lucid response with discrete point-by-point analyses and even a tiny rebuttal or two, but I had to get that out there.
Well written, Jurie. And I agree with the main point; crunch is mainly a scheduling issue, not a tools issue.
This is what happens when you try to make an inherently creative activity into an assembly line; it just doesn't work well.
It's really a tough one because it's so difficult to get an accurate reflection on this matter.
What is the specific calculated change in productivity?
And the problem here is that programming is not a laborious task, it's one of creativity. Each programmer is solving problems, and although there are systematic processes involved, each programmer is still solving a problem creatively in their own unique way.
I've experienced dramatic falls in productivity through sleep deprivation, but I've also experienced highly productive states of mind during crunches. And especially when under immense pressure I've managed to quickly create elegant solutions to problems with little effort that one would typically allocate a few days to complete.
But going back to my point, it is the lack of facts detailing the negative or positive effects of crunch that allows only the positives to be taken into account when making decisions in upper and middle management. And what's worse is that, being a creative problem solving process I suspect that results from experiments may be misleading without a thorough understanding of all the factors that need to be controlled.
Really a total nothing article. An unknown Austrian developer, whose career highlight seems to have been working for a failed studio and then some DSi games completely fails to offer anything new about how you can tackle crunch. The original article at least attempted to give practical advice on how to avoid crunching and a case study of how that worked. This just offers "I am convinced that as long as the business model of AAA console development is set up the way it is, AAA console developers will crunch" but fails to offer any advice on what that change should, or even could, be.
It's to easy to sit on the sidelines criticisng especially when you offer zero on what changes you would make.
It's opinionated people like this, that talk so loudly, but deliver so little that are killing the European games industry, in my view with big promises and failed projects / studios.
Total waste of 5 minutes of my life reading it.
Thanks Keith and Jess.
Because some other people mentioned it: I did not want to suggest Rockstar Games doesn't crunch, just that deadlines can be moved. Maybe Rockstar was a bad example.
Also, thanks Peter. The 15 seconds I took to read your comment weren't wasted. I had a good laugh. Guess I hit a nerve, eh? :)
Peter, you make a good point. The only people who should be allowed to have opinions are famous people and rich people.
I think you may have misunderstood the article:
"The cover article of Develop issue 120, about how Criterion handles crunch, can be reduced to "You can deal with crunch by investing in your build infrastructure"."
Is not true. You can deploy automation onto a team and still have a torrid time, as I've witnessed. The key tenant is:
"Our secret to alleviating crunch is to make every second of development count"
Simply deploying automation and crunch evaporates is not true.
Getting a AAA game done takes a finite amount of time to do. If you can remove dead, down time from development then you can shuffle the crunch hours into the slots left by dead time. This eases crunch.
"Only it turns out that your team cannot finish the game before the given deadline. So you make the team work harder: you crunch."
By removing blockers to production, like bust builds, things not working, diagnosing difficult bugs you arm the team with a much higher probability of getting it done.
You also missed the 2nd big idea about measuring your team's velocity and taking action to keep it high which is just as important.
That is what we did on NFS:HP and it worked for us.
I absolutely agree, you can't simply deploy build machines & auto testing and crunch dissapears, it isn't as simple as that.
I would absolutely love to see more posts on practical advice on how to kill crunch. As I said it feels it's one of the biggest blockers to AAA production.
Obviously everybody can have opinions, I just question how misguided they are. When I look @ Jurie's history there's nothing there but failure. Perhaps rather than writing articles for develop he would be better of spending his time making games that are any good ?
Sorry Peter, I don't recognise the name. Who do/have you work for again?
Well he's either a turte zooologist, or an imaginary character from an AAA title (Deus Ex). Either way, if he's so keen on proving that this is a waste of time, he seems to be gathering more waste of time. I enjoyed reading it, and Im glad that I'm not part of any AAA console or PC titles.
Paul,
let me try to explain what I didn't understand about the original article.
You're saying eliminating unproductive (dead) time reduces crunch (that's what "easing" means, right?).
Which means that on previous projects, without better build automation etc., you've crunched because of this unproductive time. (Or at least you would have had to crunch without better build automation - it doesn't matter either way.)
And that is what I don't understand. Because that HAS to be a choice that someone at Criterion consciously made.
And then I have to ask: vacations and sicknesses are unproductive time. Why aren't you crunching because of that?
Even something as nebulous as time lost due to bad builds can be estimated. How is it different from estimating the time it takes to fix bugs? Or, again, time lost due to people being ill? It's not very hard to say: right, we're losing a person-day per month due to bad builds, let's multiply that by the number of persons and months and add it to the end.
There are a lot of possible reasons for deciding to crunch instead of moving the deadline. (I'm not saying they're necessarily good reasons.)
In my opinion, unproductive time can only cause crunch because of unforseeable accidents, or because of constraints on time, scope and money. My guess is you have those constraints at Criterion. In my blog post I tried to point out these constraints and argue why they are often seen as unmoveable in AAA console development, but need not be, at least not in game development in general.
Jurie, I'm afraid you're completely missing the point. Criterion's description doesn't imply they're using tech to solve a problem. What they're using is good management to allow the cyclical nature of software development to proceed with minimal interruption.
Fundamental to software development is the edit/compile/test/debug cycle. The faster you can iterate through that loop, the faster your code development will operate. It's somewhat similar to the OODA loop famously described by John Boyd.
Criterion are doing exactly what good management should be doing. They're building an environment in which the efforts of many people can be coordinated in an efficient manner. Poor management tries to play auteur - too many producers think they're film directors - instead of solving the complex problems of financing, resource management and marketing.
Explain to me why time saved due to good management is somehow less valuable than time you gain by moving the deadline. That's nonsense. Software engineers in particular are motivated by the positive feedback of their coding actually doing something. I would contend that slowing down that feedback loop does more than waste time, it actually reduces their enthusiasm and makes them less productive as a result.
Hi Parrish,
I pretty much agree with what you are saying, and I think I made it abundantly clear that I think what Criterion did is really great, as a way of reducing wasted time. It's the link to crunching that I find strange.
"Explain to me why time saved due to good management is somehow less valuable than time you gain by moving the deadline."
I can't explain that since it makes no sense, Luckily, I said no such thing :)
Since I missed the point of the article, why don't you explain to me why reducing wasted time is a good solution to crunching? And why wasting time because of a non-optimal build process, say, is a good enough reason to make developers crunch?
> why don't you explain to me why reducing wasted time is a good solution to crunching?
lol, love it! By now everyone must agree you are on some sort of wind up Jurie, n'est pas ? How is your new studio going ? Has it failed yet or still just about struggling ? Presumably you are opening E3 2012's Nintendo / Sony / Microsoft conference with some ground breaking words of wisdom ? lololol
