Code to Joy

As audio designers, we have the potential to create the best sounds in the world: we have the full scope of creativity at our disposal, and we are no longer so limited by low sample rates, poor compression and severe memory restrictions.

But the story doesn’t end with the creation of this sonic gold: how are the sounds played? How are they used in the game world? What other aspects of the playing of this particular sound might contribute to how it works both functionally and aesthetically?

Early games were made by programmers who created the whole game – the gameplay, the art, the animations and the audio. As games have developed, all these elements have become more specialised: a higher quality was needed than could be delivered by any one person, no matter how talented. Programmers specialising in audio have always been important, but with the power and sophistication available to today’s developers, they are needed more than ever.

PROG ROCK
Take a very basic example such as footsteps. These are determined not only by the size, weight and clothing of the character, but also by the surface types they’ll be walking on; then comes running, skidding, jumping, landing, and blending between any and all of those. Designing how your sounds will be played is integral to the work of an audio designer, but the implementation of those design decisions is generally down to an audio programmer. Without good support from a role of this kind, the best sound design can ultimately be reduced to a buggy mess.

A major part of the audio programmer’s role is to give back as much freedom and control to the audio department as possible, just as in other areas of production the new tools being developed are giving artists, animators, level designers and editors more creative power.

Most advances in audio – as indeed with art, animation and so on – are driven by the demands of a specific game. Very few developers have the luxury of researching offline advances without the pressures of a paying project. Wise developers, however, while creating the particular functionality for whatever they are currently working on, will simultaneously think ahead and construct it in such a way that their work can be built on for what might be needed or possible in the future.

BLITZ CREED
At Blitz we’ve been lucky enough to have great audio support over the years from our dedicated technology team. Back in 2002, we created our own interactive music system for the BAFTA-nominated Taz: Wanted, which is a good example of creative tools coming from a programming background. These games required audio to be streamed from the disk and cross-faded during play; faster switching between music and dialogue, and multiple dialogue streams, were also becoming commonplace.

This in turn required support for features such as the ability to cache and read ahead, as well as handling multiple streams. All of this functionality was written into our proprietary technology suite BlitzTech, but at this time it still required a lot of collaboration between the audio designer who created the sounds and a programmer on the game team who triggered the sounds at the right time, at the right levels, in response to the right stimulus. This was time-consuming and frankly tedious, so we began to look at ways of giving more control to the audio designer.

One of our tech team programmers already had a lot of experience with audio coding, but crucially had also worked with audio creation software. One of his most important contributions was to give his tools user interfaces that mimicked those of sequencers and other software with which the audio designers were already familiar. This meant the audio department could work faster and more easily, which in turn encourages experimentation and results in better quality in-game audio.

 The interactive music system created back then proved so useful that we used it in many more titles. A different challenge was posed by our first Volatile Games title Reservoir Dogs which, like the film, doesn’t feature any non-diegetic – i.e. score – music at all. Instead all the soundtrack is placed within the game world and comes out of radios or other similar sources. This required considerable changes to the underlying structure of the code, as did the more advanced use of DSP (digital signal processing) effects to help achieve this. Invincible Tiger: The Legend of Han Tao also benefits from this as it needs a lot of realtime effects (distance filters and the like) and also boasts a very complicated interactive music system.

SING++
The Karaoke Revolution games, originally part of the American Idol series but now a franchise in their own right, again posed new issues for us. There’s nothing inherently difficult about supporting microphone input, but the initial challenge here was the waveform analysis which determines the pitch analysis, which has to be done on the fly to gain the correct feedback from the in-game judges and audience.

Part of our solution to this was to experiment with using more than one core. We found quite quickly that, because video and microphone are so time-critical, they performed much better when balanced across multiple cores. Again, a simple enough solution in theory, but in reality this required a lot of trial and error and some pretty nifty coding, as all of these elements have to work together, completely smoothly. No one is going to be too worried if MS Word hangs for a split second before continuing, but when you’re singing along to your favourite track, that’s really going to stand out as a bad bug.

While some people might argue that the growth of middleware solutions is a consequence of the lack of audio programmers, it’s more accurate to say that both can make a great contribution. Talented people with skills and interests in both programming and audio can work with middleware to empower the designers, and also write their own custom solutions to problems thrown up by any specific project.

So mature specialist systems such as FMOD can bring a lot to the implementation of great game audio. For example, we mentioned DSP earlier: newer general purpose processors have sufficient power to process audio signals in realtime and effects such as reverbs are implemented in this way, as opposed to being implemented as dedicated hardware processors. Programs such as FMOD and the underlying console interfaces allow us to create our own customised DSP and ‘chain’ it into the system, allowing us to perform any computation we want on the audio data as it passes through.

This had always been considered too high-end a subject for game audio, but it’s now reality at the runtime level and needs a very good grounding in maths and audio to really understand it.

This is one area in particular where certain kinds of people can make a bit of a niche for themselves. Even someone who may not have any experience in games programming but has a deep mathematical basis in signal processing and code optimisation would be ideal for producing high-quality custom audio effects.
So middleware or not, it’s absolutely true that the games industry wants and needs more audio programmers. It’s the hope of this article that more people out there who share a passion for audio and a passion for programming will consider this most rewarding and fascinating of careers.

www.blitzgamesstudios.com

…

Matt Black is audio manager at Blitz Games Studios, and was one of only 40 musicians chosen from around the world to study at the Liverpool Institute for Performing Arts, a small unique institute founded by Sir Paul McCartney. Matt was not only in the institute’s first graduating class, but one of 5 musicians picked by the ex-Beatle to work on a one-on-one basis on a song writing project. After this Matt went on to study music at Bournemouth University and since 2002 he has been creating audio for Blitz’s games.