Procedural audio with Unity

In the video games world procedural audio refers to the computational process of generating audio from nothing, or almost nothing.

The goal is to use almost no wav data (pre-recorder audio files), but rather models that generate in real-time the equivalent audio data that would be contained in pre-recorded files.

Procedural audio has many advantages: it saves memory by using code instead of wav data; it is flexible (computation can depend on all game parameters); it can lighten the burden of specific tedious tasks (sounding hundreds of interacting objects or animations) and it brings flexibility to the pipeline (since sound is directly linked to game parameters, changes will modify audio generation accordingly to keep it synced, avoiding the need for sound re-design).

Starting from version 3.5, Unity3D offers support for procedural audio. We took advantage of this during the pre-production of HolyShield, a third-person shooter targeting mobile platforms, co-produced with our top-notch partner: Dog-Box studio. The main character of the game has among its inventory, a flame-thrower.

One of the great possibilities offered by procedural audio is to have animations tightly linked with sound. Our goal is to create a procedural audio ‘flame’ sound that can be modulated by the character movement. In this short tutorial we’ll demonstrate some tricks on how to achieve this result. We’ll implement a basic ‘flame’ sound and show how to use simple unity features to implement efficient effects that are always perfectly in sync with the animation.

First Step: Send Data to the audio engine: The onAudioFilterRead method

To be able to generate sound from nothing we need at some point to write our own audio data and pass it to Unity so that it’s interpreted as audio to be played by the sound card. To do this in Unity we use the recently opened onAudioFilterRead method which is documented here.

In the following code example we compute a sine wave playing back a simple tone at 440Hz (which corresponds to the music note A, or to the tone you got when picking up a land line phone, for those of you old enough to remember):

using UnityEngine;
using System; // Needed for Math

public class Sinus : MonoBehaviour
{
// un-optimized version
public double frequency = 440;
public double gain = 0.05; private double increment;
private double phase;
private double sampling_frequency = 48000;

void OnAudioFilterRead(float[] data, int channels)
{
// update increment in case frequency has changed
increment = frequency * 2 * Math.PI / sampling_frequency;
for (var i = 0; i < data.Length; i = i + channels)
{
phase = phase + increment;
// this is where we copy audio data to make them “available” to Unity
dataundefined = (float)(gain*Math.Sin(phase));
// if we have stereo, we copy the mono data to each channel
if (channels == 2) dataundefined = dataundefined;
if (phase > 2 * Math.PI) phase = 0;
}
}
}

This Sinus script generates a sine tone at 440Hz. Calculated values have to stay between -1 and 1 to avoid distortion. To hear it you’ll need to associate it with an object in the object tree. Any object will do, even the camera if that’s the only object of your scene. Don’t forget to add an Audio Listener to your scene if you want to hear something.

Also note (directly from Unity’s documentation):
The default samplerate is dependent on the platform (typically 44.1k on OSX, and 48k on windows). You can query (or set) it with AudioSettings.outputSampleRate. The buffersize (default 2048) is also accessible thru AudioSettings.Set-/Get-DSPBufferSize().

Channels are interleaved and if you fail to fill the buffer, it will just stutter as no data is fed to the hardware.

Second step: A simple procedural fire

For this tutorial we inspired ourselves with Andy Farnell’s procedurally-generated fire. The essentials parts are a filtered noise plus distortion. We’ll take advantage of Unity’s modularity and its existing audio methods. We’ll write a white noise generator and use the filter and distortion from Unity3D. The pseudo DSP chain that generates fire looks like this:

White noise => Low-pass filter => Gain => Distortion

The following code is a white noise generator:

using UnityEngine;
using System; // Needed for Random

public class Noise : MonoBehaviour
{
// un-optimized version of a noise generator
private System.Random RandomNumber = new System.Random();
public float offset = 0;

void OnAudioFilterRead(float[] data, int channels)
{
for (int i = 0; i < data.Length; i++)
{
dataundefined = offset -1.0f + (float)RandomNumber.NextDouble()*2.0f;
}
}
}

Nothing special here except a simple trick taken from vintage analog synthesisers emulation: we add an offset to our noise. When used before a distortion it helps to shape the sound by making it asymmetrical. You’ll have to try and change the value for yourself to hear the effect.

Once we have our white noise generator, we need to add the low pass filter, the gain and the distortion. The filter and distortion are available as Unity functions, and a simple gain script example can be found here.

With all building blocks ready, we now need to add them – in order – to an object. For instance we can put everything in the main camera object. You’ll notice that all public values are available, giving instant access to specific parameters for real-time sound design. That’s a freebie.

Third site: Use animations curves to drive the synthesis

Since all the synthesis parameters are exposed via the natural MonoBehavior mechanism, we can drive all audio parameters we want directly from the animation. For instance: Noise offset, Cutoff Frequency, Lowpass Resonance, Gain Level and Distortion Level are interesting values you can control in real-time.

In this example we keep all values constant except for the ‘Level’ which is controlled by the green curve:

And here’s how it looks like in Holy Shield on the real animation curves. The purple curve being the automation of a parameter called “flamethrower power” (which in fact consists in the control of several parameters at the same time including gain, filtering frequency and distortion level).

You may notice the animation events (the small horizontal white arrows):

They are used to loop the flamethrower animation. We use the portions before and after the loop to fake the effect of the flame start and stop by generating short intense burst of flames which consists in the two spikes at the beginning and end of the purple curve. You may also notice that the flamethrower sound is constantly changing during the loop. This continuous change corresponds to the movement of the arm holding the weapon. It enables us to achieve a very realistic effect of the fire sound changing according to the arm movement in real-time. We’ll share a video showing the final result in action.

Further work

Of course, you’ll want to optimise the DSP and do some more advanced synthesis. White noise generation for example can be implemented in various ways, with different results in terms of performances. This goes beyond the scope of this tutorial but you’ll find plenty of valuable resources out there (books and forums mostly).

In our specific example, you might want to generate stereo sound (which requires uncorrelated noise for each channel) and you might want to link the sound of the flamethrower with the displacement of the player (using angular speed perhaps, generating changes in sound in the stereo field).

In HolyShield, for performance reasons, all of the code is written in one single script that includes noise, filtering, clipping and some other DSP tricks. If/when performance is an issue, it is also possible to write pure C code with hardware acceleration and use the unity plug-in feature to use it through the onAudioFilterRead method. (Documented here)

Conclusion

Procedural audio is very useful in many areas. Within Unity3D it is very easy to write c# code and test it on the fly. You can link it to animation curves or to other events. It makes Unity3D a very good sandbox for procedural audio experiments. And the good news is that it is efficient enough to be used in real games like HolyShield, coming soon.

You’ll find codes and complete examples as well as videos here.