IASIG AdaptiveAudio (Alex Brandon)

Some of us might say that video game music is finally coming into its own; EMI and other major record labels are starting to jockey to place their hot artists into the soundtracks of the best selling titles, or use their music in cross promotional deals. The movie composer Harry Gregson Williams scored the recent release “Metal Gear Solid 2: Sons of Liberty” (who we’ll be talking to later here in the IA-SIG website). Some of us may venture the opinion that video game music has finally found its voice, as journalists are comparing some of the latest soundtracks to movie and TV show quality. From the perspective of the public at large, video game music is certainly no longer necessarily hearable only as part of a game, but ‘good quality, well done’ music unto itself. More and more people who before would scoff at the (approximately) 4 voice synthesized (or less) sound of such games as “Elevator Action”, “Mario Bros.”, and the thousands of others that graced the 1980s are now buying game soundtracks by the millions (the “Final Fantasy” game series by Squaresoft, which now uses a full orchestra, sold over half a million copies of its soundtrack in Japan for its latest installment alone). They’re even reading the monthly new section in Entertainment Weekly “EWInternet” which contains at least 10 pages devoted to video games and, of course, the sound as well is written about.

Despite this new and growing worldwide popularity for video game music, something in the back of every game composer’s mind is a lurking question: “how can I make my music unique?” Game music can easily mimic the theater and TV. We all know the green producer who comes to us with the dreaded request “make my game sound like John Williams!” Well, who can blame them? It’s a benchmark to shoot from. Those industries have been around for decades, and it isn’t as though they’ll say “make my 3d immersive game sound like ‘Pac Man’!” Nevertheless, where game music truly comes into its own is in its interactivity, and I’m not just talking about whether the music switches depending on the player’s situation or not.. we all know that old trick by now and we’ll discuss it here in a moment. What do we really mean by “interactive music”? In this article we’ll discuss this meaning and not only give some examples… we’ll nail down some true definitions so that the uncertainty be removed from the minds of our colleagues. For those out there just entering the industry, for God’s sake read this. You can talk to a company and sound like an expert even if you haven’t fully ‘immersed’ yourself in the biz yet… but make sure you’re ready before you dive in. This is a pretty hefty read and a lot of pretty cool information, no matter what your level of experience, is coming your way. Grab a cup of coffee or a Jolt and sit back and enjoy the ride…

We’re going to begin with a brief history of game audio, citing among the most popular soundtracks. Note that my examples are in no way comprehensive, and I do hope some in the audience will notify me of other notable examples (What?? He didn’t include “Crysalis” as the best Nintendo soundtrack of all time? The FOOL!). The game geeks among us who have been with game music since the beginning will probably enjoy this trip down memory lane, but this might also be interesting for someone who wishes to learn ‘from whence we came’. Thanks to the magic of HTML, feel free to skip to any section of the article you’d like.

This history is categorized into generations. Remember, my friends, that these are approximations. Years before Nolan Bushnell (founder of Atari and creator of “Pong”) tinkered in his garage there may have been someone else on Earth who had already created game audio; I’ve talked to many a veteran among whose favorite phrase is “That? <snort> I did that ten years ago”. Well, it wasn’t in a national syndicated press release, so here’s your chance to let the world know you really DID do it ten years before anyone else, buckos. Write me and let me know the truth if its grossly misrepresented here.

For an excellent short history of game audio, look at “A Brief Timeline of Video Game Music” by Glenn McDonald. Also referenced here is information gleaned from my interview with Atari’s Brad Fuller in the 3^rd IA-SIG newsletter.

From the mid 1970s to 1980 or so, video game music started as the most horrendous, static filled movie soundtrack might have started in the 1920s. Simple electrical components and transistors were used to create one or two sounds at a time.

To tweak the sound, you had to actually engineer these components by hand. It is uncertain (by me, anyway, don’t count me as the expert in this era, I’d hardly been born by 1974) just when the microprocessor ‘chip’ entered the video game audio arena but it is fairly evident that by 1980 it was in fairly widespread use. Most of us shudder at the thought of such primitive days, but it was a tremendously exciting time for the engineers involved.

Before we get too far I want to make a distinction between ‘console’ systems and ‘arcade’ systems, and slapping my forehead before I forget, PCs as well. They are all separate sets of technology unto themselves and should be, in any comprehensive analysis or timeline, considered separately. For the purposes of not making my readers sick and bored to tears though, I’m going to lump them together. Just keep in mind during the years of video game audio’s history, these three markets evolved at different paces. Sometimes arcade audio held a triumphant upper hand to the poor quality of home systems and sickening quality of PCs (such as the 1980s), but at times, PCs can compete with the best of arcade and console audio (such as now, believe it or not)… just something to think about as you read this history.

Example games in this era are “Pong” of course and the other earliest arcade games: “Gunfight” (Midway, 1975), “Amazing Maze” (Midway, 1976), the ever present “Space Invaders” (Midway / Taito, 1978), “Galaxian” (Midway / Namco, 1979), “Asteroids” (Atari, 1980), and of course the #1 on the Billboard charts, “Pac Man” (Midway / Namco, 1980). Keep in mind that the United States and Japan released blockbuster hits right around the same time period. If you want to play these games I’d suggest you get MAME (the best arcade emulator for your home computer that’s available), but if you don’t contact the original game companies and buy the ROM chip of the title you play on MAME, that’s illegal. As an alternative, head to Cedar Point Amusement Park in Ohio.. they have one of the largest operating collection of vintage arcade games I’ve ever seen.

Yes indeed, by this time its obvious that Midway (who recently released a remake of their classic “Spy Hunter”) and Namco are damned old companies. A round of applause to them and the other dinosaur companies that are still around and still going strong, and a moment of silence for the (as of this writing) recently deceased “SNK”, whose first title was “Ozma Wars” in 1979. As they produced dozens of the best titles they’ll be sorely missed.

Video game music grew by leaps and bounds in this period, as did games in general in all aspects of their technology. Vector graphics began, using lines to draw objects instead of the blocky pixels, moving cockpits were used in various titles (Sega’s “Afterburner II”, for instance), and audio turned into full-fledged chip processing.

Some of the best artwork ever seen drawn digitally was done for games during this period, before 3d rendering began. The home market exploded as well. Not only did Atari flourish and then flounder, but in Christmas of 1986 the Nintendo Entertainment System was released, and thanks to its new technology and outstanding games, outsold anyone’s wildest expectations. Since this is Brad Fuller’s domain, check out his description of Atari audio and FM sound in Issue 3 of the IA-SIG newsletter to learn just what horrors the game audio folks still had to endure for arcade machines. They still did tremendously catchy pieces for the most part. The best pieces of this age still stack up compositionally to the best popular tunes of the age. ‘Tis sad that they only had the weak voice of FM synthesis to sing with, but look at some of the accomplishments…

This description of interactivity comes from Brian Schmidt, head of the audio department at Microsoft’s “Xbox” division, about his title “Black Knight 2000"):

Some sound fx (the pop bumpers on the upper playfield) are timed to 1/16th notes when you lock a ball, the sound is on a beat boundary. Also the key of the sound effect matches the underlying chord of whatever the background music is playing...if you have the glass off, try locking the ball when you know the chord's about to change, and you'll hear the sound effect transpose in mid-stream. graphics (lights, flashes, visual display) are all very, very tightly synchronized with the music.

A further trick...the vocal singing (the 'aaah's in particular)...Memory was REALLY tight. The main song is in E minor. I recorded a vocal "aaah" of an Emin chord. I use that same sample as Emin chord, CMaj 7 Chord and Bsus, so it sounds like there’s a lot more singing samples than I actually have. (listen to the part in the main music where after the "...beat the black Knight!" is sung...chords go Em, CMaj7..Bsus...B7...EMin.)

Each mode has it's own music...main play...one ball left for mball...mball...jackpot...ball in shooter (waiting to plunge)...they are all harmonically related, move from one to the other seamlessly. Also, as the music progresses, if you go to another mode (say a timed mode), when the mode's over, it doesn't always go back to the beginning of the 1st piece. It might pick up in the middle.”

Example games in this era were “Wizard of Wor” (Midway, 1980.. notable because it was among the first, among Stern’s “Berzerk”, that used voice synthesis to mimic speech.. “Vanguard” by SNK did this too but it wasn’t released until 1981), “Legend of Kage” (Taito, 1984.. notable because it was among the first video games to use far more accurate synthesis of real instruments, primitively reproduced though they were, in its soundtrack.. give it a listen, its quite impressive for the time), “Lifeforce” (Konami, 1986… not only did this game use samples in its soundtrack but it used recordings of voice.. other games that did this were “Kid Niki: Radical Ninja” by Irem / Data East, among others), “Afterburner” (Sega, 1987… used distorted guitar samples to score a very impressive soundtrack), and “Skull and Crossbones” (Atari, 1989… quote from musicians Brad Fuller and Don Diekneite: “The music becomes more intense when boss guy appears, more triumphant as his health goes down, more dire as your health goes down.”)

During this time the Atari 2600 and Colecovision home game systems were sweeping the world. The Colecovision in particular featured a Texas Instruments chip that enabled 3 tone channels and one noise channel, and while not up to the standards of the coin operated (coin-ops as they were known in those days) Gyruss class machines, it preceded only by about two years the next wave of technology, spearheaded at home by the Nintendo Entertainment System (released in February 1986 in the US), with its 2A03 integrated processor that had 2 square wave (that’s a kind of synthesizer for the boys and girls out there), a triangle wave, a noise, and sample generators, totaling five in all. Dozens of excellent soundtracks emerged that were so catchy they have been remixed by full live orchestras, among them the infamous theme to “Super Mario Bros.” By Koji Kondo and “Metroid” by Hirokazu “Hip” Tanaka.

As 1990 came round, several events marked this decade that turned games from a multimillion dollar industry to a multibillion dollar industry in a matter of a few years. This was because of changes in technology, and even more importantly, radical changes in game design itself. The first major coup was the release of “Wolfenstein 3D” and subsequently, “Doom” by Id Software. Because of the advanced technology giving legendary programmer John Carmack the power to render worlds in three dimensions instead of two, people found themselves losing thousands of jobs due to the addiction formed from this new sense of virtual reality (in fact, many games were doing 3d, but none with the use of bitmaps the way Wolfenstein 3D did… only solid colored polygons were used, and even mediocre realism was difficult to achieve with that limitation). But game music and sound was making minor leaps of its own. For the first time at home with the Commodore Amiga PC, in the living room with the Sega Genesis and Super Nintendo, and in the arcades as well, games could play back pre recorded sound from any source without much hassle. Games certainly had accomplished this before, but in bits and pieces. The 4 channel Zorro chipset on the Amiga leapfrogged the first IBM PC based “Ad-Lib” soundcards (released in 1987), which had more channels, but only used sub-standard Yamaha OPL chipsets to synthesize sound whereas Amigas could use PCM based samples in the newly developed .MOD format to mimic 80s hits such as “Axel-F” and “Rockit” with frightening precision, With the introduction of Creative Labs’ “SoundBlaster 2.0”, IBMs could play digitized sounds as well. Game music, which before had underground and closet fans in the thousands, was rising steadily into the hundreds of thousands. The music was growing with the technology and evolving with just as much stride. More from Brad Fuller and Don Diekneite on “Gauntlet: Dark Legacy”.

All the music is streamed, but we were able to still make it somewhat interactive).

Haunted House Level - Organ music fades in/out depending on how close you are to the organ - it was written to go with the existing music bed regardless of when in the music you happen to go near the organ.

Carnival of the lost - music changes as you pass through various sections of the playfield.

Some people call it “interactive audio”, but for the purposes of this article we’re talking about a segment of this broad field. Audio that isn’t just interactive, but adaptive. What’s the difference?

Thomas Dolby Robertson, who runs the company Beatnik, and who released several pop hits in the 1980s such as “She Blinded Me With Science” (he’ll never live that one down, but check out his “Retrospectacle” for other hits and lesser known gems such as “Budapest By Blimp”, for those of us that can tear our eyes away from MTV for more than ten seconds), said it best when he put it like this:

“Adaptive audio systems provide a heightened user experience through a dynamic audio soundtrack which adapts to a variety of emotional and dramatic states resulting, perhaps, from choices the user makes.”

What does this mean to the pro as well as the layman? Interactive audio is audio that happens when a user does pretty much anything with any kind of device, whether it be click a mouse or hit a key. Adaptive audio refers to something that happens most often in video games (at times in websites as well) when the user goes through more than just simple interactivity.

Mr. Robertson said this around 1994, and in the nearly eight years that have elapsed since then audio in video games has taken some very dramatic steps. In this section of the article we’ll identify how adaptive audio has progressed, and most importantly, just how effective it can be.

The simplest form of adaptive audio (AA… not to be confused with the group that uses bumper stickers that say “Easy Does It”) is found in such titles as the original arcade games. Its an easy concept to get your head around…music and sound effects would match things players did. Since sound effects are designed, for the most part, to be as closely related to actions as possible to maintain continuity, the adaptive aspect of them is instantly recognizable, but not necessarily a new concept. The explosions of “Asteroids”, the gobbling of “Pac Man”, and the heavy thud as Donkey Kong hits the girders all correspond best to the actions onscreen using whatever technology is available to reproduce them.

The next step therefore in adaptive audio is to explore music. Music in its purest form can be incidental or absolute. That is, like sound effects in that it corresponds to what is seen (incidental), or exists independently (absolute). The magic of music is that both of these techniques can work, in live music just as much as games.

Early examples of adaptive incidental music are seen in such games as “Vanguard” (SNK), when the player flies through a fuel ‘depot’ (a lovely little pixellated flashing tunnel with the word “FUEL” written above), the music changes from the main theme which begins the level (derived from Paramount’s “Star Trek: The Motion Picture”, music by Jerry Goldsmith) into a triumphant theme (derived from Thorn/EMI’s picture “Flash Gordon”, music by “Queen”) that lasts for as long as the player is invincible… around 15 seconds, during which the player can fly through anything and destroy it.

Even in “Pac Man” (Namco), there is use of a soundtrack that kicks in when the little muncher reaches an energizer and eats it, indicating that they, again, are invincible and can munch anything that stands in their way.

This technique of switching a single background soundtrack was employed by roughly 90% of games that used AA at all. Since games themselves were in their infancy, no one really thought to employ very advanced audio techniques and no one really could with the limited technology. Concentration was on adding audio, period.

Various games in the 1980s used AA in increasingly new ways, but on a very small scale. The all knowing (ha ha) author has heard tales of brilliant interactive concepts in Commodore 64 titles as well as other systems, but few, and so help me if you don’t inform ME of them, they’ll still remain shrouded in cult fantasy, so write me! It was not, however, until the early to mid 1990s that AA really started to take root and grow. The switching of a single background soundtrack was all that was used until such games as “Fade to Black”, by Delphine Software, “Ultima Underworld” by Origin Systems, and “System Shock”, by Looking Glass, actually switched the game’s music in response to events using different techniques such as fading and mixing.

For the purposes of this article, we’re going to use a few select games that have used AA in some obvious way, examine how it is used, and most importantly, try to reason where the value of it lies. For years, professionals such as myself have tried to academically catalogue interactive techniques and lump them into various terminologies. This method is proving ineffective, and I’m only just now realizing why… while it might be useful in future to label techniques used for interactivity such as ‘transition’ and ‘sequence’, people who create adaptive scores do so individually to create a unique experience on each new game.

For those who haven’t played “Parappa the Rapper”, it was, and still is a groundbreaking title. While other titles may have done the same thing in the past, Parappa and its successor, “Um Jammer Lammy”, based the entire design of the game on the musical score. As such, “adaptive audio” took command of design, and as such wins this title the top of the list for such an achievement. Check out my interview with the game’s creator, Masaya Matsuura.

Players listen to music and then tap their Playstation controller buttons to match the lyrics of a song as they are being played (the button presses scroll across the screen so the player has a visual cue). Doing so makes their character ‘rap’. The goal is to make your character rap as close to the beat as possible. If you do so, you get bonuses and higher points. If you fall behind or miss the beats entirely, you lose points.

The popularity of this game was huge. While I wasn’t able to find sound clips, check out the link above.. it explains gameplay with some poor quality streamed audio to at least give you an idea of the happy go lucky goofy soundtrack.

Its important to note that other titles have used this technique too, from Samba De Amigo to the latest music based title, Frequency.

Racing games are prime candidates for adaptive audio. About as linear a design as you can get, plus the design in general is extremely simple, with a lot of bells and whistles that can be added on. Take SSX, the game that made EA yet again loads of money because it remains the most popular snowboarding game out there (notice I didn’t say SIM… it isn’t a simulation, but an emulation).

In the videos, accessed through the link above, you will see how the soundtrack shifts when someone does a trick, most notably, the more air you get, the more the volume and low frequencies fade out until you land again, when a transition crash sound is played and everything returns to full volume.

Again, a racing title, but done long before SSX was released, this one by a long time IA-SIG member Alistair Hirst. One of the first titles to use adaptive audio techniques in this way was an Atari game with a soundtrack by Brad Fuller called “Road Blasters”. It would automatically cue a piece as the player approached the finish line, and would time a finish line piece to play perfectly in time (measurewise) with the approach piece. Need for Speed 3 uses a custom sound engine called “Pathfinder” to fade in and out tracks to a piece as various events took place, such as a chase, or as you start to go off the road. This is an example of adaptive audio used in very subtle ways, as many reviews didn’t notice how interactive the soundtrack was, and neither did the author of this article! Such techniques raise the question “if a soundtrack is interactive and the player doesn’t notice that its interactive… does it make a difference?” I think it does, but what’s a shame is that such a soundtrack doesn’t win awards, press, or accolades the way a symphony orchestra does.

The soundtrack to this, and the original, “Legacy of Kain: Soul Reaver”, were written by Kurt Harland, one of the members of Information Society. The genius of this game’s soundtrack lies again in its design.. the main character, Raziel, transfers between the spirit world and the real world with a wave of his hands. As he does, the world transforms from a bent and twisted vision in the spirit world to the real world where objects abide by physical laws. The music follows suit… in the spirit world the player is haunted by a warped version of the tracks played in the real world. The site above has a demo… definitely check it out.

Deus Ex, and Unreal as well, used similar adaptive audio models. During a level there will be an ambient piece played (anywhere from silence to a grooving techno track to a sweeping symphonic score), and when an enemy approaches, an action sequence begins. When the enemy is defeated, the music fades back into the ambient piece. In Deus Ex, there were different tracks also for when the player dies, and when the player encounters a person for a conversation. While it was open ended and very much synced with events, such a model doesn’t always work (an enemy behind you will trigger music before you’ve even seen them, for example). Still, it was a great place for the author to get his feet wet. Check out the link above for movies and a demo as well. The link below has the Liberty Island main theme followed by the action track, followed by the conversation track, rounding out with the death track.

Released two years before “Unreal” and four years before “Deus Ex”, this title followed in the footsteps of “Out Of This World” and “Flashback”, using brilliant techniques in animation to wow audiences and put Delphine Software on the map.

For F2B, Delphine used MIDI to achieve the same model used by so many subsequent titles… ambient / action / etc.. tracks, switching back and forth depending on various triggers in a 3d environment. This, along with System Shock and Ultima: Underworld helped pioneer mainstream use of adaptive audio in the 3d first person adventure genre. Sure wish I could find some demos on this sucker, but if you want to check it out, I believe its still sold in one or two places under the EA label’s “classic” lineup.

“The Dig” was one of the most unique game soundtracks LucasArts ever released. While not as popular, the game used the iMUSE system to achieve stunning interactive results.

This game was a graphic adventure, with characters moving around on a static background screen that changed when the characters exited. This style of game had been done since the late 1980s but still appealed to a large audience because of the ease of play and increasingly beautiful graphics. This title is no exception, but unlike the previous “Monkey Island” titles which used soundcard based General MIDI instruments primarily, “The Dig” used recorded stereo tracks and shifted between them in the same way that the MIDI shifted in previous titles. The results can be heard below, even though these are single pieces, trust me, they faded together completely seamlessly.

System Shock was, after Ultima Underworld 1 and 2 , among the first titles to use adaptive audio to switch music tracks. The example below gives you an idea of how it accomplished this. Unlike “Unreal” and “Deus Ex”, it created very similar tracks along a single theme per level, and each would be cued according to the measure the track was on (switching wasn’t instantaneous). System Shock was one of the best games of its day at using audio to truly envelop the player in a thick and sinister atmosphere.

System Shock Example

To finish up, let me say that these but few examples aren't necessarily the pinnacle of adaptive audio technique. One can hardly claim such a distinction for any title as adaptive audio is still a fledgling method, and not a very easy one to pull off. Plenty of composers (who I envy) are very happy to write their music, produce their sfx, and go their merry way. Indeed, most of my favorite titles have no adaptive audio at all. However, these titles do set the stage to demonstrate that a great many companies are taking adaptive audio seriously. Seriously enough to schedule into the already hectic development cycles of top game soundtracks.

So what conclusions can we draw? Certainly, the public isn't exactly clamoring for adaptive audio, but perhaps that's because there isn't much of it out there to clamor for. If we look at the increase of music based titles however, adaptive audio seems to be a good thing to keep your eye on, and if you didn't know all the games demoed above had adaptive audio and enjoyed them anyway, then the authors did a good job. Again, for all you out there who wish another title to be featured in a future article... write in, and happy AA hunting!