A report by Alexander Brandon
(pictured above: Frequency)

Some of us might say that video game music is finally coming into its own; EMI and other major record labels are starting to jockey to place their hot artists into the soundtracks of the best selling titles, or use their music in cross promotional deals. The movie composer Harry Gregson Williams scored the recent release “Metal Gear Solid 2: Sons of Liberty” (who we’ll be talking to later here in the IA-SIG website). Some of us may venture the opinion that video game music has finally found its voice, as journalists are comparing some of the latest soundtracks to movie and TV show quality. From the perspective of the public at large, video game music is certainly no longer necessarily hearable only as part of a game, but ‘good quality, well done’ music unto itself. More and more people who before would scoff at the (approximately) 4 voice synthesized (or less) sound of such games as “Elevator Action”, “Mario Bros.”, and the thousands of others that graced the 1980s are now buying game soundtracks by the millions (the “Final Fantasy” game series by Squaresoft, which now uses a full orchestra, sold over half a million copies of its soundtrack in Japan for its latest installment alone). They’re even reading the monthly new section in Entertainment Weekly “EWInternet” which contains at least 10 pages devoted to video games and, of course, the sound as well is written about.

Despite this new and growing worldwide popularity for video game music, something in the back of every game composer’s mind is a lurking question: “how can I make my music unique?” Game music can easily mimic the theater and TV. We all know the green producer who comes to us with the dreaded request “make my game sound like John Williams!” Well, who can blame them? It’s a benchmark to shoot from. Those industries have been around for decades, and it isn’t as though they’ll say “make my 3d immersive game sound like ‘Pac Man’!” Nevertheless, where game music truly comes into its own is in its interactivity, and I’m not just talking about whether the music switches depending on the player’s situation or not.. we all know that old trick by now and we’ll discuss it here in a moment. What do we really mean by “interactive music”? In this article we’ll discuss this meaning and not only give some examples… we’ll nail down some true definitions so that the uncertainty be removed from the minds of our colleagues. For those out there just entering the industry, for God’s sake read this. You can talk to a company and sound like an expert even if you haven’t fully ‘immersed’ yourself in the biz yet… but make sure you’re ready before you dive in. This is a pretty hefty read and a lot of pretty cool information, no matter what your level of experience, is coming your way. Grab a cup of coffee or a Jolt and sit back and enjoy the ride…

We’re going to begin with a brief history of game audio, citing among the most popular soundtracks. Note that my examples are in no way comprehensive, and I do hope some in the audience will notify me of other notable examples (What?? He didn’t include “Crysalis” as the best Nintendo soundtrack of all time? The FOOL!). The game geeks among us who have been with game music since the beginning will probably enjoy this trip down memory lane, but this might also be interesting for someone who wishes to learn ‘from whence we came’. Thanks to the magic of HTML, feel free to skip to any section of the article you’d like.

This history is categorized into generations. Remember, my friends, that these are approximations. Years before Nolan Bushnell (founder of Atari and creator of “Pong”) tinkered in his garage there may have been someone else on Earth who had already created game audio; I’ve talked to many a veteran among whose favorite phrase is “That? <snort> I did that ten years ago”. Well, it wasn’t in a national syndicated press release, so here’s your chance to let the world know you really DID do it ten years before anyone else, buckos. Write me and let me know the truth if its grossly misrepresented here.

For an excellent short history of game audio, look at “A Brief Timeline of Video Game Music” by Glenn McDonald. Also referenced here is information gleaned from my interview with Atari’s Brad Fuller in the 3^rd IA-SIG newsletter.

Generation One, 1970-1980

From the mid 1970s to 1980 or so, video game music started as the most horrendous, static filled movie soundtrack might have started in the 1920s. Simple electrical components and transistors were used to create one or two sounds at a time. To tweak the sound, you had to actually engineer these components by hand. It is uncertain (by me, anyway, don’t count me as the expert in this era, I’d hardly been born by 1974) just when the microprocessor ‘chip’ entered the video game audio arena but it is fairly evident that by 1980 it was in fairly widespread use. Most of us shudder at the thought of such primitive days, but it was a tremendously exciting time for the engineers involved. Before we get too far I want to make a distinction between ‘console’ systems and ‘arcade’ systems, and slapping my forehead before I forget, PCs as well. They are all separate sets of technology unto themselves and should be, in any comprehensive analysis or timeline, considered separately. For the purposes of not making my readers sick and bored to tears though, I’m going to lump them together. Just keep in mind during the years of video game audio’s history, these three markets evolved at different paces. Sometimes arcade audio held a triumphant upper hand to the poor quality of home systems and sickening quality of PCs (such as the 1980s), but at times, PCs can compete with the best of arcade and console audio (such as now, believe it or not)… just something to think about as you read this history. Example games in this era are “Pong” of course and the other earliest arcade games: “Gunfight” (Midway, 1975), “Amazing Maze” (Midway, 1976), the ever present “Space Invaders” (Midway / Taito, 1978), “Galaxian” (Midway/Namco, 1979), “Asteroids” (Atari, 1980), and of course the #1 on the Billboard charts, “Pac Man” (Midway / Namco, 1980). Keep in mind that the United States and Japan released blockbuster hits right around the same time period. If you want to play these games I’d suggest you get MAME (the best arcade emulator for your home computer that’s available), but if you don’t contact the original game companies and buy the ROM chip of the title you play on MAME, that’s illegal. As an alternative, head to Cedar Point Amusement Park in Ohio... they have one of the largest operating collection of vintage arcade games I’ve ever seen.

Yes indeed, by this time its obvious that Midway (who recently released a remake of their classic “Spy Hunter”) and Namco are damned old companies. A round of applause to them and the other dinosaur companies that are still around and still going strong, and a moment of silence for the (as of this writing) recently deceased “SNK”, whose first title was “Ozma Wars” in 1979. As they produced dozens of the best titles they’ll be sorely missed.

Generation Two, 1980-1990

Video game music grew by leaps and bounds in this period, as did games in general in all aspects of their technology. Vector graphics began, using lines to draw objects instead of the blocky pixels, moving cockpits were used in various titles (Sega’s “Afterburner II”, for instance), and audio turned into full-fledged chip processing. Some of the best artwork ever seen drawn digitally was done for games during this period, before 3d rendering began. The home market exploded as well. Not only did Atari flourish and then flounder, but in Christmas of 1986 the Nintendo Entertainment System was released, and thanks to its new technology and outstanding games, outsold anyone’s wildest expectations. Since this is Brad Fuller’s domain, check out his description of Atari audio and FM sound in Issue 3 of the IA-SIG newsletter to learn just what horrors the game audio folks still had to endure for arcade machines. They still did tremendously catchy pieces for the most part. The best pieces of this age still stack up compositionally to the best popular tunes of the age. ‘Tis sad that they only had the weak voice of FM synthesis to sing with, but look at some of the accomplishments... This description of interactivity comes from Brian Schmidt, head of the audio department at Microsoft’s “Xbox” division, about his title “Black Knight 2000"):

“From the time you press start until the games over, the beat continues. Music always changes at a musical boundary (beat, measure, 1/2 measure, etc). Some sound fx (the pop bumpers on the upper playfield) are timed to 1/16th notes. When you lock a ball, the sound is on a beat boundary. Also the key of the sound effect matches the underlying chord of whatever the background music is playing...if you have the glass off, try locking the ball when you know the chord's about to change, and you'll hear the sound effect transpose in mid-stream. Graphics (lights, flashes, visual display) are all very, very tightly synchronized with the music. A further trick...the vocal singing (the 'aaah's in particular)...Memory was REALLY tight. The main song is in E minor. I recorded a vocal "aaah" of an Emin chord.  I use that same sample as Emin chord, CMaj 7 Chord and Bsus,  so it sounds like there’s a lot more singing samples than I actually have.  (listen to the part in the main music where after the "...beat the black Knight!" is sung... chords go Em, CMaj7..Bsus...B7...EMin.) Each mode has it's own music...main play...one ball left for mball...mball...jackpot...ball in shooter (waiting to plunge)...they are all harmonically related, move from one to the other seamlessly.  Also, as the music progresses, if you go to another mode (say a timed mode), when the mode's over, it doesn't always go back to the beginning of the 1st piece. It might pick up in the middle.”

Example games in this era were “Wizard of Wor” (Midway, 1980.. notable because it was among the first, among Stern’s “Berzerk”, that used voice synthesis to mimic speech.. “Vanguard” by SNK did this too but it wasn’t released until 1981), “Legend of Kage” (Taito, 1984.. notable because it was among the first video games to use far more accurate synthesis of real instruments, primitively reproduced though they were, in its soundtrack.. give it a listen, its quite impressive for the time), “Lifeforce” (Konami, 1986… not only did this game use samples in its soundtrack but it used recordings of voice.. other games that did this were “Kid Niki: Radical Ninja” by Irem / Data East, among others), “Afterburner” (Sega, 1987… used distorted guitar samples to score a very impressive soundtrack), and “Skull and Crossbones” (Atari, 1989… quote from musicians Brad Fuller and Don Diekneite:

“The music becomes more intense when boss guy appears, more triumphant as his health goes down, more dire as your health goes down.”

During this time the Atari 2600 and Colecovision home game systems were sweeping the world. The Colecovision in particular featured a Texas Instruments chip that enabled 3 tone channels and one noise channel, and while not up to the standards of the coin operated (coin-ops as they were known in those days) Gyruss class machines, it preceded only by about two years the next wave of technology, spearheaded at home by the Nintendo Entertainment System (released in February 1986 in the US), with its 2A03 integrated processor that had 2 square wave (that’s a kind of synthesizer for the boys and girls out there), a triangle wave, a noise, and sample generators, totaling five in all. Dozens of excellent soundtracks emerged that were so catchy they have been remixed by full live orchestras, among them the infamous theme to “Super Mario Bros.” By Koji Kondo and “Metroid” by Hirokazu “Hip” Tanaka.

Generation Three, 1990-present

As 1990 came round, several events marked this decade that turned games from a multimillion dollar industry to a multibillion dollar industry in a matter of a few years. This was because of changes in technology, and even more importantly, radical changes in game design itself. The first major coup was the release of “Wolfenstein 3D” and subsequently, “Doom” by Id Software. Because of the advanced technology giving legendary programmer John Carmack the power to render worlds in three dimensions instead of two, people found themselves losing thousands of jobs due to the addiction formed from this new sense of virtual reality (in fact, many games were doing 3d, but none with the use of bitmaps the way Wolfenstein 3D did… only solid colored polygons were used, and even mediocre realism was difficult to achieve with that limitation). But game music and sound was making minor leaps of its own. For the first time at home with the Commodore Amiga PC, in the living room with the Sega Genesis and Super Nintendo, and in the arcades as well, games could play back pre recorded sound from any source without much hassle. Games certainly had accomplished this before, but in bits and pieces. The 4 channel Zorro chipset on the Amiga leapfrogged the first IBM PC based “Ad-Lib” soundcards (released in 1987), which had more channels, but only used sub-standard Yamaha OPL chipsets to synthesize sound whereas Amigas could use PCM based samples in the newly developed .MOD format to mimic 80s hits such as “Axel-F” and “Rockit” with frightening precision, With the introduction of Creative Labs’ “SoundBlaster 2.0”, IBMs could play digitized sounds as well. Game music, which before had underground and closet fans in the thousands, was rising steadily into the hundreds of thousands. The music was growing with the technology and evolving with just as much stride. More from Brad Fuller and Don Diekneite on “Gauntlet: Dark Legacy”:

"All the music is streamed, but we were able to still make it somewhat interactive. Haunted House Level - Organ music fades in/out depending on how close you are to the organ - it was written to go with the existing music bed regardless of when in the music you happen to go near the organ. Maze of illusion - music changes when playfield changes. Carnival of the lost - music changes as you pass through various sections of the playfield."

Adaptive Audio… what is it?

Some people call it “interactive audio”, but for the purposes of this article we’re talking about a segment of this broad field. Audio that isn’t just interactive, but adaptive. What’s the difference?

Thomas Dolby Robertson, who runs the company Beatnik, and who released several pop hits in the 1980s such as “She Blinded Me With Science” (he’ll never live that one down, but check out his “Retrospectacle” for other hits and lesser known gems such as “Budapest By Blimp”, for those of us that can tear our eyes away from MTV for more than ten seconds), said it best when he put it like this:

“Adaptive audio systems provide a heightened user experience through a dynamic audio soundtrack which adapts to a variety of emotional and dramatic states resulting, perhaps, from choices the user makes. What does this mean to the pro as well as the layman? Interactive audio is audio that happens when a user does pretty much anything with any kind of device, whether it be click a mouse or hit a key. Adaptive audio refers to something that happens most often in video games (at times in websites as well) when the user goes through more than just simple interactivity."

Mr. Robertson said this around 1994, and in the nearly eight years that have elapsed since then audio in video games has taken some very dramatic steps. In this section of the article we’ll identify how adaptive audio has progressed, and most importantly, just how effective it can be.

The simplest form of adaptive audio (AA… not to be confused with the group that uses bumper stickers that say “Easy Does It”) is found in such titles as the original arcade games. Its an easy concept to get your head around…music and sound effects would match things players did. Since sound effects are designed, for the most part, to be as closely related to actions as possible to maintain continuity, the adaptive aspect of them is instantly recognizable, but not necessarily a new concept. The explosions of “Asteroids”, the gobbling of “Pac Man”, and the heavy thud as Donkey Kong hits the girders all correspond best to the actions onscreen using whatever technology is available to reproduce them.

The next step therefore in adaptive audio is to explore music. Music in its purest form can be incidental or absolute. That is, like sound effects in that it corresponds to what is seen (incidental), or exists independently (absolute). The magic of music is that both of these techniques can work, in live music just as much as games.

Early examples of adaptive incidental music are seen in such games as “Vanguard” (SNK), when the player flies through a fuel ‘depot’ (a lovely little pixellated flashing tunnel with the word “FUEL” written above), the music changes from the main theme which begins the level (derived from Paramount’s “Star Trek: The Motion Picture”, music by Jerry Goldsmith) into a triumphant theme (derived from Thorn/EMI’s picture “Flash Gordon”, music by “Queen”) that lasts for as long as the player is invincible… around 15 seconds, during which the player can fly through anything and destroy it.

Even in “Pac Man” (Namco), there is use of a soundtrack that kicks in when the little muncher reaches an energizer and eats it, indicating that they, again, are invincible and can munch anything that stands in their way.

This technique of switching a single background soundtrack was employed by roughly 90% of games that used AA at all. Since games themselves were in their infancy, no one really thought to employ very advanced audio techniques and no one really could with the limited technology. Concentration was on adding audio, period.

Various games in the 1980s used AA in increasingly new ways, but on a very small scale. The all knowing (ha ha) author has heard tales of brilliant interactive concepts in Commodore 64 titles as well as other systems, but few, and so help me if you don’t inform ME of them, they’ll still remain shrouded in cult fantasy, so write me! It was not, however, until the early to mid 1990s that AA really started to take root and grow. The switching of a single background soundtrack was all that was used until such games as “Fade to Black”, by Delphine Software, “Ultima Underworld” by Origin Systems, and “System Shock”, by Looking Glass, actually switched the game’s music in response to events using different techniques such as fading and mixing.

For the purposes of this article, we’re going to use a few select games that have used AA in some obvious way, examine how it is used, and most importantly, try to reason where the value of it lies. For years, professionals such as myself have tried to academically catalogue interactive techniques and lump them into various terminologies. This method is proving ineffective, and I’m only just now realizing why, while it might be useful in future to label techniques used for interactivity such as ‘transition’ and ‘sequence’, people who create adaptive scores do so individually to create a unique experience on each new game.

Examples (Console):

Examples (PC):

To finish up, let me say that these but few examples aren't necessarily the pinnacle of adaptive audio technique. One can hardly claim such a distinction for any title as adaptive audio is still a fledgling method, and not a very easy one to pull off. Plenty of composers (who I envy) are very happy to write their music, produce their sfx, and go their merry way. Indeed, most of my favorite titles have no adaptive audio at all. However, these titles do set the stage to demonstrate that a great many companies are taking adaptive audio seriously. Seriously enough to schedule into the already hectic development cycles of top game soundtracks.

So what conclusions can we draw? Certainly, the public isn't exactly clamoring for adaptive audio, but perhaps that's because there isn't much of it out there to clamor for. If we look at the increase of music based titles however, adaptive audio seems to be a good thing to keep your eye on, and if you didn't know all the games demoed above had adaptive audio and enjoyed them anyway, then the authors did a good job. Again, for all you out there who wish another title to be featured in a future article... write in, and happy AA hunting!