Puzzle Scoring: An Adaptive Music Case Study for Russian Squares

Saturday, July 19, 2003 - 19:44

This article is an excerpt from a paper originally written  for the book, DirectX Audio Exposed: Interactive Audio Development,  published by Wordware, edited by Todd M. Fay (a.k.a. LAX). [ http://www.wordware.com/ ISBN 1-55622-288-2] This excerpt replaces DirectMusic terms with general adaptive audio  terms that can be applied to any audio technology. If you're interested in  learning about the specific DirectMusic techniques, process, and technology used  in Russian Squares, read the full length article from the book (due out late  summer or early fall 2003) The full length article is much longer than this  excerpt and contains detailed information and instruction regarding the  DirectMusic production process.

INTRODUCTION and OVERVIEW

Russian Squares is a puzzle game involving rows and columns of squares. The player eliminates rows of squares by  matching the square’s color or shape. The game’s difficulty increases as the player eliminates the rows of squares; the time given decreases and ‘blockers’  are introduced to get in the player’s way. Sounds simple enough, but the game is both challenging and extremely addictive! The game is available in “Microsoft  Plus for Windows XP”.

Figure 1: Russian Squares main screen and ‘Neon’ visual theme

Figure 1: Russian Squares main screen and ‘Neon’ visual theme
 

The Adaptive Audio Design for Russian Squares offers a wonderful opportunity to demonstrate how an  adaptive score works seamlessly with core game design elements. When creating a musical score for a game, start by discovering what the game calls for. I  approach each game without any particular adaptive technique in mind, and  brainstorm in the abstract. This way, creative ideas lead technical decisions.  Once I have an abstract idea of how the score will work, the engineering team  and I figure out how to execute the adaptive music system with the game engine.  I believe that an adaptive audio design document, which should compliment or  even be a part of the game design doc, and is an important step in creating a  highly adaptive score.

Adaptive Elements

The adaptive elements of a game determine the form of  an adaptive score. An important first step is identifying potential adaptive  elements that tie gameplay to the music. Start by asking questions about the  gameplay. What is the core gameplay element? How does it function? What are  secondary gameplay elements? In what ways can I link the music to this gameplay?

Music Cells

The core gameplay of Russian Squares is the  elimination and addition of rows of squares. We made this game design element  the core of the music functionality. As the game adds rows or the player  eliminates them, the music responds by subtly changing the music; the adaptive  music system adds or subtracts an instrument, the harmony or rhythm changes etc.  The music follows the overall pace of the player. To accomplish this, there are  about 50 music cells per composition. As the player completes rows, the music  incrementally transitions to the next music cell. Logical transition boundaries  make those transitions musical and seamless. I used mainly measure boundaries  for transitions in Russian Squares.
I use music intro, and end  cells, to kick off and end gameplay. The music transitions to an ambient break  section when the player pauses the game. The adaptive music system plays the  appropriate intros and ends based on the current music cell.

Variation

Russian Squares uses instrument level variation to keep  the individual cells from getting monotonous. Each music cell is anywhere from  two to eight measures in length, and repeats as the player works on a given row.  Within DirectMusic, each instrument can have up to 32 Variations. When combined  with other instruments, these variations increase the amount of variation  logarithmically. Most often, one to three instruments per cell use variation,  and that gives the music an organic spontaneous feel, and prevents that all too  familiar loopy feeling. Too much variation, however, can unglue the music’s  cohesion.

Music Layers

Layering short musical gestures over the currently playing  music cell accents many gameplay elements, such as the game clock, the clearing  and adding of rows added, and blockers. These elements line up rhythmically with  the music (i.e. the tempo of the clocks synchronizes with quarter note values of  the music).

Sounds

DLS2 banks allowed for maximum flexibility and variation in this  project. Some of the instrument variations controlled instrument filters (i.e.  low pass filter sweeps) in addition to notes, which was crucial for the genre of  music. The sounds used were a combination of SonicImplants and custom banks.

Cooperative Composing

Three other composers, Erik Aho, Mike Min, and  Bryan Robbins, assisted in creating the three arrangements. Working with  non-linear cells made it easy to work simultaneously. Generally, we assigned a  range of cells to each of composer. Often someone would start an idea and the  next composer would elaborate on it, or add variations. This method of working  made sharing musical ideas easy and kept everything musically coherent.

CREATING THE ADAPTIVE AUDIO DESIGN

Russian Squares: How the Game Plays

Russian Squares is fast-paced  puzzle game that grows with intensity. The object is to eliminate rows of blocks  by matching colors or shapes faster than the clock. Each level starts with a  square made up of rows of blocks. The player controls an active square with a  mouse or arrow keys. The active block can move around the perimeter of the  square or push into the square causing the square directly across from it to pop  out and become the new active square. The first level’s square starts with  blocks set up six by six. If time runs out before the player matches a row, then  the game adds a row to the square. As the player eliminates rows, the square  decreases in size (six by five, five by five, five by four etc.), until the  square disappears. This constitutes the completion of the first level. Each  subsequent level begins with a larger square with more rows to eliminate,  against a faster clock.
There is another obstacle thrown in as the game  progresses. ‘Blockers’ are blocks within the square that cannot be moved.  Blockers appear in the square for a few seconds, get in the way of the players  actions, then disappear or switch to another location in the square. They  definitely add a challenge and twist to the game.
There are nine levels  in Russian Squares. A ‘new game’ starts you on level three. This is  because there is the potential for the player to move back a few levels! The  progression of levels plays an important part in how I created the music and  integrated it into the game.

Aural and Visual Themes

There are three visual ‘themes’ for Russian  Squares: ‘Neon,’ ‘Candy,’ and ‘Shapes.’ The gameplay is identical in each,  except that with ‘Shapes’ the player matches block shapes instead of color. Each  theme has a distinct aesthetic. ‘Neon’ has a glow green high-tech feel. ‘Candy’  has a lighthearted silver look. ‘Shapes’ portray a darker red tone. My team  wrote three pieces of adaptive music that correlate with each visual theme.  ‘Spin Cycle’ sounds edgy and techy to compliment ‘Neon’, ‘Bubble Matter’ floats  on light synthesizer pads over drum and bass riffs for ‘Candy,’ and ‘Gravity  Ride’s’ down-tempo groove matches the darkness of its visual counterpart,  ‘Shapes.’

 

Figure 2: The ‘Shapes’ visual theme.
 
Figure 2: The ‘Shapes’ visual  theme.
 

Figure 3: The ‘Candy’ visual theme.

Figure 3: The ‘Candy’ visual  theme.
 

The Musical Form

Form, in a non-linear piece of music, abides by the  same basic principles as linear music. Dynamics, harmony, orchestration, and the  arc of the form are critical elements for any composition. Linear music,  however, follows a singular path created by the composer, while non-linear or  adaptive music chooses one of many potential paths. This does not mean that  adaptive music is formless. Its form lies within the boundaries set for the  music. Creating these boundaries presents a new set of challenges for game  composers.
For example, each potential outcome must make sense musically,  and adhere to the overall consistency of the piece. It is important for an  adaptive piece of music to adhere to an overarching form despite its many paths.  The composer builds a musical structure in which the adaptive elements are  contained. For example, a snowflake has a form that describes its basic  elements; size, temperature, weight, and it’s crystalline form. Every snowflake  differs slightly, yet they are all snowflakes. Adaptive music functions in this  way.
The overarching form of Russian Squares is one of increasing  intensity. Playing all of the music cells sequentially, results in a gradual  building of musical elements, yet the performances that gameplay produces vary  greatly. The adaptive element of the gameplay provides the dynamic ebb and flow  of music, while the instrument level variation adds further non-linearity. The  form takes into account the fact that the music cells will not occur  sequentially, but will slide up and down the many music cells.
Each music  cell usually has one or two musical differences from its closest neighbor. That  difference is an instrument addition/subtraction, an instrument change, a chord  change, or rhythmic shift. We arranged them in such a way that the music  gradually moves from one large musical idea to another over the course of  several music cells. This gave the music a sense of form.
We scrapped  the whole idea of tempo changes for a couple of reasons. First,  electronica does not usually feature gradual accelerando as a part of the  form, so it sounded ridiculous. Second, it added some technical hurdles given  our technical approach. The game would have to call tempo changes directly (with  ‘increase tempo’ or ‘decrease tempo’ calls), which adds another adaptive design  layer. Each music cell would need to function well within a range of tempos, and  some of the DLS banks (i.e. drum grooves) were built around a specific tempo. We  kept this aspect simple by having only one tempo per composition. To prevent it  from sounding static, we used tricks such as rhythmic modulation to increase the  music’s sense of pulse. For example, the rhythm changes dramatically over five  music cells in ‘Spin Cycle.’ The score achieves this change by introducing the  new rhythm in one music cell, and then gradually emphasizing it (while  de-emphasizing the old rhythm), over the course of the next several music cells.  Each player experiences the shift at a different rate, depending on how well  they play!

The Adaptive Goal and Core Objectives

The arc and progression to Russian  Squares is one of growing intensity. However, it is not a straight linear growth  of intensity through the levels. Each level begins calm and ends frantic,  relatively speaking, but each successive level begins and ends slightly more  intense than the last.
Our first objective for the adaptive score was to  mirror the progression of game levels. This approach involved creating a score  that increased in intensity from the start to end of each level  (micro-adaptability), while each successive level began at a higher intensity  than the last (macro-adaptability).Within the game levels, we focus on the  primary gameplay element of clearing rows. A musical change occurs each time the  player clears a row; a reward letting the player know they are on the right  track. In addition, the change indicates the growing intensity as the rows  clear. Running out of time and having a row added is the negative counterpart to  clearing a row. The score reflects this state as well.
Russian  Squares uses an additive-subtractive approach to the music, in which the  music engine can add or peel away instrument layers. Given that the time it  takes to clear a row can be anywhere from a few seconds to a few minutes, this  is a very practical solution. It allows for musical development while  maintaining direction and momentum. Implementing this technique has some  complications however. In terms of aesthetics, simply moving along a linear  curve from low intensity to high intensity does not work, as adding layer after  layer and increasing the tempo is not very musical. The music requires larger  sections, instrumentation changes, rhythmic shifts, and tonal changes to stay  interesting. In this way, the music progresses in a multidimensional manner.  This concept is the backbone for the adaptive audio design of Russian  Squares.
We mirror many other gameplay elements with musical  gestures. Most of these are musical accents indicating important events as they  occur in the game. These music layers include row clear, row add, blocker, time  running low, high score, and active square. We also use musical gestures for  start game, end game, and pause game. Another objective is creating three  compositions that share the same core functionality. Three differing pieces of  music react in the same manor to the gameplay, and their common adaptive  framework allows them to be interchangeable. Having multiple compositions  accents the need to establish a solid adaptive framework prior to composing.  Using quality instrument samples goes beyond aesthetics. Sounds also need to be  flexible in an adaptive score. DLS instruments offer this flexibility because  single note samples can react more quickly than prerecorded phrases.

Inherent Obstacles

The biggest trick was creating a score that could  gradually increase or decrease in intensity at any time, while creating a  satisfying musical experience. How could we put such a score together? How would  the layers function? How would transitions operate? The overall objective seemed  simple enough, but just thinking about execution for a few moments brought up  all kinds of tough questions. Moreover, digging into the production of the score  unearthed many more!
The music in Russian Squares needs to change  quickly. Consider that it may take the player a minute or two to clear a row at  the start of the game, yet towards the end of any level, the player can clear a  row or the game can add one every few seconds. This poses a curious problem. If  music changes every minute of so, you should use longer sections to better hold  the listener’s interest and avoid repetition. However, when the music changes  every few seconds, it must not disrupt the aesthetic flow of the phrases. A  successful adaptive score strikes a balance between the length of phrases, their  transition boundaries, and the music’s responsiveness to game changes.
In  the modern electronica genre, nothing can make the music cheesier than wimpy  sounds. Electronica lives and dies by its grooves, fat synthesizers patches, and  filter sweeps. It depends heavily on timbres. This demand for great sounds  creates a challenge for the audio producer facing severe memory  constraints.
The last challenge we needed to overcome was repetition. Any  game composer understands the challenge of avoiding undue repetition in game  scores. Russian Squares was no exception. A player can easily spend a lot  of time solving a given level. As the difficulty rises, rows are cleared, added,  cleared, and added again, triggering the same music cells. Luckily, we found  ways to deal with this and the rest of the challenges we faced during production  of our adaptive score.

THE PRODUCTION PROCESS

I kicked off the production process by gathering  the initial DLS sounds for each composition. Then, setting the tone and main  themes for the pieces, I delegated the rest of the composing between my team and  myself. Rather than having one person assigned to each composition, we all  worked on every song. Each composer was responsible for a range of music cells  in each piece. I periodically merged all of our work so that we were aware of  each other’s progress. Toward the end, I polished and mixed the pieces, ensuring  coherence.

Gathering, Creating, and Assembling DLS banks

I find working in  DirectMusic’s DLS creation environment to be straightforward. Its interface,  while not perfect, is similar to other sample-bank creation software. The  process consists of importing wave samples, creating an instrument, and  assigning samples to regions across the keyboard. The DLS Level 2 standard  allows for the layering of samples, and adds basic low pass  filtering.
The trick to good sound is not in the tool, but in the  experimentation process. Many composers like to get right to the notes, and use  whatever sounds are available. This can lead to a generic sounding piece of  music. Creating sounds with a specific piece of music in mind gives the sounds  immediate relevance. There is a lot of back-and-forth between tweaking a sound  and trying it in the context of the music. The process boils down to a few key  points:

  • Quality samples  
  • Instrument layers  
  • Real-time filters  
  • Good use of the ADSR amplitude envelope  
  • Efficient use of samples  
  • Real-time effects such as reverb, delay etc.

We created an  individual sound set for each piece of music. We honed each sound set to the  specific composition, giving each piece a unique sound. The development team set  aside 8MB for each song’s sample set, for a total of 24MB of instruments. In the  end, we kept the sample rate at 44.1 KHz, for pristine sound. The game loads all  24MB of instrument banks at start so that the player can switch between songs  without pauses. We designed the sounds from scratch or licensed from Sonic Network, Inc. (the instruments are called SonicImplants). SonicImplants have an  excellent electronica collection, which served as our point of sonic  departure.
Each composition has its own rhythmic feel and flavor that the  sound sets portray. There are live drum loops diced into quarter note sections.  These loops live in a DLS instrument, so they can playback in their original  form, or so we can mix and match them in a variety of rhythms. This loop slicing  technique is common to the electronica genre, and allows a live drum feel with  added flexibility and variation.
For percussion we use standard drum  machine sounds such as Roland 808, Yamaha R8, and other obscure beat boxes.  Hollywood Edge has a great collection of classic sounds called Beat Machine. I  got specific permission to use these and other Hollywood Edge sounds in our DLS  banks. I also use a Roland Jupiter synthesizer (a little white noise and a  filter gave an industrial flavor to ‘Gravity Ride’), and some samples generated  from percussive metal spike sounds.
We used samples of synthesizers as an  integral element of the sound pallet. We spent a lot of time building and  layering synthetic instrument sounds, to sound thick, rich, and mean when they  needed to. Many of these instruments have three or four layers, which are panned  and slightly detuned to add density to the sounds. I also took advantage of  real-time filters as the primary musical gesture on both the lead synthesizer  and the rhythmic synthesizer of ‘Spin Cycle’.

Real-time Effects

DirectMusic offers real-time effect processing and  routing. Reverb, echo, and compression, add a sense of polish to the music of  Russian Squares. Real-time reverb (as opposed to reverb mixed with  instrument samples) brings all of the instruments into the same sonic space,  creating a consistent sounding mix. In addition, reverb helps smooth transitions  between music cells. This is because the reverb tails from one cell carry over  into the next cell, blurring the transition point. Echo enhances the feel of the  drum and percussion tracks. A delay time equivalent to a dotted eighth note adds  syncopation and helps the rhythms feel solid. Compression punches up certain  instruments such as the synthesized bass in ‘Spin Cycle,’ making them more dense  sounding.
Each composition calls for different effect settings, and  therefore uses its own custom effects setup.

Continuous Controllers

Motion within instrument parts adds depth of  emotion to an otherwise static composition. Controllers 10 (pan), 11  (expression), and 74 (filter cutoff), breathe dynamic life into the Russian  Squares music. We manually entered Control Curves or points into each  instruments’ Continuous Controller tracks. Continuous Controller 11 (expression)  handles any real-time movement of an instruments’ volume. This allows the basic  mix to remain solid, while instrument volume moves dynamically relative to its  Band volume, which uses CC 7. In this way, CC 7 acts as a PChannel’s master  volume, and CC 11 as its relative volume.
CC 74 controls an instrument’s  filter cutoff, allowing for low pass filter sweeps. DLS instrument filter  parameters respond if set up appropriately; you must check the filter enable box  and turn the resonance slider. ‘Spin Cycle’s’ bass synthesizer settings are 18dB  for filter resonance, and its initial cutoff frequency is 8728 cents. Useful  filter settings vary depending on the characteristics of the DLS instrument. To  ensure good results, audition while testing various settings.

Testing, Delivery and Integration

We evaluated the initial game build  and made adjustments to the script and the music content files, and created  another build of the game. At this stage we discovered that a music layer tied  to the ‘active block’ (the block controlled by the mouse), was being triggered  too frequently and thereby was removed. We also honed the ‘clear row’ function  so that the last three rows of a level did not increment the music cell. We did  this because the player can eliminate the last three rows in a matter of  seconds, undesirably triggering a quick succession of cells. So, with three rows  to go, the current music cell continues until the level is completed. I adjusted  some instrument levels after hearing the music within the context of the game.  Overall, the music functioned as designed and even exceeded expectations. The  music cell transitions sounded musical, and the music layers were timely and  seamless.

CONCLUSION

Puzzle games offer unique opportunities to composers. Game  states change more often and quicker than in action/adventure or role-playing  games, thus the adaptive scoring must be more tightly integrated with its game  elements. An apt analogy: Scoring a puzzle game is like scoring a Saturday  morning cartoon, in which every action on screen has a musical gesture  associated with it. Where as scoring, an action game is more akin to scoring an  epic movie where broad musical gestures paint the soundscape, and follow the  overall mood of the story. The score for Russian Squares demonstrates  DirectMusic’s ability to realize the adaptive music goals of this highly active  puzzle game. For those interested in learning adaptive music techniques, I  highly recommend scoring a puzzle game. Puzzle games allow the composer to focus  on a few key gameplay elements, while utilizing a wealth of potential adaptive  solutions.

Addendum: A Guide to the Russian Squares Media Files

Six audio files  demonstrate the functionality of the Russian Squares score. Each example file  moves progressively through the main music cells of that theme. The game never  plays them as such, but they demonstrate the overarching progression of the  score. The ‘music layer’ files audition all the music layers for each theme,  over a constant music cell. The timelines below provide a guide for  listening.

Bubble Matter Music Cell Timeline

BubbleMatter_grooves.mp3

Music Cell     Time
Intro          0:00.0
20             0:01.4
21             0:07.2
22             0:13.1
23             0:19.0
24             0:24.8
25             0:30.6
26             0:42.4
27             0:48.2
28             1:00.0
29             1:05.8
30             1:11.5
31             1:23.3
32             1:35.1
33             1:46.8
34             1:52.5
35             2:04.4
36             2:16.1
37             2:33.6
38             2:43.9
39             3:01.5
40             3:13.2
41             3:30.7
42             3:36.5
43             3:48.3
44             3:54.2
45             4:00
End            4:10.2

Bubble Matter Music Layer Timeline

BubbleMatter_motifs.mp3

Music Layer    Time
Clear Row      0:05.9
Add Row        0:11.7
Blocker        0:16.7
Blocker        0:19.0
Clear Row      0:23.4
High Score     0:35.2

Gravity Ride Music Cell Timeline

GravityRide_grooves.mp3

Music Cell     Time
Intro          0:00.0
20             0:01.7
21             0:09.1
22             0:23.7
23             0:53.1
24             1:00.4
25             1:07.7
26             1:29.7
27             1:37.0
28             1:51.7
29             1:59
30             2:06.5
31             2:21.0
32             2:28.4
33             2:35.7
34             2:43.0
35             2:50.4
36             2:57.2
37             3:12.2
38             3:19.6
39             3:27.0
40             3:34.4
41             3:41.5
42             3:49.0
43             3:56.3
44             4:03.6
45             4:11.0
End            4:18.3

Gravity Ride Music Layer Timeline

GravityRide_motifs.mp3

Music Layer    Time
Clear Row      0:07.3
Add Row        0:14.6
Blocker        0:24.1
Blocker        0:27.4
Clock          0:33.3
Clear Row      0:40.3
High Score     0:45.8

Spin Cycle Music Cell Timeline

SpinCycle_grooves.mp3

Music Cell     Time
Intro          0:02.6
20             0:05.4
21             0:10.8
22             0:16.3
23             0:21.8
24             0:27.2
25             0:32.7
26             0:40.9
27             0:51.8
28             1:02.7
29             1:08.2
30             1:30.0
31             1:40.9
32             2:19.1
33             2:30.0
34             2:38.2
35             2:49.1
36             3:00.0
37             3:05.4
38             3:16.3
39             3:27.3
40             3:32.6
41             3:43.5
42             3:54.4
43             4:05.4
44             4:16.3
45             4:27.2
End            4:38.2

Spin Cycle Music Layer Timeline

SpinCycle_motifs.mp3

Music Layer    Time
Clear Row      0:05.4
Add Row        0:10.9
Blocker        0:15.4
Blocker        0:18.9
Clock          0:22.4
Clear Row      0:30.7
High Score     0:38.9