Puzzle Scoring: An Adaptive Music Case Study for Russian Squares
Saturday, July 19, 2003 - 19:44
This article is an excerpt from a paper originally written for the book, DirectX Audio Exposed: Interactive Audio Development, published by Wordware, edited by Todd M. Fay (a.k.a. LAX). [ http://www.wordware.com/ ISBN 1-55622-288-2] This excerpt replaces DirectMusic terms with general adaptive audio terms that can be applied to any audio technology. If you're interested in learning about the specific DirectMusic techniques, process, and technology used in Russian Squares, read the full length article from the book (due out late summer or early fall 2003) The full length article is much longer than this excerpt and contains detailed information and instruction regarding the DirectMusic production process.
INTRODUCTION and OVERVIEW
Russian Squares is a puzzle game involving rows and columns of squares. The player eliminates rows of squares by matching the square’s color or shape. The game’s difficulty increases as the player eliminates the rows of squares; the time given decreases and ‘blockers’ are introduced to get in the player’s way. Sounds simple enough, but the game is both challenging and extremely addictive! The game is available in “Microsoft Plus for Windows XP”.
Figure 1: Russian Squares main screen and ‘Neon’ visual theme
The Adaptive Audio Design for Russian Squares offers a wonderful opportunity to demonstrate how an adaptive score works seamlessly with core game design elements. When creating a musical score for a game, start by discovering what the game calls for. I approach each game without any particular adaptive technique in mind, and brainstorm in the abstract. This way, creative ideas lead technical decisions. Once I have an abstract idea of how the score will work, the engineering team and I figure out how to execute the adaptive music system with the game engine. I believe that an adaptive audio design document, which should compliment or even be a part of the game design doc, and is an important step in creating a highly adaptive score.
Adaptive Elements
The adaptive elements of a game determine the form of an adaptive score. An important first step is identifying potential adaptive elements that tie gameplay to the music. Start by asking questions about the gameplay. What is the core gameplay element? How does it function? What are secondary gameplay elements? In what ways can I link the music to this gameplay?
Music Cells
The core gameplay of Russian Squares is the elimination and addition of rows of squares. We made this game design element the core of the music functionality. As the game adds rows or the player eliminates them, the music responds by subtly changing the music; the adaptive music system adds or subtracts an instrument, the harmony or rhythm changes etc. The music follows the overall pace of the player. To accomplish this, there are about 50 music cells per composition. As the player completes rows, the music incrementally transitions to the next music cell. Logical transition boundaries make those transitions musical and seamless. I used mainly measure boundaries for transitions in Russian Squares.
I use music intro, and end cells, to kick off and end gameplay. The music transitions to an ambient break section when the player pauses the game. The adaptive music system plays the appropriate intros and ends based on the current music cell.
Variation
Russian Squares uses instrument level variation to keep the individual cells from getting monotonous. Each music cell is anywhere from two to eight measures in length, and repeats as the player works on a given row. Within DirectMusic, each instrument can have up to 32 Variations. When combined with other instruments, these variations increase the amount of variation logarithmically. Most often, one to three instruments per cell use variation, and that gives the music an organic spontaneous feel, and prevents that all too familiar loopy feeling. Too much variation, however, can unglue the music’s cohesion.
Music Layers
Layering short musical gestures over the currently playing music cell accents many gameplay elements, such as the game clock, the clearing and adding of rows added, and blockers. These elements line up rhythmically with the music (i.e. the tempo of the clocks synchronizes with quarter note values of the music).
Sounds
DLS2 banks allowed for maximum flexibility and variation in this project. Some of the instrument variations controlled instrument filters (i.e. low pass filter sweeps) in addition to notes, which was crucial for the genre of music. The sounds used were a combination of SonicImplants and custom banks.
Cooperative Composing
Three other composers, Erik Aho, Mike Min, and Bryan Robbins, assisted in creating the three arrangements. Working with non-linear cells made it easy to work simultaneously. Generally, we assigned a range of cells to each of composer. Often someone would start an idea and the next composer would elaborate on it, or add variations. This method of working made sharing musical ideas easy and kept everything musically coherent.
CREATING THE ADAPTIVE AUDIO DESIGN
Russian Squares: How the Game Plays
Russian Squares is fast-paced puzzle game that grows with intensity. The object is to eliminate rows of blocks by matching colors or shapes faster than the clock. Each level starts with a square made up of rows of blocks. The player controls an active square with a mouse or arrow keys. The active block can move around the perimeter of the square or push into the square causing the square directly across from it to pop out and become the new active square. The first level’s square starts with blocks set up six by six. If time runs out before the player matches a row, then the game adds a row to the square. As the player eliminates rows, the square decreases in size (six by five, five by five, five by four etc.), until the square disappears. This constitutes the completion of the first level. Each subsequent level begins with a larger square with more rows to eliminate, against a faster clock.
There is another obstacle thrown in as the game progresses. ‘Blockers’ are blocks within the square that cannot be moved. Blockers appear in the square for a few seconds, get in the way of the players actions, then disappear or switch to another location in the square. They definitely add a challenge and twist to the game.
There are nine levels in Russian Squares. A ‘new game’ starts you on level three. This is because there is the potential for the player to move back a few levels! The progression of levels plays an important part in how I created the music and integrated it into the game.
Aural and Visual Themes
There are three visual ‘themes’ for Russian Squares: ‘Neon,’ ‘Candy,’ and ‘Shapes.’ The gameplay is identical in each, except that with ‘Shapes’ the player matches block shapes instead of color. Each theme has a distinct aesthetic. ‘Neon’ has a glow green high-tech feel. ‘Candy’ has a lighthearted silver look. ‘Shapes’ portray a darker red tone. My team wrote three pieces of adaptive music that correlate with each visual theme. ‘Spin Cycle’ sounds edgy and techy to compliment ‘Neon’, ‘Bubble Matter’ floats on light synthesizer pads over drum and bass riffs for ‘Candy,’ and ‘Gravity Ride’s’ down-tempo groove matches the darkness of its visual counterpart, ‘Shapes.’
Figure 3: The ‘Candy’ visual theme.
The Musical Form
Form, in a non-linear piece of music, abides by the same basic principles as linear music. Dynamics, harmony, orchestration, and the arc of the form are critical elements for any composition. Linear music, however, follows a singular path created by the composer, while non-linear or adaptive music chooses one of many potential paths. This does not mean that adaptive music is formless. Its form lies within the boundaries set for the music. Creating these boundaries presents a new set of challenges for game composers.
For example, each potential outcome must make sense musically, and adhere to the overall consistency of the piece. It is important for an adaptive piece of music to adhere to an overarching form despite its many paths. The composer builds a musical structure in which the adaptive elements are contained. For example, a snowflake has a form that describes its basic elements; size, temperature, weight, and it’s crystalline form. Every snowflake differs slightly, yet they are all snowflakes. Adaptive music functions in this way.
The overarching form of Russian Squares is one of increasing intensity. Playing all of the music cells sequentially, results in a gradual building of musical elements, yet the performances that gameplay produces vary greatly. The adaptive element of the gameplay provides the dynamic ebb and flow of music, while the instrument level variation adds further non-linearity. The form takes into account the fact that the music cells will not occur sequentially, but will slide up and down the many music cells.
Each music cell usually has one or two musical differences from its closest neighbor. That difference is an instrument addition/subtraction, an instrument change, a chord change, or rhythmic shift. We arranged them in such a way that the music gradually moves from one large musical idea to another over the course of several music cells. This gave the music a sense of form.
We scrapped the whole idea of tempo changes for a couple of reasons. First, electronica does not usually feature gradual accelerando as a part of the form, so it sounded ridiculous. Second, it added some technical hurdles given our technical approach. The game would have to call tempo changes directly (with ‘increase tempo’ or ‘decrease tempo’ calls), which adds another adaptive design layer. Each music cell would need to function well within a range of tempos, and some of the DLS banks (i.e. drum grooves) were built around a specific tempo. We kept this aspect simple by having only one tempo per composition. To prevent it from sounding static, we used tricks such as rhythmic modulation to increase the music’s sense of pulse. For example, the rhythm changes dramatically over five music cells in ‘Spin Cycle.’ The score achieves this change by introducing the new rhythm in one music cell, and then gradually emphasizing it (while de-emphasizing the old rhythm), over the course of the next several music cells. Each player experiences the shift at a different rate, depending on how well they play!
The Adaptive Goal and Core Objectives
The arc and progression to Russian Squares is one of growing intensity. However, it is not a straight linear growth of intensity through the levels. Each level begins calm and ends frantic, relatively speaking, but each successive level begins and ends slightly more intense than the last.
Our first objective for the adaptive score was to mirror the progression of game levels. This approach involved creating a score that increased in intensity from the start to end of each level (micro-adaptability), while each successive level began at a higher intensity than the last (macro-adaptability).Within the game levels, we focus on the primary gameplay element of clearing rows. A musical change occurs each time the player clears a row; a reward letting the player know they are on the right track. In addition, the change indicates the growing intensity as the rows clear. Running out of time and having a row added is the negative counterpart to clearing a row. The score reflects this state as well.
Russian Squares uses an additive-subtractive approach to the music, in which the music engine can add or peel away instrument layers. Given that the time it takes to clear a row can be anywhere from a few seconds to a few minutes, this is a very practical solution. It allows for musical development while maintaining direction and momentum. Implementing this technique has some complications however. In terms of aesthetics, simply moving along a linear curve from low intensity to high intensity does not work, as adding layer after layer and increasing the tempo is not very musical. The music requires larger sections, instrumentation changes, rhythmic shifts, and tonal changes to stay interesting. In this way, the music progresses in a multidimensional manner. This concept is the backbone for the adaptive audio design of Russian Squares.
We mirror many other gameplay elements with musical gestures. Most of these are musical accents indicating important events as they occur in the game. These music layers include row clear, row add, blocker, time running low, high score, and active square. We also use musical gestures for start game, end game, and pause game. Another objective is creating three compositions that share the same core functionality. Three differing pieces of music react in the same manor to the gameplay, and their common adaptive framework allows them to be interchangeable. Having multiple compositions accents the need to establish a solid adaptive framework prior to composing. Using quality instrument samples goes beyond aesthetics. Sounds also need to be flexible in an adaptive score. DLS instruments offer this flexibility because single note samples can react more quickly than prerecorded phrases.
Inherent Obstacles
The biggest trick was creating a score that could gradually increase or decrease in intensity at any time, while creating a satisfying musical experience. How could we put such a score together? How would the layers function? How would transitions operate? The overall objective seemed simple enough, but just thinking about execution for a few moments brought up all kinds of tough questions. Moreover, digging into the production of the score unearthed many more!
The music in Russian Squares needs to change quickly. Consider that it may take the player a minute or two to clear a row at the start of the game, yet towards the end of any level, the player can clear a row or the game can add one every few seconds. This poses a curious problem. If music changes every minute of so, you should use longer sections to better hold the listener’s interest and avoid repetition. However, when the music changes every few seconds, it must not disrupt the aesthetic flow of the phrases. A successful adaptive score strikes a balance between the length of phrases, their transition boundaries, and the music’s responsiveness to game changes.
In the modern electronica genre, nothing can make the music cheesier than wimpy sounds. Electronica lives and dies by its grooves, fat synthesizers patches, and filter sweeps. It depends heavily on timbres. This demand for great sounds creates a challenge for the audio producer facing severe memory constraints.
The last challenge we needed to overcome was repetition. Any game composer understands the challenge of avoiding undue repetition in game scores. Russian Squares was no exception. A player can easily spend a lot of time solving a given level. As the difficulty rises, rows are cleared, added, cleared, and added again, triggering the same music cells. Luckily, we found ways to deal with this and the rest of the challenges we faced during production of our adaptive score.
THE PRODUCTION PROCESS
I kicked off the production process by gathering the initial DLS sounds for each composition. Then, setting the tone and main themes for the pieces, I delegated the rest of the composing between my team and myself. Rather than having one person assigned to each composition, we all worked on every song. Each composer was responsible for a range of music cells in each piece. I periodically merged all of our work so that we were aware of each other’s progress. Toward the end, I polished and mixed the pieces, ensuring coherence.
Gathering, Creating, and Assembling DLS banks
I find working in DirectMusic’s DLS creation environment to be straightforward. Its interface, while not perfect, is similar to other sample-bank creation software. The process consists of importing wave samples, creating an instrument, and assigning samples to regions across the keyboard. The DLS Level 2 standard allows for the layering of samples, and adds basic low pass filtering.
The trick to good sound is not in the tool, but in the experimentation process. Many composers like to get right to the notes, and use whatever sounds are available. This can lead to a generic sounding piece of music. Creating sounds with a specific piece of music in mind gives the sounds immediate relevance. There is a lot of back-and-forth between tweaking a sound and trying it in the context of the music. The process boils down to a few key points:
- Quality samples
- Instrument layers
- Real-time filters
- Good use of the ADSR amplitude envelope
- Efficient use of samples
- Real-time effects such as reverb, delay etc.
We created an individual sound set for each piece of music. We honed each sound set to the specific composition, giving each piece a unique sound. The development team set aside 8MB for each song’s sample set, for a total of 24MB of instruments. In the end, we kept the sample rate at 44.1 KHz, for pristine sound. The game loads all 24MB of instrument banks at start so that the player can switch between songs without pauses. We designed the sounds from scratch or licensed from Sonic Network, Inc. (the instruments are called SonicImplants). SonicImplants have an excellent electronica collection, which served as our point of sonic departure.
Each composition has its own rhythmic feel and flavor that the sound sets portray. There are live drum loops diced into quarter note sections. These loops live in a DLS instrument, so they can playback in their original form, or so we can mix and match them in a variety of rhythms. This loop slicing technique is common to the electronica genre, and allows a live drum feel with added flexibility and variation.
For percussion we use standard drum machine sounds such as Roland 808, Yamaha R8, and other obscure beat boxes. Hollywood Edge has a great collection of classic sounds called Beat Machine. I got specific permission to use these and other Hollywood Edge sounds in our DLS banks. I also use a Roland Jupiter synthesizer (a little white noise and a filter gave an industrial flavor to ‘Gravity Ride’), and some samples generated from percussive metal spike sounds.
We used samples of synthesizers as an integral element of the sound pallet. We spent a lot of time building and layering synthetic instrument sounds, to sound thick, rich, and mean when they needed to. Many of these instruments have three or four layers, which are panned and slightly detuned to add density to the sounds. I also took advantage of real-time filters as the primary musical gesture on both the lead synthesizer and the rhythmic synthesizer of ‘Spin Cycle’.
Real-time Effects
DirectMusic offers real-time effect processing and routing. Reverb, echo, and compression, add a sense of polish to the music of Russian Squares. Real-time reverb (as opposed to reverb mixed with instrument samples) brings all of the instruments into the same sonic space, creating a consistent sounding mix. In addition, reverb helps smooth transitions between music cells. This is because the reverb tails from one cell carry over into the next cell, blurring the transition point. Echo enhances the feel of the drum and percussion tracks. A delay time equivalent to a dotted eighth note adds syncopation and helps the rhythms feel solid. Compression punches up certain instruments such as the synthesized bass in ‘Spin Cycle,’ making them more dense sounding.
Each composition calls for different effect settings, and therefore uses its own custom effects setup.
Continuous Controllers
Motion within instrument parts adds depth of emotion to an otherwise static composition. Controllers 10 (pan), 11 (expression), and 74 (filter cutoff), breathe dynamic life into the Russian Squares music. We manually entered Control Curves or points into each instruments’ Continuous Controller tracks. Continuous Controller 11 (expression) handles any real-time movement of an instruments’ volume. This allows the basic mix to remain solid, while instrument volume moves dynamically relative to its Band volume, which uses CC 7. In this way, CC 7 acts as a PChannel’s master volume, and CC 11 as its relative volume.
CC 74 controls an instrument’s filter cutoff, allowing for low pass filter sweeps. DLS instrument filter parameters respond if set up appropriately; you must check the filter enable box and turn the resonance slider. ‘Spin Cycle’s’ bass synthesizer settings are 18dB for filter resonance, and its initial cutoff frequency is 8728 cents. Useful filter settings vary depending on the characteristics of the DLS instrument. To ensure good results, audition while testing various settings.
Testing, Delivery and Integration
We evaluated the initial game build and made adjustments to the script and the music content files, and created another build of the game. At this stage we discovered that a music layer tied to the ‘active block’ (the block controlled by the mouse), was being triggered too frequently and thereby was removed. We also honed the ‘clear row’ function so that the last three rows of a level did not increment the music cell. We did this because the player can eliminate the last three rows in a matter of seconds, undesirably triggering a quick succession of cells. So, with three rows to go, the current music cell continues until the level is completed. I adjusted some instrument levels after hearing the music within the context of the game. Overall, the music functioned as designed and even exceeded expectations. The music cell transitions sounded musical, and the music layers were timely and seamless.
CONCLUSION
Puzzle games offer unique opportunities to composers. Game states change more often and quicker than in action/adventure or role-playing games, thus the adaptive scoring must be more tightly integrated with its game elements. An apt analogy: Scoring a puzzle game is like scoring a Saturday morning cartoon, in which every action on screen has a musical gesture associated with it. Where as scoring, an action game is more akin to scoring an epic movie where broad musical gestures paint the soundscape, and follow the overall mood of the story. The score for Russian Squares demonstrates DirectMusic’s ability to realize the adaptive music goals of this highly active puzzle game. For those interested in learning adaptive music techniques, I highly recommend scoring a puzzle game. Puzzle games allow the composer to focus on a few key gameplay elements, while utilizing a wealth of potential adaptive solutions.
Addendum: A Guide to the Russian Squares Media Files
Six audio files demonstrate the functionality of the Russian Squares score. Each example file moves progressively through the main music cells of that theme. The game never plays them as such, but they demonstrate the overarching progression of the score. The ‘music layer’ files audition all the music layers for each theme, over a constant music cell. The timelines below provide a guide for listening.
Bubble Matter Music Cell Timeline
Music Cell Time Intro 0:00.0 20 0:01.4 21 0:07.2 22 0:13.1 23 0:19.0 24 0:24.8 25 0:30.6 26 0:42.4 27 0:48.2 28 1:00.0 29 1:05.8 30 1:11.5 31 1:23.3 32 1:35.1 33 1:46.8 34 1:52.5 35 2:04.4 36 2:16.1 37 2:33.6 38 2:43.9 39 3:01.5 40 3:13.2 41 3:30.7 42 3:36.5 43 3:48.3 44 3:54.2 45 4:00 End 4:10.2
Bubble Matter Music Layer Timeline
Music Layer Time Clear Row 0:05.9 Add Row 0:11.7 Blocker 0:16.7 Blocker 0:19.0 Clear Row 0:23.4 High Score 0:35.2
Gravity Ride Music Cell Timeline
Music Cell Time Intro 0:00.0 20 0:01.7 21 0:09.1 22 0:23.7 23 0:53.1 24 1:00.4 25 1:07.7 26 1:29.7 27 1:37.0 28 1:51.7 29 1:59 30 2:06.5 31 2:21.0 32 2:28.4 33 2:35.7 34 2:43.0 35 2:50.4 36 2:57.2 37 3:12.2 38 3:19.6 39 3:27.0 40 3:34.4 41 3:41.5 42 3:49.0 43 3:56.3 44 4:03.6 45 4:11.0 End 4:18.3
Gravity Ride Music Layer Timeline
Music Layer Time Clear Row 0:07.3 Add Row 0:14.6 Blocker 0:24.1 Blocker 0:27.4 Clock 0:33.3 Clear Row 0:40.3 High Score 0:45.8
Spin Cycle Music Cell Timeline
Music Cell Time Intro 0:02.6 20 0:05.4 21 0:10.8 22 0:16.3 23 0:21.8 24 0:27.2 25 0:32.7 26 0:40.9 27 0:51.8 28 1:02.7 29 1:08.2 30 1:30.0 31 1:40.9 32 2:19.1 33 2:30.0 34 2:38.2 35 2:49.1 36 3:00.0 37 3:05.4 38 3:16.3 39 3:27.3 40 3:32.6 41 3:43.5 42 3:54.4 43 4:05.4 44 4:16.3 45 4:27.2 End 4:38.2
Spin Cycle Music Layer Timeline
Music Layer Time Clear Row 0:05.4 Add Row 0:10.9 Blocker 0:15.4 Blocker 0:18.9 Clock 0:22.4 Clear Row 0:30.7 High Score 0:38.9
