IASIG Interactive Composition WG

GDC 1999 - Roundtable Moderator Report
GDC Audio Track: Thursday 11:30a - 12:30a, Room D
Chris Grigg This email address is being protected from spambots. You need JavaScript enabled to view it.

Intro

My fondest hopes for the roundtable session were for the most part met, and I had a lot more fun doing it than I thought I would- in fact, I'd likely do it again if asked. Of particular personal satisfaction was the nice way the roundtable fit into the flow of the conference's terribly successful Audio Track by giving attendees an opportunity both to talk back and to access learned opinion regarding what they'd been getting fed in all the other sessions. We had a lively session, folks had fun, and several people said nice things to me afterwards about how much they enjoyed hearing the wide range of perspectives that the folks who spoke up brought to the table. So to speak.

What follows is a recap of the session, cobbled together from the script I read, from the list of topics I scrawled on the easel, and from my memories which I admit are at best fragmentary.

The welcome & bio that I read

"I'd like to welcome you all to the Interactive Composition Roundtable.
"My name is Chris Grigg, and I've been involved in music and sound for games since 1985, both as an artist, as a tool designer, and a sound department director. Companies would include LucasFilm Games, Electronic Arts, Epyx, Sega, Rocket Science, and Pixar. Following an excursion into the film sound world, I'm now a consultant to the game sound industry, a member of IA-SIG and the 3DWG, a developer of sound libraries for Kurzweil and Poke in the Ear, and I develop media production tools under the name Control-G.

The intro that I read

"Interactive Composition might be defined as the study of how one goes about making a soundtrack interactive. Unlike most presentations at the Conference where some guy gets up and talks about what she or he cares about, the aim of a roundtable is to give the attendees a chance to talk about what interests them. The point of this session is to provide game audio artists, the game engineers that support them, and providers of soundcards and runtime libraries an opportunity to assess the current state of the tools, and to think about what's next.

"As you can see, I've drawn up a list of topics that I thought you might find germane to discuss, and then we'll have about 10 minutes for just plain open discussion time at the end. In a minute we can vote on which of these topics y'all would like to take up, but I'd like to say just a couple framing things first.

"We're at a very interesting point in the history of game sound. This year we're seeing the fruition of lots of Interactive Audio technology initiatives all hitting at the same time. So now, kind of suddenly, there's a whole new range of creative possibilities.

"Do you feel prepared to take advantage of them?

"Do you feel the need?

"Is what you got what you wanted?

"…or maybe you're still just digesting all the new information.

"My feeling is that, cool as all this new stuff is, it still doesn't address some of our age-old real-world sound department problems like incomplete authoring tools and over reliance on our programmers to get our ideas into the game.

"Many of the other Conference sessions have been taking the long-term, high-level view of these emerging technologies, so I'm hoping we can balance that out by focusing specifically on the present situation."

Audience profile

A show of hands at this point revealed the audience to break down as follows:

• 70% game sound practitioners- working sound designers, composers, and other folks working with developers making media for titles
• 5% vendors- API, hardware, and OS makers
• 20% game engineers - game title programmers and game engine architects

But first a message from our sponsor…

At this point, Tom White spoke up to point out that not only was Room D really just uncomfortably full of people, but there were also another twenty folks lined up out in the hall and wanting to get in. Since the bigger B1/4 room that most of the Audio Track sessions had been happening in was not being used at the time, we took a brief recess and moved the whole session over there.
This worried me at first because the new room was set up for paper and panel presentations (with rows of chairs all facing front rather, the roundtable format where everyone faces each other) which seemed to me less conducive to open discussion. On reflection I think that because the group was so large by this point, this format actually helped to keep the session orderly (my rough estimate is that by the time we started again in B1/4, we had around 100 - 120 people in the room, which strikes me as kind of a lot for a roundtable). I decided to talk loudly rather than use the microphone, in hopes of keeping the session more like a town meeting and less like the Rikki Lake show (for any number of reasons…).

Topics Discussed

Be advised that from this point on I can rely only on my recollections, so the notes for each topic are necessarily going to be spotty and at least a wee bit error-prone. Anyone with differing memories, please do set me straight (This email address is being protected from spambots. You need JavaScript enabled to view it.).

MIDI- Experts agree it lives; do you?

There seemed to be a sense that despite DLS, digital audio music (meaning mainly Yellow Book) is where many sound creators would rather be (or at least the sound creators who spoke up, including Tommy Tallarico). The usual obvious advantages of having access to real human players, studio production techniques, and multiple playback platforms were recited.

Sound Design for 3D Worlds- Is reality just a crutch for people who can't handle games?

This got into a discussion about the way 3D (both positional audio ala A3D and I3DL1, and environmental reverberation ala EAX and I3DL2) has one again inserted the game programmer between the sound creator's audio media and the audience's ears, and how bad a thing that might be. I remember one person suggested that with 3D the programmer becomes like the sound designer; at which point several people including yours truly wondered out loud whether, gee, it might not be a better choice to let the Sound Designer be the sound designer.

There was also a short discussion about why it might or might not be a good idea to accept a physically accurate representation of how sound works in the real world- for example, in real life it takes sound time to travel in air, so for events in the far distance the sound to be correct should lag the animation in proportion to the distance; this however would be contrary to the convention used in film and TV production, where the sound always happens when the picture changes. There was also a mention of the broadly held experience among sound creators that creating an aesthetic sense of real-ISM is not always best served by strictly modeling real-ITY.

There seemed to be consensus a) that since in our games we find drama/fantasy preferable to reality in all other regards, there's little reason to accept mere reality in the area of 3D audio; b) that the cinema production models and conventions are in most cases going to be the better choice than the strictly correct physical approach (reverb, as in EAX and I3DL2, were mentioned in this connection); and c) that nonetheless, doubtless someone will some day do something just spectacular with strictly physically accurate 3D sound.

3D Audio Support- Design goal or consumer expectation?

I know this continued from the last topic, but somehow I have little sense of what anyone said. Maybe it was that some games do need it, but not all do, and that we're all big kids and can each speak up for ourselves and tell the marketing and engineering types when it just isn't called for. (Lots of anecdotes have been floating around lately about game sound folks who Just Say No when pressed to go 3D unnecessarily.) Put more diplomatically, there seemed to be a view that positional 3D and environmental reverb add to the palette of options- but that it should be left to the sound designer to pick the paints that make the painting. To force a metaphor.

More than one creator of 3D audio infrastructure spoke up to endorse the more cinematic view of how this stuff should work, and to make it clear to the gathered game sound practitioners that they were sincerely interested in working with us to make sure that the controls of the stuff they build can be put squarely in game sound creators' hands, not just the game programmers'. That was refreshing in the extreme.

Multichannel & Surround Formats- ???

Following some amusing but not in the end terribly informative interchange that I hesitate to characterize as sniping, exactly, between several hardware and software technology vendors and one or two game sound practitioners, a show of hands revealed that out of the audience of over 100 folks, only a couple would be doing anything with surround or more-than-two-speaker formats for Christmas 1999 titles. This surprised a lot of people. Perhaps some folks are just being cagey about their plans, but (someone please write to correct me if I'm misremembering here) there seemed to be a feeling that while in the fullness of time there might indeed be a huge installed base of home theater setups with multiple speaker and amplifier channels (…the letters T, H, and X were uttered), and while there are signs that multichannel/surround sound cards might become more predominant in future, this is still at an early-adopter stage for now, both from an installed-base standpoint and from a game sound authoring standpoint.

Programmer Dependence- It's 3 AM. Do you know where your programmer is?

This was my mildly provocative way of introducing perhaps the oldest gripe in our craft, the boatload of problems (or, for the lucky few, the cornucopia of opportunities) that arise from the fact that our ideas about how our sounds should be used in the game mean absolutely nothing until and unless we can secure the help and (usually more problematically) the time of sympathetic local game code programmers to implement, test, and improve the ideas.

As it turned out, no special provocation was necessary to launch this liveliest volley of the session.

Agreement was both immediate and fervent that for most of us life pretty well sucks in this regard, with the exception of the one programmer who was able to get a couple sentences out about what a great collaboration it can be when the sound artist and the programmer can spend quality time together dreaming up ideas and honing implementations, before most of the rest of the audience became physically ill with envy and the sounds of their discontent drowned out the one programmer everybody wished they worked with but don't.

There was further general agreement that since the local programmer situation both a) doesn't meet our needs, and b) is not expected to improve by itself any time soon, technical solutions to the problem are indicated. The conversation then kinda slid back and forth between the related issues of a) the sound practitioner needing to be able to specify dynamic sound behavior for runtime; b) the sound practitioner needing to have complete tools allowing the authoring & auditioning of content that takes advantage of all of the runtime APIs' features, not just the ones it's easy to write an editor for; and c) various abstract and specific approaches to tools, standardization, and conceiving and implementing playback APIs in a cross-platform sense.

The notion of some sort of a 'Universal API' (to use Rob Hubbard's term) that's been floating around over the last year or so in game sound architecture circles (to the extent that such circles can be said to exist) surfaced in this connection. The idea here is to find some way of getting all the runtime audio APIs talking a similar language- which would then make it feasible to build some sort of an authoring environment to drive them, thereby putting the knobs (or the reins, if you prefer that metaphor) back into the sound designer's hands. Cooksie Thomas pointed out the necessity of making the interface between sound and engineering data-driven, i.e. to simplify the logistical aspect by avoiding source-code-level dependencies. We also heard from a few folks who've spent more than a few years creating and working with custom in-house cross-platform tools about the creative and logistical advantages of having a common way to author (and audition) for multiple playback contexts. An unexpected minority opinion was voiced that attempting to standardize in this area is a bad idea because it would tend to stifle competition (which, upon reflection in the context of the recent introduction of DirectMusic as a game music engine, I now see as humor of the most subtle and wry sort).

Again there was a nice sense from the programmers and tool & API vendors in the room that they're sincerely interested in seeing that we get what we want, and that they're sincerely interested in listening to what we have to say- and again this was highly gruntling to the attendees.

(Personally, I would suggest that we should speak to them carefully, and only after much thought and deliberation among ourselves, if we want to avoid a lot of expensive dead-ends and the hurt feelings that are bound to result. Say, did I mention that the IA-SIG has an Interactive Composition working group that thinks about these kinds of things?)

DirectMusic Performance Layer- How do we like the interactivity model?

This was a surprisingly dead topic. Given the high percentage of all game development that the Windows platform represents, and given the unusually strong spotlight that the DirectMusic track-switching/track-muting/motifs/chords/etc. scheme has received, I had expected there to be more discussion about how people like the idea of doing things that way (of all possible ways) in their games. My guess is that people are still getting their heads around how to think in the DM structure, and haven't yet got to the point where they've formed strong opinions about it. The few comments included a note that full access to the interactivity features is not available to the sound content creator without a programmer's assistance in extending the DM system.

So I mentioned a couple things, which I'll repeat here to fill the space, if I may: a) this is a technology that finds its fullest advantage if the music creator is someone who is already thinking in terms of musical notes, chords, etc.- whereas many music creators (indeed many musics generally today) think more in terms of timbral structures & rhythmic patterns, i.e. music as audio production; and b) it's a remarkable and in many ways unexpected development to see these kinds of features that heretofore have been available only in custom, internal developer drivers (and some 3rd party libraries) now a part of The Only Operating System There Is™.

New Technologies & Budget, Schedule- One from Column A?

That impenetrably obliquely heading was intended to introduce the question of whether developers attempting to support the plethora of exotic new audio technologies (positional 3D, environmental reverb, DLS, multichannel/surround, DirectMusic) this year were going to have an adverse effect on their sound departments by trying to squeeze more content development out of a budget the same size as last year's- will we be faced with choosing just those technologies we can afford to support? Will the sound department be sleeping in the office all summer?

Once again, the consensus was that people would have the good sense to use only those technologies appropriate to the particular game design, and that nobody was too concerned with the budget stuff, and assumed that in practice that'd just kinda work itself out. I guess we'll see.

DLS- A sampler in every PC… but what about the samples?

This wasn't our hottest topic either, but then again by this point the needle on the ol' clock on the wall was starting to point at the big red E. There was one opinion that forcing the music people to create custom samples was an unwelcome burden, but other folks thought that a) music people seem to come in a pretty remarkable variety of capabilities, and for lots of them it's actually no problem at all, and b) if there's truly a vacuum there, the invisible and omniscient hand of capitalism will find some way to fill it, and soon. Here we talked a bit about the puzzling (to some) unwillingness of many sampler library vendors to sell licenses for the DLS format for use in games, and how remarkable it is that this is one of the first times that legal and/or regulatory customs originating from other areas of creativity have caused a sticking point for progress in game sound. I think someone also offered a general endorsement of the timbral freedom that DLS brings, and someone else offered further skepticism about even DLS-arranged MIDI as compared with just playing some solid, studio-recorded digital audio mixes.

Interactive Composition in 1999- What did you expect to be able to do by now that hasn't happened yet?

I have absolutely no recollection what anyone said about this. I do know time was getting awfully short by this point.

Interactive Composition in 2000- What more do you expect to be able to do by a year from now?

Ditto, but even shorter.