From: Mark Miller
Sent: Wednesday, September 25, 1996 7:07 PM
To: iasig-icwg-members@world.std.com
Subject: IMA Expo Report

Hello,

Thanks to all of you who came to the ICWG meeting at the IMA Expo.

First let me begin with what I hope will be good news to all... I have
spoken with the folks here at Crystal Dynamics and I am now able to put the
CDIAE document out to the working group with out the copyright notice.

It is our intent to put forth this document as the basis for an industry
standard and hope that someone out there will build this and license it
back to us!

Most of the meeting consisted of a quick run through of the proposal as it
stands. There was a generally good response to it in principle. Following
are the main points that emerged from the discussion:

One: What we are really describing here is 'middleware'. In other words, we
are not describing a specific sound driver for a specific platform. Rather
we are describing a module of code that would live somewhere between the
main application and the platform specific sound driver. It's main
functions would be:
- to specify and carry out the extended functions that will become
(hopefully) an industry standard for interactive composition

- to specify a standard method for addressing these extended functions from
the application side

two: The scripting language was found to be a good starting point. It was
decided that the scripting can take place either within the segments or
outside of the segments in a file that is interpreted and run on the
application side. This decision will be made by the implementers of any
specific IAE depending upon platform and desired functionality.

three: it was decided that the basic functionality as defined in the
proposal was a good starting point even though it represented a fairly
minimal set of capabilities. It was also noted that this limitation might
also allow quick implementation of extended functionality into existing
systems.

four: it was discussed that the concept of player 'slots' would allow a
sort of plug in architecture so that more advanced interactive audio
systems like Audio Active could develop a 'player' to be plugged into slots
as modules and addressed in a standardized fashion through the common
middleware layer by the application. This was by no means a conclusive or
exhaustive discussion on this point.

five: the bit depth of the registers was questioned, defended as being MIDI
friendly, but ultimately it was decided that it would be best to make this
standard register size larger so as to support a wider range of uses.

I am very interested in peoples thoughts on what the bit depth of the
registers should be. Anyone who has a suggestion should jump right in...

six: it was suggested that perhaps this proposal has gone too far and
should have some features removed from the standard set and moved to call
back functions. Call back functions were also sited as another open area
for plug in like extensibility.

I am re sending the original proposal without any significant changes so
that you can have a record of the removal of the copy right notice...

Crystal's Dynamic Interactive Audio Environment (CDIAE):
Public Domain version 9/25/96

Written by:
Fred Mack (fred@metalithic.com)
Mark Miller (mmiller@crystald.com)

With contributions by:
Jon Miller and
Crystal Dynamics, Inc.

Following is a proposal for a standard set of functionality for I.A.E.s.
For this proposal it is necessary to add to the definitions of existing
terms or to add new terms to the lexicon previously established. As I have
in the past, I will capitalize all words that are defined in the ICWG
lexicon.

New or amended terminology: (at the conclusion of discussion, more terms
will be added...)

Segment: (add): For a system that plays audio by constructing an output
stream of MIDI or digital audio data (in real-time) based upon a set of
parametrized data and variables, a Segment would be any logical set of
such parameters and registered variables. Such Segments may divide this
output data into separate Tracks either in the data structure itself or
during the output phase.

Slot: A player type construct within the driver capable of playing one
Segment at a time. In a standard I.A.E., multiple Slots are capable of
playing Segments simultaneously and in synchronization with each other.

Crystal's Dynamic Interactive Audio Environment (CDIAE):

CDIAE contains the following logical runtime components:

Segment Bank:

A bank of up to 256 segments.

Sound Banks:

For MIDI and Parameter based systems, instrument sounds or voices should be
stored in a format appropriate to the sound playback hardware. The common
bond will be that all sound bank formats must respond to MIDI program
change commands. For digital audio systems, this may be irrelevant as the
voice or instrument data is contained and iterated within the digital audio
data.

Pitch Map Bank:

A bank of up to 8 Pitch Maps may be defined by the musician or programmer
and registered by the programmer. A Pitch Map is a table which defines a
translation of every note value of a Track to another note value. The
Musician or the Programmer can select from the available Pitch Maps using
embedded commands or the API.

User Tools:

Tools for building Segments, Segment Banks, Sounds and Sound Banks, Pitch
Map Banks, for integrating commands into segments, and simulating the
run-time driver and command structure. Some of these tools will be
profesionally available, other will need to be built specifically for the
IAE being designed. Some thought has gone into making the command structure
MIDI friendly so as to leverage off of existing tools wherever possible.

Run Time Driver (the middle-ware):

A module of code that sits between the main application and the sound
hardware. The RTD receives commands and performs the low level tasks of
sound playback and real time sound manipulation. The RTD must support the
following functionality and commands.

Segment Slots:

There are 7 Segment Slots. Each Slot plays one Segment at a time. Segments
are played by associating a Segment with an available Slot and then
starting playback of that Slot. All Slots play in sync. and at the same
tempo. Segments are assigned to Slots by the programmer via the API or by
the musician via MIDI commands. Each Segment can contain up to 14 Tracks
for a total of 98 possible Tracks. While there are 98 possible Tracks
defined by CDIAE, it will most likely be the case that there will be a
limit of how many Tracks may actually play simultaneously at any given
time. In this case, Track conflicts should be resolved via a priority based
dynamic Track allocation scheme.

NOTE: The other alternative to a priority based dynamic Track allocation
scheme would be to limit the number of Tracks allowable for each Slot, or
the number of Slots. While this type of limitation may be more strait
forward, it imposes some unwieldy limitations especially when incorporating
previously authored non-interactive work into the IAE....

Commands embedded within the Segment data (such as system exclusive data
for a MIDI Segment) can control playback of Segment Slots. Most commands
have both a local format and an absolute format. The local format commands
operate on the Segment they are issued in, regardless of which Slot that
Segment is played on. Absolute format commands reference an absolute Slot
number, without regard for what Segment is playing in that Slot. A Segment
which contains mostly absolute format commands and no note or audio data
can be created as a master 'control segment' or Structure (see lexicon)
used to load, play, and control Segments in other Slots.

Segment Slot Attributes:

For each Track of each Segment Slot, the following attributes are
maintained. These attributes can be directly controlled by the musician or
the programmer.

Mute Status
Volume
Pan Position
Transpose Amount
Current Program (voice; 0-127)
Current Pitch Map Table
Transpose Status

Commands:

Commands may be embedded into Segments. Commands will take a form
appropriate to the data type of the segment (e.g. MIDI Segments should use
system exclusive data). These commands control playback of the Segment they
are issued within, or they can control Segments in other Slots (local vs.
absolute).

Note: For MIDI segments, commands could be entered and edited using Sysex
data and a sysex editor. We have also discussed a tool that would convert
MIDI files to a time stamped, channelized, text format and allow the
musician to enter commands in an English like programming language. The
tool would then convert the text file back to a MIDI file. In this
conversion, the English like commands would be converted back to Sysex data
so that the resultant MIDI file could be edited freely in a MIDI sequencer
without losing the command structure.

Assign Segment <Segment number> <Slot number>

The referenced Segment number, from the current Segment Bank, is assigned
to the referenced Slot. Playback does not start until a start or resume
command is issued. Only one Segment is assigned to a Slot at any given
time. Segments may be reassigned as often as necessary, without regard for
what has been previously assigned to that Slot. If the referenced Slot is
already playing a Segment, any sustaining notes are keyed-off before the
Slot is reassigned. Assign Sequence does not reset the Sequence Slot
Attributes for the individual Tracks, although these may be reset from the
data within the newly referenced Segment.

Note: This can be quite useful when, for example, volume of a particular
Track is being updated continuously in an IAE, but the IAE is also choosing
a new Segment from a Pool of Segments with common Track and voice
structures at each bar boundary.

Get Segment Number <Slot number> <Variable number>

Gets the Segment number assigned to the referenced Slot and copies it into
the referenced user variable. See description of user variables below.

Set Tempo <beats per minuet>

Sets playback tempo for all Segment Slots. Initial tempo is set from the
first Segment assigned to any slot. There are 128 tempo values which range
from 48 to 175 BPM.

Note: This may not be useful in some digital audio based systems unless
pitch shifting and time compression and expansion can be accomplished in
real time.

Get Tempo <variable number>

The current playback tempo is copied to referenced variable

Start Slot <Slot list>
Stop Slot <Slot list>
Pause Slot <Slot list>
Resume Slot <Slot list>

The Slot number(s) referenced in <Slot list> are simultaneously started,
etc. The commands will most likely be issued within a control Segment (the
Structure) which is playing on a Slot not included in <Slot list>

Get Slot Status <Slot number> <variable number>

Gets a copy of the playback status for the referenced Slot. The copy is
stored in the referenced variable. This command stores a 1 in the variable
if a Segment is playing. A value of 0 is stored if the slot is not playing,

Mute/Unmute Track Local <Track mute list low> <Track mute list high>
<Track unmute list low> <Track unmute list high>

Mute Track Local <Track mute list low> <Track mute list high>
Unmute Track Local <Track unmute list low> <Track unmute list high>

Mute/Unmute Track Absolute <Slot list> <Track mute list low> <Track mute
list high> <Track unmute list low> <Track unmute list high>

Mute Track Absolute <Slot list> <Track mute list low> <Track mute list high>
Unmute Track Absolute <Slot list> <Track unmute list low> <Track unmute
list high>

Tracks of the issuing Segment or specified Segment are muted and/or
unmuted. The 'low' and 'high' distinction refers to Tracks 1 through 7 and
Tracks 8-14 respectively. Use Mute/Unmute Tracks to simultaneously change
multiple Tracks.

Get Track Mute <Slot number> <Track number> <variable number>

Gets a copy of the Track mute status for the referenced Slot. This copy is
stored in the referenced user variable. If the Track is muted, this command
stores a 1 in the variable. If this Track is not muted, a 0 is stored.

Set Track Volume Local < Track list low> < Track list high> <volume>
Set Track Volume Absolute <Slot list> <Track list low> < Track list high>
<volume>

Set the playback volume of Tracks within the referenced or implied Slot(s).
A volume of 127 plays the Track data at recorded velocities (or sample
amplitude values) 64 is half...

Change Track Volume Local < Track list low> < Track list high> <volume change>
Change Track Volume Absolute <Slot list> < Track list low> < Track list
high> <Volume change>

Changes the playback volume of Tracks within the referenced or implied
Slot(s) by the specified amount, relative to the previous Track volume.. A
volume change of 64 is no change, 127 is +63, 0 is -64....

Get Track Volume <Slot number> <Track number> <variable number>

Gets a copy of the Track volume for the referenced Track within the
referenced Slot. The copy is stored in the referenced user variable.

Set Track Pan Local < Track list low> < Track list high> <pan>
Set Track Pan Absolute <Slot list> <Track list low> < Track list high> <pan>

Sets the left/right pan position of Tracks within the referenced or implied
Slot(s). A pan position is defined such that a 0 is full left, 127 is full
right and 64 is center.

Change Track Pan Local < Track list low> < Track list high> <pan change>
Change Track Pan Absolute <Slot list> < Track list low> < Track list high>
<pan change>

Changes the left/right pan position of Tracks within the referenced or
implied Slot(s) by the specified amount, relative to the previous Track
left/right pan position.. A volume change of 64 is no change, 127 is +63, 0
is -64....

Get Track Pan <Slot number> <Track number> <variable number>

Gets a copy of the Track pan position for the referenced Track within the
referenced Slot. The copy is stored in the referenced user variable.

Transpose Tracks Local < Track list low> < Track list high> <Transpose amount>
Transpose Track Absolute <Slot list> < Track list low> < Track list high>
<transpose amount>

Notes played on the referenced Track(s) within the referenced or implied
Slot(s) will be transposed be the specified amount. A transpose amount of
64 is no change, 127 is +63 semitones, 0 is -64 semitones....

Get Track Transpose <Slot number> <Track number> <variable number>

Gets a copy of the Track transpose amount for the referenced Track within
the referenced Slot. The copy is stored in the referenced user variable

Ignore Transpose Local < Track list low> < Track list high>
Ignore Transpose Absolute <Slot list> < Track list low> < Track list high>
Respect Transpose Local < Track list low> < Track list high>
Respect Transpose Absolute <Slot list> < Track list low> < Track list high>

Use these commands to temporarily ignore transposition on the referenced
Track(s) of the referenced Slot(s). This command can also be used by the
musician from within a Track to insure that it will not be transposed by
mistake

Program Change Local < Track list low> < Track list high> <program number>
Program Change Absolute <Slot list> < Track list low> < Track list high>
<program number>

A program change is issued for the referenced Track number(s) on the
referenced Slot number(s). For digital audio systems, this may be
irrelevant as the voice or instrument data is contained and iterated within
the digital audio data.

Select Pitch Map Local < Track list low> < Track list high> <Pitch Map number>
Select Pitch Map Absolute <Slot list> < Track list low> < Track list high>
<Pitch Map number>

Subsequent notes played on the referenced Track number(s) of the referenced
or implied Slot number(s) will be remapped using the specified pitch map
table. A pitch map number of 127 indicates that no more pitch mapping is to
be done on the referenced Track(s)

Ignore Pitch Map Local < Track list low> < Track list high>
Ignore Pitch Map Absolute <Slot list> < Track list low> < Track list high>
Respect Pitch Map Local < Track list low> < Track list high>
Respect Pitch Map Absolute <Slot list> < Track list low> < Track list high>

Use these commands to temporarily ignore pitch mapping on the referenced
Track(s) of the referenced Slot(s) without changing which Pitch Map is in
effect. This command can also be used by the musician from within a Track
to insure that it will not be Pitch mapped by mistake.

User Variables:

Up to 128, 7-bit variables may be registered. A 7-bit variable contains a
value from 0 to 127. Variables must be registered by the programmer via the
API before they can be used. Variables are used for communications between
a Segment and the programmer, and by the musician for communication between
two Segments.

Set Variable <variable number> <constant value>
Copy Variable <source variable number> <destination variable number>
Add Variable <variable number> <constant value>
Subtract Variable <variable number> <constant value>

Set, Copy, increment, or decrement a variable. Variables must be registered
by the API before they can be used.

Set Variable Bits <variable number> <constant value>
Clear Variable Bits <variable number> <constant value>

Bits or flags within a variable are set. Use these command to build Track
or Slot lists. For such lists, each of the seven bits corresponds to one
Track or Slot with bit 0 being Track/Slot 1 and bit 6 being Track/Slot 7
for a Slot list or Track list low. For a Track list high, bit 0 would be
Track 8 and bit 6 would be Track 14... The idea of these commands is to
give you 7 flag variables packed into one variable.
Set Variable Bits does an "OR" of the variable and the parameter to set the
bits which are 1 in the parameter, leaving other bits unchanged.Clear
Variable Bits does an "AND" with the variable and the 1's compliment of
the parameter. Bits set to 1 in the parameter will clear corresponding bits
in the variable, while leaving other bits unchanged.

Substitute Variable <variable number> <parameter number>

This is a prefix command, meaning that it affects the way the next command
will operate. Substitute variable causes the next command to use the value
of <variable number> as the value for the parameter referenced by
<parameter number>. Two or more contiguous substitute variable commands may
be used to substitute multiple parameters of a command. Variable
substitution is ignored by the substitute variable command itself. All
variables and parameters are single 7-bit values.

Substitutes variable is a powerful command. It can be used to execute
commands with variable parameters instead of constant value parameters. It
can also be used on a parameter which is already a variable to indirectly
address another variable.

Segment Flow Control:

The following commands control the flow of a Segment as it plays in a given
Slot. These commands have local formats only. A branch cannot be directly
effected in one Segment by another. This must be accomplished by using
variables to communicate between Segments.

Loop Start <loop count>

Marks a loop start with a loop count being the number of times to play the
following loop section. A loop can execute from 1 to 127 times. A loop
count of 0 is used to loop forever. Loops may be nested up to 4 deep per
segment.

Loop End

Marks the end of a loop section. Flow immediately branches back to the most
recent loop start if the loop count has not been exhausted.

Define Label <label number>

Defines a label at the current location within a Segment. Up to 128 unique
labels may be defined for each Segment.

Goto Label <label number>

Parsing of Segment data branches immediately (on the same tick) to the
specified label

Note: It is the responsibility of the musician to make sure that no notes
are left sustaining after a branch in a sequence. No key-offs are issued
upon branching.

Decision Control:

These commands allow Segments to 'make decisions' and conditionally execute
blocks of commands and/or MIDI/Audio data.

If Equal <variable number> <variable number>
If Equal Const <variable number> <constant number>
If Equal Goto <variable number> <variable number> <label number>
If Equal Const Goto <variable number> <constant number> <label number>

Compares the previously registered 7-bit variable with another variable or
constant value. If equal, sequence flow continues with the following
command or MIDI/Audio data. If not equal, flow jumps to the matching End If
or End Else command. The Goto versions branch to a specific Label and do
not use an End If or End Else.

End If

Ends the most recent If Equal or If Equal Const command. Flow continues
here when the matching condition fails.

Else

Begins an Else block which is matched to the most recent If Equal or If
Equal Const command. Flow continues here when the condition fails, however
the Else block is skipped if the condition passes

End Else

Ends the most recent Else block. Flow continues here when the most recent
matching condition prior to the Else block is true and the Else block is
skipped.

Callback Functions:

For extended functionality, callback functions allow the musician to invoke
custom commands or functions created by the programmer. Callback functions
must be registered with the API before they are used.

Note:

The callback function will be most useful when retrofitting existing IAE
type systems to this new model.

Invoke Callback <callback number>

The referenced callback function is invoked.

Invoke Callback Function With Variable <callback function> <variable number>
Invoke callback Function With Const <callback function> <constant number>

The referenced callback function is invoked and is passed the 7-bit
variable or constant. Callback functions retrieve the passed variable via a
function in the API