1.
Sonification of Complex Data Sets: An Example from Basketball
Frances L. Van Scoy
West Virginia University
Morgantown, West Virginia 26506-6845 USA
fvanscoy@wvu.edu
Abstract
We are seeking ways to display via music relationships in complex data sets. As initial data sets we study some
simple mathematical functions and scores from basketball games.
Goals
New multimedia virtual systems use a combination of visualization and sonification to convey information to users.
The term visualization has been in widespread usage; the term sonification, meaning "the transformation of
numerical data into sound for purposes of observing that data" (NCSA ADG, 1999), or, more simply, "data
representation through audio" (ICAD, 1999), is somewhat less widely used.
In describing goals in visualization, Edward R. Tufte (1983) wrote:
Excellence in statistical graphics consists of complex ideas communicated with clarity, precision, and efficiency.
Graphical displays should
• show the data
• induce the viewer to think about the substance rather than about methodology, graphic design, the
technology of graphic production, or something else
• avoid distorting what the data have to say
• present many numbers in a small space
• make large data sets coherent
• encourage the eye to compare different pieces of data
• reveal the data at several levels of detail, from a broad overview to the fine structure
• serve a reasonably clear purpose: description, exploration, tabulation or decoration
• be closely integrated with the statistical and verbal descriptions of a data set.
We ask how well these requirements can be fulfilled by a good sonification application. Our initial approach
investigates approaches to mathematical representations of mathematical functions and data from basketball games.
Previous Work by Others
Others have addressed this issue in various ways.
Crumbs
The NCSA Biological Imaging Group (Brady, 1996) observed that many early attempts at sonification were difficult
to listen to for long periods of time, because the tones generated were repetitious and the chords typically dissonant.
Their goal was to build a system that would generate sound which would be both melodic and unobtrusive. The
2.
Crumbs software they developed rotates through several different melodies, all based on the dorian scale. As the
user begins searching with a probe for fibrous tissue in the data set (generated from an MRI scan of a horse's leg),
the sound of a guitar playing a melody is generated. As the user gets closer to the goal, other instruments are added
in order, a piano in single register, a piano in multiple registers, a voice-like synthesized instrument, and a flute. All
five instruments are heard when the user reaches the most dense region of fiber.
The vOICe
Peter Meijer has developed The vOICe, an experimental system for auditory image representation (Meijer). His
system coverts a picture into a 64 x 64 pixel digital image in gray scale. The picture is then scanned from left to
right, one column at a time. Each row of the picture is interpreted as representing a particular sound frequency.
While a particular column is being scanned, the 64 sound frequencies are generated with the volume of each
frequency based on the brightness (within 16 levels) of the corresponding pixel. The result is a changing sound
pattern with duration of about 1.5 minutes. The goal of his work is to assist blind people in understanding the
environment around them (for example, recognizing the position of doors in a large room).
Adam Burford's Masters Project
The work presented in this paper has an intellectual origin in the masters project of Adam Burford (1989).
Combining his undergraduate major in music and his graduate work in computer science, Adam generated music
from changes in cellular automata over time. In particular, he generated one measure of music from each
configuration in an instance of the Game of Life. Figure 1 shows an example of an oscillating sequence of
configurations, and Music 1 presents the music generated from that sequence.
Figure 1 Oscillating Sequence of Game of Life Configurations
Music 1 Music Generated from Oscillating Sequence of Game of Life Configurations
Representation of Simple Functions
We initially began looking at ways to represent minute-by-minute scores from basketball games. We mapped each
nonnegative integer between 0 and 108 to a note in the chromatic scale. That is, we had two functions f and g each
with domain the integers between 0 and 40 and range the integers between 0 and 108. The value of f(t) corresponded
to the score of one team when 40-t minutes of playing time in the game remained and g(t) to the score of the other
team at that time. For example, Music 2 shows the music generated this way from the first 10 minutes of the
basketball game described later in this paper.
Music 2 Music from Basketball Game Scores
3.
This approach has several disadvantages: nine octaves are required (beyond the range of a piano keyboard or other
instrument), the two notes representing the current score of each team often are dissonant, and unless a different
instrument is chosen for each team it is difficult to determine which team is ahead.
We then began looking at simpler functions.
Chromatic Scale for Simple Functions
Simple functions of a single variable can be represented by selected notes of the chromatic scale. For example the
function f(t) = t can be represented by the chromatic scale.
Figure 2 Graph of f(t) = t
Music 3 Musical representation of f(t) = t
A semicircle eliminates some notes of the chromatic scale and repeats others. We show this in the following table in
which we compute the value of sqrt(144-t2) for 0 <= t <=12 as we determine how to represent a quarter circle.
x 0 1 2 3 4 5 6 7 8 9 10 11 12
x2 0 1 4 9 16 25 36 49 64 81 100 121 144
144-x2 144 143 140 135 128 119 108 95 80 63 44 23 0
sqrt( 144 - x2) 12.00 11.96 11.83 11.62 11.31 10.91 10.39 9.75 8.94 7.94 6.63 4.80 0.00
rounded 12 12 12 12 11 11 11 10 9 8 7 5 0
note C C C C B B B Bb A Ab G F C
4.
A semicircle, then can be represented by the notes C F G G# A A# B B B C C C C C C C B B B Bb A Ab G F C.
Figure 3 Graph of f(t) = sqrt(k2 -t2)
Music 4 Musical representation of f(t) = sqrt(k2 -t2)
Functions over the Same Domain
Next, let us draw in music a full circle. That is, we will present the two functions f(t) = sqrt(k 2 - t2) and g(t) = -
sqrt(k2 - t2).
5.
Figure 4 Graph of f(t) = sqrt(k2 -t2) and g(t) = sqrt(k2 -t2)
We now play two notes simultaneously, one representing the value of f(t) and one representing the value of g(t).
Music 5 Musical representation of graph of f(t) = sqrt(k2 -t2) and g(t) = sqrt(k2 -t2)
This works, to a certain extent, but has the disadvantage of producing sounds which are disharmonic.
Consider the pair of functions f(t) = t and g(t) = -t.
Figure 6 Graph of either f(t) = t and g(t) = -t or f(t) = abs(t) and g(t) = - abs(t)
These two functions can be represented by two chromatic scales, one ascending and one descending. The resulting
music has the same problem of that generated by the circle above of being unpleasant to the Western ear.
Music 6 Musical representation of graph of f(t) = t and g(t) = -t
It has another problem, however, in that the music is indistinguishable from that generated by the pair of functions
f(t) = abs(t) and g(t) = - abs(t).
Two possible solutions are choosing easily distinguished instruments for the two functions, or using pitch for one
function and duration for the other. For example, we might represent a relatively small value by a whole note and a
relatively large value by eight eighth notes. Simplifying the number of pitches for this example, we have two
distinguishable melodies for f(t) = t, g(t) = -t and for f(t) = abs(t), g(t) = - abs(t).
6.
Music 7 Musical representation using pitch and duration of f(t)=t and g(t)=-t
Music 8 Musical Representation using Pitch and Duration of f(t) = abs(t )and g(t) = - abs(t)
Permutations as Proposed Solution
Our interest in sonifying basketball scores minute-by-minute was more than showing which team is ahead by how
much. We were looking for a way to show which combinations of five players are more effective (in terms of
increasing their team's score relative to the other team's).
We tried to represent each of the twelve players by a different musical instrument. We also tried to represent each
player by a different note in a chord, and each minute by a measure, in which the number of notes played indicated
roughly the difference in score between the two teams. Neither of these generated music which was pleasant to listen
to or easy to understand.
To avoid producing dissonant chords we decided to represent the players on the court in a given minute not by a
chord but by a permutation of the notes (from a major scale) representing those players. In this way we do not have
to be concerned with generating chords or sequences of chords which are pleasing to the ear but rather with
generating short melodies which change over time as function values change.
The permutation varies according to any of several patterns. For example, if we present the permutations in
lexicographic order (defined by pitch) we would play the series of notes (representing three players):
C D E, C E D, D C E, D E C, E C D, E D C
Any sequence of permutations of these notes can indicate that the team of interest is ahead in points. To indicate that
this team is tied or behind, we sharpen or flatten one note to make the sequence of notes be from a minor scale.
This work is inspired by work in comma-free codes (Ridley, 1993; Buchanan, 1992). In constructing a comma-free
code, an n element alphabet and a set of code words each of length k over that alphabet are chosen in such a way
that if any sufficiently long fragment of a series of code words is selected the code words can be recognized
uniquely. For example when n=4 and k=3 there are potentially 43 = 64 possible code words but it can be shown that
the largest possible set of code words contains only 20 in order to maintain this "comma-free" property.
However, in our work we are not looking for comma-free sequences which can be recognized logically but rather for
7.
melodies which are recognized more subtly and intuitively by a listener.
There are two main kinds of permutations we are currently studying: permutations in lexicographic order and in bell
change ringing order. We are also beginning to look at a subset of permutations, those forming a comma-free code.
Permutations in Lexicographic Order
We generate permutations of a set of notes not in strict alphabetical order but in lexicographical order based on
pitch.
For example, with base set {C, D, E }, we generate the string of notes CDE CED DCE DEC ECD ECD (with spaces
added here for legibility).
Music 9 Musical Representation of the Permutation in Lexicographic Order of C D E
Note that this same perumutation on a different set of notes, such as {C, E, G} or {C, D#, E} sounds somewhat
similar but different.
Music 10 Musical Representation of the Permutation in Lexicographic Order of C E G
Music11 Musical Representation of the Permutation in Lexicographic Order of C D# E
We represent a function by beginning with the melody of the permutations of a set of notes in lexicographic order
and then as the value of the function changes we change, generally one at a time, the notes in the base set.
8.
Change Ringing
We have also begun to study those permutations that correspond to the notes produced during change ringing of
bells. Over simplifying the definition, we begin with a downward scale and then for each successive permutation
exchange some pairs of adjacent notes. The result is a fairly complex melody with a recognizable structure to it. We
perturb this structure by gradually sharpening or flattening certain notes in the base set. Appendix B shows an
example of change ringing.
Permutations Based on Comma-Free Codes
One student who joined the project in summer 1999 has a degree in psychology. We hope to use him to design and
perform experiments to determine whether computer users can distinguish among melodies generated by different
comma-free codes over the same alphabet.
This will be a change from our current work of conveying information by choice of notes. In theory, we could use
comma-free codes to represent 20 different values using only 4 notes.
A Specific Example of Representation of a Basketball Game
We are searching for a scheme for music generation which is robust enough to take data collected at basketball
games which show which five players of one team are on the playing court at a given time and how the relative
score changes with each combination of players. This problem is interesting because it is simple enough to
understand (help the coaching staff recognize which combinations of players are most effective) but because the
data is relatively complex (there are 12 players on the team, only 5 play at a given time, and player performance is
of course not consistent).
As a case study we initially hand coded simplified data from the first half (20 minutes) of the West Virginia
Universitgy versus University of Georgia basketball game of December 2, 1998.
Table 1 shows which five players (by player number) were playing at each minute of the first half of the game as
well as the score of each team at the end of that minute.
minute Center Guard Guard Forward Forward WVU Georgia
1 35 25 33 0 32 3 3
2 35 25 33 0 32 3 5
3 35 25 33 0 32 8 7
4 35 25 33 0 32 8 10
5 35 25 33 0 32 10 10
6 35 25 33 0 32 12 10
7 35 34 33 0 32 15 11
8 35 25 33 0 32 17 11
9 35 25 33 0 32 19 17
10 35 25 33 0 32 24 22
11 31 25 13 0 30 26 22
12 31 34 33 0 32 26 23
9.
13 31 25 33 0 32 26 25
14 31 25 33 0 32 26 27
15 31 25 33 0 32 28 27
16 31 25 33 0 32 28 34
17 31 25 33 0 32 30 34
18 31 25 33 0 32 34 36
19 31 25 33 0 32 36 36
20 31 25 33 0 32 39 38
Table 1 First Half Data for WVU versus Georgia Basketball Game of December 2, 1999
We somewhat arbitrarily assigned notes to each of the players as shown in Table 2.
Player 35 0 32 31 25 34 13 33 30
Note middle C G E high C A D B F D above high C
Table 2 Assignment of Notes to Players
The melody generated from this first assignment of notes to players is given in Music 13. In this example the
number of notes per measure indicates the relative amount by which our team is ahead.
We chose a permutation of notes based on change ringing:
1 2 3 4 5; 2 1 4 3 5; 2 4 1 5 3; 4 2 5 1 3; 4 5 2 3 1; 5 4 3 2 1; 5 3 4 1 2; 3 5 1 4 2; 3 1 5 2 4; 1 3 2 5 4
We chose this particular permutation in large part because it repeats every ten steps, or four times in a 40 minute
basketball game.
In the first measure we play the notes for the starting players in order of center, two guards, and two forwards. In the
second measure we permute the position of the notes for the center and the first guard and the notes for the second
guard and first forward. As one player is substituted for another we change the pitch associated with that playing
role but continue with the permutation. For example, in minute 7, player 34 replaces player 25. By our permutation
sequence, measure 7 of the music should have contained the notes in order E F G C A but instead uses D for A,
giving E F G C D. Appendix C gives the music generated by this scheme.
We used Finale Allegro to transpose measures in which our team was behind into a minor key, trying several keys
until we found one that for this combination of players/notes sounded "sad." The resulting music does indicate even
to first time listeners that something is different in that minute/measure, but by choosing the minor key arbitrarily we
lost the consistent mapping of a player to a note.
In another attempt, rather than transposing losing measures to a minor key we lowered the pitch by an octave. The
result is given in Appendix D.
We have generated various other melodies based on this same basketball game. Other variations include changing
how notes are associated with players, using one instrument when our team is ahead and a different instrument when
our team is behind, and using percussion instrument accompaniment to indicate relative score.
10.
While we would prefer to have losing measures in a minor key, we have not yet developed an algorithm which
regardless of the combination of players always produces a measure in a minor key while retaining the identification
of players and notes.
Status and Goal
We have now written a C++ application which takes as input our rules for generating music and the data for a
particular basketball game and generates a MIDI file for the resulting music which is then played on a Macintosh G3
computer using the Finale Allegro 98 package. We are now constructing by hand data files from hand written notes
taken at WVU basketball games during the 1997-8 and 1998-9 seasons. We anticipate that the music generated from
these two seasons will be noticably different. During 1997-8 the team had five strong seniors who played almost the
entire game, with rest breaks provided by two other seniors, and the team was quite successful, going to the Sweet
Sixteen. In 1998-9 the team had few experienced players, and a core of five to seven strong players never emerged.
This team lost more games than it won. We anticipate that more frequent substitutions and playing time by more
different players during the 1998-9 season will lead to obviously different sounding music than that generated from
games of the 1997-8 season. (To test this, we will have to disable indicating relative score and which team is
leading, or compare only games in which WVU won.)
So far our evaluation of different schemes for generating music has been quite informal, testing melodies on each
other, on others who work in our lab, and on visitors to the lab. One team member has an undergraduate degree in
psychology, and during summer 1999 he will develop and perform more formal experiments with naive users.
More importantly than improving our method of representing basketball games, we intend to apply the lessons
learned from working with basketball to analysis of three-dimensional data sets. In particular we want to provide a
viewer of a 2-d projection of 3-d data with clues about how to change viewing perspective to see interesting
properties of the data. Figure 7 shows a simple example in which we project into the x-y plane two "clouds" of
points, each cloud parallel to the x-y plane.
Figure 7 Projection of 3-Data into the x-y Plane
This projection could represent data arranged in many ways in Cartesian space, including those shown in Figure 8,
in which the two data clouds are either disjoint or intersecting.
Figure 8 3-D Views of Data, Showing Disjoint and Intersecting Red anGreen Clouds
11.
We plan to generate music that will give auditory clues about how far apart these clouds are along the z axis by
generating music in which the relative distance of a point in a particular cloud from the x-y plane is mapped to pitch.
This will require a more sophisticated scheme for generating music than is currently being used with the basketball
data
As in Meijer's The vOICe, we will scan the 2-d projection from left to right, generating music (as distinguished from
sound) for each column. We will provide a bar which shows which column in the image is producing the current
music. The user may click on the mouse button to mark that particular column and may later zoom in to the data
using two of these bars to bracket the region of interest. The user may also choose to scan the projection from top to
bottom. We hope to generate music that will guide the user in choosing viewing perspectives of the data.
Bibliography
Brady, Rachel, Robin Bargar, Insook Choi, and Joseph Reitzer, "Auditory Bread Crumbs for Navigating Volumetric
Data," presented at IEEE Visualization '96, San Francisco, October 1996, available on-line at
http://mayflower.ncsa.uiuc.edu/sonification.html
Buchanan, Hollie L. and Michael E. Mays, "Retrograde codes and bounded synchronization delay," Information and
Computation 96 (1992) 55-64
Burford, Adam, Cellular Automaton Systems/Self-Reproducing Turing Machine, masters problem report, MS CS,
West Virginia University, Department of Statistics and Computer Science,1989.
ICAD (International Community for Auditory Display), "Joint ICAD/ASA/EAA Workshop on Auditory Display,"
http://www.santafe.edu/~icad/websiteV2.0/Conferences/Berlin99/cfp.html
Meijer, Peter, "The vOICE - Seeing with Sound,"
http://ourworld.compuserve.com/homepages/Peter_Meijer/voice.htm
NCSA Audio Development Group, "Sonification," http://cage.ncsa.uiuc.edu/adg/Sonification/
Ridley, J. M. and Michael E. Mays, "Generalized Witt formulae and comma-free codes," Journal of Pure and
Applied Algebra 88 (1993)177-185
Tufte, Edward R., The Visual Display of Quantitative Information, Cheshire, Connecticut: Graphics Press, 1983.
Acknowledgements
This work is supported by the EPSCoR programs of the National Science Foundation and the State of West Virginia
and by the Department of Computer Science and Electrical Engineering of West Virginia University.
Undergraduate students contributing to this project are Eric Martino, Radhika Mukamala, and Young Oh of West
Virginia University and James Keefover of Bluefield State College.
Macintosh and G3 are registered trademarks of Apple Computer, Inc.
Finale Allegro 98 is a registered trademark of Coda Music Technology.
Appendix A Basketball
Some relevant facts of basketball for this paper:
(1)there are two teams
(2) each team typically has twelve members, but only five from each team are on the court at a given time
12.
(3) at the university level, a game consists of two periods of twenty minutes each
At the end of the season the National Collegiate Athletic Association holds a single elimination tournament in which
the ostensibly best 64 college teams in the country play. Those sixteen teams remaining in the tournament after two
rounds of play are informally called the "Sweet Sixteen."
13.
Appendix B Musical Representation of a Permutation of the C Major Scale based on
Change Ringing
14.
Appendix C Music Based on Basketball Game, Losing Minutes in Arbitrary Minor Key
15.
Appendix D Music Based on Basketball Game, Losing Minutes Lowe r Octave
Be the first to comment