In 1999 IEEE Symposium on Visual LanguagesTokyo, Japan, pp. (2)

In 1999 IEEE Symposium on Visual Languages
Tokyo, Japan, pp. 111-118
Visual Music in a Visual Programming Language
Fred Collopy
Case Western Reserve University
[email protected]
Robert M. Fuhrer
IBM Watson Research Center
[email protected]
David Jameson
DigiPortal, Inc.
Abstract
Sonnet was designed as a visual language for
implementing real-time processes. Early design and
development of behavioral components has largely
focused on the domain of music programming. However,
Sonnet’s architecture is well suited to expressing many
kinds of real-time activities. In particular, Sonnet is easily
extended with new kinds of data types and behavioral
components.
We have developed a collection of visual output
components for Sonnet, referred to collectively as
Sonnet+Imager. Its design embodies aesthetically
grounded representations of color, form, and rhythm, as
well as dynamics for each. Moreover, its value is

enhanced by a flexible, modular architecture that treats
these graphic entities and operations as first-class
objects.
1. Introduction
There is growing interest in the design of instruments
to play graphics in the way that musicians play sound.
Such visual instruments could assume a wide variety of
forms and functions. Indeed, there is no reason to expect
less variety than there is among musical instruments.
When the visual instruments are computer-based, we refer
to them as imagers. The pieces that are produced using
them we refer to as lumia.1
Interest in instruments that could integrate graphics
with music is not new. In the early 18th century Louis-
Bertrand Castel produced his clavecin oculaire, a light
organ inspired by Newton’s work on color theory. Since,
there have been numerous experiments by painters,
composers and film-makers. At the beginning of this
century, the physicist Albert Michelson wrote: “so
strongly do these color phenomena appeal to me that I
venture to predict that in the not very distant future there
may be a color art analogous to the art of sound--a color
music [9]”.
Around the same time, composers and filmmakers
experimented with, and invented the elements of, such an
art. Among the interesting developments were those of
members of the Bauhaus, where Wassily Kandinsky and
1 This term was suggested by Thomas Wilfred in 1947 ([13]).
Paul Klee speculated on similarities between music and

painting and Moholy-Nagy built kinetic sculptures to
explore light in motion. In the United States, modern
artists Morgan Russell and Stanton Macdonald-Wright
built kinetic light machines, which they saw as a natural
extension of their interest in color and movement. Film-
makers including Oskar Fischinger, John and James
Whitney, Mary Ellen Bute, and Harry Smith explored
some of the same issues in abstract films.
Powerful graphic computers, MIDI, and high-
bandwidth distribution media have led to an increased
interest in computer-based graphic instruments. Scott
Draves’ Bomb, Mark Dank’s GEM, Sydney Fels, et al.’s
MusiKalscope, Sandy Cohen’s Bindhu, and Greg
Jalbert’s Bliss Paint represent modern attempts to
integrate graphics and music.2 These and similar
programs each implements some aesthetic model that
characterizes it.
Any instrument imposes certain features of an
aesthetic on its users. One cannot do with a trumpet
precisely what one can with a piano. One cannot achieve
in oils the effects that can be produced easily using
watercolors. Thus, an instrument’s design both limits
expression and favors particular types of expression.
Instrument designers must identify which choices are
passed to the player. Choices provide the potential for
expressiveness, though generally at the cost of added
complexity. In other words, a balance must be achieved.
When expressive potential is enhanced, and the resulting
complexity is not too burdensome, the instrument
succeeds.
With Imager we have designed an environment in
which a variety of graphic instruments can be built.
Details about particular aesthetics and range of

expressiveness are left to a circuit designer. This
decoupling of aesthetic choices from the underlying
rendering engine can be used to hide some complexity
from the player. Nevertheless, we cannot shirk
responsibility for having made certain aesthetic choices.
The architecture described in this paper builds upon
ideas embodied in a previous version of Imager that was
constructed by the first author. That version runs under
MAX on Macintosh, and is largely implemented as a
single monolithic object. Nonetheless, it is capable of
2 URLs for these and similar developments can be found at
http://imagers.cwru.edu
producing a significant variety of visuals, as illustrated in
several color images located at http://imagers .cwru.edu.
One of these is reproduced in black and white as Figure 1.
That version also serves as a baseline for the aesthetic
quality, variety of form, and range of expressiveness of
the new version.
However, its monolithic design has several
shortcomings. First, it is not easily extended with new
capabilities (e.g., new kinds of forms, motions, and other
dynamic behavior). This problem stems in part from the
fact that the design does not incorporate a modular
architecture that encourages the development of reusable
components. Second, several of its features force hard-
coded constraints on the number and behavior of various
kinds of objects.
The version presently under construction represents an

effort to overcome these limitations. In particular, our
new design focuses on: 1) devising a more coherent,
modular and robust architecture for building alternative
instruments, and 2) providing the artist with a more useful
environment by exposing key aesthetic properties in the
form of intuitive and powerful controls.
The remainder of the paper is structured as follows.
First, we discuss the critical role of visual aesthetics in
devising an architecture for producing visual music. Then,
we briefly introduce the Sonnet programming
environment, upon which Imager is built. In Section 4 we
provide a detailed description of our architecture. Some
extensions that aid in realizing complex compositions are
briefly presented in Section 5. Section 6 discusses how
Sonnet+Imager differs from other dynamic visual
generation systems. Finally, we briefly identify some
future directions and make some concluding remarks.
2. The role of visual aesthetics in defining an
architecture
Many approaches can be taken to devising an aesthetic
for visual experience. One that has a close historical link
with efforts to integrate visual and sonic experiences is
the constructivist aesthetic employed by such modern
artists as Kandinsky, Klee, Max Bill, and Karl Gerstner.
For them and other modern artists, form and color
assumed preeminent roles in expressiveness. Form was
built up from simple elements, such as points, lines,
planes, angles, and conic sections. Color was of interest in
its own right, not merely as a way of rendering objects in
the world.
Given that Imager is intended to deal with color and
form in time, it is necessary to add a third element to

describe changes in these two. In 1914, the painter
Leopold Survage wrote about a new visual art in time, an
art of “colored rhythm and of rhythmic color ([11], p.
36).” He believed that colored visual form could play a
role analogous to that of sound in music and that these
forms could be described by three factors: color, the
visual form proper, and rhythm. So, we add this third
element to the constructivists’ notions of color and form.
Through rhythm, graphics and music can become linked,
compositionally and improvisationally. These three
elements (form, color and rhythm) completely describe
the space of dynamic visuals.
For each of these three domains, color, form, and
rhythm, it is necessary to make some choices about how
composers and players will control them. In effect, one
needs to decide what knobs will be available. In making
these decisions, we have been guided by aesthetic
considerations wherever possible. Our choice of the hue,
saturation, and value (HSV) color model serves as an
illustration.
Hue is the principal way in which one color is
distinguished from another. Describing and managing
hues is generally taken to be the central problem for color
theory. Indeed, the very language we use to denote colors
is associated primarily with their hues. A hue is denoted
by its angle around a color wheel, for example, red at 0°,
yellow at 60°, green at 120°, blue at 240°, and purple at
300°. In a well-behaved color wheel, complementary
colors would appear at 180°-opposite positions.
Saturation describes how pure a particular hue is. It is
also referred to as the intensity, strength, or chroma of a
color. A particular hue becomes less saturated by mixing

gray with it. Reducing saturation at a constant value has
the effect of adding white pigment, producing what artists
call tints.
Value is the quality that differentiates a light color
from a dark one. It is also referred to as lightness. A
particular color moves toward black by a reduction in its
value. Low valued colors are less visible than higher
valued ones. Decreasing value while leaving saturation
alone has the effect of adding black pigment, producing
what are referred to as different shades. Finally, what
Figure 1. Single frame from a visual performance
generated by Imager under MAX
http://www.imagers.cwru.edu/
artists refer to as tones can be created by decreasing both
saturation and value. There is substantial literature that
uses these concepts to describe art history and technique.
Decisions about Imager’s color model were, in short,
driven by artistic considerations. The choice of the right
color model makes achieving certain aesthetic choices,
such as surface textures, lighting effects, and perspective,
much easier. We have chosen the hue, saturation, value
(HSV) color model because those concepts and the
related concepts of tint, tone, and shade have artistic
meaning. We have taken a similar approach to the design
of Imager’s other components.
The design of Sonnet, however, is such that a choice of
a particular model does not preclude the use of alternative
models. For example, a lumianist could use components
to transform to and from RGB or CYMK space.

3. The Sonnet Environment
With the design of Imager, we are interested in
providing artists with the ability to define instruments that
can be used to play with color and form as musicians play
with sounds. Artists will wish to bring ideas and forms of
their own to this process. Enabling that is achieved by
embedding Imager’s facilities in a programming
environment.
Sonnet was originally designed as a visual
environment for associating runtime actions with running
programs. It has since evolved into a visual programming
language for the rapid development of real-time
applications [6]. Sonnet uses a circuit metaphor, and
embodies event-flow semantics. Sonnet behavior is
expressed in two forms: as primitive “components”, and
as “circuits”, which are interconnections of components.
We often refer to Sonnet programs and Sonnet circuits
interchangeably. The programming activity in Sonnet
consists in constructing different arrangements of
components into circuits that perform some computation.
Components are entities that have 1) a set of strongly-
typed input and output “ports” through which data
packets flow, and 2) an Execute() method. An input
port may be designated a “trigger”; it then causes the
Execute() method to be invoked whenever a data
packet arrives on that input.
As shown in Figure 2, components are interconnected
using “wires” which attach one output port to one or more
input ports. It is permissible for a component to have no
inputs or to have no outputs (as is often the case with
interface components).

A circuit can be collapsed into a single component,
known as a “chip”, and used in the same manner as a
primitive component. Chips allow the designer to
structure Sonnet programs hierarchically.
Sonnet is a strongly typed language. That is, all ports
accept or produce a well-defined type of data packet.
Sonnet normally disallows the connection of ports of
incompatible types, to ensure type safety. At present, each
component’s implementation is trusted to conform to its
declared type signature.
Data packet types are arranged in a hierarchy. Thus,
each port is compatible with its specified type and all of
that type’s ancestors in the hierarchy. At the root of the
hierarchy is the “Generic” type, which is compatible with
all types.
Sonnet is very modular, a fact which manifests in
several ways. First, everything in Sonnet is represented by
a software object. Data exists as quanta known as
“packets”. Components are also objects that support a
certain set of pre-defined interfaces.
Second, the key language semantics are primarily
defined by a replaceable module called the “semantic
policy module”. Thus, although the standard module
implements event-flow semantics, it is a straightforward
matter to replace this with one that implements pure data
flow semantics.
Third, the scheduler is also a replaceable module.
Currently, we have implemented simple round-robin
scheduling. However, we recognize that specific real-time

applications require other algorithms, such as rate-
monotonic, earliest deadline first, best effort, and so forth.
We have therefore designed in the appropriate hooks for
other algorithms.
Finally, the Sonnet execution engine is defined and
implemented as a server. As a result, the GUI may be
replaced, or may run on a different machine. In fact, no
GUI need exist; the engine can run “headless”.
The Sonnet environment supplies a basic set of
components for control, logic, and mathematics, and for
string, list, and matrix manipulation. A set of MIDI I/O
components is also included, so that Sonnet programs
may be written that produce and respond in real-time to
MIDI events. Additional component sets include TCP/IP
communication, file I/O, and GUI controls (pushbuttons,
checkboxes, sliders, and so on).
Sonnet is easily extended by adding new component
types and/or data packet types. No data types or
component types are treated specially by the Sonnet
infrastructure (with the exception of chip I/O ports).
Figure 2. A program expressed as a Sonnet circuit
Components are implemented as objects that support a
small number of standard interfaces. Most interfaces are
supported by base classes, so that essentially all that a
component writer needs to supply is an appropriate
Execute() method.
Likewise, data packets are implemented as objects that
support a simple, standard interface. Beyond this, data

packets typically support at least one additional interface
to grant access to the particular kind of data that the
packet holds.
4. Visual Compositions in Sonnet+Imager
Visual compositions in Sonnet+Imager have four
major components: visual objects, rhythm, interaction,
and orchestration. Each of these components builds on the
previous one to provide larger-scale compositional
elements. The components are treated in the four
subsequent sections.
4.1. Visual objects: form and color
In the aesthetic model we have chosen, the abstract
elements of mathematics are also the formal elements of
art. In his book Point and Line to Plane, for example,
Kandinsky established such elements as points, lines, and
planes as more than tools to represent other objects.
Instead, forms became the content of his art, and that of
the modern artists who followed [7]. He, Klee [8],
Gerstner [5], and others wrote extensively about how
these simple elements can be combined and juxtaposed to
represent such abstract ideas as causality, tension, and
growth. The modern artists’ basic strategy of constructing
works from simple elements is well-suited to computer-
based art.
The choice of a model for defining and representing
form is one of the most fundamental that must be faced in
the design of a graphic instrument. Some designers have
elected to base instruments on a single family of forms,
such as kaleidoscopic imagery [4]. Others have provided
a direct mapping of certain musical parameters onto
visual ones. Sonovision, for example, related the

frequency and size of an ellipse to the music’s frequency
and loudness respectively [12]. Some systems simply
expose the forms provided by the underlying computer
graphics toolkit, such as QuickDraw or OpenGL, leaving
it to the composer to build up higher-level forms. Our
approach calls for building a vocabulary from the simple
geometric elements of abstract art. Interfaces are provided
to critical parameters, such as the number of sides for
polygons, or the eccentricity for conic sections. Functions
and envelopes can be applied to these parameters.
Mappings between these parameters and musical events
are then defined at the time of composition or
performance. Given Sonnet’s structure, other aesthetics
could replace and extend the ones we have provided.
Three foundation classes of forms that Imager provides
are regular polygons, irregular polygons, and conic
sections. Other classes, such as string art, fractals, and
ambient forms, can (and will) be added in modular
fashion. This is thought to be one of the ways in which
visual instruments can be personalized and customized.
Artists can make the foundation elements of whatever
aesthetic they wish to work with first-class elements in
the environment. Their newly created types can inherit the
color, movement, and other methods that have been
defined to apply to our basic classes of forms.
We have already described the basic color model that
Imager uses. Other issues arise in controlling color. In
particular, it is necessary to provide high-level constructs
that make it easy to use color to achieve tensions,
resolutions, harmonies, and similar affects. Color can then
be used to connect musical and visual rhythms.
Once we have a color model that corresponds well to

the way the lumianist thinks about color, we can define
interfaces to manage it. To do this effectively, we must
identify appropriate manners in which the instrument
permits the lumianist to control an item’s color (beyond
directly specifying its hue, saturation, and value).
One kind of control was suggested by the work of
color theorists. Ogden Rood, for example, suggested that
harmonious color combinations are found in pairs
separated by 90 degrees on the color wheel, as well as by
triads of colors that are 120 degrees apart [10]. Joseph
Albers noted that strong complementary colors (those
separated by 180 degrees) produce after-images and
vibrations [1]. Faber Birren observed that people find
pleasure in harmonies of color based on analogy (adjacent
hues) and on extreme contrast (complementar y hues) [2].
When adjacent colors are used, the effect is to produce
color schemes that are predominantly warm or cool in
feeling. When complementary colors are used, the result
is more startling and compelling. For color to be played
improvisationally, it is useful to be able to move rapidly
and easily from dissonant to harmonious, or from warm to
cool, motifs.
Sonnet+Imager facilitates manipulating objects’ colors
in just such a manner. For example, a line’s and a ring’s
colors can be controlled by a Sonnet circuit so as to
maintain a complementary relationship. When the ring’s
hue is changed from red to yellow, the line’s hue goes
from blue to purple.
In fact, Sonnet+Imager allows for maintaining many
kinds of relationships among visual forms. The circuit
structure described above can also be applied to shape,
thickness, and texture. These interactions form a basis for
compositional structure. As a result, our architecture

supports both establishing and controlling the structure
and mood of visual compositions.
4.2. Rhythm: navigating parameter spaces
Because Imager is intended to facilitate the creation of
dynamic graphic images that interact with musical
performances, the temporal dimension plays an important
role in its architecture.
Rhythms can be produced through changes in any of
the dimensions (e.g., color, shape, location, and
orientation). Considerations of rhythm pervade Imager’s
design. Manipulation of scale, location, and orientation
are all ways in which objects become animated. Similarly,
changes in the colors, textures, pen shapes, and other
characteristics of objects can define rhythms. In short,
dynamic changes in any attribute of a visual object
contribute to the rhythm of the visual performance.
Our model of visual rhythm has three components:
where, when and what. By decomposing visual rhythm
into these three components, we gain significant
flexibility and opportunity for reuse.
The first component, where, refers generically to a
path through N-dimensional space for any collection of N
visual parameters. Any path through such a space can
form the basis of a visual rhythm. Paths need not be
smooth or continuous in the mathematical sense. Rather, a
path is represented by an arbitrary mapping from real
numbers on the closed interval [0, 1] onto points in N-
space.

This model constitutes a very general and powerful
view of rhythm. For example, the visual parameters hue,
thickness, and x-coordinate form one such 3-dimensional
space. The individual dimensions need not be real-valued;
strings and symbol sets also define valid dimensions.
In practice, many interesting traversals of N-space can
be implemented using a set of N distinct functions. E.g.:
)cos(),sin(,)(,, 321 ttkttfzzz =>=<
describes a path through 3-space in which the x
coordinate follows a linear path, while the y and z
coordinates follow phase-inverted sinusoidal paths.
The second component of rhythm, “when,” associates
temporal information with the path, thereby specifying
dynamics. In particular, a distinct function maps a portion
of the temporal dimension onto the parameter of the path-
mapping function mentioned above. A trivial example
linearly maps the time interval [5, 10] onto the parameter
domain [0, 1]. A more interesting example maps the time
interval [5, 10] onto [0,1] using the quadratic function
20333.7.67.2)( xxxft −+−==
This has the effect of dilating time so that it flows more
quickly in the early portion of the time interval than in the
latter portion, as shown in Figure 3.
Although the most obvious temporal mappings are
monotonic and smooth, mappings need not be. For
example, a sinusoidal temporal mapping defines cyclic
motion along an arbitrary path. That is, the traversal
moves back and forth along the entire path.
Finally, the “what” component associates the dynamic

path with the concrete parameters to be affected. That is,
the above 3-D path can be associated with hue, saturation,
and value, or alternatively with hue, thickness, and x-
coordinate. By decoupling the path’s shape from the
parameters that it affects, interesting paths can be applied
to different sets of parameter, without modification.
This micro-architecture provides considerable
potential for reuse, an essential aspect of any toolbox.
Any particular path through N-space is trivially
substituted for any other, given that the number and types
of the dimensions match. Thus, a sinusoidal path in 2-
space can be used to affect hue and saturation, or x- and
y-coordinates, and so on. Likewise, one can vary the rate
of traversal of a path by substituting another temporal
component. The architecture is illustrated in Figure 4.
Because our objective is to create a platform for
devising a variety of dynamic and interactive visual
instruments, it is important to have ‘knobs’ that are at
once simple to understand and use, and powerful. We
achieve this by expressing all of the above functions using
the same construct -- functors.
4.2.1. Functors. Functors are software objects that
encapsulate arbitrary functions. A functor’s clients need
know only 1) its domain and range, and 2) how to ask it
to compute its value given a value in that domain. In this
manner, the specific function embedded in the functor
(say, a linear curve, piecewise-linear envelope, or sine
wave) is hidden.
Figure 4. A modular architecture for realizing rhythm
Figure 3. Temporal mapping realizing dilation

Path
Functor
("where")
Temporal
Functor
("when")
Parameter
Selection
("what")
x
time
t
<z1,...,zN>
Apply
to Visual
Object(s)
0
0.2
0.4
0.6
0.8
1
5 6 7 8 9 10

More specifically, a class of functors is represented at
the lowest level by a trivial interface with a single method
to evaluate the underlying function. An example appears
in Figure 5, roughly in C++ syntax. The functor shown
embodies a 1-dimensional path, since it maps the real
numbers onto the real numbers.
A particular functor simply needs to support the
appropriate interface (such as OneArgFunctor above),
a to
m
l
F
c
b
e
r
p
p
f
l
w
c
F
c

i
p
s
t
s
f
4.2.2. The Functor Toolbox. The Imager toolbox
furnishes a set of simple functors, including one-
parameter functors such as polynomial (linear, quadratic,
etc.), transcendental (sine, cosine), as well as various
algebraic two-parameter functors, used for combining
functors in intuitive ways.
Perhaps the most flexible type of functor is the
envelope functor. This type of functor encapsulates a set
of samples of a function. Its appeal derives in part from
the fact that the function can be drawn graphically (in
“freehand”), without needing to know its precise
mathematical form. It can be particularly effective in
describing motion trajectories in 2-space: the artist simply
draws the desired path.
4.2.3. Visual Representation of Functors in
Sonnet+Imager. In the context of Sonnet’s visual
language, functors appear in three forms: as individual
primitive components, as circuits or chips, and as packets.
This representation has two advantages. First, the
circuit metaphor employed by Sonnet allows functors to
be “hooked up” to suitable clients using simple “wires”.
No code needs to be written. Type safety is guaranteed by
Sonnet’s normal type-checking mechanism.

Second, more elaborate functors can be composed
interface OneArgFunctor {
double Evaluate(double t);
};
Figure 5. A simple interface defining a class
of functors
nd to support interfaces that provide access
class LinearFunctor: OneArgFunctor {
public:
void SetOffset(double k0);
void SetSlope(double k1);
};
Figure 6. An interface that provides access to both
a linear functor’s parameters and its evaluation
anipulate the function’s definition. An example of a
inear functor that uses OneArgFunctor appears in
igure 6.
The decoupling of client from function has two
onsequences. First, functors over the same domain can
e interchanged without affecting a compatible client. For
xample, given a linear functor mapping real numbers to
eal numbers (e.g. tkktf 10)( += ), we can change the
articular linear function in use by manipulating the
arameters (in this case, 0k and 1k ) that define the
unction. Alternatively, a quadratic functor (such as
2
210)( tktkktf ++= ) can be substituted for the

inear functor. In both cases, the client continues to work
ithout modification.
Second, a given functor can be used with any
ompatible client. For example, a functor embodying a
ermat spiral over 2D space can be used by any client that
onforms to the evaluation interface. The client is free to
nterpret the function as navigating any pair of
arameters, such as x and y-coordinates, hue and
aturation, or saturation and x-coordinate.
Functors can be constructed to encapsulate functions
hat compute non-numeric values, such as strings, discrete
ymbols, visual objects, and so on.
It is also possible to construct functors that embody
unctions of multiple arguments, such as
2
11021 ),( tktkttf += .
from simpler ones by wiring functors together into
circuits. For example, a “wobbling spiral” path can be
constructed by combining two primitive functors (a
wobble functor, and a Fermat’s spiral functor) with an
additive composition functor, as shown in Figure 7. The
packet produced at the adder’s output is itself a functor,
which is called by an additional component at the
appropriate times to produce its result.
This circuit can now be collapsed into a chip, and
reused as a single component.
4.3. Interaction: input and synchronization
Imager’s improvisational performance capability relies

on the ability to accomplish two things based on a
performer’s input: 1) modulating visual parameters, and
2) triggering visual rhythms. Likewise, a significant
aspect of any integrated musical and visual composition
Fermat's
spiral
Wobblemagni-
tude
"wobbly
spiral"
tight-
ness
Figure 7. Compound trajectory built from simple
components
lies in the interplay between the rhythms in the two
domains. The key to establishing this interplay is the
ability to synchronize musical and visual rhythms.
Sonnet furnishes an ideal platform upon which to build
and combine the necessary components. In particular,
interface components can be constructed (e.g. for MIDI,
data glove, dance suits) to allow data flows from external
sources to modulate visual parameters. Likewise, because
data packets can trigger component execution, these …
Journal of Visual Arts Practice Volume 7 Number 1 © 2008

Intellect Ltd
Article. English Language. doi: 10.1386/jvap.7.1.11/1
Intellectual process, visceral result:
human agency and the production of
artworks via automated technology
Michael Betancourt
Abstract
Automated technology that produces art presents specific issues
for interpreta-
tion: where should the artwork be situated – in the objects the
machine pro-
duces, the machine itself, or the design for the machine? Central
to this question
is the issue of human agency in the creation of art. This paper
examines these
issues in relation to the implications of Sol LeWitt’s ‘Sentences
on Conceptual
Art’, and frames the question of human agency in relation to the
work of con-
temporary artist Roxy Paine, and the historical artists Mary
Hallock-Greenewalt
and Richard M. Craig who created autonomous systems for
making visual
music. These artists’ work involves automated technology that
functions without
their intervention. These works suggest a framework that
recognizes the artist’s
role as the designer of art, rather than as necessarily the
fabricator of those works.
In the New York Times art review ‘Computer Generated
Pictures’, published

on 8 April 1965, Stuart Preston made an eerily prescient
prediction in his
review of the first computer art exhibit:
[Some day] almost any kind of painting can be computer-
generated. From then
on all will be entrusted to the ‘deus ex machina.’ Freed from the
tedium of
techniques and the mechanics of picture making, the artist will
simply create.
(Preston 1965)
When one considers the painting- and sculpture-making
machines of Roxy
Paine, for example, it is tempting to believe this situation has
come to pass,
raising the implicit question of Preston’s comment: what does
the artist do
that qualifies as ‘simply create’ if the artist actually needs to do
nothing to
produce a particular artwork? This question is the problematic
provoked by
autonomous systems: if computers can produce art without the
need of the
artist to guide them, then what could be called ‘Preston’s
problematic’
appears: the artist may surrender their agency in the creation of
art to a
machine, rendering themselves irrelevant to the creation of art.
Automated systems for making art present specific problematics
for human
agency, and they apply equally well to earlier developments in
automated
technological art, such as Mary Hallock-Greenewalt’s Sarabet, a

technological
instrument that began as an automated, mechanical system for
visual
accompaniment to music; or in the work of other artists who
constructed
11JVAP 7 (1) pp. 11–18 © Intellect Ltd 2008
Keywords
new media art
conceptual art
process
computer-
generated art
JVAP_7.1_02_art_Betancourt 6/26/08 3:05 PM Page 11
automated colour organs such as Richard M. Craig (Betancourt
2004). The
implications of Sol LeWitt’s comments on ‘conventions’
suggest a possible
framework for thinking about art making in a way divorced
from human
activity, if not human agency.
LeWitt’s ‘Sentences on Conceptual Art, 1968’ includes four
sentences that
directly mention working with/within the ‘conventions of art’.
These sentences
are significant because they propose that art making is
explicitly a matter of
conditions that determine the work of art by creating a network
of potential

approaches to its facture. As a whole, the twenty statements
suggest that the
conventions of art are historically contingent objects (Poggioli
1968: 56),
directly influenced by the art they describe. This is a dynamic
situation,
where the conventions constrain the potential objects, which can
be art as
they are affected by those objects (Eco 1984: 66–67). These
sentences are
representative of a shift in conceptual art from the creation of
objects to
the generation of specific conditions for the creation of art
based on ‘the artist
or somebody else fabricating or describing the piece as equal
conditions
for the production of [the art], thereby abolishing the notion of
artist
centered production’ (Alberro 2000: xxii–xxv), a situation that
has explicit
implications for automated art, since the role of the artist is
extremely atten-
uated when compared to, for example, a handmade (manually
produced)
painting.
However, where conceptual art abolishes the object as well as
the artist’s
physical labour in creating the artwork, LeWitt’s ‘Sentences on
Conceptual
Art’ still assumes that there will be some type of object
produced from the
artist’s activity; this feature of his articulation makes his
comments more
directly applicable to any theorization of human agency in
relation to

automated art-making systems. The significant factor in such a
framework
is how the human agency is separated from the mechanical
aspects of
actually making the work. LeWitt’s comment that ‘The idea
becomes a
machine that makes the art’ (Lippard 1997: 28) is a good
example of this
framework in action and provides a point of reference for
considering these
statements as the basis for a theory of human agency:
(17) All ideas are art if they are concerned with art and fall
within the conven-
tions of art.
(18) One usually understands the art of the past by applying the
conventions
of the present, thus misunderstanding the art of the past.
(19) The conventions of art are altered by works of art.
(20) Successful art changes our understanding of the
conventions by altering
our perceptions.
(LeWitt 1972: 174–175)
While LeWitt mentions ‘conventions’, and uses the term as if it
is unques-
tioned what these ‘conventions’ may be, the precise nature of
that con-
cept is unclear. Understanding the specific nature of these
conventions
clarifies their application to contemporary work by artists, such
as Roxy
Paine, where the question of human agency is central: when
devices create
art autonomously, determining what constitutes human agency

is neces-
sary in order to decide what constitutes the art. What is at stake
in this sit-
uation is the status of the objects produced; this interpretation
assumes
12 Michael Betancourt
that art is dependent on human agency, so the relationship
between human
agency and the autonomously produced object is crucial.
Creating a framework for human agency in relation to
autonomous sys-
tems depends on understanding the meaning and role of
conventions. The
two most apparent approaches to the meaning of this term are
not directly
opposed, but neither do they provide an understanding of what
can be done
with the conventions. The first would take conventions to mean
those rules
for selecting from possible objects for inclusion in the category
‘art’, and
would provide guidelines for ranking those objects
hierarchically. The second
understanding of this term shows methods to interpret the works
already in
the category called ‘art’, and aids in developing arguments
about their
significance and meaning (Danto 1990: 217–231).

Both approaches to the meaning of conventions assume that the
received meaning of art (i.e. the traditional definition) is not in
doubt. Such
a usage of conventions takes the form of a lexicon of
established ways
and means – practices having particular meanings as a result of
their
repeated use. Making art with this view of conventions is
essentially
academic; the conventions are received dicta not open to
question, debate
or investigation.
However, within LeWitt’s statements there is a third
understanding of
these sentences that suggests that conventions are directly
connected to
the process of making the works themselves: a set of rules
invented by the
artist based on their particular interests, using history as source
and refer-
ent. This approach can be understood in terms of (a) avant-
gardism and
those objects that may be included as art; or (b) interpretation,
work com-
menting on its historical embeddedness; or (c) as a comment on
the direct
influence that an artwork can have on subsequent work – how
all art is
interpreted by each new work’s expansion/comment on the
history of art.
The third interpretation is one which emphasizes process. This
would be
the ‘productive’ interpretation of the ‘Sentences on Conceptual
Art’.

This third interpretation raises (again) the question of human
agency in
making art. Approaching conventions as a set of rules whose
application
will determine a particular work enables a view of human
agency separated
from the fabrication of the artwork: an approach that has ample
historical
support ranging from Marcel Duchamp’s readymades through
the use of
commercial reproduction, and even the historical debate about
photogra-
phy as art. It is a history where the removal of direct, clear
human agency
(as in manually produced artworks) results in a denial of the
status ‘art’.
Each of these historical examples raises the question of human
agency
anew. They are all examples of Preston’s problematic where
deciding a work
is art demands a new consideration of human agency in regard
to fabricat-
ing artworks.
The concept of ‘axioms’ from the field of mathematics provides
a useful
parallel for conventions in art. Viewing conventions as axioms
makes them
into a function of the framework used in the working process.
LeWitt states
that these constraints – when drawn from the history of art –
form the con-
ventions an artist employs in the process of making artwork.
The crux of his
sentences is the relationship an artist has to the source of those
conven-

tions: given the breadth of art history, it is arguable that these
conventions
could take potentially any form; his comments suggest that
creating the
13Intellectual process, visceral result: human agency and the
production …
boundaries necessary for working becomes part of the artist’s
work (or is
the art itself ).
Art objects exist simultaneously as objects for interpretation
and as
expressions of specific conventions whose relationship may not
be self-
reflexive: the conventions themselves are separate from any
meaning that
may result from following those conventions. The concept of
‘style’, as
used to describe traditional art, can be understood to perform
this role. The
creation of a convention set (whether explicitly as conventions
or implicitly
through style) provides the logic of the sequence. These
conventions pro-
duce a network of potentials that enables the creation of the
individual
work. This framework necessarily constrains the physical form
of the work
and the possibilities for depiction within that form.

LeWitt’s conception of conventions suggests a radically
different view of
the artistic creative process, one much closer to mechanical
design than the
action of ‘genius’. The artist chooses/creates the conventions
that are the
operational parameters for the artwork, then constructs objects
within that
field of potentials. Whether this fabrication is manually or
autonomously
done makes no difference to the issue of human agency because
the crucial
decisions about the art takes the form of conventions, rather
than the
artist’s labour in making objects. LeWitt’s sentences propose a
typical
approach to conceptual art that can allow the instructions to
become the
work, eschewing objects in favour of the audience mentally
creating the art;
Yoko Ono’s fabrication of her ‘instruction paintings’ in the
1990s (Ono 1995)
demonstrates the feasibility of these instructions in the actual
creation of
objects.
When making art becomes an autonomous process, the
aesthetics of
the work are literally built into the machine itself. This issue is
especially
clear with an entire category of art-making devices called
‘colour organs’.
Mary Hallock-Greenewalt, an early developer of colour organs
(and inventor
of new technologies), described two specific potential
approaches to colour

music (Hallock-Greenewalt 1946): (a) the live, virtuoso
performance and
(b) automated performance, where a machine performs using a
pre-
arranged set of instructions for colour music (Betancourt 2005).
The auto-
mated approach (b) has been employed by various other artists
whose
machinery responds to music following a predetermined set of
potentials;
this latter variety is the ‘music visualizer’ now a common
feature on most
computers.
What these machines all have in common is that the decisions
about
the relationship between sound and image must be made by the
artist a pri-
ori to the construction of the device: the visualizer cannot exist
without
these conventions. The determining factor separating one music
visualizer
from another is what it does – i.e. what conventions it
establishes in how it
visualizes music.
Hallock-Greenewalt’s machines create very different effects
from Richard
M. Craig’s Radio Color Organ. His machine was specifically
designed to gen-
erate autonomous visual accompaniment to broadcast music
(Betancourt
2004: 167–171). Whether the artist or someone else then
manufactures the
machine is irrelevant since the significant choices – the artist’s
agency as

artist – determine the machine’s function. The artist chooses
which con-
ventions to follow and which to ignore, constrained only by the
technology
available; Hallock-Greenewalt was forced to invent new
technologies to
enable the construction of her visual music instrument, the
Sarabet. This
technologic limitation is not one of aesthetics derived from art
history, but
an engineering problem.
The historical development proposed by Clement Greenberg in
‘Modernist
Painting’ where each successive new art is a historically
mandated reduc-
tion towards purity (Greenberg 1986: 85–93) cannot apply to
works created
within this framework because the conditions which would
allow such
development no longer apply (Danto 1990: 46). Any art that
asserts such a
framework would (ironically) appear either ahistorical or
quotational, rather
than simply a manifestation of a historically mandated art.
LeWitt explicitly
states this situation in ‘(19) The conventions of art are altered
by works of
art’. As Arthur Danto has observed,

The philosophical question of the nature of art, rather, was
something that
arose within art [following the 1960s] when artists pressed
against boundary
after boundary, and found that all the boundaries gave way.
(Danto 1990: 15)
These boundaries were defined by traditions that the avant-
garde systemat-
ically attacked and destroyed between the 1860s and the 1960s,
with the
resulting effect that artists entered into the new historical
moment uncon-
strained by the tradition that Danto describes; LeWitt’s
sentences are a
product of this changed relationship to the past.
Implicit in this framework is an understanding that conventions
are
specifically serial; a particular historical framework presents
constraints on
the potentially available forms (called ‘aesthetics’). Making art
requires the
elaboration of specific groupings of conventions whose choice
is funda-
mentally arbitrary (a matter of the individual artist’s choice
rather than
mandated by any external authority such as an academy or
tradition).
Umberto Eco explains what this conception of ‘seriality’ means:
The real problem is that what is of interest is not so much the
single variation
as ‘variability’ as a formal principle, the fact that one can make

variations to
infinity. Variability to infinity has all the characteristics of
repetition, and very
little of innovation. But it is the ‘infinity’ of the process that
gives a new sense
to the device of variation.[…] What becomes celebrated here is
a sort of vic-
tory of life over art, with the paradoxical result that the era of
electronics,
instead of emphasizing the phenomena of shock, interruption,
novelty, and
frustration of expectations, would produce a return to the
Cyclical, the Periodical,
the Regular.
(Eco 1984: 46)
In describing serial forms, Eco also identifies the atomization
effected by
conventions. The return to repetitious, continuous forms is not
as surpris-
ing as it sounds from Eco’s description. The demands imposed
by the
adaptation of aesthetic methods towards procedures adoptable
by auto-
mated production, with its historical foundations in the
assembly line’s
breakdown of production into modular components joined into a
seamless
whole. Eco’s observation that the ‘era of electronics’ efface s
fragmentation
production …

in favour of continuity is a return of the classical idea of
‘transparency’
where the actual manufacture of the artwork is hidden from
view, instead
presenting an illusion of seamless facture; this transparency is
also a goal
of designers who develop computer software (Pold 2005: par.
6), suggest-
ing a possible inverse correlation between the experience of
interruption
provided by the technology and transparency, i.e. the greater the
interrup-
tion, the more it will be used to create a transparent experience.
As Paine’s machines or Hallock-Greenewalt’s Sarabet
demonstrate, con-
structing a machine of any type is an issue of engineering, and
building a
device capable of producing the desired result requires an
explicit descrip-
tion of any physical product that particular machine will make.
An explicit
iteration of conventions is required for the design of machinery
for the auto-
mated creation of artworks. Machines for fabricating art
therefore make the
conventions of art an immanent part of their design; i.e. their
conventions
(the aesthetics they employ) are an emergent property available
through
examining their design. These conventions are also apparent
through the
product (the art) created by the machine’s operation. At the

same time, the
approach to creating art this framework describes is one in
which the artist
assumes the role of designer, in effect separating the fabrication
of the art
(including the manufacture of the machinery) from the decision-
making
process that leads to that art.
Situating human agency at the decision-making stage of the
art’s pro-
duction creates two distinct ways to view the function of the
framework in
producing the art. Emphasis can be placed on either (a) the
machine itself,
or on (b) what the machine produces. Robert Scott’s discussion
of Paine’s
robots for making art – built to function like an automated
assembly line –
clarifies this issue:
Paine’s PMU (Painting Manufacture Unit), 1999–2000 provides
a perfect
example of this rigorous engineering… . To make just one
painting, the
process [the machine follows] may repeat itself between 80 and
200 times,
without any intervention whatsoever from the artist… . Paine’s
insistence on
this technical exactitude and lack of superfluous detail reveals
his surprisingly
mechanistic understanding of the inventions he aptly regards as
‘labor saving
devices’. By automating processes lasting from many hours to
many days,
they spare the artist’s time and attention, rendering his engaged

presence
irrelevant to the creation of the work.
(Rothkoff 2001: 21–25)
Paine’s use of the computer as a technical device avoids the
physical labour
of the artist, since once set in motion it will make art without
any further
need for assistance. If the artist’s programming and design of
the machine
gains emphasis, as in Scott’s description of the Painting
Manufacture Unit,
it makes the result simply a proof that the machine does what it
should.
This ‘proof of function’ interpretation literalizes LeWitt’s
comment that ‘the
idea becomes a machine that makes the art’. Viewing what the
machine
makes as a proof of function renders those works simply an
iteration of fol-
lowing the instructions, specifically displacing the art from the
fabricated
works onto the machine itself and its processes, something
Paine’s concept
of labour-saving devices argues against. The same lingering
scepticism that
denied photography the status of art for the first century of its
existence is

evident here in the tendency to deny the art produced by a
machine status
as an artwork, even though that machine follows the artist’s
specifications
precisely.
What is at issue with these works is the question of human
agency in
the creation of art: what role must the artist have in relation to
the creation
of the art for it to be immediately acceptable as art? Paine’s
work, while
provocative in this regard, ‘plays it safe’ by exhibiting both the
mechanism
and the results of its functioning. The presentation of both
autonomous
machine and its products together avoids the problematic this
work sug-
gests by allowing the proof of function interpretation, an
approach that is
less readily available if the machine is not exhibited in
conjunction with the
work produced.
This paper proposed a framework for understanding art making
based
on human agency divorced from direct human action that is
implicit in the
idea of conventions suggested by Sol LeWitt in his ‘Sentences
on Conceptual
Art, 1968’. The ability to break the process of making art into a
set of
descriptive instructions (the network of potentials that
conventions
produce) is a prerequisite to being able to produce art-making
machines.

By focusing on the decision-making aspects of human agency in
determining
the final artwork rather than the active, physically engaged
dimension of art
creation, it becomes possible to consider the distinctions
between these
two activities and the ways they combine to literally inscribe
aesthetic con-
cerns and beliefs about the nature of the work in the
construction of
machineries for making art. This framework recognizes the
artist’s role as
the designer of art, rather than as necessarily the fabricator of
those works.
Understanding the creation of art in these terms extends the
recognition
that skilled labourers employed by an artist (in the form or
technicians or
assistants) in making art can be applied and understood as in the
use of
automated technology as well.
References
Alberro, Alexander (2000), ‘Reconsidering Conceptual Art’, in
Alberro, A. and
Stimson, B. (eds), Conceptual Art: A Critical Anthology,
Cambridge: The MIT
Press.
Betancourt, Michael (ed.) (2004), Visual Music Instrument
Patents, Vol. 1, Holicong:
Borgo Press.
——, (ed.) (2005), Mary Hallock-Greenewalt: The Complete
Patents, Wildside Press.

Danto, Arthur (1990), Beyond the Brillo Box, Berkeley:
University of California Press.
Eco, Umberto (1984), Postscript to The Name of the Rose,
(trans. William Weaver),
New York: Harcourt Brace Jovanovich.
Greenberg, Clement (1986), The Collected Essays and
Criticism, Volume 4, Chicago:
University of Chicago Press.
Hallock-Greenewalt, Mary (1946), Nourathar: The Fine Art of
Light-Color Playing,
Philadelphia: Westbrook Publishing.
LeWitt, Sol (1972), ‘Sentences on Conceptual Art, 1968’ in Art-
Language, 1:1, 1969,
reprinted in Meyer, U. (ed), Conceptual Art ed. Ursula Meyer,
New York: Dutton.
Lippard, Lucy (1997), Six Years: The Dematerialization of the
Art Object, Berkeley: The
University of California Press.
production …
Ono, Yoko (1995), Instruction Paintings, New York:
Weatherhill.
Poggioli, Renato (1968), The Theory of the Avant-Garde,

Cambridge: Harvard
University Press.
Pold, Soren (2005), ‘Interface Realisms: The Interface as
Aesthetic Form’ in Post Modern
Culture, 15:1, par. 6.,
www.iath.virginia.edu/pmc/current.issue/15.2pold.html.
Preston, Stuart (1965), ‘Reputations Made and in the Making’,
in The New York
Times, 18 April.
Rothkoff, Scott (2001), Roxy Paine, New York: James Cohan
Gallery.
Suggested citation
Betancourt, M. (2008), ‘Intellectual process, visceral result:
human agency and the
production of artworks via automated technology’, Journal of
Visual Arts Practice
7: 1, pp. 11–18, doi: 10.1386/jvap.7.1.11/1.
Contributor details
Michael Betancourt is an independent scholar, curator and new
media/installation
artist who writes critical theory. His papers have appeared in
Leonardo, CTheory, The
Millenium Film Journal and Semiotica, and he has edited
several anthologies on
visual music technology and art, as well as having written the
book Structuring Time,
a theory of motion pictures based on the historical avant-garde.
Contact: 1718 Summit Street, Sioux City IA 51105, 712-252-
9595.

E-mail: [email protected]

In 1999 IEEE Symposium on Visual LanguagesTokyo, Japan, pp. (2)

Recommended

Recommended

More Related Content

Similar to In 1999 IEEE Symposium on Visual LanguagesTokyo, Japan, pp. (2)

Similar to In 1999 IEEE Symposium on Visual LanguagesTokyo, Japan, pp. (2) (20)

More from he45mcurnow

More from he45mcurnow (20)

Recently uploaded

Recently uploaded (20)

In 1999 IEEE Symposium on Visual LanguagesTokyo, Japan, pp. (2)