Animation and Sound: Visualizing Character Performance

By Mikhail Lebedev
AKV St Joost
Tags: Music video, sound design,
character animation, acting
Animation and Sound

2
Abstract
This research paper analyzes the close relationship between sound and visuals. The majority of the
research is focused on how to use sound theory to improve and explore possibilities of storytelling,
character animation. The research method is particularly selected to study sound trough the perspective of
animator and/or unexperienced sound designer. Anyone who is going to deal with sound in their practice
can benefit from reading this paper. All of the interviews in this paper are presented in a way to support or
supplement the main discussion. Furthermore, this paper refers to different examples of music videos
including my own as well as films that help to demonstrate the theory better. I should mention that this
research helped me to enhance my graduation project that I will describe further in this paper. I believe
that sound and visuals should be studied together because they are interconnected and each one of them
add something useful to the project. Hopefully, this research will help to bridge the gap between
animation and sound design fields without going deep into technical side of sound design.
Main research questions and themes:
-When do music and sound ‘marry’ visuals?
-How does sound speak about visuals and the other way around?
-How can animation benefit from sound design?
-How can performance of characters in animation be supported and exaggerated by music
and sound design?
Sub-questions:
-How to tell a story by performance of the character? Acting for character animation.
-How to achieve believability in character animation?

3
Acknowledgements
My research paper would not be successfully completed without guidance of my supervisor Sarah
Lugthart. She supported my project and gave constructive advices along the way of writing my novelty
research paper. I would like to thank her for spending her valuable time giving me advice on writing my
research paper. Also I would like to express my sincere gratitude to Leon van Roy. Without his support I
would have spent a lot of time figuring out a way to solve some problems of my graduation project.
Additionally, huge thanks goes to all of the Avans students that shared some valuable advices with me.
Big thanks to people from animation department that helped me out with the story development process of
my animation project and supported me along the way.
Finally, I would like to give credits to my family and friends who supported me throughout my research.
Their support and advice was very helpful in achieving my goal.

4
Contents
Abstract................................................................................................................................................................. 2
Acknowledgements..............................................................................................................................................3
About the author...................................................................................................................................................5
Director statement & brief project description...................................................................................................6
Origin of project...........................................................................................................................................6
Synopsis/Themes .........................................................................................................................................6
Story..............................................................................................................................................................6
Technique and style.....................................................................................................................................6
Introduction ..........................................................................................................................................................8
Benefits of studying sound design theory ..................................................................................................8
Brief history of sound and animation.........................................................................................................9
Brief sound theory and cognitive theory..................................................................................................11
Music video possibilities...........................................................................................................................14
Visualizing story and environment through music ..........................................................................................17
Sound as a story telling device..................................................................................................................17
Determining a visual style and mood of the story ...........................................................................................19
Visualizing the character & promoting character through sound ...................................................................21
Purpose of the character or characters......................................................................................................21
Believable character animation performance...........................................................................................21
Realism in animation.................................................................................................................................22
Multizvuk ‘Sound Design’ in animation..................................................................................................24
Visualizing camera movements & editing of the video...................................................................................27
Conclusion..........................................................................................................................................................28
References ..........................................................................................................................................................30

5
About the author
Let me introduce myself and tell about my position in animation field. I was born in Kyrgyzstan. It
happened that I started my journey in animation spontaneously, due to my child’s curiosity. One of my
friends was experimenting with animating ‘stick-figures’ in ‘Gif-Animator’ program on the PC. That was
the beginning of my journey into the world of animation. There were several animation studios in my
country at that time which I applied as intern and learned more about the ‘pipeline’ of animation. This
helped me to produce my own short cartoons. As everyone else around the world I was inspired by
‘Disney-studio’ animation. In addition to that I was also inspired by a lot of soviet Russian cartoons like
‘Бременские Музыканты’ (The Bremen Town Musicians) by ‘Soyuzmultfilm-studio, that’s why in my
research I mention some of them. For me ever since that time technological and esthetic advancement in
animation was the key. Unfortunately, I was not opened to experimentation with as many styles of
animation as possible and most of my focus was on 2d and 3d-animation. One of my goals was to enrich
the field of animation in my home country by doing a study abroad. My goal during master of animation
program was to develop and apply my acting skills, animation skills and music skills to create more
believable and astonishing character animation as possible. The main idea is to learn how to deliver
emotions through body mechanics and expression of the character and not necessarily through dialog,
where sound and music plays an important role to add certain ‘texture’ to the main storyline and character
animation. In other words, with music and sound design I would like to amplify the character animation.
MA Animation was my chance to connect with professionals in the field as well as to grow as character
animator. I decided that a good way to get experience of working with music and visuals would be by
doing animated music video.
‘The Aqua Life’ Here music and sound sets
up the mood and rhythm for animation.
‘Сон ли это' (‘Is it a dream?’) Here
composer had to emphasize the mood and
tell story through music.
Is it a dream?

6
Director statement & brief project description
Origin of project
After working on a small video mapping cooperation project with Bachelor student I was asked to
think about working on a bigger video mapping project for a German music band’s release of the album
“FRÉ”. Before this proposal I had an idea of integrating character-driven animation and video mapping. I
thought that it might trigger people’s emotions better and successfully help to tell a story. Unfortunately, it
was not possible to connect live performance with character animation because It would take a lot of
unnecessary attention. Finally, I decided to work on more abstract animation for the performance, and
implement my character animation in a music video on one of the songs from their album that’s called
‘Sun’.
Synopsis/Themes
The project is about many subject matter focusing on Environment in particular “colors, the sea,
the sun, weeds, grains of sand, raindrops, bees, trees, the moon and ice”. These songs are very light on the
lyrics considering that they are quite easy to grasp and about things that everyone is familiar with. The
topic of this project is focused on the awareness and appreciation of environment. I constructed a story
that is not literal translation of the songs nevertheless brings the topic of the songs in an interesting way.
My focus is to tell the story that is consistent and more importantly influenced by the music and lyrics.
Story
The story for the song “Sun” is about realization of the main character that sad moments are
subjective and that we should always look on the brighter side. The song has a very light jazz accompany
that suggest imaginative world of the character. The story starts with a sad girl walking in abstract almost
completely dark environment. At the beginning of the music video she wonders in the sky realizing that
sun is always shining even though it might be hidden behind dark clouds that serves as a metaphor. Then
she walks on the ground (rain starts) and see that these rainy days are very important for nature and thus
we must appreciate then and go along. Finally, she realizes that it’s important to see positive things and
help others.
Technique and style
The style I decided to work on is 2d with additional 3D for depth effects. I don’t want the style to
be complicated, character animation is important focus thus I keep background simple. It is also motivated
by the story, because it happens to be her imagination thus could be abstract. (See visual examples on the
next page for character design and environment design mood boards)

7
CharacterDesign Moodboard for the music video
Environment Design for the music video done by me

8
Introduction
It is almost impossible to imagine a film or animation without a sound. Even though visual and
auditory perceptions are two distinct processes nevertheless they are interconnected. You might have
noticed that when you listen one of your favorite songs you can instantly recall important events or
emotions that you had the first time while listening to the song. Scientists also find out that when we read
something, we audition it in our head that suggests a close connection between the sound and visuals.
Sometimes I notice that certain movements in animation with a muted sound can still generate rhythms or
sounds in my head. Our perception of visuals and sounds are interconnected even though it varies from
person to person. Sound design is a communication tool and a universal language. It is quite easy to
distinguish sad from happy leitmotif. Many professional animation studios such as Disney know it very
well and use it in their films to bring certain emotions, exaggerate character performance/acting and
finally make people emotionally connect with the story. Even though it sounds very robust, that there are
ways to manipulate human emotions, knowing theory and its practical implementation is beneficial. I hope
that my input in the development of this topic is going to be useful for artists who will have to work with
sound.
My main research question is how sound theory can be connected to visuals? After reading this
research paper you would not be able to become a sound designer but you will find information on how to
use sound theory to improve and explore possibilities of storytelling, character animation. Therefore, this
research helps to bridge the gap between two fields without going deep into technical side of sound design
such as tools, software & hardware. My approach it to study sound trough the perspective of animator and
not the other way around. This paper is going to be very useful for music video production, because it
will be connected with practical implementation of sound and animation theory through my animated
music video project. In addition, any animator or visual artist might find useful information in this paper
because they will deal with sound one or the other way. Finally, I would like to point out that both visual
and auditory are equally important and should not be compered in terms of dominance over each other but
rather should be seen as added value to each other. Thus I believe we should study both, and analyze how
they interconnect.
Benefits of studying sound design theory
Sound can add seriousness to your project or make it look funny, it can add additional information
and help you clear things out or it can make it unbearable and confusing. Sound is a general term for
everything like music, sound effects etc. Learning more about sound design can help to deliver your ideas
successfully. Music can add texture and mood, describe environment and characters, play with feelings
and tie down all the scenes. Sound effects can help deliver certain physical characteristics of the objects in
your scenes or add additional level of physicality and make almost any motion look believable. Like
Beauchamp says “Sounds have the potential of revealing or clarifying the underlying meaning or subtext
of a scene.” (Beauchamp, 2013) Carefully selected sound/sounds can deliver complex idea without the
necessity of showing it visually. For example, in TV series Fargo – season 1 when Molly (Allison
Tolman) realizes who is possible a killer, we see close up of her face with a subtle facial expression that
would not be able to convey the full emotional confusion without subtle musical score. Sound can touch
much deeper unconscious emotions. I think that we are much more critical and conscious when dealing
with visuals in comparison to sound. Imagine hearing a very loud siren, without any doubt it will strike

9
your attention. There are much more approaches toward conveying ideas and emotions with sounds than
what I can say in my research. Most important is to be open to this wonderful and universal language that
is evolving every day!
There is a saying that goes something like this “Film is the gun and sound is the bullet”, which
means that they are both interconnected and the gun/film could not function without bullet/sound. There is
a bit of truth in that saying, but I like to think that every sound designer or animator working with the
sound should approach sound consciously and creatively, especially in animation. We should not always
audition everything that happens in the scene, but rather think about how it is going to effect the scene and
what does it adds. Silence is our friend in sound design too, because it can provide tension,
mysteriousness, discomfort and all the other feelings depending on the context. In addition to that take
into consideration that bad/poor audio quality can ruin a good visual/video. So keep it simple and never
underestimate the influence of audio on visuals!
When you work from sound to visuals, like when you work on music video production for
example, knowing sound design and music theory can help you deconstruct the song, find out cue points,
define rhythm, texture etc. that can be used to visualize the song better and simplify many things. There
are no rules there are only ‘conventions’ that can help ‘smooth out’ production process. Remember that
creativity is important but theory and practice goes hand in hand. Theory in this paper hopefully can help
simplify pre-production and post-production phases of the animation projects that is related to sound. Just
like being exposed to a new language grammar, being exposed to a sound design theory will help you to
recognize patterns and use it in your project!
Brief history of sound and animation
I discovered two main approaches: from sound to visuals and from visual to sounds. Fist one is
used mostly in music video production, where creative team get song or music from artist that they have
to visualize. This approach was also used in many Disney cartoons such as Fantasia (1940) which was
“…honoring of the organic life of music to which the trickery of animated imagery could only aspire”.
(Philip Brophy, 1991) Unlike Mikey Mouse and Silly Symphonies, Fantasia was more poetic visualization
of classical music. This approach is also used by music video artists, because they need to come up with
visuals for the song. Second approach is what most of the animation studios use. Sound design in this case
goes after production or usually after the ‘animatic’ is ready. Basically visuals and story in this case is
much easy to lay at the beginning and not after the sound. Sound thus adds subtle layer of mood, texture,
additional information and supports visuals instead of playing the dominant role of storytelling.
Many passionate artist and scientist like Norman Mclaren, Evgeny Sholpo, Oskar Fischinger were
investigating sound and its relationship with visuals. ‘Drawn Sound’ was a method of creating sounds
using for example ‘Pen Point Percussion’ method or ‘Variophone’ mechanism. I find it interesting how
Oskar Fischinger in “An Optical Poem” (1938) was able to describe sounds and music in general by
animating different patterns, shapes and colors. Take for example the moment at ‘2:15’ in “An Optical
Poem” (1938) where very smooth string instruments fade away and percussive-string instrument starts
playing its part, we see how it is reflected through visuals. First of all, contrast of the shape colors defines
instruments ‘timbre’ sound is bright and visuals are bright. But notice also how those patterns start to
flicker instead of slowly fading away which defines sound quality. Last but not least the size of those

10
shapes connect with ‘pitch’ of the melody. Usually highest notes are seen more fragile therefore small
shapes.
Scientific investigation in brain study brought many interesting suggestions on why some people
can ‘see sound’ more than other. One of the examples is sound-induced flash illusion that demonstrate
how our brain can trick us into visualizing things that are not there (Charles Q. Choi, 2012). Beeping
sound is played over visuals of a circle appearance on screen. Even though circle appears once over two
beeping sounds, some people tell that they see circle second time. Researcher Benjamin de Haas, a
neuroscientist at University College London says that smaller visual brain can make use of additional
information provided by ears (Charles Q. Choi, 2012). By analyzing sound our brain localizes sound,
define it and connect it with visuals. But analyzing music is far more interesting subject, in my view,
because it is a universal language that is constructed by humans to define many things but also tell stories
and transmit emotions. I relate to the idea in the article ‘Commercial Salad of imaginary’ that “Interacting
layers of perception may be instinctive, inter-subjective and individual, which in turn activate social
aspects such as family, peer group, region, country, language etc.” (Sven E Carlsson, 1999) That suggests
that analyzing music can tell us what visuals the audience might expect.
Early in the history, sound designers where real live instrument performers such as piano, organ
players or the whole orchestra. Each time they had to play it live which was limited in terms of quantity
and quality. In anyways mood that musicians create was important for entertainment and storytelling.
During that time people were not able to record ‘diegetic sounds’ such as dialogues, thus music performed
live was important delivering certain emotions. It “should make the audience feel more deeply about the
situation.” (Frank Thomas and Ollies Johnston, 1981, p.298) In the early 20th
Disney were experimenting
with recording sounds. The most known early animation with sound by Disney is Steamboat Willie
(1928). Disney animators had to work with sound and meet character gestures in an appropriate rhythm.
They had to develop a workflow that would allow them to “bring the music and picture together.” (Frank
Thomas and Ollies Johnston, 1981, Ch.” Timing, Spacing and Metronome”) When Disney were creating
Fantasia (1940) they had to really on music in a greater extend. Music was providing many different
aspects including mood that was influencing further development of the movie. “It was decided to record
the sound first!” (Frank Thomas and Ollies Johnston, 1981, Ch.” Timing, Spacing and Metronome”) This
approach could be compared to the one that is used in the music video production. Animator in that case
need to deconstruct the music or song into instruments, lyrics, rhythm, melody and structure that would
provide him with a base to analyze the music and develop visuals. This made their work a lot harder but
trained them to animate precisely to music.
Earliest sound design was limited to the analog instruments, that could be found in orchestra, and
other stuff that ‘Foley sounds’ resemble. You will notice the use of ‘Foley sounds’ in Silly Symphony the
Skeleton Dance (1929) as well as hear instruments that add rhythm and mood. Further sounds were
recorded digitally, that opened a lot of doors for experimentation in sound design. Warner Brothers was
using real world sounds such as acceleration of the car or sudden stop. “This out-of-context use of real
world sounds soon became the hallmark sonic characteristic of Warner Bros. Animation.” (Kate Finan,
2015) Syncing of sound and animation became much easier. “In the late 1950’s, television animation
blossomed largely due to a more economical style of animation which was dialogue rather than action-
based.” (Kate Finan, 2015) Thus classical method of sound design and animation that was used in Fantasia
for example was not considered commercially valid anymore. Animating to sound approach is mostly use
in music videos nowadays and non-dialog cartoons driven by music and sound design are mostly seen in

11
student or ‘author animation’. Hanna-Barbera (1960) for example resembles economic style both visually
and sound-wise. ‘Non-diegetic’ laugh sound is often used in such series or movies to add studio comedy
effect. The role of the sound designer changed through history from “percussionist, to mechanical
engineer, to digital expert”. (Kate Finan, 2015) With the development of technologies possibilities of
creating something new and unordinary are expanding. Almost everyone can record high quality audio in
their home studio without any need to spend a lot of money on equipment. Anyways, creativity plays the
most important role in animation and sound design as usual, and it requires experimentation and research.
Brief sound theory and cognitive theory
When we are watching film or listening to the music we often not ask questions why do we enjoy
it or feel certain emotions and why are we being manipulated by the sound. In this chapter I would like to
quickly go over some of the main sound theory that is going to help us understand sound better. “If we are
to derive meaning from sound, we must first perceive and respond to the sound through active listening.”
(Robin Beauchamp, 2013, p. 2) That’s why we will look at different aspects of sound and how they can be
used in practice.
As I mentioned earlier there were many experimentations with sound and visuals. Norman
McLaren was drawing sound and he was drawing it in a particular way. Sound is a physical vibration that
go through medium such as air into receiver which are our ears. We can visualize sound with a sinus
wave, saw wave, square wave and other basic sound waves. (pic below)
McLaren was drawing sounds in a similar fashion, by playing with spacing between paint strokes, their
width, height and by varying their shapes. (pic below)
Therefore, many sound designers were able to create their own sounds by taking into account theory of the
sound wave. Further we will learn about the theory sound wave in more detail.
Let’s start by examining different properties of sound. Acoustic sounds are much more
complicated than any synthetic/digital sounds. Even though you can get very close to recreate any musical
instrument’s sound digitally, there are a lot of properties of the sound that our ear will find missing and
easily recognize if its digital. It is possible to play any note using sin wave by stretching it to the sides, but

12
even though ‘tones’ of the sin wave and the guitar are going to be same, in other words notes they will
produce are similar, the ‘timbre’ qualities are going to be different. (pic below is a visual representation)
Some sounds might be generated digitally or mixed out of many other instruments to create the desired
‘timbre’ quality. But what is most important to know about ‘timbre’ is that it helps to describe the sound
as being soft, hard, synthetic or analog and so on. That is especially helpful to know when you
communicate to a sound designer what do you want to hear in your animation.
‘Timbre’ can describe texture or character of the visuals. Let me explain what I mean. For
example, Marsha Onderstijn in her animation ‘The life of death’ (2012) picked guitar sound and in the
interview told me that it was not a random decision. First of all, she said that the sound must be acoustic
and very minimalistic because of the topic connected with the nature. Second she wanted everything
played by guitar instead of piano for example. ‘Texture’ includes ‘timbre’ and several other qualities
of sound. “Texture is about density [of the sound].” (International School Bangkok, 2012) I will come
back to other qualitied of sound that are under the ‘texture’ further, but for now let’s find out why and
where you will need ‘timbre’. We might prefer certain kind of sound based on ‘timbre’. It is closely
connected to genres in music such as EDM, rock, classics etc. because each one of these genres
incorporate specific instruments and set specific sound quality based on the expectations of the genre. If it
is EDM you would usually use drums that are prerecorded or computer generated that usually sound
artificial and less real. In classical music you will have much less experimentation with sound because it
tends to leave the sound natural even though you might come across some variation and experimentation
with ganres such as classical dubstep or orchestral trance music. ‘Timbre’ can influence the mood and the
physicality of the scene. For example, Marsha Onderstijn had in mind specific mood/feel for her movie.
She tried 3 times until she was able to make it sound the way she wanted. First improvisation and solo that
sound designer sent to her was done with electric ‘overdrive’ guitar that sounded a bit scary. So even one
instruments can have many ‘timbres’ depending on how you adjust the sound. You can make guitar sound
like a ‘slappy’ bass or like a very soft violin. Tinus Kardolus from ‘animate the world’ studio told me very
interesting things about why he decided to build the whole score on organ and keyboards for the
animation “The Bald and The Beautiful” created by Jeoffrey van Overveld. “[Sound designers] use organ
and piano in soap opera to exaggerate dramatic scene.” Because the main character from the movie acts as
if he was in a soap opera, dramatically reacting to insecurity of not having hair on his head and always
following trends it was decided to use specifically this instruments and ‘timbre’. Of course you might
notice that there are some places where we hear guitar, but it plays over the organ and is used in addition
to the main instrument to add contrast. So it’s important to think about ‘timbre’ of the sound in order to
set certain mood in a scene.
‘Pitch’ could be low, high and everything in-between, but there are limits of what human can hear.
Even though our sound system ‘frequency range’ can be wide for example 20hz to 20khz not everyone is
able to hear it with their ears. But even though someone will be able to hear the whole range nobody
would use the whole range at once but rather use it selectively depending on the song, genre and so on. If

13
you use a lot of bass and a lot of highs, you might have some problems playing it on your low budget
sound set up. That’s why it’s good to have separate speakers like subwoofer for lows and normal speakers
for highs and middle frequencies for professional sound design. So do not stack up sounds of the same
volume and frequencies. Further in this paper I am going to mention how ‘pitch’ can suggest the
physicality of the objects in the scene. Simply put, lower ‘pitch’ usually describes bigger, terrifying things
in comparison to higher ‘pitch’.
One of the interesting thing about our hearing is that we are able to filter sounds. If you would go
to a very loud place your brain will try to focus on the specific sounds and filter other that why can miss
certain sounds. This effect is called ‘cocktail effect’ and is used in sound design when you need to focus
audience attention on specific things or usually when camera goes from wide shot to close-up of
something. That’s one of the reasons why sometimes few sounds could make visuals stand out better
instead of putting all ‘diegetic’ – sounds that are heard by actors in the scene, such as footsteps and ‘non-
diegetic’ – sounds that are set over the existing footage, that are not heard by the actors, such as score
everywhere. Sometimes you only need the ambience sound with some ‘hard sounds’ and sometimes you
only need the score that also incorporate the rhythm that can represent the rhythm of the visuals and the
mood of the scene. If you take a lot at “Ball Like Beijing” by Bionik Feat. Lizzo at (0:40) you will see
how just a drum kick can substitute the actual sound of a punch. For music video production it is even
more important that music is not going to be overwhelmed with additional sound effects. It is in fact very
rare to hear any additional sound effects in the music video. In addition to that voice and score if used
together in a scene should be mixed in a way that the voice remains clear. When I was on the FMX in
Germany (2017) we had a nice talk with one of the representative of sound design department from
Disney. He said that voice is the priority in their animation and if they feel that the sound is too loud they
usually leave the voice level untouched but lower or remove other sounds. Thinking about the frequencies
of the sounds that are mixed together is very useful in that case. If the score occupies the whole frequency
range, there will be no space for mixing voice in the way that it will be clearly heard.
‘Dynamic range’ is important part of sound design, because it helps to focus audience attention on
the specific things in the scene. It can also make silence more unexpected and loudness more loud.
Basically ‘dynamic range’ can be seen when we analyze the track you will see how the loudness vary
trough time. If you first have a silent scene with just slight ambience sounds with few ‘hard sounds’ such
as footsteps and then suddenly will have loud action scene you would notice how it has a powerful effect.
‘Dynamic range’ is very important in music production. It is common in the music to have an intro
slowly building up until the bass drop that make bass sound more powerful. Having one loudness for the
whole song makes it hard to focus and boring to listen; same thing applies to the sound design in the
movies. In music videos by analyzing ‘Dynamic range’, or accents in a music, artists can deliver visuals
in a more exciting way. One of my favorite examples is music video “Could This Be” by Noisia at (1:40)
where the plane crashes. We can see how visuals support music and how music adds additional power to
the visuals!

14
Last but not least you could add certain effects to the sound that will make your visuals look more
powerful. “Direct the eye; by using volume, panning, delay and reverb.” (Lisan Peters, 2014, p. 17)
Special effects like delay, reverb, echo and others can describe space and contrast reality. “A dream
sequence for example, is often reinforced with additional reverb to distance the sound from reality.”
(Lisan Peters, 2014, p.17) Visuals are closely related to the mood and space suggested by the music in
Music video “Re” by Nils Frahm. ‘Reverb’ in this music video makes visual world a bit more unreal and
also suggest big space. Further in this paper I will elaborate on the ways of visualizing sound effects from
the music video.
Music video possibilities
Since my main practice is based around music video production I will mention possibilities of
working with visuals for music videos (MV). I would like talk about theory and support it with examples
of some old and more recent clips. This chapter is mostly focused on deconstructing MV and going
through different approaches of creating MV. In the following chapters about pre-production, production
and post-production of visuals for music videos I will go deeper into steps of creating MV.
Music video (MV) – is a form of audio-visual communication in which the meaning is created via
carriers of information such as; (1) the music, (2) the lyrics and (3) the moving images (Sven E Carlsson,
1999) Even though it seems like a special way of communicating ideas, it is constructed out of different
elements of story-telling, cinematography, animation and more. The main difference that sets it apart, for
example from film, is that it places singing performance as the main ‘diver’ of storytelling and visuals.
Even though it might be similar to musicals, where singing is intertwined with performance, MV is far
more open for experimentation with storytelling and visuals. And even though both can show ‘singing
performance’ of actors, MV is usually extended with additional motion picture. The oldest music video
goes back to 20th century and it is called ‘illustrated song’. It has lyrics of the song connected to the
imagery. Many of the music videos were not very popular until the time when major television stations
like MTV were successfully merchandizing songs by showing them on TV. Music video helped song and
artist become popular. One of the approaches was to take a “singing performer and place him or her in a
setting either literally suggested by the song’s lyrics or in one that mirrors the escapist pleasantries
common to movie musicals.” (Sven E Carlsson, 1999)
There are many variations of creating music videos, but standard music videos can be sorted in
three main categories/types. These are performance, narrative and abstract/art clips. One of the typical as
well as popular ways of approaching music video is by making a performance clip. Performance contains
mostly filmed performance of the band or singer in some usual setting like stage or unusual settings like
forest, also possibly conceptual one that is created and modeled by artists specifically for the music video.
Most of these videos are low budget and require just an ability to film actors, although sometimes it
requires big crew, cast and planning like in the clip “Upside Down & Inside Out” by OK Go. There are
several goals of performance music video that we can highlight. First one is based on telling story of a life
and death. If we look at “I want to break free" by Queen, we will notice how Freddie Mercury is
performing a character with a distinctive characteristic that tells a story during singing performance.
Carlsson stated that in this type of MV artist often “contemplates her or his own greatness and unfulfilled
wishes.” (1999) One of the animated music video with performance is “Clint Eastwood” by Gorillaz that
combines it with narrative. Although sometimes artists make it too abstract that seems like there is no
clear narrative and it was done purely for the entertainment. Second goal of performance music video is

15
very common one to monetize everything in the clip such as him or herself, lifestyle “to evoke the
charisma of stardom and sexuality, he or she wishes to embody dreams of celebrity, to be an icon, the
center of procreative wishes.” (Sven E Carlsson, 1999) Dana Summerfield suggests that ignoring common
narrative is “important in their role of advertising” (2013, slide 4). It helps to sort of give a glimpse on the
life of the popular singers without sharing the whole story. Almost every clip in the genre of pop music is
made in such way, for example look at Nickie Minaj, Justin Bieber. This way they only ‘amplify’ the
mysteriousness and incredibility of the artist in contrast to their ordinary life.
Performance clip can work well with a narrative, but Narrative music video type in contrast
usually does not contain any lip-syncing singing and might not even have the singer filmed. Most of the
animated music videos fall into this category. Even I do with my graduation music video. Narrative can be
very complex and might either follow the lyrics or completely disagree with it visually. But most common
way of approaching such music videos is to ‘amplify’ the lyrics to create an interesting story that can be
visualized in a creative way. This is where “the meaning and effects [of the song lyrics] are manipulated.”
(Sven E Carlsson, 1999) One of my favorite examples is “Let's Go” by Stuck in The Sound where you can
clearly see how visuals drive the story by amplifying and manipulating the lyrics of the song. I think it is a
good way of making sense out of the song that might not have it. It also makes song interesting by
visualizing it in a creative way within a story. Usually it is the most consuming way of producing music
videos not just because it takes a lot of time but it also hard to make it authentic.
Building a narrative or a story is the most complicated stage in my view. As I said MV contains
out of parts from cinematography, story-telling and more. In order to make your narrative in a MV more
interesting it is a good practice to adopt certain principals of story-telling. For example, Roland Barthes
described five codes that are useful in making a cohesive and interesting story: The Hermeneutic Code
(HER), The Enigma/Proairetic Code (ACT), The Symbolic Code (SYM), The Culture Code (REF), The
Semantic Code (SEM). Let me just describe several of them that I find useful in my practice. Hermeneutic
Code can be used to add mysteriousness to the narrative. In short it means that we can treat our music
video narrative as a detective novel where in order to understand the idea or solve a problem the reader
have to collect all the clues. The Symbolic Code is a combination of many Semantic Codes that suggest
additional meaning by way of connotation. Symbol is then understood if all the semantics are clear. Music
video “Re” by Nils Frahm visualize successfully selected symbols, that make the narrative more
interesting. Cultural Code is quite common in music videos that represents certain section of population:
religious people, couples, rockers, skateboarders, break-dancers and so on. If MV producer wants to make
all the teenagers interested in his clip, he has to visualize certain cultural codes that they can relate to.
Carlsson suggests that “layers of perception may be instinctive, inter-subjective and individual, which in
turn activate social aspects such as family, peer group, region, country, language etc.” (1999) So the type
of targeted audience will influence the creative decisions for MV.
Animators can show simple story in a very complex abstract visual. Take a look at “Two Against
One” by Danger Mouse, Daniele Luppi where story of a twisted mental state of the narrator in the song is
spiced up with crazy visuals. Another one is “Du Monde Tout Autour” by Louise Attaque that also have a
very weird way of visual narration. The main reason for that is that most of the music videos are made to
entertain the viewer. “Abstract Montage is a combination of images or artwork that may have no
relationship to the music but are visually stimulating.” (H. Wolfgang Porter, 2010) Art/abstract music
video is another category/type that “contains no perceptible visual narrative and contains no lip-
synchronized singing.” (Sven E Carlsson, 1999) Take for example Norman McLaren’s work “percussion”

16
and make a music video out of it that will be pure art clip. Of course not every genre of music will work
over such visuals or at least not throughout the whole song. I think it’s good to use this approach in a very
selective way to entertain the audience and not to bore them. Abstract visuals might also be used as a
‘Pictorial elements’ that appear in the clip and together develop ‘visual leitmotif’. For example, if you
have pure rain footage and a singing performance of an actor in a room, by different combination and use
of the rain footage throughout the MV you develop a ‘visual leitmotif’ and assign a certain meaning to it.
Of course such things require a lot of time and experimentation with the medium but the rewards are
greater too.
In my graduation project I was struggling to make the story of my music video visually clear. I
was pretty sure on what emotions each part of the music need to convey and was able to successfully
place each scene of the story in the right place, but there was not enough of visual clues for the viewer to
understand the complex idea. So I decided to leave the most important parts of the story and think of
alternative ways of delivering the main idea. I asked for advice of my fellow students on what exactly was
not clear in my MV. I managed to restructure my story in a way that all of the parts were logically
connected and what most important deliver the main idea in a simple and clear way.

17
Visualizing story and environment through music
Now when we are familiar with different categories/types of music videos such as narrative,
performance and abstract we can go through main decisions artist make to come up with ideas, stories and
styles. We will look at how by analyzing the song or the music you will be able to stimulate your mind for
creative decisions. There is no one way of creating music video thus I encourage you to critically think
about what might suite your needs. The most straightforward way of coming up with ideas is to just listen
and let your mind interpreted the music and visualize. Music is a global language but it might raise certain
feelings or even events and imageries that you would be happy to use in your music video. So it might be
a nice idea at the beginning of the process to relax and enjoy the music and only after that, approach it
conceptually.
Sound as a story telling device
Most of the music follow structural patterns that we can deconstruct. There are several aspects that
make music sound the way it is: rhythm, melody and dynamics. In addition to that we need look for lyrics,
but let’s just focus on music for now. I tend to think of melody, harmony, phrase, theme, motif etc. as one
thing even though in music theory they are separate. The reason for that is that they all share the same
function in the process of analyzing the sound for music videos. Melody can vary, repeat, be reversed, it
can sound sad, happy, mysterious, random or sound uncompleted. Melody consists out of notes but they
only gain meaning in combination with rhythm. As Robin Beauchamp pointed out we can “derive
additional meaning from the rhythm and tempo of [sound] delivery.” (2013, p. 5) Rhythm can suggest
where to put a cut at the editing stage, and it can also suggest time & space. Notice how the rhythm of
different people walking can suggest their character, in the similar fashion rhythm in music can suggest
characteristics such as calm, rushed, normal etc.
Melody and rhythm together can define genre that can be used in defining mood, theme and style
for the MV. Carlsson suggests that “connections between the music genre and the visual genre of music
video are weak.” (1999) But still points out most often made decisions based on music genres. I think that
you can have a lot of visual variations on one genre but usually specific genre has his own personal
characteristics that makes it different. Those characteristics can be used to suggest what kind of visuals or
story feel right or wrong. Melody and rhythm can help in story development, because it contains cultural
codes that can suggest era and environment where the story happens. One of the examples is “Amaro and
Walden’s Joyride” by The Line studio that clearly has a very underground, hip-hop, club sound that is
reflected in the visuals, suggesting that culture.
Finally, dynamics makes melody sound more interesting even though melody might be repeated
multiple times in the song. Dynamics adds ‘saturation’ and ‘contrast’ to the melody and can have
correlation with the story developments. Usually song builds up slow and ends the same way, but it is
important also to look for the dynamics in the body of the song and identify which parts of the song sound
intense or relaxed. As we know from storytelling we must introduce the audience to the world, situations
and characters we created and then buildup the conflict and resolution. However, it’s not always the case
in music videos because music dynamics can have a different structure from traditional storytelling. It
might start very loud, expressive and then go into slow or build up. Certain intense moments in the music

18
can correlate with intense story moments visually in the MV. Dynamics of the sound can suggest in which
part of the song it would be nice to show realization or climax of the story or show conflict or defeat.
One of the useful approaches to find important cues for your visual storytelling is to deconstruct
the song using ‘song anatomy’. Most of the songs will have ‘intro’, ‘verse’ and ‘chorus’. These are the
building blocks for any pop song. In order to make it more clear I will use music video “Boss Wave” by
Xilent. The reason I chose this music video is that it has a very basic structure without ‘break’ or ‘solo’
parts of the song structure that would have made it complicated to understand. Such structure is very
prominent and very easy to notice in this EDM dubstep music. ‘Intro’ is strait forward in comprehension,
as it can, for example correlate to the introduction of the movie. Usually introduction is less loud and
might only have a melody in comparison to other parts. Visually you could put your wide shots or
establishing shots introducing the environment, sometimes characters and situations. In the “Boss Wave”
it is quite long but I think it is because it also blends with a verse (.0-1.19). ‘Verse’ would be the
‘leitmotif’ that is repeated in the beginning (.0 – .20) in the ‘verse’ sections at (.55-1:19), (2.15 – 2.29),
(3.12 – 3.30). In the music video these are the parts where certain things are introduced that lead into
action, for example at (.55-1:19) protagonist is in the robot suit preparing to do his duty or at the (2.15 –
2.29) where his fiancé is introduced that leads to unexpected plot twist. It can also be described as a
transitional moment. The action/fight scenes happen to be at the ‘chorus’ as it has more dynamic sound
(1.20 – 2.15) & (2.57 – 3.11). This part is usually fast in rhythm thus is used as a tension moment after
release/calm moment. As we can see structure of the song can be used for visual storytelling. By finding
patterns, dynamics and repetitions in the song it’s much easier to plan your story. It is very important to be
sensitive to the ‘story’ that musicians include in the song that is not necessarily based on lyrics, rather on
accumulative of the melody, style, mood, dynamics etc. Below you can see how I graph the song in order
to understand it better. I drew the line demonstrating the whole duration of the song and then divided it
based on the repeating patterns such as verse, chorus etc. Furthermore, I added notes on top and bottom to
make it clear for me what and where are these lyrics cues. As you can see I used red color pencil to show
the dynamics of the song. Some places on the graph have notes about the instruments that are useful for
‘isomorphism’ in animation.
Usually song structure is not enough to come up with the story or theme, thus ‘artist’s integrity’
and his/her story might help. If your goal is to represent artist with your visuals then it’s important to
make an informal interview asking what was the main motivation to write music, what mood she/he is
trying to portray. ‘Artistic integrity’ is “animated performance [that] must remain absolutely consistent,
exactly as its creator committed to creating it, throughout all viewings and screenings.” (Lisan Peters,
2014, p. 8) In my interview with Tinus Kardolus about “Space Sheep’s” music video by Oliver Heldens &
Chocolate Puma he mentioned that by “the way [artist] want to show themselves to the world it’s very
easy to say what you can or cannot do and how far you can go with your ideas without compromising their
own image.” So if you don’t have resources to conduct an interview or get a response on your ideas you

19
can make a research about artist representation such as how he performs on stage what is he talking about
in the social networks etc.
Determining a visual style and mood of the story
We already talked about ‘timbre’ in music that is definitive part of the genre and mood. If you take
popular band “Coldplay” you would notice that they use acoustic instrument such as piano, drums and
guitar that makes music expressive and sound natural and ambient synthesizer with ‘reverb’ to add
atmosphere and make it sound uplifting. Robin Beauchamp pointed out how “low-frequency sound effects
to rouse our "fight or flight" response.” (2013, p. 9-10) If you take a look at music by LORN or Infected
Mushroom they both use a very psytrance, glitch, retro, darkwave-like sounds, which definitely influence
the mood the style of the visual piece that in this case could be depressing, sad, dark, nostalgic etc. But
LORN is also slowing down the music but not using traditional EDM beat loop instead compensate it with
glitch sounds and other sound effects. Therefore, mood, style and time & space for MV can be determined
by analyzing melody and ‘timbre’ of the song.
Several sound characteristics to take in mind while analyzing the song:
-Is sound suggest narrative or ambience
-Does it sound realistic or fantasy-like
-What physical characteristics does it suggest/sounds like
-What are the dynamics of sound (panning, delay, echo, reverb etc.)
-What kind of style or design, color would suite this sound
It is important to build a visual guide, mood-board in the preproduction stage of music video.
“Since the music is the force driving the video, you'll need to decide what style of video you want to
make.” (H. Wolfgang Porter, 2010) Going back to three ways of promoting song; you can illustrate what
is in the song lyrics or you can amplify it and for example make a guy singing underwater being half fish
half human like in “iSpy” music video by KYLE to add comedic context and finally you could
‘Disjuncture’ by completely ignoring what is in the lyrics and make your own sense based on music and
your imagination. ‘Tone’, ‘melody’, ‘timbre’, ‘rhythm’ can already suggest a lot of things about the
mood/style/theme for music video.
In my graduation project I was certain about the mood of the song. In addition to that artist of the
song told me about how she came up with the idea for the melody of the song. She told me that she was
flying through soft clouds in the sky on the plane when the idea came to her mind. Sound quality in the
song also suggested to add a lot of softness to the visualization. She mentioned that in her album she
wanted to show the appreciation of life around us. That information was also helpful in story and character
development for MV. The song ‘Sun’ mostly reflects her realization that we should look at the bright side
of the life. The main character in the MV change from having a sad perception of the life to a happier one.
I guess in other words from pessimist to optimist. The visual look is influenced by the music rhythm,
instruments, melody. For example, in the intro of the song melody and voice has a very uplifting feel that
is reflected in the music video, where the cloud lifts the character up into the skies. It might be very
obvious to use clouds in the music video, but I was not able to think of anything else that would reflect the

20
song very well. Rhythm of the song as well as melody has a lot of repeating patterns that can be visualized
by different environment things/props passing the camera, just like when you go by car and you see trees
passing by in the window, but instead of trees I used clouds.

21
Visualizing the character & promoting character through sound
Sound and story can contribute to the development of characters for music video. Sound can
suggest age, size of the character, his feelings, inner characteristics such as evil or good. Sound can
stimulate character movements and make supper-hyperbolized or unrealistic animation look believable.
‘Leitmotif’ can add to the recognition of certain events or characters in the MV. Certain genres of the
music can contribute to the MV story therefore also to the character development. Following chapter is
about all these mentioned things and character animation in general.
Purpose of the character or characters
First of all, it is important to decide whether characters are really mandatory in the MV. Some MV
are successful even without characters because it is not based on narrative but on abstract visualization of
the song. Take a look at “The Road” by ALB where they connect visuals using rhythm and melody
repetitions therefore tell the story using only objects and environment. There is other type of MV where
lyrics are shown in a ‘motion graphics’ ‘typography’ way. For example, “Sad Machine” by Porter
Robinson uses Kinetic Typography. But of course it is possible to combine narrative with mentioned
approaches. Characters in the MV might also just play a role of crowd or additional entertainment element
and not necessarily have to be thoroughly developed. For example, take a look at the music video
“Alfonso Muskedunder” by TODD TERJE or “The Music Scene” by Blockhead or “Papercut” by Zedd ft.
Troye Sivan. Images and animation in that case is stimulated by music rhythm and sound dynamics (ex.
Pitch, loudness, effects). If you are doing narrative and you need character to drive your MV, then you
have to think about their purpose. Is your character the one who sings the lyrics like in “Freak of the
Week” by Freak Kitchen or the one who is similar to traditional story telling character that evolves, gets
into certain obstacles and not necessarily represents the singer. There are many examples of MV that
follow such approach: “Rocket Man” by Elton John, “Boys Latin” by Panda Bear, “The Veldt” by
deadmau5 feat. Chris James, “Let's Go” by Stuck in The Sound, “Paper Plane” by Giangrande and more.
Usually the purpose of the character then would be to represent the artist and his story in that song. Just
like in traditional story telling characters in the MV drive the story and when visualized correctly can
receive sympathy of the audience. You might also use several characters in your music video, and show
the interaction between them like in “Jenny” by Studio Killers. Usually genuine characters can add to the
believability of the story and success of the MV. One of the popular examples would be Gorillaz band
characters.
Believable character animation performance
Here I would like to accentuate the topic of character animation through acting and performance of
the characters and music & sound design. First of all, good animation is relative, so character animation
should not be seen as good or bad, but should be perceived and analyzed by subjective and objective
factors. Subjective in this case will be just a personal ‘involvement’ and sympathy with character.
Objective factors are based on questioning acting, context, pantomime, rhythm of the characters.
“Animation is both a technical and an artistic endeavor” (What Separates the Good Animator from
the Bad Animator?). Technical part requires familiarity with knowledge about physics and mechanics of
character animation such as 12 principals of animation by Walt Disney. Artistic is far more subjective and

22
require some sort of talent and sense of design. One of the good advice is to constantly and consciously
observe different actions. Making or finding reference videos as well as thumb nailing for the animation
can lead to believable character animation. I also find Dana Boadway Masson’s advice on “Layering
rhythm in the performance” very useful. Rhythm is vital part of character animation. Acting tips can also
enhance believability of characters. One of the main popular Stanislavski acting tip is to sympathize with
character by ‘getting in his shoes’. For example, if character is scared try not to just imagine that you are
scared but remember the moment in your life when you felt the same as this character and include it in
your acting. Another tip is always to question character’s thoughts and actions. Thought must drive
actions. In most of the cases non-dialog scenes speak more about the character’s feelings, but they are also
hard to make.
Bertolt Brecht’s approach in acting differs from Stanislavski’s in that he wanted to achieve
Verfremdungseffekt for example by "breaking the fourth wall". Who Killed Who? (1933) by Tex Avery
cartoon demonstrates Bert Brecht’s approach in acting that makes animation interesting and funny.
Character animation can also be spiced up by optional choices of life-or-death. (Song of the Sea Acting
Analysis, By Ed Hooks) This way we also try to think ourselves any possible solution to achieve a goal
while overcoming obstacles. Relationship between protagonist and antagonist as well as their world views
must be clear and not just obvious or cliché. Because for example evil character is not absolutely evil he is
good in his own world and bad only in relation to a protagonist. In general characters must contain their
integrity throughout the film. They could change their mind but that will not be so easy and obvious. And
most important thought drives action and not vice versa. In conclusion knowing how to apply animation
principals and not least important being creative in storytelling and character development can make
character animation performance believable.
Realism in animation
Character animator in my view requires and understanding of ongoing global discussion and
expansion of animation medium in general, because it can be implemented in terms of artistic expression
and technical innovation. By learning more about possibilities and the origins of animation, I get clearer
perspective on the limitations and possible expansion of the medium. It is important and relevant for
character animator because it shed light on the apparent confrontation between realism implemented in
Disney and subversion implemented by Dusan Vukotic in his cartoons and explains why both are
relevant and important. Due to digitalization, animation became accessible to more creators. And in order
to make your animation look interesting, engaging and
appropriate it’s not enough to just follow a cliché in character
Animation and Animation in general. Making Disney style
character animation is not enough to become popular.
Everything in the
scene should be
thought through
and should
support the main
idea.
Environment (colors, composition, set, camera etc.) sets the
tone, and character supports it and the story by the emotions
Dusan Vukotic

23
through acting. Most often the problem is that movements/actions do not follow thoughts, and are present
there just for the sake of action. Thoughts should drive movements of the character, and those thoughts are
built by the many factors such as environment other characters and story. A lot of artists trying to get
noticed by copying trending animation, when in my opinion trying you own way is more interesting and
rewarding.
Animator should critically approach every animation
and be opened to different types of animation styles. Realism
is a relative and subjective term. It can be based on
psychological truth of the performances or physical reality.
(drawing for animation, ch.2 p.40) For example Disney uses
12 principals of animation as a way to make animation more
believable and realistic. On the other hand, characters can
react in a realistic manner defined by the context and location.
For example, in “Spirited away” (2001) by Hayao Miyazaki,
main character dives into a new dimension/world that has its own mystical/unrealistic characters that act
and react in terms of their own world forces and rules. But even when we never been to that world we
instantly ‘believe it’, empathize with the characters and enjoy it. Other good example that I personally
enjoy watching is cartoon “Surogat” by Dusan Vukotic
where he breaks many rules of time & space, forces and
common sense. Going back to the versatility of
animation styles, Dusan Vukotic is very good at timing
character movements based on music rhythm and
emotion. This approach exaggerate comedy and makes
it interesting to watch. Glen Kean disagrees with the
argument that animation purpose is only for subversion
and impossible tasks for live action. (understanding
animation, p.8) “Beowulf”(2007) by Robert Zemeckis style of ‘realistic’ animation, in my opinion sets up
a very strong artificial realism that is supported in every piece of the story except maybe the script itself.
Environment, colors, lighting, modeling, motion capture animation and other aspects support each other
and makes it believable. Definitely throughout the story we understand that it is not real, but in general it
looks and feels believable. Can we say that motion capture combined with CGI realism will produce a
good character acting? Perhaps it all comes down to psychological truth that is built on artificial reality
that makes character actions look real and not dissonant. So character animation alone cannot be judged in
terms of ‘realism’ or believability, unless there is a context.
Disney’s reality is full of exaggeration and anthropomorphize, but that does not mean that
everyone else should do the same. Hayao Miyazaki for example adopts his own alternative naturalistic
style, even though he enjoys Disney character animation.
Robert Zemeckis
Hayao Miyazaki

24
Multizvuk ‘Sound Design’ in animation
I find it interesting to experiment with sound and
animation similar to what Norman Mclaren did with
Pen Point Percussion or Evgeny Sholpo’s
‘Variophone’ ‘Drawn Sound’. The only difference is
that I try to connect sound design with character
animation, to exaggerate, and set certain mood. Further
I would like to discuss some ways of exaggerating
physicality or add believability to motion of characters.
In 20th century Russian animators were creating
beautiful cartoons where they implemented music in storytelling. For example, cartoon ‘snow trails’
(Снежные Дорожки) by Борис Дёжкин (Boris Dezkin) in 1963 incorporated sound design and music as
a tool to support main characteristics of the story: emotions, purpose and rhythm of the characters in
relation to other characters, but also the context. There are
two groups of characters that are not only contraposed by
their design (colors, clothes etc.) but also by the music. In
the cartoon Music sets up the mood while sounds add to
the physicality/physical properties of the objects and
characters. In the cartoon ‘snow trail’ there are certain
moments where character almost dance to the music like
he is on stage. Characters might break laws, move in an
unusual way, but sound and music will be able to support,
exaggerate it and help to fool viewers to believe it.
Similarly, Disney used this ‘isomorphism’ approach in
their earlier animation. ‘Isomorphism’ can be observed in
a short surrealist cartoon “Le Merle” (The Blackbird) by Norman McLaren (1958). It is wonderful how
surreal character comes alive and dance or deform according to the sound. Some people call it mickey-
mousing for example “hearing xylophone ‘plings’ when someone raises his eyebrow etc. feels very
cartoony however it can be great in slapstick moments and when done well even in dramatic situations.”
(Robin Hoffmann, 2011) Melody can guide the viewer perception. If you take an ascending melody and
make a visual of a bouncing or rolling ball it will feel like the ball is going up. In addition to that if the
character is falling his actions might be distorted by music rhythm and instruments. Character can jump up
and down, do crazy things but sound if it is done correct will justify it. So sound design when done
correctly can help tell story and deliver performance of the characters successfully.
Recently in Russia I noticed the decrees of cartoons with the same idea of experimenting with
medium in terms of sound and visuals. More and more we see talking characters that do not ‘shut up’. We
get used to it and take it for granted. I think there is something unusual and wonderful in old cartoons.
Sound and visual comes together and makes the viewer think about the story and ‘build up’ the mood and
McLaren
Snow Trails

25
characters feelings. In my view it engages the
viewer much more then recent cartoons like
‘Волки и овцы: бе-е-е-зумное
превращение’ (‘Sheep and Wolves: crazy
transformations’) by Russian studio Wizart.
(photo on the right)
If you think of the orchestra each
musical instrument plays certain role in the
composition. Violin can be secondary and add
elegance or raise certain emotions. Bass can
be the base/fundament and support the melody, almost like ‘blocking stage’ in animation. I cannot leave
without proper attention drums. Rhythm in animation as I said earlier is very important, because they
provide rhythm and cohesiveness. Music/score can change just like a story in a movie, to evoke new
feeling or change mood. Depending on what role the character plays, certain music instruments might be
used. Let’s take the easiest example, good and evil characters. The seriousness of the evil character can be
added using the lower pitch, and unpleasant sounds of the other instruments like violin or tuba even piano.
Rhythm can also change and bring contrast. For example, character runs and then suddenly stops. Rhythm
of the music/score will also slow down. I think that we have more attention to visuals and idea of the story
when sound supports it. Music and sound design can make character performance much more interesting
and unique.
In the “Peter and the Wolf” that was musically accompanied by Serge Prokofiev certain ‘timbre’
of the instruments are able to imply size relationships:
“A little boy walking alone through a huge city is just a scene that might be very suitable for this
use of film music. In this case the music can enlarge the visual differences between these things by
giving the boy a little innocent flute motif on top of a very low, boiling music which might
symbolize the huge city. In this way, things that can’t be seen like that on the screen can be set into
relationship. If this little innocent boy from above might actually be a really brave, heroic person
who has done or is about to do big things the music might set him into another “size” dimension to
the city even though the images don’t reveal that yet.” (Robin Hoffmann, 2011)
It is possible to subvert the perception of the physicality of the object by sound. Although sound is
not able to influence visuals, in other words if we see cow and we hear dog we will not perceive it as dog,
but we might perceive a cow huge or made out of metal if we accompany its movement with low pitch
and metallic sounds. Therefore, we can change the substance and the perception of the object in addition
to visuals.
Sound effects like pan, reverb and echo can add to the believability of motion. If you have for
example a bouncing ball moves off the screen you can extend the camera field by continuation of the
sound and use of pan effect. “Tom and Jerry” series is full of unrealistic character animation, but it looks
believable because of the ‘music score’ that adds mood and describe characters’ motion and feelings as
well as ‘sound effects’ that add another layer to the physicality of the objects. Jerry or Tom can fly
through the screen and crash into the wall with a sound of flying plane and crashing car.
Wizart

26
"If this speculation holds, it would make perfect sense for smaller visual brains to make more use
of the additional information provided by the ears…in the real world, sources of light and sound
are often identical, and combining them will be advantageous. Imagine you take a twilight walk in
a forest and scare up some animal in the undergrowth. The best strategy for finding out whether
you are dealing with a hedgehog or a bear will involve combining visual information, like moving
twigs and branches, with auditory information, like cracking wood."(Charles Q. Cho, 2012)

27
Visualizing camera movements & editing of the video
Post-production stage is crucial in finalizing the look and the feel of the animation. This stage
requires fitting the rhythm of the animation to the music and adjusting the footage to fit the mood of the
song. Tinus Kardolus in the interview said that the most important advice he could give is to count music.
It means that we must look for accents and rhythm in song. “Cuts are important in a music video, they
must fit in with the rhythm and beat of the song. This attracts the audience as it makes the music video
more entertaining.” (Dana Summerfield, 2013, slide 6) Audience will most likely remember the song that
has music video visuals resembling rhythm and melody. If the cuts are motivated by either music or
actions/animation than they will be invisible and the audience will not be distracted. Dynamics of the
sound is another cue editor can use in his advantage. For example, if the song has some instruments silent
and other play a bit louder solo, then it can be used for a specific story moment. Take a look at "The
Ancestor" by Darlingside right after ‘chorus’ (1:52) we only hear nondescript rhythm with no guitar just
voices that makes it more intimate and special, almost suggesting what main character feels at that
moment. Marsha Onderstijn in her animation “The Life of Death” at (1:57) wanted the main character’s
‘leitmotif’ to be this ‘ticking clock’ melody. This melody also suggest time passing and main character
making a very important decision either take life or wait.
In other chapter I mentioned how ‘song structure’ can be useful for storytelling. Usually you
would have cuts on ‘verse’ and ‘chorus’. Last one would have a faster or uplifting melody and rhythm so
it must be supported by camera placement, movement and cuts. In “Express” by Hidetake Takayama you
can notice that ‘chorus’ is at the beginning and suggest uplifting part there. Second ‘verse’ at (2:38) is
setting anticipation by colors, lighting, and editing so that ‘chorus’ that follows will look more bright and
powerful. In my MV project I had to adjust camera position, cuts and story many times. Doing it at the
beginning, at the animatic stage is less risky and much faster. It is possible to visualize the whole story
with just a few thumb nails and 3d primitives that represent the movement of the characters and tells the
story. Only when previsualization or animatic is complete and the story is clear production stage is
initiated. Finally, it is also much easier to get feedback from people when you have at least some
visualization done. In addition to that script is not enough to be confident about the success of the
animation.

28
Conclusion
Here I would like to summarize the above and give some advices on working with sound. If you
are working with sound designer, it is always hard to communicate your ideas especially if you do not
know anything about sound design. This is what Marsha Onderstijn suggested on how to approach sound
designer or musician:
-I really like sending [sound or music] examples it really helps to communicate; this is
what I am looking for; this is what I want to hear. And whenever he sends me [recorded]
examples, I would point the things I did like; I like this chords or I don’t like the electric
guitar or those sound are too high or it’s a little bit too creepy; pointing out specifics of
what he made. So first ask him send me those initial ideas/sketches that he made. I do think
that it helps if you give [sound designer] some direction of where to look otherwise he can
go everywhere. But I do think that you must give some freedom not exactly this sound, this
melody or this chords. What I just did; I [said that I] want it a little bit curious, sad; give
directions like that.
Tinus Kardolus also added that it is important to give freedom to sound designer, but give him directions
or limits to start creative process. He said that some clients ask him either ‘acoustic’ or ‘digital’ sound that
helps him limiting his research and inspire creative process. I found it interesting how he defined freedom
in animation/sound design as creativity within boundaries. He said that the hardest part of any sound
design process is to start from scratch.
-It helps if you make sound mood boards, think about what you like, and why you like it.
Find some examples. It’s important to be critical [about sound]. Don’t through sound
designer a lot but also not too little. Set some borders to give something to hold on to.
When I asked Tinus Kardolus about tips for animators who work on music video this is what he told:
-Learn to count music. It’s very helpful to feel the beat [and tempo] of the song. It helps
when it comes to editing; to make the video go with the flow of the music. I think in the
medium of animation it is more well suited to go out of the ordinary. It is good to
exaggerate truth or see the lyrics in another way.
As you can see there are many ways to apply sound design and music theory in your animation.
You can look at the structure of the song and deduct the theme or the mood for your story. Furthermore,
you can use song structure to structure your story, and align it to the repeating patterns such as verse and
chorus. You can use the theory of sound to your advantage when you need to communicate your idea to
the musician or when you trying to create animation for music video or any other type of animation where
the sound is involved. Audience have more attention to visuals and idea of the story when sound supports
it. Animators can use symbolical symbiosis of musical/compositional/structural components and
narrative/textual/temporal. (Philip Brophy, 1994) Music can reflect the mood and suggest symbolism. One
of the examples is the composition “the rites of spring” by Igor Stravinsky that was visualized in Disney
animation Fantasia symbolizing the creation and cycle of life on the earth. In the cartoon Music sets up the
mood while sounds can add to the physicality/physical properties of the objects and characters as in many
examples of early Disney classics. For example, Silly Symphonies “The Skeleton Dance” (1929) “features

29
an array of skeletons coming to life to perform music upon themselves, where their bodies are the very
instruments they play in order to make their bodies move” (Philip Brophy, 1994).
To conclude sound can help to tell the story, it can create the background, and it can add emotion
to something you see. You can either create sound for animation or animation for sound. First method is
very common one, where sound plays a role of supplement to the main story line. The variations of using
sound can be compared to the variations of using words in literature; it can hyperbolize/exaggerate,
contradict, supply to the meaning or story and much more. Sound can add so much to the visuals that it
requires careful exploration from the creator of the work. In my graduation project I was mostly following
the second method. Sound and lyrics of the song in my case were driving the visuals the story. In this
paper we looked at many different examples of marrying sound and visuals from the basic isomorphism
by fitting the diegetic or non-diegetic sounds with visuals wither by rhythm or melody. We looked at how
sound is constructed and how we can find out many helpful things for the development of the project
visuals and story from analyzing it.
Luckily there is more to study in this field. I plan to continue my research of sound and animation
by doing more interviews with sound designers and animators. I find that every studio, animator or sound
designer have their own approach when it comes to marring sound and visuals. It suggests that analyzing
more work can make it easier to build different workflows and compare them. Majority of the informers
do not rely or at least do not think about whether they rely or not on a certain workflow. Most of the
important steps such as analyzing the sound doing research are getting lost in production. So whenever I
have conducted an interview mentioning this in my questions, informants would say “oh, I haven’t
thought about it”. Unfortunately, I notice how majority of the music videos that are popular, and most of
them are EDM, have a very trivial and sometimes boring visualization. In my view commercialism/selling
and artistry/creativity should go hand in hand but that’s a topic for another discussion.

30
References
1. Edwin Carels, Animation = Multiplication of artforms, LUX, London in collaboration with Arts
Council 2006
2. Paul Wells, Drawing for animation, chapter 2, Types of Performance, 2008
3. Paul Wells, Understanding animation, p. 10-28, London: Routldge 1998
4. Margaret F. Callcott and Wei-Na Lee, A Content Analysis of Animation and Animated Spokes-
Characters in Television Commercials, Taylor & Francis, Ltd. 1994
5. Victor Navone, What Separates the Good Animator from the Bad Animator?,
(http://www.animationtipsandtricks.com/2009/08/what-separates-good-animator-from-bad.html)
6. Dana Boadway, Masson 10 Advanced Acting Performance Tips for Animators,
(http://blog.animationmentor.com/10-advanced-acting-performance-tips-for-animators/)
7. Ed Hooks, Oscar Ballot Guide: ‘Big Hero 6’ Acting and Performance Analysis,
(http://www.cartoonbrew.com/ideas-commentary/oscar-ballot-guide-big-hero-6-acting-and-
performance-analysis-109226.html)
8. Ed Hooks, Oscar Ballot Guide: ‘Song of the Sea’ Acting and Performance Analysis,
(http://www.cartoonbrew.com/ideas-commentary/oscar-ballot-guide-song-of-the-sea-acting-and-
performance-analysis-109124.html)
9. H. Wolfgang Porter, Making Music Videos, 2010
(https://www.videomaker.com/article/f5/14214-making-music-videos)
10. Philip Brophy, The Animation of Sound, published in The Illusion of Life: 1st International
Conference on Animation, Power Publications, Sydney, 1994
(http://www.philipbrophy.com/projects/sncnm/AnimationSound.html)
11. Charles Q. Choi, Why Some People See Sound, 2012
(https://www.livescience.com/24191-why-some-people-see-sound.html)

31
12. Frank Thomas and Ollie Johnston, Chapter 11 The Disney Sounds, p.285 The Illusion of Life by
Disney Animation
13. Kate Finan, The History Of Animation Sound, 2015
(https://www.milleranimation.com/blog-posts/2015/11/25/history-animation-sound)
14. Lisan Peters, Sound in animation, 2013-2014, Master of Animation AKV. St. Joost
15. Robin Beauchamp, Designing Sound for Animation second edition, 2013
16. Robin Hoffmann, What is the function of film music?, 2011
(http://www.robin-hoffmann.com/tutorials/guide-to-working-with-a-film-composer/12-what-is-the-
function-of-film-music/)
17. Sven E Carlsson, Audiovisual poetry or Commercial Salad of Images?, 1999
(http://filmsound.org/what_is_music_video/)
18. Dana Summerfield, music video theories, 2013
(https://www.slideshare.net/DanaSummerfield/music-video-theorists-26982606)
19. International School Bangkok, musical Analysis Writing Guide, 2012
(https://isbangkok.haikulearning.com/thomasm/ibmusicsl2/cms_file/show/39843637.pdf)

Animation and Sound: Visualizing Character Performance

Recommended

Recommended

More Related Content

Similar to Animation and Sound: Visualizing Character Performance

Similar to Animation and Sound: Visualizing Character Performance (20)

More from Jennifer Roman

More from Jennifer Roman (20)

Recently uploaded

Recently uploaded (20)

Animation and Sound: Visualizing Character Performance