sound/tracks:Real-Time Synaesthetic Sonification and Visualisation of Passing Landscapes

sound/tracks: Real-Time Synaesthetic Sonification and Visualisation of Passing Landscapes 聲音，軌跡火車上的即時的視覺與聽覺的通感翻譯系統 NTUST 指導老師 - 梁容輝設計研究所資設組鄭智云

PAPER FROM http://mmart.iat.sfu.ca/exhibition.html

ABSTRACT

When travelling on a train, many people enjoy

looking out of the window at the landscape passing by.

The continuously changing view outside the window

is captured with a camera and translated intoMIDI events

that are replayed instantaneously.

This allows for a reflection of the visual impression, adding

a sound dimension to the visual experience and deepening

the state of contemplation.

The application is intended to

be run on both mobile phones (with built-in camera) and

on laptops (with a connected Web-cam).

1. MOTIVATION AND CONTEXT

我們常常在坐火車時，看著窗外，看風景略過。另外常做的的事情是聽手機裡的 mp3 。

然而，這兩個活動印象中常常同時的上演，不相應對方，視覺和聲音保持獨立的實體，沒有邏輯和美學的連接。

該項目的目標是在這裡建立一個結合視覺和聽覺藝術的應用

建立一個一致的旅遊體驗 - 一種共通經驗 , 在視覺和聽覺印象互相對應

2. GENERAL SCENARIO AND USER INTERFACE

our goal is to give the passing landscape an active role, to make it the central player that controls the real-time production of music. In a sense, the passing landscape with its fleeting visual impressions acts as the musical score which is going to be interpreted based on outside conditions such as weather, season, lighting, the speed of the train, and the quality of the camera. Every journey is different and will yield a unique composition

3. TRANSFORMING VIDEO TO SOUND Figure 2: Four example screenshots taken from the mobile version of the software running on a Nokia 6120. The right half of the screen displays the current video taken from the camera. The left half is the condensed history – the ‘score image’.

4. TRANSFORMATION APPROACHES (IFFT)The filter is realized by applying an inverse Fast Fourier Transformation

4.1 Method 1: Filter Approach Figure 4: Filter approach. Top: Short excerpt of a journey as taken by the camera (all middle columns of the processed frames). Bottom: Spectrogram of a generated sound.

4.2 Method 2: Piano Roll Approach Figure 5: Tone-to-colour mapping of Skrjabin’s Clavier `a lumi`eres.

4.3 Method 3: Colour-based Approaches Figure 6: The basic colour mapping used in the per- ceptually motivated approach.

5. EVALUATION OF THE SONIFICATION ALGORITHMS 1. 過濾器方法不使人信服與喜愛因為產生的聲音像是噪音以及吹口哨聲，此外計算每個 ifft 圖片與圖片的過渡耗費太久時間 2.piano roll 沒有經過精密計算所以導致複雜的風景呈現上聽起來很簡單 3.color-based 法 : 目前最好的結果穩定率與基本節奏方面產生的樂曲和諧度為最高

Six short excerpts from visual ‘score images’.

8. APPLICATION AND PRESENTATION SCENARIOS

8.1 Private Enjoyment

8.2 Public Installations and Performances

9. DISCUSSION 1. 未來將近一步研究其他圖像的色調翻譯方法 2. 計畫軟件開發提供一種新的有趣的小工具使人們的旅途中更加的愉快

BONUS 靈感來自於 Chemical Brothers - Star Guitar Chemical Brothers http://www.thechemicalbrothers.com/home

BONUS 化學兄弟 / 有志一同：新歌＋精選 The Chemical Brothers / Brotherhood 【 Limited 2CD 】 Chemical Brothers: Let Forever Be http://www.youtube.com/watch?v=Hmpxsk3dHaA&feature=related