1. The document describes a system that sonifies landscapes in real-time as they pass by a train window. A camera captures the changing visuals which are translated into MIDI events and sounds. 2. Three approaches to the visual to auditory transformation are discussed: a filter approach, a piano roll approach, and color-based approaches. Evaluation found the color-based method produced the most harmonious and rhythmic music. 3. The goal is to create an integrated sensory experience that connects the visual and auditory impressions of traveling landscapes out a train window. The system is intended for use on mobile phones and laptops.

1. sound/tracks: Real-Time Synaesthetic Sonification and Visualisation of Passing Landscapes 聲音，軌跡 火車上的即時的視覺與聽覺的通感翻譯系統 NTUST 指導老師 - 梁容輝 設計研究所資設組 鄭智云

2. PAPER FROM http://mmart.iat.sfu.ca/exhibition.html

7. 3. TRANSFORMING VIDEO TO SOUND Figure 2: Four example screenshots taken from the mobile version of the software running on a Nokia 6120. The right half of the screen displays the current video taken from the camera. The left half is the condensed history – the ‘score image’.

8. 4. TRANSFORMATION APPROACHES (IFFT)The filter is realized by applying an inverse Fast Fourier Transformation

9. 4.1 Method 1: Filter Approach Figure 4: Filter approach. Top: Short excerpt of a journey as taken by the camera (all middle columns of the processed frames). Bottom: Spectrogram of a generated sound.

10. 4.2 Method 2: Piano Roll Approach Figure 5: Tone-to-colour mapping of Skrjabin’s Clavier `a lumi`eres.

11. 4.3 Method 3: Colour-based Approaches Figure 6: The basic colour mapping used in the per- ceptually motivated approach.

12. 5. EVALUATION OF THE SONIFICATION ALGORITHMS 1. 過濾器方法不使人信服與喜愛 因為產生的聲音像是噪音以及吹口哨聲， 此外計算每個 ifft 圖片與圖片的過渡耗費太久時間 2.piano roll 沒有經過精密計算 所以導致複雜的風景呈現上聽起來很簡單 3.color-based 法 : 目前最好的結果 穩定率與基本節奏方面產生的樂曲和諧度為最高

13. Six short excerpts from visual ‘score images’.

15. 9. DISCUSSION 1. 未來將近一步研究其他圖像的色調翻譯方法 2. 計畫軟件開發 提供一種新的有趣的小工具使人們的旅途中更加的愉快

16. BONUS 靈感來自於 Chemical Brothers - Star Guitar Chemical Brothers http://www.thechemicalbrothers.com/home

17. BONUS 化學兄弟 / 有志一同：新歌＋精選 The Chemical Brothers / Brotherhood 【 Limited 2CD 】 Chemical Brothers: Let Forever Be http://www.youtube.com/watch?v=Hmpxsk3dHaA&feature=related