Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
A Picking Interface Seamlessly Connecting Passive and Active Performance on Guitar
- 1. A Picking Interface Seamlessly Connecting Passive and Active Performance on Guitar IWAIT2020 Nozomu Yoshida, Katsutsugu Matsuyama Faculty of Science and Engineering, Iwate University, 4-3-5, Ueda, Morioka-shi 020-8551, Japan
- 2. Guitar Performance Sound / Visual Effects Seamlessly Connects Passive and Active Performance
- 3. Guitar Performance Automatically adapts Sound / Visual Effects Passive performance
- 4. Guitar Performance Manipulates effects Active performance Sound / Visual Effects
- 5. Guitar Performance Reflects the performer's intentions visually and musically Sound / Visual Effects
- 6. RELATED WORKS (PATTERN RECOGNITION) F0 sequence pattern recognition [2] Guitar recording -> Melody extraction -> Estimate fundamental frequency (F0) -> Analyze and estimate techniques [1] L.Reboursière et al., "Left and right-hand guitar playing techniques detection, " NIME 2012. [2] Y. Chen et al., "Electric Guitar Playing Technique Detection in Real-World Recording Based on F0 Sequence Pattern Recognition," ISMIR 2015. Left & right-hand techniques detection [1] Detection and classification of (1) Attacks/note onsets (2) Left-hand and right-hand discrimination (3) Guitar techniques: e.g. normal, mute, bend, slide, hammer-on, pull-off, harmonic, palm muting
- 9. Pressure sensor Acceleration sensor Appearance
- 13. Weak = 1 if Strong nor Jitter
- 19. Stationary = 1 if Narrow nor Wide
- 22. 'Jitter' and 'Wide’ -> 'Chord stroke’ 'Jitter' and 'Narrow’ -> 'Single note’ Otherwise -> 'Non-picking'
- 23. Rest -> Single note -> Chord stroke -> Rest (Short interval) 60 FPS
- 24. (Short interval) Rest -> Single note -> Chord stroke -> Rest 60 FPS
- 25. Rest -> Single note -> Chord stroke -> Rest 60 FPS (Short interval)
- 26. Rest -> Single note -> Chord stroke -> Rest (Short interval) 60 FPS
- 27. Rest -> Single note -> Chord stroke -> Rest (Short interval) 60 FPS
- 28. Rest -> Single note -> Chord stroke -> Rest (Short interval) 60 FPS
- 29. APPLICATION
- 32. CONCLUSION • Proposed a method to discriminate ways of picking by sensing pressure and rotation angle in a player’s stroke movement. • As a result of the experiment, 3 picking states can be correctly distinguished through our interface. • Introduced an application example which manipulates visual and sound effects.
- 33. A Picking Interface Seamlessly Connecting Passive and Active Performance on Guitar Nozomu Yoshida, Katsutsugu Matsuyama Faculty of Science and Engineering, Iwate University, 4-3-5, Ueda, Morioka-shi 020-8551, Japan IWAIT2020
- 34. Pressure Change Rotation angle Change
Editor's Notes
- Hello everyone, I’m Nozomu Yoshida from Iwate University in Japan. Now I have a presentation about A Picking Interface Seamlessly Connecting Passive and Active Performance on Guitar.
- We have designed an interface that allows a guitar player to manipulate another channel, such as the manipulation of sound and visual effects, while playing the guitar. Our interface connects passive and active performance seamlessly.
- Passive performance refers to a state in which the performer normally plays the guitar and the interface automatically applies sound and visual effects. For example, when playing the guitar, the picking motion itself is sensed and the parameters of the sound effects are automatically adjusted.
- On the other hand, active performance refers to a state in which the player actively manipulates the effects through the interface when NOT playing the guitar. For example, when the performer is not playing the guitar, visual effects can be switched by some active action, like pressing the pick or by shaking the player's wrist.
- Our goal is to create an interface that immediately reflects the performer's intentions visually and musically without turning away from the guitar playing motion.
- There have been many studies in pattern recognition on the guitar. For instance, one study proposed a method that detects guitar techniques by analyzing note, pitch, and attack of guitar sound. Another study investigated a recognition method of guitar techniques based on F0 sequence. However, these sensing techniques cannot be used directly for the purpose of this study, picking state identification.
- In this paper, we propose a method to recognize guitar picking state by using pressure and acceleration sensors. In addition, we present an audio-visual application. This application enables performers to extend their expression by converting their picking hand motion into visual and sound effects.
- For our goal, recognizing whether picking occurs or not is needed firstly to clarify whether ongoing performance is passive or active. We think it is also useful to recognize picking styles (such as Chord stroke, Single note) to give effects variation to active performance. So, we judge the current state as ‘Non-picking’, ‘Chord stroke’ or ‘Single note’.
- Our interface consists of a pressure sensor and a 3-axis acceleration sensor. The pressure sensor is fixed onto a guitar pick. It senses pressure force between the thumb and index fingers of the player’s picking hand. The acceleration sensor is fixed on the first knuckle joint of the index finger. It measures how much the picking hand has rotated.
- Table II shows the classification of pressure patterns. We separated pressure force into three categories: ‘Strong’, ‘Weak’ or ‘Jitter’.
- Firstly, we judge the state as ‘Strong’ when pressing the pick strongly.
- Secondly, the state is set to ‘Jitter’ if the pressure rapidly changes.
- Finally, a state that does not satisfy the both of ‘Strong’ and ‘Jitter’ is determined as ‘Weak’, which means a non-picking state in which the player is just pinching the pick.
- This is a movie of 'Strong' recognition. When the pressure value is higher than the threshold, 'Strong' flag turns on
- This is a movie of ‘Jitter' recognition. Specifically, it indicates a state that the difference between the original pressure value and the pressure value after applying the low pass filter exceeds a certain threshold.
- Table III shows the rotation angle pattern classification. We determined the rotation states to be ‘Narrow’, ‘Wide’, or ‘Stationary’.
- Firstly, we judge the state as ‘Wide’ if the rotation angle fluctuates greatly.
- Secondly, the state is set to ‘Narrow’ if the rotation angle vibrates quickly.
- Finally, a state that does not satisfy the conditions of ‘Wide’ and ‘Narrow’ is determined as ‘Stationary’, it means a non-picking state, in which the player is not moving their wrist in a picking motion.
- This is a movie of ‘Wide' recognition. Concretely, we calculate the Root Mean Square (RMS) of the difference between the original rotation value and the rotation value after applying the low pass filter, and then judge the state as ‘Wide’ if the RMS exceeds a certain threshold. RMS was adopted to measure the continuous intensity of wrist rotation during the chord stroke.
- This is a movie of ‘Narrow' recognition. Specifically, it indicates a state that the difference between the original rotation value and the rotation value after applying the low pass filter exceeds a certain threshold. All of the threshold values depends on the individual picking style.
- By combining pressure and rotation angle, we judge the ongoing picking state. Firstly, if the picking state is 'Jitter' and 'Wide', we determine it as 'Chord stroke’. Secondly, if the picking state is 'Jitter' and 'Narrow', we determine it as 'Single note’. Otherwise, we define the picking state as 'Non-picking’.
- We informally tested our prototype interface. One of the authors has been playing the guitar for 5 years. He sequentially played the guitar with our interface in the order of rest, single note, short interval, chord stroke, and rest for about 12 seconds. In this graph, each column shows the value of the pressure sensor and acceleration sensor respectively. The horizontal axis indicates the time at 60 frame per seconds.
- This graph shows the result. Each column shows non-picking flag, single note flag, and chord stroke flag respectively. The value takes 1 or 0 on each flag. As a result, basically the three picking states are classified correctly.
- This graph shows the comparison between sensing data and the flag.
- Comparing the start point of single note picking around the 50th frame shows that the latency is very low.
- In the vicinity of the 320th frame, the 'Non-picking flag' rises for a moment and then falls immediately. This is because there was an interval when moving from single note playing to chord playing in actual performance.
- On the other hand, latency of about 30 to 40 frames occurred in the recognition of the chord playing. This is presumed to be due to the use of the RMS calculation for ‘Chord stroke’ recognition.
- 5. APPLICATION EXAMPLE At last, we present an audio-visual application example. This application enables performers to augment their visual and sound expression through their picking hand motion.
- 5.1 Visual Expression As an active performance, the performer can manipulate CG effects by associating the pressure with the expansion rate, and the rotation angle with the colorfulness of the sphere. Also in the picking state, as a passive performance, the interface automatically applies CG effects. For example, during the chord stroke, the RMS value affects the rotation speed of the sphere, therefore the intensity of performance and rotation of the sphere are synchronized.
- 5.1 Sound Expression The performer can also manipulate sound effect. In the non-picking state, as an active performance, the performer can manipulate sound effects by associating the pressure with the dynamics effect, and small wrist shaking with the delay effect. Also in the picking state, as a passive performance, the interface automatically applies sound effects.
- 6. CONCLUSION In this research, we proposed a method to discriminate ways of picking by sensing pressure and rotation angle in a player’s stroke movement. As a result of the experiment, 3 picking states can be correctly distinguished through our interface. We also introduced an application example which manipulates visual and sound effects. We will improve the latency in the future.
- Thank you for listening.
- 5. APPLICATION EXAMPLE We present an audio-visual application example. This application enables performers to augment their visual and sound expression through their picking hand motion. 5.1 Visual Expression In the non-picking state, as an active performance, the performer can manipulate CG effects by associating the pressure with the expansion rate, and the rotation angle with the colorfulness of the sphere. Also in the picking state, as a passive performance, the interface automatically applies CG effects. For example, during the chord stroke, the RMS value affects the rotation speed of the sphere, therefore the intensity of performance and rotation of the sphere are synchronized.
- 5.2 Sound Expression The performer can also manipulate sound effect parameters by picking. In the non-picking state, as an active performance, the performer can manipulate sound effects by associating the pressure with the dynamics effect, and small wrist shaking with the delay effect. Also in the picking state, as a passive performance, the interface automatically applies sound effects. For example, during the chord stroke, the RMS value affects the degree of distortion, therefore the intensity of performance and the intensity of the guitar sound are synchronized.