CoSECiVi'14 - E-Cecilia. Implementation of a music game
•
1 like•295 views
Trabajo Regular presentado en CoSECiVi'14. Resumen: This paper describes a serious game intended to teach singing to children. The system shows the students a virtual world they can explore, and the evolution of the world is based on their performance. Automatic content generation techniques are used to present the player with different exercises, and their singing is evaluated using state of the art techniques.
Recommended
Recommended
More Related Content
What's hot
What's hot (17)
Viewers also liked
Similar to CoSECiVi'14 - E-Cecilia. Implementation of a music game
Similar to CoSECiVi'14 - E-Cecilia. Implementation of a music game (20)
More from Sociedad Española para las Ciencias del Videojuego
More from Sociedad Española para las Ciencias del Videojuego (20)
Recently uploaded
Recently uploaded (20)
CoSECiVi'14 - E-Cecilia. Implementation of a music game
- 1. D’ IMATGE I GRÀFICS! TECNOLOGIE S! D’ IMATGE I GRÀFICS! TECNOLOGIES! http://gilab.udg.edu E-cecilia: implementation of a music game R. García1,5, I. Barbancho2, L. Tardón2, J. Arambarri3, M. Magdics1,4, M. Sbert1 1University of Girona 2University of Malaga 3Virtualware 4Budapest University of Technology and Economics 5iMinds / University of Hasselt
- 2. D’ IMATGE I GRÀFICS! TECNOLOGIE S! • Introduction • Game description • Architecture, client-server model • Server • Music composition & analysis • Client • Graphics, non-photorealistic rendering • Natural interaction • Immersive devices Contents
- 3. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Introduc-on • Learning to sing requires long practising hours • The early stages use very simple songs → boring • Good singers start very young, so they tire more easily • Solution: Make the practising into a game • Related work: • SingStar: full songs, too complex for starting young children • Other games: • focused on keyboard / mouse interaction • don't do audio analysis
- 4. D’ IMATGE I GRÀFICS! TECNOLOGIE S! • Serious game to practice singing and rhythm • The user is presented with a small village to explore • The houses are small, and can be enhanced by the user: • After clicking a house, a song is presented • The user sings to the best of his abilities • The house evolves in response to the user singing • The user can continue exploration and singing until he has practiced enough. Game descrip-on
- 6. D’ IMATGE I GRÀFICS! TECNOLOGIE S! System architecture • Three main components: • Game client (Unity) • Assets (on a web server) • Recommendation and Evaluation server • Audio and rhythm evaluation (matlab) • Recommendation system (C++ library) • Interconnection: • TCP sockets (game-server) • HTML (game-webserver)
- 7. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Game client • Object oriented, asynchronous architecture using the Unity game engine. Object behaviour uses C# scripts. • The main objects are: • World Creation (terrain, house creation and evolution) • World View: Camera, NPR effects, Dome transforms, GUI • World control: Kinect, mouse interaction • Asset downloading, communication with web server. • Microphone: song recording • ServerConnect: Communication with the server, evolving the GameState
- 8. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Music server • The game server is responsible for the automatic generation of musical excercises (songs) and the assessment of the user performance Server
- 9. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Music synthesis • A set of training songs is used and analyzed • Created songs replicate the selected style and complexity of that of the corresponding exemplars Training Generation
- 10. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Music assessment • Tempo, rythm and melody is compared against the reference • Works with inexpensive microphones too
- 11. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Graphics, Non-‐photorealis-c Rendering (NPR) • Post processing stylization – State-of-the-art image processing (on the color and depth) • Flexible, a wide range of effects • Commercial Unity package under finalization Original rendering (“Photo-realistic”) Color Comic: Image simplification + edges
- 12. D’ IMATGE I GRÀFICS! TECNOLOGIE S! NPR: edge effects Intensity + depth-normal edges Colored painting: edge color is driven by the (original) rendered color
- 13. D’ IMATGE I GRÀFICS! TECNOLOGIE S! • Replacement of the shadowed pixels by another color NPR: Shadow effects Complementary shadows: Shadow color is the complementary color of the shadowed surface
- 14. D’ IMATGE I GRÀFICS! TECNOLOGIE S! • Increasing the level of abstraction by depth NPR: depth sensa-on Depth-based desaturation Depth-based simplification: Texture details are removed in the background
- 16. D’ IMATGE I GRÀFICS! TECNOLOGIE S! • Augmented reality: mobile device, augment camera image with controls (e.g. buttons) • Kinect: exergame (imitating real movements, like walking or turning) or control by hand movements Natural interac-on AR button Exergame: walk
- 17. D’ IMATGE I GRÀFICS! TECNOLOGIE S! • Different projection/camera settings – E.g. stereo: 2 cameras, dome: 360º projection with cube-map Stereo, top-bottom Stereo, side-by-side Immersive visualiza-on
- 18. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Immersive visualiza-on Spherical dome source: immersivedisplay.co.uk Immersapod
- 19. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Immersive visualiza-on • NPR+immersive: NPR on the final image – Regular 2D image processing • No added computational complexity for using immersive displays w.r.t. NPR Dome+color comic Immersive+B&W comic
- 20. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Conclusions • Music teaching game • State-of-the-art automatic music synthesis and singing assessment • Controllable graphics style (NPR) as post- processing • Support for natural interaction, immersive visualization
- 21. D’ IMATGE I GRÀFICS! TECNOLOGIE S! Thank you! Questions? D’ IMATGE I GRÀFICS! TECNOLOGIES!