Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Query-by-Dancing: A Dance Music Retrieval System Based on Body-Motion Similarity

81 views

Published on

This paper presents Query-by-Dancing, which is a dance music retrieval system that enables a user to retrieve music by dance motion. When dancers search for music to play when dancing, they sometimes refer to music in online dance videos in which motions similar to their own dance are used. Previous music retrieval systems, however, cannot support retrieval specialized for dancing because they do not accept dance motions as the query. We therefore developed our Query-by-Dancing system that uses a video of a dancing user as the input query to search a database of dance videos. This query video is recorded using an ordinary RGB camera that does not obtain depth information, such as one attached to a smartphone. The pose and motions in the query are then analyzed and used to retrieve a set of dance videos with similar pose and motions. The system then enables the user to browse the musical pieces attached to those retrieved videos so that the user can find a piece that is appropriate for his/her dancing. An interesting problem here is that simply finding the most similar videos results in getting results not similar in terms of genres of dance motions. We solved this by using a novel measure similar to tf-idf to weight the importance of dance motions when retrieving videos. We conducted the comparative experiments on four dance genres and confirmed that the system gained an average of three points or more evaluation scored for three dance genres, Waack, Pop, Break, and that our proposed method was able to deal with different dance genres.

Published in: Entertainment & Humor
  • Be the first to comment

  • Be the first to like this

Query-by-Dancing: A Dance Music Retrieval System Based on Body-Motion Similarity

  1. 1. Query-by-Dancing: A Dance Music Retrieval System Based on Body-Motion Similarity Shuhei Tsuchida, Satoru Fukayama, Masataka Goto National Institute of Advanced Industrial Science and Technology (AIST) 10.01.2019 MMM2019
  2. 2. It is important for dancers to find appropriate musical pieces. Dancers often look for musical pieces Stage performance Practice DJ, freestyling Music video source: https://youtu.be/lRY-rdCQ7eI source: https://youtu.be/pcTBr64Elew
  3. 3. How does a dancer usually find musical pieces? Keywords • Hiphop dance • Break dance • etc. Musical pieces Existing Retrieval System Dancer Dance Style
  4. 4. How does a dancer usually find musical pieces? Keywords • Hiphop dance • Break dance • etc. Musical pieces Existing Retrieval System Dancer Dance Style
  5. 5. How does a dancer usually find musical pieces? Keywords • Hiphop dance • Break dance • etc. Musical pieces Existing Retrieval System Dancer Dance Style
  6. 6. How does a dancer usually find musical pieces? Keywords • Hiphop dance • Break dance • etc. Musical pieces Existing Retrieval System Dancer Dance Style Too vague
  7. 7. How does a dancer usually find musical pieces? Keywords • Favorite dancer • Favorite group • Dance events • etc. Existing Retrieval System Dancer Dance Style Musical piece Compare
  8. 8. How does a dancer usually find musical pieces? Keywords • Favorite dancer • Favorite group • Dance events • etc. Existing Retrieval System Dancer Dance Style Compare Musical piece
  9. 9. How does a dancer usually find musical pieces? Keywords • Favorite dancer • Favorite group • Dance events • etc. Existing Retrieval System Dancer Dance Style Musical piece Compare
  10. 10. How does a dancer usually find musical pieces? Keywords • Favorite dancer • Favorite group • Dance events • etc. Existing Retrieval System Dancer Dance Style Musical piece Compare Tedious and time-consuming
  11. 11. How does a dancer usually find musical pieces? Query-by-Dancing Dancer Dance motion Musical pieces
  12. 12. How does a dancer usually find musical pieces? Query-by-Dancing Dancer Dance motion Musical pieces
  13. 13. How does a dancer usually find musical pieces? Query-by-Dancing Dancer Dance motion Musical pieces
  14. 14. System overview
  15. 15. Query-by-Dancing: retrieval results
  16. 16. Query-by-Dancing: retrieval results Input video
  17. 17. Query-by-Dancing: retrieval results Retrieval results
  18. 18. Query-by-Dancing: retrieval results (Break)
  19. 19. Query-by-Dancing: retrieval results (Break)
  20. 20. Query-by-Dancing: retrieval results Similarity
  21. 21. Query-by-Dancing: retrieval results 0,1,2, … … 97, 98, 99
  22. 22. Query-by-Dancing: retrieval results Top 10
  23. 23. Query-by-Dancing: retrieval results Retrieval rank, Dance genre
  24. 24. Query-by-Dancing: retrieval results (Pop)
  25. 25. Query-by-Dancing: retrieval results (Pop)
  26. 26. Query-by-Dancing: retrieval results (Waack)
  27. 27. Query-by-Dancing: retrieval results (Waack)
  28. 28. System flow vθ , vΔθ , vΔ θ d, Wα U α 2 (i) Input video Dance video database Video1 Video2 Video3 C Select a dancer Calculate and integrate joint angles vθ, vΔθ, and vΔ θ 2 Pre-processing Calculate similarity Calculate weight W Calculate weighted Euclidean distances of body-motion U(i) Output musical pieces
  29. 29. System flow vθ , vΔθ , vΔ θ d, Wα U α 2 (i) Input video Dance video database Video1 Video2 Video3 C Select a dancer Calculate and integrate joint angles vθ, vΔθ, and vΔ θ 2 Pre-processing Calculate similarity Calculate weight W Calculate weighted Euclidean distances of body-motion U(i) Output musical pieces
  30. 30. Pre-processing: select a dancer The system first estimates the person's skeleton information in all video frames using the OpenPose library. Z. Cao et al.: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields (2017).
  31. 31. Pre-processing: select a dancer The system first estimates the person's skeleton information in all video frames using the OpenPose library. Z. Cao et al.: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields (2017). A dancer was defined as a person who is close to the center of the screen and appears big.
  32. 32. Pre-processing: calculate joint angles • Pose • Motion
  33. 33. Pre-processing: calculate joint angels • Pose • Motion
  34. 34. Pre-processing: calculate joint angels • Pose • Motion
  35. 35. System flow vθ , vΔθ , vΔ θ d, Wα U α 2 (i) Input video Dance video database Video1 Video2 Video3 C Select a dancer Calculate and integrate joint angles vθ, vΔθ, and vΔ θ 2 Pre-processing Calculate similarity Calculate weight W Calculate weighted Euclidean distances of body-motion U(i) Output musical pieces
  36. 36. Calculate dance motion similarity: Euclidean distance t
  37. 37. Calculate dance motion similarity: Euclidean distance t
  38. 38. Calculate dance motion similarity: Euclidean distance t
  39. 39. Calculate dance motion similarity: Euclidean distance t
  40. 40. Calculate dance motion similarity: Euclidean distance The system calculates the Euclidean distances between all frames of an input video and all frames of a video in the video database. d(vin(n), v(i)(m)) 1 ≤ n ≤ Nin 1 ≤ m ≤ N(i) 1 ≤ i ≤ I
  41. 41. Calculate dance motion similarity: Euclidean distance The system calculates the Euclidean distances between all frames of an input video and all frames of a video in the video database. d(vin(n), v(i)(m)) 1 ≤ n ≤ Nin 1 ≤ m ≤ N(i) 1 ≤ i ≤ I • If the system uses these calculation results (Euclidean distance) as a similarity distance, → The system doesn't work well. Reason: Most of dance videos include several common movements. • The system uses weighting methods to consider these common movements.
  42. 42. Small Weight Unimportant dance motion Many common motions in the database Calculate dance motion similarity: weight Large Weight Important dance motion Rare motions in the database The system uses a method similar to tf-idf often used in natural language processing.
  43. 43. Small Weight Unimportant dance motion Many common motions in the database Calculate dance motion similarity: weight Large Weight Important dance motion Rare motions in the database The system uses a method similar to tf-idf often used in natural language processing. Database Video 1 Video 2 Video 3 Video 4 Video 5 … …
  44. 44. Small Weight Unimportant dance motion Many common motions in the database Calculate dance motion similarity: weight Large Weight Important dance motion Rare motions in the database The system uses a method similar to tf-idf often used in natural language processing. Database Video 1 Video 2 Video 3 Video 4 Video 5 … …
  45. 45. Small Weight Unimportant dance motion Many common motions in the database Calculate dance motion similarity: weight Large Weight Important dance motion Rare motions in the database The system uses a method similar to tf-idf often used in natural language processing.
  46. 46. System overview vθ , vΔθ , vΔ θ d, Wα U α 2 (i) Input video Dance video database Video1 Video2 Video3 C Select a dancer Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2 Pre- processing Calculate similarity Calculate weight W Calculate weighted Euclidean distances of body-motion U(i) Output musical pieces
  47. 47. System overview vθ , vΔθ , vΔ θ d, Wα U α 2 (i) Input video Dance video database Video1 Video2 Video3 C Select a dancer Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2 Pre- processing Calculate similarity Calculate weight W Calculate weighted Euclidean distances of body-motion U(i) Output musical pieces Weighted- Euclidean distance 0 IVideos in database
  48. 48. System overview vθ , vΔθ , vΔ θ d, Wα U α 2 (i) Input video Dance video database Video1 Video2 Video3 C Select a dancer Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2 Pre- processing Calculate similarity Calculate weight W Calculate weighted Euclidean distances of body-motion U(i) Output musical pieces Weighted- Euclidean distance 0 IVideos in database
  49. 49. System overview vθ , vΔθ , vΔ θ d, Wα U α 2 (i) Input video Dance video database Video1 Video2 Video3 C Select a dancer Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2 Pre- processing Calculate similarity Calculate weight W Calculate weighted Euclidean distances of body-motion U(i) Output musical pieces Weighted- Euclidean distance 0 IVideos in database The system finds the videos with the top k values among weighted- Euclidean distances and presents the candidate musical pieces.
  50. 50. Evaluation experiment Input video Output musical pieces
  51. 51. Evaluation experiment Input video Output musical pieces Strongly disagree  1 - 2 - 3 - 4 - 5  Strongly agree
  52. 52. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre
  53. 53. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre
  54. 54. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre
  55. 55. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre
  56. 56. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre
  57. 57. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre
  58. 58. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre
  59. 59. Results * : p<.05 * * * Averaged score Averagedevaluationscore Output musical pieces Input Dance Genre The participants can get the musical pieces that they dance easily.
  60. 60. Results * : p<.05 * * * Averaged score Averagedevaluationscore Input Dance Genre Output musical pieces
  61. 61. • We interviewed participants who gave a low score to pop to determine the reason for this. Discussion • The dance motions used as the query included "vibration" and "wave" dance techniques that match specific sounds.
  62. 62. • We interviewed participants who gave a low score to pop to determine the reason for this. Discussion • The dance motions used as the query included "vibration" and "wave" dance techniques that match specific sounds.
  63. 63. • We interviewed participants who gave a low score to pop to determine the reason for this. Discussion • The dance motions used as the query included "vibration" and "wave" dance techniques that match specific sounds.
  64. 64. • The dance motions used as the query included "vibration" and "wave" dance techniques that match specific sounds. → (Future work) An interface that supports more advanced motion matching. • We interviewed participants who gave a low score to pop to determine the reason for this. Discussion
  65. 65. Summary Query-by-Dancing • enables a user to retrieve a musical piece using dance motions. • can get better musical pieces by weighting the importance of dance motions. Future work • We plan to add a wider range of dance genres to the database of dance videos.

×