Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Human Motion Forecasting (Generation) with RNNs


Published on

"On human motion prediction using recurrent neural networks", Julieta Martinez, Michael J. Black, Javier Romero. CVPR2017

Published in: Engineering
  • Be the first to comment

Human Motion Forecasting (Generation) with RNNs

  1. 1. Terry Taewoong Um ( University of Waterloo Department of Electrical & Computer Engineering Terry T. Um ON HUMAN MOTION PREDICTION USING RNNS (2017) 1
  2. 2. MOTIVATION TO CHOOSE THIS PAPER Terry Taewoong Um ( • I have applied convolutional neural networks (CNNs) to classify wearable sensor data in my research, but haven’t applied recurrent neural networks (RNNs) in my research. Exercise Motion Classification from Large- Scale Wearable Sensor Data Using CNNs (2016) Classified 50 gym exercises with 92% Data Augmentation of Wearable Sensor Data for Parkinson’s Disease Monitoring using CNNs (2017) classification accuracy 77%  92%
  3. 3. 2 baseline papers
  4. 4. MOTION FORECASTING • Motion forecasting (Motion prediction) : Given a person’s past motions, forecast the most likely future 3D poses Terry Taewoong Um ( • e.g.) Sentence completion motion forecasting ≈ a high-dimensional and nonlinear version of sentence completion
  5. 5. BACKGROUND: RNN Terry Taewoong Um ( • Recurrent Neural Networks (RNNs) (unfold)  vanishing or exploding gradient problem  solve by using gate units (Xavier Giro,
  6. 6. BACKGROUND: LSTM & GRU Terry Taewoong Um ( • Solution : Let nodes to decide whether forget or bypass the information  Gate units: Long short-term memory(LSTM) or gated recurrent unit (GRU) (Christopher Olah, LSTM GRU similar performance, but less computation
  7. 7. BACKGROUND: RNN Terry Taewoong Um ( (Andrej Karpathy,
  8. 8. SEQUENCE GENERATION (Andrej Karpathy,
  9. 9. SEQUENCE GENERATION (Andrej Karpathy,
  10. 10. SIMPLEST APPROACH Terry Taewoong Um ( • Just apply a LSTM to joint angle data (2015) ERD Encoder-Recurrent-Decoder LSTM ch?v=CvaKD1NGcBk [Result] Contribution : It’s the first LSTM work with skeleton data
  11. 11. RELATED WORK ERD Terry Taewoong Um (
  12. 12. RELATED WORK SRNN Terry Taewoong Um (
  13. 13. C.F.) HIERARCHICAL RNN MHAD dataset (11 actions) HDM05 dataset (65 actions) MSR-Action3D dataset (20 actions) (2016)
  14. 14. MOTION FORECASTING USING RNN Terry Taewoong Um ( • Evaluation criteria • Problem of RNN-based methods for short-term (<=0.5s) for long-term (>=1s) Learning Human Motion Models for Long-term Predictions (2017), P. Ghosh et al.
  15. 15. WHAT’S THE PROBLEMS? Other problems: - Model is so complicated that large data is needed - Action-specific network : use a certain-action data Terry Taewoong Um (
  17. 17. PROPOSED SOLUTION (from K. He’s ResNet tutorial at ICML2016)
  18. 18. EXPERIMENT SETTING Terry Taewoong Um ( • Details
  19. 19. RESULTS (SA : single action data, MA: multiple action data)
  20. 20. RESULTS • Zero-velocity shows a good performance • Sampling-based loss gives plausible motion generation + no noise scheduling is needed • Residual connection improves the performance • Using single action data < Using all action data (data quality < data quantity) • Aperiodic motions are hard to model with RNNs • Action labels helps the learning process • Small loss != good qualitative long-term motion  need to propose a new loss • Unsupervised approach gives a comparative result • This research area hasn’t been matured, so, we have a chance . Terry Taewoong Um ( Idea: (for t+1 prediction) Rather than residual input 𝑋𝑡 residual input 𝑋𝑡 + 𝑋𝑡 𝑑𝑡, or explicitly exploiting 𝑋 and 𝑋
  21. 21. BONUS: MORE RESEARCHES FROM 2017 Terry Taewoong Um (
  22. 22. BONUS: MORE RESEARCHES FROM 2017 Terry Taewoong Um (
  23. 23. BONUS: MORE RESEARCHES FROM 2017 Terry Taewoong Um ( enevillegas/hierch_vid