Introduction to
Action Recognition
in Python
@wideio

Bertrand NOUVEL, bn@wide.io
Jonathan KELSEY, jk@wide.io
Bernard HERN...
Outline
- (While you download the data) Forewords & Overview
- PART 1: Video-processing in python
- PART 2: The pipeline i...
FOREWORDS

WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HE...
GETTING READY

1)
2)

GET THE ABSOLUTELY ESSENTIAL PACKAGES :
Python 2.x / 3.x
Numy/Scipy
PIL
OpenCV2

ANACONDA
WAKARI

GE...
WHO WE ARE
WIDE IO - Democratising the best algorithms
startup

+

consultancy

WIDE IO - DEMOCRATISING THE BEST ALGORITHM...
Why do Computer Vision in Python?
Reflexive
Multiparadigm

Computer vision is difficult.

“””
Readable

Python is 1000 slo...
ACTION RECOGNITION

WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - B...
What is action recognition ?
Classification task

SEMANTIC GAP
KTH, Human action dataset (Laptev)
Classify actions

WIDE I...
Types of Systems
FEATURE BASED
ACTION SPECIFIC

HOLISTIC
APPROACHES

APPEARANCE
BASED

Many priors
WIDE IO - DEMOCRATISING...
The Very-Traditional Pipeline
RAW DATA

FEATURE-EXTRACTION

MACHINE LEARNING

WIDE IO - DEMOCRATISING THE BEST ALGORITHMS
...
The Very-Traditional Pipeline

VIDEO
FRAMES

SPARSIFICATION

CLASSIFICATION
METHOD

WIDE IO - DEMOCRATISING THE BEST ALGOR...
The Very-Traditional Pipeline

VIDEO
FRAMES

BAGS OF KEYPOINTS

SVM

WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUC...
PART I
PROCESSING
VIDEOS

TOWARDS FRAMEWORKS
Reading Videos
-

PYFFMPEG (outdated)

-

OPENCV
MLT

-

ON WAKARI

anaconda$ vi io/player_mlt.py
WIDE IO - DEMOCRATISING ...
Reading Videos
-

PYFFMPEG (outdated)

-

OPENCV
MLT

-

ON WAKARI

anaconda$ vi io/player_cv.py
WIDE IO - DEMOCRATISING T...
Displaying Images
Alternatives

Recommended

anaconda$

python io/display_pyglet.py

WIDE IO - DEMOCRATISING THE BEST ALGO...
Harris Corner Detector
Detecting interest points:
imsmooth=scipy.ndimage.gaussian_filter
def harris(I,alpha=0.04,si=1):
Ix...
Optical flow - wrapper based code

anaconda$

python keyponats/custom_feature.py

WIDE IO - DEMOCRATISING THE BEST ALGORIT...
Spatio-temporal keypoints
Use the two previous elements to compute keypoints that contain information about the movement

...
PART II
UNDERSTANDING THE KEY
ELEMENTS OF THE PIPELINE
Feature extractions

WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - ...
Feature extractions

SIFT (128)
Descriptor vector
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECO...
Algorithms:
Type (density, connectivity, …)
Dimension (descriptor, position, …)

Recursive clustering
Connectivity (ward)
...
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - J...
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - J...
WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 2014 - BERNARD HERNANDEZ - J...
Support Vector Machines

WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

INTRODUCTION TO ACTION RECOGNITION
PYDATA LONDON 201...
Support Vector Machines
Weighted-Support Vector Machines
Different support

WIDE IO - DEMOCRATISING THE BEST ALGORITHMS

I...
Support Vector Machines
Weighted-Support Vector Machines
Different support
Different relevance (outliers)

WIDE IO - DEMOC...
PART III
PUTTING EVERYTHING
TOGETHER
Feature detection
PCA Projection

Clustering
Eigen Vectors & Mean

Graphic Model
Model.model

centers

BOW

WIDE IO - DEMO...
Feature detection stack for image

PCA project with graphic model

Distance of projection to BOW cluster centers with metr...
Training data

Training Labels

Testing data

Testing labels

Feature computer
SVM predict all

Feature stack
SVM auto tra...
Introduction to
Action Recognition
in Python
@wideio

Bertrand NOUVEL, bn@wide.io
Jonathan KELSEY, jk@wide.io
Bernard HERN...
Upcoming SlideShare
Loading in …5
×

Wide IO Presentation PyData London

1,101 views

Published on

Tutorial from PyData London: An Introduction to Video Action Recognition by Bertrand Nouvel, Jonathan Kelsey and Bernard Hernandez from Wide IO

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,101
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Wide IO Presentation PyData London

  1. 1. Introduction to Action Recognition in Python @wideio Bertrand NOUVEL, bn@wide.io Jonathan KELSEY, jk@wide.io Bernard HERNANDEZ, bh@wide.io PYDATA LONDON 2014
  2. 2. Outline - (While you download the data) Forewords & Overview - PART 1: Video-processing in python - PART 2: The pipeline in details - PART 3: Putting it together WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  3. 3. FOREWORDS WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  4. 4. GETTING READY 1) 2) GET THE ABSOLUTELY ESSENTIAL PACKAGES : Python 2.x / 3.x Numy/Scipy PIL OpenCV2 ANACONDA WAKARI GET THE SOURCE CODE git clone https://bitbucket.org/wideio/pydata.git WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  5. 5. WHO WE ARE WIDE IO - Democratising the best algorithms startup + consultancy WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  6. 6. Why do Computer Vision in Python? Reflexive Multiparadigm Computer vision is difficult. “”” Readable Python is 1000 slower than C++ Lots of packages Lightweight Best community Operator overloading WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  7. 7. ACTION RECOGNITION WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  8. 8. What is action recognition ? Classification task SEMANTIC GAP KTH, Human action dataset (Laptev) Classify actions WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  9. 9. Types of Systems FEATURE BASED ACTION SPECIFIC HOLISTIC APPROACHES APPEARANCE BASED Many priors WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL DEEP LEARNING Less priors
  10. 10. The Very-Traditional Pipeline RAW DATA FEATURE-EXTRACTION MACHINE LEARNING WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  11. 11. The Very-Traditional Pipeline VIDEO FRAMES SPARSIFICATION CLASSIFICATION METHOD WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  12. 12. The Very-Traditional Pipeline VIDEO FRAMES BAGS OF KEYPOINTS SVM WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  13. 13. PART I PROCESSING VIDEOS TOWARDS FRAMEWORKS
  14. 14. Reading Videos - PYFFMPEG (outdated) - OPENCV MLT - ON WAKARI anaconda$ vi io/player_mlt.py WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  15. 15. Reading Videos - PYFFMPEG (outdated) - OPENCV MLT - ON WAKARI anaconda$ vi io/player_cv.py WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  16. 16. Displaying Images Alternatives Recommended anaconda$ python io/display_pyglet.py WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  17. 17. Harris Corner Detector Detecting interest points: imsmooth=scipy.ndimage.gaussian_filter def harris(I,alpha=0.04,si=1): Ix,Iy = scipy.gradient(I) H11 = imsmooth(Ix*Ix, si) H12 = imsmooth(Ix*Iy, si) H22 = imsmooth(Iy*Iy, si) return ((H11*H22 - H12**2) - alpha*(H11+H22)**2) anaconda$ python keypoints/harris.py WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  18. 18. Optical flow - wrapper based code anaconda$ python keyponats/custom_feature.py WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  19. 19. Spatio-temporal keypoints Use the two previous elements to compute keypoints that contain information about the movement anaconda$ python keyponats/custom_feature.py Ideas for extension: Make the same with a pyramidal approach. WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  20. 20. PART II UNDERSTANDING THE KEY ELEMENTS OF THE PIPELINE
  21. 21. Feature extractions WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  22. 22. Feature extractions SIFT (128) Descriptor vector WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL SURF (64)
  23. 23. Algorithms: Type (density, connectivity, …) Dimension (descriptor, position, …) Recursive clustering Connectivity (ward) Centroid (k-means) WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL Density (optics)
  24. 24. WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  25. 25. WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  26. 26. WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  27. 27. Support Vector Machines WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  28. 28. Support Vector Machines Weighted-Support Vector Machines Different support WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  29. 29. Support Vector Machines Weighted-Support Vector Machines Different support Different relevance (outliers) WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  30. 30. PART III PUTTING EVERYTHING TOGETHER
  31. 31. Feature detection PCA Projection Clustering Eigen Vectors & Mean Graphic Model Model.model centers BOW WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  32. 32. Feature detection stack for image PCA project with graphic model Distance of projection to BOW cluster centers with metric Invert with 'discriminator' to turn into weighting Normalise (Area=1) Mean of stack to create histogram WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  33. 33. Training data Training Labels Testing data Testing labels Feature computer SVM predict all Feature stack SVM auto train Param grid WIDE IO - DEMOCRATISING THE BEST ALGORITHMS INTRODUCTION TO ACTION RECOGNITION PYDATA LONDON 2014 - BERNARD HERNANDEZ - JONATHAN KESLEY - BERTRAND NOUVEL
  34. 34. Introduction to Action Recognition in Python @wideio Bertrand NOUVEL, bn@wide.io Jonathan KELSEY, jk@wide.io Bernard HERNANDEZ, bh@wide.io PYDATA LONDON 2014

×