Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
SelfieCap
By Bin Shen, Jennifer Rubinovitz, Michał Wolski
DRAW
1. People love comics!
2. Most people can’t draw
3. We could try to use
computer vision to
automate the process
Origi...
Proof
Momentcam
10,000,000 - 50,000,000 installs
Cartoon Camera
10,000,000 - 50,000,000 installs
Otaku Camera
1,000,000 - ...
Ran into some issues
1. State of the art in human pose estimation:
http://www.vision.ee.ethz.ch/~hpedemo/
More issues
2. We couldn’t find a
single API to do the
drawing for us
3. So we tried to do the
drawing ourselves
Turns out we can’t draw either
4. We used Cocos2D to
draw and animate this
guy =>
Pivot
Motion capture without the markers
Face Tracking
How it works
● Learn a model for each
keypoint on the face
○ considering small patches in
the image
● Then constrain their...
Constrained Local Models
Point Distribution Model
image
point scaling
parameter
rotation
matrix
translationbasis of
variationsmean
location
of
obje...
Global shape prior
● The mean point locations are
computed by taking the
average of their locations in a
labeled dataset
●...
Graphics
Turns out to be pretty hard:
● Had to find a good graphics
engine for iOS
● Then learn how to use it
(ended up pi...
Model & Rigging
More things to learn:
● Using Maya
● Rigging Models
Then we had some
issues porting them to
iOS
Model & Rigging
● Maya => .pod => iOS
using PVRGeoPOD
converter
● Most of the advanced
maya functionality
didn’t work on i...
Model & Rigging
● Normal directions
● Smooth meshes
● Texturing
● Texture mapping
● Skinning(weight
painting)
iOS Application
● Facetracker
○ iOS integration
○ OpenCV
● Graphics
○ Cocos3d
○ Model display
○ Model tweak
● Video stream...
Graphics on iOS
Video Stream
+ Face Tracking
+ 3D Graphics =>
Possible Improvements
● Use animation blending instead of key
points
○ smoother control
● Integrate more things into 3D
○ ...
Demo!
Upcoming SlideShare
Loading in …5
×

SelfieCap

A project I worked on in a Computer Vision and Machine Learning for Mobile Devices class at Columbia for the Spring 2014 term.

  • Login to see the comments

  • Be the first to like this

SelfieCap

  1. 1. SelfieCap By Bin Shen, Jennifer Rubinovitz, Michał Wolski
  2. 2. DRAW 1. People love comics! 2. Most people can’t draw 3. We could try to use computer vision to automate the process Originally wanted to draw comics
  3. 3. Proof Momentcam 10,000,000 - 50,000,000 installs Cartoon Camera 10,000,000 - 50,000,000 installs Otaku Camera 1,000,000 - 5,000,000 installs
  4. 4. Ran into some issues 1. State of the art in human pose estimation: http://www.vision.ee.ethz.ch/~hpedemo/
  5. 5. More issues 2. We couldn’t find a single API to do the drawing for us 3. So we tried to do the drawing ourselves
  6. 6. Turns out we can’t draw either 4. We used Cocos2D to draw and animate this guy =>
  7. 7. Pivot Motion capture without the markers
  8. 8. Face Tracking
  9. 9. How it works ● Learn a model for each keypoint on the face ○ considering small patches in the image ● Then constrain their arrangement using a point distribution model
  10. 10. Constrained Local Models
  11. 11. Point Distribution Model image point scaling parameter rotation matrix translationbasis of variationsmean location of object point non-rigid motion parameter
  12. 12. Global shape prior ● The mean point locations are computed by taking the average of their locations in a labeled dataset ● The basis of variation can be found by running PCA on the covariance matrix of the points PCA Components Demo
  13. 13. Graphics Turns out to be pretty hard: ● Had to find a good graphics engine for iOS ● Then learn how to use it (ended up picking Cocos3D)
  14. 14. Model & Rigging More things to learn: ● Using Maya ● Rigging Models Then we had some issues porting them to iOS
  15. 15. Model & Rigging ● Maya => .pod => iOS using PVRGeoPOD converter ● Most of the advanced maya functionality didn’t work on iOS ○ could only use joints
  16. 16. Model & Rigging ● Normal directions ● Smooth meshes ● Texturing ● Texture mapping ● Skinning(weight painting)
  17. 17. iOS Application ● Facetracker ○ iOS integration ○ OpenCV ● Graphics ○ Cocos3d ○ Model display ○ Model tweak ● Video streaming ○ AVFoundation
  18. 18. Graphics on iOS Video Stream + Face Tracking + 3D Graphics =>
  19. 19. Possible Improvements ● Use animation blending instead of key points ○ smoother control ● Integrate more things into 3D ○ Interaction ● Performance tuning
  20. 20. Demo!

×