Kinect Motion Capture    for Filmmaking         3/23/13
How does the Kinect work?                            Photo by y3rdua
Getting Started: HardwareYou will need:• A Kinect, or generic equivalent such as the Asus Xtion• USB/AC adapter cable, if ...
Getting Started: SoftwareOpenNI driver bundle:simple-openni.googlecode.comKinectToPin:kinecttopin.com
Two competing SDKsOpenNI• Free and open source• Works with all models of Kinect as well as other devices such as the Asus ...
KinectToPinKinect Motion Capture for After Effects
What is KinectToPin?                              +       A standalone motion                                  A UI panel ...
Rig 2D character models in 3D
“Working Backwards”Traditional Animation                 Motion Capture Workflow1. Start with scene already           1. S...
“Working Backwards”Think of it like working with raw footage.
Character Design2D models in 3D space
Why multi-layered characters?Single-layer model:   Multi-layered:
Design Tips• “Break” your model at every  joint.• Build in overlap so joints have  room to rotate without  disconnecting.•...
CutoutsRaster-based characters drawn or cut out of existing images or photographs.• Work big if you want to be able to zoo...
Vector-Based CharactersDrawn with math, not pixels. Easier to break into layers – but watch out.• Can be built in Illustra...
Motion CaptureRecording with KinectToPin
Setting up your space to record                6 – 10 feet
Capturing• Record 15-point 3D skeletal  tracking data• Capture in sync to dialogue• Remote control with a standard  presen...
Acting for Kinect• The Kinect can’t record what it can’t see.   • Keep your entire body in the camera’s range   • Face the...
Data formats for exportKinectToPin can output quite a few data, but the ones we’ll beusing today are:• XML (Native recordi...
Lunch.
Importing tracking dataNative import or copy & paste
Meet the KinectToPin UI Panel.• Create character  templates• 2D and 3D setups• Native XML import  (SLOW)• Automatically ad...
The Puppet ToolTHREE OR MORE PINS ON ONE LAYER:   TWO PINS EACH, MULTIPLE LAYERS:Elbows don’t bend this way. Ow.    Rotati...
AE Puppet best practices• Work BIG. Start high-res, scale down in the puppet comp, then  scale back up in the project comp...
Character RiggingUsing the KinectToPin UI panel
Retargeting for arbitrary charactershapes
It’s easy!Just grab a red control null and move it to the appropriateposition.
Quick auto-lip sync1. Select audio layer and choose“Convert Audio to Keyframes.”
Quick auto-lip sync2. Alt-click the stopwatch to add an expression. Use thepickwhip to connect the audio keyframes to the ...
Quick auto-lip sync3. Size the audio data to fit. Use this expression:      audio = [your audio keyframes];      linear(au...
One last thought:Embrace the limitations of      the format.
Further non-K2P toolsFREE                                  COMMERCIALBrekel Kinect                         iPi Soft   Open...
Get in TouchVictoria Necewww.victorianece.com@FakeGreenDress
K2P workshop 3-23-13
Upcoming SlideShare
Loading in …5
×

K2P workshop 3-23-13

353 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
353
On SlideShare
0
From Embeds
0
Number of Embeds
110
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Why DIY MoCap?Motion capture is now cheap and accessibleNo longer need a giant studioIt’s FAST.Ideal use cases: short-turnaround animation, rapid prototyping etc.
  • Microsoft SDK vs. OpenNI
  • Can add multiple motion tracks and duplicate nullsPin names need to match control nulls
  • 2D vs. 3D: Avoiding the “Uncanny Valley”Awesome glitches
  • K2P workshop 3-23-13

    1. 1. Kinect Motion Capture for Filmmaking 3/23/13
    2. 2. How does the Kinect work? Photo by y3rdua
    3. 3. Getting Started: HardwareYou will need:• A Kinect, or generic equivalent such as the Asus Xtion• USB/AC adapter cable, if you have a Kinect that was bundled with the Xbox (don’t buy the Microsoft adapter, the cheap ones work fine)• Enough space to perform
    4. 4. Getting Started: SoftwareOpenNI driver bundle:simple-openni.googlecode.comKinectToPin:kinecttopin.com
    5. 5. Two competing SDKsOpenNI• Free and open source• Works with all models of Kinect as well as other devices such as the Asus Xtion.• Cross platform – even supports ARM processors and AndroidMicrosoft (official) SDK• More tracking points• Estimates occluded joints and hand/foot rotation• Special case capture modes (seated figure etc.)• Only works with Kinect For Windows – on Windows
    6. 6. KinectToPinKinect Motion Capture for After Effects
    7. 7. What is KinectToPin? + A standalone motion A UI panel for After Effects capture application
    8. 8. Rig 2D character models in 3D
    9. 9. “Working Backwards”Traditional Animation Motion Capture Workflow1. Start with scene already 1. Start with full color, full-motion storyboarded and timed out characters2. Create sketches of key poses for 2. Refine blocking and movement; each character add facial animation3. Add in-between drawings for 3. Choose shots and timing of movement edit, add camera moves4. Color and composite characters 4. (Optional) Re-record any in final scene performances that aren’t working
    10. 10. “Working Backwards”Think of it like working with raw footage.
    11. 11. Character Design2D models in 3D space
    12. 12. Why multi-layered characters?Single-layer model: Multi-layered:
    13. 13. Design Tips• “Break” your model at every joint.• Build in overlap so joints have room to rotate without disconnecting.• Parts that fall in front of and behind another layer (e.g. the hair of the model at left) should be two layers.• The hands, feet and head can each have multiple elements.
    14. 14. CutoutsRaster-based characters drawn or cut out of existing images or photographs.• Work big if you want to be able to zoom in on your characters• When cutting out layers, use “Refine Edge” to smooth your edges and clean up any extra junk that got selected• Clone/heal/paint in portions that will be missing or problematic when the model changes pose
    15. 15. Vector-Based CharactersDrawn with math, not pixels. Easier to break into layers – but watch out.• Can be built in Illustrator, Photoshop or with AE Shape Layers• Vector-based characters are not infinitely scalable as you’d expect: rigging with AE’s Puppet Tool generates a mesh at a set resolution.• Building puppets with vector layers adds an extra step: they need to be precomposed before rigging.
    16. 16. Motion CaptureRecording with KinectToPin
    17. 17. Setting up your space to record 6 – 10 feet
    18. 18. Capturing• Record 15-point 3D skeletal tracking data• Capture in sync to dialogue• Remote control with a standard presentation remote• Open source and cross-platform: runs on OpenNI, built in Processing, also works with generic sensors like the Asus Xtion
    19. 19. Acting for Kinect• The Kinect can’t record what it can’t see. • Keep your entire body in the camera’s range • Face the camera when capturing • Watch out for occluded joints (sitting down, putting your hands behind your back etc. can cause it to lose the track)• It’s surprisingly good at picking up subtle posture shifts, but not smaller motions like laughter.• Use a different performer for each character if possible
    20. 20. Data formats for exportKinectToPin can output quite a few data, but the ones we’ll beusing today are:• XML (Native recording format, no export required)• After Effects 3D Point Controls (CS5.5+)• After Effects 2D Point Controls (CS4+)Tip: Edit KinectToPin’s settings.txt file to choose your outputformats.
    21. 21. Lunch.
    22. 22. Importing tracking dataNative import or copy & paste
    23. 23. Meet the KinectToPin UI Panel.• Create character templates• 2D and 3D setups• Native XML import (SLOW)• Automatically add expressions to smooth tracking data• Automate rigging
    24. 24. The Puppet ToolTHREE OR MORE PINS ON ONE LAYER: TWO PINS EACH, MULTIPLE LAYERS:Elbows don’t bend this way. Ow. Rotation with stretch and squash!
    25. 25. AE Puppet best practices• Work BIG. Start high-res, scale down in the puppet comp, then scale back up in the project comp (enable Collapse Transformations).• Precompose if you’re using vectors or shape layers• Keep multiple angles of the same character in sync High-res Start big. Character Layers Project Comp End big. Rigging Template
    26. 26. Character RiggingUsing the KinectToPin UI panel
    27. 27. Retargeting for arbitrary charactershapes
    28. 28. It’s easy!Just grab a red control null and move it to the appropriateposition.
    29. 29. Quick auto-lip sync1. Select audio layer and choose“Convert Audio to Keyframes.”
    30. 30. Quick auto-lip sync2. Alt-click the stopwatch to add an expression. Use thepickwhip to connect the audio keyframes to the mouth animation.
    31. 31. Quick auto-lip sync3. Size the audio data to fit. Use this expression: audio = [your audio keyframes]; linear(audio,sourceMin,sourceMax,targetMin,targetMax) Lowest value of source data Highest value of source data Closed mouth (probably 0) Widest possible open mouth
    32. 32. One last thought:Embrace the limitations of the format.
    33. 33. Further non-K2P toolsFREE COMMERCIALBrekel Kinect iPi Soft Open-source capture for 3D Uses two Kinects to record animation (Windows-only) www.ipisoft.com www.brekel.com NI MateDuik Kinect for OSC/MIDI/Blender Inverse kinematics plugin for AE www.ni-mate.com www.duduf.net
    34. 34. Get in TouchVictoria Necewww.victorianece.com@FakeGreenDress

    ×