Kinect motion capture allows filmmakers to capture full-body motion and facial animation data using a Kinect sensor. The KinectToPin software records 3D skeletal tracking data from the Kinect and exports it to After Effects where it can be used to drive 2D character rigs. Rigging characters involves breaking them into multiple layers that map to the tracked joints, allowing for natural movement and animation. Motion capture with Kinect provides a more efficient workflow than traditional frame-by-frame animation by starting with live-action movement that can then be applied to animated characters.

1. Kinect Motion Capture
for Filmmaking
3/23/13

2. How does the Kinect work?
Photo by y3rdua

3. Getting Started: Hardware
You will need:
• A Kinect, or generic equivalent such as the Asus Xtion
• USB/AC adapter cable, if you have a Kinect that was bundled
with the Xbox (don’t buy the Microsoft adapter, the cheap ones
work fine)
• Enough space to perform

4. Getting Started: Software
OpenNI driver bundle:
simple-openni.googlecode.com
KinectToPin:
kinecttopin.com

5. Two competing SDKs
OpenNI
• Free and open source
• Works with all models of Kinect as well as other devices such as the Asus Xtion.
• Cross platform – even supports ARM processors and Android
Microsoft (official) SDK
• More tracking points
• Estimates occluded joints and hand/foot rotation
• Special case capture modes (seated figure etc.)
• Only works with Kinect For Windows – on Windows

6. KinectToPin
Kinect Motion Capture for After Effects

7. What is KinectToPin?
+
A standalone motion
A UI panel for After Effects
capture application

8. Rig 2D character models in 3D

9. “Working Backwards”
Traditional Animation Motion Capture Workflow
1. Start with scene already 1. Start with full color, full-motion
storyboarded and timed out characters
2. Create sketches of key poses for 2. Refine blocking and movement;
each character add facial animation
3. Add in-between drawings for 3. Choose shots and timing of
movement edit, add camera moves
4. Color and composite characters 4. (Optional) Re-record any
in final scene performances that aren’t working

10. “Working Backwards”
Think of it like working with raw footage.

11. Character Design
2D models in 3D space

12. Why multi-layered characters?
Single-layer model: Multi-layered:

13. Design Tips
• “Break” your model at every
joint.
• Build in overlap so joints have
room to rotate without
disconnecting.
• Parts that fall in front of and
behind another layer (e.g. the
hair of the model at left)
should be two layers.
• The hands, feet and head can
each have multiple elements.

14. Cutouts
Raster-based characters drawn or cut out of existing images or photographs.
• Work big if you want to be able to zoom in on your characters
• When cutting out layers, use “Refine Edge” to smooth your
edges and clean up any extra junk that got selected
• Clone/heal/paint in portions that will be missing or problematic
when the model changes pose

15. Vector-Based Characters
Drawn with math, not pixels. Easier to break into layers – but watch out.
• Can be built in Illustrator, Photoshop or with AE Shape Layers
• Vector-based characters are not infinitely scalable as you’d
expect: rigging with AE’s Puppet Tool generates a mesh at a set
resolution.
• Building puppets with vector layers adds an extra step: they
need to be precomposed before rigging.

16. Motion Capture
Recording with KinectToPin

17. Setting up your space to record
6 – 10 feet

18. Capturing
• Record 15-point 3D skeletal
tracking data
• Capture in sync to dialogue
• Remote control with a standard
presentation remote
• Open source and cross-platform:
runs on OpenNI, built in
Processing, also works with
generic sensors like the Asus
Xtion

20. Acting for Kinect
• The Kinect can’t record what it can’t see.
• Keep your entire body in the camera’s range
• Face the camera when capturing
• Watch out for occluded joints (sitting down, putting your hands behind
your back etc. can cause it to lose the track)
• It’s surprisingly good at picking up subtle posture shifts, but not
smaller motions like laughter.
• Use a different performer for each character if possible

21. Data formats for export
KinectToPin can output quite a few data, but the ones we’ll be
using today are:
• XML (Native recording format, no export required)
• After Effects 3D Point Controls (CS5.5+)
• After Effects 2D Point Controls (CS4+)
Tip: Edit KinectToPin’s settings.txt file to choose your output
formats.

22. Lunch.

23. Importing tracking data
Native import or copy & paste

24. Meet the KinectToPin UI Panel.
• Create character
templates
• 2D and 3D setups
• Native XML import
(SLOW)
• Automatically add
expressions to smooth
tracking data
• Automate rigging

25. The Puppet Tool
THREE OR MORE PINS ON ONE LAYER: TWO PINS EACH, MULTIPLE LAYERS:
Elbows don’t bend this way. Ow. Rotation with stretch and squash!

26. AE Puppet best practices
• Work BIG. Start high-res, scale down in the puppet comp, then
scale back up in the project comp (enable Collapse
Transformations).
• Precompose if you’re using vectors or shape layers
• Keep multiple angles of the same character in sync
High-res
Start big. Character
Layers
Project Comp
End big.
Rigging
Template

27. Character Rigging
Using the KinectToPin UI panel

28. Retargeting for arbitrary character
shapes

29. It’s easy!
Just grab a red control null and move it to the appropriate
position.

30. Quick auto-lip sync
1. Select audio layer and choose
“Convert Audio to Keyframes.”

31. Quick auto-lip sync
2. Alt-click the stopwatch to add an expression. Use the
pickwhip to connect the audio keyframes to the mouth animation.

32. Quick auto-lip sync
3. Size the audio data to fit. Use this expression:
audio = [your audio keyframes];
linear(audio,sourceMin,sourceMax,targetMin,targetMax)
Lowest value of source data
Highest value of source data
Closed mouth (probably 0)
Widest possible open mouth

33. One last thought:
Embrace the limitations of
the format.

34. Further non-K2P tools
FREE COMMERCIAL
Brekel Kinect iPi Soft
Open-source capture for 3D Uses two Kinects to record
animation (Windows-only) www.ipisoft.com
www.brekel.com
NI Mate
Duik
Kinect for OSC/MIDI/Blender
Inverse kinematics plugin for AE
www.ni-mate.com
www.duduf.net

35. Get in Touch
Victoria Nece
www.victorianece.com
@FakeGreenDress