Social media has saturated our lives (for better or worse) and redefined the ways people communicate. The rules for how we tie these interactions into stories on screen, however, are still being written. From House of Cards to Searching, The Circle to The Social Dilemma, the evolution of characters on screen interacting with their digital world has moved far beyond the clunky phone screen cutaways of the early digital era. Augmented reality and spatialized computing concepts unlock a whole new level of storytelling for filmmakers, where their characters can communicate in seamless ways that keep the audience immersed in their worlds and feel more subjectively true to life. Learn best practices for visualizing our digital world in innovative ways, from script to screen and beyond.

1. AUGMENTED STORYTELLING
SOCIAL MEDIA ON SCREEN
An Emergent Language in Film and Television

2. Inspiration

3. Panel Outline
• Historical Overview - Digital Interaction Portrayed on Screen from the Desktop through the Mobile Era
• Trailblazers: The Net, House of Cards, Sherlock, Stranger Than Fiction, You, The Circle, Cat
fi
sh, Searching, Unfriended, etc
• Creative Breakdown
• Unique Challenges of Mobile Digital Interaction
• Phone Screen Cutaways vs. Pop-Up Graphics
• Pros & Cons of Each Approach
• Dembanger Creative Direction = Pop-Up Augmented Reality Overlays within Environments
• Integration into the fabric of 3D Space
• Planning within a scene & blending into the environment
• Real-Time Actor/Character-Driven Interactions
• Digital interaction as a new graphical layer overlaid on a visual medium to deepen characters
• Story decisions captured on set by actors actions, but malleable through the end of post
• Technical Overview
• On Set Production Process - Dedicated Supervisor for Live Screen Recordings, Actors Interactions Captured for Each Take
• Post-Production Process - VFX Breakdowns, Interface Design Iteration, After E
ff
ects Scripting
• Future Roadmap
• On-Set Visualization & Live Compositing of Phone Interactions using Real-Time Game Engines - Virtual Production
• Automated Pipeline for Production Assets to Post-VFX
• Reuse of on-set social media assets for Marketing & Experiential Advertising

4. INITIATION Case Study
The film centers on a social media game taken too far — phone interaction was crucial to the story and the audience’s understanding of the characters
With 150+ graphical overlays throughout the film, we needed to develop an innovative pipeline for
seamlessly translating the on-set interactions to our remote post-production and VFX teams in Australia.

5. Social Media Pipeline (beta)
Pre-Production Production Post-Production
Phone Interaction Script Breakdown
Character Account Creation
Character Phone Setup
Virtual Production Supervisor
Handles Phone Prep for each Scene
Actors Perform Real Phone Interactions During Takes
Data Captured as Screen Recordings + Raw Assets
Screen Recordings Synched
with Dailies for Editorial
Raw Photo/Social Media Assets
Synched with Cloud for VFX Team
+

6. Sleek Modern Aesthetic
Monochromatic Tones
Minimalist UI to Forefront the Content
CREATIVE
DIRECTION

7. Framing in Negative Space
CREATIVE
DIRECTION

8. Frameless 3D Depth Layers
Foreground + Background Elements
Individual Pieces of Content with different Z-Depth
CREATIVE
DIRECTION

9. Aligned with Environment Planes
CREATIVE
DIRECTION

10. Fluid Motion
CREATIVE
DIRECTION

11. Organization
Account
Globals
Shot
Breakdown
Cloud Access to
Screen Recordings
& GFX Assets
TECHNICAL
EXECUTION

12. On-Set Capture
Virtual Production Supervisor
Managed Phones, Character Accounts,
and Screen Recordings for each Take
TECHNICAL
EXECUTION
Actors performed Phone Interactions
in Real-Time with Real (Private)
Social Media Accounts
Virtual Production Team
Uploaded Screen Recordings
& Digital Assets to Cloud

13. Design Evolution
TECHNICAL
EXECUTION
Iterative process from 2D Layout to 3D Tracking to pare down UI elements, forefront visuals and reinforce characters in audience’s awareness

14. Content Evolution
TECHNICAL
EXECUTION
Flexibility to adjust content and tighten web of connections and dynamics between characters over several Work-In-Progress test screenings

15. Content Evolution
TECHNICAL
EXECUTION
Freedom to ramp up the emotion of pivotal moments.
Here we progressively added more and more graphics to highlight the character’s sense of overwhelm.

16. Final Integration
On-Set Screen Capture:
TECHNICAL
EXECUTION
Final VFX Shot:

17. Final Integration
TECHNICAL
EXECUTION
Reuse of Media Generated During On-Set Phone Capture:

18. Final Integration
TECHNICAL
EXECUTION
Use of Z-Space to add Depth:

19. Final Integration
TECHNICAL
EXECUTION
Fluid Motion and Real-Time Popups for Breaking News Coverage:

20. Best Practices
• Dedicated supervisor and synched screen recording on set allows
Editorial and VFX departments to assemble the film with easy
graphical reference and character interactions clear
• Saves time and money in VFX spotting and construction of
elements from scratch, reduces cycles to final graphics
• Frame with negative space in mind
• Shoot with a fast shutter angle and markers (if necessary) for easier
3D tracking
• Find horizon lines within each shot and make graphical elements
parallel or perpendicular to planes of environment for more natural
integration
• Motion Blur, Semi-Transparency and DOF help further integrate with
environment
TECHNICAL
EXECUTION

21. Future Roadmap
TECHNICAL
EXECUTION
Real-Time Compositing
On-Set Visualization of Character Phone Interactions Through
Social Media APIs Using Tracked Camera + Game Engine
Automated Ingest/Transcoding
Upload of Social Media APIs & Screen Recordings
to Cloud Asset Database for Instant Remote Access