The document discusses the evolution of DreamWorks Animation's production pipeline, highlighting the transition from a monolithic architecture to a microservices-based approach to enhance scalability and efficiency. Key technologies adopted include Spring Boot, Spring Data, and various databases, with an emphasis on agility and cloud-native solutions. Lessons learned stress the importance of generalists, collaboration, and not fearing change as they adapt to new technologies.

1. Dragon2
Technical Directors
November 15, 2013
How to Train Your
Microservice

2. DOUG SHERMAN
Principal Engineer,
DreamWorks Animation

3. FAMILY
ENTERTAINMENT
COMPANY
focused on telling
amazing stories
through beloved characters
and state-of-the-art
computer graphics imagery

4. BY THE NUMBERS

6. • more ambitious than the last
• increased visual richness
MORE
• complex characters
• detailed environments
and storylines
• cinematography
and lighting
• more compelling:
UNBOUNDED CREATIVE AMBITION

13. STORY

14. LAYOUT

15. ANIMATION

16. LIGHTING

18. • 90 minutes: 130,000 individual frames
to produce
one film
4YRS
• each frame: 100’s of assets and elements
• each character: 1,000’s of control points(+)
500,000,000over digital
files
PRODUCTION PROCESS

19. DREAMWORKS ANIMATION
is a DIGITAL MANUFACTURER
Creating high value in a high-end
digital product that is distributed and
consumed worldwide

20. • 75,000,000 CPU hours
• 10,000+ cores
• 200+ terabytes of data
• 500,000,000 files
• 250,000,000,000 pixels
IN JUST ONE FILM…

22. Where did we start?
• One show released every
few years
• Each show typically had
their own pipeline
• C++, Perl scripts (later
Python) were the norm
• There was one lone Java
application with Swing
• Workflows were very
client-heavy

23. How did that go?
• Was hard to monitor
• Where are these things
running?
• Risky to make changes to
monolithic code
• Assuming devs were
still around
• Difficult to scale
• Some software couldn’t
work between sites

24. Where did we go next?
• More Movies perYear == New Architecture
• New Architecture == Adoption Challenges
• SOA & Microservices introduced
• SOAP… then REST
• C++/Python... then Java
• JEE… then Spring

25. Transitions
• SOA felt overwhelming at the start
• SOAP, XML, ESB, BPEL,App Servers
• Java wins for sure… and Spring helped
• Microservices via REST / JSON easier
• Spring leaned on for REST, DBs,Testing
• Spring Messaging… was... interesting

26. Transition for Clients
• Clients provided language specific wrappers
• Pros
• Clients could call upon a familiar API
• Cons
• Clients could call upon a familiar API
• Education and experience helped

27. Ultimately, how did it all go?
• Foreign architecture required “Social Engineering”
• Encouraged client & server developers
• Things improved with new Platform & QA teams
• Microservices scaled to meet multi-show demand
• Optimal databases could be used per service
• Smaller teams + smaller targets = agility

28. What are we doing now?
• Spring Boot
• Convention over configuration
• Where are my beans???!
• “Make Jar, not War” – Josh Long
• JEE App Servers on way out
• Actuator … yes, please!
• But what about “other” services
in other languages?
• http://spring.io has great examples
• Move everything to Spring Boot?

29. What else are we doing now?
• Spring Data
• Simple… maybe too simple?
• Supports many of our DB choices
• Oracle/PostgreSQL
• Cassandra
• MongoDB
• ElasticSearch
• Move everything to Spring Data?!
• Spring Cloud
• Configuration for starters

30. Lessons learned so far?
• Generalists are key to our success
• So many frameworks
• So many databases
• So many containers
• So many languages
• Social Engineering is important
• Take time to share ideas
• Build with your consumers
• But offer opinions
• Keep your vendors honest!

31. Spring specific lessons learned
• Don’t fear change
• Using XML config in Java??!
• Stop using XML config in Java?
• Spring Boot is wonderfully
different
• Don’t settle
• Spring Boot, Spring Data and other
packages may not offer it all
• Leverage the experts when you can
• Do more with less

32. Where are we going next?
• Going Cloud-Native
• Exposing more RESTful APIs
• Embracing the container
• Trying to make CI / CD a reality

33. Questions?
Open Source communication
“Let’s do lunch!”
doug.sherman@dreamworks.com