Video: https://www.youtube.com/watch?v=OGjaXtauO-c Jozef will talk about building a new generation of distributed services framework called Amanda at MPC. MPC is a leading visual FX company that works with extremely high volumes of data and a vast array of computing power. This talk gives a behind the scenes (sorry) look at how art assets get processed at scale. And like all good talks it includes plenty of metrics!

1. Providing scalability for
Pirates, Lizards and
Zombies.
August 2014
Amanda&
Pla)orm&for&distributed&services

2. Who am I
Software developer at MPC
!
Python since 2009
!
Loves services
!
Obsessed by monitoring
!
@JozefVanE on twitter

3. Who is MPC
We create Visual Effects
!
»Advertising
»Feature films

4. We create Visual Effects
!
»Across 8 sites
»Fully integrated cross
site pipeline

5. What are visual effects?
Visual effects involve the integration of live-action footage
and generated imagery to create environments which look
realistic, but would be dangerous, expensive, impractical, or
simply impossible to capture on film.
(source wikipedia)

6. What are visual effects?

7. What are visual effects?

8. What are visual effects?

9. What are visual effects?

10. Production workflow.

11. Production considerations
» Management
» Assets, Production, Users
» Various sources
» Used by:
» In house tools
» 3rd party applications
» Multiple environments
» Users
» Artists
» Developers

12. MPC’s approach - Amanda
» Service based architecture
» Platform as a service
» Multi protocol, transport and concurrency
» Ecosystem to write a service for developers of any level
» 1st generation 2007
» 2nd generation 2012-2013

13. Some statistics.
!
» Godzilla
» 450 years if rendered on a single machine

14. Some statistics.
!
» Godzilla
» 450 years if rendered on a single machine
» Or, 15 years after Henry 8th died

15. Some statistics.
!
» Godzilla
» 650 TB of data

16. Some statistics.
!
» Godzilla
» 650 TB of data
» Or, 41000 mobile phones with 16GB drives.

17. Some statistics.
!
» Godzilla
» 250.000 Amanda req/min.

18. Some statistics.
!
» Godzilla
» 250.000 Amanda req/min.
» Or, 4 times Germany VS Brazil tweets

19. Amanda - building the platform

20. Amanda - building the platform

21. Amanda - building the platform
Services
class MakeMovieService(object):!
!
@public!
def greet_director(self, director_name):!
return "hello %s" % (director_name,)!

22. Amanda - building the platform
We have a service, but lets make it do something useful

23. Amanda - building the platform
Infrastructures
class MakeMovieService(object):!
storage = Dependency(“storage")!
def _check_data_on_disk(self, files):!
""" Check data is available!
"""!
return self.storage.check_exists(files)!

24. Amanda - building the platform
Infrastructures
class MakeMovieService(object):!
storage = Dependency(“storage")!
def _check_data_on_disk(self, files):!
""" Check data is available!
"""!
return self.storage.check_exists(files)!
def _get_shot(self, shot):!
""" Get shot data!
"""!
query = "<insert db query>"!
return self._db.execute(query, (shot,))!
!

25. Amanda - building the platform
Infrastructures
class MakeMovieService(object):!
storage = Dependency(“storage")!
def _check_data_on_disk(self, files):!
""" Check data is available!
"""!
return self.storage.check_exists(files)!
def _get_shot(self, shot):!
""" Get shot data!
"""!
query = "<insert db query>"!
return self._db.execute(query, (shot,))!
!
@public!
def make_movie_magic(self, shotInfo):!
""" Do some magic!
"""!
shot = self._get_shot(shotInfo)!
if not self._check_data_on_disk(shot["files"]):!
self._log.error(‘files not found')!
raise DataNotAvailable()!
self._do_magic(shot["files"], self._config.production.jobsite())

26. Amanda - building the platform
Ok now it is doing something but that is one user,
one service and not very useful at scale…

27. Amanda - building the platform

28. Amanda - building the platform
Service providers
services = ServiceProvider(!
make_movies=MakeMovieService,!
storage=StorageService!
config=ConfigService!
logging=LoggingService!
__infrastructure__={!
"_db": DbInfrastructure,!
"_config": ConfigInfrastructure,!
"_xmesh": XmeshInfrastructure,!
"_log": LoggingInfrastructure!
}!
)!
services.make_movies.make_movie_magic("shot1")!
services.logging.change_logging_level("make_movies", "debug")!

29. Amanda - building the platform

30. Amanda - building the platform
Proxies
class Proxy(object):!
!
def __getattr__(self, method):!
def method_stand_in(*args, **kwargs):!
return self._transport.publish(self._service, method, *args, **kwargs)!
return method_stand_in!
!
services = ServiceProvider(!
make_movies=Proxy("make_movies", transport)!
)!
services.make_movies.do_movie_magic(“shot1”)!
!

31. Amanda - building the platform

32. Amanda - building the platform
Ok, at this point we can scale a bit but what about
concurrency/parallel processing?

33. Amanda - building the platform

34. Amanda - building the platform

35. Amanda - building the platform

36. Amanda - building the platform
This gives us all the building blocks to assemble a
distributed system with the possibility to use the
best technologies for the given tasks

37. Amanda - assembling & maintaining the platform

38. Amanda - assembling & maintaining the platform
Setting up for a cross language pipeline.

39. Amanda - assembling & maintaining the platform

40. Amanda - assembling & maintaining the platform

41. Amanda - assembling & maintaining the platform
Setting up for a cross language, fault tolerant
pipeline.

42. Amanda - assembling & maintaining the platform
Setting up for cross language, fault tolerant,
monitorable pipeline.

43. Amanda - assembling & maintaining the platform
request: userAdmin, getUserIdByUserName, [u'jozef'], {}
request id: 1393351037.307669-userAdmin-staffdirectory-3500,
by jozef from staffdirectory
exception: <type 'NoneType'>, None
handled by: amanda-host-1_28368-GeventConcurrencyPool_worker_37
stats:
sent on Tue Feb 25 17:57:17 2014
received on Tue Feb 25 17:57:17 2014
frontend->backend delivery took 0.003s`
internal queuing took 0.0000015s
request execution took 0.005s
total time: 0.011s

44. Amanda - assembling & maintaining the platform
internal
queuing:
0.0000015s
frontend->backend: 0.003s
total time: 0.011s
request
execution:
0.005s

45. Amanda - assembling & maintaining the platform
» Maintenance.
» Salt
» Extended with an Amanda module.
» Salt client wrapped inside a service so we can access it from anywhere not
just the master node.
» Predefined, repeatable configuration.

46. Amanda
» Adaptable
» Extendable
» Configurable
» Best of breeds

47. Amanda
» Looking ahead
» Containerisation
» Auto scaling
» Open sourcing possibilities

48. In Production
Montréal

49. We’re Now Hiring!
!
Opportunities across all our Studios and across all
disciplines
!
Check out www.moving-picture.com for vacancies
!
Or email recruitment@moving-picture.com

50. Amanda
Thank you, any questions?
Jozef van Eenbergen - twitter: @JozefVanE