Video: https://www.youtube.com/watch?v=OGjaXtauO-c
Jozef will talk about building a new generation of distributed services framework called Amanda at MPC. MPC is a leading visual FX company that works with extremely high volumes of data and a vast array of computing power. This talk gives a behind the scenes (sorry) look at how art assets get processed at scale. And like all good talks it includes plenty of metrics!
Top 10 Most Downloaded Games on Play Store in 2024
Jozef Ve Providing Scalability for Pirates, Lizards and Zombies at #DOXLON
1. Providing scalability for
Pirates, Lizards and
Zombies.
August 2014
Amanda&
Pla)orm&for&distributed&services
2. Who am I
Software developer at MPC
!
Python since 2009
!
Loves services
!
Obsessed by monitoring
!
@JozefVanE on twitter
3. Who is MPC
We create Visual Effects
!
»Advertising
»Feature films
4. We create Visual Effects
!
»Across 8 sites
»Fully integrated cross
site pipeline
5. What are visual effects?
Visual effects involve the integration of live-action footage
and generated imagery to create environments which look
realistic, but would be dangerous, expensive, impractical, or
simply impossible to capture on film.
(source wikipedia)
11. Production considerations
» Management
» Assets, Production, Users
» Various sources
» Used by:
» In house tools
» 3rd party applications
» Multiple environments
» Users
» Artists
» Developers
12. MPC’s approach - Amanda
» Service based architecture
» Platform as a service
» Multi protocol, transport and concurrency
» Ecosystem to write a service for developers of any level
» 1st generation 2007
» 2nd generation 2012-2013
13. Some statistics.
!
» Godzilla
» 450 years if rendered on a single machine
14. Some statistics.
!
» Godzilla
» 450 years if rendered on a single machine
» Or, 15 years after Henry 8th died
21. Amanda - building the platform
Services
class MakeMovieService(object):!
!
@public!
def greet_director(self, director_name):!
return "hello %s" % (director_name,)!
22. Amanda - building the platform
We have a service, but lets make it do something useful
23. Amanda - building the platform
Infrastructures
class MakeMovieService(object):!
storage = Dependency(“storage")!
def _check_data_on_disk(self, files):!
""" Check data is available!
"""!
return self.storage.check_exists(files)!
24. Amanda - building the platform
Infrastructures
class MakeMovieService(object):!
storage = Dependency(“storage")!
def _check_data_on_disk(self, files):!
""" Check data is available!
"""!
return self.storage.check_exists(files)!
def _get_shot(self, shot):!
""" Get shot data!
"""!
query = "<insert db query>"!
return self._db.execute(query, (shot,))!
!
25. Amanda - building the platform
Infrastructures
class MakeMovieService(object):!
storage = Dependency(“storage")!
def _check_data_on_disk(self, files):!
""" Check data is available!
"""!
return self.storage.check_exists(files)!
def _get_shot(self, shot):!
""" Get shot data!
"""!
query = "<insert db query>"!
return self._db.execute(query, (shot,))!
!
@public!
def make_movie_magic(self, shotInfo):!
""" Do some magic!
"""!
shot = self._get_shot(shotInfo)!
if not self._check_data_on_disk(shot["files"]):!
self._log.error(‘files not found')!
raise DataNotAvailable()!
self._do_magic(shot["files"], self._config.production.jobsite())
26. Amanda - building the platform
Ok now it is doing something but that is one user,
one service and not very useful at scale…
36. Amanda - building the platform
This gives us all the building blocks to assemble a
distributed system with the possibility to use the
best technologies for the given tasks
41. Amanda - assembling & maintaining the platform
Setting up for a cross language, fault tolerant
pipeline.
42. Amanda - assembling & maintaining the platform
Setting up for cross language, fault tolerant,
monitorable pipeline.
43. Amanda - assembling & maintaining the platform
request: userAdmin, getUserIdByUserName, [u'jozef'], {}
request id: 1393351037.307669-userAdmin-staffdirectory-3500,
by jozef from staffdirectory
exception: <type 'NoneType'>, None
handled by: amanda-host-1_28368-GeventConcurrencyPool_worker_37
stats:
sent on Tue Feb 25 17:57:17 2014
received on Tue Feb 25 17:57:17 2014
frontend->backend delivery took 0.003s`
internal queuing took 0.0000015s
request execution took 0.005s
total time: 0.011s
44. Amanda - assembling & maintaining the platform
internal
queuing:
0.0000015s
frontend->backend: 0.003s
total time: 0.011s
request
execution:
0.005s
45. Amanda - assembling & maintaining the platform
» Maintenance.
» Salt
» Extended with an Amanda module.
» Salt client wrapped inside a service so we can access it from anywhere not
just the master node.
» Predefined, repeatable configuration.
49. We’re Now Hiring!
!
Opportunities across all our Studios and across all
disciplines
!
Check out www.moving-picture.com for vacancies
!
Or email recruitment@moving-picture.com
50. Amanda
Thank you, any questions?
Jozef van Eenbergen - twitter: @JozefVanE