Building the world with Elastic Map Reduce

328 views

Published on

Talk given by Oliver Norton and Tim Jenks at AWS for the Gaming Industry November 2012

Terrabyte geometry processing to build a 3D interactive model of the real world from GIS data.

https://aws.amazon.com/campaigns/aws-for-gaming-london/

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Building the world with Elastic Map Reduce

  1. 1. apps on maps...
  2. 2. Building the World with Elastic Map Reduce Oliver Norton, Technical Director Tim Jenks, Technical Lead
  3. 3. appsonmaps.com AS3 SDK – Apps in Browser JS API – Embed on your website IOS SDK – Mobile, coming Q4 2012
  4. 4. social media
  5. 5. social commerce
  6. 6. traffic updates
  7. 7. journey planning
  8. 8. worldflightclub.com flying like a bird World Flight Club on YouTube
  9. 9. flipping the bird! Flipper on YouTube
  10. 10. photographic maps Photographic-based maps ….
  11. 11. layered data We fuse layered data to procedurally generate our maps (using AWS’ Elastic Map Reduce)
  12. 12. streamed All built & served from off-the-shelf Amazon Web Service infrastructure
  13. 13. pipeline
  14. 14. data size Over 2TB Data Terrain: GB (10m, ¼ million Km2) US (10m, 40x GB) Buildings : GB (full coverage) US (120 cities) Roads: GB (¼ million miles) US (4 million miles) Processing this can start to be expensive $$$
  15. 15. before • Limited scalability -> 60 desktop spec machines • Multi-TB SAN with a £10k/year maintenance cost • In house build that needed maintaining • 10mbit/sec symmetric internet to upload TBs of data • 3 developers knew how to run builds • Electric costs -> who knows…
  16. 16. now On Amazon Elastic Map Reduce
  17. 17. now • Scalability -> 800 m1.large instances • Off shelf tech that’s discoverable (hadoop, MRJob) • Maintenance reduced • Data is already in cloud (source, and destination) • More predictable costs (and happier costs, with spot pricing) • DevOps benefits: Now any engineer can write and run jobs, not just 3
  18. 18. pipeline
  19. 19. mrjob • MRjob from Yelp • http://github.com/Yelp/mrjob
  20. 20. 800 machines in 20lines class MyMapReduceJob(MRJob): def mapper_init(self): self.__mapper = # wire up mapper def mapper(self, key, line): # perform map work for key, value in self.__mapper.map(line, None): yield str(key), value def reducer_init(self): self.__reducer = # wire up reducer def reducer(self, key, values): # perform reduce work result = self.__reducer.reduce(key, values) if result: yield key, "built successfully" else: yield key, "failed" if __name__ == '__main__': MyMapReduceJob.run()
  21. 21. amazon emr Processing Complexity X Data Size
  22. 22. thanks appsonmaps.com Twitter: @eeGeo

×