apps on maps...
Building the World with Elastic
Map Reduce
Oliver Norton, Technical Director
Tim Jenks, Technical Lead
appsonmaps.com
AS3 SDK – Apps in Browser
JS API – Embed on your website
IOS SDK – Mobile, coming Q4 2012
social media
social commerce
traffic updates
journey planning
worldflightclub.com
flying like a bird
World Flight Club on YouTube
flipping the bird!
Flipper on YouTube
photographic maps
Photographic-based maps ….
layered data
We fuse layered data to procedurally
generate our maps
(using AWS’ Elastic Map Reduce)
streamed
All built & served from off-the-shelf Amazon Web Service
infrastructure
pipeline
data size
Over 2TB Data
Terrain:
GB (10m, ¼ million Km2)
US (10m, 40x GB)
Buildings :
GB (full coverage)
US (120 cities)
R...
before
• Limited scalability -> 60 desktop spec machines
• Multi-TB SAN with a £10k/year maintenance cost
• In house build...
now
On Amazon Elastic Map Reduce
now
• Scalability -> 800 m1.large instances
• Off shelf tech that’s discoverable (hadoop, MRJob)
• Maintenance reduced
• D...
pipeline
mrjob
• MRjob from Yelp
• http://github.com/Yelp/mrjob
800 machines in 20lines
class MyMapReduceJob(MRJob):
def mapper_init(self):
self.__mapper = # wire up mapper
def mapper(se...
amazon emr
Processing Complexity X Data Size
thanks
appsonmaps.com
Twitter: @eeGeo
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Building the world with Elastic Map Reduce
Upcoming SlideShare
Loading in...5
×

Building the world with Elastic Map Reduce

167

Published on

Talk given by Oliver Norton and Tim Jenks at AWS for the Gaming Industry November 2012

Terrabyte geometry processing to build a 3D interactive model of the real world from GIS data.

https://aws.amazon.com/campaigns/aws-for-gaming-london/

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
167
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Building the world with Elastic Map Reduce

  1. 1. apps on maps...
  2. 2. Building the World with Elastic Map Reduce Oliver Norton, Technical Director Tim Jenks, Technical Lead
  3. 3. appsonmaps.com AS3 SDK – Apps in Browser JS API – Embed on your website IOS SDK – Mobile, coming Q4 2012
  4. 4. social media
  5. 5. social commerce
  6. 6. traffic updates
  7. 7. journey planning
  8. 8. worldflightclub.com flying like a bird World Flight Club on YouTube
  9. 9. flipping the bird! Flipper on YouTube
  10. 10. photographic maps Photographic-based maps ….
  11. 11. layered data We fuse layered data to procedurally generate our maps (using AWS’ Elastic Map Reduce)
  12. 12. streamed All built & served from off-the-shelf Amazon Web Service infrastructure
  13. 13. pipeline
  14. 14. data size Over 2TB Data Terrain: GB (10m, ¼ million Km2) US (10m, 40x GB) Buildings : GB (full coverage) US (120 cities) Roads: GB (¼ million miles) US (4 million miles) Processing this can start to be expensive $$$
  15. 15. before • Limited scalability -> 60 desktop spec machines • Multi-TB SAN with a £10k/year maintenance cost • In house build that needed maintaining • 10mbit/sec symmetric internet to upload TBs of data • 3 developers knew how to run builds • Electric costs -> who knows…
  16. 16. now On Amazon Elastic Map Reduce
  17. 17. now • Scalability -> 800 m1.large instances • Off shelf tech that’s discoverable (hadoop, MRJob) • Maintenance reduced • Data is already in cloud (source, and destination) • More predictable costs (and happier costs, with spot pricing) • DevOps benefits: Now any engineer can write and run jobs, not just 3
  18. 18. pipeline
  19. 19. mrjob • MRjob from Yelp • http://github.com/Yelp/mrjob
  20. 20. 800 machines in 20lines class MyMapReduceJob(MRJob): def mapper_init(self): self.__mapper = # wire up mapper def mapper(self, key, line): # perform map work for key, value in self.__mapper.map(line, None): yield str(key), value def reducer_init(self): self.__reducer = # wire up reducer def reducer(self, key, values): # perform reduce work result = self.__reducer.reduce(key, values) if result: yield key, "built successfully" else: yield key, "failed" if __name__ == '__main__': MyMapReduceJob.run()
  21. 21. amazon emr Processing Complexity X Data Size
  22. 22. thanks appsonmaps.com Twitter: @eeGeo
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×