Distributed
   bulk rendering
on the Amazon cloud

           Dr. Raphael Volz
  nogago GmbH, Gilching, Germany
State of t...
Another 6 years into the project, the
      planet will have 5 TB (compressed).
                                          ...
OSM is power law distributed with
     top 10% covering only 1 ‰ of planet
                                          Obser...
Lower barriers of entry and increase the
            number of eyes !
                             How to achieve the 5 TB...
Ride the smartphone wave !
                                                Smartphones in use (in Mio.)




Source: Nogago...
For example nogago outdoor is a smartphone app
    that accompanies your outdoor activities




          Plan            ...
Our customers will download offline maps
          for regions of choice

                                                ...
Generating offline maps takes
                        significant time and space
                                 Using th...
Basic approach to parallelization using
               Amazon cloud
  Render
  Image                                      ...
To make things fit consider corner cases

                                                                     Example:
  ...
Optimization: Do not always get planet
       Prepare chunks periodically
• Two step approach, first:
    – Download plane...
What have we learned ?

                         Rendering the world on
                              zoom level 15

     ...
EU FP7                               Thank
 project partner                            you !
   sought for
mapping-related...
Upcoming SlideShare
Loading in …5
×

nogago distributed bulk rendering

1,791 views
1,651 views

Published on

Presentation at State of the Map 2010 by nogago.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,791
On SlideShare
0
From Embeds
0
Number of Embeds
167
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

nogago distributed bulk rendering

  1. 1. Distributed bulk rendering on the Amazon cloud Dr. Raphael Volz nogago GmbH, Gilching, Germany State of the map, Girona, July 9, 2010
  2. 2. Another 6 years into the project, the planet will have 5 TB (compressed). Hypothesis July 2010 10 GB July 2016 5 TB Distributed bulk rendering on the Amazon cloud
  3. 3. OSM is power law distributed with top 10% covering only 1 ‰ of planet Observation Top 100 6°x6° areas Size (Bytes GZ-compressed) # 1 / 1 GB West 6 South 48 # 2 / 0,6 GB West -84 South 30 6x6 splits by size Distributed bulk rendering on the Amazon cloud
  4. 4. Lower barriers of entry and increase the number of eyes ! How to achieve the 5 TB planet Distributed bulk rendering on the Amazon cloud
  5. 5. Ride the smartphone wave ! Smartphones in use (in Mio.) Source: Nogago analysis of Juniper Research, Gartner Spring 2009 forecasts Note: Assumption: Devices sold are in use for two years Distributed bulk rendering on the Amazon cloud
  6. 6. For example nogago outdoor is a smartphone app that accompanies your outdoor activities Plan Share activities experiences • View maps and • Download guides anytime • View tracks Maps and offline • View photos • Choose from • Record and (geo-referenced) 400.000 tracks follow Tracks • Share with • Plan new family, friends tracks Enjoy the or the world ! Outdoors ! Distributed bulk rendering on the Amazon cloud
  7. 7. Our customers will download offline maps for regions of choice Raster map POI DB Ways DB Max. 200 MB per 1°x1° Distributed bulk rendering on the Amazon cloud
  8. 8. Generating offline maps takes significant time and space Using the OSM map rendering tool chain… Get Load into Render XML planet PNG Database Maps osm.bz2 wget osm2pgsql mapnik … takes space and time … Zoom Level 15 1 Billion 5 621 (1:14000 @eq.) Tiles TB Days Zoom Level 18 68 Billion 350 109 (1:1700 @eq.) Tiles TB Years Assumptions: 16 KB per Tile on average, 20 Tiles per Second to render on modern hardware Distributed bulk rendering on the Amazon cloud
  9. 9. Basic approach to parallelization using Amazon cloud Render Image Nogago.com Ubuntu 10.04 Start Msg. Msg. Shut off Apache Instance started finished instance Postgres 8.4 User Amazon EC2 osm2pgsql Data osmosis mapnik Render Image TimKay‘s aws Get planet Load into Render Store Database Maps mapnik-OSM osm.bz2 on S3 Nogago tools Amazon EC2 runs images Rarely Changing Amazon S3 Content stores files Total: 3 GB Amazon S3 Note: UD = User Data, thanks to Eric Hammond (http://ec2-run-user-data.notlong.com) Distributed bulk rendering on the Amazon cloud
  10. 10. To make things fit consider corner cases Example: 1 x1 to be rendered at zoom level 10 1,352 x 1,352 to be loaded into database 190% overhead at zoom level 10 Data to load into Postgres Area to be rendered (depends on zoom levels) by mapnik Distributed bulk rendering on the Amazon cloud
  11. 11. Optimization: Do not always get planet Prepare chunks periodically • Two step approach, first: – Download planet, periodically 1 Day to process – Split into chunks and copy on S3 – Store chunks on EC2 • Then, many rendering images – Get appropriate chunk from S3 Max. – Load (parts of) chunk into Postgres 3 hours per – Render from Postgres 1 x1 Distributed bulk rendering on the Amazon cloud
  12. 12. What have we learned ? Rendering the world on zoom level 15 in 28 hours instead of 621 days ( 2 ‰ of the time) By distributing bulk rendering on the Amazon cloud Distributed bulk rendering on the Amazon cloud
  13. 13. EU FP7 Thank project partner you ! sought for mapping-related project proposal Distributed bulk rendering on the Amazon cloud

×