Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Mappy-ing Open Source
09/21/2013, FOSS4G2013
@AudreyMalherbe
Audrey Malherbe
Head of geo platform
at Mappy
audrey.malherbe@mappy.com
367 millions
route requests in 2012
4 millions
referenced business
Mappy, a brand of SoLocal Group
About 10 millions
month...
Feedback on the overhaul of
the mapping engine of Mappy
around Mapnik and PostGIS
Plan
The path
Our journey
Feedback
– Les enjeux du projet
The path
The path – starting
point
Background
– Map processing for 16 years with TomTom/AND databases
– Maps are used by Mappy & Fr...
Geographic data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
Data processing
Mostly pre-...
The path - goals
– Create a map design able to support content
– Be ready to release with the new version of our website
f...
Take a Deep Breath.
Start Again.
Geographic
data
API AJAX
NewMappy
SDK Mobile
New Mappy
KyotoCabinetKyotoCabinet
New platform
in production
Reduced Prepara...
– Le chemin
Our journey
Our journey - Starting point
– Heterogeneous knowledge of the team
on open source technologies
– Training & getting help f...
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobil...
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobil...
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobil...
Geographic
data
SDK Mobile
Mappy
API AJAX
Mappy
Stack Mappy
Engine
Templates
Stack Mappy
Cache
API AJAX
NewMappy
SDK Mobil...
Our journey – key points
– Be able to challenge historical choices
– Reach a better consensus with our customers (Web/Mobi...
Our journey – technical choices
– PostGIS: transactional database/scalable/standard compliant
– Mapnik : approved by marke...
Our journey –
Contribution to Mapnik
– Mappy was legitimate to participate in an open source project
– Develop an open sou...
Feedback
Feedback– Key figures
– Production platform handles 85% of the load (All media except
Mobile YP)
– 14 servers:
– Varnish :...
Feedback… team
– Positive
– Team motivation, proud and in control of our new platform
– PostGIS and Mapnik tools
– Contrib...
Thank
you
@AudreyMalherbe https://github.com/Mappy
Upcoming SlideShare
Loading in …5
×

Foss4G2013 Mappy-ing Open Source

3,080 views

Published on

Feedback on our overhaul of map engine around Mapnik and PostGIS

Published in: Technology, Education
  • Be the first to comment

Foss4G2013 Mappy-ing Open Source

  1. 1. Mappy-ing Open Source 09/21/2013, FOSS4G2013
  2. 2. @AudreyMalherbe Audrey Malherbe Head of geo platform at Mappy audrey.malherbe@mappy.com
  3. 3. 367 millions route requests in 2012 4 millions referenced business Mappy, a brand of SoLocal Group About 10 millions monthly users 27 millions monthly visits Provide route planner and local services on the web and mobiles devices
  4. 4. Feedback on the overhaul of the mapping engine of Mappy around Mapnik and PostGIS
  5. 5. Plan The path Our journey Feedback
  6. 6. – Les enjeux du projet The path
  7. 7. The path – starting point Background – Map processing for 16 years with TomTom/AND databases – Maps are used by Mappy & French YellowPages media – « home-made » processing mastered but ageing The future – Repositionnning map to fit new requirements : the purpose of the map is now to support content – Open-source-oriented strategy (moving towards) – Anticipate emerging challenges (vector tiles, HD maps, relief, ...)
  8. 8. Geographic data SDK Mobile Mappy API AJAX Mappy Stack Mappy Engine Templates Stack Mappy Cache Data processing Mostly pre-computation Duration : 10 days Non standard proprietary format 12 zoom levels Software system Fully mastered (behaviour) Some dead code Costly evolutions Non negociable performances 6 Billions req/month BACK OFFICE PRODUCTION
  9. 9. The path - goals – Create a map design able to support content – Be ready to release with the new version of our website fr.mappy.com – Reduce the computational time – address weaknesses of the platform and keep in mind its strengths – Streamlining map tile production – Standardizing (REST API, etc.) – Build for the future (Able to scale and support new services)
  10. 10. Take a Deep Breath. Start Again.
  11. 11. Geographic data API AJAX NewMappy SDK Mobile New Mappy KyotoCabinetKyotoCabinet New platform in production Reduced Preparation Time 2days (vector) + 2days (relief) Standard Live code Mapnik contribution Reduced operating cost BACK OFFICE PRODUCTION
  12. 12. – Le chemin Our journey
  13. 13. Our journey - Starting point – Heterogeneous knowledge of the team on open source technologies – Training & getting help from expert – Bring a culture of open source – Split project – Migrate brick by brick in order to learn and master the platform – « Eat our own food » – Test it fast Innophi
  14. 14. Geographic data SDK Mobile Mappy API AJAX Mappy Stack Mappy Engine Templates Stack Mappy Cache API AJAX NewMappy SDK Mobile New Mappy KyotoCabinetKyotoCabinet BACK OFFICE PRODUCTION BACK OFFICE PRODUCTION Templates Step 0 Map design By marketing and Map designer
  15. 15. Geographic data SDK Mobile Mappy API AJAX Mappy Stack Mappy Engine Templates Stack Mappy Cache API AJAX NewMappy SDK Mobile New Mappy KyotoCabinetKyotoCabinet BACK OFFICE PRODUCTION BACK OFFICE PRODUCTION Step 1 Migration of database and map engine Keep Front interface Work on database performance
  16. 16. Geographic data SDK Mobile Mappy API AJAX Mappy Stack Mappy Engine Templates Stack Mappy Cache API AJAX NewMappy SDK Mobile New Mappy KyotoCabinetKyotoCabinet BACK OFFICE PRODUCTION BACK OFFICE PRODUCTION Step 2 Cache migration New APIs Add relief /HD tiles Work on performance and configuration (Cache/Service/APIs)
  17. 17. Geographic data SDK Mobile Mappy API AJAX Mappy Stack Mappy Engine Templates Stack Mappy Cache API AJAX NewMappy SDK Mobile New Mappy KyotoCabinetKyotoCabinet BACK OFFICE PRODUCTION BACK OFFICE PRODUCTION Step 3 Streamlining databases Work on scaling Optimizing process
  18. 18. Our journey – key points – Be able to challenge historical choices – Reach a better consensus with our customers (Web/Mobile) – Less time on tools implementation than configuration – We built prototype in two weeks – We worked on configuration during 4 months – Real issue : association of metadata with map tiles (dynamic display on map icons)
  19. 19. Our journey – technical choices – PostGIS: transactional database/scalable/standard compliant – Mapnik : approved by marketing and technical teams! – Tornado : – High performance reputation, already used by mobile team – Varnish : – Highyield – Already used by front teams – Nothing to prove
  20. 20. Our journey – Contribution to Mapnik – Mappy was legitimate to participate in an open source project – Develop an open source policy at Mappy – Arrange technical organization (Open Source Manager, Github website : https://github.com/Mappy) – Contribute to Mapnik : implementation of parallelism with PostGIS (available on master). Performance gain: 25% – Some tools availables on Github like pycnik
  21. 21. Feedback
  22. 22. Feedback– Key figures – Production platform handles 85% of the load (All media except Mobile YP) – 14 servers: – Varnish : 2 / Tornik : 8 / PostGIS : 4 – Output without cache: – Peak 240 req/s – Average throughput 150 req/s – Output with cache: 2500 req/s (peak) – Cache Hit Ratios: 94% – Cache preload: 2 millions objects – Postgis database: 75 Go – Relief database: 300 Go
  23. 23. Feedback… team – Positive – Team motivation, proud and in control of our new platform – PostGIS and Mapnik tools – Contribution/open source world – Negative – Difficult to identify correct projects splitting – Contribute lately to Mapnik – Contributing requires available time (discussion with community/ pull requests) – Deployment by our customer not as fast as expected
  24. 24. Thank you @AudreyMalherbe https://github.com/Mappy

×