Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Mapbox.com
Globally serving maps from 8
datacenters
@freenerd
johan@mapbox.com
Berlin DevOps 2015-01
1
• What is Mapbox
• Serving maps from 8 regions
• Deploying to 8 regions
2
3
4
5
6
7
8
9
Application requirements
• Global high availability
• Global low latency
10
Application requirements
• Global high availability
• Global low latency
Usage pattern
• "The long tail of map requests"
11
What is a map?
12
13
Serving tiles (layer-by-layer)
14
15
Client
• Browser loads an embedded map
• Browser loads tile-by-tile
https://tiles.mapbox.com/v4/
map.id/17/70428/42997.png...
Client
https://tiles.mapbox.com/v4/
map.id/17/70428/42997.png?access_token=pk.xxx
17
Client
DNS
• Route53: Amazon DNS service
https://tiles.mapbox.com
18
Client → DNS
CDN (Content Distribution
Network)
• Cloudfront: AWS CDN
• Terminate TLS
• Cache based on url & query string
...
Client → DNS
CDN (Content Distribution
Network)
• find the nearest edge location
20
Client → DNS
CDN (Content Distribution
Network)
21
Client → DNS
CDN (Content Distribution
Network)
• find the nearest edge location
• TTL of 5 minutes
• ~35% cache hits
22
Client → DNS
CDN (Content Distribution
Network)
• On cache miss:
• Continue request to origin server
23
24
Client → DNS → CDN
DNS
• Route53: Amazon DNS
25
Client → DNS → CDN
DNS
• Route53: Amazon DNS
• Mapbox.com serves maps from 8 regions
• Route to "best region for request b...
Client → DNS → CDN
DNS
27
Client → DNS → CDN -> DNS
Load Balancer
• ELB: Amazon Elastic Load Balancer
• Terminate TLS
• Route requests to applicatio...
Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
• EC2: Amazon Elastic Compute Cloud
• Virtual Machines
• EC...
Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
• Node.js/Express
• Authentication
• Determine location of ...
Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
Fetching tiles
• check simultanously in local cache (redis)...
Client → DNS → CDN -> DNS -> Load Balancer
Application Servers
Cache vs. Object Store
• redis: 50% cache hits
• (60% non-c...
33
34
35
36
Deploy to many regions
37
Deploy to many regions
• constantly have around 300 EC2 instances running
• last months spike was 1000 EC2 instances (batc...
39
Deploying cloudformation stacks
mapbox-cli
• the tool, the infrastructure team gives to developers
• abstracts and automat...
Screencast
41
Screencast
Link to video on youtube
https://www.youtube.com/watch?v=wmDvcDvc6Jw
42
Deployment strategies
• Sequential
• Parallel
• Probed
• one region first
• wait for alerts
• all other regions parallel
43
Deploys with interruptions
• Take region out of Route 53
• Wait for traffic to cease
• Update stack
• Put region back into ...
• mapbox-cli is built with node.js
• mostly based around the aws-js sdk
• github.com/mapbox
• cfn-config
• cfn-stack-event-...
Future directions
• deploybot
• send some traffic to staging/canary stacks
• docker for speeding up instance start time
46
We are hiring
• work asynchronously
• github & hipchat
• no emails
47
Thanks
• github.com/mapbox
• cfn-config
• cfn-stack-event-stream
• as-replace-instances
@freenerd
johan@mapbox.com
48
Upcoming SlideShare
Loading in …5
×

Mapbox.com: Serving maps from 8 regions

5,200 views

Published on

Talk given at DevOps Berlin Usergroup January 2015
http://www.meetup.com/blndevops/events/219636613/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Mapbox.com: Serving maps from 8 regions

  1. 1. Mapbox.com Globally serving maps from 8 datacenters @freenerd johan@mapbox.com Berlin DevOps 2015-01 1
  2. 2. • What is Mapbox • Serving maps from 8 regions • Deploying to 8 regions 2
  3. 3. 3
  4. 4. 4
  5. 5. 5
  6. 6. 6
  7. 7. 7
  8. 8. 8
  9. 9. 9
  10. 10. Application requirements • Global high availability • Global low latency 10
  11. 11. Application requirements • Global high availability • Global low latency Usage pattern • "The long tail of map requests" 11
  12. 12. What is a map? 12
  13. 13. 13
  14. 14. Serving tiles (layer-by-layer) 14
  15. 15. 15
  16. 16. Client • Browser loads an embedded map • Browser loads tile-by-tile https://tiles.mapbox.com/v4/ map.id/17/70428/42997.png?access_token=pk.xxx 16
  17. 17. Client https://tiles.mapbox.com/v4/ map.id/17/70428/42997.png?access_token=pk.xxx 17
  18. 18. Client DNS • Route53: Amazon DNS service https://tiles.mapbox.com 18
  19. 19. Client → DNS CDN (Content Distribution Network) • Cloudfront: AWS CDN • Terminate TLS • Cache based on url & query string https://tiles.mapbox.com/v4/ map.id/17/70428/42997.png?access_token=pk.xxx 19
  20. 20. Client → DNS CDN (Content Distribution Network) • find the nearest edge location 20
  21. 21. Client → DNS CDN (Content Distribution Network) 21
  22. 22. Client → DNS CDN (Content Distribution Network) • find the nearest edge location • TTL of 5 minutes • ~35% cache hits 22
  23. 23. Client → DNS CDN (Content Distribution Network) • On cache miss: • Continue request to origin server 23
  24. 24. 24
  25. 25. Client → DNS → CDN DNS • Route53: Amazon DNS 25
  26. 26. Client → DNS → CDN DNS • Route53: Amazon DNS • Mapbox.com serves maps from 8 regions • Route to "best region for request based on latency" 26
  27. 27. Client → DNS → CDN DNS 27
  28. 28. Client → DNS → CDN -> DNS Load Balancer • ELB: Amazon Elastic Load Balancer • Terminate TLS • Route requests to application servers 28
  29. 29. Client → DNS → CDN -> DNS -> Load Balancer Application Servers • EC2: Amazon Elastic Compute Cloud • Virtual Machines • EC2 instances are provisioned via Auto Scaling Group • Auto Scaling is based on instance CPU load • Scale up/down if CPU load over/under 55%/20% for 2 minutes 29
  30. 30. Client → DNS → CDN -> DNS -> Load Balancer Application Servers • Node.js/Express • Authentication • Determine location of tiles • Fetch tiles and return them 30
  31. 31. Client → DNS → CDN -> DNS -> Load Balancer Application Servers Fetching tiles • check simultanously in local cache (redis) and object store (s3) • return from where is found first • if only found in object store, update local cache 31
  32. 32. Client → DNS → CDN -> DNS -> Load Balancer Application Servers Cache vs. Object Store • redis: 50% cache hits • (60% non-cdn = 30% total requests) • redis is used as least-recently used cache, thus popular tiles for a region are usually cached • redis is provisioned by AWS via ElastiCache 32
  33. 33. 33
  34. 34. 34
  35. 35. 35
  36. 36. 36
  37. 37. Deploy to many regions 37
  38. 38. Deploy to many regions • constantly have around 300 EC2 instances running • last months spike was 1000 EC2 instances (batch jobs) • can't care for individual EC2 instances • every developer handles deployment themselves 38
  39. 39. 39
  40. 40. Deploying cloudformation stacks mapbox-cli • the tool, the infrastructure team gives to developers • abstracts and automates common tasks • encodes our conventions for using AWS 40
  41. 41. Screencast 41
  42. 42. Screencast Link to video on youtube https://www.youtube.com/watch?v=wmDvcDvc6Jw 42
  43. 43. Deployment strategies • Sequential • Parallel • Probed • one region first • wait for alerts • all other regions parallel 43
  44. 44. Deploys with interruptions • Take region out of Route 53 • Wait for traffic to cease • Update stack • Put region back into Route 53 44
  45. 45. • mapbox-cli is built with node.js • mostly based around the aws-js sdk • github.com/mapbox • cfn-config • cfn-stack-event-stream • as-replace-instances 45
  46. 46. Future directions • deploybot • send some traffic to staging/canary stacks • docker for speeding up instance start time 46
  47. 47. We are hiring • work asynchronously • github & hipchat • no emails 47
  48. 48. Thanks • github.com/mapbox • cfn-config • cfn-stack-event-stream • as-replace-instances @freenerd johan@mapbox.com 48

×