Planet Labs's Troy Toman shows how they're using Kubernetes to run the infrastructure needed to manage and process petabytes of satellite imagery.

1. Mailiao Refinery, Taiwan May 31, 2016
Troy Toman
@troytoman
troy@planet.com
Managing 6PB of Earth Imagery
with Kubernetes

2. Dongtaijinai’er Salt Lake, China MAY 3, 2016
You think you have cloud problems?
Let me show you cloud problems

3. Uluru, Australia, DEC 2, 2015
and some Kubernetes
SPACE!

4. AGENDA
• Planet’s Mission 1
• Where we started
• Why Kubernetes
• Where we are going

5. To image the whole world
every day, making change
visible, accessible and
actionable.

6. CAPE TOWN, SOUTH AFRICA
MARCH 9, 2016
MONITORING
PORT ACTIVITY

7. CAPE TOWN, SOUTH AFRICA
MARCH 14, 2016
MONITORING
PORT ACTIVITY

8. CAPE TOWN, SOUTH AFRICA
MARCH 15, 2016
MONITORING
PORT ACTIVITY

9. SAN GABRIEL MOUNTAINS
JUNE 4, 2016

10. SAN GABRIEL MOUNTAINS
JUNE 4, 2016
SAN GABRIEL
WILDFIRES
VEGETATION INDEX MAP
VEGETATION
bare sparse densemoderate

11. SAN GABRIEL
WILDFIRES
FIRE RETARDENT LINES
ARE IN RED
JUNE 22, 2016
SAN GABRIEL MOUNTAINS

12. JUNE 22, 2016
SAN GABRIEL MOUNTAINS
FALSE COLOR
COMOPOSITE
REVEALS BURN SCARS

13. Muir Woods & Mt. Tamalpais, California, USA DEC 23, 2015
OUR APPROACH

14. HISTORICAL APPROACH
• Tasking
• Low coverage
• Weeks to gain access
• Expensive
PLANET APPROACH
• Monitoring
• Global coverage
• Daily online delivery
• Affordable

15. 13 Builds in only 3 years
BUILD 1
APR 2012
BUILD 6
APR 2013
BUILD 13
JUN 2015
AGILE AEROSPACE

16. 12+
SATELLITES
3 - 5M
RESOLUTION
4-BAND IMAGERY
10
GROUND STATIONS
4
SITES
10,000
IMAGES PER DAY
API
FOR INTERNAL DATA
PIPELINE
100S
OF VIRTUAL
MACHINES
TBs
of STORAGE
The early days …

17. Early approach
• One service team
• 2 basic services
• Storage
• Jobs
• Everything packaged as debs
• Deployed on VMs w/Ansible
• Everything tested/deployed as one

18. Planet’s Evolving Infrastructure
Jobs
Worker Spot
Instances
Imagery
Storage
Storage
Elastic Search
Planet API v0

19. 100+
SATELLITES
3 - 5M
RESOLUTION
4-BAND IMAGERY
26
GROUND STATIONS
9
SITES
10+TB
PER DAY
EVERY DAY
API
FOR INTERNAL DATA
PIPELINE AND
PLATFORM ACCESS,
CHANGE DETECTION,
BASE MAPS,
MOSAICS,
and MORE
10,000s
OF VIRTUAL
MACHINES
PBs
of STORAGE
The very near future …

20. PLATFORMAUTOMATED
DATA
PIPELINE
DATA AND FUNCTIONAL EXPLOSION!
MOSAIC &
IMAGE
PROCESSING
ANALYTICS
CONTAINER
API – RESTFUL & JSON
PlanetScope
~150 Mkm2/DAY
RapidEye
~6 Mkm2/DAY
Processing
Engine
Ingestion
API
WEB TILES
API
CATALOG
SCENES, MOSAIC, TILE API
ANALYTICS
ANALYSIS API
ARCHIVE GUI
FIND, DOWNLOAD
DATA VIA WEB
BROWSER
TOOLS
SAMPLE CODE
3RD PARTY
APPLICATIONS
More to come …
Landsat 8

21. Large Scale, Dynamic Environment

22. We’ve Seen Change
• Growing number of service teams
• Dozens of services
• Different scales
• Different maturity
• Different markets
• Unintentional monolith and/or
unmanageable fringe services
• Infrastructure as impediment

23. How our software world is changing
• Containers
• Flexible per service packaging
• Faster, simpler, consistent deploys
• Kubernetes for orchestration
• Dynamic reallocation of resources
• Isolate development teams from infra
changes and each other
• Couple service changes
where necessary

24. Planet’s Evolving Infrastructure
Jobs
Worker Spot
Instances
Imagery
Storage
Storage
Elastic Search
Planet API v0
Planet API v1
Infra
Apps

25. Planet Kubernetes Tooling
• Klondike
• Bootstraps k8s on AWS for Planet
• https://github.com/planetlabs/klondike
• Focused on infrastructure deployment
• Planet-Express
• Environments, Services, Deployments, Secrets
• Template driven manifests and artifacts
kubectl create -f build/<platform>/<service>/
kubectl replace -f build/<platform>/<service>/
• Focused on services development and deployment

26. Key Learnings
• Kubernetes increases our flexibility
and responsiveness
• Move can be evolutionary
• Tooling is leveraged across teams
• More good stuff to come
• More change on the way - better
prepared to exploit opportunity

27. London Array Wind Farm, United Kingdom, APR 17, 2016
See Change.
Change the World.
(applies to software and infrastructure too!)
troy@planet.com
@troytoman

28. London Array Wind Farm, United Kingdom, APR 17, 2016