1. What It Really Takes to Build a Container
Platform
10 Things We Learned the Hard Way

2. Matt Butcher
Platform Architect at Deis
Matt is responsible for
defining Deis v2 and
overseeing what he likes to
call…
The Great Kubernetes Rebase
11/18/2015 Putting Containers into Production

3. This is not a commercial for Deis
11/18/2015 Putting Containers into Production

4. Our View (Subject to Change)
• Component architecture is haute couture.
• Containers are the idea vehicle for
contemporary architecture
• Avoid the “thrown over the wall” problem by
building good tools
11/18/2015 Putting Containers into Production

5. Background
Deis v1
• Focus: Better Heroku for
You
• Docker to the max
• Mostly Go
• Flexibility, but no services
• Big success
Deis v2
• Focus: PaaS as a Set of
Microservices
• Kubernetes to the max
• Still mostly Go
• Building blocks
• About 40% complete
11/18/2015 Putting Containers into Production

6. Top Ten List of Things We Learned the
Wrong Way
11/18/2015 Putting Containers into Production
Hard

7. SAY “YES” TO OBJECT STORAGE
Lesson 10
11/18/2015 Putting Containers into Production

8. … and probably
“no” to network
file systems
11/18/2015 Putting Containers into Production

9. 11/18/2015 Putting Containers into Production
Mistake: We assumed everyone wanted a
single storage solution.
Solution: Focus on object storage for core
components

10. Object Storage vs. Network FS
Object Storage
• Standard interface: S3 API
• Dozens of nearly uniform
implementations
• Unit of work: object (file)
Filesystem/Block Storage
• POSIX-ish, lots of standards
• Too many choices for
implementation
• Lots of variants with
important discrepancies
• But can handle streaming
11/18/2015 Putting Containers into Production

11. TURTLES ALL THE WAY DOWN
Lesson 9
11/18/2015 Putting Containers into Production

12. The strange things we do for security
11/18/2015 Putting Containers into Production

13. 11/18/2015 Putting Containers into Production
Mistake: We built our containers inside of
containers (insecure).
Solution: Well…

14. Docker and Security
• Running containers:
relatively small set of
issues
• Building containers is
much riskier
11/18/2015 Putting Containers into Production

15. “What if we ran Docker in a VM in a container?”
11/18/2015 Putting Containers into Production
Kubernetes Node
Container
VM (Qemu)
Build Container

16. THE CLUSTERBOMB
Lesson 8
11/18/2015 Putting Containers into Production

17. HA clustering is hard. Ask etcd.
11/18/2015 Putting Containers into Production

18. 11/18/2015 Putting Containers into Production
Mistake: We used Etcd in a naïve way,
assuming clustering was fairly dynamic.
Solution: Spend lots more time
hardening clustering solutions.

19. DOCKER IS A WHALE ON THE MOVE
Lesson 7
11/18/2015 Putting Containers into Production

20. 11/18/2015 Putting Containers into Production
Mistake: We committed to stay on the
latest release of Docker within the 1.x line
(SemVer).
Solution: Hang back until a particular
version emerges as stable.

21. Docker doesn’t do
SemVer
11/18/2015 Putting Containers into Production

22. SINK THE TITANIC WHILE IT’S STILL AT
HARBOR
Lesson 6
11/18/2015 Putting Containers into Production

23. 11/18/2015 Putting Containers into Production
Mistake: Underestimate the complexity
of a fully operational container platform.
Solution: Testing of as many varieties as
possible.

24. Testing with vim and vigor
11/18/2015 Putting Containers into Production
http://is.gd/GWKfGv

25. GOPHERS ARE AWESOME…ISH
Lesson 5
11/18/2015 Putting Containers into Production

26. Five Things I Love About Go
1. Great tools
2. Static compilation is huge for containers
3. Rich libraries, esp. for networking
4. Easy to learn
5. Cross-platform (for very small definitions of
“platform”)
11/18/2015 Putting Containers into Production

27. 11/18/2015 Putting Containers into Production
Mistake: We might have been just a little
starry eyed.
Solution: Accept the limitations.

28. O Go, Y U NO…
11/18/2015 Putting Containers into Production
http://meme.wikia.com/wiki/File:Y_u_no_sketch.png

29. Trouble in Paradise
• “Go is easy” == simple
syntax, verbose code
• Package management
• Package management
• Package management
11/18/2015 Putting Containers into Production

30. YOUR TEAM IS LIKE YOUR UNDERPANTS
Lesson 4
11/18/2015 Putting Containers into Production

31. They can only stretch so far
11/18/2015 Putting Containers into Production

32. Microservice Architecture
11/18/2015 Putting Containers into Production
http://assets2.ignimgs.com/2014/06/24/ready-player-one-book-art-1280jpg-b68ca1_1280w.jpg

33. 11/18/2015 Putting Containers into Production
Mistake: Microservice all the things, but
without a plan for maintenance.
Solution: Process saves the day.

34. ALL YOUR SCHEDULERS ARE BELONG TO
US
Lesson 3
11/18/2015 Putting Containers into Production

35. Somebody set up us the bomb!
11/18/2015 Putting Containers into Production

36. 11/18/2015 Putting Containers into Production
Mistake: “Support all of the schedulers”
means “support the weakest scheduler”
Solution: Choose just one, and choose
based on power & flexibility.

37. MONOREPOS ARE EVIL
Lesson 2
11/18/2015 Putting Containers into Production

38. Monorepos
Pros
• All the code is in just one
place.
• Everyone gets alerted on
everything.
• Simplifies builds,
documentation, issue
queue, etc.
Cons
• Huge dependency tree
• Everyone gets alerted on
everything
• Stable and “preview” mixed
• Monorepo vs. microservice
• Our users couldn’t
effectively re-use
11/18/2015 Putting Containers into Production

39. 11/18/2015 Putting Containers into Production
Mistake: One repo with everything in it!
(Hey! Google does it!)
Solution: Break out microservices into
their own repo.

40. KUBERNETES FLOATS MY BOAT
Lesson 1
11/18/2015 Putting Containers into Production

41. But it’s a very big and complex boat
11/18/2015 Putting Containers into Production
http://is.gd/bUGF1J

42. 11/18/2015 Putting Containers into Production
Mistake: “Kubernetes is easy! After all,
we only have to support one scheduler!”
Solution: Educate, cooperate, and build
really good tooling!

43. Package Management
Make working with
Kubernetes as easy
as…
$ helm install redis-
cluster
11/18/2015 Putting Containers into Production
Github.com/deis/helm

44. Conclusion
• Our guiding philosophy: containers for
microservices without “throwing over the
wall”
• We’ve learned some lessons along the way
• But the bottom line: Containers are the right
bet for the microservice architecture.
11/18/2015 Putting Containers into Production

45. 11/18/2015 Putting Containers into Production