The Container Revolution: Reflections after the first decade

1. The Container Revolution Reﬂections after the ﬁrst decade CTO bryan@joyent.com Bryan Cantrill @bcantrill

2. The Container Revolution Reﬂections after the ﬁrst third decade CTO bryan@joyent.com Bryan Cantrill @bcantrill

3. Containers are born: March 18, 1982 • Containers are not a new idea, having originated via ﬁlesystem containers with chroot in Seventh Edition Unix • chroot originated with Bill Joy, but speciﬁcs are blurry; according to Kirk McKusick, via Poul-Henning Kamp and Robert Watson:

4. Containers ca. 2000 • Seeking to provide a security mechanism, FreeBSD extended chroot into jails:

5. • To provide workload consolidation, Sun introduced complete operating system virtualization with zones (née Project Kevlar) Containers ca. 2002

6. Containers ca. 2006

7. Elsewhere ca. 2006...

8. Aside: Hardware-level virtualization • Since the 1960s, the preferred approach for operating legacy stacks unmodiﬁed has been to virtualize the hardware • A virtual machine is presented upon which each tenant runs an operating system that they choose (but must also manage) • Effective for running legacy stacks, but with a clear inefﬁciency: there are as many operating systems on a machine as tenants: • Operating systems are heavy and don’t play well with others with respect to resources like DRAM, CPU, I/O devices, etc.! • Still, hardware-level virtualization became de facto in the cloud

10. Containers ca. 2011, cont. • Via SmartOS zones, Joyent has run containers in multi-tenant production since ~2006... • Adding support for hardware-based virtualization circa 2011 strengthened our resolve with respect to OS-based virtualization! • We saw that the operational characteristics of OS containers — performance, elasticity, tenancy — opened up new possibilities...

11. Containers ca. 2012/2013

12. Containers ca. 2012/2013, cont. • Over 2012 and early 2013, we built Manta, a ZFS- and container- based internet-facing object store offering in situ compute • Containers allow the compute to be moved to the data instead of having to backhaul data to transient compute • And as a reminder, the OS — Unix! — was built around the notion of ad hoc computation

13. Aside: Unix • When Unix appeared in the early 1970s, it was not merely a new system, but a new way of thinking about systems • The Unix philosophy, as articulated by Doug McIlroy: • Write programs that do one thing and do it well • Write programs to work together • Write programs that handle text streams, because that is a universal interface • Four decades later, this philosophy remains as relevant as ever!

14. Aside: Doug McIlroy v. Don Knuth • In 1986, Jon Bentley posed the challenge that became the Epic Rap Battle of computer science history: Read a ﬁle of text, determine the n most frequently used words, and print out a sorted list of those words along with their frequencies. • Don Knuth’s solution: a purpose-built algorithm in WEB, a Pascal- like literate programming system of his own invention • Doug McIlroy’s solution shows the power of the Unix philosophy: tr -cs A-Za-z 'n' | tr A-Z a-z | sort | uniq -c | sort -rn | sed ${1}q

15. Containers ca. 2012/2013, cont. • Manta allows for an arbitrarily scalable variant of McIlroy’s solution to Bentley’s challenge: mfind -t o /bcantrill/public/v7/usr/man | mjob create -o -m "tr -cs A-Za-z 'n' | tr A-Z a-z | sort | uniq -c" -r "awk '{ x[$2] += $1 } END { for (w in x) { print x[w] " " w } }' | sort -rn | sed ${1}q" • As with McIlroy’s solution, the container allows for not just for terser expression but faster execution!

16. Containers ca. 2012/2013 • Manta served to strengthen our belief in the power of containers as not merely better infrastructure, but as transformative • Our biggest challenge with Manta was that containers were not well understood: it felt “easy” to us, but was confusing to many • Would the world ever really ﬁgure out containers?!

18. Containers ca. 2013/2014: Revolution at last! • Docker used the rapid provisioning + shared underlying ﬁlesystem of containers to allow developers to think operationally • Operational dependencies are encoded via an image • Images can be reliably and reproducibly deployed as a container • Images can be quickly deployed — and re-deployed • Coupled with the mainstream adoption of open source and the leading adoption of microservices... • Containers accelerate software development

19. Containers ca. 2014/2015

20. Containers ca. 2014/2015, cont. • Triton combines SmartOS and our cloud management software with a Docker Remote API endpoint • With Triton, the notion of a Docker host is virtualized: to the Docker client, the datacenter is a large Docker host • One never allocates VMs with Triton; one only allocates containers — and all Triton containers run directly on-the-metal • All of Triton is open source — you can download it and run it for yourself (or run it on our public cloud)

26. 2016 and beyond: Frameworks v. Libraries • The crowded container ecosystem can be viewed broadly as two different approaches: • A framework approach in which the framework is in control, but alternatives are “pluggable” • A library approach in which the developer/operator is in control, composing functionality out of well-defined modules • The “all in” nature of frameworks may make initial adoption easier, but they ultimately sacrifice flexibility!

27. 2016 and beyond: Frameworks v. Libraries • For example, a framework approach to orchestration may not accommodate applications that demand local persistence • We opted for orchestration that lives not in an all-knowing framework, but rather with the application, in the container • In-container logic allows for service registration/discovery, health checks, etc. — but does not handle restart • This is application-centric orchestration, and we have dubbed this the autopilot pattern: https://autopilotpattern.io

28. 2016 and beyond: Containers on autopilot • We have developed ContainerPilot, an embodiment of the autopilot pattern for Docker containers • All open source (https://github.com/joyent/containerpilot) — and not Triton or Joyent speciﬁc; can be used anywhere

29. 2016 and beyond: Container failure modes

32. 2016 and beyond: Failure matters • When deploying containers + microservices, there is an unstated truth: you are developing a distributed system • While more resilient to certain classes of force majeure failure, distributed systems remain vulnerable to software defects • We must be able to debug such systems; hope is not a strategy! • Distributed systems are hard to debug — and are more likely to exhibit behavior non-reproducible in development • With container-based systems, we must think in terms of the system, not merely the program!

33. 2016 and beyond: The Jevons paradox • The Jevons paradox seems very likely to hold for containers: greater efficiency will result in a net increase in consumption! • Efficiency gains from containers are in terms of developer time... • ...but requiring containers to be scheduled in VMs induces operational inefficiencies: every operator must now think like a cloud operator — maximizing density within fixed-cost VMs • Greater consumption + operational inefficiencies threaten to slow the container revolution — or make it explosive in terms of cost • To realize the full economic promise of the container revolution, we need container-native infrastructure!

34. 2016 and beyond: Public and on-prem • Death of on-prem computing is greatly exaggerated! • There are three key determinants for public v. on-premises: • Economics: Rent vs. buy; OPEX vs. CAPEX • Risk Management: Security/compliance — and also risk factors associated with operator-as-threat • Latency: The speed of light is a constant! • Economics dominates: “private cloud” efforts that do not deliver public cloud economics are doomed to (continue to) fail!

35. 2016 and beyond: #serverless • On the one hand, this term is clearly meaningless… • ...but on the other, it highlights that containers have liberated us to think in terms of higher-level functionality • Thinking in functions is great (viz. Manta), but conﬁning functions to containers to single-tenant VMs undermines their economics • The virtual machine is a vestigial abstraction; we must reject container-based infrastructure that implicitly assumes it! • So don’t say “#serverless” when you mean...

36. 2016 and beyond #vmless