Couchbase Containers with
Bare Metal Performance
Elastic infrastructure, circa Dot Com Boom
• In the late 1990s, the only way to meaningfully
scale a database was up — and the physical
infrastructure had to scale with it
• This was excruciatingly expensive — and
became a non-starter in the post-apocalyptic
nuclear winter of the early 2000s...
Elastic infrastructure, circa Dot Com Bust
• The rise of rack-and-stack commodity servers brought with it new
distributed software architectures like memcached that were
designed to scale across many machines
• The rise of these architectures afforded new operational
possibilities: if the computer itself is a commodity, why buy it at
all? Why not rent from someone who runs it cheaper and better?
• From the perspective of compute providers, economies of scale
could only be realized if hardware is shared across tenants...
• Multi-tenancy demands virtualization, but where in the stack to
• The historical answer to virtualization — since the 1960s — has
been to virtualize the hardware:
• A virtual machine is presented upon which each tenant runs an
operating system that they choose (and must manage)
• There are as many operating systems on a machine as tenants!
• Can run entire legacy stacks unmodiﬁed...
• ...but operating systems are heavy and don’t play well with others
with respect to resources like DRAM, CPU, I/O devices, etc.
• Hardware-level virtualization limits tenancy and performance!
• Virtualizing at the application platform layer addresses the
tenancy challenges of hardware virtualization, and presents a
much more nimble (& developer friendly!) abstraction...
• ...but at the cost of dictating abstraction to the developer
• This is the “Google App Engine” problem: developers are in a
straightjacket where toy programs are easy — but sophisticated
applications are impossible
• Virtualizing at the application platform layer poses many other
challenges with respect to security, containment, etc.
• Virtualizing at the operating system hits a sweet spot:
• A single operating system (i.e. a single kernel) allows for efﬁcient use of
hardware resources, maximizing tenancy and performance
• Disjoint instances are securely compartmentalized by the operating system
• Gives tenants what appears to be a virtual machine (albeit a very fast one)
on which to run higher-level software: PaaS ease with IaaS generality
• Applications run directly on the hardware!
• Model was pioneered by FreeBSD jails and taken to their logical
extreme by Solaris zones — and then aped by Linux containers
OS-level virtualization at Joyent
• Joyent runs OS containers in the cloud via SmartOS — and we
have run containers in multi-tenant production since ~2006
• Adding support for hardware-based virtualization circa 2011
strengthened our resolve with respect to OS-based virtualization
• We emphasized their operational characteristics — performance,
elasticity, tenancy — and for many years, we were a lone voice...
Containers as PaaS foundation?
• Some saw the power of OS containers to facilitate up-stack
• For example, dotCloud — a platform-as-a-service provider — built
their PaaS on OS containers
• Struggling as a PaaS, dotCloud pivoted — and open sourced
their container-based orchestration layer...
• Docker has used the rapid provisioning + shared underlying
ﬁlesystem of containers to allow developers to think operationally
• Developers can encode deployment procedures via an image
• Images can be reliably and reproducibly deployed as a container
• Images can be quickly deployed — and re-deployed
• Docker will do to apt what apt did to tar
Broader container revolution
• The Docker model has pointed to the future of containers
• Docker’s challenges today are largely operational: network
virtualization, persistence, security, etc.
• Security concerns are not due to Docker per se, but rather to the
architectural limitations of the Linux “container” substrate
• For multi-tenancy, state-of-the-art for Docker containers is to run
in hardware virtual machines (!!)
• Deploying OS containers in hardware virtual machines negates
their economic advantage!
• SmartOS has been container-native since its inception — and
running in multi-tenant, internet-facing production for many years
• Can we achieve an ideal world that combines the development
model of Docker with the container-native model of SmartOS?
• This would be the best of all worlds: agility of Docker coupled with
production-proven security and on-the-metal performance of
• But there are some obvious obstacles...
Docker + SmartOS: Linux binaries?
• First (obvious) problem: while it has been designed to be cross-
platform, Docker is Linux-centric — and the encyclopedia of
Docker images will likely forever remain Linux binaries
• SmartOS is Unix — but it isn’t Linux…
• Fortunately, Linux itself is really “just” the kernel — which only has
one interface: the system call table
• We resurrected (and ﬁnished) a Sun technology for Linux system
call emulation, LX-branded zones, the technical details of which
are beyond the scope of this presentation...
Docker + SmartOS: Provisioning?
• With the binary problem being tackled, focus turned to the
mechanics of integrating Docker with SmartOS provisioning
• Provisioning a SmartOS zone operates via the global zone that
represents the control plane of the machine
• docker is a single binary that functions as both client and server
— and with too much surface area to run in the global zone,
especially for a public cloud
• docker has also embedded Go- and Linux-isms that we did not
want in the global zone; we needed to ﬁnd a different approach...
Docker Remote API
• While docker is a single binary that can run on the client or the
server, it does not run in both at once…
• docker (the client) communicates with docker (the server) via
the Docker Remote API
• The Docker Remote API is expressive, modern and robust (i.e.
versioned), allowing for docker to communicate with Docker
backends that aren’t docker
• The clear approach was therefore to implement a Docker Remote
API endpoint for SmartDataCenter, our (open source!)
orchestration software for SmartOS
Triton: Docker + SmartOS
• In March, we launched Triton, which combines SmartOS and
SmartDataCenter with our Docker Remote API endpoint
• With Triton, the notion of a Docker host is virtualized: to the
Docker client, the datacenter is a large Docker host
• All of the components to Triton are open source: you can
download and install SmartDataCenter and run it yourself
• Triton is currently in early access at Joyent — and moving into
general availability in Q2CY2015
• It is becoming broadly clear that containers are the future of
application development and (especially) deployment
• But the upstack ramiﬁcations are entirely unclear — there are
many rival frameworks for service discovery, deployment, etc.
• The rival frameworks are all open source:
• Unlikely to be winner-take-all
• Productive mutation is not just possible but highly likely
• Triton takes a deliberately modular approach: the container as
general-purpose foundation, not prescriptive framework
Containers and Couchbase
• Couchbase is particularly appropriate for containers: its scale-out
architecture demands elastic infrastructure — and its use cases
demand on-the-metal performance
• But hardware-virtualized Docker hosts undermine the efﬁcacy of
containers — and force an allocation-oriented disposition instead
of allowing a consumption-oriented one
• The Triton model represents Couchbase containers without
compromise: like Couchbase itself, the infrastructure can grow as
needed — while still delivering bare-metal performance!
Future of containers
• For nearly a decade, we have believed that OS-virtualized
containers represent the future of computing — and with the rise
of microservices + Docker, this is no longer controversial
• But to achieve the full promise of containers, they must run
directly on-the-metal — multi-tenant security is a constraint!
• The virtual machine is a vestigial abstraction; we must reject
container-based infrastructure that implicitly assumes it
• Triton represents our belief that containers needn’t compromise:
multi-tenant security, operational elasticity and on-the-metal
• @joshwilsdon, @trentmick, @cachaﬂa, @orlandov, @fredfkuo
and @notmatt and Todd Whiteman for their work on sdc-docker
• Jerry Jelinek, @pfmooney, @jmclulow, @rmustacc and @jperkin
for their work on LX branded zones
• The countless engineers who have worked on or with SmartOS
because they believed in OS-based virtualization before the rest
of the world ﬁgured it out