Microcontainers, Microservices, Microservers? Less [Linux] is more!
Microcontainers, Microservices, Microservers.
Less [Linux] is more!
1993-1997 Set up first ISP in Northern Ireland (Genesis Project Ltd)
Senior Technical Support Engineer @
Mobile Technology Consultant @ SLA Mobile, on-site at
Vodafone Global in Germany
Contractor @ Alcatel-Lucent Network Services in
2012-2015 Contractor @ Vodafone Group Services in Germany
2015 April-August Sabbatical/career break
2015 September Senior Linux Administrator @ Proofpoint
In a Microservices vision large scale services can be built by combining
Application Containers like Lego bricks. However App Containers are not
usually lightweight – a typical Docker container uses Ubuntu as its base.
Alternatively if based on minimal Linux distributions or (in a more extreme
case) applications are just statically linked, the resulting containers can be
far smaller. Lightweight means faster startup & migration time between
hosts and less RAM use. And as for the container hosts? Whether VM
instances or bare metal, the underlying OS can also be stripped right down
– CoreOS and RancherOS were designed with exactly this in mind.
Together lightweight containers and hosts mean savings – fewer, smaller,
cheaper cloud VM instances or bare metal servers (such as Microservers)
and potentially improved security.
The progression from physical machines to
● Real machines hosting applications
● Individual machines often dedicated to hosting only a
single or a few applications
● Typically machines are underutilised (CPU, RAM use)
IBM HPDELL* * *
* apparently use of their logos requires prior written permission!
Provides a degree of application isolation allowing
several physical machines to be replaced by 1 physical
machine hosting multiple VMs
Results in better physical machine utilisation
Single kernel shared between containers – better I/O
(disk, network) performance than VMs but “weaker”
security isolation between containers
Faster startup time than VMs
An application plus “just enough” OS to support the app
(i.e. no sshd, no syslogd, no init, etc)
Immutable Infrastructure – no need to use
Puppet/Chef/Ansible/Salt to manage them
The same container is portable across physical hosts,
VMs, and multiple cloud providers
The exact same container can be run on a developer's
laptop, in a testlab, and in production environment
Various architectures of container use
Host OSHost OS
Bare Metal Virtualisation Cloud for VMs Cloud for
The range of sizes
Normal “full fat” Linux distribution
Lean but generic Linux distribution
Minimal application-specific container
(perhaps just a statically-linked binary)
Minimal App Containers (1)
An application-specific container where the application
is compiled & either statically linked (resulting in just a
single binary file in the container) or dynamically
linked (resulting in a binary plus dependant libraries).
Minimal App Containers (2)
“dockerize” is a utility that analyses a dynamically-
linked binary to create a container with the binary plus
just the libraries it needs.
“gockerize” is a utility for building statically linked
golang binaries and creating Docker container
Alpine Linux (1)
● Distribution based on musl and Busybox
● Official base image size in Docker Hub is only 5MB in size!
● Comes with a package manager. Large number of common
applications already packaged.
● Provides a useful middle ground between normal “full fat”
distributions and handcrafted minimal app-specific containers.
Alpine Linux (2)
NGINX – 8MB
OpenJDK Java 7 JRE, 123MB
Oracle Java 8 JRE, 173MB
Example sizes of Alpine-based containers:
Host OS Options (2)
TinyCore Linux, 10MB+
Alpine Linux, 87MB+
Size is of base distribution, does not take into account size of Docker tools etc...
Combined small App Containers with small Host
Benefits of combined reductions
● Smaller spec Cloud VMs or physical servers can be utilised
● Cost savings!
● Quicker app container and/or host VM migration
● Smaller attack surface
Special case: App Container & slim VM merged
Intel Clear Containers
● KVM hypervisor
● Uses kvmtool (no need for BIOS/UEFI)
● Stripped-down Linux kernel (only Virtio devices)
● Systemd as init
● Rkt containers supported (docker support soon)
● Startup almost as fast (150ms) as native container
● Per-container RAM overhead of 18-20MB
● Container running inside a lightweight VM
“Microservices is a software architecture style in which
complex applications are composed of small, independent
processes communicating with each other using language-
agnostic APIs. These services are small, highly decoupled and
focus on doing a small task, facilitating a modular approach to
“small independent processes”
“modular approach to system building”
App Containers fit the Microservices model perfectly:
Micro Data Centre (2)
● Small rack cabinet with multiple compute nodes,
storage nodes, admin node(s), network switch, UPS,
● 10U cabinet capable of hosting 4000+ containers?
Micro Data Centre (3)
● No special (physical, power, cooling) room
requirements unlike a normal data centre
● Easier to keep data on-site for legal or privacy
● Potentially economical to have multi-site redundancy
with 2 MDCs in separate but geographically-close
(low network latency) locations
Personal experimentation – my current Micro
Compute Nodes (1)
Off-the-shelf Mini-ITX motherboards with low power
CPUs (with passive heatsinks or heatsink/fans)
Small amounts of RAM, 4-8Gb. Low power (1.35V)
DDR3L if possible.
No local storage
Low wattage small power supply
Machines PXE boot the Host OS and run directly from
Compute Nodes (3)
My intention is to make a custom low-cost blade-type
chassis for nodes.
Chassis will be approx. 6U high and 250mm deep with
12 compute nodes per chassis.
Individual nodes will be approx. 6Ux 35mm x 245mm
Low power CPUs on Mini-ITX motherboards
Mix of SSDs and HDDs
Using CEPH to provide storage services
Pair of HDDs used with software mirroring
Provides DHCP/TFTP/HTTP services for PXE booting
Will run InfluxDB, ElasticSearch and Kibana for metric
and log analysis
Gigabit Ethernet managed switch with VLAN, IPv6, and
Use Ansible, via SSH, to configure the switch
Plan is for all containers & hosts to use only IPv6
addressing. The entry point to the “cloud” (i.e. Load
Balancer) will handle both IPv4 and IPv6 external
addresses and forward traffic onwards to internal IPv6