The Internet of things (IoT) requires a transition of many paradigms that are commonly used today. One of them is the shift from *scaling up*, i.e. from a system design towards bigger machines with higher computational power to *scaling down*, i.e. a system design based on tiny devices with small computational power. Heavy virtual machines and energy-hungry data center technology are not suited for sensor networks at the edge of the IoT. With the latest release of Docker (August 2016), there is an orchestration tool available that is energy-efficient and lightweight enough to handle heavy load on small, but many IoT devices. This is because Docker has now been made available on the ARM CPU architecture, which is already used in a large share of small devices available today because ARM is optimized for small scale and energy-efficient computing. Thus, with the new Docker ported to ARM, the power of container virtualization technology is now available for the IoT, thereby simplifying the development, deployment and operation of applications. In the mean time, Kubernetes, Google's production-ready orchestration tool, has also been ported to the ARM architecture and therefore is also available to run large sensor networks at little computational power. In this talk I present the results of a research project that evaluated the high availability performance of Kubernetes and Docker in an IoT-like environment: A multi-node Raspberry Pi cluster. This talk includes a live demo if requested.

1. Professorship for Computer Science
Communication Services, Telecommunication Systems and Computer Networks
High Availability Performance
of Kubernetes and Docker Swarm
on a Raspberry Pi Cluster
HighLoad++ Conf | Moskow
November 8, 2016
@MathiasRenner | Hypriot.com

2. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Content
1 Motivation
2 Why Kubernetes and Docker?
3 Related Work
4 Test Setup
5 Testing Scenarios
6 Results
7 Live Demo
8 Outlook
9 Lessons Learned
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 2/30

3. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

4. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

5. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
Energy efficiency increases
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

6. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
Energy efficiency increases
Low-cost hardware availability increases
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

7. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
Energy efficiency increases
Low-cost hardware availability increases
→ Internet of Things
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

8. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
Energy efficiency increases
Low-cost hardware availability increases
→ Internet of Things
Contributing factors
Educational added value (physical, bare metal). Same as big machines, just less
powerful.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

9. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
Energy efficiency increases
Low-cost hardware availability increases
→ Internet of Things
Contributing factors
Educational added value (physical, bare metal). Same as big machines, just less
powerful.
Cluster for Distributed Systems/Microservices/SOA
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

10. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
Energy efficiency increases
Low-cost hardware availability increases
→ Internet of Things
Contributing factors
Educational added value (physical, bare metal). Same as big machines, just less
powerful.
Cluster for Distributed Systems/Microservices/SOA
Very popular
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

11. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Motivation
Observing paradigm shift: From large to small devices, in size and power.
Architecture size shrinks
Energy efficiency increases
Low-cost hardware availability increases
→ Internet of Things
Contributing factors
Educational added value (physical, bare metal). Same as big machines, just less
powerful.
Cluster for Distributed Systems/Microservices/SOA
Very popular
Own contribution for Docker on ARM
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 3/30

12. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

13. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

14. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

15. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

16. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

17. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
Recently support for ARM has been extended
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

18. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
Recently support for ARM has been extended
Known as very reliable (based on Google’s Borg DC Manager)
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

19. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
Recently support for ARM has been extended
Known as very reliable (based on Google’s Borg DC Manager)
Not that ressource-intensive
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

20. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
Recently support for ARM has been extended
Known as very reliable (based on Google’s Borg DC Manager)
Not that ressource-intensive
What about...?
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

21. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
Recently support for ARM has been extended
Known as very reliable (based on Google’s Borg DC Manager)
Not that ressource-intensive
What about...?
Mesos: Requires too much resources (Netflix got it working), complex setup
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

22. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
Recently support for ARM has been extended
Known as very reliable (based on Google’s Borg DC Manager)
Not that ressource-intensive
What about...?
Mesos: Requires too much resources (Netflix got it working), complex setup
Nomad: Still experimental, in general and on ARM arch
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

23. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Why Kubernetes and Docker
Docker
Official support on ARM available since Aug 18, 2016
Lightweight
Announced HA features that are very basic, but complete for enough use cases
Kubernetes (K8s)
Recently support for ARM has been extended
Known as very reliable (based on Google’s Borg DC Manager)
Not that ressource-intensive
What about...?
Mesos: Requires too much resources (Netflix got it working), complex setup
Nomad: Still experimental, in general and on ARM arch
OpenStack, Pacemaker, Marathon, Apache Spark, Open MPI, ROOT...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 4/30

24. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

25. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
In 2012: first public project about cloud computing with Raspberry Pis (the first
model) has been started by researchers at the University of Glasgow
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

26. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
In 2012: first public project about cloud computing with Raspberry Pis (the first
model) has been started by researchers at the University of Glasgow
LXC was state of the art, everything non-container virtualization too
resource-hungry
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

27. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
In 2012: first public project about cloud computing with Raspberry Pis (the first
model) has been started by researchers at the University of Glasgow
LXC was state of the art, everything non-container virtualization too
resource-hungry
Largest project: Bolzano University with 300 RPis
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

28. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
In 2012: first public project about cloud computing with Raspberry Pis (the first
model) has been started by researchers at the University of Glasgow
LXC was state of the art, everything non-container virtualization too
resource-hungry
Largest project: Bolzano University with 300 RPis
In 2015: Turning point: Lucas K¨aldstr¨om and University of Glasgow started at
same time porting Kubernetes to ARM - independently.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

29. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
In 2012: first public project about cloud computing with Raspberry Pis (the first
model) has been started by researchers at the University of Glasgow
LXC was state of the art, everything non-container virtualization too
resource-hungry
Largest project: Bolzano University with 300 RPis
In 2015: Turning point: Lucas K¨aldstr¨om and University of Glasgow started at
same time porting Kubernetes to ARM - independently.
Lucas implemented to most features and recently became Kubernetes maintainer.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

30. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
In 2012: first public project about cloud computing with Raspberry Pis (the first
model) has been started by researchers at the University of Glasgow
LXC was state of the art, everything non-container virtualization too
resource-hungry
Largest project: Bolzano University with 300 RPis
In 2015: Turning point: Lucas K¨aldstr¨om and University of Glasgow started at
same time porting Kubernetes to ARM - independently.
Lucas implemented to most features and recently became Kubernetes maintainer.
In June 2016: Kaspar Nissen and Martin Jensen from Ahrhus University were first
to run comprehensive load and high availability tests with Kubernetes on Raspberry
Pis. Most versatile investigation about Cloud Computing at small scale.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

31. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Related Work
In 2012: first public project about cloud computing with Raspberry Pis (the first
model) has been started by researchers at the University of Glasgow
LXC was state of the art, everything non-container virtualization too
resource-hungry
Largest project: Bolzano University with 300 RPis
In 2015: Turning point: Lucas K¨aldstr¨om and University of Glasgow started at
same time porting Kubernetes to ARM - independently.
Lucas implemented to most features and recently became Kubernetes maintainer.
In June 2016: Kaspar Nissen and Martin Jensen from Ahrhus University were first
to run comprehensive load and high availability tests with Kubernetes on Raspberry
Pis. Most versatile investigation about Cloud Computing at small scale.
This work’s contribution: Extend tests of Nissen/Jensen and compare K8s
against Docker Swarm.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 5/30

32. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Hardware Setup with PicoCluster
Figure 1: Pico Cluster 5xRPi 3, https://www.picocluster.com
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 6/30

33. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

34. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

35. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

36. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

37. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

38. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

39. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

40. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

41. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

42. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
Monitoring system that supports all required capabilities
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

43. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
Monitoring system that supports all required capabilities
Configure Desktop machine: No network interference, appropriate software to
manage the cluster (terminator)
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

44. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
Monitoring system that supports all required capabilities
Configure Desktop machine: No network interference, appropriate software to
manage the cluster (terminator)
Setup NTP
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

45. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
Monitoring system that supports all required capabilities
Configure Desktop machine: No network interference, appropriate software to
manage the cluster (terminator)
Setup NTP
SSH listen on two interfaces
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

46. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
Monitoring system that supports all required capabilities
Configure Desktop machine: No network interference, appropriate software to
manage the cluster (terminator)
Setup NTP
SSH listen on two interfaces
Setup Swapspace
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

47. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
Monitoring system that supports all required capabilities
Configure Desktop machine: No network interference, appropriate software to
manage the cluster (terminator)
Setup NTP
SSH listen on two interfaces
Setup Swapspace
Automate, automate, automate!
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

48. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Setup aspects
Goal: Meet all requirements for good experiments.
Hardware
All SD cards of same type
Check network switch
Isolate network
...
Software
Which OS? Raspbian vs. HypriotOS vs. ArchLinux
Flashing (I stopped counting), device-init...
Network: 2 interfaces per device, check DHCPs ...
Update Software
Monitoring system that supports all required capabilities
Configure Desktop machine: No network interference, appropriate software to
manage the cluster (terminator)
Setup NTP
SSH listen on two interfaces
Setup Swapspace
Automate, automate, automate!
...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 7/30

49. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Software Setup: Docker
Easy since included in HypriotOS.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 8/30

50. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Software Setup: Kubernetes
Using project: ”Kubernetes-on-arm”by Lucas K¨aldstr¨om
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 9/30

51. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Software Setup: Kubernetes
Using project: ”Kubernetes-on-arm”by Lucas K¨aldstr¨om
Bleeding edge: Setup non-deterministic and unreliable. Many trials, experiments
and bug shooting.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 9/30

52. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Software Setup: Kubernetes
Using project: ”Kubernetes-on-arm”by Lucas K¨aldstr¨om
Bleeding edge: Setup non-deterministic and unreliable. Many trials, experiments
and bug shooting.
Installed Kubernetes via automated interactive call:
/bin/echo -e " rpi-3nhypriotosnnode-greennEurope/Berlinn
nnynY" | sudo kube-config install
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 9/30

53. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Software Setup: Kubernetes
Using project: ”Kubernetes-on-arm”by Lucas K¨aldstr¨om
Bleeding edge: Setup non-deterministic and unreliable. Many trials, experiments
and bug shooting.
Installed Kubernetes via automated interactive call:
/bin/echo -e " rpi-3nhypriotosnnode-greennEurope/Berlinn
nnynY" | sudo kube-config install
Recently, setup became as easy as with Docker.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 9/30

54. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Software stack for experiments
Kubernetes Stack Docker Swarm Stack
rpicloud/greeting-rpi rpicloud/greeting-rpi
K8s@ARM
Docker Docker
HypriotOS HypriotOS
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 10/30

55. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Testing Scenarios
Resource Usage
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 11/30

56. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Testing Scenarios
Resource Usage
Load Test
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 11/30

57. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Testing Scenarios
Resource Usage
Load Test
Resilience Test
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 11/30

58. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – Resource Usage
Table 1: First results of resource consumption measurements
Sc. CPU usage, % Mem usage, % Proc CPU D., % Proc mem D., MB
1 0.44 4.21 0.05 2.47
2 1.43 / 0.83 7.87 / 7.4 1.1 / 0.44 57.45 / 2.42
3 14.74 / 2.01 34.21 / 12.68 5.76 / 1.31 479.51 / 160.77
Legend:
Scenario 1: Fresh install, monit and Docker daemon running
Scenario 2: After docker init
Scenario 3: After starting K8s
Also, in Scenario 3, Docker created 41 process children. In Scenario 1 & 2, Docker has 1.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 12/30

59. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
And then the Big Bang...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 13/30

60. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – HA
Kernel panics and systemd services/daemons crashed. Often reconfiguration and
reboots required.
1http://blog.bugsnag.com/container-orchestration-with-docker-swarm-mode
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 14/30

61. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – HA
Kernel panics and systemd services/daemons crashed. Often reconfiguration and
reboots required.
Mesh routing: Produce timeouts in both K8s & Docker.
1http://blog.bugsnag.com/container-orchestration-with-docker-swarm-mode
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 14/30

62. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – HA
Kernel panics and systemd services/daemons crashed. Often reconfiguration and
reboots required.
Mesh routing: Produce timeouts in both K8s & Docker.
Bugsnag: ”The intelligent routing was very spotty, sometimes hanging during a
request, and sometimes not even accepting inbound connections at all.”1
1http://blog.bugsnag.com/container-orchestration-with-docker-swarm-mode
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 14/30

63. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – HA
Kernel panics and systemd services/daemons crashed. Often reconfiguration and
reboots required.
Mesh routing: Produce timeouts in both K8s & Docker.
Bugsnag: ”The intelligent routing was very spotty, sometimes hanging during a
request, and sometimes not even accepting inbound connections at all.”1
1http://blog.bugsnag.com/container-orchestration-with-docker-swarm-mode
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 14/30

64. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – HA
Kernel panics and systemd services/daemons crashed. Often reconfiguration and
reboots required.
Mesh routing: Produce timeouts in both K8s & Docker.
Bugsnag: ”The intelligent routing was very spotty, sometimes hanging during a
request, and sometimes not even accepting inbound connections at all.”1
1http://blog.bugsnag.com/container-orchestration-with-docker-swarm-mode
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 14/30

65. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – HA
Kernel panics and systemd services/daemons crashed. Often reconfiguration and
reboots required.
Mesh routing: Produce timeouts in both K8s & Docker.
Bugsnag: ”The intelligent routing was very spotty, sometimes hanging during a
request, and sometimes not even accepting inbound connections at all.”1
→ Both tools were basically not working to the minimum extent.
1http://blog.bugsnag.com/container-orchestration-with-docker-swarm-mode
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 14/30

66. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Network Issues
Figure 2: Timeout of curl command against the cluster
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 15/30

67. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Network Issues
Figure 3: Incoming network traffic
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 16/30

68. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Network Issues
Figure 4: Incoming network traffic
Figure 5: Incoming network traffic
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 17/30

69. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Network Issues
Figure 6: One of the resolved routing issues for Docker 1.12.2
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 18/30

70. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Plan B...
Throw most of the effort away
and figure out a Plan B...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 19/30

71. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Final Software stack of experiments
Kubernetes Stack Docker Swarm Stack
(Nissen/Jensen)
rpicloud/greeting-rpi rpicloud/greeting-rpi
K8s v1.2 + K8s@ARM v0.7
Docker v1.11.x Docker v1.12.3
ArchLinux ARM latest HypriotOS v1.1.0
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 20/30

72. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results - Load Test
Figure 7: Breakpoints of load test with Kubernetes
Test has performed with 10.000 req at eatch step. Stepsize=100
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 21/30

73. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results - Load Test
Figure 8: Breakpoints of load test with Docker Swarm Mode
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 22/30

74. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – Load Test
Table 2: Breakpoints of Kubernetes and Docker Swarm Mode
Replica Count Kubernetes Docker Swarm Mode
1 ca. 900 reqests/seconds ca. 600 reqests/seconds
5 ca. 1400 reqests/seconds ca. 900 reqests/seconds
10 ca. 1500 reqests/seconds ca. 900 reqests/seconds
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 23/30

75. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results - Resilience Test
Figure 9: Successful requests onto the Kubernetes cluster with 1 replica
Kubernetes: Total outage of 47 seconds
Docker Swarm Mode: Total outage of 72 seconds (+53%)
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 24/30

76. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results - Resilience Test
Figure 10: Successful requests onto the cluster with 2 replica
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 25/30

77. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Results – Resilience Test
Table 3: Success rates of Kubernetes and Docker Swarm Mode
Replica Count Kubernetes Docker Swarm Mode
1 74.65% 61.74%
2 99.87% 99.85%
5 99.98% 99.93%
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 26/30

78. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Live Demo:
”Let’s crash the cluster!”
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 27/30

79. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Outlook
Add and test high available storage (GlusterFS, Ceph, HDFS) and extend
applications to a microservices level
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 28/30

80. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Outlook
Add and test high available storage (GlusterFS, Ceph, HDFS) and extend
applications to a microservices level
Use single board computers with higher network I/O
...
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 28/30

81. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Outlook
Add and test high available storage (GlusterFS, Ceph, HDFS) and extend
applications to a microservices level
Use single board computers with higher network I/O
...
Paper of this work will soon be released.
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 28/30

82. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Lessons Learned
Don’t trust! Measure!
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 29/30

83. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Lessons Learned
Don’t trust! Measure!
Don’t trust! Measure!
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 29/30

84. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Lessons Learned
Don’t trust! Measure!
Don’t trust! Measure!
Don’t trust! Measure!
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 29/30

85. Motivation
Why Kubernetes and
Docker?
Related Work
Test Setup
Testing Scenarios
Results
Live Demo
Outlook
Lessons Learned
Questions ?
@MathiasRenner
www.Hypriot.com
High Availability Performance of Kubernetes and Docker Swarm on a Raspberry Pi Cluster | @MathiasRenner | KTR 30/30