This document is a presentation on clustered computing with CoreOS, fleet and etcd given by Jonathan Boulle of CoreOS. It begins with introducing the speaker and his background. The bulk of the presentation covers CoreOS Linux, its self-updating operating system design and use of application containers. It also discusses fleet, CoreOS's cluster management system, and how it allows applications to remain highly available during server updates using atomic operating system updates and active/passive root file systems.
Kubernetes와 Kubernetes on OpenStack 환경의 비교와 그 구축방법에 대해서 알아봅니다.
1. 클라우드 동향
2. Kubernetes vs Kubernetes on OpenStack
3. Kubernetes on OpenStack 구축 방벙
4. Kubernetes on OpenStack 운영 방법
Kubernetes와 Kubernetes on OpenStack 환경의 비교와 그 구축방법에 대해서 알아봅니다.
1. 클라우드 동향
2. Kubernetes vs Kubernetes on OpenStack
3. Kubernetes on OpenStack 구축 방벙
4. Kubernetes on OpenStack 운영 방법
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...SaltStack
Google is making the power of its datacenter, network, and technology innovations available to the world through its Cloud services. This presentation will provide an overview of the Google Cloud Platform and a deeper dive on Google Compute Engine. Google recently made an open source contribution to SaltStack and now you can now use Salt Cloud to manage your Compute Engine resources (IaaS virtual machine services). Come find out more about Google's Cloud Platform and how you can leverage Google scale with SaltStack.
EFK Stack이란 ElasticSearch, Fluentd, Kibana라는 오픈소스의 조합으로, 방대한 양의 데이터를 신속하고 실시간으로 수집/저장/분석/시각화 할 수 있는 솔루션입니다. 특히 컨테이너 환경에서 로그 수집을 위해 주로 사용되는 기술 스택입니다.
Elasitc Stack에 대한 소개와 EFK Stack 설치 방법에 대해 설명합니다.
runC: The little engine that could (run Docker containers) by Docker Captain ...Docker, Inc.
With the announcement of the OCI by Solomon Hykes at last summer's DockerCon, a Docker-contributed reference implementation of the OCI spec, called runC, was born. While some of you may have tried runC or have a history of poking at the OS layer integration library to Linux namespaces, cgroups and the like (known as libcontainer), many of you may not know what runC offers. In this talk Phil Estes, Docker engine maintainer who has also contributed to libcontainer and runC, will show what's possible using runC as a lightweight and fast runtime environment to experiment with lower-level features of the container runtime. Phil will introduce a conversion tool called "riddler", which can inspect and convert container configurations from Docker into the proper OCI configuration bundle for easy conversion between the two environments. He'll also demonstrate how to make custom configurations for trying out security features like user namespaces and seccomp profiles.
2018년 10월 19일 금요일, 오픈스택 한국 커뮤니티 정기 세미나에서 발표주셨던 자료입니다.
- 행사 정보: http://festa.io/events/118
- 발표자: 김용기 부장님
> Sr. Solution Architect, Red Hat
> Administrator, Ansible Facebook Usergroup
OSv is a new, high-performance OS for virtual machines in the cloud. Designed to run one application per guest with minimal overhead, OSv eliminates important bottlenecks for NoSQL applications through improvements in memory management, network I/O, and scheduling. And many important bottlenecks for NoSQL applications are tunable on a conventional OS, but do not require tuning in the OSv environment.
OSv is fully stateless and can be configured at runtime with cloud-init or through a REST API, with zero configuration files. OSv offers unified tracing from the application layer through the JVM and the OS kernel. Attendees will learn how to boot Cassandra in one second, and create a simple cluster in a minute.
It's been three years since Netflix's Brendan Gregg described the Berkeley Packet Filter as "Superpowers for Linux". Since then there has been an explosion of capabilities and tools based on eBPF, so you've probably heard the term, but do you know what it is and how to use it? In this demo-rich talk we'll explore some of the powerful things we can do with this technology, especially in the context of containers.
Small, Simple, and Secure: Alpine Linux under the MicroscopeDocker, Inc.
Alpine Linux is a distro that has become popular for Docker images. Why do we need another distro? Why does Alpine matter? How does it differ from other distros?
In this talk, we'll answer all these questions – and a few more.
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Henning Jacobs
Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...SaltStack
Google is making the power of its datacenter, network, and technology innovations available to the world through its Cloud services. This presentation will provide an overview of the Google Cloud Platform and a deeper dive on Google Compute Engine. Google recently made an open source contribution to SaltStack and now you can now use Salt Cloud to manage your Compute Engine resources (IaaS virtual machine services). Come find out more about Google's Cloud Platform and how you can leverage Google scale with SaltStack.
EFK Stack이란 ElasticSearch, Fluentd, Kibana라는 오픈소스의 조합으로, 방대한 양의 데이터를 신속하고 실시간으로 수집/저장/분석/시각화 할 수 있는 솔루션입니다. 특히 컨테이너 환경에서 로그 수집을 위해 주로 사용되는 기술 스택입니다.
Elasitc Stack에 대한 소개와 EFK Stack 설치 방법에 대해 설명합니다.
runC: The little engine that could (run Docker containers) by Docker Captain ...Docker, Inc.
With the announcement of the OCI by Solomon Hykes at last summer's DockerCon, a Docker-contributed reference implementation of the OCI spec, called runC, was born. While some of you may have tried runC or have a history of poking at the OS layer integration library to Linux namespaces, cgroups and the like (known as libcontainer), many of you may not know what runC offers. In this talk Phil Estes, Docker engine maintainer who has also contributed to libcontainer and runC, will show what's possible using runC as a lightweight and fast runtime environment to experiment with lower-level features of the container runtime. Phil will introduce a conversion tool called "riddler", which can inspect and convert container configurations from Docker into the proper OCI configuration bundle for easy conversion between the two environments. He'll also demonstrate how to make custom configurations for trying out security features like user namespaces and seccomp profiles.
2018년 10월 19일 금요일, 오픈스택 한국 커뮤니티 정기 세미나에서 발표주셨던 자료입니다.
- 행사 정보: http://festa.io/events/118
- 발표자: 김용기 부장님
> Sr. Solution Architect, Red Hat
> Administrator, Ansible Facebook Usergroup
OSv is a new, high-performance OS for virtual machines in the cloud. Designed to run one application per guest with minimal overhead, OSv eliminates important bottlenecks for NoSQL applications through improvements in memory management, network I/O, and scheduling. And many important bottlenecks for NoSQL applications are tunable on a conventional OS, but do not require tuning in the OSv environment.
OSv is fully stateless and can be configured at runtime with cloud-init or through a REST API, with zero configuration files. OSv offers unified tracing from the application layer through the JVM and the OS kernel. Attendees will learn how to boot Cassandra in one second, and create a simple cluster in a minute.
It's been three years since Netflix's Brendan Gregg described the Berkeley Packet Filter as "Superpowers for Linux". Since then there has been an explosion of capabilities and tools based on eBPF, so you've probably heard the term, but do you know what it is and how to use it? In this demo-rich talk we'll explore some of the powerful things we can do with this technology, especially in the context of containers.
Small, Simple, and Secure: Alpine Linux under the MicroscopeDocker, Inc.
Alpine Linux is a distro that has become popular for Docker images. Why do we need another distro? Why does Alpine matter? How does it differ from other distros?
In this talk, we'll answer all these questions – and a few more.
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Henning Jacobs
Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.
Etcd is an open source distributed consistent key-value store developed by CoreOS. It has become a mature cornerstone of a variety of systems in the container ecosystem for networking, service discovery, configuration management and load balancing. This talk will dive into what etcd is and its history, production use cases and how it powers reliable distributed systems including Kubernetes, the container orchestration engine.
Par Jonathan Boulle (Head of Containers and Berlin site lead @ CoreOS)
Toutes les vidéos des conférences seront disponibles sur Xebia.tv
Docker orchestration using core os and ansible - Ansible IL 2015Leonid Mirsky
The last couple of years have seen an increasing interest in Docker and related technologies. One of these technologies is CoreOS, a new operating system built from the ground up for running Docker containers at scale.
In this talk we will learn about CoreOS main concepts and tools. We will get our hands dirty as we work together toward a goal of running a CoreOS cluster on AWS (using Ansible) and running docker containers on it.
The talk will conclude with a discussion on the place of Ansible (and configuration management tools in general) in the "next-generation" stack.
While etcd was born in the cloud era, it does not really play well in a dynamic environment where nodes come and go and where IP addresses are ephemeral. Moreover, etcd is meant – with its RAFT algorithm at the core – as a consistent key-value store. It rather refuses to form or join a cluster than putting concistency at risk.
As a consequence it is very conservative in implementing advanced cluster member management mechanisms – or even heuristics to make the operation of an etcd cluster more comfortable for the admin.
The elastic-etcd binary experiments with those advanced member management heuristics. It is meant as a frontend to the etcd binary, applying these heuristics and then creating a matching command line for etcd itself.
Database firewall is a useful tool that monitor databases to identify and protect against database specific attacks that mostly seek to access sensitive information stored in the databases. However the commercial database firewalls are expensive and needs specific product knowledge, while the opensource database firewalls are designed for specific opensource database servers.
In order to fulfill the need of inexpensive database firewall, Snort - an opensource IDS/IPS - is possible to achieve the goal in some scenarios with familiar rule writing. The paper will explain the limitation of Snort as a database firewall, constraints in commercial database statement and some example implementation.
OSDC 2014: Nat Morris - Open Network Install EnvironmentNETWAYS
ONIE defines an open source “install environment” that runs on this management subsystem utilizing facilities in a Linux/BusyBox environment. This environment allows end-users and channel partners to install the target network OS as part of data center provisioning, in the fashion that servers are provisioned.
ONIE enables switch hardware suppliers, distributors and resellers to manage their operations based on a small number of hardware SKUs. This in turn creates economies of scale in manufacturing, distribution, stocking, and RMA enabling a thriving ecosystem of both network hardware and operating system alternatives.
Learn how to improve the performance of your Cognos environment. We cover hardware and server specifics, architecture setup, dispatcher tuning, report specific tuning including the Interactive Performance Assistant and more. See the recording and download this deck: https://senturus.com/resources/cognos-analytics-performance-tuning/
Senturus offers a full spectrum of services for business analytics. Our Knowledge Center has hundreds of free live and recorded webinars, blog posts, demos and unbiased product reviews available on our website at: https://senturus.com/resources/
Openstack Summit Tokyo 2015 - Building a private cloud to efficiently handle ...Pierre GRANDIN
What do you do when your usual setup or turnkey solution isn’t suited for your workload?
Most of the documentation and user feedback that you can find about OpenStack is written for the use-case of running a public facing cloud serving several external customers. When you want to host a single tenant with a single application the problem is completely different, you don't want publicly exposed APIs. You want to ensure optimal resource allocation to maximize your application performance. You want to leverage the fact that you own the infrastructure layer to optimize your instance placement strategy, and to get the best latency and to avoid creating SPOFs using affinity (or anti affinity rules).
This talk will focus on what we learned during a two years journey; from getting OpenStack up and running reliably, to investigating performance bottlenecks, to maximizing the performance of our private cloud.
While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies.
This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.
Alfresco Environment Validation and "Day Zero" ConfigurationAlfresco Software
This session will commence with the environmental checks that should be performed prior to the installation of Alfresco, and then describe the "day zero" configuration changes that should be made to ensure that the installed Alfresco instance is optimally configured.
Make stateful apps in Kubernetes a no brainer with Pure Storage and GitOpsWeaveworks
The need for scale and acceleration of code to production are the main drivers behind the rapid adoption of Kubernetes in modern enterprises. Organizations are aiming to deploy thousands of cloud native applications, including stateful applications on premise, in a single cloud or across multiple clouds. Managing these workloads are complex and can often be a challenging task when it comes to automating operational tasks, rolling updates or migrations.
In this webinar Weaveworks and Pure will show you how integrated solutions such as the Weave Kubernetes Platform and Pure Service Orchestrator can save you valuable time through efficient, predictable and reliable operations.
The presentation which I was using during my talk at EPAM Lviv JS community about offline-first applications. Contains high-level review of tools and web platform to submerge folks in a world of offline-first thinking.
Rudder 3.0 was released in January 2015. This talk will bring attendees up to date on recent evolutions in Rudder and show off some of the latest and greatest features like the new compliance dashboard and graphs, redesigned web interface, built-in Technique editor (that automatically builds CFEngine code), basic command line interface, ...
We will then discuss ideas for future features. Last but not least, we should have some time to dig deeper into any parts of Rudder attendees want to know more about - examples could include reporting, ncf, OS support, CFEngine integration, ...
Practical virtual network functions with Snabb (SDN Barcelona VI)Igalia
By Andy Wingo.
SDN and Network Programmability Meetup in Barcelona (VI)
21 June 2017
https://www.meetup.com/es-ES/SDN-and-Network-Programmability-Meetup-in-Barcelona
/events/239667457/?eventId=239667457
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
4. Jonathan Boulle
@baronboulle
jonboulle
Who am I?
South Africa -> Australia -> London -> San Francisco
Red Hat -> Twitter -> CoreOS
5. Jonathan Boulle
@baronboulle
jonboulle
Who am I?
South Africa -> Australia -> London -> San Francisco
Red Hat -> Twitter -> CoreOS
Linux, Python, Go, FOSS
9. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
10. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
11. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
12. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
– cluster-level init system
13. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
– cluster-level init system
– etcd + systemd
14. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
– cluster-level init system
– etcd + systemd
● fleet and...
15. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
– cluster-level init system
– etcd + systemd
● fleet and...
– systemd: the good, the bad
16. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
– cluster-level init system
– etcd + systemd
● fleet and...
– systemd: the good, the bad
– etcd: the good, the bad
17. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
– cluster-level init system
– etcd + systemd
● fleet and...
– systemd: the good, the bad
– etcd: the good, the bad
– Golang: the good, the bad
18. Agenda
● CoreOS Linux
– Securing the internet
– Application containers
– Automatic updates
● fleet
– cluster-level init system
– etcd + systemd
● fleet and...
– systemd: the good, the bad
– etcd: the good, the bad
– Golang: the good, the bad
● Q&A
24. Why?
● CoreOS mission: “Secure the internet”
● Status quo: set up a server and never touch it
25. Why?
● CoreOS mission: “Secure the internet”
● Status quo: set up a server and never touch it
● Internet is full of servers running years-old
software with dozens of vulnerabilities
26. Why?
● CoreOS mission: “Secure the internet”
● Status quo: set up a server and never touch it
● Internet is full of servers running years-old
software with dozens of vulnerabilities
● CoreOS: make updating the default, seamless
option
27. Why?
● CoreOS mission: “Secure the internet”
● Status quo: set up a server and never touch it
● Internet is full of servers running years-old
software with dozens of vulnerabilities
● CoreOS: make updating the default, seamless
option
● Regular
28. Why?
● CoreOS mission: “Secure the internet”
● Status quo: set up a server and never touch it
● Internet is full of servers running years-old software
with dozens of vulnerabilities
● CoreOS: make updating the default, seamless option
● Regular
● Reliable
29. Why?
● CoreOS mission: “Secure the internet”
● Status quo: set up a server and never touch it
● Internet is full of servers running years-old software
with dozens of vulnerabilities
● CoreOS: make updating the default, seamless option
● Regular
● Reliable
● Automatic
31. How do we achieve this?
● Containerization of applications
32. How do we achieve this?
● Containerization of applications
● Self-updating operating system
33. How do we achieve this?
● Containerization of applications
● Self-updating operating system
● Distributed systems tooling to make applications
resilient to updates
41. Automatic updates
How do updates work?
● Omaha protocol (check-in/retrieval)
– Simple XML-over-HTTP protocol developed by
Google to facilitate polling and pulling updates
from a server
42. Omaha protocol
Client sends application id and
current version to the update server
<request protocol="3.0" version="CoreOSUpdateEngine-0.1.0.0">
<app appid="{e96281a6-d1af-4bde-9a0a-97b76e56dc57}"
version="410.0.0" track="alpha" from_track="alpha">
<event eventtype="3"></event>
</app>
</request>
43. Omaha protocol
Client sends application id and
current version to the update server
Update server responds with the
URL of an update to be applied
<url codebase="https://commondatastorage.googleapis.com/update-storage.
core-os.net/amd64-usr/452.0.0/"></url>
<package hash="D0lBAMD1Fwv8YqQuDYEAjXw6YZY=" name="update.gz"
size="103401155" required="false">
44. Omaha protocol
Client sends application id and
current version to the update server
Update server responds with the
URL of an update to be applied
Client downloads data, verifies
hash & cryptographic signature,
and applies the update
45. Omaha protocol
Client sends application id and
current version to the update server
Update server responds with the
URL of an update to be applied
Client downloads data, verifies
hash & cryptographic signature,
and applies the update
Updater exits with response code
then reports the update to the
update server
47. Automatic updates
How do updates work?
● Omaha protocol (check-in/retrieval)
– Simple XML-over-HTTP protocol developed by
Google to facilitate polling and pulling updates
from a server
● Active/passive read-only root partitions
48. Automatic updates
How do updates work?
● Omaha protocol (check-in/retrieval)
– Simple XML-over-HTTP protocol developed by
Google to facilitate polling and pulling updates from
a server
● Active/passive read-only root partitions
– One for running the live system, one for updates
60. Active/passive /usr partitions
● Single image containing most of the OS
– Mounted read-only onto /usr
– / is mounted read-write on top (persistent data)
61. Active/passive /usr partitions
● Single image containing most of the OS
– Mounted read-only onto /usr
– / is mounted read-write on top (persistent data)
– Parts of /etc generated dynamically at boot
62. Active/passive /usr partitions
● Single image containing most of the OS
– Mounted read-only onto /usr
– / is mounted read-write on top (persistent data)
– Parts of /etc generated dynamically at boot
– A lot of work moving default configs from /etc to /usr
63. Active/passive /usr partitions
● Single image containing most of the OS
– Mounted read-only onto /usr
– / is mounted read-write on top (persistent data)
– Parts of /etc generated dynamically at boot
– A lot of work moving default configs from /etc to /usr
# /etc/nsswitch.conf:
passwd: files usrfiles
shadow: files usrfiles
group: files usrfiles
64. Active/passive /usr partitions
● Single image containing most of the OS
– Mounted read-only onto /usr
– / is mounted read-write on top (persistent data)
– Parts of /etc generated dynamically at boot
– A lot of work moving default configs from /etc to /usr
# /etc/nsswitch.conf:
passwd: files usrfiles
/usr/share/baselayout/passwd
shadow: files usrfiles
/usr/share/baselayout/shadow
group: files usrfiles
/usr/share/baselayout/group
66. Atomic updates
● Entire OS is a single read-only image
core-01 ~ # touch /usr/bin/foo
touch: cannot touch '/usr/bin/foo': Read-only file system
67. Atomic updates
● Entire OS is a single read-only image
core-01 ~ # touch /usr/bin/foo
touch: cannot touch '/usr/bin/foo': Read-only file system
– Easy to verify cryptographically
● sha1sum on AWS or bare metal gives the same result
68. Atomic updates
● Entire OS is a single read-only image
core-01 ~ # touch /usr/bin/foo
touch: cannot touch '/usr/bin/foo': Read-only file system
– Easy to verify cryptographically
● sha1sum on AWS or bare metal gives the same result
– No chance of inconsistencies due to partial updates
● e.g. pull a plug on a CentOS system during a yum update
● At large scale, such events are inevitable
71. Automatic, atomic updates are great! But...
● Problem:
– updates still require a reboot (to use new kernel and
mount new filesystem)
72. Automatic, atomic updates are great! But...
● Problem:
– updates still require a reboot (to use new kernel and
mount new filesystem)
– Reboots cause application downtime...
73. Automatic, atomic updates are great! But...
● Problem:
– updates still require a reboot (to use new kernel and
mount new filesystem)
– Reboots cause application downtime...
● Solution: fleet
74. Automatic, atomic updates are great! But...
● Problem:
– updates still require a reboot (to use new kernel and
mount new filesystem)
– Reboots cause application downtime...
● Solution: fleet
– Highly available, fault tolerant, distributed process
scheduler
75. Automatic, atomic updates are great! But...
● Problem:
– updates still require a reboot (to use new kernel and
mount new filesystem)
– Reboots cause application downtime...
● Solution: fleet
– Highly available, fault tolerant, distributed process
scheduler
● ... The way to run applications on a CoreOS cluster
76. Automatic, atomic updates are great! But...
● Problem:
– updates still require a reboot (to use new kernel and mount
new filesystem)
– Reboots cause application downtime...
● Solution: fleet
– Highly available, fault tolerant, distributed process scheduler
● ... The way to run applications on a CoreOS cluster
– fleet keeps applications running during server downtime
78. fleet – the “cluster-level init system”
fleet is the abstraction between machine and
application:
79. fleet – the “cluster-level init system”
fleet is the abstraction between machine and
application:
– init system manages process on a machine
– fleet manages applications on a cluster of machines
80. fleet – the “cluster-level init system”
fleet is the abstraction between machine and
application:
– init system manages process on a machine
– fleet manages applications on a cluster of machines
Similar to Mesos, but very different architecture
(e.g. based on etcd/Raft, not Zookeeper/Paxos)
81. fleet – the “cluster-level init system”
fleet is the abstraction between machine and
application:
– init system manages process on a machine
– fleet manages applications on a cluster of machines
Similar to Mesos, but very different architecture
(e.g. based on etcd/Raft, not Zookeeper/Paxos)
Uses systemd for machine-level process
management, etcd for cluster-level co-ordination
83. fleet – low level view
fleetd binary (running on all CoreOS nodes)
84. fleet – low level view
fleetd binary (running on all CoreOS nodes)
– encapsulates two roles:
85. fleet – low level view
fleetd binary (running on all CoreOS nodes)
– encapsulates two roles:
• engine (cluster-level unit scheduling – talks to etcd)
86. fleet – low level view
fleetd binary (running on all CoreOS nodes)
– encapsulates two roles:
• engine (cluster-level unit scheduling – talks to etcd)
• agent (local unit management – talks to etcd and systemd)
87. fleet – low level view
fleetd binary (running on all CoreOS nodes)
– encapsulates two roles:
• engine (cluster-level unit scheduling – talks to etcd)
• agent (local unit management – talks to etcd and systemd)
fleetctl command-line administration tool
88. fleet – low level view
fleetd binary (running on all CoreOS nodes)
– encapsulates two roles:
• engine (cluster-level unit scheduling – talks to etcd)
• agent (local unit management – talks to etcd and systemd)
fleetctl command-line administration tool
– create, destroy, start, stop units
89. fleet – low level view
fleetd binary (running on all CoreOS nodes)
– encapsulates two roles:
• engine (cluster-level unit scheduling – talks to etcd)
• agent (local unit management – talks to etcd and systemd)
fleetctl command-line administration tool
– create, destroy, start, stop units
– Retrieve current status of units/machines in the cluster
90. fleet – low level view
fleetd binary (running on all CoreOS nodes)
– encapsulates two roles:
• engine (cluster-level unit scheduling – talks to etcd)
• agent (local unit management – talks to etcd and systemd)
fleetctl command-line administration tool
– create, destroy, start, stop units
– Retrieve current status of units/machines in the cluster
HTTP API
94. systemd
Linux init system (PID 1) – manages processes
– Relatively new, replaces SysVinit, upstart, OpenRC, ...
95. systemd
Linux init system (PID 1) – manages processes
– Relatively new, replaces SysVinit, upstart, OpenRC, ...
– Being adopted by all major Linux distributions
96. systemd
Linux init system (PID 1) – manages processes
– Relatively new, replaces SysVinit, upstart, OpenRC, ...
– Being adopted by all major Linux distributions
Fundamental concept is the unit
97. systemd
Linux init system (PID 1) – manages processes
– Relatively new, replaces SysVinit, upstart, OpenRC, ...
– Being adopted by all major Linux distributions
Fundamental concept is the unit
– Units include services (e.g. applications), mount points,
sockets, timers, etc.
98. systemd
Linux init system (PID 1) – manages processes
– Relatively new, replaces SysVinit, upstart, OpenRC, ...
– Being adopted by all major Linux distributions
Fundamental concept is the unit
– Units include services (e.g. applications), mount points,
sockets, timers, etc.
– Each unit configured with a simple unit file
102. fleet + systemd
systemd exposes a D-Bus interface
– D-Bus: message bus system for IPC on Linux
103. fleet + systemd
systemd exposes a D-Bus interface
– D-Bus: message bus system for IPC on Linux
– One-to-one messaging (methods), plus pub/sub abilities
104. fleet + systemd
systemd exposes a D-Bus interface
– D-Bus: message bus system for IPC on Linux
– One-to-one messaging (methods), plus pub/sub abilities
fleet uses godbus to communicate with systemd
105. fleet + systemd
systemd exposes a D-Bus interface
– D-Bus: message bus system for IPC on Linux
– One-to-one messaging (methods), plus pub/sub abilities
fleet uses godbus to communicate with systemd
– Sending commands: StartUnit, StopUnit
106. fleet + systemd
systemd exposes a D-Bus interface
– D-Bus: message bus system for IPC on Linux
– One-to-one messaging (methods), plus pub/sub abilities
fleet uses godbus to communicate with systemd
– Sending commands: StartUnit, StopUnit
– Retrieving current state of units (to publish to the
cluster)
117. fleet + systemd
systemd takes care of things so we don't have to
118. fleet + systemd
systemd takes care of things so we don't have to
fleet configuration is just systemd unit files
119. fleet + systemd
systemd takes care of things so we don't have to
fleet configuration is just systemd unit files
fleet extends systemd to the cluster-level, and adds
some features of its own (using [X-Fleet])
120. fleet + systemd
systemd takes care of things so we don't have to
fleet configuration is just systemd unit files
fleet extends systemd to the cluster-level, and adds
some features of its own (using [X-Fleet])
– Template units (run n identical copies of a unit)
121. fleet + systemd
systemd takes care of things so we don't have to
fleet configuration is just systemd unit files
fleet extends systemd to the cluster-level, and adds
some features of its own (using [X-Fleet])
– Template units (run n identical copies of a unit)
– Global units (run a unit everywhere in the cluster)
122. fleet + systemd
systemd takes care of things so we don't have to
fleet configuration is just systemd unit files
fleet extends systemd to the cluster-level, and adds
some features of its own (using [X-Fleet])
– Template units (run n identical copies of a unit)
– Global units (run a unit everywhere in the cluster)
– Machine metadata (run only on certain machines)
125. systemd is... not so great
Problem: unreliable pub/sub
– fleet agent initially used a systemd D-Bus subscription
to track unit status
126. systemd is... not so great
Problem: unreliable pub/sub
– fleet agent initially used a systemd D-Bus subscription
to track unit status
– Every change in unit state in systemd triggers an event
in fleet (e.g. “publish this new state to the cluster”)
127. systemd is... not so great
Problem: unreliable pub/sub
– fleet agent initially used a systemd D-Bus subscription
to track unit status
– Every change in unit state in systemd triggers an event
in fleet (e.g. “publish this new state to the cluster”)
– Under heavy load, or byzantine conditions, unit state
changes would be dropped
128. systemd is... not so great
Problem: unreliable pub/sub
– fleet agent initially used a systemd D-Bus subscription
to track unit status
– Every change in unit state in systemd triggers an event
in fleet (e.g. “publish this new state to the cluster”)
– Under heavy load, or byzantine conditions, unit state
changes would be dropped
– As a result, unit state in the cluster became stale
131. systemd is... not so great
Problem: unreliable pub/sub
Solution: polling for unit states
132. systemd is... not so great
Problem: unreliable pub/sub
Solution: polling for unit states
– Every n seconds, retrieve state of units from systemd,
and synchronize with cluster
133. systemd is... not so great
Problem: unreliable pub/sub
Solution: polling for unit states
– Every n seconds, retrieve state of units from systemd,
and synchronize with cluster
– Less efficient, but much more reliable
134. systemd is... not so great
Problem: unreliable pub/sub
Solution: polling for unit states
– Every n seconds, retrieve state of units from systemd,
and synchronize with cluster
– Less efficient, but much more reliable
– Optimize by caching state and only publishing changes
135. systemd is... not so great
Problem: unreliable pub/sub
Solution: polling for unit states
– Every n seconds, retrieve state of units from systemd,
and synchronize with cluster
– Less efficient, but much more reliable
– Optimize by caching state and only publishing changes
– Any state inconsistencies are quickly fixed
137. systemd (and docker) are... not so great
Problem: poor integration with Docker
138. systemd (and docker) are... not so great
Problem: poor integration with Docker
– Docker is de facto application container manager
139. systemd (and docker) are... not so great
Problem: poor integration with Docker
– Docker is de facto application container manager
– Docker and systemd do not always play nicely together..
140. systemd (and docker) are... not so great
Problem: poor integration with Docker
– Docker is de facto application container manager
– Docker and systemd do not always play nicely together..
– Both Docker and systemd manage cgroups and
processes, and when the two are trying to manage the
same thing, the results are mixed
142. systemd (and docker) are... not so great
Example: sending signals to a container
143. systemd (and docker) are... not so great
Example: sending signals to a container
– Given a simple container:
[Service]
ExecStart=/usr/bin/docker run busybox /bin/bash -c
"while true; do echo Hello World; sleep 1; done"
144. systemd (and docker) are... not so great
Example: sending signals to a container
– Given a simple container:
[Service]
ExecStart=/usr/bin/docker run busybox /bin/bash -c
"while true; do echo Hello World; sleep 1; done"
– Try to kill it with systemctl kill hello.service
145. systemd (and docker) are... not so great
Example: sending signals to a container
– Given a simple container:
[Service]
ExecStart=/usr/bin/docker run busybox /bin/bash -c
"while true; do echo Hello World; sleep 1; done"
– Try to kill it with systemctl kill hello.service
– ... Nothing happens
146. systemd (and docker) are... not so great
Example: sending signals to a container
– Given a simple container:
[Service]
ExecStart=/usr/bin/docker run busybox /bin/bash -c
"while true; do echo Hello World; sleep 1; done"
– Try to kill it with systemctl kill hello.service
– ... Nothing happens
– Kill command sends SIGTERM, but bash in a Docker
container has PID1, which happily ignores the signal...
148. systemd (and docker) are... not so great
Example: sending signals to a container
149. systemd (and docker) are... not so great
Example: sending signals to a container
OK, SIGTERM didn't work, so escalate to SIGKILL:
systemctl kill -s SIGKILL hello.service
150. systemd (and docker) are... not so great
Example: sending signals to a container
OK, SIGTERM didn't work, so escalate to SIGKILL:
systemctl kill -s SIGKILL hello.service
● Now the systemd service is gone:
hello.service: main process exited, code=killed,
status=9/KILL
151. systemd (and docker) are... not so great
Example: sending signals to a container
OK, SIGTERM didn't work, so escalate to SIGKILL:
systemctl kill -s SIGKILL hello.service
● Now the systemd service is gone:
hello.service: main process exited, code=killed, status=9/KILL
● But... the Docker container still exists?
# docker ps
CONTAINER ID COMMAND STATUS NAMES
7c7cf8ffabb6 /bin/sh -c 'while tr Up 31 seconds hello
# ps -ef|grep '[d]ocker run'
root 24231 1 0 03:49 ? 00:00:00 /usr/bin/docker run -name hello ...
154. systemd (and docker) are... not so great
Why?
– Docker client does not run containers itself; it just sends
a command to the Docker daemon which actually forks
and starts running the container
155. systemd (and docker) are... not so great
Why?
– Docker client does not run containers itself; it just sends
a command to the Docker daemon which actually forks
and starts running the container
– systemd expects processes to fork directly so they will
be contained under the same cgroup tree
156. systemd (and docker) are... not so great
Why?
– Docker client does not run containers itself; it just sends
a command to the Docker daemon which actually forks
and starts running the container
– systemd expects processes to fork directly so they will
be contained under the same cgroup tree
– Since the Docker daemon's cgroup is entirely separate,
systemd cannot keep track of the forked container
157. systemd (and docker) are... not so great
# systemctl cat hello.service
[Service]
ExecStart=/bin/bash -c 'while true; do echo Hello World; sleep 1; done'
# systemd-cgls
...
├─hello.service
│ ├─23201 /bin/bash -c while true; do echo Hello World; sleep 1; done
│ └─24023 sleep 1
158. systemd (and docker) are... not so great
# systemctl cat hello.service
[Service]
ExecStart=/usr/bin/docker run -name hello busybox /bin/sh -c
"while true; do echo Hello World; sleep 1; done"
# systemd-cgls
...
│ ├─hello.service
│ │ └─24231 /usr/bin/docker run -name hello busybox /bin/sh -c while
true; do echo Hello World; sleep 1; done
...
│ ├─docker-
51a57463047b65487ec80a1dc8b8c9ea14a396c7a49c1e23919d50bdafd4fefb.scope
│ │ ├─24240 /bin/sh -c while true; do echo Hello World; sleep 1; done
│ │ └─24553 sleep 1
160. systemd (and docker) are... not so great
Problem: poor integration with Docker
Solution: ... work in progress
161. systemd (and docker) are... not so great
Problem: poor integration with Docker
Solution: ... work in progress
– systemd-docker – small application that moves cgroups
of Docker containers back under systemd's cgroup
162. systemd (and docker) are... not so great
Problem: poor integration with Docker
Solution: ... work in progress
– systemd-docker – small application that moves cgroups
of Docker containers back under systemd's cgroup
– Use Docker for image management, but systemd-nspawn
for runtime (e.g. CoreOS's toolbox)
163. systemd (and docker) are... not so great
Problem: poor integration with Docker
Solution: ... work in progress
– systemd-docker – small application that moves cgroups
of Docker containers back under systemd's cgroup
– Use Docker for image management, but systemd-nspawn
for runtime (e.g. CoreOS's toolbox)
– (proposed) Docker standalone mode: client starts
container directly rather than through daemon
166. etcd
A consistent, highly available key/value store
167. etcd
A consistent, highly available key/value store
– Shared configuration, distributed locking, ...
168. etcd
A consistent, highly available key/value store
– Shared configuration, distributed locking, ...
Driven by Raft
169. etcd
A consistent, highly available key/value store
– Shared configuration, distributed locking, ...
Driven by Raft
Consensus algorithm similar to Paxos
170. etcd
A consistent, highly available key/value store
– Shared configuration, distributed locking, ...
Driven by Raft
Consensus algorithm similar to Paxos
Designed for understandability and simplicity
171. etcd
A consistent, highly available key/value store
– Shared configuration, distributed locking, ...
Driven by Raft
Consensus algorithm similar to Paxos
Designed for understandability and simplicity
Popular and widely used
172. etcd
A consistent, highly available key/value store
– Shared configuration, distributed locking, ...
Driven by Raft
Consensus algorithm similar to Paxos
Designed for understandability and simplicity
Popular and widely used
– Simple HTTP API + libraries in Go, Java, Python, Ruby, ...
175. fleet + etcd
● fleet needs a consistent view of the cluster to make
scheduling decisions: etcd provides this view
176. fleet + etcd
● fleet needs a consistent view of the cluster to make
scheduling decisions: etcd provides this view
– What units exist in the cluster?
177. fleet + etcd
● fleet needs a consistent view of the cluster to make
scheduling decisions: etcd provides this view
– What units exist in the cluster?
– What machines exist in the cluster?
178. fleet + etcd
● fleet needs a consistent view of the cluster to make
scheduling decisions: etcd provides this view
– What units exist in the cluster?
– What machines exist in the cluster?
– What are their current states?
179. fleet + etcd
● fleet needs a consistent view of the cluster to make
scheduling decisions: etcd provides this view
– What units exist in the cluster?
– What machines exist in the cluster?
– What are their current states?
● All unit files, unit state, machine state and
scheduling information is stored in etcd
182. etcd is great!
● Fast and simple API
● Handles all cluster-level/inter-machine
communication so we don't have to
183. etcd is great!
● Fast and simple API
● Handles all cluster-level/inter-machine
communication so we don't have to
● Powerful primitives:
184. etcd is great!
● Fast and simple API
● Handles all cluster-level/inter-machine
communication so we don't have to
● Powerful primitives:
– Compare-and-Swap allows for atomic operations and
implementing locking behaviour
185. etcd is great!
● Fast and simple API
● Handles all cluster-level/inter-machine
communication so we don't have to
● Powerful primitives:
– Compare-and-Swap allows for atomic operations and
implementing locking behaviour
– Watches (like pub-sub) provide event-driven behaviour
188. etcd is... not so great
● Problem: unreliable watches
– fleet initially used a purely event-driven architecture
189. etcd is... not so great
● Problem: unreliable watches
– fleet initially used a purely event-driven architecture
– watches in etcd used to trigger events
190. etcd is... not so great
● Problem: unreliable watches
– fleet initially used a purely event-driven architecture
– watches in etcd used to trigger events
● e.g. “machine down” event triggers unit rescheduling
191. etcd is... not so great
● Problem: unreliable watches
– fleet initially used a purely event-driven architecture
– watches in etcd used to trigger events
● e.g. “machine down” event triggers unit rescheduling
– Highly efficient: only take action when necessary
192. etcd is... not so great
● Problem: unreliable watches
– fleet initially used a purely event-driven architecture
– watches in etcd used to trigger events
● e.g. “machine down” event triggers unit rescheduling
– Highly efficient: only take action when necessary
– Highly responsive: as soon as a user submits a new
unit, an event is triggered to schedule that unit
193. etcd is... not so great
● Problem: unreliable watches
– fleet initially used a purely event-driven architecture
– watches in etcd used to trigger events
● e.g. “machine down” event triggers unit rescheduling
– Highly efficient: only take action when necessary
– Highly responsive: as soon as a user submits a new
unit, an event is triggered to schedule that unit
– Unfortunately, many places for things to go wrong...
196. ● Example:
Unreliable watches
– etcd is undergoing a leader election or otherwise
unavailable, watches do not work during this period
197. ● Example:
Unreliable watches
– etcd is undergoing a leader election or otherwise
unavailable, watches do not work during this period
– Change occurs (e.g. a machine leaves the cluster)
198. ● Example:
Unreliable watches
– etcd is undergoing a leader election or otherwise
unavailable, watches do not work during this period
– Change occurs (e.g. a machine leaves the cluster)
– Event is missed
199. ● Example:
Unreliable watches
– etcd is undergoing a leader election or otherwise
unavailable, watches do not work during this period
– Change occurs (e.g. a machine leaves the cluster)
– Event is missed
– fleet doesn't know machine is lost!
200. ● Example:
Unreliable watches
– etcd is undergoing a leader election or otherwise
unavailable, watches do not work during this period
– Change occurs (e.g. a machine leaves the cluster)
– Event is missed
– fleet doesn't know machine is lost!
– Now fleet doesn't know to reschedule units that were
running on that machine
202. etcd is... not so great
● Problem: limited event history
203. etcd is... not so great
● Problem: limited event history
– etcd retains a history of all events that occur
204. etcd is... not so great
● Problem: limited event history
– etcd retains a history of all events that occur
– Can “watch” from an arbitrary point in the past, but..
205. etcd is... not so great
● Problem: limited event history
– etcd retains a history of all events that occur
– Can “watch” from an arbitrary point in the past, but..
– History is a limited window!
206. etcd is... not so great
● Problem: limited event history
– etcd retains a history of all events that occur
– Can “watch” from an arbitrary point in the past, but..
– History is a limited window!
– With a busy cluster, watches can fall out of this window
207. etcd is... not so great
● Problem: limited event history
– etcd retains a history of all events that occur
– Can “watch” from an arbitrary point in the past, but..
– History is a limited window!
– With a busy cluster, watches can fall out of this window
– Can't always replay event stream from the point we
want to
209. ● Example:
Limited event history
– etcd holds history of last 1000 events
210. ● Example:
Limited event history
– etcd holds history of last 1000 events
– fleet sets watch at i=100 to watch for machine loss
211. ● Example:
Limited event history
– etcd holds history of last 1000 events
– fleet sets watch at i=100 to watch for machine loss
– Meanwhile, many changes occur in other parts of the
keyspace, advancing index to i=1500
212. ● Example:
Limited event history
– etcd holds history of last 1000 events
– fleet sets watch at i=100 to watch for machine loss
– Meanwhile, many changes occur in other parts of the
keyspace, advancing index to i=1500
– Leader election/network hiccup occurs and severs
watch
213. ● Example:
Limited event history
– etcd holds history of last 1000 events
– fleet sets watch at i=100 to watch for machine loss
– Meanwhile, many changes occur in other parts of the
keyspace, advancing index to i=1500
– Leader election/network hiccup occurs and severs
watch
– fleet tries to recreate watch at i=100 and fails:
214. ● Example:
Limited event history
– etcd holds history of last 1000 events
– fleet sets watch at i=100 to watch for machine loss
– Meanwhile, many changes occur in other parts of the
keyspace, advancing index to i=1500
– Leader election/network hiccup occurs and severs watch
– fleet tries to recreate watch at i=100 and fails:
err="401: The event in requested index is outdated and
cleared (the requested history has been cleared [1500/100])
216. etcd is... not so great
● Problem: unreliable watches
– Missed events lead to unrecoverable situations
● Problem: limited event history
– Can't always replay entire event stream
217. etcd is... not so great
● Problem: unreliable watches
– Missed events lead to unrecoverable situations
● Problem: limited event history
– Can't always replay entire event stream
● Solution: move to “reconciler” model
220. Reconciler model
In a loop, run periodically until stopped:
1.Retrieve current state (how the world is) and desired state
(how the world should be) from datastore (etcd)
221. Reconciler model
In a loop, run periodically until stopped:
1.Retrieve current state (how the world is) and desired state
(how the world should be) from datastore (etcd)
2.Determine necessary actions to transform current state -->
desired state
222. Reconciler model
In a loop, run periodically until stopped:
1.Retrieve current state (how the world is) and desired state
(how the world should be) from datastore (etcd)
2.Determine necessary actions to transform current state -->
desired state
3.Perform actions and save results as new current state
223. Reconciler model
Example: fleet's engine (scheduler) looks something like:
for { // loop forever
select {
case <- stopChan: // if stopped, exit
return
case <- time.After(5 * time.Minute):
units = fetchUnits()
machines = fetchMachines()
schedule(units, machines)
}
}
225. etcd is... not so great
● Problem: unreliable watches
– Missed events lead to unrecoverable situations
● Problem: limited event history
– Can't always replay entire event stream
● Solution: move to “reconciler” model
226. etcd is... not so great
● Problem: unreliable watches
– Missed events lead to unrecoverable situations
● Problem: limited event history
– Can't always replay entire event stream
● Solution: move to “reconciler” model
– Less efficient, but extremely robust
227. etcd is... not so great
● Problem: unreliable watches
– Missed events lead to unrecoverable situations
● Problem: limited event history
– Can't always replay entire event stream
● Solution: move to “reconciler” model
– Less efficient, but extremely robust
– Still many paths for optimisation (e.g. using watches to
trigger reconciliations)
232. ● Standard language for all CoreOS projects (above OS)
– etcd
– fleet
golang
233. golang
● Standard language for all CoreOS projects (above OS)
– etcd
– fleet
– locksmith (semaphore for reboots during updates)
234. golang
● Standard language for all CoreOS projects (above OS)
– etcd
– fleet
– locksmith (semaphore for reboots during updates)
– etcdctl, updatectl, coreos-cloudinit, ...
235. golang
● Standard language for all CoreOS projects (above OS)
– etcd
– fleet
– locksmith (semaphore for reboots during updates)
– etcdctl, updatectl, coreos-cloudinit, ...
● fleet is ~10k LOC (and another ~10k LOC tests)
238. ● Fast!
Go is great!
– to write (concise syntax)
239. ● Fast!
Go is great!
– to write (concise syntax)
– to compile (builds typically <1s)
240. ● Fast!
Go is great!
– to write (concise syntax)
– to compile (builds typically <1s)
– to run tests (O(seconds), including with race detection)
241. ● Fast!
Go is great!
– to write (concise syntax)
– to compile (builds typically <1s)
– to run tests (O(seconds), including with race detection)
– Never underestimate power of rapid iteration
242. ● Fast!
Go is great!
– to write (concise syntax)
– to compile (builds typically <1s)
– to run tests (O(seconds), including with race detection)
– Never underestimate power of rapid iteration
● Simple, powerful tooling
243. ● Fast!
Go is great!
– to write (concise syntax)
– to compile (builds typically <1s)
– to run tests (O(seconds), including with race detection)
– Never underestimate power of rapid iteration
● Simple, powerful tooling
– Built in package management, code coverage, etc.
246. Go is great!
● Rich standard library
– “Batteries are included”
247. Go is great!
● Rich standard library
– “Batteries are included”
– e.g.: completely self-hosted HTTP server, no need for
reverse proxies or worker systems to serve many
concurrent HTTP requests
248. Go is great!
● Rich standard library
– “Batteries are included”
– e.g.: completely self-hosted HTTP server, no need for
reverse proxies or worker systems to serve many
concurrent HTTP requests
● Static compilation into a single binary
249. Go is great!
● Rich standard library
– “Batteries are included”
– e.g.: completely self-hosted HTTP server, no need for
reverse proxies or worker systems to serve many
concurrent HTTP requests
● Static compilation into a single binary
– Ideal for a minimal OS with no libraries
251. Go is... not so great
● Problem: managing third-party dependencies
252. Go is... not so great
● Problem: managing third-party dependencies
– modular package management but: no versioning
253. Go is... not so great
● Problem: managing third-party dependencies
– modular package management but: no versioning
– import “github.com/coreos/fleet” - which SHA?
254. Go is... not so great
● Problem: managing third-party dependencies
– modular package management but: no versioning
– import “github.com/coreos/fleet” - which SHA?
● Solution: vendoring :-/
255. Go is... not so great
● Problem: managing third-party dependencies
– modular package management but: no versioning
– import “github.com/coreos/fleet” - which SHA?
● Solution: vendoring :-/
– Copy entire source tree of dependencies into repository
256. Go is... not so great
● Problem: managing third-party dependencies
– modular package management but: no versioning
– import “github.com/coreos/fleet” - which SHA?
● Solution: vendoring :-/
– Copy entire source tree of dependencies into repository
– Slowly maturing tooling: goven, third_party.go, Godep
258. Go is... not so great
● Problem: (relatively) large binary sizes
259. Go is... not so great
● Problem: (relatively) large binary sizes
– “relatively”, but... CoreOS is the minimal OS
260. Go is... not so great
● Problem: (relatively) large binary sizes
– “relatively”, but... CoreOS is the minimal OS
– ~10MB per binary, many tools, quickly adds up
261. ● Problem: (relatively) large binary sizes
– “relatively”, but... CoreOS is the minimal OS
– ~10MB per binary, many tools, quickly adds up
● Solutions:
Go is... not so great
262. Go is... not so great
● Problem: (relatively) large binary sizes
– “relatively”, but... CoreOS is the minimal OS
– ~10MB per binary, many tools, quickly adds up
● Solutions:
– upgrading golang!
263. Go is... not so great
● Problem: (relatively) large binary sizes
– “relatively”, but... CoreOS is the minimal OS
– ~10MB per binary, many tools, quickly adds up
● Solutions:
– upgrading golang!
● go1.2 to go1.3 = ~25% reduction
264. Go is... not so great
● Problem: (relatively) large binary sizes
– “relatively”, but... CoreOS is the minimal OS
– ~10MB per binary, many tools, quickly adds up
● Solutions:
– upgrading golang!
● go1.2 to go1.3 = ~25% reduction
– sharing the binary between tools
266. Sharing a binary
● client/daemon often share much of the same code
267. Sharing a binary
● client/daemon often share much of the same code
– Encapsulate multiple tools in one binary, symlink the
different command names, switch off command name
268. Sharing a binary
● client/daemon often share much of the same code
– Encapsulate multiple tools in one binary, symlink the
different command names, switch off command name
– Example: fleetd/fleetctl
269. Sharing a binary
● client/daemon often share much of the same code
– Encapsulate multiple tools in one binary, symlink the
different command names, switch off command name
– Example: fleetd/fleetctl
func main() {
switch os.Args[0] {
case “fleetctl”:
Fleetctl()
case “fleetd”:
Fleetd()
}
}
270. Sharing a binary
● client/daemon often share much of the same code
– Encapsulate multiple tools in one binary, symlink the
different command names, switch off command name
– Example: fleetd/fleetctl
func main() {
switch os.Args[0] {
case “fleetctl”:
Fleetctl()
case “fleetd”:
Fleetd()
}
}
Before:
9150032 fleetctl
8567416 fleetd
After:
11052256 fleetctl
8 fleetd -> fleetctl
272. Go is... not so great
● Problem: young language => immature libraries
273. Go is... not so great
● Problem: young language => immature libraries
– CLI frameworks
274. Go is... not so great
● Problem: young language => immature libraries
– CLI frameworks
– godbus, go-systemd, go-etcd :-(
275. ● Problem: young language => immature libraries
– CLI frameworks
– godbus, go-systemd, go-etcd :-(
● Solutions:
Go is... not so great
276. Go is... not so great
● Problem: young language => immature libraries
– CLI frameworks
– godbus, go-systemd, go-etcd :-(
● Solutions:
– Keep it simple
277. Go is... not so great
● Problem: young language => immature libraries
– CLI frameworks
– godbus, go-systemd, go-etcd :-(
● Solutions:
– Keep it simple
– Roll your own (e.g. fleetctl's command line)
278. type Command struct {
Name string
Summary string
Usage string
Description string
Flags flag.FlagSet
Run func(args []string) int
}
fleetctl CLI
288. Thank you :-)
● Everything is open source – join us!
– https://github.com/coreos
● Any more questions, feel free to email
– jonathan.boulle@coreos.com
● CoreOS stickers!
291. Brief side-note: locksmith
● Reboot manager for CoreOS
● Uses a semaphore in etcd to co-ordinate reboots
● Each machine in the cluster:
1. Downloads and applies update
2. Takes lock in etcd (using Compare-And-Swap)
3. Reboots and releases lock