KubeCon EU 2016: Killing containers to make weather beautiful

•

0 likes•267 views

The Met Office Informatics Lab includes scientists, developers and designers. We build prototypes exploring new technologies to make environmental data useful. Here we describe a recent project to process multi-dimensional weather data to create a fully interactive 4D browser application. We used long-running containers to serve data and web pages and short-running processes to ingest and compress the data. Forecast data is issued every three hours so our data ingestion goes through regular and predictable bursts (i.e. perfect for autoscaling). We built a Kubernetes cluster in an AWS group which auto-scales based on load. We used replication controllers to process the data. Every three hours ingestion jobs are added to a queue and the number of ingestion containers are set in proportion to the queue length. Each worker completes exactly one ingestion job from the queue and then exits, at which point Kubernetes creates a new one to process the next message. This has allowed us to remove the lifespan logic from the containers and keep them light, fast and massively scalable. We are now in the process of using this in our production systems. Sched Link: http://sched.co/6BWQ

Technology

Killing containers to
make weather
beautiful
Jacob Tomlinson
Met Oﬃce Informatics Lab
@_jacobtomlinson

Visualising
4D data in a
web
browser
Confidence
in
integration

Supercomputer
Binary files
PNG images
Videos
The browser

Long running daemons
Short running workers

Docker Host
nginx
SystemCapacity
Docker Host
nginxSystemCapacity
Docker Host
nginx
SystemCapacity

Docker Host
SystemCapacity
Docker Host
SystemCapacity
Docker Host
SystemCapacity
nginx nginx nginx

Docker Host
nginx
SystemCapacity
Docker Host
nginxSystemCapacity
Docker Host
nginx
SystemCapacity
worker
worker
worker
worker
worker
worker
worker
worker
worker
worker
worker
worker

Docker Host
nginx
SystemCapacity
Docker Host
SystemCapacity
Docker Host
SystemCapacity
worker
nginx nginx
workerworker

Docker
nginx
System
worker
Docker
System
worker
nginx
Docker
System
worker
nginx
Docker
System
worker
worker
worker
worker
worker

Docker
System
worker
Docker
System
worker
Docker
System
worker
Docker
System
worker
nginx
worker
nginxnginxnginx
workerworkerworker

Docker Host
mongo
EBS Volume
Docker Host Docker Host

Docker Host
EBS Volume
Docker Host Docker Host
mongo mongo

Docker Host
EBS Volume
Docker Host Docker Host
mongo

Short containers
Long containers
Daemon processes which listen to requests
Asynchronous processes doing repetitive
workloads

Pros
Conceptual simplicity
Confidence
Cons
Overhead

Questions
Jacob Tomlinson @_jacobtomlinson

rkt is a modern container runtime, built for security, efficiency, and composability. Kubernetes is a modern cluster orchestration system allowing users. Kubernetes doesn't directly execute application containers but instead delegate to a container runtime, which is integrated at the kubelet (node) level. When Kubernetes first launched, the only supported container runtime was Docker - but in recent months, we've been hard at work integrating rkt as an alternative container runtime, aka "rktnetes". The goal of "rktnetes" is to have first-class integration between rkt and the kubelet, and allow Kubernetes users to take advantage of some of rkt's unique features. This talk will describe how rkt works, some of the features that make it unique as a container runtime, and some of the process of integrating an alternative container runtime with Kubernetes, as well as the latest state of "rktnetes."Introduction to rkt, including special/unique features. Sched Link: http://sched.co/6BY7

DockerCon EU 2015: The Glue is the Hard Part: Making a Production-Ready PaaS

Docker, Inc.

Presented by Evan Krall, Site Reliability Engineer, Yelp Docker is an amazing technology. In particular, its build-once-run-anywhere model unlocks the world of cluster schedulers like Mesos and Kubernetes. These solve many of the problems of running high-scale websites, but introduce new challenges that need addressing. In this talk, Evan will describe PaaSTA, a PaaS built on top of open source tools including Docker, Mesos, Marathon, and Chronos. PaaSTA provides tooling for developers to quickly turn their microservice into a monitored, highly available application spanning multiple datacenters and cloud regions. Evan will give an overview of the open-source technologies that power PaaSTA, discuss how Yelp has glued these together to give developers control without burdening them with the complexities of the infrastructure, and show the workflow used by developers to update and maintain their services on PaaSTA.

DockerCon EU 2015: Trading Bitcoin with Docker

Docker, Inc.

Presented by Sebastien Goasguen, VP, Apache CloudStack and Mathieu Buffenoir, co-founder, SBEX Bity is an internet money gateway built by Swiss Bitcoin Exchange ( SBEX ). To trade bitcoin the entire infrastructure of Bity is running in Docker containers. All the components of the infrastructure are using Docker, from the frontend applications and load balancer, the Django based backend, replicated Postgres database, Bitcoin daemon and remittance engine. All software goes through a CI pipeline that starts with Docker images being built on private repositories in Docker hub. Developers take also advantage of a docker-compose definition that allows them to run the entire infrastructure on a single laptop. Finally the production deployments happen thanks to the Ansible Docker module on a CloudStack based public cloud. Everything has been automated to ease re-deployment and operations. This presentation will go through every component and how Docker has enabled us to go production in 4 months.

How to Achieve Canary Deployment on Kubernetes

HanLing Shen

Kernel load-balancing for Docker containers using IPVS

Docker, Inc.

Many companies use expensive proprietary hardware and software to provide load-balancing and routing for their users and services. I'm going to demonstrate how the same or even exceeding performance and feature set can be achieved using an open-source technology which has been a part of the mainline Linux kernel for over a decade – IPVS. Specifically, you'll see how IPVS can be used to automatically configure load balancing and routing for Docker containers using a simple Go daemon and a Docker plugin.

Thinking Inside the Container: A Continuous Delivery Story by Maxfield Stewart

Docker, Inc.

Riot builds a lot of software. At the start of 2015 we were looking at 3000 build jobs over a hundred different applications and dozens of teams. We were handling nearly 750 jobs per hour and our build infrastructure needed to grow rapidly to meet demand. We needed to give teams total control of the “stack” used to build their applications and we needed a solution that enabled agile delivery to our players. On top of that, we needed a scalable system that would allow a team of four engineers to support over 250. After as few explorations, we built an integrated Docker solution using Jenkins that accepts docker images submitted as build environments by engineers around the company . Our “containerized” farm now creates over 10,000 containers a week and handles nearly 1000 jobs at a rate of about 100 jobs an hour. In this occasionally technical talk, we’ll explore the decisions that led Riot to consider Docker, the evolutionary stages of our build infrastructure, and how the open source and in-house software we combined to achieve our goals at scale. You’ll come away with some best practices, plenty of lessons learned, and insight into some of the more unique aspects of our system (like automated testing of submitted build environments, or testing node.js apps in containers with Chromium and xvfb).

Introduction kubernetes 2017_12_24

Sam Zheng

Share the continuity of Société Générale's journey with Docker Enterprise from different points of view, from executives to devops, with CD platform as an enabler. Creating a Dockerfile that runs a container on a developer's laptop is pretty straightforward. But extending that to stacks of containers running on a dozen environments (development, integration, testing, staging, production, etc.) with different configuration and topologies can be a challenge. This talk will cover aspects of our journey to Docker Enterprise: What configuration should go in an image? Where to put different types of configuration? Images, environment variables, entrypoint, ...? How to store assets for building images and configuration for deployment in version control. We will discuss how Société Générale has implemented these, and what we plan next for Docker Enterprise deployment.

Deep dive in container service discovery

Docker, Inc.

Service discovery and traffic load-balancing in the container ecosystem relies on different technologies, such as IPVS and iptables, and container orchestrators use different approaches. This talk will present in details how Docker Swarm and Kubernetes achieve this. The talk will continue with a demo showing how applications that are not managed by Kubernetes can take advantage of its native load-balancing. Finally, it will compare these approaches to service-mesh solutions.

Tectonic Summit 2016: The Origins of Kubernetes

CoreOS

Kubernetes - Starting with 1.2

William Stewart

This talk will focus on a brief history, including a demo and overview of how we at Superbalist use Kubernetes, and how Kubernetes uses Docker, does load balancing, deployments, and data migrations. Talk from Cape Town DevOps meetup on Jun 21, 2016: https://www.meetup.com/Cape-Town-DevOps/events/231530172/ Code: https://github.com/zoidbergwill/kubernetes-examples Slides as markdown: http://www.zoidbergwill.com/presentations/2016/kubernetes-1.2-and-spread/index.md

Docker Summit 2016 - Kubernetes: Sweets and Bitters

smalltown

Secure your K8s cluster from multi-layers

Jiantang Hao

Docker on docker leveraging kubernetes in docker ee

Docker, Inc.

Understanding Kubernetes

Tu Pham

DockerCon EU 2015: Speed Up Deployment: Building a Distributed Docker Registr...

Docker, Inc.

KubeCon EU 2016: A Practical Guide to Container Scheduling