Docker allows building portable software that can run anywhere by packaging an application and its dependencies in a standardized unit called a container. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes can replicate containers, provide load balancing, coordinate updates between containers, and ensure availability. Defining applications as Kubernetes resources allows them to be deployed and updated easily across a cluster.
A Comprehensive Introduction to Kubernetes. This slide deck serves as the lecture portion of a full-day Workshop covering the architecture, concepts and components of Kubernetes. For the interactive portion, please see the tutorials here:
https://github.com/mrbobbytables/k8s-intro-tutorials
Kubernetes for Beginners: An Introductory GuideBytemark
An introduction to Kubernetes for beginners. Includes the definition, architecture, benefits and misconceptions of Kubernetes. Written in plain English, ideal for both developers and non-developers who are new to Kubernetes.
Find out more about Kubernetes at Bytemark here: https://www.bytemark.co.uk/managed-kubernetes/
A basic introduction to Kubernetes. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.
A Comprehensive Introduction to Kubernetes. This slide deck serves as the lecture portion of a full-day Workshop covering the architecture, concepts and components of Kubernetes. For the interactive portion, please see the tutorials here:
https://github.com/mrbobbytables/k8s-intro-tutorials
Kubernetes for Beginners: An Introductory GuideBytemark
An introduction to Kubernetes for beginners. Includes the definition, architecture, benefits and misconceptions of Kubernetes. Written in plain English, ideal for both developers and non-developers who are new to Kubernetes.
Find out more about Kubernetes at Bytemark here: https://www.bytemark.co.uk/managed-kubernetes/
A basic introduction to Kubernetes. Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.
History and Basics of containers, LXC, Docker and Kubernetes. This presentation is given to Engineering colleage students at VIT DevFest 2018. Beginner to Intermediate level.
Docker is the world’s leading software container platform. Developers use Docker to eliminate “works on my machine” problems when collaborating on code with co-workers. Operators use Docker to run and manage apps side-by-side in isolated containers to get better compute density. Enterprises use Docker to build agile software delivery pipelines to ship new features faster, more securely and with confidence for both Linux and Windows Server apps.
Learn More: http://www.collabnix.com
Learn, Collaborate & Dockerize. Docker is an open platform that helps you build, ship and run applications anytime and anywhere.
Join Docker Jaipur:
Docker Page: events.docker.com/jaipur
Telegram Group: t.me/dockerjaipur
Twitter: @JaipurDocker
** Kubernetes Certification Training: https://www.edureka.co/kubernetes-certification **
This Edureka tutorial on "Kubernetes Architecture" will give you an introduction to popular DevOps tool - Kubernetes, and will deep dive into Kubernetes Architecture and its working. The following topics are covered in this training session:
1. What is Kubernetes
2. Features of Kubernetes
3. Kubernetes Architecture and Its Components
4. Components of Master Node and Worker Node
5. ETCD
6. Network Setup Requirements
DevOps Tutorial Blog Series: https://goo.gl/P0zAfF
A basic introductory slide set on Kubernetes: What does Kubernetes do, what does Kubernetes not do, which terms are used (Containers, Pods, Services, Replica Sets, Deployments, etc...) and how basic interaction with a Kubernetes cluster is done.
Traditional virtualization technologies have been used by cloud infrastructure providers for many years in providing isolated environments for hosting applications. These technologies make use of full-blown operating system images for creating virtual machines (VMs). According to this architecture, each VM needs its own guest operating system to run application processes. More recently, with the introduction of the Docker project, the Linux Container (LXC) virtualization technology became popular and attracted the attention. Unlike VMs, containers do not need a dedicated guest operating system for providing OS-level isolation, rather they can provide the same level of isolation on top of a single operating system instance.
An enterprise application may need to run a server cluster to handle high request volumes. Running an entire server cluster on Docker containers, on a single Docker host could introduce the risk of single point of failure. Google started a project called Kubernetes to solve this problem. Kubernetes provides a cluster of Docker hosts for managing Docker containers in a clustered environment. It provides an API on top of Docker API for managing docker containers on multiple Docker hosts with many more features.
History and Basics of containers, LXC, Docker and Kubernetes. This presentation is given to Engineering colleage students at VIT DevFest 2018. Beginner to Intermediate level.
Docker is the world’s leading software container platform. Developers use Docker to eliminate “works on my machine” problems when collaborating on code with co-workers. Operators use Docker to run and manage apps side-by-side in isolated containers to get better compute density. Enterprises use Docker to build agile software delivery pipelines to ship new features faster, more securely and with confidence for both Linux and Windows Server apps.
Learn More: http://www.collabnix.com
Learn, Collaborate & Dockerize. Docker is an open platform that helps you build, ship and run applications anytime and anywhere.
Join Docker Jaipur:
Docker Page: events.docker.com/jaipur
Telegram Group: t.me/dockerjaipur
Twitter: @JaipurDocker
** Kubernetes Certification Training: https://www.edureka.co/kubernetes-certification **
This Edureka tutorial on "Kubernetes Architecture" will give you an introduction to popular DevOps tool - Kubernetes, and will deep dive into Kubernetes Architecture and its working. The following topics are covered in this training session:
1. What is Kubernetes
2. Features of Kubernetes
3. Kubernetes Architecture and Its Components
4. Components of Master Node and Worker Node
5. ETCD
6. Network Setup Requirements
DevOps Tutorial Blog Series: https://goo.gl/P0zAfF
A basic introductory slide set on Kubernetes: What does Kubernetes do, what does Kubernetes not do, which terms are used (Containers, Pods, Services, Replica Sets, Deployments, etc...) and how basic interaction with a Kubernetes cluster is done.
Traditional virtualization technologies have been used by cloud infrastructure providers for many years in providing isolated environments for hosting applications. These technologies make use of full-blown operating system images for creating virtual machines (VMs). According to this architecture, each VM needs its own guest operating system to run application processes. More recently, with the introduction of the Docker project, the Linux Container (LXC) virtualization technology became popular and attracted the attention. Unlike VMs, containers do not need a dedicated guest operating system for providing OS-level isolation, rather they can provide the same level of isolation on top of a single operating system instance.
An enterprise application may need to run a server cluster to handle high request volumes. Running an entire server cluster on Docker containers, on a single Docker host could introduce the risk of single point of failure. Google started a project called Kubernetes to solve this problem. Kubernetes provides a cluster of Docker hosts for managing Docker containers in a clustered environment. It provides an API on top of Docker API for managing docker containers on multiple Docker hosts with many more features.
This presentation by Andrew Aslinger discusses best practices and pitfalls of integrating Docker into Continuous Delivery Pipelines. Learn how Andrew and his team used Docker to replace Chef to simplify their development and migration processes.
Tell the history of Container/Docker/Kubernetes, and show the key elements of them.
After view this document, you could know the main feature of Container Docker and Kubernetes.
Very basic infomation about how these technique work together.
Best Practices for Running Kafka on Docker ContainersBlueData, Inc.
Docker containers provide an ideal foundation for running Kafka-as-a-Service on-premises or in the public cloud. However, using Docker containers in production environments for Big Data workloads using Kafka poses some challenges – including container management, scheduling, network configuration and security, and performance.
In this session at Kafka Summit in August 2017, Nanda Vijyaydev of BlueData shared lessons learned from implementing Kafka-as-a-Service with Docker containers.
https://kafka-summit.org/sessions/kafka-service-docker-containers
Docker Kubernetes Istio
Understanding Docker and creating containers.
Container Orchestration based on Kubernetes
Blue Green Deployment, AB Testing, Canary Deployment, Traffic Rules based on Istio
Get you Java application ready for Kubernetes !Anthony Dahanne
In this demos loaded talk we’ll explore the best practices to create a Docker image for a Java app (it’s 2019 and new comers such as Jib, CNCF buildpacks are interesting alternatives to Docker builds !) - and how to integrate best with the Kubernetes ecosystem : after explaining main Kubernetes objects and notions, we’ll discuss Helm charts and productivity tools such as Skaffold, Draft and Telepresence.
Michigan IT Symposium 2017 - Container BOFJeffrey Sica
Development with Containers
Moderator: Jeffery Sica
Orchestration and Management with Containers
Moderator: Bob Killen
The Container has revolutionized how many industries and enterprises develop and deploy software and services. While the promise of containers to improve reliability, reproducibility and sharing application is helping to drive this adoption, there are still questions and concerns that are holding back broader adoption. In this BOF we will provide an opportunity for the Michigan IT community to engage with the local leaders in this area that are developing solutions and promoting the container model in their units
This BoF has been broken into two high-level topics: Development with Containers that is geared toward developers working with and implementing containerized applications, and Orchestration and Management with Containers that is geared toward sysadmins that are hosting and maintaining production-level containers. There will be an overview followed by an interactive period where the audience can ask questions of our experts and share their experiences with others. A key goal of these sessions is to understand and develop an agenda for solving questions and concerns that are holding back broader adoption.
Dockerized containers are the current wave that promising to revolutionize IT. Everybody is talking about containers, but a lot of people remain confused on how they work and why they are different or better than virtual machines. In this session, Black Duck container and virtualization expert Tim Mackey will demystify containers, explain their core concepts, and compare and contrast them with the virtual machine architectures that have been the staple of IT for the last decade.
Containers in depth – Understanding how containers work to better work with c...All Things Open
Presented by: Brent Laster
Presented at the All Things Open 2021
Raleigh, NC, USA
Raleigh Convention Center
Abstract: Containers are all the rage these days – from Docker to Kubernetes and everywhere in-between. But to get the most out of them it can be helpful to understand how containers are constructed, how they depend and interact with the operating system, and what the differences and interactions are between layers, images, and containers. Join R&D Director, Brent Laster as he does a quick, visual overview of how containers work and how applications such as Docker work with them.
Topics to be discussed include:
• What containers are and the benefits they provide
• How containers are constructed
• The differences between layers, images, and containers
• What does immutability really mean
• The core Linux functionalities that containers are based on • How containers reuse code
• The differences between containers and VMs
• What Docker really does
• The Open Container Initiative
• A good analogy for understanding all of this
This presentation covers the basics of dockers, its security related features and how certain misconfigurations can be used to escape from container to host
We present a web service named FLOW to let users do FLink On Web. FLOW aims to minimize the effort of handwriting streaming applications similar in spirit to Hortonworks Stream Analytics Manager, StreamAnalytix, and Nussknacker by letting users drag and drop graphical icons representing streaming operators on GUI.
FLOW builds on Flink Table API and lets users assemble graphical icons associated with not only basic SQL operations but also advanced SQL operations like window aggregation, temporal join, and pattern recognition (MATCH_RECOGNIZE clause). Its data preview function enables to observe how sample data changes before and after applying each operation on screen. In addition, FLOW shows the sample data as time-series charts and geographical maps by interacting with Elasticsearch and Kibana. Therefore, domain experts with basic knowledge of SQL can design their streaming applications easily on GUI without understanding of Flink DataStream API and Flink CEP library.
In this talk, we first present what motivates the development of FLOW, then show how FLOW can be used to figure out the "Popular Places" exercise in its own style, and lastly explain how FLOW leverages Flink Table API.
This presentation is to help you understand https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/ without having to read all the concepts in a number of Kubernetes documents.
A Comparative Performance Evaluation of Apache FlinkDongwon Kim
I compare Apache Flink to Apache Spark, Apache Tez, and MapReduce in Apache Hadoop in terms of performance. I run experiments using two benchmarks, Terasort and Hashjoin.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
2. Big Data Tech. Lab in SK telecom
• Discovery Group
• Predictive Maintenance Group
• Manufacturing Solution Group
• Groups making own solutions
• Technology and Architecture Leading Group
• Big data processing engine
• Advanced analytics algorithms
• Systematize service deployment and service operation on cluster
• Docker
• Kubernetes
3. Prepare for an era of cloud with Docker and Kubernetes
oracle
ubuntu
cloud
Major technologies
Docker
Kubernetes
Amazon Web Service
Microsoft azure
Cloud technologies
for service providers
icloud
Cloud services
for users
one drive
dropbox
google
drive
Trend
- Buy both SW & HW
- Buy HW and DIY
- Run your SW on cloud
Ubiquitous cloud services
around us
Enabling technologies
for custom cloud services
* technology trend in USA (2004-2017)
4. Overview & Conclusion
• Docker to build portable software
• Build your software upon Docker
• Then distribute it anywhere (even on MS Azure and Amazon Web Service)
• Kubernetes to orchestrate multiple Docker instances
• Start using Docker and Kubernetes before too late!
• Google has been using container technologies more than 10 years
Docker
Kubernetes
Hadoop
The Enterprise IT Adoption CyclePopularity of Docker and Kubernetes
8. Few choices left to you
1. Convince your customer (a.k.a. 甲)
2. Install all the dependencies manually (without the package manager)
3. Modify your program to make it depend v1
9. Docker container
Package manager in host OS
Use Docker for isolating your application
Package manager in guest OS
Your program
program1
v2
program2
v2
program3
v2
depends ondepends on
Customer program
program1
v1
program2
v1
program3
v1
Host operating system
Linux kernel must be ≥3.10 (such as Ubuntu 14.04 and CentOS 7)
Docker engine (daemon)
10. Virtual machines and docker containers
Host Operating System
Kernel
Hypervisor Docker engine
Virtual machines Docker containers
Device drivers
Host Operating System
Kernel Device drivers
CentOS-like
container
yum
Libraries
App
Ubuntu-like
container
apt
App
Libraries
CentOS
virtual machine
Kernel Device
drivers
yum
Libraries
App
Ubuntu
virtual machine
Kernel
apt
App
Libraries
Device
drivers
Containers share the kernel in the host
11. Linux namespaces – what makes isolated environments in a host OS
Host Operating System
Docker engine
Container
pid
ipc
uts
net
mnt
user
Various ipc objects
- POSIX message queue
- SystemV IPC objects
(mq, sem, shm)
System identifiers
- hostname
- NIS domain name
Network devices
- Network devices
- IPv4, IPv6 stacks
- Routing tables, Firewall
Mount points
(directory hierarchy)
Security-related identifiers
- User IDs
- Group IDs
Process ID number space
(staring from 1)
Container
pid
ipc
uts
net
mnt
user
Container
pid
ipc
uts
net
mnt
user
Six namespaces are enough to give an illusion of running inside a virtual machine
12. Analogy between program and docker
Dockerfile Docker image
(read-only layers)
Docker container
(read-only layers + writable layer)
Source code Byte/machine code
(read only)
Process
(read only)
text
data
heap
stack
compile execute
build run
Program
Docker
13. How to define an image and run a container from it?
1) Write Dockerfile
- Specify to install python with pip on ubuntu
- Tell pip to install numpy
2) Build an image from Dockerfile
- Execute each line of Dockerfile to build an image
3) Execute a Docker container from the image
14. 1 to N relationship between image and container
Execute five containers from an image
Q) Five containers take up 2,445MB (=489MB*5) in the host?
A) No due to image layering & sharing
15. Images consists of layers each of which is a set of files
• Instructions (FROM, RUN, CMD, etc) create layers
• Base images (imported by “FROM”) also consist of layers
• If a file exists in multiple layers, the one in the upper layer is seen
Dockerfile
Base ubuntu image
Layer (apt-get install python-dev python-pip)
Layer (pip install numpy)
Layer
Layer (files)
Layer (files)
Layer (files)
Image
16. Docker container
• A container is just a thin read/write layer
• base images are not copied to containers
• Copy-On-Write (COW)
• When a file in the base image is modified,
• copy the file to the R/W layer
• and then modify the copied file
17. Image sharing between containers
ubuntu:15.04 image (~188MB) does not copied to all containers
18. Layer sharing between images
If multiple Dockerfiles
1. start from the same base image
2. share a sequence of instructions (one RUN instruction in a below example)
, then docker engine automatically reuses existing layers
numpy Dockerfile matplotlib Dockerfile
19. Example of stacking docker images
Kafka broker PdM engine
kafka
(with scala)
Zookeeper
container
Kafka
container
PdM engine
(librdkafka, avro, flask)
cuda
PdM engine
(librdkafka, avro, flask)
scipy
(numpy, scipy, matplotlib, ipython, jupyter, pandas, scikit-learn, h5py)
theano-gpu (theano, keras)
theano-cpu
(theano, keras)
openjdk:8
zookeeper
buildpack-deps:jessie
python:2.7
buildpack-deps:jessie-curl
official
official
official
official
Zookeeper cluster
zk
zk
zkzk
zk
broker
broker
broker
Kafka
consumer
Kafka
producer
Web server
scipy libraries has nothing to
do with GPU, so share it
theano compiles
its expression graphs into
CPU/GPU instructions
PdM container (cpu) PdM container (gpu)
buildpack-deps:jessie-scm
debian:jessie
official
official
jessie is the latest, stable
Debian release
buildpack-deps contains
essential tools to
download/compile softwares
20. Enabling technologies for docker (wrap-up)
• Linux namespaces (covered)
• To isolate system resources
• pid, net, ipc, mnt, uts, user
• It makes a secure & isolate environment (like a VM)
• Advanced multi-layer unification File System (covered)
• Image layering & sharing
• Linux control groups (not covered)
• To track, limit, and isolate resources
• CPU, memory, network, and IO
* https://mairin.wordpress.com/2011/05/13/ideas-for-a-cgroups-ui/
21. Docker topics not covered here
• How to install Docker engine
• What are the docker instructions other than FROM, RUN, and CMD
• ENV / ADD / ENTRYPOINT / LABEL / EXPOSE / COPY / VOLUME / WORKDIR /
ONBUILD
• How to push local Docker images to docker hub
• How to pull remote images from docker hub
• ...
Consult with https://docs.docker.com/engine/getstarted/
23. Disclaimer
• The purpose of this section is
to briefly explain Kubernetes without details
• For a detailed explanation
with the exact Kubernetes terminology,
see the following slide
• https://www.slideshare.net/ssuser6bb12d/kubernetes-introduction-
71846110
24. What is Kubernetes for?
Container-based virtualization + Container orchestration
To satisfy common needs in production
replicating application instances
naming and discovery
load balancing
horizontal auto-scaling
co-locating helper processes
mounting storage systems
distributing secrets
application health checking
rolling updates
resource monitoring
log access and ingestion
...
from the official site : https://kubernetes.io/docs/whatisk8s/
25. Why Docker with Kubernetes?
• A mission of our group
• Systematize service deployment and service operation on cluster
• I believe that systematizing smth. is to minimize human efforts on smth.
• How to minimize human efforts on service deployment?
• Make software portable using a container technology
• Docker (chosen for its maturity and popularity)
• Rkt from CoreOS (alternative)
• Build images and run containers anywhere
• Your laptop, servers, on-premise clusters, even cloud
• How to minimize human efforts on service operation?
• Inform a container orchestration runtime of service specification
• Kubernetes from Google (chosen for its maturity and expressivity)
• Docker swarm from Docker
• Define your specification and then the runtime operates your services as you wish
26. Kubernetes architecture
Server
- REST API server with a K/V store
- Scheduler
- Find suitable machines for containers
- Controller manager
- Current state Desired state
- Make changes if states go undesirable
Service specification
(written in yaml)
- Execute a web-server image
- Two replicas for LB & HA
- 3GB memory each
Docker engine
Node agent
container
(3GB)
Docker engine
Node agent
container
(3GB)
Docker engine
Node agent
container
(3GB)
Ensure a specified
# of replicas running
all the time
27. Web server example
node 2
webserver
node 1
webserver
node 3
webserver
Want to launch 3 replicas
for high availability and load balancing
How to achieve the followings?
• Users must be unaware of the replicas
• Traffic is evenly distributed to replicas
webserver
4bp80
webserver
6dk12
webserver
g1sdf
a well-known address
It’s a piece of cake with Kubernetes!
28. How to replicate your service instances
node 2
webserver
6dk12
node 1
webserver
4bp80
node 3
webserver
g1sdfapp=web1 app=web1 app=web1
Server
Node agent Node agent Node agent
Docker engine Docker engine Docker engine
Specify your Docker image and a replication factor
using Deployment
Specify a common label
to group containers with
different names
29. node 2node 1 node 3
Define a service to do round-robin forwarding
Server
<service>
webserver:80
webserver
6dk12
webserver
4bp80
webserver
g1sdfapp=web1 app=web1 app=web1
33% 33% 33%
<ingress>
metatron:80
External traffic
over internet
Internal traffic
Kubernetes runs its own DNS server for name resolution
Kubernetes manipulates iptables on each node to proxy traffic
30. Kubernetes
How to guarantee a certain # of running containers during maintenance
node1
zk-0
Containers
Volumes
node2
zk-2
Containers
Volumes
node3
zk-3
Containers
Volumes
Drain node1
Operation is permitted
because allowed-disruptions=1
Kubernetes
Drain node2
3 replicas have to be running
due to StatefulSet,
so try scheduling zk-0
on other nodes!
Oops!
cannot schedule zk-0
on node2 and node3
due to anti-affinity!
Operation not permitted
because allowed-disruptions=0
(Note that minAvailable=2)
Please wait until
node1 is up and zk-0 is rescheduled!
node1
zk-0
Containers
Volumes
node2
zk-2
Containers
Volumes
node3
zk-3
Containers
Volumes
Define disruption budget
to specify requirement for
the minimum available containers
Hold on for a while
31. PdM Kubernetes cluster
Zookeeper headless service Kafka headless service
PdM service
QuorumPeer
Main
QuorumPeer
Main
QuorumPeer
Main
Pod Pod Pod
Kafka
(broker)
Kafka
(broker)
Kafka
(broker)
Pod Pod Pod2181
2888
3888
2181
2888
3888
2181
2888
3888
9092
9092
9092
Statefulset Statefulset
PdM engine
Kafka
consumer
Kafka
producer
Web
server
Pod (Deployment)
Ingress
rule
8080
Persistent
storage
Attached
volume
Volume
80
32. Overview & Conclusion
• Docker to build portable software
• Build your software upon Docker
• Then distribute it anywhere (even on MS Azure and AWS)
• Kubernetes to orchestrate multiple Docker instances
• Start using Docker and Kubernetes before too late!
• Google has been using container technologies more than 10 years
Docker
Kubernetes
Hadoop
The Enterprise IT Adoption CyclePopularity of Docker and Kubernetes