OpenShift v3
Docker, Kubernetes, Containers
Peter Larsen, Sr. Solutions Architect
plarsen@redhat.com
October 2014

Agenda
● Recent Highlights
● OpenShift Online
● OpenShift v3 – Overview
● OpenShift – detailed archiecture
● Q&A
● Demo

Recent Highlights
● OpenShift wins InfoWorld Bossie 2014 Award
● Gartner gives OpenShift edge over Cloud Foundry in report on Open
Source PaaS Frameworks
● Hortonworks announces Yarn integration with v3 / Kubernetes
● Paychex added to public reference customers (CA, Boeing, FICO,
Cisco, UNC)

COMPUTE RESOURCES
(CPU, RAM, NETWORK, DISK)
VIRTUAL GUEST IMAGES
OPERATING SYSTEM
APPLICATION PLATFORM
(App Server, Middleware, Languages)
APPLICATION
Automated and Managed
by the Cloud Provider
Provided and Controlled
by Cloud Consumer
IaaS PaaS SaaS
Increased Control
Increased Automation
DEVELOPER TOOLING
(IDE, Source Control, Build Tools, CI)
Cloud Service Models

COMPUTE RESOURCES
(CPU, RAM, NETWORK, DISK)
VIRTUAL GUEST IMAGES
OPERATING SYSTEM
APPLICATION PLATFORM
(App Server, Middleware, Languages)
APPLICATION
Automated and Managed
by the Cloud Provider
Provided and Controlled
by Cloud Consumer
IaaS+ IaaS++
Increased Control
Increased Automation
DEVELOPER TOOLING
(IDE, Source Control, Build Tools, CI)
What is PaaS?
PaaS

COMPUTE RESOURCES
(CPU, RAM, NETWORK, DISK)
VIRTUAL GUEST IMAGES
OPERATING SYSTEM
APPLICATION PLATFORM
(App Server, Middleware, Languages)
APPLICATION
Automated and Managed
by the Cloud Provider
Provided and Controlled
by Cloud Consumer
OpenStack CloudForms
Increased Control
Increased Automation
DEVELOPER TOOLING
(IDE, Source Control, Build Tools, CI)
Red Hat Offerings
OpenShift

Red Hat’s Cloud Portfolio

PaaS = Platform as a Service
A Cloud Application Platform
Code Deploy Run
Save Time and Money
Code your app
Push-
button
Deploy, and
your App is
running in
the Cloud!

Today’s IT Challenge

PaaS Lets You Streamline App Dev
With PaaS
How to Build an App:
1. Have Idea
2. Get Budget
3. Code
4. Test
5. Launch
6. Automatically Scale
How to Build an App:
1. Have Idea
2. Get Budget
3. Submit VM Request request
4. Wait
5. Deploy framework/appserver
6. Deploy testing tools
7. Code
8. Test
9. Configure Prod VMs
10. Push to Prod
11. Launch
12. Request VMs to meet demand
13. Wait
14. Deploy app to new VMs
15. Etc.
Virtualized
How to Build an App:
1. Have Idea
2. Get Budget
3. Submit hardware acquisition request
4. Wait
5. Get Hardware
6. Rack and Stack Hardware
7. Install Operating System
8. Install Operating System Patches
9. Create user Accounts
10. Deploy framework/appserver
11. Deploy testing tools
12. Code
13. Test
14. Buy and configure Prod servers
15. Push to Prod
16. Launch
17. Order more servers to meet demand
18. Wait…
19. Deploy new servers
20. Etc.
Physical
“The use of Platform-as-a-Service technologies will
enable IT organizations to become more agile and
more responsive to the business needs.” –Gartner*

Velocity and Efficiency enable Scalability
●
Scalable Applications
●
Scalable Infrastructure
●
Scalable Workflows/Processes
Scale IT Like a Factory with PaaS

DevOps? Continuous Delivery?
What is DevOps? - The application of
software development principles to the tasks of
IT Operations, often automating Operations
tasks such as application code deployment and
promotion.
What is Continuous Delivery? – The
automation of code promotion through the
application lifecycle so that code changes can
be pushed into production environments very
often.
But how do we do these things?

Code Deploy Run
PaaS leverages automation technologies
and a cloud architecture…
to drive Velocity, Efficiency and
Scalability in IT.
Accelerate IT

OpenShift is
PaaS by Red Hat
 Multi-language
 Auto-scaling
 Self-service
 Open Source
 Enterprise-grade
 Secure
 Built on Red Hat

Red Hat’s PaaS Strategy
Public
PaaS
Service
On-premise
or Private
PaaS
Software
Open
Source
PaaS
Project

OpenShift Online

OpenShift Online Adoption

Docker Intro
● Great intro here:
https://goldmann.pl/presentations/2013-red-hat-docker
-introduction
● https://www.docker.com/whatisdocker/

OpenShift v3 – Big Picture

OpenShift Architecture Today

Node
Developer
Node (RHEL)
Replication
Controller
Python
Pod
JBoss
Pod
Postgres
Pod
pgadmin
Tomcat
Pod
JBoss
Pod
Node
(RHEL 7 / Atomic)
Node
(RHEL 7 / Atomic)
Ruby
Pod
Node.js
Pod
Master
(RHEL 7 / Atomic)
etcd
Scheduler
OAuth
REST
Web Console
CLI
IDE
Service Layer
Pod
Pod
Pod
Pod
Routing Layer
Application Users
GIT / SSH
OpenShift v3 Architecture

What's Different?
● New base OS
● RHEL 7 + Atomic vs. RHEL 6
● New container model
● Docker vs. v2 “Gears”
● New orchestration engine
● Kubernetes vs. v2 “Broker”
● New packaging model
● Docker images vs. v2 “Cartridges”
● New routing tier
● Platform routing layer vs. v2 Node-based routing
● More services and a better developer experience

Why Is This Better?
● Standard containers API
● Container-optimized OS
● Web-scale orchestration
● Expanded choice of services
● Enhanced developer and
operator experience
● Industry standard PaaS
stack!

OpenShift v3 – Key Principles

Redefine the “Application”
● Networked components wired together
● Not just a “web frontend” anymore
● Service-oriented-Architectures / microservices are real
● HTTP frontends are just one type of component
● Critical: relationships between components
● If you can’t abstract the connection between
components you can’t evolve them independently

Immutable Images as Building Blocks
● Image based deployment (Docker)
● Create once, test everywhere
● Build a single artifact containing the dependency chain
● Needs tools to manage the build process, manage
security updates
● Declarative application descriptions
● Record “intent” - this links to this, this should be
deployed like this, and let system converge to that intent

Decouple Dev and Ops
● Reduce complexity of application topology
● Less need for complicated post-deploy setup
● Allow teams to share common stacks
● Better migration from dev stacks to production stacks
● Reduce vendor lock-in
● Ops and Devs can use same tools
● OpenShift is an app, so are most organizational tools
● Ensure patterns work cleanly for both

Decouple Dev and Ops (cont.)
● Templatize / blueprint everything
● Ensure that organizations have clear configuration
chains
● Define common patterns for rolling out changes
● Easily provision new resources
● Allow infrastructure teams to provision at scale
● Subdivide resources for organizational teams with hard
and soft limits

Abstract Operational Complexity
● Networking
● App deployers should see flat networks
● Define private vs public, internal vs external, fast vs
slow
● Storage
● Most components need *simple* persistent storage
● Ensure storage is not coupled to the host
● Health
● Every component should expose health information

Multi-Level Security
● Ensure containers “contain”
● SELinux, user namespaces, audit
● Decompose the Docker daemon over time
● Fine grained security controls on SSH access
● Allow easy integration with existing security tools
● Kerberos, system wide security, improved scoping of
access
● More customization possible
● Allow application network isolation

Architectural Overview

Major components
● Docker image: Defines a filesystem for running an isolated
Linux process (typically an application)
● Docker container: Running instance of a Docker image with its
own isolated filesystem, network, and process spaces.
● Pod: Kubernetes object that groups related Docker containers
that need to share network, filesystem or memory together for
placement on a node. Multiple instances of a Pod can run to
provide scaling and redundancy
● Replication Controller: Kubernetes object that ensures N (as
specified by the user) instances of a given Pod are running at
all times.
● Service: Kubernetes object that provides load balanced access
to all instances of a Pod from another container in another Pod.

Pod Design examples
● A single pod with two
containers, each exposing
a port on the pod’s IP
address
● Three different pods each
running a set of related
containers

Service Example
● Multiple instances of a
single pod are load
balanced and accessed via
a Service

Architectural OpenShift V3 Overview

Getting Started

Architecture Details
Shamefully borrowed from:
https://github.com/openshift/openshift-pep/blob/maste
r/openshift-pep-013-openshift-3.md
● This is work in progress. Absolutely subject to change!

Building and Managing images

Docker Image
● An executable image that represents a runnable
component (web app, mysql, etc), a set of exposed ports,
and an set of directories that represent persistent storage
across container restarts
– an image may have extended metadata beyond that specified in the
Dockerfile/registry
– how many instances of the image are required for normal function (1,
1..N)
– the environment variables it exposes that must be generated or specified
(beyond the defaults in the Dockerfile itself)
– additional port metadata describing what type of network services are
exposed
– named commands that may be run inside of an image for performing
specific actions

Docker images
● each image (contents and metadata) are immutable and changing
them requires creating a new image (the reference to older image
data or older metadata may be preserved)
● image metadata may change over successive versions of an
image - multiple images may share the same metadata
● any image can be executed (which runs its default
command/entrypoint), but different images may have different
roles and thus not be suitable for execution as a network service
● An example is an image that contains scripts for backing up a
database - the image might be used by a service like cron to
periodically execute tasks against a remote DB, but the image
itself is not useful for hosting the DB
● images are protected by reference - if you have access to a
resource that refers to an image, you can view that image

Build
● A build creates a docker image as output and places that image into a single image
repository.
● there are two defined types of build:
– Docker build takes as input as input a source repository or an archive containing a resource
– Source build takes as input a builder image and a source repository
● A source build is typically based on either an image, OR an image repository
● a build is owned by a project
● a build result is recorded whenever a build happens
– a build provides a webhook resource for remote servers to notify a build that an underlying image or repo or
archive has changed (allows consumers to trigger builds)
– build results history may be removed when it reaches certain limits
● a build has configuration to control whether an automatic build occurs when an input
changes
● a build exposes an endpoint that allows a consumer to POST an archive that results
in a build
– Example: post a WAR file to a source-to-images build that results in that WAR being deployed
– Example: post a zip of the source contents of a Git repo to the build endpoint to result in a Docker image
build using those contents as the Docker build context

Image repository
A set of images with a given name corresponding to a Docker repository in a registry (a set of images
where a single tag represents the latest image that people pull by default). A repository has 1..N tags -
the implicit "latest" tag refers to the image that would be retrieved / used if the user did not specify a tag
●
An image repository has a set of default metadata which is applied to every image created under that
stream, which allows an operator to define an image repository and then push images to the stream and
have that metadata set
●
Use case: allows operators to provide security updates to end users - allows end users to rely on a
continuously maintained piece of software
●
Use case: allow developers and operators to share a set of images across multiple applications and
environments
● An image repository would expose a webhook that allows a consumer to register a new image, as well
as integrating into a Docker registry so that a docker push to a certain repository (probably uniquely
identified by name and a short id) would add that image * Image streams are owned by a project or are
public, and an administrator of that project can grant view access to that stream to other projects.
Anyone who can view an image repository can use it. If access is revoked, any builds or services that
reference that stream can still use it.
●
Retention of images is controlled by reference - deployed services, service deployment history, and
recent builds all take strong references on individual images in a stream and control when images pass
out of scope.
●
Image repositories may be global in scope to a deployment as well as per project

Source Code Repository
● OpenShift 2.x embeds a git repository in the first gear of every application
● OpenShift 3.x, source code repositories are referenced or owned by a service, but are not
physically located inside the gears of that service.
● The APIs for repositories should be flexible to other types of source code repositories, but
OpenShift will focus on Git integration
● When creating a service a user may
– specify that a source code repository be allocated for that service
● the repository will be secured with the keys and tokens of the owning project
● a build will be defined using the "source to images flow" if the image(s) used by the service support STI
● a git postreceive hook will be configured that triggers the build hook, if a build exists
– reference an external source code repository and optionally provide authentication credentials for the repository
(private key, username or password)
● a build will be defined using the "source to images flow" if the image(s) used by the service support STI
● the user will be able to download a commit hook that they can put into their git repository
– specify no repository
● When a source code repository is connected to a build, there is a set of config on that reference that what source code specific
options apply (similar to 2.x):
– Trigger build only a specific branch (optional, defaults to false)
● Whether to automatically build (is the link present or not)
– Any service with a referenced source code repository has a source code webhook exposed via the API which can be used
to trigger the default flow

Scenarios
● A user should be able to easily push a Docker image to
OpenShift and have applications be redeployed
● An integrator should be able to easily notify OpenShift
when the inputs to a build have changed (source code or
base image) as well as notify OpenShift of the existence
of a new image generated by an external build
● Workflows on top of images should be able to refer to an
image repository instead of having to directly specify an
image
● A system administrator can manage the total images in
use across the system by setting retention policies on
accounts

Integration
● Images may be referenced externally (via their full name
<registry>/<user>/<repository>) or pulled into the integrated OpenShift
registry
● An image repository should support a direct docker push with access
control from a Docker client to the integrated OpenShift registry
docker push
my.openshift.server.com/<some_generated_user_name>/mysql-
58343
● An image repository should have a webhook endpoint which allows a
new image to be created when a 3rd party builds an external image
(like the DockerHub)
● In the future it would be nice to be able to automatically pull that
image into the integrated registry
● A build can be triggered by a DockerHub build webhook (for a base
image) or a GitHub commit webhook (for an external source repo)

Multi-tenancy and Isolation
● In OpenShift 2.x, the primary unit of access control was
the domain, which has a membership list and each
member is granted a specific role.
● In 3.x this pattern will continue, but we will rename the
domain to a project to more closely associate the concept
with its common use.

Everything scales
● Stateful – copy state between systems
● Stateless
● 2/3 tier architecture

What is: Container / Gear?
● A running execution environment based on an image and
runtime parameters
– a container encapsulates a set of processes and manages their lifecycle
and allows a unit of software to be deployed repeatably to a host
– gear is the historical OpenShift term for a container and will continue to
be exposed via the user interface

What is: Pod ?
● A set of related containers that should be run together on
a host as a group. The primary motivation of a pod is to
support co-located, co-managed helper programs.
– a pod enables users to group containers together on a host and share
disk storage, memory, and potentially access each other's processes.
– images beyond the first in a pod can be thought of as plugins - providing
additional functionality through composition
– a pod template is a definition of a pod that can be created multiple times

What is: Service ?
● A pod definition that can be replicated 0..N times onto
hosts. A service is responsible for defining how a set of
images should be run and how they can be replicated and
reused by other services, as well as defining a full
software lifecycle (code, build, deploy, manage).
– All services have environment variables, aliases, an optional internal or
external source code repository, a build flow (if they have a source code
repository), and a set of deployments
– a service is a template for a sequence of deployments which create
actual containers. The service has 0..N current deployments, but typically
transitions from one deployment to another.
– a user may customize the ports exposed by an image in a service

What is: Stateful Service ?
● Some images may depend on persistent disk data (such
as a database) - these images have special rules that
restrict how a pod containing them may be replicated,
moved, or configured
– in a stateful service, a "move" operation is exposed for each pod that will
relocate the pod onto another host while preserving the data of that pod
– stateful services may incur downtime when a host is rebooted or fails, but
the system will be designed to minimize that downtime
– by design, stateless services may be aggressively moved and the system
will prefer to create extra pods in new locations and delete existing pods
in the old locations

What is: Service Environment ?
● Each service has 0..N environment variables (key value string
pairs) which may be set by the user, automatically generated
at creation, or automatically provided by a link, that are
available at runtime in each container in the service
– Automatically generated environment variables may represent passwords or
shared secrets, and can be overriden by the user post-creation
– Link environment variables may be overridden by a user environment variable
(to the empty string or a different value) but if the user unsets the user
environment variable the link variable will return
– The pod template for a service may also define container level variables, which
override service environment variables.
– If an image requires a variable be generated, but does not publish it (via a link),
the variable is only defined on the pod template. A published variable is added
to the service environment.
– Changing an environment variable may require a deployment (if only to update
environment settings). It may be desirable to offer a faster change process.

What is: Deployment ?
● A historical or in-progress rolling update to a service that replaces one service
configuration with another (primarily updating the image, but also changing ports,
environment, or adding new images). A deployment is a snapshot of the state of a
service at a point in time, and as such records the history of the service.
– Recording deployments allows a user to see the history of the deployments of a service, rollback to a previous
configuration, and to react to ongoing deployments
– A deployment may be automatically triggered by a build, by manual intervention with a known image, or via an API
call such as a DockerHub webhook.
– Some deployments may fail because the ports the old image exposes differ from the ports the new image exposes
– A deployment in the model records the intent of what a deployment should be - a deployment job carries out the
deployment and may be cancelled or stopped. There may be pods representing multiple deployments active in a
service at any one time.
– A deployment retains a reference to an image - the retention policy of deployments controls which references are
valid, and as long as an image is referenced it will not be deleted.
– A deployment does not control the the horizontal scale of a service, and during a deployment the number of pods
created may exceed the current horizontal scale depending on the type of deployment requested.
– There are different deployment types:
– External - user defines the new deployment, and creates or manipulates the underlying replication controllers
corresponding to the service to add or remove pods for old and new versions
– Simple rolling deploy - pods corresponding to the new template are created, added to the load balancer(s), and
then the old pods are removed from the load balancer and then deleted

What is: Link ?
● A relationship between two services that defines an
explicit connection, how that connection is exposed
environment, a proxy or load balancer), and potentially
whether start order is significant.
– Use Case: A link allows one service to export environment variables to
another service
– Use Case: A link may be used by the infrastructure to create or configure
proxies to service a load balanced access point for the instances of a
service
– A link represents an intent to connect two components of a system - the
existing "add cartridge" action is roughly translated to "create a new
service and link to the existing service with a set of defaults".
– A link as described is different than a Docker link, which links individual
containers. Service links define how sets of Docker links may be created.

What is: Template ?
● A template defines how one or more services can be
created, linked, and deployed
– Use case: A template allows users to capture existing service
configurations and create new instances of those services
– A template also may have a deployment script (an embedded text blob,
URL, or command in a known image), which is invoked as a job after the
service definitions, links, and environment variables have been created
– A template merges the existing quickstart and cartridge concepts to allow
precanned applications or units of software to be deployed.
– A template may be the target of the "create and link" action, which allows
complex components to be templatized and reused. The template must
identify the components that should be linked in this fashion.
– Templates may be global in scope to a deployment as well as per project

Relationships
● Resources are nested:
– 0..N public image repositories
● 1 account
– 0..N projects
● 0..N image repositories
● 0..X owned image repositories
● 0..Y shared image repositories
● 0..M references to images (images are one-to-many repositories)
● 0..N services
● 1 active pod definition (1..M images)
● references 1..M image repositories or explicit images
● additional per image config that becomes the template for containers
● 0..N instances of a pod
● 0..N deployments
● 0..1 source code repositories
● 0..1 builds
● 0..N aliases
● 0..N environment variables
● 0..N links

New Code Service

Build Image using Jenkins and OpenShif

● System Components

System Components – the big picture

Subsystem: API and Images (Purple)
● Implements a high level API that allows end users and
clients to offer an application centric view of the world, vs
a container centric view. Performs project validation on
incoming requests and invokes/triggers the orchestrator
and scheduler as necessary.
● The API allocates source repositories via an API and
configures them to bind to build hooks. The build hooks
can trigger builds via the orchestrator onto hosts as
containers, and then after completion push the new
images into the registry and trigger other orchestrations
like deployments.

Subsystem: Orchestrator (green)
● A job API that schedules sequential execution of tasks on
the cluster (which may run for seconds, minutes, or
hours). Examples include triggering a build, running a
long deployment across multiple services, or scheduled
operations that run at intervals.
● A job creates a "run-once" pod and has an API for
callback, blocking, or status.
● The orchestrator encapsulates all operations that run to
completion that a client may wish to block on.

Subsystem: Scheduler (red)
● The scheduler (and cluster manager) would be Google's
Kubernetes and exposes an API for setting the desired
state of a cluster (via a replication controller) and
managing the cluster.
● The scheduler is related to ensuring containers are
running with a certain config, for autohealing the cluster
as issues develop, and for ensuring individual hosts are in
sync with the master.
● Over time, integration at a lower level with Mesos for fine
grained resource allocation is desirable.

Events (Pink)
● The event subsystem aggregates and exposes events from many
sources to enable administrators, users, and third party observers to
make decisions about the health of a system or container.
●
The policy engine represents a component that may make decisions
automatically based on events from the cluster. Examples of events
include container exit, host unreachable, router detecting failed
backend, node removed, or an alert generated by a hardware
monitoring agent.
● The policy engine is able to decide on the appropriate action in a
customizable fashion. Some events are core to other subsystems and
may flow through alternate means (as an example, Kubernetes may
aggregate exit status of containers via its own API), but the general
goal is to expose as much of the activity of the system to external
observation and review as possible.
● System integrators may choose to expose the event bus to containers.

Subsystem: Routing and Load Balancing
(blue)
● Routers provide external DNS mapping and load balancing to
services over protocols that pass distinguishing information
directly to the router (HTTP, HTTPS, TLS with SNI, and
potentially SSH).
● Routers subscribe to configuration changes and
automatically update themselves with new configuration, and
routers may be containerized or virtual (converting those
changes to API calls to a system like an F5).
● Other automatic capabilities exist to load balance individual
services within the cluster - these would be exposed via
configuration on link relations between services and would
ensure a set of services would be available. Implementations
may choose to implement these as local proxies per host, or
to reuse the shared routing infrastructure.

Subsystem: Containers (yellow)
● The execution of containers is handled by systemd unit
files generated by geard.
● Key access for SSH or additional linking information
would be propagated to hosts similar to the container
configuration.
● The host may also be configured to run one or more
additional containers that provide monitoring,
management, or pluggable capabilities.
● The container subsystem is responsible for ensuring logs
are aggregated and available to be centralized in the
cluster.

Subsystem: Cluster Health (orange)
● The status of containers and hosts is monitored by active
and passive checks over TCP or through process level
calls.
● The state reconciler has the responsibility of acting on
failing heath checks on containers, while the host health
monitor reports failing hosts to the event subsystem for
the policy engine to act on.

Git Repository Hosting
● Rather than forcing Git to be directly integrated into the
hosting infrastructure, OpenShift 3.x will allow external
integration to Git hosting.
● The requirements on a host are to be able to dynamically
create a repository on demand, set git postreceive hooks,
and assign security to those repositories that matches the
ownership model in OpenShift.

Docker Registry
● OpenShift should utilize any server implementing the
Docker registry API as a source of images, including the
canonical DockerHub, private registries run by 3rd
parties, and self hosted registries that implement the
registry API
– In order to connect to private registries, it is expected that authorization
information may need to be associated with the registry source
– It would be desirable to develop plugins that allow smooth integration of
an external access control model with individual repositories in the
registry
– In many systems, imposing quota limits on repositories (size of image,
number of images, total image size) may be necessary

Example Flow: Application Deployment Flow

Example Flow: Git Push Path

Thank You
And now let's DEMO things