What does it take to make sure your containerized application is ready for production

1. Creating Production-Ready, Secure and
Scalable Applications in IBM Cloud Private
(#4789A)
—
Shikha Srivastava
IBM Senior Technical Staff Member
@shikhasthoughts
Michael Elder
IBM Distinguished Engineer - IBM Multicloud Platform
@mdelder
Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM Corporation

2. Please note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice and at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise,
or legal obligation to deliver any material, code or functionality. Information about potential
future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for our
products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in
a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an
individual user will achieve results similar to those stated here.
2

3. IBM Confidential
Notices and disclaimers
3Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM Corporation
© 2019 International Business Machines Corporation. No part of this
document may be reproduced or transmitted in any form without
written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to
products that have not yet been announced by IBM) has been reviewed
for accuracy as of the date of initial publication and could include
unintentional technical or typographical errors. IBM shall have no
responsibility to update this information. This document is distributed
“as is” without any warranty, either express or implied. In no event,
shall IBM be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity. IBM products and
services are warranted per the terms and conditions of the agreements
under which they are provided.
IBM products are manufactured from new parts or new and used parts.
In some cases, a product may not be new and may have been previously
installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product
plans are subject to change or withdrawal without notice.
Performance data contained herein was generally obtained in a
controlled, isolated environments. Customer examples are presented as
illustrations of how those
customers have used IBM products and the results they may have
achieved. Actual performance, cost, savings or other results in other
operating environments may vary.
References in this document to IBM products, programs, or services does
not imply that IBM intends to make such products, programs or services
available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared
by independent session speakers, and do not necessarily reflect the views
of IBM. All materials and discussions are provided for informational
purposes only, and are neither intended to, nor shall constitute legal or
other guidance or advice to any individual participant or their specific
situation.
It is the customer’s responsibility to insure its own compliance with legal
requirements and to obtain advice of competent legal counsel as to
the identification and interpretation of any relevant laws and regulatory
requirements that may affect the customer’s business and any actions the
customer may need to take to comply with such laws. IBM does not
provide legal advice or represent or warrant that its services or products
will ensure that the customer follows any law.

4. IBM Confidential
Notices and disclaimers
continued
4
Information concerning non-IBM products was obtained from the
suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products about this
publication and cannot confirm the accuracy of performance, compatibility
or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of
those products. IBM does not warrant the quality of any third-party
products, or the ability of any such third-party products to
interoperate with IBM’s products. IBM expressly disclaims all
warranties, expressed or implied, including but not limited to, the
implied warranties of merchantability and fitness for a purpose.
The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com and [names of other referenced IBM
products and services used in the presentation] are trademarks of
International Business Machines Corporation, registered in many
jurisdictions worldwide. Other product and service names might
be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at “Copyright and trademark
information” at: www.ibm.com/legal/copytrade.shtml.
Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM Corporation

5. What is Ready for production application
• Secure
• Installation, authentication and access
• Resilient, Highly Available and scale
• Repeated deployment
• with safe upgrades and configuration changes
• Performance
• Observable
• Upgradeable
• more …..
• And AGILE too
Its not MAGIC 

6. 6
What is a
12-factor
app?
https://12factor.net/
• “12-Factor” is a software
methodology for building scalable
microservice applications
• Originally created by Heroku
• Best practices designed to enable
applications to be built with
portability, resilience, and
scalability when deployed to the
web

7. 7
I. Codebase
One codebase tracked in revision control, many deploys
II. Dependencies
Explicitly declare and isolate dependencies
III. Config
Store config in the environment
IV. Backing services
Treat backing services as attached resources
V. Build, release, run
Strictly separate build and run stages
VI. Processes
Execute the app as one or more stateless processes
VII. Port binding
Export services via port binding
VIII. Concurrency
Scale out via the process model
IX. Disposability
Maximize robustness with fast startup and graceful shutdown
X. Dev/prod parity
Keep development, staging, and production as similar as
possible
XI. Logs
Treat logs as event streams
XII. Admin processes
Run admin/management tasks as one-off processes
Why
12 factor
apps?
• Make it easier to run, scale, and
deploy applications
• Keep parity between development
and production
• Provide strict separation between
build, release, and run stages

8. I. Codebase
One codebase tracked in
revision control, many deploys
II. Dependencies
Explicitly declare and isolate
dependencies
III. Config
Store config in the environment
IV. Backing services
Treat backing services as
attached resources
V. Build, release, run
Strictly separate build and run
stages
VI. Processes
Execute the app as one or more
stateless processes
VII. Port binding
Export services via port binding
VIII. Concurrency
Scale out via the process model
IX. Disposability
Maximize robustness with fast
startup and graceful shutdown
X. Parity between dev & prod
Keep development, staging,
and production as similar as
possible
XI. Logs
Treat logs as event streams
XII. Admin processes
Run admin/management tasks
as one-off processes
Code Deploy Operate

9. Developers Dream – code Factors
9
• One codebase for my application
tracked in revision that runs
anywhere: build, ship and run
anywhere
AND
• I can offload deployment, HA,
scaling, upgrade strategy and not
worry about it
Test and
automation
Release AGILE
Design
• Container Images built from
Dockerfiles using trusted small image.
Kubernetes Deployments, etc
managed as YAML (F#I- Codebase)
• Having a strong artifact-driven model
makes it easier to follow a
Continuous Delivery lifecycle (F#V-
Build, release, run)
• Using the same images and YAML
objects make it easier for dev teams
to match what’s running in
production
(F#X- Dev/prod parity)
Develop

10. Pod
(Single IP Address)
Deploy factors
10
• ConfigMaps and Secrets managed in source
repositories or built dynamically via commands
(F#III: Config ). Containers retrieve during runtime
• A collection of Pods can expose or consume
Services via Service port bindings (F#IV : Backing
Services, F#VII: Port binding)
• Container image runs as a container process in a
Pod with other containers (F#VI: Processes )
• Explicitly declare and isolate dependencies(F#II:
Dependencies)
• Running app as a container makes it possible to
capture all logs, metrics, and other management
functions in a consistent way (F#XII: Admin
Process )
Pod
(Single IP Address)
Volume
Volume
container
container
container
Volume
Secret
ConfigMap
Service
Persistent
Volume

11. • Ensure scale for your app
• Replica set ensures specified number of pods are always
running
Service
Replica set
container
Deployment
kind: Deployment
metadata:
name: nginx
spec:
replicas: 2
template:
metadata:
labels:
service: http-server
spec:
containers:
- name: nginx
image: nginx:1.10.2
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
• Is this enough?
Remember load is never constant in the real world
container
Operate factors: Concurrency (F#VIII) & Disposability (F#IX)

12. Operate factors: Concurrency (F#VIII)
Leverage autoscaling to automate computation resources based on load
• Horizontal Pod Scaler (HPA)
• Controls the number of replicas
• Use cpu or memory as a trigger or use
custom metric
• Applicable for stateless app
• Vertical Pod Scaler (HPA)
• Controls the memory and cpu for pod
• Use cpu or memory as a trigger or use
custom metric
• Applicable for statefull apps
Service
Replica set
Pod
Deployment
Pod Pod---
HPA
(policy)
Resource
metric
(usage)
Service
Replica set
Deployment
VPA
(policy)
Resource
metric
(usage)
Pod
Pod
Pod
Pod

13. 13
7
missing
factors
XIII. Observable
Apps should provide visibility about current health and metrics
XIV. Schedulable
Apps should provide guidance on expected resource constraints
XV. Upgradable
Apps must upgrade data formats from prior generations
XVI. Least privileged
Apps should provide guidance on expected resource constraints
XVII. Auditable
Apps should provide appropriate audit logs for compliance needs
XVIII. Access Control (Identity, Network, Scope, Certificates)
Protect app and resources from the world
XIX. Measurable
Apps usage should be measurable for quota or chargebacks

14. IBM Cloud / © 2018 IBM Corporation
Observable: Application health (F#XIII)
Know your application health
• Kubernetes probes
• Is the app ready to accept
traffic?: Readiness
• Is the app responsive? :
Liveliness
• Is this enough?
• What about transactions,
traffic, memory usage ?
App instance 1
App instance 2
Requests
/metric
/metric
livenessProbe:
# an http probe
httpGet:
path: /healthcheck
port: 8080
initialDelaySeconds: 15
timeoutSeconds: 1
readinessProbe:
# an http probe
httpGet:
path: /readiness
port: 8080
initialDelaySeconds: 20
timeoutSeconds: 5

15. Cluster
IBM Cloud / © 2018 IBM Corporation
Schedulable: Resource requests, limits, & quotas (F#XIV)
• Guarantee resources for your containers: Specify request and limits for the compute resources
CPU request: 150 CPU limit: 200
Guaranteed
CPU
Throttle limit
for K8
Max CPU resource for
container
CPU request: 0 CPU limit:0
Max CPU resource for
container
No request and limits are set. It defaults to 0
No guarantees, pods can be preempted any time
Once quota in a namespace for compute resources set, the users are forced to set requests or limits for
those values
• Set resource quota
Namespace 1
Resource Quota :
CPU Limit:500mi
Memory Limit: 1024
MIB
Namespace 2
Resource Quota:
CPU Limit:500mi
Memory Limit: 1024 MIB

16. IBM Cloud / © 2018 IBM Corporation
Upgradable (F#XV)
Applications should be able to roll out updates for cases where
backward compatible updates ( security or feature updates )needs
to be made
Servicecontainer
Deployment
Replica set V2Replica set V1
containercontainer container container container
minReadySeconds: 5
strategy:
# indicate which strategy
# we want for rolling update
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0

17. Cluster
IBM Cloud / © 2018 IBM Corporation
Least Privilege (F#XVI)
• Limit container access to hosts
• Every permission is an attack vector
• Use Pod Security Policy and Network Policy
to
• Limit access to filesystem
• Limit access to Kernel capabilities
• Use a non-privileged user
• Limit access to volume types
• Limit access to ports
container
container
container
container
#sample-psp.yaml
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: example
spec:
privileged: false
# Don't allow
# privileged pods!
# The rest fills in some
# required fields.
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
volumes:
- '*'
Compromised

18. IBM Cloud / © 2018 IBM Corporation
Auditable (F#: XVII)
• Know WHAT/WHEN/WHO/WHERE for all CRUD operations
• Chronological set of records documenting sequence of events affecting
system and application by users or components
• Use cloud agnostic industry standard format – CADF (Cloud Auditing Data
Federation)
• Control the quantity of logs
CADF event:
<initiator_id>: ID of the user that performed the operation
<target_uri>: CADF specific target URI, (for example: data/security/project)
<action>: The action being performed, typically: <operation>. <resource_type>

19. IBM Cloud / © 2018 IBM Corporation
Access Control -Identity, Network, Scope F#: XVIII )
• Protect app and resources from the world
• Authentication and Authorization
• Certificate Management
• Data Protection
• Network security
• Network policy
• Network Isolation
• Admission Controller
• Example: Image admission controller

20. Access Control :Identity, Network, Scope (F#: XVIII)
• Ensure secure communication
• Generate Certificates
• Enable TLS / mTLS
• Manage Certificates
letsencrypt-stagingletsencrypt-prod icp-root-ca
signed
keypair
signed
keypair
Certificate Manager
signed
keypair
k8s Secret
Issuer
Certificates
Secrets
1. Issuer creates Certificate
2. Certificate creates secret
3. Secret mounts to
Pod
IP: 9.37.239.158
Issuer: icp-root-ca
IP: 9.37.239.158
Issuer: icp-root-ca
Example.com
Issuer: letsencrypt-prod
# sample issuer.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: demo1-nginx-ca
namespace: demo
spec:
ca:
secretName: demo1-nginx-ca-key-pair
# sample certificate.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
Metadata:
name: demo1-nginx-cert
spec:
secretName: demo1-nginx-cert
issuerRef:
name: demo1-nginx-ca
kind: Issuer
commonName: "foo1.bar
dnsNames:
foo1.bar1
Pod

21. Cluster container
container
container
container
container
container
---
IBM Cloud / © 2018 IBM Corporation
Measurable(F#: XIX)
Dept/org 1 Dept /orgn
Total usage
Cluster containercontainercontainercontainer containercontainer
IT cost
Dept /org 2 ---
---
• Know the cost of the application
• Compute resources allocated to run the containers should be measurable
• Org / department using the cluster should be accountable

22. So, What really
makes a
production-
ready app 22

23. IBM Cloud / © 2018 IBM Corporation
A production grade application
Production
thinking needs
to be through
the entire
process
Attention to
Building containers and what's inside the containers
Example: Factor I : codebase , Factor X: dev/prod
parity, Factor XV
Attention to
Kubernetes configuration
Example: Factor III: Config, Factor II Config, Factor
XIV: Schedulable
Attention to
Cloud provider configurations
Example XII: Observable, Example: XVIII: Access
Control. Factor XIX:: Measurable

24. IBM Cloud / © 2018 IBM Corporation
IBM Cloud Private (ICP)
24
• Self-service rich catalog of IBM MW
• Helm based parameterized install to
simplify complex K8 apps
• Logging : ELK + filebeat
• Monitoring : Prometheus + Grafana
• Usage : IBM Metering Service
• IBM Vulnerability Advisor
• IBM Mutation Advisor
• Authentication/ Authorization
• Certificate Management
• Network security
• Audit trail for any CRUD operations
• Team based organization of resources
Enterprise Content Catalog
Open Source and IBM Middleware, DevOps,
Data, Analytics, and AI Software
Core Operational Services
Logging, Monitoring, Metering, Security, Alerting
Kubernetes Container
Orchestration Platform
IBM Z
Choice of
infrastructure:
All communication enabled over TLS.
Data secured in transit and at rest
Provides the capabilities to run containerized application in secure, scalable and resilient environment

25. Enough
talking,
let’s see it
LIVE! 25

26. Leverage the IBM Cloud Garage
Method to change how you work.
26Think 2018
Provides an in-depth collection of
practices, tutorials, and
architectures to help you on this
journey.
Completely open forum for learning
at your own pace.
We offer hands-on guidance and
services, if needed.
Defined
Practices
Business
Benefits
Technical
Benefits
ibm.com/cloud/garage >

27. 27
Learn
more in
our new
book!
#7678A: Tech Talk:
Deploying Kubernetes in
the Enterprise (with the
authors)
When: Wednesday, 11:30
AM - 12:10 PM
Where: Table Top Tap Room
at the Metreon | Code Cafe
Tech Talks Area
Get a signed copy with all
of the authors at the Code
Café Mezzaine on
Wednesday (7 – 7:30PM)!
ibm.biz/BdYA4i >Now available online compliments of IBM:

28. Read the white paper!
28
Think 2019 / DOC ID / Month XX, 2019 / © 2019 IBM
Corporation
http://ibm.biz/Ovum-WP
84% improvement in speed to market
75% reduction in operational expenses
60% increase in projects IT
Administrators can handle
50% reduction in service desk calls

30. 30
®
https://www.ibm.com/legal/us/en/copytrade.shtml