SlideShare a Scribd company logo
1 of 39
Download to read offline
ING Container Hosting Platform, 3 years onward
ING Tech Infra&Engineering
Growing and sustaining a Private Cloud Container Hosting Platform
October 2022
2
Introductions
ING is a global bank with a strong European base. Our more than
57,000 employees serve around 38 million customers, corporate
clients and financial institutions in over 40 countries. Our
purpose is to empower people to stay a step ahead in life and in
business.
* ING is undergoing a transition to close our Wholesale Banking offices in Argentina, Brazil and Kazakhstan as
announced on 5 November 2020. ING has exited the Austrian and Czech retail banking markets as of the end of
2021. We announced in December 2021 that we will leave the retail banking market in France after a strategic
review of our retail banking operations there. We announced June 24th 2022 we will leave the Philippines retail
banking market as of then end of 2022.
For INGā€™s position regarding Russia and the Ukraine see our Feb 28 2022 statement
Thijs Ebbers
Architecting Cloud Native @ING
since 2016 (employee since 2001)
Architecture Lead for the
Runtime Domain (ā€œVM & Container
Hostingā€), for ING Private & Public
Clouds
Sandeep Kaul
IT Area Lead (ā€œHead of ITā€) for
Cloud/PaaS, a.o. responsible for
the delivery of ICHP towards
internal customers.
3
Short Recap
At 2019ā€™s San Diego OpenShift Commons ING presented ā€œRunning Containers in Production, the ING Storyā€
where we introduced INGā€™s Container Hosting Platform (ā€œICHPā€) and explained:
ā€¢ the drivers behind our transformation
ā€¢ the ā€œCloud Native ecosystem Kubeā€ we use(d) to communicate with our stakeholders
ā€¢ ICHPā€™s deployment model & its delivery patterns
ā€¢ which topics took the most time during the first 2 years of engineering and operation.
So for the remainder of this presentation we assume this content is known ;-)
and if not catch up afterwards :
4
So what happened since ? (aka agenda)
ā€¢ Covidā€¦
ā€¢ Expansions
ā€¢ Divisions
ā€¢ Security
ā€¢ LCM
ā€¢ Exponential growth
ā€¢ Bugs
ā€¢ Resilience
ā€¢ Partnerships
ā€¢ (Refined) Strategy
5
ICHP ā€“> ICH* (aka abstracting for multi cloud)
ING Container Hosting Platform (ā€œICHPā€) was INGā€™s first standardized Container Hosting environment, but
it has evolved into a family we now call ICH which consists of:
ā€¢ The private cloud ICHP platform based on OCP
ā€¢ The ICHA platform on Azure AKS (still being engineered)
ā€¢ The ICHG platform on GCP GKE (still being engineered)
All these platforms abide to the same defined interfaces to enable application portability between the
ICH family members, for both re-useability as well as regulatory reasons. And making deployments to
multiple target clouds with defined interfaces mandatory will empower our engineers with the skills they
need.
Why not use (managed) OpenShift on Azure & GCP ?
ā€¢ ING believes the value of public clouds is in the integrated ecosystem and the cloud vendors native
services will be the 1st to benefit from innovations in those ecosystems. This means we should
consume SaaS rather then PaaS rather then IaaS to make use of those innovations as soon as possible.
ā€¢ Hence we consume Azure and GCP native services wherever possible and we try to avoid IaaS services
For the remainder of this presentation weā€™ll focus on ICHP
Note : 2 of our ING colleagues are presenting this KubeCon on ā€œHPC Batch computing for Analytics Workloads ā€œ
(Wednesday 2:30 pm), attend and learn but be aware this environment is not (yet) hosted on ICH*
K8s technology = Amazing
ā€¢ Almost infinite # use casesā€¦
But
ā€¢ Mixing multiple distinct use cases
together in the same clusters =
a recipe for (regulatory) disasterā€¦
This table shows the attempt of ING to
categorize in order to give each service
its appropriate (risk&security) treatment.
white = (partially) available as a service
grey = currently not available as a service
This table is not intended to be staticā€¦
Meaning ING is very opinionated on
what workloads should be hosted on our
Immutable container hosting serviceā€¦
Donā€™t make any wrong assumptions,
we do get a lot of requests for hosting
software which in our opinion belongs
on services currently unavailable (grey)
ā€œNoā€ is a valid answerā€¦, as ING Tech chooses to focus on a limited set (since we have scarce engineering capacity). Spinning up those other
services would require additional DevOps squads to engineer and run those services. Requesters can still run that software on VMā€™s...
Service Typical use case(s)
Immutable
Containers
Capability to host immutable container workloads like 12 Factor Apps (commonly: ā€œAPIā€™sā€)
Functions as a
Service
Capability to execute backend code directly without having to manage any servers
Data Services
Capability to provide data persistence as a service e.g. Object Storage, Elastic, Kafka, MS-SQL,
Oracle, Cassandra, Redis, etc.
HPC
Capability to provide High Performance Compute (e.g. GPU, fast local cache, etc.) as a service
for machine learning and analytics.
Centralized End-
User
Capability to provide a managed, dedicated set-up for specific user(s) e.g. for data science,
educational and training set-ups etc.
Infrastructure
Management
Capability to manage IT assets as Desired State Custom Resources (e.g. Crossplane)
Non-12 factor
apps (ā€œVMā€s) ā€“ K8S
Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images
containing monolithic applications, not ready for hands off operations, could be requiring
managed local persistency. Hosted on a K8s or equivalent platform.
Standalone - Non-
12 factor apps
(ā€œVMā€s)
Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images
containing monolithic applications, not ready for hands off operations, could be requiring
managed local persistency. Hosted on Standalone ā€œDockerā€ instances.
6
Divide & Conquer (aka Distinct Services deserve Distinct Treatment)
7
Security
ā€¢ ING Container Security Standard in place as of Q2
2019
(ā€œZero Privilegeā€, do not trust ā€œZero Trustā€ā€¦)
ā€¢ Comprehensive Vulnerability Scanning in
OnePipeline enforced as of Q4 2021
(thank you Log4J for the priorityā€¦)
ā€¢ Immutable Workloads, enforced as of Q1 2022
(privileged NPAā€™s removed from production namespaces,
Break Glass available, mandatory redeploy afterwards)
ā€¢ Immutable Platform (Clusters), as of Q3 2022
(OCP 4.10.30 IPI for Bare Metal dependency)
ā€¢ Anomaly Detection & Policy-as-Code capability as
of Q3 2022 (OCP 4.10/ K8s kernel version dependency)
ā€¢ As of Q3 2022 ICHP has implemented all controls
as specified in the Container Security Standard
8
OpenShift LCM
Generic learnings:
ā€¢ Not all consuming DevOps teams understand the full scope of going Cloud Native and might be unaware of their
shortcomings
ā€¢ In particular we noticed the value of automated testing wasn't always recognized, which did hamper our Platform
LCM
ā€¢ Application redeployment should be ā€œthe new normalā€, not ā€œsomething to be scared ofā€ā€¦
Migrating OpenShift 3.11 OKD to OpenShift 3.11 Enterprise (Q1 2020):
ā€¢ Clusters were redeployed (1 DC at a time), forcing downtime (in practice a DR-excersize) upon our consumers. This
wasnā€™t well received and led us to rethink our migration strategy for the next LCM momentā€¦
Migrating OpenShift 3.11 Enterprise to OpenShift 4.10 Enterprise (ā€œICHP v2ā€) (Q3 2022):
ā€¢ Running on Bare Metal nodes doesnā€™t help if new OpenShift
versions are first released for VM based clustersā€¦ It took Red Hat
until version 4.10.30 to deliver the quality ING needed to do a
fully automated IPI based install in our DCā€™s (which was kind of
challenging looking at the EOS of OCP 3.11ā€¦)
ā€¢ This time we opted to spin up new clusters in parallel to the old
environment, and offer our customers a migration window to
redeploy their workloads. We opened the migration Sep 15th 2022
for our Non-Prod clusters and our Prod clusters are planned to go live this week
More details on how we build our NaaS on top of OCP 4.10 IPI in the other ING presentation today
In Q3 2019 ICHP had 18 FTE and hosted :
ā€¢ 250 Prod namespaces whose APIā€™s only
processed a marginal number of transactions
(most still preparing for actual workloads)
ā€¢ about 1000 Non-Prod Namespaces
In Q3 2022 ICHP hosted:
ā€¢ 600 namespaces in Production whose
APIā€™s processed 50% of ING retail customer
transactions
ā€¢ 1690 namespaces in Non-Production
And did this with the same 18 FTE, we only
added an additional Ops squad to manage
the environment (+6 FTE) from a partner
(hired as of Q1 2022, for new total of 24 FTE)
9
Growing pains : exponential growth for Immutable App hosting
So looking back ICHP coped with a sustained 100% growth each year (measured in installed CPU/memory
capacity) (for the last 3 years) & this growth is expected to continue into 2023. As a result the price/unit
has now become very competitive and ICHPā€™s Immutable Container hosting in this way supports the
broader ING Tech Strategy to enable to grow our business at marginal cost
Green line is actual ICHP Prod DC1
usage (Prod DC2 usage is similar)
Red line is available ICHP Prod DC1
capacity (Prod DC2 capacity is identical)
Drop in capacity end of August ā€™21
due to bug in OpenShift 3.11 icw
Atomic requiring ICHP to scale
back from 250 pods to 160 pods
per node to become stable again
(with OpenShift 4/CoreOS its fixed)
Customers were impacted (mainly
in the non-Prod environment) in
the Q3 2021 due to this shortage
As a provider we had to scavenge
additional nodes everywhere to
remain stableā€¦
The drop in Feb 2022 was due to an
unintended reboot of multiple nodes by
one of our suppliers
Growing pains : Bugs (pending pods / pods per node)
10
Pending pods ->
Pods/node back to 160
Running out of
capacityā€¦
Blue Line
measuring
ā€œpending podsā€
introduced
Q1 2022
Multiple outages impacting ING customers on workloads hosted on ICHP (For the record : ICHP has had zero
unplanned platform downtimeā€¦). Analysis concluded most were avoidableā€¦ by properly defining K8s
application deployment configs. Lesson learned : Developer Autonomy is good thing, but never forget to
validate the resultsā€¦
Q2 2022
ā€¢ ING CTO&CIOā€™s decided on mandatory (not yet enforced) use of :
canary releasing
resilient pod placement
liveness & readiness probes
ā€¢ ICHP published ING specific workload resilience documentation
Q3 2022
ā€¢ ICHP provides Dashboards giving ING CIOā€™s (& developers) insight in
the quality of the deployments on ICHP
ā€¢ Additional internal templates & resilience training materials being developed (by partner enabling teams)
Future Step : Policy enforcement via Policy-as-Code (K8s deployment config insufficient -> deploy job fails)
11
Growing pains : workload resilience
12
ICHP Partnerships
Ingested products
(products or data provided by other platforms
that are exposed through this platform)
supplied-services
(provided by other platforms
required for platform functions)
engineered product builds
(developed and maintained by one or multiple
squads)
CONSUME ENGINEER
INGEST
Exposed platform
services
EXPOSE
Exposed product builds
(semi finished product used as supplied
product build by the consuming
platform)
Complement
(add value)
ING ā€œBTPā€ (Topics & Service Mesh & Workload Deployment Templates)
ING ā€œIPCā€ (DBaaS, Keyspace-aaS, Object Storage, Cloud Automation)
ING ā€œSecurity Tribeā€ (IAM & Workload Security Monitoring & Workload Certs)
ING ā€œMDPLā€ (Workload Observability)
ING ā€œOne Pipelineā€ (Workload CI/CD)
ING Banking Technology Platform
ING Group Services
ING Retail Banks
ING Wholesale
ING Infra&Engineering
ING ā€œDCNā€ (Networking)
ING ā€œIPCā€ (Object Storage)
ING ā€œMDPLā€ (Platform Observability)
ING ā€œOne Pipelineā€ (Platform CI/CD)
ING ā€œSecurity Tribeā€ (IAM & Platform Security Monitoring & Platform Certs)
ING ā€œIPCā€ (Nodes BM & VM)
Red Hat (OpenShift)
Portworx (local persistency)
13
Strategy (part 1 : why do we do what we do ?)
INGā€™s Tech strategy : ā€œTo build Scaleable Tech which enables Growing our business at marginal costā€
ā€¢ ICHP is a core example of technologies engineered by ING to support this goal, as explained earlier
Additional Strategy goals are ā€œSelf-Serviceā€, ā€œRe-useabilityā€ & Automation
ā€¢ The immutable container hosting is delivered as a Namespace-aaS from our private cloud portal
ā€¢ The ICHP platform hosts applications from almost every ING segment and for most of the countries
ā€¢ The ICHP platform is fully automated, both for Namespace & Cluster provisioning (ā€œImmutableā€)
ICHPā€™s Unique Selling Point : Compliance-out-of-the-Box
ā€¢ Immutability is the key tool in achieving this goal. Hence ICHP focused on Immutable container hosting
over the past period and this will stay itā€™s core delivery
ING Tech is looking into ā€œTeam Topologiesā€ as a way to optimize the internal
organization and as it turns out ICHP ticks all the boxes for a
ā€œThinnest Viable Platformā€ (since 2018, the book was published in 2019ā€¦)
Our platform is compelling for consumers, with our NaaS approach which reduces
cognitive load, it simplifies the risk evidencing burden & we are strongly partnered
with key ā€œstream alignedā€, ā€œenablingā€ & ā€œplatformā€ teams in different ING segments
to help guide our backlog/roadmap
14
Strategy (part 2 : whatā€™s next?)
For the Immutable container hosting service thereā€™s still work to be done:
ā€¢ Pets -> Cattle for the clusters, which is aided by
ā€¢ Hypervisor based Nodes (currently all production is on Bare Metal)
ā€¢ Multi-Cluster Management (ā€œMCMā€) : Pattern(s)/Solution(s) to be investigated
ā€¢ Anomaly Detection and Policy-as-Code capabilities have been released with ICHPv2 based on
OpenShift 4.10, but scope wise we have lots of opportunities to expand rulesetsā€¦ Workload resilience
will be high on that backlogā€¦
But ING does recognize other K8s use cases as explained before:
ā€¢ ICHP is hosting Elastic & Prometheus environments on dedicated
OpenShift clusters (with local persistency)
ā€¢ ICHP is working on a Cluster-aaS delivery for Platform Teams
ā€¢ And ICHP is tracking the maturity of the market:
ā€¢ To determine when best to start investing our scarce engineering resources (Scarcity does create focus...).
ā€¢ The Multi-Cluster Services (ā€œMCSā€) developments have our special attention (perhaps Skupper + KCPā€¦?)
In the mean time non-Immutable workloads can still be hosted on our Private Cloud VM offeringā€¦
(no exceptions/no specials in our multi-tenant Immutable Container hosting, or else we risk losing our USPā€¦)
Last but not least we strive to be a more engaged participant in the community, more on that topic in the
other ING presentation todayā€¦
Of course weā€™re hiring ;-)
https://www.ing.jobs/tech
15
Thank you / Questions
Slides will be made available at
https://www.slideshare.net/ThijsEbbers/
Appendices
Additional materials for your understanding
Whilst we treaded the eye of the needle
in our Production clusters when we hit the
ā€œPending Pods bugā€, we werenā€™t so ā€œluckyā€
with our Non-Production clusters.
On several occasions consumer (DTA)
workloads did encounter negative impact,
and since hardware replenishment was
difficult we moved D/T workloads to
newly added VM based nodes (which
explains the decline in the green line) in
order to free up Bare Metal nodes to be
used to stabilize the Production clusters
as well as to keep the Acceptance
workloads as stable as possible on the
remaining Bare Metal nodes
Meanwhile, in Non-Prodā€¦
17
Pods/node
back to 160
Pending
pods
Introduction
of additional
(VM based)
nodes
Non-Prod
Environment
stabilized
ā€¢ From INGā€™s Daily Banking Tribe talk at GOTO 2022 conference (ā€œSpotify Model : On Radical
Improvements From The Trenches Of Changeā€):
ā€¢ Deployment time : 62% build & deploy speed improvement
ā€¢ Development capacity : 27% more Backlog items done (measured per year in 2020/21)
ā€¢ More Coding : 16% more coding time per team
ā€¢ No more VMā€™s : Less infra costs, no patching, more secure, less issues with auditing the stack
(More cool ING Tech talks on our YouTube Channel)
ā€¢ Building a Global Investments Platform in 12 Months (2020-21)
ā€¢ Building a Digital Platform for Insurance Products (ā€œING-AXA partnershipā€) in 9 Months (2019)
ā€¢ Building a (Mobile Only Digital) Bank in 9 Months (ā€œThe Manila Miracleā€) (2018)
ā€¢ And more to comeā€¦
ICHP(ā€˜s-ecosystem) enabled our Internal ING Customers to achieve :
18
Platform (ā€œICHPā€) Adds e.g. :
ā€¢ ING Compliance
ā€¢ ING Integrations (SEM,
Monitoring, AZDO, ā€¦)
ā€¢ ING Hardening
Distribution (e.g. ā€œOpenShiftā€, ā€œRKEā€, ā€œGKEā€, ā€œAKSā€)
Adds e.g. : RBAC, SDN
Container Management Framework
(ā€œKubernetesā€)
Adds e.g. :
Scaling, resilience
Application Containers to Container Hosting Platforms
19
Containers
ā€¢ Important aspect to be taken into consideration: The ING Container Hosting Platform Immutable
Container hosting is NOT aiming to rehost VMā€™s to containersā€¦ Purpose is to have the best possible
hosting environment for immutable workloads!
ā€¢ If a DevOps teams manages to properly refactor their VM into a set of container images they are
welcome
ā€¢ Hands-off approach in production (which implies a mature team in all aspects, e.g. CI/CD !)
If teams feel not comfortable with this they should stay on VMā€™s!
ICHP Immutable App hosting Intended Use
20
ING aims to make our DevOps teams autonomous (As explained in INGā€™s Way-of-Working in 2019) .
We also want to enable those DevOps teams to deliver maximum value to the business, by not bothering them
with IT-Infrastructure concerns, nor bothering them with having to deliver compliancy evidence for the hosting
platform.
Hence offering a Namespace-as-a-Service is for us the sweet spot:
ā€¢ Clear demarcation between (Infra)Provider and Consumer
ā€¢ Enabling hand-over of compliance evidence
ā€¢ Enabling multi-tenancy (hence fast time-to-market/self-service consumption, and the potential for efficient
utilization of resources)
ā€¢ The DevOps team can assume responsibility for (almost) the full stack (only the kernel stays shared). They have
liberty/responsibility to choose/maintain their (versions of) base image, runtime engine, libraries, etc. (within
the boundaries set by the ING corporate risk&compliancy rules !)
ING will offer a K8S Cluster-as-a-Service in the future, however this will be a limited offering only available to
selected platform teams.
Why ICHP only offers a ā€œNamespace-as-a-Serviceā€
21
Key Objective for ING is to remain capable to exit a Public Cloud provider.
Determining the right abstractions is key to meeting this objective for a container hosting platform,
hence:
ā€¢ We choose an open/industry standard (in this case: the Kubernetes ecosystem)
ā€¢ We choose an intended use (in this case: Immutable Workloads)
ā€¢ We clearly define the allowed interfaces (in the ICH ā€“ Interfaces standard)
ā€¢ We automate our deployments by the mandatory use of One Pipeline
Note that these abstractions are design time and not runtime (as we still use the AKS service native to
Azure or the GKE service native to GCP)
However there will always be debate on the details as we try to balance innovation, time-to-market,
risk/security and/or cost-income ambitions. Therefore actually running production workloads on at
least one Kubernetes stack outside of the primary (Public) Cloud provider is the best recipe to ā€œkeep
everyone honestā€, as this will prevent Cloud Provider specific ā€œoptimizationsā€ from slipping into ING
code. Meaning there should always be at least 2 ICH standard solutions in use.
Cloud Abstractions
22
We do know how to build those services in greyā€¦ Our engineers are more then capable to build and
run those. But not all of them together at the same timeā€¦
Hence for each year ING determines where to allocate that engineering capacity:
ā€¢ Improving existing services
ā€¢ e.g. Pets -> Cattle
ā€¢ Supporting additional use cases on existing services
ā€¢ e.g. Data Services we today only support ELK & Prometheus, if we start to support Data Persistence Services
more critical to the bank (e.g. message bus, (non-)relational databases) weā€™d actually like the availability of
those Services to be independent from K8s cluster uptime (hence our interest in CNCF MCS developments)
ā€¢ Introducing New Services
ā€¢ We do have active instantiations of Centralized End-User & HPC use cases on K8s in ING (one is even presenting
@KubeCon Detroit). But those are currently not hosted as a centralized/standardized Service. So we could also
choose to include those in the ICH catalogue
ā€¢ The one defined service weā€™re not likely to invest time in any time soon is FaaS, that has everything to do
with the controls currently enforced from a regulatory perspective and little with the actual technologyā€¦
Dive and Conquer additional information
23
IPC/ICHP hosting services (1/2) (Grey is currently not available)
24
Service Typical use case(s)
Immutable Containers Capability to host immutable container workloads like 12 Factor Apps (commonly: ā€œAPIā€™sā€)
Functions as a Service Capability to execute backend code directly without having to manage any servers
Data Services
Capability to provide data persistence as a service e.g. Object Storage, Elastic, Kafka, MS-SQL, Oracle,
Cassandra, Redis, etc.
HPC
Capability to provide High Performance Compute (e.g. GPU, fast local cache, etc.) as a service for machine
learning and analytics.
Centralized End-User
Capability to provide a managed, dedicated set-up for specific user(s) e.g. for data science, educational
and training set-ups etc.
Infrastructure
Management
Capability to manage IT assets as Desired State Custom Resources (e.g. Crossplane)
Non-12 factor apps
(ā€œVMā€s) ā€“ K8S
Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images containing
monolithic applications, not ready for hands off operations, could be requiring managed local persistency.
Hosted on a K8s or equivalent platform.
Standalone - Non-12
factor apps (ā€œVMā€s)
Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images containing
monolithic applications, not ready for hands off operations, could be requiring managed local persistency.
Hosted on Standalone ā€œDockerā€ instances.
IPC/ICHP hosting services (2/2) (Grey is currently not available)
25
Service Unit of Delivery
(/Security)
Local
Persistency
GPU Ingress Exposed
interface(s)
End-user access to
SSH/Terminal Interfaces
Immutable
Containers
Namespace Not Available Not Available
Shared per
Traffic type
(Internal/DMZ/ā€¦
)
API (HTTPS ingress),
Eventing (ā€œKafkaā€),
File (/Object) (ā€œS3ā€)
Not Allowed (in Production)
(currently still tolerated in Non-
Production clusters)
Functions as a Service Function
Not Available
TBD Shared
See ā€œImmutable
Containersā€
Not Possible
Data Services
(set of)
Namespace(s)
Available Not Available
Dedicated per
workload
TBD Not Allowed
HPC Namespace Optional Optional TBD
See ā€œImmutable
Containersā€
Not Allowed
Centralized End-User Namespace TBD TBD TBD SSH/https/ā€¦ Allowed
Infrastructure
Management
Control plane Not Available Not Available Not Applicable K8s API Not Allowed
Non-12 factor apps
(ā€œVMā€s) ā€“ K8S
Namespace, VM ? TBD Not Available TBD TBD TBD
Standalone - Non-12
factor apps (ā€œVMā€s)
VM Available Not Available Dedicated No restrictions Available
ā€œImmutableā€ =
ā€¢ Any change can only be executed via an ING pipeline which has all required checks &
balances in place (and which facilitates automated evidencing for regulatory purposes)
ā€¢ In short : no direct access for natural persons, no change privileges for other systems
And yes, INGā€™s Container Security Standard was in place before ā€œSolarwindsā€ reached the
headlines in 2020ā€¦
And although most people in the IT(-Security) industry are now aware of the software
lineage issue exposed by it (that doesnā€™t mean its anywhere near solvedā€¦), far too few
understand itā€™s 2nd lesson about the dangers of overprivileged softwareā€¦ (what enabled
those attackers to move laterally so easilyā€¦?)
For these 2 reason ING does not trust ā€œZero Trustā€ labelled implementations as very few of
those have actually covered these concerns properly...
Security additional explanation
26
Range Of Container Technology Usage Models
27
28
Container Technology ING Security Reference Architecture
Image Storage and Retrieval
Container Deployment,
Management, and Operations.
Image Creation, Testing, and
Accreditation
DevOps
DevOps
DevOps
Host with
Containers
Host with
Containers
Host with
Containers
Host with
Containers
Build, Deploy,
Test, Release
Development
Registry
Any Other
Registry
Authorized
Registry
Deployment
Registry
Orchestrator
Container
Management
Framework
No Connection
No Connection
W.01 W.02 W.03
W.04 W.05
I.06 I.07 I.08 R.09 R.10 O.11
P.14 P.17 P.20 H.21
P.15 C.12
P.18 H.22
P.16 C.13
P.19 H.23
29
Container Technology Security Measures Overview
Design & Code
(digitize)
Build & Test
(create)
Operate & Maintain
(detect & correct)
Deploy & Test
(assembly)
Release
(orchestrate)
Onboard and Plan
(define)
Engineering Journey
R.09
R.10
Registry
Measures
O.11
Orchestrator
Measures
I.06
I.07
I.08
Image
Measures
Measures to be implemented by
providers of CI/CD Pipeline for container
technology e.g. GEP
P.14
P.15
P.16
P.17
P.18
P.19
P.20
Platform
Measures
H.21
H.22
H.23
Host
Measures
C.12
C.13
Container
Measures
Measures to be implemented by
providers of container hosting platform
(except H.23) e.g. ICHP
W.01
W.02
W.03
W.04
W.05
Workload
Measures
Measures to be implemented by
providers of container workloads e.g.
DevOps
ā€¢ Want to understand how to :
ā€¢ Use Canary (Blue/Green) releasing using K8s cluster external loadbalancing services
ā€¢ Use Canary (Blue/Green) releasing using K8s cluster internal capabilities
ā€¢ Revert my traffic 100% to my stable version in case of unexpected issues
ā€¢ Rollback my release to the N-1 version in case of unexpected issues with version N
ā€¢ Perform Resilient Placement so a single Red or Blue outage in the underlying IaaS
layers will not affect my application availability
ā€¢ Implement Liveness Probes so my pods will restart automatically in case of applicative
issues
ā€¢ Implement Readiness Probes so I have the input to automatically direct my traffic in
case of applicative issues
ā€¢ And would like to see examples & templates how to use above features in INGā€™s One
Pipeline
Workload Resilience : As an ICHP Consuming DevOps Team, I
30
The following slides come from ING Investor Update June 13th 2022 Tech& Operations
33
Strategy additional explanation
34
35
Construct your own Kube
36
Follow us
SlideShare.net/ING
@ING_News LinkedIn.com/company/ING
YouTube.com/ING Flickr.com/INGGroup
Facebook.com/ING
ing.com
ingwb.com
Medium.com/ing-blog
ING Container Hosting Platform - 3 years onward_with Kube_for distribution.pdf
ING Container Hosting Platform - 3 years onward_with Kube_for distribution.pdf

More Related Content

What's hot

KCD Italy 2022 - Application driven infrastructure with Crossplane
KCD Italy 2022 - Application driven infrastructure with CrossplaneKCD Italy 2022 - Application driven infrastructure with Crossplane
KCD Italy 2022 - Application driven infrastructure with Crossplanesparkfabrik
Ā 
Kubernetes Introduction
Kubernetes IntroductionKubernetes Introduction
Kubernetes IntroductionMartin Danielsson
Ā 
Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...
Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...
Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...SlideTeam
Ā 
Cloud Native Application
Cloud Native ApplicationCloud Native Application
Cloud Native ApplicationVMUG IT
Ā 
12 factor app an introduction
12 factor app an introduction12 factor app an introduction
12 factor app an introductionKrishna-Kumar
Ā 
Cloud native principles
Cloud native principlesCloud native principles
Cloud native principlesDiego Pacheco
Ā 
Introduction to openshift
Introduction to openshiftIntroduction to openshift
Introduction to openshiftMamathaBusi
Ā 
Introduction to Docker - 2017
Introduction to Docker - 2017Introduction to Docker - 2017
Introduction to Docker - 2017Docker, Inc.
Ā 
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREAraf Karsh Hamid
Ā 
Combining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityCombining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityElasticsearch
Ā 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupDATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupSaewoong Lee
Ā 
ING microServices
ING   microServicesING   microServices
ING microServicesAndrei Rugina
Ā 
Kubernetes Basics
Kubernetes BasicsKubernetes Basics
Kubernetes BasicsEueung Mulyana
Ā 
Introduction to Red Hat OpenShift 4
Introduction to Red Hat OpenShift 4Introduction to Red Hat OpenShift 4
Introduction to Red Hat OpenShift 4HngNguyn748044
Ā 
Container Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with TrivyContainer Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with TrivyFaheem Memon
Ā 
Microservices Architecture - Cloud Native Apps
Microservices Architecture - Cloud Native AppsMicroservices Architecture - Cloud Native Apps
Microservices Architecture - Cloud Native AppsAraf Karsh Hamid
Ā 
Transforming Organizations with CI/CD
Transforming Organizations with CI/CDTransforming Organizations with CI/CD
Transforming Organizations with CI/CDCprime
Ā 
Pivotal Cloud Foundry: A Technical Overview
Pivotal Cloud Foundry: A Technical OverviewPivotal Cloud Foundry: A Technical Overview
Pivotal Cloud Foundry: A Technical OverviewVMware Tanzu
Ā 
Kubernetes #2 monitoring
Kubernetes #2   monitoring Kubernetes #2   monitoring
Kubernetes #2 monitoring Terry Cho
Ā 

What's hot (20)

KCD Italy 2022 - Application driven infrastructure with Crossplane
KCD Italy 2022 - Application driven infrastructure with CrossplaneKCD Italy 2022 - Application driven infrastructure with Crossplane
KCD Italy 2022 - Application driven infrastructure with Crossplane
Ā 
Kubernetes Introduction
Kubernetes IntroductionKubernetes Introduction
Kubernetes Introduction
Ā 
Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...
Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...
Devops Strategy Roadmap Lifecycle Ppt Powerpoint Presentation Slides Complete...
Ā 
Cloud Native Application
Cloud Native ApplicationCloud Native Application
Cloud Native Application
Ā 
12 factor app an introduction
12 factor app an introduction12 factor app an introduction
12 factor app an introduction
Ā 
Cloud native principles
Cloud native principlesCloud native principles
Cloud native principles
Ā 
Elastic-Engineering
Elastic-EngineeringElastic-Engineering
Elastic-Engineering
Ā 
Introduction to openshift
Introduction to openshiftIntroduction to openshift
Introduction to openshift
Ā 
Introduction to Docker - 2017
Introduction to Docker - 2017Introduction to Docker - 2017
Introduction to Docker - 2017
Ā 
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SREMicroservices Docker Kubernetes Istio Kanban DevOps SRE
Microservices Docker Kubernetes Istio Kanban DevOps SRE
Ā 
Combining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observabilityCombining logs, metrics, and traces for unified observability
Combining logs, metrics, and traces for unified observability
Ā 
DATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backupDATABASE AUTOMATION with Thousands of database, monitoring and backup
DATABASE AUTOMATION with Thousands of database, monitoring and backup
Ā 
ING microServices
ING   microServicesING   microServices
ING microServices
Ā 
Kubernetes Basics
Kubernetes BasicsKubernetes Basics
Kubernetes Basics
Ā 
Introduction to Red Hat OpenShift 4
Introduction to Red Hat OpenShift 4Introduction to Red Hat OpenShift 4
Introduction to Red Hat OpenShift 4
Ā 
Container Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with TrivyContainer Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with Trivy
Ā 
Microservices Architecture - Cloud Native Apps
Microservices Architecture - Cloud Native AppsMicroservices Architecture - Cloud Native Apps
Microservices Architecture - Cloud Native Apps
Ā 
Transforming Organizations with CI/CD
Transforming Organizations with CI/CDTransforming Organizations with CI/CD
Transforming Organizations with CI/CD
Ā 
Pivotal Cloud Foundry: A Technical Overview
Pivotal Cloud Foundry: A Technical OverviewPivotal Cloud Foundry: A Technical Overview
Pivotal Cloud Foundry: A Technical Overview
Ā 
Kubernetes #2 monitoring
Kubernetes #2   monitoring Kubernetes #2   monitoring
Kubernetes #2 monitoring
Ā 

Similar to ING Container Hosting Platform - 3 years onward_with Kube_for distribution.pdf

Pivotal Container Service Overview
Pivotal Container Service Overview Pivotal Container Service Overview
Pivotal Container Service Overview VMware Tanzu
Ā 
Making Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfMaking Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfRakuten Group, Inc.
Ā 
[API World 2021 ] - Understanding Cloud Native Deployment
[API World 2021 ] - Understanding Cloud Native Deployment[API World 2021 ] - Understanding Cloud Native Deployment
[API World 2021 ] - Understanding Cloud Native DeploymentWSO2
Ā 
ANIRBAN_GHOSH_RESUME
ANIRBAN_GHOSH_RESUMEANIRBAN_GHOSH_RESUME
ANIRBAN_GHOSH_RESUMEAnirban Ghosh
Ā 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kuberneteskloia
Ā 
Building Cloud Native Applications with Oracle Autonomous Database.
Building Cloud Native Applications with Oracle Autonomous Database.Building Cloud Native Applications with Oracle Autonomous Database.
Building Cloud Native Applications with Oracle Autonomous Database.Oracle Developers
Ā 
Infrastructure design for Kubernetes
Infrastructure design for KubernetesInfrastructure design for Kubernetes
Infrastructure design for KubernetesGuillaume Morini
Ā 
Continuous Everything in a Multi-cloud and Multi-platform Environment
Continuous Everything in a Multi-cloud and Multi-platform EnvironmentContinuous Everything in a Multi-cloud and Multi-platform Environment
Continuous Everything in a Multi-cloud and Multi-platform EnvironmentVMware Tanzu
Ā 
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020InfluxData
Ā 
Migrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 months
Migrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 monthsMigrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 months
Migrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 monthsKonveyor Community
Ā 
Transforming mission-critical applications on mainframes for innovation
Transforming mission-critical applications on mainframes for innovationTransforming mission-critical applications on mainframes for innovation
Transforming mission-critical applications on mainframes for innovationEranea
Ā 
Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)
Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)
Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)Samy Fodil
Ā 
GCP Meetup #3 - Approaches to Cloud Native Architectures
GCP Meetup #3 - Approaches to Cloud Native ArchitecturesGCP Meetup #3 - Approaches to Cloud Native Architectures
GCP Meetup #3 - Approaches to Cloud Native Architecturesnine
Ā 
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Elasticsearch
Ā 
Agile integration: Decomposing the monolith
Agile integration: Decomposing the monolithAgile integration: Decomposing the monolith
Agile integration: Decomposing the monolithJudy Breedlove
Ā 
Technology insights: Decision Science Platform
Technology insights: Decision Science PlatformTechnology insights: Decision Science Platform
Technology insights: Decision Science PlatformDecision Science Community
Ā 
OpenStack and Kubernetes - A match made for Telco Heaven
OpenStack and Kubernetes - A match made for Telco HeavenOpenStack and Kubernetes - A match made for Telco Heaven
OpenStack and Kubernetes - A match made for Telco HeavenTrinath Somanchi
Ā 
Preparing for Neo - Singapore OutSystems User Group October 2022 Meetup
Preparing for Neo - Singapore OutSystems User Group October 2022 MeetupPreparing for Neo - Singapore OutSystems User Group October 2022 Meetup
Preparing for Neo - Singapore OutSystems User Group October 2022 MeetupYashrajNayak4
Ā 
Pivotal Container Service (PKS) at SF Cloud Foundry Meetup
Pivotal Container Service (PKS) at SF Cloud Foundry MeetupPivotal Container Service (PKS) at SF Cloud Foundry Meetup
Pivotal Container Service (PKS) at SF Cloud Foundry Meetupcornelia davis
Ā 

Similar to ING Container Hosting Platform - 3 years onward_with Kube_for distribution.pdf (20)

Pivotal Container Service Overview
Pivotal Container Service Overview Pivotal Container Service Overview
Pivotal Container Service Overview
Ā 
Making Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdfMaking Cloud Native CI_CD Services.pdf
Making Cloud Native CI_CD Services.pdf
Ā 
[API World 2021 ] - Understanding Cloud Native Deployment
[API World 2021 ] - Understanding Cloud Native Deployment[API World 2021 ] - Understanding Cloud Native Deployment
[API World 2021 ] - Understanding Cloud Native Deployment
Ā 
ANIRBAN_GHOSH_RESUME
ANIRBAN_GHOSH_RESUMEANIRBAN_GHOSH_RESUME
ANIRBAN_GHOSH_RESUME
Ā 
Ultimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on KubernetesUltimate Guide to Microservice Architecture on Kubernetes
Ultimate Guide to Microservice Architecture on Kubernetes
Ā 
Building Cloud Native Applications with Oracle Autonomous Database.
Building Cloud Native Applications with Oracle Autonomous Database.Building Cloud Native Applications with Oracle Autonomous Database.
Building Cloud Native Applications with Oracle Autonomous Database.
Ā 
Infrastructure design for Kubernetes
Infrastructure design for KubernetesInfrastructure design for Kubernetes
Infrastructure design for Kubernetes
Ā 
Continuous Everything in a Multi-cloud and Multi-platform Environment
Continuous Everything in a Multi-cloud and Multi-platform EnvironmentContinuous Everything in a Multi-cloud and Multi-platform Environment
Continuous Everything in a Multi-cloud and Multi-platform Environment
Ā 
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020
Tim Hall [InfluxData] | InfluxDB Roadmap | InfluxDays Virtual Experience NA 2020
Ā 
Migrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 months
Migrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 monthsMigrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 months
Migrating a Large Fortune 100 Healthcare Company to Kubernetes in 7 months
Ā 
Transforming mission-critical applications on mainframes for innovation
Transforming mission-critical applications on mainframes for innovationTransforming mission-critical applications on mainframes for innovation
Transforming mission-critical applications on mainframes for innovation
Ā 
Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)
Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)
Connectivity is here (5 g, swarm,...). now, let's build interplanetary apps! (1)
Ā 
GCP Meetup #3 - Approaches to Cloud Native Architectures
GCP Meetup #3 - Approaches to Cloud Native ArchitecturesGCP Meetup #3 - Approaches to Cloud Native Architectures
GCP Meetup #3 - Approaches to Cloud Native Architectures
Ā 
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
Ā 
Agile integration: Decomposing the monolith
Agile integration: Decomposing the monolithAgile integration: Decomposing the monolith
Agile integration: Decomposing the monolith
Ā 
Javantura v4 - Self-service app deployment with Kubernetes and OpenShift - Ma...
Javantura v4 - Self-service app deployment with Kubernetes and OpenShift - Ma...Javantura v4 - Self-service app deployment with Kubernetes and OpenShift - Ma...
Javantura v4 - Self-service app deployment with Kubernetes and OpenShift - Ma...
Ā 
Technology insights: Decision Science Platform
Technology insights: Decision Science PlatformTechnology insights: Decision Science Platform
Technology insights: Decision Science Platform
Ā 
OpenStack and Kubernetes - A match made for Telco Heaven
OpenStack and Kubernetes - A match made for Telco HeavenOpenStack and Kubernetes - A match made for Telco Heaven
OpenStack and Kubernetes - A match made for Telco Heaven
Ā 
Preparing for Neo - Singapore OutSystems User Group October 2022 Meetup
Preparing for Neo - Singapore OutSystems User Group October 2022 MeetupPreparing for Neo - Singapore OutSystems User Group October 2022 Meetup
Preparing for Neo - Singapore OutSystems User Group October 2022 Meetup
Ā 
Pivotal Container Service (PKS) at SF Cloud Foundry Meetup
Pivotal Container Service (PKS) at SF Cloud Foundry MeetupPivotal Container Service (PKS) at SF Cloud Foundry Meetup
Pivotal Container Service (PKS) at SF Cloud Foundry Meetup
Ā 

Recently uploaded

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...gurkirankumar98700
Ā 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science
Ā 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
Ā 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel AraĆŗjo
Ā 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
Ā 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays
Ā 
šŸ¬ The future of MySQL is Postgres šŸ˜
šŸ¬  The future of MySQL is Postgres   šŸ˜šŸ¬  The future of MySQL is Postgres   šŸ˜
šŸ¬ The future of MySQL is Postgres šŸ˜RTylerCroy
Ā 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer
Ā 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreternaman860154
Ā 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
Ā 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
Ā 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi
Ā 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
Ā 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer
Ā 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
Ā 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
Ā 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
Ā 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
Ā 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
Ā 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
Ā 

Recently uploaded (20)

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...
Kalyanpur ) Call Girls in Lucknow Finest Escorts Service šŸø 8923113531 šŸŽ° Avail...
Ā 
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Ā 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
Ā 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Ā 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
Ā 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Ā 
šŸ¬ The future of MySQL is Postgres šŸ˜
šŸ¬  The future of MySQL is Postgres   šŸ˜šŸ¬  The future of MySQL is Postgres   šŸ˜
šŸ¬ The future of MySQL is Postgres šŸ˜
Ā 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
Ā 
Presentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreterPresentation on how to chat with PDF using ChatGPT code interpreter
Presentation on how to chat with PDF using ChatGPT code interpreter
Ā 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
Ā 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Ā 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Ā 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Ā 
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024Tata AIG General Insurance Company - Insurer Innovation Award 2024
Tata AIG General Insurance Company - Insurer Innovation Award 2024
Ā 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Ā 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
Ā 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
Ā 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Ā 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
Ā 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
Ā 

ING Container Hosting Platform - 3 years onward_with Kube_for distribution.pdf

  • 1. ING Container Hosting Platform, 3 years onward ING Tech Infra&Engineering Growing and sustaining a Private Cloud Container Hosting Platform October 2022
  • 2. 2 Introductions ING is a global bank with a strong European base. Our more than 57,000 employees serve around 38 million customers, corporate clients and financial institutions in over 40 countries. Our purpose is to empower people to stay a step ahead in life and in business. * ING is undergoing a transition to close our Wholesale Banking offices in Argentina, Brazil and Kazakhstan as announced on 5 November 2020. ING has exited the Austrian and Czech retail banking markets as of the end of 2021. We announced in December 2021 that we will leave the retail banking market in France after a strategic review of our retail banking operations there. We announced June 24th 2022 we will leave the Philippines retail banking market as of then end of 2022. For INGā€™s position regarding Russia and the Ukraine see our Feb 28 2022 statement Thijs Ebbers Architecting Cloud Native @ING since 2016 (employee since 2001) Architecture Lead for the Runtime Domain (ā€œVM & Container Hostingā€), for ING Private & Public Clouds Sandeep Kaul IT Area Lead (ā€œHead of ITā€) for Cloud/PaaS, a.o. responsible for the delivery of ICHP towards internal customers.
  • 3. 3 Short Recap At 2019ā€™s San Diego OpenShift Commons ING presented ā€œRunning Containers in Production, the ING Storyā€ where we introduced INGā€™s Container Hosting Platform (ā€œICHPā€) and explained: ā€¢ the drivers behind our transformation ā€¢ the ā€œCloud Native ecosystem Kubeā€ we use(d) to communicate with our stakeholders ā€¢ ICHPā€™s deployment model & its delivery patterns ā€¢ which topics took the most time during the first 2 years of engineering and operation. So for the remainder of this presentation we assume this content is known ;-) and if not catch up afterwards :
  • 4. 4 So what happened since ? (aka agenda) ā€¢ Covidā€¦ ā€¢ Expansions ā€¢ Divisions ā€¢ Security ā€¢ LCM ā€¢ Exponential growth ā€¢ Bugs ā€¢ Resilience ā€¢ Partnerships ā€¢ (Refined) Strategy
  • 5. 5 ICHP ā€“> ICH* (aka abstracting for multi cloud) ING Container Hosting Platform (ā€œICHPā€) was INGā€™s first standardized Container Hosting environment, but it has evolved into a family we now call ICH which consists of: ā€¢ The private cloud ICHP platform based on OCP ā€¢ The ICHA platform on Azure AKS (still being engineered) ā€¢ The ICHG platform on GCP GKE (still being engineered) All these platforms abide to the same defined interfaces to enable application portability between the ICH family members, for both re-useability as well as regulatory reasons. And making deployments to multiple target clouds with defined interfaces mandatory will empower our engineers with the skills they need. Why not use (managed) OpenShift on Azure & GCP ? ā€¢ ING believes the value of public clouds is in the integrated ecosystem and the cloud vendors native services will be the 1st to benefit from innovations in those ecosystems. This means we should consume SaaS rather then PaaS rather then IaaS to make use of those innovations as soon as possible. ā€¢ Hence we consume Azure and GCP native services wherever possible and we try to avoid IaaS services For the remainder of this presentation weā€™ll focus on ICHP Note : 2 of our ING colleagues are presenting this KubeCon on ā€œHPC Batch computing for Analytics Workloads ā€œ (Wednesday 2:30 pm), attend and learn but be aware this environment is not (yet) hosted on ICH*
  • 6. K8s technology = Amazing ā€¢ Almost infinite # use casesā€¦ But ā€¢ Mixing multiple distinct use cases together in the same clusters = a recipe for (regulatory) disasterā€¦ This table shows the attempt of ING to categorize in order to give each service its appropriate (risk&security) treatment. white = (partially) available as a service grey = currently not available as a service This table is not intended to be staticā€¦ Meaning ING is very opinionated on what workloads should be hosted on our Immutable container hosting serviceā€¦ Donā€™t make any wrong assumptions, we do get a lot of requests for hosting software which in our opinion belongs on services currently unavailable (grey) ā€œNoā€ is a valid answerā€¦, as ING Tech chooses to focus on a limited set (since we have scarce engineering capacity). Spinning up those other services would require additional DevOps squads to engineer and run those services. Requesters can still run that software on VMā€™s... Service Typical use case(s) Immutable Containers Capability to host immutable container workloads like 12 Factor Apps (commonly: ā€œAPIā€™sā€) Functions as a Service Capability to execute backend code directly without having to manage any servers Data Services Capability to provide data persistence as a service e.g. Object Storage, Elastic, Kafka, MS-SQL, Oracle, Cassandra, Redis, etc. HPC Capability to provide High Performance Compute (e.g. GPU, fast local cache, etc.) as a service for machine learning and analytics. Centralized End- User Capability to provide a managed, dedicated set-up for specific user(s) e.g. for data science, educational and training set-ups etc. Infrastructure Management Capability to manage IT assets as Desired State Custom Resources (e.g. Crossplane) Non-12 factor apps (ā€œVMā€s) ā€“ K8S Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images containing monolithic applications, not ready for hands off operations, could be requiring managed local persistency. Hosted on a K8s or equivalent platform. Standalone - Non- 12 factor apps (ā€œVMā€s) Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images containing monolithic applications, not ready for hands off operations, could be requiring managed local persistency. Hosted on Standalone ā€œDockerā€ instances. 6 Divide & Conquer (aka Distinct Services deserve Distinct Treatment)
  • 7. 7 Security ā€¢ ING Container Security Standard in place as of Q2 2019 (ā€œZero Privilegeā€, do not trust ā€œZero Trustā€ā€¦) ā€¢ Comprehensive Vulnerability Scanning in OnePipeline enforced as of Q4 2021 (thank you Log4J for the priorityā€¦) ā€¢ Immutable Workloads, enforced as of Q1 2022 (privileged NPAā€™s removed from production namespaces, Break Glass available, mandatory redeploy afterwards) ā€¢ Immutable Platform (Clusters), as of Q3 2022 (OCP 4.10.30 IPI for Bare Metal dependency) ā€¢ Anomaly Detection & Policy-as-Code capability as of Q3 2022 (OCP 4.10/ K8s kernel version dependency) ā€¢ As of Q3 2022 ICHP has implemented all controls as specified in the Container Security Standard
  • 8. 8 OpenShift LCM Generic learnings: ā€¢ Not all consuming DevOps teams understand the full scope of going Cloud Native and might be unaware of their shortcomings ā€¢ In particular we noticed the value of automated testing wasn't always recognized, which did hamper our Platform LCM ā€¢ Application redeployment should be ā€œthe new normalā€, not ā€œsomething to be scared ofā€ā€¦ Migrating OpenShift 3.11 OKD to OpenShift 3.11 Enterprise (Q1 2020): ā€¢ Clusters were redeployed (1 DC at a time), forcing downtime (in practice a DR-excersize) upon our consumers. This wasnā€™t well received and led us to rethink our migration strategy for the next LCM momentā€¦ Migrating OpenShift 3.11 Enterprise to OpenShift 4.10 Enterprise (ā€œICHP v2ā€) (Q3 2022): ā€¢ Running on Bare Metal nodes doesnā€™t help if new OpenShift versions are first released for VM based clustersā€¦ It took Red Hat until version 4.10.30 to deliver the quality ING needed to do a fully automated IPI based install in our DCā€™s (which was kind of challenging looking at the EOS of OCP 3.11ā€¦) ā€¢ This time we opted to spin up new clusters in parallel to the old environment, and offer our customers a migration window to redeploy their workloads. We opened the migration Sep 15th 2022 for our Non-Prod clusters and our Prod clusters are planned to go live this week More details on how we build our NaaS on top of OCP 4.10 IPI in the other ING presentation today
  • 9. In Q3 2019 ICHP had 18 FTE and hosted : ā€¢ 250 Prod namespaces whose APIā€™s only processed a marginal number of transactions (most still preparing for actual workloads) ā€¢ about 1000 Non-Prod Namespaces In Q3 2022 ICHP hosted: ā€¢ 600 namespaces in Production whose APIā€™s processed 50% of ING retail customer transactions ā€¢ 1690 namespaces in Non-Production And did this with the same 18 FTE, we only added an additional Ops squad to manage the environment (+6 FTE) from a partner (hired as of Q1 2022, for new total of 24 FTE) 9 Growing pains : exponential growth for Immutable App hosting So looking back ICHP coped with a sustained 100% growth each year (measured in installed CPU/memory capacity) (for the last 3 years) & this growth is expected to continue into 2023. As a result the price/unit has now become very competitive and ICHPā€™s Immutable Container hosting in this way supports the broader ING Tech Strategy to enable to grow our business at marginal cost
  • 10. Green line is actual ICHP Prod DC1 usage (Prod DC2 usage is similar) Red line is available ICHP Prod DC1 capacity (Prod DC2 capacity is identical) Drop in capacity end of August ā€™21 due to bug in OpenShift 3.11 icw Atomic requiring ICHP to scale back from 250 pods to 160 pods per node to become stable again (with OpenShift 4/CoreOS its fixed) Customers were impacted (mainly in the non-Prod environment) in the Q3 2021 due to this shortage As a provider we had to scavenge additional nodes everywhere to remain stableā€¦ The drop in Feb 2022 was due to an unintended reboot of multiple nodes by one of our suppliers Growing pains : Bugs (pending pods / pods per node) 10 Pending pods -> Pods/node back to 160 Running out of capacityā€¦ Blue Line measuring ā€œpending podsā€ introduced
  • 11. Q1 2022 Multiple outages impacting ING customers on workloads hosted on ICHP (For the record : ICHP has had zero unplanned platform downtimeā€¦). Analysis concluded most were avoidableā€¦ by properly defining K8s application deployment configs. Lesson learned : Developer Autonomy is good thing, but never forget to validate the resultsā€¦ Q2 2022 ā€¢ ING CTO&CIOā€™s decided on mandatory (not yet enforced) use of : canary releasing resilient pod placement liveness & readiness probes ā€¢ ICHP published ING specific workload resilience documentation Q3 2022 ā€¢ ICHP provides Dashboards giving ING CIOā€™s (& developers) insight in the quality of the deployments on ICHP ā€¢ Additional internal templates & resilience training materials being developed (by partner enabling teams) Future Step : Policy enforcement via Policy-as-Code (K8s deployment config insufficient -> deploy job fails) 11 Growing pains : workload resilience
  • 12. 12 ICHP Partnerships Ingested products (products or data provided by other platforms that are exposed through this platform) supplied-services (provided by other platforms required for platform functions) engineered product builds (developed and maintained by one or multiple squads) CONSUME ENGINEER INGEST Exposed platform services EXPOSE Exposed product builds (semi finished product used as supplied product build by the consuming platform) Complement (add value) ING ā€œBTPā€ (Topics & Service Mesh & Workload Deployment Templates) ING ā€œIPCā€ (DBaaS, Keyspace-aaS, Object Storage, Cloud Automation) ING ā€œSecurity Tribeā€ (IAM & Workload Security Monitoring & Workload Certs) ING ā€œMDPLā€ (Workload Observability) ING ā€œOne Pipelineā€ (Workload CI/CD) ING Banking Technology Platform ING Group Services ING Retail Banks ING Wholesale ING Infra&Engineering ING ā€œDCNā€ (Networking) ING ā€œIPCā€ (Object Storage) ING ā€œMDPLā€ (Platform Observability) ING ā€œOne Pipelineā€ (Platform CI/CD) ING ā€œSecurity Tribeā€ (IAM & Platform Security Monitoring & Platform Certs) ING ā€œIPCā€ (Nodes BM & VM) Red Hat (OpenShift) Portworx (local persistency)
  • 13. 13 Strategy (part 1 : why do we do what we do ?) INGā€™s Tech strategy : ā€œTo build Scaleable Tech which enables Growing our business at marginal costā€ ā€¢ ICHP is a core example of technologies engineered by ING to support this goal, as explained earlier Additional Strategy goals are ā€œSelf-Serviceā€, ā€œRe-useabilityā€ & Automation ā€¢ The immutable container hosting is delivered as a Namespace-aaS from our private cloud portal ā€¢ The ICHP platform hosts applications from almost every ING segment and for most of the countries ā€¢ The ICHP platform is fully automated, both for Namespace & Cluster provisioning (ā€œImmutableā€) ICHPā€™s Unique Selling Point : Compliance-out-of-the-Box ā€¢ Immutability is the key tool in achieving this goal. Hence ICHP focused on Immutable container hosting over the past period and this will stay itā€™s core delivery ING Tech is looking into ā€œTeam Topologiesā€ as a way to optimize the internal organization and as it turns out ICHP ticks all the boxes for a ā€œThinnest Viable Platformā€ (since 2018, the book was published in 2019ā€¦) Our platform is compelling for consumers, with our NaaS approach which reduces cognitive load, it simplifies the risk evidencing burden & we are strongly partnered with key ā€œstream alignedā€, ā€œenablingā€ & ā€œplatformā€ teams in different ING segments to help guide our backlog/roadmap
  • 14. 14 Strategy (part 2 : whatā€™s next?) For the Immutable container hosting service thereā€™s still work to be done: ā€¢ Pets -> Cattle for the clusters, which is aided by ā€¢ Hypervisor based Nodes (currently all production is on Bare Metal) ā€¢ Multi-Cluster Management (ā€œMCMā€) : Pattern(s)/Solution(s) to be investigated ā€¢ Anomaly Detection and Policy-as-Code capabilities have been released with ICHPv2 based on OpenShift 4.10, but scope wise we have lots of opportunities to expand rulesetsā€¦ Workload resilience will be high on that backlogā€¦ But ING does recognize other K8s use cases as explained before: ā€¢ ICHP is hosting Elastic & Prometheus environments on dedicated OpenShift clusters (with local persistency) ā€¢ ICHP is working on a Cluster-aaS delivery for Platform Teams ā€¢ And ICHP is tracking the maturity of the market: ā€¢ To determine when best to start investing our scarce engineering resources (Scarcity does create focus...). ā€¢ The Multi-Cluster Services (ā€œMCSā€) developments have our special attention (perhaps Skupper + KCPā€¦?) In the mean time non-Immutable workloads can still be hosted on our Private Cloud VM offeringā€¦ (no exceptions/no specials in our multi-tenant Immutable Container hosting, or else we risk losing our USPā€¦) Last but not least we strive to be a more engaged participant in the community, more on that topic in the other ING presentation todayā€¦
  • 15. Of course weā€™re hiring ;-) https://www.ing.jobs/tech 15 Thank you / Questions Slides will be made available at https://www.slideshare.net/ThijsEbbers/
  • 17. Whilst we treaded the eye of the needle in our Production clusters when we hit the ā€œPending Pods bugā€, we werenā€™t so ā€œluckyā€ with our Non-Production clusters. On several occasions consumer (DTA) workloads did encounter negative impact, and since hardware replenishment was difficult we moved D/T workloads to newly added VM based nodes (which explains the decline in the green line) in order to free up Bare Metal nodes to be used to stabilize the Production clusters as well as to keep the Acceptance workloads as stable as possible on the remaining Bare Metal nodes Meanwhile, in Non-Prodā€¦ 17 Pods/node back to 160 Pending pods Introduction of additional (VM based) nodes Non-Prod Environment stabilized
  • 18. ā€¢ From INGā€™s Daily Banking Tribe talk at GOTO 2022 conference (ā€œSpotify Model : On Radical Improvements From The Trenches Of Changeā€): ā€¢ Deployment time : 62% build & deploy speed improvement ā€¢ Development capacity : 27% more Backlog items done (measured per year in 2020/21) ā€¢ More Coding : 16% more coding time per team ā€¢ No more VMā€™s : Less infra costs, no patching, more secure, less issues with auditing the stack (More cool ING Tech talks on our YouTube Channel) ā€¢ Building a Global Investments Platform in 12 Months (2020-21) ā€¢ Building a Digital Platform for Insurance Products (ā€œING-AXA partnershipā€) in 9 Months (2019) ā€¢ Building a (Mobile Only Digital) Bank in 9 Months (ā€œThe Manila Miracleā€) (2018) ā€¢ And more to comeā€¦ ICHP(ā€˜s-ecosystem) enabled our Internal ING Customers to achieve : 18
  • 19. Platform (ā€œICHPā€) Adds e.g. : ā€¢ ING Compliance ā€¢ ING Integrations (SEM, Monitoring, AZDO, ā€¦) ā€¢ ING Hardening Distribution (e.g. ā€œOpenShiftā€, ā€œRKEā€, ā€œGKEā€, ā€œAKSā€) Adds e.g. : RBAC, SDN Container Management Framework (ā€œKubernetesā€) Adds e.g. : Scaling, resilience Application Containers to Container Hosting Platforms 19 Containers
  • 20. ā€¢ Important aspect to be taken into consideration: The ING Container Hosting Platform Immutable Container hosting is NOT aiming to rehost VMā€™s to containersā€¦ Purpose is to have the best possible hosting environment for immutable workloads! ā€¢ If a DevOps teams manages to properly refactor their VM into a set of container images they are welcome ā€¢ Hands-off approach in production (which implies a mature team in all aspects, e.g. CI/CD !) If teams feel not comfortable with this they should stay on VMā€™s! ICHP Immutable App hosting Intended Use 20
  • 21. ING aims to make our DevOps teams autonomous (As explained in INGā€™s Way-of-Working in 2019) . We also want to enable those DevOps teams to deliver maximum value to the business, by not bothering them with IT-Infrastructure concerns, nor bothering them with having to deliver compliancy evidence for the hosting platform. Hence offering a Namespace-as-a-Service is for us the sweet spot: ā€¢ Clear demarcation between (Infra)Provider and Consumer ā€¢ Enabling hand-over of compliance evidence ā€¢ Enabling multi-tenancy (hence fast time-to-market/self-service consumption, and the potential for efficient utilization of resources) ā€¢ The DevOps team can assume responsibility for (almost) the full stack (only the kernel stays shared). They have liberty/responsibility to choose/maintain their (versions of) base image, runtime engine, libraries, etc. (within the boundaries set by the ING corporate risk&compliancy rules !) ING will offer a K8S Cluster-as-a-Service in the future, however this will be a limited offering only available to selected platform teams. Why ICHP only offers a ā€œNamespace-as-a-Serviceā€ 21
  • 22. Key Objective for ING is to remain capable to exit a Public Cloud provider. Determining the right abstractions is key to meeting this objective for a container hosting platform, hence: ā€¢ We choose an open/industry standard (in this case: the Kubernetes ecosystem) ā€¢ We choose an intended use (in this case: Immutable Workloads) ā€¢ We clearly define the allowed interfaces (in the ICH ā€“ Interfaces standard) ā€¢ We automate our deployments by the mandatory use of One Pipeline Note that these abstractions are design time and not runtime (as we still use the AKS service native to Azure or the GKE service native to GCP) However there will always be debate on the details as we try to balance innovation, time-to-market, risk/security and/or cost-income ambitions. Therefore actually running production workloads on at least one Kubernetes stack outside of the primary (Public) Cloud provider is the best recipe to ā€œkeep everyone honestā€, as this will prevent Cloud Provider specific ā€œoptimizationsā€ from slipping into ING code. Meaning there should always be at least 2 ICH standard solutions in use. Cloud Abstractions 22
  • 23. We do know how to build those services in greyā€¦ Our engineers are more then capable to build and run those. But not all of them together at the same timeā€¦ Hence for each year ING determines where to allocate that engineering capacity: ā€¢ Improving existing services ā€¢ e.g. Pets -> Cattle ā€¢ Supporting additional use cases on existing services ā€¢ e.g. Data Services we today only support ELK & Prometheus, if we start to support Data Persistence Services more critical to the bank (e.g. message bus, (non-)relational databases) weā€™d actually like the availability of those Services to be independent from K8s cluster uptime (hence our interest in CNCF MCS developments) ā€¢ Introducing New Services ā€¢ We do have active instantiations of Centralized End-User & HPC use cases on K8s in ING (one is even presenting @KubeCon Detroit). But those are currently not hosted as a centralized/standardized Service. So we could also choose to include those in the ICH catalogue ā€¢ The one defined service weā€™re not likely to invest time in any time soon is FaaS, that has everything to do with the controls currently enforced from a regulatory perspective and little with the actual technologyā€¦ Dive and Conquer additional information 23
  • 24. IPC/ICHP hosting services (1/2) (Grey is currently not available) 24 Service Typical use case(s) Immutable Containers Capability to host immutable container workloads like 12 Factor Apps (commonly: ā€œAPIā€™sā€) Functions as a Service Capability to execute backend code directly without having to manage any servers Data Services Capability to provide data persistence as a service e.g. Object Storage, Elastic, Kafka, MS-SQL, Oracle, Cassandra, Redis, etc. HPC Capability to provide High Performance Compute (e.g. GPU, fast local cache, etc.) as a service for machine learning and analytics. Centralized End-User Capability to provide a managed, dedicated set-up for specific user(s) e.g. for data science, educational and training set-ups etc. Infrastructure Management Capability to manage IT assets as Desired State Custom Resources (e.g. Crossplane) Non-12 factor apps (ā€œVMā€s) ā€“ K8S Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images containing monolithic applications, not ready for hands off operations, could be requiring managed local persistency. Hosted on a K8s or equivalent platform. Standalone - Non-12 factor apps (ā€œVMā€s) Hosting of non-12 factor apps (ā€œVMā€™s in a container packageā€), typically large images containing monolithic applications, not ready for hands off operations, could be requiring managed local persistency. Hosted on Standalone ā€œDockerā€ instances.
  • 25. IPC/ICHP hosting services (2/2) (Grey is currently not available) 25 Service Unit of Delivery (/Security) Local Persistency GPU Ingress Exposed interface(s) End-user access to SSH/Terminal Interfaces Immutable Containers Namespace Not Available Not Available Shared per Traffic type (Internal/DMZ/ā€¦ ) API (HTTPS ingress), Eventing (ā€œKafkaā€), File (/Object) (ā€œS3ā€) Not Allowed (in Production) (currently still tolerated in Non- Production clusters) Functions as a Service Function Not Available TBD Shared See ā€œImmutable Containersā€ Not Possible Data Services (set of) Namespace(s) Available Not Available Dedicated per workload TBD Not Allowed HPC Namespace Optional Optional TBD See ā€œImmutable Containersā€ Not Allowed Centralized End-User Namespace TBD TBD TBD SSH/https/ā€¦ Allowed Infrastructure Management Control plane Not Available Not Available Not Applicable K8s API Not Allowed Non-12 factor apps (ā€œVMā€s) ā€“ K8S Namespace, VM ? TBD Not Available TBD TBD TBD Standalone - Non-12 factor apps (ā€œVMā€s) VM Available Not Available Dedicated No restrictions Available
  • 26. ā€œImmutableā€ = ā€¢ Any change can only be executed via an ING pipeline which has all required checks & balances in place (and which facilitates automated evidencing for regulatory purposes) ā€¢ In short : no direct access for natural persons, no change privileges for other systems And yes, INGā€™s Container Security Standard was in place before ā€œSolarwindsā€ reached the headlines in 2020ā€¦ And although most people in the IT(-Security) industry are now aware of the software lineage issue exposed by it (that doesnā€™t mean its anywhere near solvedā€¦), far too few understand itā€™s 2nd lesson about the dangers of overprivileged softwareā€¦ (what enabled those attackers to move laterally so easilyā€¦?) For these 2 reason ING does not trust ā€œZero Trustā€ labelled implementations as very few of those have actually covered these concerns properly... Security additional explanation 26
  • 27. Range Of Container Technology Usage Models 27
  • 28. 28 Container Technology ING Security Reference Architecture Image Storage and Retrieval Container Deployment, Management, and Operations. Image Creation, Testing, and Accreditation DevOps DevOps DevOps Host with Containers Host with Containers Host with Containers Host with Containers Build, Deploy, Test, Release Development Registry Any Other Registry Authorized Registry Deployment Registry Orchestrator Container Management Framework No Connection No Connection W.01 W.02 W.03 W.04 W.05 I.06 I.07 I.08 R.09 R.10 O.11 P.14 P.17 P.20 H.21 P.15 C.12 P.18 H.22 P.16 C.13 P.19 H.23
  • 29. 29 Container Technology Security Measures Overview Design & Code (digitize) Build & Test (create) Operate & Maintain (detect & correct) Deploy & Test (assembly) Release (orchestrate) Onboard and Plan (define) Engineering Journey R.09 R.10 Registry Measures O.11 Orchestrator Measures I.06 I.07 I.08 Image Measures Measures to be implemented by providers of CI/CD Pipeline for container technology e.g. GEP P.14 P.15 P.16 P.17 P.18 P.19 P.20 Platform Measures H.21 H.22 H.23 Host Measures C.12 C.13 Container Measures Measures to be implemented by providers of container hosting platform (except H.23) e.g. ICHP W.01 W.02 W.03 W.04 W.05 Workload Measures Measures to be implemented by providers of container workloads e.g. DevOps
  • 30. ā€¢ Want to understand how to : ā€¢ Use Canary (Blue/Green) releasing using K8s cluster external loadbalancing services ā€¢ Use Canary (Blue/Green) releasing using K8s cluster internal capabilities ā€¢ Revert my traffic 100% to my stable version in case of unexpected issues ā€¢ Rollback my release to the N-1 version in case of unexpected issues with version N ā€¢ Perform Resilient Placement so a single Red or Blue outage in the underlying IaaS layers will not affect my application availability ā€¢ Implement Liveness Probes so my pods will restart automatically in case of applicative issues ā€¢ Implement Readiness Probes so I have the input to automatically direct my traffic in case of applicative issues ā€¢ And would like to see examples & templates how to use above features in INGā€™s One Pipeline Workload Resilience : As an ICHP Consuming DevOps Team, I 30
  • 31.
  • 32.
  • 33. The following slides come from ING Investor Update June 13th 2022 Tech& Operations 33 Strategy additional explanation
  • 34. 34
  • 35. 35
  • 37. Follow us SlideShare.net/ING @ING_News LinkedIn.com/company/ING YouTube.com/ING Flickr.com/INGGroup Facebook.com/ING ing.com ingwb.com Medium.com/ing-blog