Elastic-Engineering

@arafkarsh arafkarsh
a tech presentorial
Combination of
presentation &
tutorial
ARAF KARSH HAMID
Co-Founder / CTO
MetaMagic Global Inc., NJ, USA
@arafkarsh
arafkarsh
1
Create
Elastic
Engineering
(Pods) Teams
To Build Cloud Native Apps
Using Composable Enterprise Architecture
To Manage
SpecOps : From Specs to Ops

Objectives / Goals of Engineering Pods
2
• Train and Build Cloud Native App Engineering Team
• Team Structure Follows Capability Centric Model and
Microservices based Architecture
• Optimum Team Size for Engineering Pod
• Identify Key Roles within Engineering Pod
• Define Roles and Responsibilities
• Faster GoTo Market Capabilities (4-6 Weeks to 1-Day to Hours)
• Fully Automated DevOps Culture

@arafkarsh arafkarsh 3
Design Thinking
Lean Startup / Agile
User Stories / MVP
1
Architecture
Kafka, Containers,
Kubernetes, Istio
2
Testing Strategies,
BDD & Automation
Network / Security
3
CI/CD
Observability
DevSecOps
4
Setting up the
Context
For Elastic Engineering
A
Define
Elastic
Engineering
Pods
B
How

Setting up the Context
Why do we need Elastic Engineering?
4
A

Pioneers in Cloud Native App Implementation
5
New Entrants

Cloud Native Apps Technology Landscape
1999
Commercial
Virtual Machine
2003
VM Monitor
Hypervisor
2004
Architecture Pattern
Domain
Driven Design
2006
Cloud Services
Amazon AWS
2013
Containers
Docker
2014
Container
Orchestrator
Kubernetes
2005
Architecture Pattern
Event Sourcing
& CQRS
1995s 2020s
2000s
Cloud Native Apps
Infrastructure Evolution
1. Virtual Machines
2. Containers
3. Kubernetes (Orchestrator)
4. Istio (Service Mesh)
5. Kafka (Messaging)
Architecture Patterns
1. API Gateways / LB
2. Service Discovery
3. Event Driven
4. Service Mesh
5. Domain Driven Design
6. Event Sourcing & CQRS
7. Reactive Programming
8. Distributed Tx
2015
Service Mesh
Istio
2011
Messaging
Kafka
1998
Architecture Style
3 Tier Architecture
2003
Architecture Style
SOA
2020
Service Mesh
Open Service Mesh
2007
Linux Kernel
cgroups
2008
Cloud Services
Google Cloud
2010s
2010
Cloud Services
Microsoft Azure
2011
Hybrid Cloud Services
RedHat OpenShift
1999
Software Process
XP (Agile)
1987
Design Pattern
Saga Pattern
2005s 2015s
2004
Software Process
Kanban
1985s
2010
Cloud Services
OpenStack
2009
PaaS Services
Cloud Foundry

Agile
Scrum (4-6 Weeks)
Developer (Migration – Brown Field or Green Field) Journey
Monolithic
Domain Driven Design
Event Sourcing and CQRS
Waterfall
Optional
Design
Patterns
Continuous Integration (CI)
6/12+ Months
Enterprise Service Bus
Relational Database [SQL] / NoSQL
Development QA / QC Ops
7
Microservices
Scrum / Kanban (1-5 Days)
Mandatory
Design
Patterns
Infrastructure Design Patterns
Continuous Integration - CI
DevOps
Event Streaming / Replicated Logs
SQL NoSQL
Continuous Delivery - CD
Container Orchestrator Service Mesh

Application Modernization – 3 Transformations
8
Monolithic SOA Microservice
Physical
Server
Virtual
Machine
Cloud
Waterfall Agile DevOps
Source: IBM: Application Modernization > https://www.youtube.com/watch?v=RJ3UQSxwGFY
Architecture
Infrastructure
Delivery

Application Modernization – 3 Transformations
9
Monolithic SOA Microservice
Physical
Server
Virtual
Machine
Cloud
Waterfall Agile DevOps
Architecture
Infrastructure
Delivery
Modernization
1
2
3
Source: IBM: Application Modernization > https://www.youtube.com/watch?v=RJ3UQSxwGFY

Maturation of SDLC Best Practices
10
Source:
Page 16
US DoD Enterprise
DevSecOps 2.0
Fundamentals

DevSecOps
11
DevSecOps is a culture and philosophy that
must be practiced across the organization,
realized through the unification of a set of
Software development (Dev), Security (Sec)
and Operations (Ops) personnel into a
singular team.
Source: Page 17. US DoD Enterprise DevSecOps Fundamentals

Summary: Why do we need Elastic Engineering?
12
1. Scalability: Example Walmart Black Friday Sales,
Uber, NetFlix etc..
2. Application Modernization Benefits / Composable
3. Diverse Technology Stack
4. Near Realtime Data Streaming / Analytics
5. Continuous Delivery Pipelines
6. Faster Go To Market – Adaptive

Elastic Engineering
• Business Capability Centric Design
• Shift Left Operations
• SpecOps Workflow for Green Field / Brown Field Apps
• Engineering Pods
• Roles and Responsibilities
13
B

Composable Enterprise Architecture
14
A composable enterprise is an
organization that delivers
business outcomes and adapts
to the pace of business
change.
It does this through the
assembly and combination of
Packaged Business Capabilities
(PBCs).
PBCs are application building
blocks that have been
purchased or developed.
Source: Gartner: Future of Applications: Delivering the Composable Enterprise | Why does it matter?

Business Capability Centric Design
Business Centric Development
• Focus on Business Capabilities
• Entire team is aligned towards
Business Capability.
• From Specs to Operations – The
team handles the entire spectrum
of Software development.
• Every vertical will have its own
Code Pipeline, Build Pipeline
Front-End-Team Back-End-Team Database-Team
In a typical Monolithic way, the team is
divided based on technology / skill set
rather than business functions. This leads
to not only bottlenecks but also lack of
understanding of the Business Domain.
QA Team
QA = Quality Assurance
PO = Product Owner
Vertically sliced Product Team
15
Front-End
Back-End
Database
Business
Capability 1
QA
PO
Ops
Front-End
Back-End
Database
Business
Capability 2
QA
PO
Ops
Front-End
Back-End
Database
Business
Capability - n
QA
PO
Ops

Production Environment
Continuous Monitoring
Fully
Automated
Continuous Deployment
Shift Left – Operational Concerns
16
• Operations Concerns move earlier in software delivery life cycle, towards development.
• The Goal is to enable Developers and QC Team to Develop and Test the software that
behave like Production System in fully automated way.
Development Environment
Build
Build
Build
Test Environment
Continuous Integration
Unit
Testing
Component
Testing
Contract
Testing
Integration
Testing
Continuous Testing
Shift Left moves operations earlier in development cycle.
Stage Environment
Acceptance Testing
Pull Request / Merge
Continuous Delivery
GitOps – CD/CD

Management
Pipeline Automation
Architecture
SpecOps Workflow - SDLC
17
Green
Field
Brown
Field
Event Sourcing / CQRS
Migration Patterns
Strangler Fig, CDC…
Build
Design Develop Test Stage Ops
Cloud
• Fault Tolerance
• Reliability
• Scalability
• Traffic Routing
• Security
• Policies
• Observability
• Unit Testing
• Component
• Integration
• Contract
• Package
Repositories
• Mvn, npm,
docker hub
• Containers
• Orchestration
• Serverless
• Traffic Routing
• Security (mTLS, JWT)
• Policies (Network / Security
• Observability
Infra Code
• Feature
Code
• Configs
Source Code
Specs

Management
Pipeline Automation
Architecture
Engineering Pods
18
Build
Design Develop Test Stage Production
Specs
Cloud
Manager
Cloud
Architect
Cloud
Sr. Engineer
Cloud
Engineers
Product
Owner
Elastic Engineering Pod
2-6 Engineers
Customer 1-3 Engineers

Engineering Pod (Team) Size
19
5 7
Manager
Architect Sr. Engineer
Manager
Engineers
Engineers
Engineering Pods
are created based
on Various Team
Size Configurations
As per Business
Capability
Requirement
• 5 Members
• 7 Members
• 9 Members

Engineering Pod (Team) Size
20
9 9
Manager
Manager
Engineers Engineers
9
Manager
Engineers

Roles & Responsibilities
21
Category Role Type Responsibilities
1 Customer Product Owner External
• Complete Product Vision
• Specifications
2 Engineering Pod
Cloud Manager /
Analyst
Internal
• Analyst & Specifications (User Stories, Acceptance Criteria)
• Feature Management in Production & Focused on Outcome
• Scrum Master & Team Management
3 Engineering Pod Cloud Architect Internal
• Technology Stack, Mentor / Guiding Team on Technology
• Data Models and Interface Definitions (Hexagonal Architecture)
• Research & Coding (Feature and Infra Code)
4 Engineering Pod Cloud Sr. Engineer Internal
• Mentor / Guiding Team on Technology
• All the Responsibilities of Cloud Engineer also
5 Engineering Pod
Cloud Engineer /
SRE
Internal
• Feature Coding (Open API Specs)
• Feature Test Cases (Unit, Component, Contract, Integration)
• Infra Coding (Infra, Security, Network)
• Build Pipeline Automation
6 Engineering Pod
Cloud Network /
Security Lead
External
• Define Network Policies
• Define Security Policies
• Review Infra Code for Network & Security Policies
7 Engineering Pod
Cloud User
Experience Lead
External
• Usability Guidelines
• Data Flow Guidelines

Summary Elastic Engineering
22
1. Team Built around Business Capabilities
2. Smaller Team with Full Stack Developers
3. Fully Automated SDLC Pipeline
4. Shift Left: All Coding done at Development time
5. Easily Add More Elastic Engineering Pods to Scale
Up the team size.

Design Thinking
Lean Startup / Agile
User Stories / MVP
1
Architecture
Kafka, Containers,
Kubernetes, Istio
2
Testing Strategies,
BDD & Automation
Networking / Security
3
DevSecOps & SRE
Observability
Case Studies
4
How
Cloud Manager / Analyst
Cloud Architect / Engineer
Cloud Architect / Cloud Engineer
Cloud Architect / Cloud Engineer

Workshop Plan
24
Day Topic
Day 1
Design Thinking, Lean Startup & Agile
Epics, User Stories, MVP, Release Plans
Day 2 Domain Driven Design
Day 3
Kafka – Pub / Sub and Streaming
Day 4
Big Data and Design Patterns
Replication and Sharding
Day 5
Microservices Architecture
Migration Design Patterns
Day Topic
Day 6
Microservices Testing Strategies
Unit, Component, Contract, Integration
Day 7
Docker and Containers
Kubernetes
Day 8
Advanced Kubernetes,
Service Mesh (Istio), Security
Day 9 Cloud Architecture / Terraform
Day 10 CI/CD Pipeline, Observability, DevOps
Day 11 Zero Trust, SASE, DevSecOps
All the section contains real-world examples with more than 10 GitHub Repositories
as Reference Implementation for all the theoretical concepts.
Fully Functional e-Commerce Application available Reference Implementation
based on Microservices architecture, Kafka and Big Data (MongoDB) Deployed
using Containers, Kubernetes and Istio.

11
Days
Workshop
25

1
Outcome oriented
o Design Thinking / Lean Startup / Agile
o Measurable Outcomes
o Specs: Themes, Epics and User Stories
o Story Map, Minimum Viable Product & Agile Scrum / Kanban
26

Three Mindsets of Product Development
27
Design
Thinking
Lean
Startup Agile
Source: Jonny Schneider, Thought Works
Explore the
Problem
Build the
right things
Build the
things right
Hypothesis
Validation
New Business Requirements
Product Evolutions
Agile
MVP

From
Project Based
Activity
Oriented
To
Product Based
Outcome
Oriented
Source: Sriram Narayan– https://martinfowler.com/bliki/BusinessCapabilityCentric.html
28

ShopEasy – eCommerce Portal
29
Theme Epic User Story Sprint
ShopEasy – eCommerce Application
1. Customer Management
2. Search Product
3. Catalogue
4. Shopping Cart
5. Order Processing
6. Payments
2. Search Product
Release 1
1. Global Search
Release 2
1. Search by Brand
2. Search by Price
Range
Release 3
1. Search by Model
2. Search by Rating
Stories
1. Global Search
2. Search by Brand
3. Search by Price
Range
4. Search by Model
5. Search by Rating

Epic – Customer
30
As a Consumer
I want to register eCommerce Portal
So that I can buy products
Role-Feature-Reason Matrix
User Story – 1 : Registration
BDD Acceptance Criteria – 1: Save User
Given The fields First Name, Last Name, DOB
Address, Email Address, Phone No.
When User enters values in the fields First
Name, Last Name, DOB Address, Email
Address, Phone No.
Then If the following fields contains values
First Name, Last Name, Address, Email
Address and Phone No.
AND Age is greater than 18
Save the Data.
BDD Acceptance Criteria – 2 : Generate
Password
Given User Info Available
When Email Address is a valid email
Then Generate the password
AND Send mail with user email address as
login id the URL of the portal
AND Send Password in a separate email
address.
AND Store data on mail status as mail send
or failed.
BDD Acceptance Criteria – 3 : Resend Mail
Given User Registration mail status is available
When The Mail status is failed.
Then Send the mail again
AND stored the attempt number.

Microservices / User Journey / Story Map & Release Cycles
31
Browse Products Add to Shopping Cart Select Shipping Address Confirm Order Make Payment
Catalogue Shopping Cart Order Payment
Customer
View Product
Search
User Journey
Search by Price Image Gallery Update Qty Use PayPal
R2
Global Search Product Details Add to Cart
Delete Item
Select Address Confirm Order
Pay Credit Card
Make
Payment
R1
Registration
Minimum Viable Product
Scrum Sprint Cycle
Search by Price Image Gallery Update Qty Use PayPal
Kanban Cycle: Each of the Story can be released without waiting for other stories to be completed resulting
in Shorter Releases as all the stories are independent!

Summary: Outcome Oriented
32
1. Iterative model based on Design Thinking
2. Specs as User Stories with Acceptance
Criteria
3. Create Minimum Viable Product with User
Journey
4. Agile with Sprint and Kanban for Faster
Releases

2
Architecture & Design
o Composable Enterprise Architecture
o Domain Driven Design
o Event Sourcing and CQRS
o Monolithic to Microservices
o Event Streaming & Pub / Sub – Apache Kafka
o Event Streaming & Analytics – Apache Flink
o Container Orchestration
o Hybrid with Edge and Multi Cloud Auto Scaling
33

Composable Enterprise Architecture
34
Source: Gartner: Future of Applications: Delivering the Composable Enterprise | Why does it matter?
On Demand
Scalability
Service & Apps
Integrated
with Clients &
Devices
Automated
On Demand
Services
Self Service
Options for
App & Data
MASA
Mesh App & Service
Architecture
Enterprise Data
Available, Accessible, &
Integrated into Data Flow
Delivers > Packaged Business Capabilities

Domain Driven Design – Tactical Design
35
Source: Domain-Driven Design Reference by Eric Evans

Process
• Define your Business Processes. Eg. Various aspects of Order
Processing in an E-Commerce Site, Movie Ticket Booking,
Patient visit in Hospital.
1
Commands • Define the Commands (End-User interaction with your App) to
execute the Process. Eg. Add Item to Cart is a Command.
2
Event Sourced
Aggregate
• Current state of the Aggregate is always derived from the Event
Store. Eg. Shopping Cart, Order etc. This will be part of the Rich
Domain Model (Bounded Context) of the Micro Service.
4
Projections
• Projections focuses on the View perspective of the Application.
As the Read & Write are different Models, you can have
different Projections based on your View perspective.
5
Write
Data
Read
Data
Events • Commands generates the Events to be stored in Event Store.
Eg. Item Added Event (in the Shopping Cart).
3
Event Storming – Concept

Event Sourcing Intro
37
Standard CRUD Operations – Customer Profile – Aggregate Root
Profile
Address
Title
Profile Created
Profile
Address
New Title
Title Updated
Profile
New
Address
New Title
New Address added
Derived
Profile
Address
Notes
Notes Removed
Time T1 T2 T4
T3
Event Sourcing and Derived Aggregate Root
Commands
1. Create Profile
2. Update Title
3. Add Address
4. Delete Notes
2
Events
1. Profile Created Event
2. Title Updated Event
3. Address Added Event
4. Notes Deleted Event
3
Profile
Address
New Title
Current State of the
Customer Profile
4
Event store
Single Source of Truth
Greg
Young

Summary User Journey: CCD / DDD / Event Sourcing & CQRS
38
User Journey
Bounded
Context
1
Bounded
Context
2
Bounded
Context
3
1. Bounded Contexts
2. Entity
3. Value Objects
4. Aggregate Roots
5. Domain Events
6. Repository
7. Service
8. Factory
Process
1
Commands
2
Projections
5
ES Aggregate
4
Events
3
Event Sourcing & CQRS
Product
Owner
Cloud
Manager
Cloud
Architect
Cloud
Engineer
Epics / User Stories
Acceptance Criteria
Automated
Test Cases
Source / Infra
Code
Cloud
Sr. Engineer
DDD – Domain Driven Design
Ubiquitous Language
Core
Domain
Sub
Domain
Generic
Domain
Vertically sliced Product Team
FE
BE
DB
Business
Capability 1
QA
Team
PO
FE
BE
DB
Business
Capability 2
QA
Team
PO
FE
BE
DB
Business
Capability n
QA
Team
PO
CCD – Capability Centric Design

Design Pattern
1 Decomposition Identify the Modules and Boundaries
2 Strangler Fig Product Service (Query Only with Apache Solr)
3 Branch By Abstraction Shopping Cart as Service using Monolithic DB
4 Decorating Collaborator New Shipping Service with Database
5 Change Data Capture Enhance Loyalty Module as a new Service
6 Aggregate API Expose Customer API on Monolith
7 Split Table
Product Split into 2 Services as Product Management
and Warehouse
8 Change Data Ownership Order Service using Monolithic DB
9 Move Foreign Key to Code
Foreign Key in Order (item) with Product Table moved to
Order Service Code
10 Strangler Fig Move out Customer Module and decommission the Monolith
Legacy Migration - Design Patterns
39
• Prioritize the Modules / Services For Implementation & Deployment
Setup
Transition Plan
UI Layer
WS
BL
DL
Database
Cart
Order
Customer
Product
Monolith

Strangler Fig Pattern
40
API Gateway / Proxy
Web Services
Business Logic
Database Layer
Product
Micro
Service
Product
SE 8
UI Layer
WS
BL
DL
Database
Cart
Order
Customer
Product UI Layer
WS
BL
DL
Database
Cart
Order
Customer
Product
UI Layer
WS
BL
DL
Database
Cart
Order
Customer
Product
Web Services
Business Logic
Database Layer
Product
Micro
Service
Product
SE 8
API Gateway / Proxy
Step 1
Identity the Module
Step 2
Move the Service
Step 3
Redirect the Calls to New Service
Source: Monolith to Microservices By Sam Newman
1. New Service
(Code)
2. Better Search
Capabilities
Product DB will be used
by other modules in
Monolithic App
Monolith Monolith Monolith
Only used for Search
Capabilities and
Scalability.

Async Calls : Kafka – Scalable Multi Subscribers
41
Check Out
Cart
4. Data Durability (Replication)
5. Ordering Guarantee (Per
Partition)
Use Partition Key
Kafka
Producer API
Kafka
Consumer API
1 2 3 4
1 2 3 4
3 4
Service
Instances
Order Topic (Total Partitions 6)
Kafka Storage
Replicated Logs
Kafka Cluster
5 6 7 8
7 8
What will happen to
Inventory Instance 7 and 8?
Order Consumer Group Inv Consumer Group Multiple Subscriber
As there are only 6 Partitions
Kafka can serve ONLY 6
consumers within a partition
2 5
1
Broadcast Orders to following
Consumers
All the above Consumers will get same
orders available in the Order Topic
1. Highly Scalable
2. Multi Subscriber
3. Loosely Coupled
Systems

Apache Flink Architecture for Stream Analytics
42

Servers / Virtual Machines / Containers
43
Hardware
Host OS
HYPERVISOR
App 1 App 1 App 1
Guest
OS
BINS
/ LIB
Guest
OS
BINS
/ LIB
Guest
OS
BINS
/ LIB
Type 2 Hypervisor
App 2
App 3
App 2
OS
Hardware
Desktop / Laptop
BINS
/ LIB
App
BINS
/ LIB
App
Container 1 Container 2
Type 1 Hypervisor
Hardware
HYPERVISOR
App 1 App 1 App 1
Guest
OS
BINS
/ LIB
Guest
OS
BINS
/ LIB
Guest
OS
BINS
/ LIB
App 2
App 3
App 2
Guest OS
Hardware
Type 1 Hypervisor
BINS
/ LIB
App
BINS
/ LIB
App
BINS
/ LIB
App
Container 1 Container 2 Container 3
HYPERVISOR
Virtualizes the OS
Create Secure Sandboxes in OS
Virtualizes the Hardware
Creates Virtual Machines
Hardware
OS
BINS / LIB
App
1
App
2
App
3
Server
Data Center
No Virtualization
Cloud Elastic Computing

Deployment – Updates and rollbacks, Canary Release
D
ReplicaSet – Self Healing, Scalability, Desired State
R
Worker Node 1
Master Node (Control Plane)
Kubernetes
Architecture
44
POD
POD itself is a Linux
Container, Docker
container will run inside
the POD. PODs with single
or multiple containers
(Sidecar Pattern) will share
Cgroup, Volumes,
Namespaces of the POD.
(Cgroup / Namespaces)
Scheduler
Controller
Manager
Using yaml or json
declare the desired
state of the app.
State is stored in
the Cluster store.
Self healing is done by Kubernetes using watch loops if the desired state is changed.
POD POD POD
BE
1.2
10.1.2.34
BE
1.2
10.1.2.35
BE
1.2
10.1.2.36
BE
15.1.2.100
DNS: a.b.com 1.2
Service Pod IP Address is dynamic, communication should
be based on Service which will have routable IP
and DNS Name. Labels (BE, 1.2) play a critical role
in ReplicaSet, Deployment, & Services etc.
Cluster
Store
etcd
Key Value
Store
Pod Pod Pod
Label Selector selects pods based on the Labels.
Label
Selector
Label Selector
Label Selector
Node
Controller
End Point
Controller
Deployment
Controller
Pod
Controller
….
Labels
Internet
Firewall K8s Virtual
Cluster
Cloud Controller
For the cloud providers to manage
nodes, services, routes, volumes etc.
Kubelet
Node
Manager
Container
Runtime
Interface
Port 10255
gRPC
ProtoBuf
Kube-Proxy
Network Proxy
TCP / UDP Forwarding
IPTABLES / IPVS
Allows multiple
implementation of
containers from v1.7
RESTful yaml / json
$ kubectl ….
Port 443
API Server
Pod IP ...34 ...35 ...36
EP
• Declarative Model
• Desired State
Key Aspects
N1
N2
N3
Namespace
1
N1
N2
N3
Namespace
2
• Pods
• ReplicaSet
• Deployment
• Service
• Endpoints
• StatefulSet
• Namespace
• Resource Quota
• Limit Range
• Persistent
Volume
Kind
Secrets
Kind
• apiVersion:
• kind:
• metadata:
• spec:
Declarative Model
• Pod
• ReplicaSet
• Service
• Deployment
• Virtual Service
• Gateway, SE, DR
• Policy, MeshPolicy
• RbaConfig
• Prometheus, Rule,
• ListChekcer …
@
@
Annotations
Names
Cluster IP
Node
Port
Load
Balancer
External
Name
@
Ingress

Shopping Portal
45
/ui
/productms
/auth
/order
Gateway
Virtual Service
Deployment / Replica / Pod Nodes
Istio Sidecar - Envoy
Load Balancer
Kubernetes
Objects
Istio Objects
Firewall
P M C
Istio Control Plane
UI Pod N5
v2
Canary
v2
User X = Canary
Others = Stable
A / B Testing using
Canary Deployment
v1 UI Pod
UI Pod
UI Pod
UI
Service
N1
N2
N2
Destination
Rule
Stable / v1
EndPoints
Internal
Load Balancers
Source:
https://github.com/meta-magic/kubernetes_workshop
Users
Product Pod
Product Pod
Product Pod
Product
Service
MySQL
Pod
N4
N3
Destination
Rule
EndPoints
Review Pod
Review Pod
Review Pod
Review
Service
N1
N4
N3
Service Call
Kube DNS
EndPoints

Hybrid Cloud with Edge
46
Customer Data
Financials App
On Premise
GDPR
On Cloud
IaaS
SaaS
PaaS
E-Comm
Portal
Notification
Service
Inventory
App
Lift & Shift
Edge
K8s K8s
Delivery &
Shipping

Multi cluster
Multi / Hybrid / Edge cloud
47
Gateway
Service-1 Service-3 Service-6 Service-8
Service-3
Service-1
Cluster 1
Secure Communication
Container (Pod),
Deployment
Strategy, Replicas
Hardware Specs:
Memory, CPU,
GPU, QoS
K8s Cluster
Multi Cloud Auto Scaling
(Clone Cluster 1)
Commit Infra Code to
Service Infra Code Repository
Service as
Serverless
Service
Definitions,
Ports, Load
Balancing Algo
GW
Horizontal
Scaling
within cluster
or across
cluster
Gateway and Routing Rules
North-South
Communication
Auto Scaling across cluster
Service-3
Service-1
Cluster 2
GW
Service-7
Service-6
Cluster 3
Service-9
Service-8
Cluster 4
On-Premise Edge

Microservices Characteristics
48
We can scale our operation independently, maintain
unparalleled system availability, and introduce new
services quickly without the need for massive
reconfiguration. —
Werner Vogels, CTO, Amazon Web Services
Modularity ... is to a technological economy
what the division of labor is to a
manufacturing one.
W. Brian Arthur,
author of e Nature of Technology
The key in making great and growable systems is
much more to design how its modules communicate
rather than what their internal properties and
behaviors should be.
Alan Kay, 1998 email to the Squeak-dev list
Components
via
Services
Organized around
Business
Capabilities
Products
NOT
Projects
Smart
Endpoints
& Dumb Pipes
Decentralized
Governance &
Data Management
Infrastructure
Automation
Design for
Failure
Evolutionary
Design

Summary: Cloud Native Apps (Microservices)
49
Pros
1. Adds Complexity
2. Skillset shortage
3. Confusion on getting the
right size
4. Team need to manage
end-to-end of the
Service (From UI to
Backend to Running in
Production).
1. Robust
2. Scalable
3. Testable (Local)
4. Easy to Change and
Replace
5. Easy to Deploy
6. Cloud & Technology
Agnostic
Cons

3
Test Automation / Security
o Testing Pyramid / Test Categories
o Test Tools
o Cloud Security Architecture
o Perimeter Security Vs Zero Trust
o NIST 800-207, SDP
50

Microservices Testing Strategies
51
E2E
Testing
Integration
Testing
Contract Testing
Component Testing
Unit Testing
Number of Tests
Speed
Cost
Time
Mike Cohen’s Testing Pyramid
Test Pyramid: https://martinfowler.com/bliki/TestPyramid.html
70%
20%
10%
Ubiquitous
Language
Product
Owner
Cloud
Manager
Cloud
Architect
Cloud
Engineer
Design
Docs
Test Cases
Code
Cloud
Sr. Engineer

Microservices Testing Strategy
52
Unit Testing
A unit test exercises the
smallest piece of testable
software in the application
to determine whether it
behaves as expected.
Source: https://martinfowler.com/articles/microservice-testing/#agenda
Component Testing
A component test limits the
scope of the exercised
software to a portion of the
system under test,
manipulating the system
through internal code
interfaces and using test
doubles to isolate the code
under test from other
components.
Integration Testing
An integration test verifies
the communication paths
and interactions between
components to detect
interface defects
Integration Contract Testing
An Integration Contract test is a
test at the boundary of an
external service verifying that it
meets the contract expected by a
consuming service.
End 2 End Testing
An end-to-end test verifies that a
system meets external
requirements and achieves its
goals, testing the entire system,
from end to end
Say NO to More End 2 End Tests - Mike
Walker April 22, 2015. Google Test Blog

Microservices Testing Scenarios / Tools
53
Testing Tools
Contract Testing Scope
Integration Testing
Verifies the communication
paths and interactions between
components to detect interface
defects
Contract Testing
It is a test at the boundary of an
external service verifying that it
meets the contract expected by a
consuming service.
Payment Mock
Integration
Contract
Testing
Scope
Test Double
Montebank
Cart
Component Testing
Unit
Testing
Integration
Testing
Scope
Order
REST / HTTP or
Events / Kafka
Item ID,
Quantity,
Address..
Mock Order
Component Testing
A component test limits the
scope of the exercised
software to a portion of the
system under test.
Order
Payment
Unit
Testing
Firewall
Integration Testing Scope
REST / HTTP
Payment
Sandbox
Component
Testing
U

Summary: Microservices Testing Strategy
54
1. Unit Testing
A unit test exercises the smallest piece of testable software.
2. Component Testing
A component test limits the scope of the exercised software to a portion of the system
under test.
3. Contract Testing
It is a test at the boundary of an external service verifying that it meets the contract
expected by a consuming service
4. Integration Testing
It verifies the communication paths and interactions between components to detect
interface defects.

Security
o SANS: Cloud Security Architecture
o Perimeter Security Vs. Zero Trust
o Software Defined Perimeter
o Service Mesh
55

SANS Cloud Security Architecture Principles
56
Source: RSA Conference 2019 – A Cloud Security Architecture workshop. Dave Shackleford Sr. Instructor SANS Institute
Think
Components
Design for
Failure
Always
Think of
Feedback Loops
Use Different
Storages
Options
Built-In
Security
at every Layer
CENTRALIZATION
Focus on
Centralization
Standards & Automation
Design for
Elasticity

Perimeter Security Vs. Zero Trust
57
Classic Security Model
Perimeter Security
• Location Based (External /
Internal)
• Anyone inside the network is
always trusted.
• Based on Layered Security
Never Trust,
Always Verify 1
Implement
Least Privilege 2
(Always)
Assume Breach 3
Forrester's John Kindervag 2010: No More Chewy Centers: Introducing
The Zero Trust Model Of Information Security
Inspired from Jericho Forum Commandments v1.2 May 2007
Source: Microsoft: Jericho & Modern Security
Restrict everything to a secure Network
Zero Trust
Protect Assets
anywhere with
Central Policy

NIST 800-207: Zero Trust Architecture
58
Source: NIST SP 800-207:Zero Trust Architecture https://csrc.nist.gov/publications/detail/sp/800-207/final
A User, An Application, or a Device – Operating on (or with) a Computer System which has access to an
Enterprise Resource
Subject
Is an Application, Document, Data, Database, Workload that’s under the Enterprise Control protected
by the Zero Trust System
Resource
Policy Enforcement Point
Policy Engine Policy Administrator
Policy Decision Point
Control
Plane
Data Plane Resource
Subject
User
App Device
UnTrusted Trusted
CDM
System
GRC
System
Threat
Intelligence
Activity
Logs
Data
Access
Policy
PKI
IAM
SIEM
1 2
3

Software Defined Perimeter: Architecture
59
Cloud Security Alliance: May 27, 2020: SDP and Zero Trust
Policy
Enforcement Point
SDP Gateway
SDP Controller
Policy Decision Point
Control Plane
Data Plane
Resource
Subject
User
App
Device
SDP
Client
Source: https://cloudsecurityalliance.org/artifacts/sdp-architecture-guide-v2/
IH: Initiating Host
Control Messages
Data
SDP requires
2 Security
modules
1. mTLS
2. SPA
AH
AH: Accepting Host
The model depicted below is Similar to Enclave Resource model from NIST 800-207 Architecture. NIST team
defined that based on Cloud Security Alliance SDP Architecture.

Istio Security Architecture – Defense in Depth
60
Source: https://istio.io/docs/concepts/security/
• Citadel for key and
certificate management
• Sidecar and perimeter
proxies to implement
secure communication
between clients and
servers
• Pilot to
distribute authentication
policies and secure
naming information to the
proxies
• Mixer to manage
authorization and auditing

4
DevOps & SRE / Observability
o DevOps Principles
o Site Reliability Engineering
o Benefits of DevOps
o DevSecOps
61

Is DevOps - A Technology or Collection
of technologies?
Answer: NO
Is DevOps - A way of programming?
Answer: NO
is DevOps - A Process?
Answer: NO
Can you become a DevOps Engineer?
Answer: NO - its not a skill set
It’s the Destination
What is DevOps?

What Dev & Ops want?
63
Development Team
Agility
Operations Team
Stability
Developers Keep
throwing releases
over the wall and
get pushed back by
the operations
team.

5 DevOps Principles – CALMS Framework
64
Source: https://www.atlassian.com/devops/frameworks/calms-framework
DevOps isn’t a
process, or a different
approach to
development.
It’s a culture change.
DevOps culture is
collaboration.
Build, Test, Deploy, and Provisioning
automation are typical starting points for
teams.
Another major contribution of DevOps is
“configuration as code.” Developers strive to
create modular, composable applications
because they are more reliable and maintainable.
CULTURE AUTOMATION
LEAN MEASUREMENT SHARING
Continuous
Improvement
with Canary
Releases and A/B
Testing
Continuous
Improvement
requires Data to
measure the
changes
Sharing responsibility,
success, failure goes a
long way toward
bridging that divide
between Dev and Ops.
You built it, You run it.

Implementing CALMS – DevOps Principles
65
Capability Centric Design
Reduce
Organization Silos
CULTURE
Leverage
Tooling &
Automation
CI/CD Pipeline & Container Orchestration
AUTOMATION
Implement
Gradual
Change
& Agile: Kanban
LEAN
Measure
Everything
Service Mesh: Observability
MEASUREMENT
Accept
Failure as
Normal
Design for Failure
SHARING
Source: IBM DevOps Vs. SRE https://www.youtube.com/watch?v=KCzNd3StIoU Google: https://www.youtube.com/watch?v=uTEL8Ff1Zvk

class SRE implements DevOps – CALMS
66
Capability Centric Design
Reduce
Organization Silos
CULTURE
Leverage
Tooling &
Automation
CI/CD Pipeline & Container Orchestration
AUTOMATION
Implement
Gradual
Change
& Agile: Kanban
LEAN
Measure
Everything
Service Mesh: Observability
MEASUREMENT
Accept
Failure as
Normal
Design for Failure
SHARING
 Share Ownership
 SLOs & Blameless PM
 Canary Deployment, A/B Testing
 Automate this years Job  Measure toil & reliability

Pillars of Observability
67
Immutable records of
discrete events that
happen over time
Logs/events
Numbers describing a
particular process or
activity measured over
intervals of time
Metrics
Data that shows, for
each invocation of each
downstream service,
which instance was called,
which method within that
instance was invoked, how
the request performed, and
what the results were
Traces
Source: A Beginners guide to Observability by Splunk

Observability in Kubernetes Worker Node
68
eBPF Programs Network Flow Log
K-Probe
Connection
Tracker
Linux Kernel
Prometheus Envoy Proxy Log Collector FluentD
Pods Pods Pods
Pods Pods Pods
Service
Pods Pods Pods
Pods Pods Pods
Service
Namespace
Pods Pods Pods
Pods Pods Pods
Service
Namespace
Observability Tools

Summary: Benefits of DevOps
70
 Velocity
o Agile / Kanban,
o Capability Centric Design
o Domain Driven Design
o Event Sourcing & CQRS
o Microservices Architecture
Code Build Manage Learn
Idea
 Quality / Stability
o Test Automation
o Build Pipeline Automation
o Continuous Integration
o Continuous Delivery
o Continuous Deployment
o Observability
People Process Tools

DevSecOps
71
Source: https://www.atlassian.com/devops/devops-tools/devsecops-tools

DevSecOps
72
Recommended by US DoD DevSecOps Best Practices
Source:
Page 17
US DoD
Enterprise
DevSecOps
Fundamentals

6
DevSecOps Playbook
73
1 Adopt a DevSecOps Culture
2 Adopt Infrastructure as Code
3
Adopt Containerized
Microservices
4
Adopt a Capability Model, not a
Maturity Model
5
Drive Continuous Improvement
through Key Capabilities
Establish a Software Factory
7
Define a meaningful
DevSecOps pipeline
8
Adapt an Agile Acquisition
Policy for Software
9
Tirelessly Pursue Cyber
Resilience
10
Shift Left: Operational Test &
Eval
Source: US DoD DevSecOps Fundamentals Playbook

SecOps / DevOps
74
Source: SCI – Your Eyes in the Sky By AI Huger, Nov 15, 2021
While SecOps starts on the left with security posture and attack surface
management as its entry point, DevOps start at the far right with
continuous integration and continuous delivery (CI/CD) pipeline and
application/API security as their main care about.
As SecOps moves right and begins to influence the other
stakeholders within a mature organization, DevOps shifts
left to include pre-deploy checks by using runtime security
inputs.

Case Studies
• NetFlix
• Spotify
• EE Implementation Guidelines
75

100s Microservices
1,000s Releases / Day
10,000s Virtual Machines
100K+ User actions / Second
81 M Customers Globally
1 B Time series Metrics
10 B Hours of video streaming
every quarter
Source: NetFlix: : https://www.youtube.com/watch?v=UTKIT6STSVM
10s OPs Engineers
0 NOC
0 Data Centers
So what do NetFlix think about DevOps?
No DevOps
Don’t do lot of Process / Procedures
Freedom for Developers & be Accountable
Trust people you Hire
No Controls / Silos / Walls / Fences
Ownership – You Build it, You Run it.
76

50M Paid Subscribers
100M Active Users
60 Countries
Cross Functional Team
Full, End to End ownership of features
Autonomous
1000+ Microservices
Source: https://microcph.dk/media/1024/conference-microcph-2017.pdf
1000+ Tech Employees
120+ Teams
77

Implementing Elastic Engineering (Day wise)
78
2. Domain Driven Design
Deep Dive into Domain-Driven Design for Design &
Development of the Software. Understand how to
Create Bounded Context based on Epics and User
Stories. How create Entities and Value Objects. How to
refactor monolithic Entity into Service Specific Entities.
3. Event Sourcing & Kafka
Event Sourcing is based on Commands and
(Immutable) Events combined with CQRS it's a very
powerful pattern to create asynchronous Event based
services. Kafka can be used to fully implement an Event
Sourced system. Kafka is a distributed Fault-Tolerant
replicated log.
4. Big Data - Redis, MongoDB, Data Lakes
This section focuses on a Comparison between SQL and
NoSQL Databases and shows various design patterns for
Redis and MongoDB. It dives deep into the concept of
Partitions and Sharding and Geo Partitions. It shows
examples of Distributed Transactions in Event based
system.
1. Design Thinking, Lean Startup & Agile
This section focuses on Design Thinking, Lean Startup
and Agile (Scrum / Kanban). Write Specs using Epics,
User Stories, Acceptance Criteria, Concept of Minimum
Viable Product (MVP) and release plans and write test
cases based on the Acceptance Criteria defined.

79
6. Microservices Testing Strategies
Microservices Testing Strategies includes
examples from
• JUnit 5 / Springboot Test
• Cucumber
• Selenium
• Mockito
• WireMock
• Pact
7. Docker, Kubernetes
Containers are the de-facto standard for deploying the
services. Kubernetes is the cloud-agnostic container
Orchestration solution. Kubernetes focuses on Container
deployment, Multi Version Rollouts, Traffic Routing, Auto Pod
Scaling, Application Setup using Secrets, Config Maps etc.
5. Microservices Architecture
Microservices Architecture focuses on various
infra structure patterns like API Gateway, Service
Discovery apart from the fundamental principles.
Migration from Monolithic shows 10+ Design
patterns (strangler Fig, Change Data Capture,
etc.,) for the transformation into Microservices.
8. Service Mesh (Istio) & Security
Istio is one of the popular Service mesh implementations
that run on top of Kubernetes. It addresses the following
functionality - Advanced Traffic routing, Security, Policy,
& Observability.

80
10. DevOps & SRE / DevSecOps
DevOps is the 3rd phase of the Application Modernization
• Architecture
• Infrastructure
• Delivery
Continuous Delivery is a critical requirement in DevOps. Without CD
what you have is a bunch of tools deployed.
10. Observability
Observability is a critical feature in a container-based
deployment. Following are the key tools that can be enabled
in Istio (Service Mesh).
• Prometheus
• Zipkin / Jaeger
• Kiali
• Grafana / Kibana
10. CI/CD Pipeline
CI/CD Pipeline is critical in fully automated development
cycles. Building software, running test cases, Building
infrastructure, packaging, and deploying the App/Service in
the Kubernetes cluster. Jenkins, GitHub Actions and Tekton are
popular tools for CI/CD pipeline automation
9. Cloud Architecture & Terraform
Cloud Architecture focuses on various Cloud solutions like
IaaS, PaaS, SaaS, FaaS, and multi-cloud environments.
Connecting On-Premise, Edge, and Multi-Cloud
Environments and the challenges and network routing and
security policies. Use Terraform to build cloud
infrastructures.

81
11. Networking / Security & Zero Trust
Understand the Concepts of Overlay Networking along with
Software Defined Networking, & SD-WAN. Understand the
Concepts of classic Perimeter based Security and Security
based on Zero Trust principles and how DevOps needs to be
extended with Security – DevSecOps.

Source Code: https://github.com/MetaArivu

DREAM | AUTOMATE | EMPOWER
Araf Karsh Hamid :
India: +91.999.545.8627
http://www.slideshare.net/arafkarsh
https://www.linkedin.com/in/arafkarsh/
https://www.youtube.com/user/arafkarsh/playlists
http://www.arafkarsh.com/
@arafkarsh
arafkarsh

Elastic-Engineering

More Related Content

What's hot

Similar to Elastic-Engineering

More from Araf Karsh Hamid

Recently uploaded

Elastic-Engineering

Editor's Notes