stackconf 2023 | Bringing Order to Chaos: Make Your Systems More Resilient with Chaos Engineering by Sayan Mondal.pdf

Confidential / © Harness Inc. 2020
Bringing order to Chaos
Make your systems more resilient
with Chaos Engineering

Sayan Mondal
Senior Software Engineer II at
Maintainer at
@s_ayanide
@s-ayanide ● Senior Software Engineer 2 at Harness
● Maintainer of LitmusChaos (CNCF incubating
project) for 2 years, contributing since 3.5 years
● Volunteer and mentor at Linux Foundation
● Chaos Carnival organizing team member

Confidential / © Harness 2022
What causes Downtime
Application Failures
Reputational Impact Financial Impact Poor User Experience
Slack’s Outages Est. >$55M in losses to WF 75,000+ passengers travel
plans impacted
Infrastructure Failures Operational Failures

The cloud native problem
Proliferation of applications
into micro services leads to a
RELIABILITY challenge
In cloud native, your code depends on hundreds
of other microservices and runs on many
platforms. The potential of being subjected to a
dependent component failure is huge.
1
Your Application
3
Cloud Native Services
CoreDNS, Envoy,
Prometheus, OpenEBS, etc.
5
Platform Services
Infrastructure
2
Your Application’s
Dependencies
MongoDB, Kafka, TiKV,
Vitess, Postgres, etc.
4
Kubernetes Services

Problems with existing solutions
Not automated
Not collaborative
Reactive Approach
● No proactive investments for
failure testing
● Generally driven by root
cause analysis
● No proactive investments for
failure testing
● Generally driven by root
cause analysis
● Driven by Ops
● Not integrated into CI/CD or
Gamedays

The solution? Chaos Engineering
SREs + Developers
Experiments are in Git just like code
Chaos engineering is collaborative
Collaborative chaos experiments in
a centralized control plane
Optimize initial investment
Reduce the inertia for starting chaos
Robust Experiments
Public and private chaos hubs with
ready to use experiments
Find weaknesses during build/test phase
Verifying at dev stage saves money
Integrate into CI/CD systems
Rollout automated and controlled
chaos experiments across
prod/non-prod environments
Measure the impact of inducing chaos
Build confidence by starting small
Enables observability for Chaos
Chaos metrics used to assess
impact and manage SLOs/Errors

How to do Chaos Engineering?
Why Resilience? Achieving Resilience Litmus 101 Litmus Experiments Contributing

Project
LitmusChaos -
a CNCF incubating project
Started in 2017; 4+ years of active development
350K+ Litmus installations; 30x usage growth in the last 3
quarters, 50+ chaos experiments, 100+ contributors
Stable platform : 2.0 released
50+ enterprises using 2.0
CNCF Incubating project
Litmus is an open source platform for practicing chaos
engineering in a cloud native way.
CNCF
Incubating
project 30x growth in per-day installations of Litmus
in the last 3 quarters; 1500 installations per day
Litmus is adopted by

What’s Exciting about LitmusChaos?
Chaos
Workflow editor
User
ChaosCenter
Management &
Teaming
ChaosAgent
Custom Image
Registry
GitOps
ChaosCenter
Monitoring and
Observability
Resilience
Score
Calculation
ChaosHub
Scheduling
More Control
over Chaos
Results
New &
Enhanced
Experiments
Support for
Cri-o,
ContainerD and
Docker

Architecture Overview of LitmusChaos 2.0

Chaos Workflow Deep Dive

The ChaosCenter is a single source of truth to control
all the different Chaos Activities happening around
Litmus. From the ChaosCenter you get the freedom to
manage every single part of Litmus and shape your
workflows exactly the way you want it.
A ChaosAgent in Litmus is nothing but the target
cluster where Chaos would be injected via Litmus.
There should always be at least one or more than
one ChaosAgents connected to the ChaosCenter.
Each individual ChaosAgent can be chosen to be
the Target Agent for Chaos Injection.
Core Components of LitmusChaos
“ “
ChaosCenter
ChaosAgent

Variety of faults offered in LitmusChaos
Pod Chaos Node Chaos Network Chaos Stress Chaos Cloud Services Application Chaos
Pod Failure
Container Kill
Pod Autoscale
Node Drain
Forced Eviction (Node Taints)
Node Restart/PowerOff
Network Latency
Packet Loss
Network Corruption,
Duplication
Pod, Node CPU Hog
Pod, Node Memory Hog
Pod, Node Disk Stress
Pod Ephemeral Storage
Fill
AWS EKS EC2 Termination
AWS EBS Disk Detach
GCP GPD Disk Detach
Kafka Leader Broker Failure
Cassandra Ring Disruption
OpenEBS Control Plane / Volume
Failure

The Features
Let’s take a look at the core
features offered

Chaos Center
Chaos Workflows
Automate dependency setup,
create complex chaos
scenarios, support definition of
load/validation jobs along with
chaos injection
Multiple options
From Templates, Custom
Workflows from Scratch (using
ChaosHubs), From pre-created
YAMLs
Crontrol
Chaos Experiments Sequence
Control (Parallel as well as
Sequential steps creation)
Schedules
Creation of either Singular or
Cron Workflows as Schedules
Experiment Priority
Attaching priority to Chaos
Experiments based on your use
cases

Workflow Management
GitOps
Rolling out automated changes
using GitOps
Custom Image
Allowing image addition from
custom image server (both
public and private)
Resilience Score
Measure and Analyse the
Resilience Score of each
workflow

Multi Tenancy
Scope Support
Supports setup (control plane &
agents) and execution of chaos
experiments in both cluster
scoped and namespace scoped
modes.
Authentication
Authentication and a smooth
onboarding process. Choose
between email and password
auth or OAuth with Google or
GitHub for your teams.
Create Teams
Creating a Team of multiple
Users and Project Management
Fine-Grained RBAC
Flexible RBAC to drill down
and grant correct privileges
to users.

Monitoring & Observability
Connect Datasource
Connect a Data Source (from
any Agent) and monitor
workﬂows
Visualization
Visualize workflow run statistics
and aggregated schedules
Comparison
Compare two or more
Workﬂows
Upload Dashboards
Upload shared/downloadable
dashboards available in the
community
Tune Dashboards
Edit queries, Tune dashboards
to create a custom one from
scratch
Monitor in Real Time
Monitor effect of chaos in real
time with interleaved events
and metrics from Prometheus
Datasource

GitOps for Chaos
Git based SCM
Integrates with Git-based SCM
to provide a single source of
truth for chaos artifacts
(workflows), such that changes
are synchronized bi-directionally
b/w the git source and the chaos
center - thereby pulling the
latest artifact for execution.
Tracking
Provides an event-tracker
microservice to automatically
launch “subscribed” chaos
workflows upon app upgrades
affected by GitOps tools like
ArgoCD, Flux

Non Kubernetes Chaos
Chaos on Infra
Inject chaos on infrastructure
resources such as
VMs/instances and disks (AWS,
GCP, Azure, VMWare)
Attack Baremetal
Introduces chaos experiments
to bring down baremetal nodes
that provide IPMI-based
out-of-band access.
Chaos on Machine
Litmus has developed m-agent:
a platform generic daemon
agent for orchestrating chaos
into any computing node.

Hands On
Demo Time

Install the components

Pick the faults

Inject Chaos in your application

Observe Impact

Seamless support for cross cloud connectivity
and interactions. Target you applications running
on your preferred cloud provider with Litmusctl.
Multi-Cloud Support with LitmusChaos 2.0

Future Roadmap What’s ahead for us?
● Increased support for chaos against Non-Kubernetes infrastructure components
● More Application speciﬁc chaos experiments with native faults and health checks
● Improved Chaos SDK for creation of user-deﬁned experiments
● Additional probe types for diverse steady state-hypothesis validation
● Improved Observability for chaos experiments
● More community supported Chaos Types

How to Contribute?
We welcome contributions of all kinds
● Development of features, bug fixes, and
other improvements.
● Documentation including reference
material and examples.
● Bug and feature reports
You can choose from a list of sub-dependent
repos to contribute to, a few highlighted repos
that Litmus uses are:
● Chaos-charts
● Chaos-workflows
● Test-tools
● Litmus UI
● Litmus-go
● website-litmuschaos

Conclusion
● Chaos Engineering
● A tool called LitmusChaos
● Architecture principle
● Core Components of Chaos Induction
● Demo
Conclusion

/LitmusChaos
/litmuschaos
Follow Litmus on
Thank You
@s_ayanide
/s-ayanide
Contact me on

stackconf 2023 | Bringing Order to Chaos: Make Your Systems More Resilient with Chaos Engineering by Sayan Mondal.pdf

Recommended

Recommended

More Related Content

Similar to stackconf 2023 | Bringing Order to Chaos: Make Your Systems More Resilient with Chaos Engineering by Sayan Mondal.pdf

Similar to stackconf 2023 | Bringing Order to Chaos: Make Your Systems More Resilient with Chaos Engineering by Sayan Mondal.pdf (20)

Recently uploaded

Recently uploaded (20)

stackconf 2023 | Bringing Order to Chaos: Make Your Systems More Resilient with Chaos Engineering by Sayan Mondal.pdf