SlideShare a Scribd company logo
Building and Monitoring
Services at Lithium
(fault tolerance, resiliency and monitoring)

Paul Cichonski, Senior Software Engineer
@paulcichonski
Services at Lithium Use:

2
Failure is a Constant, Need to
Avoid Cascading Failure

Image Source: Netflix Hystrix: https://github.com/Netflix/Hystrix/wiki

3
We All Know How to Simulate
Failure:

4
But how do we develop code to
deal with failure?

5
Need to build fault tolerant and
resilient services... How?

Clustering, for high-availability, is
not enough to protect against
cascading failure
6
#1 Fail Fast: use timeouts
aggressively

7
#2 Use circuit breakers on
network calls

8
#3 Use async communication
when possible

9
#4 Have well thought-out
backpressure mechanisms

10
#5 Use cross-region (or crossdatacenter) replication

11
#6 Failure models should be built
into the business requirements of a
service

12
Read:

13
Even with all of that, your app will
still fail, so how do you recover
quickly?

14
Devops/Cloudops Model:
OODA

15
Observe and Orient: you need
metrics and dashboards

16
You Need Metrics
• Reduce “map/territory” confusion
• We use Yammer Metrics
– Timers
– Meters
– Histograms

• We use them a lot
– Every class has at least one metric, most
have multiple
17
You Need to Visualize the Metrics

18
You Need Dashboards Keyed to
Business Functionality

19
Use alerting as a last resort
(because sometimes we need to
sleep)

20
Decide and Act: you need robust
CI and fast code roll-outs

21
Rinse and Repeat

22

More Related Content

What's hot

.NET Security (Radu Vunvulea)
.NET Security (Radu Vunvulea).NET Security (Radu Vunvulea)
.NET Security (Radu Vunvulea)
Radu Vunvulea
 
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code DeploysDevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
Andreas Grabner
 
Soluciones Dynatrace
Soluciones DynatraceSoluciones Dynatrace
Soluciones Dynatrace
Innovation Strategies
 
Observability, Distributed Tracing, and Open Source: The Missing Primer
Observability, Distributed Tracing, and Open Source: The Missing PrimerObservability, Distributed Tracing, and Open Source: The Missing Primer
Observability, Distributed Tracing, and Open Source: The Missing Primer
VMware Tanzu
 
DevNet UX Creative Design 101 workshop
DevNet UX Creative Design 101 workshopDevNet UX Creative Design 101 workshop
DevNet UX Creative Design 101 workshop
Cisco DevNet
 
NashTech - Azure Application Insights
NashTech - Azure Application InsightsNashTech - Azure Application Insights
NashTech - Azure Application Insights
Phi Huynh
 
DevOps for AI Apps
DevOps for AI AppsDevOps for AI Apps
DevOps for AI Apps
Richin Jain
 
Building Observable Infrastructure and Code
Building Observable Infrastructure and CodeBuilding Observable Infrastructure and Code
Building Observable Infrastructure and Code
UllyCarolinneSampaio
 
Automate threat detections and avoid false positives
Automate threat detections and avoid false positivesAutomate threat detections and avoid false positives
Automate threat detections and avoid false positives
Elasticsearch
 
Modern App Architecture - Microservices, API Friendly
Modern App Architecture - Microservices, API FriendlyModern App Architecture - Microservices, API Friendly
Modern App Architecture - Microservices, API Friendly
DevOps Indonesia
 
Metrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
Metrics Driven DevOps - Automate Scalability and Performance Into your PipelineMetrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
Metrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
Andreas Grabner
 
Humana digitally transforming health and well-being with Pivotal cloud foundr...
Humana digitally transforming health and well-being with Pivotal cloud foundr...Humana digitally transforming health and well-being with Pivotal cloud foundr...
Humana digitally transforming health and well-being with Pivotal cloud foundr...
Dynatrace
 
WestJet Customer Presentation
WestJet Customer PresentationWestJet Customer Presentation
WestJet Customer Presentation
Splunk
 
Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019
Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019
Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019
Eclipse IoT
 
Dynatrace: Going beyond APM and soaring to the future
Dynatrace: Going beyond APM and soaring to the futureDynatrace: Going beyond APM and soaring to the future
Dynatrace: Going beyond APM and soaring to the future
Dynatrace
 
Automate Your Container Deployments Securely
Automate Your Container Deployments SecurelyAutomate Your Container Deployments Securely
Automate Your Container Deployments Securely
DevOps.com
 
Habitat for Reals
Habitat for RealsHabitat for Reals
Habitat for Reals
Mandi Walls
 
Threat hunting with Elastic APM
Threat hunting with Elastic APMThreat hunting with Elastic APM
Threat hunting with Elastic APM
FaithWestdorp
 
Construção de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizadaConstrução de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizada
Elasticsearch
 
Api Management and Demo
Api Management and DemoApi Management and Demo
Api Management and Demo
DevOps Indonesia
 

What's hot (20)

.NET Security (Radu Vunvulea)
.NET Security (Radu Vunvulea).NET Security (Radu Vunvulea)
.NET Security (Radu Vunvulea)
 
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code DeploysDevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
DevOps Days Toronto: From 6 Months Waterfall to 1 hour Code Deploys
 
Soluciones Dynatrace
Soluciones DynatraceSoluciones Dynatrace
Soluciones Dynatrace
 
Observability, Distributed Tracing, and Open Source: The Missing Primer
Observability, Distributed Tracing, and Open Source: The Missing PrimerObservability, Distributed Tracing, and Open Source: The Missing Primer
Observability, Distributed Tracing, and Open Source: The Missing Primer
 
DevNet UX Creative Design 101 workshop
DevNet UX Creative Design 101 workshopDevNet UX Creative Design 101 workshop
DevNet UX Creative Design 101 workshop
 
NashTech - Azure Application Insights
NashTech - Azure Application InsightsNashTech - Azure Application Insights
NashTech - Azure Application Insights
 
DevOps for AI Apps
DevOps for AI AppsDevOps for AI Apps
DevOps for AI Apps
 
Building Observable Infrastructure and Code
Building Observable Infrastructure and CodeBuilding Observable Infrastructure and Code
Building Observable Infrastructure and Code
 
Automate threat detections and avoid false positives
Automate threat detections and avoid false positivesAutomate threat detections and avoid false positives
Automate threat detections and avoid false positives
 
Modern App Architecture - Microservices, API Friendly
Modern App Architecture - Microservices, API FriendlyModern App Architecture - Microservices, API Friendly
Modern App Architecture - Microservices, API Friendly
 
Metrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
Metrics Driven DevOps - Automate Scalability and Performance Into your PipelineMetrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
Metrics Driven DevOps - Automate Scalability and Performance Into your Pipeline
 
Humana digitally transforming health and well-being with Pivotal cloud foundr...
Humana digitally transforming health and well-being with Pivotal cloud foundr...Humana digitally transforming health and well-being with Pivotal cloud foundr...
Humana digitally transforming health and well-being with Pivotal cloud foundr...
 
WestJet Customer Presentation
WestJet Customer PresentationWestJet Customer Presentation
WestJet Customer Presentation
 
Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019
Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019
Exploring the Trend Toward the Edge | Eclipse IoT Day Santa Clara 2019
 
Dynatrace: Going beyond APM and soaring to the future
Dynatrace: Going beyond APM and soaring to the futureDynatrace: Going beyond APM and soaring to the future
Dynatrace: Going beyond APM and soaring to the future
 
Automate Your Container Deployments Securely
Automate Your Container Deployments SecurelyAutomate Your Container Deployments Securely
Automate Your Container Deployments Securely
 
Habitat for Reals
Habitat for RealsHabitat for Reals
Habitat for Reals
 
Threat hunting with Elastic APM
Threat hunting with Elastic APMThreat hunting with Elastic APM
Threat hunting with Elastic APM
 
Construção de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizadaConstrução de uma plataforma de observabilidade centralizada
Construção de uma plataforma de observabilidade centralizada
 
Api Management and Demo
Api Management and DemoApi Management and Demo
Api Management and Demo
 

Viewers also liked

Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...
Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...
Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...
Nagios
 
LabVIEW Based Monitoring the Building in wireless communication
LabVIEW Based Monitoring the Building in wireless communicationLabVIEW Based Monitoring the Building in wireless communication
LabVIEW Based Monitoring the Building in wireless communication
Sathish Kumar
 
Site Operation Manual for a Typical Air Monitoring Site
Site Operation Manual for a Typical Air Monitoring SiteSite Operation Manual for a Typical Air Monitoring Site
Site Operation Manual for a Typical Air Monitoring Site
TAMUK
 
Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...
Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...
Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...
Torben Haagh
 
Khulisa Management Services- ECD Site Monitoring Instrument
Khulisa Management Services- ECD Site Monitoring InstrumentKhulisa Management Services- ECD Site Monitoring Instrument
Khulisa Management Services- ECD Site Monitoring Instrument
kaleylemottee
 
The Benefits of Having Nerds On Site Monitoring Your Technology
The Benefits of Having Nerds On Site Monitoring Your TechnologyThe Benefits of Having Nerds On Site Monitoring Your Technology
The Benefits of Having Nerds On Site Monitoring Your Technology
Kevin Lloyd
 
River monitoring site 7
River monitoring site 7River monitoring site 7
River monitoring site 7
John Hoopman
 
How to build a budget transparency site: 5 easy steps
How to build a budget transparency site: 5 easy steps How to build a budget transparency site: 5 easy steps
How to build a budget transparency site: 5 easy steps
Lucy Chambers
 
The Drupal Ecosystem for Drupal Services
The Drupal Ecosystem for Drupal ServicesThe Drupal Ecosystem for Drupal Services
The Drupal Ecosystem for Drupal Services
Vardot
 
Big Data and Social Monitoring: Building Meaningful Relationships
Big Data and Social Monitoring: Building Meaningful RelationshipsBig Data and Social Monitoring: Building Meaningful Relationships
Big Data and Social Monitoring: Building Meaningful Relationships
Emanuela Zaccone
 
How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...
How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...
How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...
Target Health, Inc.
 
Low power wireless sensor network for building monitoring
Low power wireless sensor network for building monitoringLow power wireless sensor network for building monitoring
Low power wireless sensor network for building monitoring
ecwayerode
 
Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...
Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...
Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...
Target Health, Inc.
 
#lspe Building a Monitoring Framework using DTrace and MongoDB
#lspe Building a Monitoring Framework using DTrace and MongoDB#lspe Building a Monitoring Framework using DTrace and MongoDB
#lspe Building a Monitoring Framework using DTrace and MongoDB
dan-p-kimmel
 
ECD monitoring instrument
ECD monitoring instrumentECD monitoring instrument
ECD monitoring instrument
Khulisa Management Services
 
Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...
Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...
Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...
ivteam
 
Meeting Enrollment Goals in a Competitive Environment
Meeting Enrollment Goals in a Competitive EnvironmentMeeting Enrollment Goals in a Competitive Environment
Meeting Enrollment Goals in a Competitive Environment
jbarag
 
Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...
Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...
Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...
Jesús Fontecha
 
Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...
Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...
Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...
Kevin Perry
 

Viewers also liked (19)

Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...
Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...
Nagios Conference 2013 - Thomas Dunbar - Building Technology for Storage Syst...
 
LabVIEW Based Monitoring the Building in wireless communication
LabVIEW Based Monitoring the Building in wireless communicationLabVIEW Based Monitoring the Building in wireless communication
LabVIEW Based Monitoring the Building in wireless communication
 
Site Operation Manual for a Typical Air Monitoring Site
Site Operation Manual for a Typical Air Monitoring SiteSite Operation Manual for a Typical Air Monitoring Site
Site Operation Manual for a Typical Air Monitoring Site
 
Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...
Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...
Presentation Mrs.Smolka Ursula, Ramboll: costs and benefits when monitoring s...
 
Khulisa Management Services- ECD Site Monitoring Instrument
Khulisa Management Services- ECD Site Monitoring InstrumentKhulisa Management Services- ECD Site Monitoring Instrument
Khulisa Management Services- ECD Site Monitoring Instrument
 
The Benefits of Having Nerds On Site Monitoring Your Technology
The Benefits of Having Nerds On Site Monitoring Your TechnologyThe Benefits of Having Nerds On Site Monitoring Your Technology
The Benefits of Having Nerds On Site Monitoring Your Technology
 
River monitoring site 7
River monitoring site 7River monitoring site 7
River monitoring site 7
 
How to build a budget transparency site: 5 easy steps
How to build a budget transparency site: 5 easy steps How to build a budget transparency site: 5 easy steps
How to build a budget transparency site: 5 easy steps
 
The Drupal Ecosystem for Drupal Services
The Drupal Ecosystem for Drupal ServicesThe Drupal Ecosystem for Drupal Services
The Drupal Ecosystem for Drupal Services
 
Big Data and Social Monitoring: Building Meaningful Relationships
Big Data and Social Monitoring: Building Meaningful RelationshipsBig Data and Social Monitoring: Building Meaningful Relationships
Big Data and Social Monitoring: Building Meaningful Relationships
 
How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...
How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...
How to Efficiently and Effectively Balance Central Monitoring with On-Site Mo...
 
Low power wireless sensor network for building monitoring
Low power wireless sensor network for building monitoringLow power wireless sensor network for building monitoring
Low power wireless sensor network for building monitoring
 
Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...
Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...
Experience from Phase 3 Study Using Risk- Based Monitoring and eSource Method...
 
#lspe Building a Monitoring Framework using DTrace and MongoDB
#lspe Building a Monitoring Framework using DTrace and MongoDB#lspe Building a Monitoring Framework using DTrace and MongoDB
#lspe Building a Monitoring Framework using DTrace and MongoDB
 
ECD monitoring instrument
ECD monitoring instrumentECD monitoring instrument
ECD monitoring instrument
 
Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...
Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...
Notes to support the presentation 'Introduction to the Visual Infusion Phlebi...
 
Meeting Enrollment Goals in a Competitive Environment
Meeting Enrollment Goals in a Competitive EnvironmentMeeting Enrollment Goals in a Competitive Environment
Meeting Enrollment Goals in a Competitive Environment
 
Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...
Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...
Exploring the AmIHEALTH paradigm. Monitoring in Healthcare: Building mHealth ...
 
Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...
Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...
Pendergrass, Gary, GeoEngineers, CCR Rule Compliance Innovative Geophysics to...
 

Similar to Building and Monitoring Services at Lithium

Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)
VMware Tanzu
 
Webinar : Microservices and Containerization
Webinar : Microservices and ContainerizationWebinar : Microservices and Containerization
Webinar : Microservices and Containerization
Newt Global Consulting LLC
 
Evolving to Cloud-Native - Nate Schutta 2/2
Evolving to Cloud-Native - Nate Schutta 2/2Evolving to Cloud-Native - Nate Schutta 2/2
Evolving to Cloud-Native - Nate Schutta 2/2
VMware Tanzu
 
Application Modernisation through Event-Driven Microservices
Application Modernisation through Event-Driven Microservices Application Modernisation through Event-Driven Microservices
Application Modernisation through Event-Driven Microservices
confluent
 
Microservices Architecture, Monolith Migration Patterns
Microservices Architecture, Monolith Migration PatternsMicroservices Architecture, Monolith Migration Patterns
Microservices Architecture, Monolith Migration Patterns
Araf Karsh Hamid
 
Securing a Cloud Migration
Securing a Cloud MigrationSecuring a Cloud Migration
Securing a Cloud Migration
Carlos Andrés García
 
Securing a Cloud Migration
Securing a Cloud MigrationSecuring a Cloud Migration
Securing a Cloud Migration
VMware Tanzu
 
How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...
How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...
How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...
MuleSoft
 
The Future of Cloud Innovation, featuring Adrian Cockcroft
The Future of Cloud Innovation, featuring Adrian CockcroftThe Future of Cloud Innovation, featuring Adrian Cockcroft
The Future of Cloud Innovation, featuring Adrian Cockcroft
Dun & Bradstreet Cloud Innovation Center
 
Think Small To Go Big - Introduction To Microservices
Think Small To Go Big - Introduction To MicroservicesThink Small To Go Big - Introduction To Microservices
Think Small To Go Big - Introduction To Microservices
Ryan Baxter
 
Introduction to Serverless through Architectural Patterns
Introduction to Serverless through Architectural PatternsIntroduction to Serverless through Architectural Patterns
Introduction to Serverless through Architectural Patterns
Mathieu Mailhos
 
Availability in a cloud native world v1.6 (Feb 2019)
Availability in a cloud native world v1.6 (Feb 2019)Availability in a cloud native world v1.6 (Feb 2019)
Availability in a cloud native world v1.6 (Feb 2019)
Haytham Elkhoja
 
From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018
Christophe Rochefolle
 
Cloud-Native Fundamentals: An Introduction to 12-Factor Applications
Cloud-Native Fundamentals: An Introduction to 12-Factor ApplicationsCloud-Native Fundamentals: An Introduction to 12-Factor Applications
Cloud-Native Fundamentals: An Introduction to 12-Factor Applications
VMware Tanzu
 
We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?
Jonas Bonér
 
APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?
APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?
APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?
FABERNOVEL TECHNOLOGIES
 
Cloud computing for microprocessor tools
Cloud computing for microprocessor toolsCloud computing for microprocessor tools
Cloud computing for microprocessor tools
GowthamRider
 
ThatConference 2016 - Highly Available Node.js
ThatConference 2016 - Highly Available Node.jsThatConference 2016 - Highly Available Node.js
ThatConference 2016 - Highly Available Node.js
Brad Williams
 
Resilient microservices
Resilient microservicesResilient microservices
Resilient microservices
Maxim Shelest
 
Technology insights: Decision Science Platform
Technology insights: Decision Science PlatformTechnology insights: Decision Science Platform
Technology insights: Decision Science Platform
Decision Science Community
 

Similar to Building and Monitoring Services at Lithium (20)

Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)
 
Webinar : Microservices and Containerization
Webinar : Microservices and ContainerizationWebinar : Microservices and Containerization
Webinar : Microservices and Containerization
 
Evolving to Cloud-Native - Nate Schutta 2/2
Evolving to Cloud-Native - Nate Schutta 2/2Evolving to Cloud-Native - Nate Schutta 2/2
Evolving to Cloud-Native - Nate Schutta 2/2
 
Application Modernisation through Event-Driven Microservices
Application Modernisation through Event-Driven Microservices Application Modernisation through Event-Driven Microservices
Application Modernisation through Event-Driven Microservices
 
Microservices Architecture, Monolith Migration Patterns
Microservices Architecture, Monolith Migration PatternsMicroservices Architecture, Monolith Migration Patterns
Microservices Architecture, Monolith Migration Patterns
 
Securing a Cloud Migration
Securing a Cloud MigrationSecuring a Cloud Migration
Securing a Cloud Migration
 
Securing a Cloud Migration
Securing a Cloud MigrationSecuring a Cloud Migration
Securing a Cloud Migration
 
How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...
How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...
How Cisco is Leveraging MuleSoft to Drive Continuous Innovation​ at Enterpris...
 
The Future of Cloud Innovation, featuring Adrian Cockcroft
The Future of Cloud Innovation, featuring Adrian CockcroftThe Future of Cloud Innovation, featuring Adrian Cockcroft
The Future of Cloud Innovation, featuring Adrian Cockcroft
 
Think Small To Go Big - Introduction To Microservices
Think Small To Go Big - Introduction To MicroservicesThink Small To Go Big - Introduction To Microservices
Think Small To Go Big - Introduction To Microservices
 
Introduction to Serverless through Architectural Patterns
Introduction to Serverless through Architectural PatternsIntroduction to Serverless through Architectural Patterns
Introduction to Serverless through Architectural Patterns
 
Availability in a cloud native world v1.6 (Feb 2019)
Availability in a cloud native world v1.6 (Feb 2019)Availability in a cloud native world v1.6 (Feb 2019)
Availability in a cloud native world v1.6 (Feb 2019)
 
From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018
 
Cloud-Native Fundamentals: An Introduction to 12-Factor Applications
Cloud-Native Fundamentals: An Introduction to 12-Factor ApplicationsCloud-Native Fundamentals: An Introduction to 12-Factor Applications
Cloud-Native Fundamentals: An Introduction to 12-Factor Applications
 
We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?We are drowning in complexity—can we do better?
We are drowning in complexity—can we do better?
 
APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?
APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?
APIDays 2018 - APIOps & Microservices - What is MICRO by the Way ?
 
Cloud computing for microprocessor tools
Cloud computing for microprocessor toolsCloud computing for microprocessor tools
Cloud computing for microprocessor tools
 
ThatConference 2016 - Highly Available Node.js
ThatConference 2016 - Highly Available Node.jsThatConference 2016 - Highly Available Node.js
ThatConference 2016 - Highly Available Node.js
 
Resilient microservices
Resilient microservicesResilient microservices
Resilient microservices
 
Technology insights: Decision Science Platform
Technology insights: Decision Science PlatformTechnology insights: Decision Science Platform
Technology insights: Decision Science Platform
 

Recently uploaded

BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
Neo4j
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
ssuser1915fe1
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
chetankumar9855
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
WhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdf
WhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdfWhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdf
WhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdf
ArgaBisma
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
ishalveerrandhawa1
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
SynapseIndia
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
ChristopherTHyatt
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Zilliz
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
Lidia A.
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
Emerging Tech
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
Edge AI and Vision Alliance
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 

Recently uploaded (20)

BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdfBT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
BT & Neo4j: Knowledge Graphs for Critical Enterprise Systems.pptx.pdf
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Feature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptxFeature sql server terbaru performance.pptx
Feature sql server terbaru performance.pptx
 
Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...Amul milk launches in US: Key details of its new products ...
Amul milk launches in US: Key details of its new products ...
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
WhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdf
WhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdfWhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdf
WhatsApp Image 2024-03-27 at 08.19.52_bfd93109.pdf
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
Calgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptxCalgary MuleSoft Meetup APM and IDP .pptx
Calgary MuleSoft Meetup APM and IDP .pptx
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
How RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptxHow RPA Help in the Transportation and Logistics Industry.pptx
How RPA Help in the Transportation and Logistics Industry.pptx
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and OllamaTirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
Tirana Tech Meetup - Agentic RAG with Milvus, Llama3 and Ollama
 
WPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide DeckWPRiders Company Presentation Slide Deck
WPRiders Company Presentation Slide Deck
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Implementations of Fused Deposition Modeling in real world
Implementations of Fused Deposition Modeling  in real worldImplementations of Fused Deposition Modeling  in real world
Implementations of Fused Deposition Modeling in real world
 
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 

Building and Monitoring Services at Lithium

Editor's Notes

  1. “these are the technologies we use at lithium, you can see that we use a few different technologies for storing data...mostly for different use cases (i.e., batch vs realtime vs transactional). On top of all these data-storage technologies we are also building up services as we move towards a service oriented architecture and horizontally scalable, highly-available services. As we move towards SOA, and more importantly towards cloud the design space changes and we must deal with failure more realistically......transition, failure is constant”.
  2. - As dependencies for fulfilling a request goes up, so does the probability of failure, which is fine, we just can’t have cascading failure.
  3. - Everyone has now heard of netflix simian army for simulating failure, but we don’t always talk about the coding practices to be able to withstand its wrath.
  4. This is not really a problem of “cloud” this is a problem associated with building distributed, horizontally scalable applications.The only time you don’t have to worry about these things is if your chosen method of scaling is “up” and not “out”, but event then, does it connect to users (i.e., the wider-system context is distributed).In the past generation of “scale-up” failure meant everything was dead, now it just means functionality gets degraded.
  5. - Especially on network calls, this is about protecting the client
  6. - Also about protecting the client- See hystrix from netflix, this is about protecting the client and allowing the downstream service to heal
  7. - Beware that it is harder to reason about anything async.
  8. - This is about signaling to upstream traffic that something is wrong downstream and they may want to take evasive action.
  9. Or at least fail-over.Easy in cloud, harder in datacenter
  10. - They should be explicit, how is this app going to deal with failure in these dependencies (always from the client-side perspective).
  11. - Now that your apps have all of the previous concepts built in bad stuff will still happen. How do you manage the service in production to know when things are going wrong?
  12. Find the most critical calls in your app (i.e., network calls, client calls)Figure out a way to visualize them to gain instant awareness as to what is going wrong in prodService should be small enough for a single engineer to gain full insight (assuming he has the baseline).
  13. - Create alerts around specific log levels (ERROR) or system usage outside of a well-known baseline.- An alert should mean that something needs immediate attention (i.e., keep noise to a minimum)- Alerts should be a last resort (because sometimes you need to sleep).- Alerts should not be a substitute for continuous monitoring of the service through dashboards.
  14. Or at least fail-over.Easy in cloud, harder in datacenter