SlideShare a Scribd company logo
Building reliable, scalable services with
Clojure and Core.async
Kapil Reddy - Helpshift
Building reliable, scalable services with
Clojure and Core.async
Kapil Reddy - Helpshift
About me - Kapil
Staff Engineer @ Helpshift
Clojure
Distributed Systems
Games
Music
Books/Comics
Football
Helpshift is a Mobile CRM SaaS product. We help connect app developers with their customers. Since everything is now on mobile.
• 600M+ MAU
• 60k RPS
• 500GB / day
Scale
These are some of the scale numbers we have reached at Helpshift.
Reliable
• Fail fast
• Detect non-recoverable errors
• Resilient
• Retries recoverable errors
• Backpressure
• Detect degraded state
At this scale services need to be reliable. We need to have exact control on how things behave under failure conditions.
Let’s take a look at the problem. We will build the solution to the problem iteratively once we understand the scope 

We do a lot of writes to ElasticSearch but those writes can be done asynchronously. So application servers just push updates to a Kafka topic. We need write a Kafka
consumer that reads from the topic and performs writes to ElasticSearch. But wait! Elasticsearch has bulk api. So we need to write a Kafka consumer that bulk writes to
Elasticsearch.
Testing becomes simpler. It’s just putting things in channel and verifying FSM state

Test generate signals and data. Assertion is checking what state FSM goes into.
Scale
• V1 - 150 rps
• Today - 5k rps
Scaling a reliable service which recovers from error states is very simple. It’s basically Kafka consumer that can handle all the happy and unhappy paths. Scaling it means
just adding more instances of these services or FSMs in the same service.
Extensibility / Maintainability
• First version - 5 weeks
• MongoDB - 2 weeks
• Active maintainer - 1 engineer - 20% time
• In Production - 2 years
• LOC - 2k
• Project as a library
Summary
• Reliable system == Predictable failures and happy
paths
• Use CSP / core.async to decouple components
• Central FSM that receives all data and control
signals to take decisions
Thank you!
@kapilreddy

More Related Content

What's hot

AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...
AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...
AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...
AWS User Group - Thailand
 
AWS - Lambda Fundamentals
AWS - Lambda FundamentalsAWS - Lambda Fundamentals
AWS - Lambda Fundamentals
Piyush Agrawal
 
Shipping apps to eks with code pipeline and lambda functions
Shipping apps to eks with code pipeline and lambda functionsShipping apps to eks with code pipeline and lambda functions
Shipping apps to eks with code pipeline and lambda functions
Štěpán Vraný
 
Integrating API Gateway and internal ELB
Integrating API Gateway and internal ELBIntegrating API Gateway and internal ELB
Integrating API Gateway and internal ELB
Juan Diaz
 
AWS Community Day Bangkok 2019 - Hello ClaudiaJS
AWS Community Day Bangkok 2019 - Hello ClaudiaJSAWS Community Day Bangkok 2019 - Hello ClaudiaJS
AWS Community Day Bangkok 2019 - Hello ClaudiaJS
AWS User Group - Thailand
 
Choosing the right messaging service for your serverless app [with lumigo]
Choosing the right messaging service for your serverless app [with lumigo]Choosing the right messaging service for your serverless app [with lumigo]
Choosing the right messaging service for your serverless app [with lumigo]
Dhaval Nagar
 
Azure functions
Azure functionsAzure functions
Azure functions
vivek p s
 
Ben Kehoe - Serverless Architecture for the Internet of Things
Ben Kehoe - Serverless Architecture for the Internet of ThingsBen Kehoe - Serverless Architecture for the Internet of Things
Ben Kehoe - Serverless Architecture for the Internet of Things
ServerlessConf
 
Is Serverless The New Swiss Cheese?
Is Serverless The New Swiss Cheese?Is Serverless The New Swiss Cheese?
Is Serverless The New Swiss Cheese?
Chase Douglas
 
Serverless Code Deployments in AWS
Serverless Code Deployments in AWSServerless Code Deployments in AWS
Serverless Code Deployments in AWS
Marko Tomic
 
Managing the Continuous Delivery of Code to AWS Lambda
Managing the Continuous Delivery of Code to AWS LambdaManaging the Continuous Delivery of Code to AWS Lambda
Managing the Continuous Delivery of Code to AWS Lambda
Amazon Web Services
 
AWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 mins
AWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 minsAWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 mins
AWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 mins
AWS User Group - Thailand
 
AWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase Productivity
AWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase ProductivityAWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase Productivity
AWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase Productivity
AWS User Group - Thailand
 
The guardian and app engine
The guardian and app engineThe guardian and app engine
The guardian and app engine
Michael Brunton-Spall
 
AWS Lambda
AWS LambdaAWS Lambda
AWS Lambda
Muhammed YALÇIN
 
AWS Jungle - Lambda
AWS Jungle - LambdaAWS Jungle - Lambda
AWS Jungle - Lambda
Squash Apps Pvt Ltd
 
DevOps for the Enterprise: Continuous Integration
DevOps for the Enterprise: Continuous IntegrationDevOps for the Enterprise: Continuous Integration
DevOps for the Enterprise: Continuous Integration
Amazon Web Services
 
Serverless by examples and case studies
Serverless by examples and case studiesServerless by examples and case studies
Serverless by examples and case studies
CodeOps Technologies LLP
 
An Introduction to Microservices
An Introduction to MicroservicesAn Introduction to Microservices
An Introduction to Microservices
Ad van der Veer
 
Artificial Intelligence & Machine learning foundation topic in AWS
Artificial Intelligence & Machine learning foundation topic in AWS Artificial Intelligence & Machine learning foundation topic in AWS
Artificial Intelligence & Machine learning foundation topic in AWS
Varun Manik
 

What's hot (20)

AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...
AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...
AWS Community Day Bangkok 2019 - DevOps Cost Reduction using Jenkins & AWS Sp...
 
AWS - Lambda Fundamentals
AWS - Lambda FundamentalsAWS - Lambda Fundamentals
AWS - Lambda Fundamentals
 
Shipping apps to eks with code pipeline and lambda functions
Shipping apps to eks with code pipeline and lambda functionsShipping apps to eks with code pipeline and lambda functions
Shipping apps to eks with code pipeline and lambda functions
 
Integrating API Gateway and internal ELB
Integrating API Gateway and internal ELBIntegrating API Gateway and internal ELB
Integrating API Gateway and internal ELB
 
AWS Community Day Bangkok 2019 - Hello ClaudiaJS
AWS Community Day Bangkok 2019 - Hello ClaudiaJSAWS Community Day Bangkok 2019 - Hello ClaudiaJS
AWS Community Day Bangkok 2019 - Hello ClaudiaJS
 
Choosing the right messaging service for your serverless app [with lumigo]
Choosing the right messaging service for your serverless app [with lumigo]Choosing the right messaging service for your serverless app [with lumigo]
Choosing the right messaging service for your serverless app [with lumigo]
 
Azure functions
Azure functionsAzure functions
Azure functions
 
Ben Kehoe - Serverless Architecture for the Internet of Things
Ben Kehoe - Serverless Architecture for the Internet of ThingsBen Kehoe - Serverless Architecture for the Internet of Things
Ben Kehoe - Serverless Architecture for the Internet of Things
 
Is Serverless The New Swiss Cheese?
Is Serverless The New Swiss Cheese?Is Serverless The New Swiss Cheese?
Is Serverless The New Swiss Cheese?
 
Serverless Code Deployments in AWS
Serverless Code Deployments in AWSServerless Code Deployments in AWS
Serverless Code Deployments in AWS
 
Managing the Continuous Delivery of Code to AWS Lambda
Managing the Continuous Delivery of Code to AWS LambdaManaging the Continuous Delivery of Code to AWS Lambda
Managing the Continuous Delivery of Code to AWS Lambda
 
AWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 mins
AWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 minsAWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 mins
AWS Community Day Bangkok 2019 - Build a Serverless Web Application in 30 mins
 
AWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase Productivity
AWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase ProductivityAWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase Productivity
AWS Community Day Bangkok 2019 - Dev Ops Philosophy Increase Productivity
 
The guardian and app engine
The guardian and app engineThe guardian and app engine
The guardian and app engine
 
AWS Lambda
AWS LambdaAWS Lambda
AWS Lambda
 
AWS Jungle - Lambda
AWS Jungle - LambdaAWS Jungle - Lambda
AWS Jungle - Lambda
 
DevOps for the Enterprise: Continuous Integration
DevOps for the Enterprise: Continuous IntegrationDevOps for the Enterprise: Continuous Integration
DevOps for the Enterprise: Continuous Integration
 
Serverless by examples and case studies
Serverless by examples and case studiesServerless by examples and case studies
Serverless by examples and case studies
 
An Introduction to Microservices
An Introduction to MicroservicesAn Introduction to Microservices
An Introduction to Microservices
 
Artificial Intelligence & Machine learning foundation topic in AWS
Artificial Intelligence & Machine learning foundation topic in AWS Artificial Intelligence & Machine learning foundation topic in AWS
Artificial Intelligence & Machine learning foundation topic in AWS
 

Similar to Building a reliable, scalable service with Clojure and Core.async

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
HostedbyConfluent
 
Building and Scaling a WebSockets Pubsub System
Building and Scaling a WebSockets Pubsub SystemBuilding and Scaling a WebSockets Pubsub System
Building and Scaling a WebSockets Pubsub System
Kapil Reddy
 
Five Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data ApplicationsFive Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data Applications
Lightbend
 
Using Kubernetes to deliver a “serverless” service
Using Kubernetes to deliver a “serverless” serviceUsing Kubernetes to deliver a “serverless” service
Using Kubernetes to deliver a “serverless” service
DoKC
 
AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)
Amazon Web Services
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million Users
Amazon Web Services
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million Users
Amazon Web Services
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
confluent
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
Monal Daxini
 
Testing at Stream-Scale
Testing at Stream-ScaleTesting at Stream-Scale
Testing at Stream-Scale
All Things Open
 
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
confluent
 
Building Asynchronous Applications
Building Asynchronous ApplicationsBuilding Asynchronous Applications
Building Asynchronous Applications
Johan Edstrom
 
AWS Startup Webinar | Developing on AWS
AWS Startup Webinar | Developing on AWSAWS Startup Webinar | Developing on AWS
AWS Startup Webinar | Developing on AWS
Amazon Web Services
 
Apinizer - Full API Lifecycle and Integration Platform
Apinizer - Full API Lifecycle and Integration Platform Apinizer - Full API Lifecycle and Integration Platform
Apinizer - Full API Lifecycle and Integration Platform
Mustafa Yildiz
 
Koby holzer live person-openstack_story
Koby holzer   live person-openstack_storyKoby holzer   live person-openstack_story
Koby holzer live person-openstack_story
Cloud Native Day Tel Aviv
 
ENT309 scaling up to your first 10 million users
ENT309 scaling up to your first 10 million usersENT309 scaling up to your first 10 million users
ENT309 scaling up to your first 10 million users
Amazon Web Services
 
Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !!
Couchbase Singapore Meetup #2:  Why Developing with Couchbase is easy !! Couchbase Singapore Meetup #2:  Why Developing with Couchbase is easy !!
Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !!
Karthik Babu Sekar
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million Users
Amazon Web Services
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEAData Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEA
Andrew Morgan
 
Webinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDBWebinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDB
MongoDB
 

Similar to Building a reliable, scalable service with Clojure and Core.async (20)

Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
 
Building and Scaling a WebSockets Pubsub System
Building and Scaling a WebSockets Pubsub SystemBuilding and Scaling a WebSockets Pubsub System
Building and Scaling a WebSockets Pubsub System
 
Five Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data ApplicationsFive Early Challenges Of Building Streaming Fast Data Applications
Five Early Challenges Of Building Streaming Fast Data Applications
 
Using Kubernetes to deliver a “serverless” service
Using Kubernetes to deliver a “serverless” serviceUsing Kubernetes to deliver a “serverless” service
Using Kubernetes to deliver a “serverless” service
 
AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)AWS re:Invent 2016: AWS Database State of the Union (DAT320)
AWS re:Invent 2016: AWS Database State of the Union (DAT320)
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million Users
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million Users
 
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with K...
 
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017
 
Testing at Stream-Scale
Testing at Stream-ScaleTesting at Stream-Scale
Testing at Stream-Scale
 
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019 Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
 
Building Asynchronous Applications
Building Asynchronous ApplicationsBuilding Asynchronous Applications
Building Asynchronous Applications
 
AWS Startup Webinar | Developing on AWS
AWS Startup Webinar | Developing on AWSAWS Startup Webinar | Developing on AWS
AWS Startup Webinar | Developing on AWS
 
Apinizer - Full API Lifecycle and Integration Platform
Apinizer - Full API Lifecycle and Integration Platform Apinizer - Full API Lifecycle and Integration Platform
Apinizer - Full API Lifecycle and Integration Platform
 
Koby holzer live person-openstack_story
Koby holzer   live person-openstack_storyKoby holzer   live person-openstack_story
Koby holzer live person-openstack_story
 
ENT309 scaling up to your first 10 million users
ENT309 scaling up to your first 10 million usersENT309 scaling up to your first 10 million users
ENT309 scaling up to your first 10 million users
 
Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !!
Couchbase Singapore Meetup #2:  Why Developing with Couchbase is easy !! Couchbase Singapore Meetup #2:  Why Developing with Couchbase is easy !!
Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !!
 
ENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million UsersENT309 Scaling Up to Your First 10 Million Users
ENT309 Scaling Up to Your First 10 Million Users
 
Data Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEAData Streaming with Apache Kafka & MongoDB - EMEA
Data Streaming with Apache Kafka & MongoDB - EMEA
 
Webinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDBWebinar: Data Streaming with Apache Kafka & MongoDB
Webinar: Data Streaming with Apache Kafka & MongoDB
 

Recently uploaded

一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
dakas1
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Green Software Development
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
TaghreedAltamimi
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
gapen1
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
Grant Fritchey
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
mz5nrf0n
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
Peter Muessig
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
Rakesh Kumar R
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Peter Muessig
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
Patrick Weigel
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 
YAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring detailsYAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring details
NishanthaBulumulla1
 

Recently uploaded (20)

一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, FactsALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
 
Lecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptxLecture 2 - software testing SE 412.pptx
Lecture 2 - software testing SE 412.pptx
 
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
如何办理(hull学位证书)英国赫尔大学毕业证硕士文凭原版一模一样
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 
Using Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query PerformanceUsing Query Store in Azure PostgreSQL to Understand Query Performance
Using Query Store in Azure PostgreSQL to Understand Query Performance
 
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
在线购买加拿大英属哥伦比亚大学毕业证本科学位证书原版一模一样
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
UI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design SystemUI5con 2024 - Bring Your Own Design System
UI5con 2024 - Bring Your Own Design System
 
What next after learning python programming basics
What next after learning python programming basicsWhat next after learning python programming basics
What next after learning python programming basics
 
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemUI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 
YAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring detailsYAML crash COURSE how to write yaml file for adding configuring details
YAML crash COURSE how to write yaml file for adding configuring details
 

Building a reliable, scalable service with Clojure and Core.async

  • 1. Building reliable, scalable services with Clojure and Core.async Kapil Reddy - Helpshift
  • 2. Building reliable, scalable services with Clojure and Core.async Kapil Reddy - Helpshift
  • 3. About me - Kapil Staff Engineer @ Helpshift Clojure Distributed Systems Games Music Books/Comics Football
  • 4. Helpshift is a Mobile CRM SaaS product. We help connect app developers with their customers. Since everything is now on mobile.
  • 5. • 600M+ MAU • 60k RPS • 500GB / day Scale These are some of the scale numbers we have reached at Helpshift.
  • 6. Reliable • Fail fast • Detect non-recoverable errors • Resilient • Retries recoverable errors • Backpressure • Detect degraded state At this scale services need to be reliable. We need to have exact control on how things behave under failure conditions.
  • 7. Let’s take a look at the problem. We will build the solution to the problem iteratively once we understand the scope We do a lot of writes to ElasticSearch but those writes can be done asynchronously. So application servers just push updates to a Kafka topic. We need write a Kafka consumer that reads from the topic and performs writes to ElasticSearch. But wait! Elasticsearch has bulk api. So we need to write a Kafka consumer that bulk writes to Elasticsearch.
  • 8. Testing becomes simpler. It’s just putting things in channel and verifying FSM state Test generate signals and data. Assertion is checking what state FSM goes into.
  • 9. Scale • V1 - 150 rps • Today - 5k rps Scaling a reliable service which recovers from error states is very simple. It’s basically Kafka consumer that can handle all the happy and unhappy paths. Scaling it means just adding more instances of these services or FSMs in the same service.
  • 10. Extensibility / Maintainability • First version - 5 weeks • MongoDB - 2 weeks • Active maintainer - 1 engineer - 20% time • In Production - 2 years • LOC - 2k • Project as a library
  • 11. Summary • Reliable system == Predictable failures and happy paths • Use CSP / core.async to decouple components • Central FSM that receives all data and control signals to take decisions