SlideShare a Scribd company logo
1 of 22
Scalable Service Architectures
Lessons learned
Zoltán Németh
Engineering Manager, Core Systems
Agenda
 Our scalability experience
 What is Scalability?
 Requirements in detail
 Tips and tools
 Extras, Closing remarks
Our experience
Streaming stack
Defining scalability
Scalability is the ability to handle increased workload
by repeatedly applying a costeffective strategy for
extending a system’s capacity.
(CMU paper, 2006)
How well a solution to some problem will work when
the size of the problem increases. When the size
decreases, the solution must fit. (dictionary.com and
Theo Schlossnagle, 2006)
Self-contained
service
 Explicitly declare and
isolate dependencies
 Isolation from the outside
system
 Static linking
 Do not rely on system
packages
Disposability  Maximize robustness with
fast startup and graceful
shutdown
 Disposable processes
 Graceful shutdown on
SIGTERM
 Handling sudden death:
robust queue backend
Startup and
Shutdown
 Automate all the things
 Chef
 Docker
 Gold image based
deployment
 Immutable
 Handling tasks before
shutdown
Backing Services  Treat backing services as
attached resources
 No distinction between
local and third party
services
 Easily swap out resources
 Export services via port
binding
 Become the backing
service for another app
Processes,
concurrency
 Stateless processes (not
even sticky sessions)
 Process types by work type
 We <3 linux process
 Shared-nothing  adding
concurrency is safe
 Process distribution
spanning machines
Statelessness  Store everything in a
datastore
 Aggregate data
 Chandra
 Scalable datastores
 Redis
 Cassandra
 Aerospike
 Handling user sessions
Monitoring  Application state and
metrics
 Dashboards
 Alerting
 Health
 Remove failing nodes
 Capacity
 Act on trends
Monitoring  Metrics collecting
 Graphite, New Relic
 Self-aware checks
 Cluster state
 Zookeeper, Consul
 Scaling decision types
 Capacity amount
 Graph derivative
 App requests
Load Balance and
Resource
Allocation
 Load Balance: distribute
tasks
 Utilize machines
efficiently
 VM compatible apps
 Flexibility
 Adapting to available
resources
Load Balance  DNS or API
 App level balance
 Uniform entry point or
proxy
 Balance decisions
 Load
 Zookeeper state
 Resource policies
Service
Separation
 Failure is inevitable
 Protect from failing
components
 Cascading failure
 Fail fast
 Decoupling
 Asynchronous operations
 Message queues
Service
Separation
 Rate limiting
 Circuit Breaker pattern
 Stop cascading failure,
allow recovery
 Hystrix
 Fail fast, fail silent
 Service decoupling
Extras  Debugging features
 Logs
 Clojure / JS consoles
 Runtime configuration
via env
 Scaling API
 Integrating several
cloud providers
 Automatic start / stop
Reading
 Scalable Internet Architectures by Theo Schlossnagle
 The 12-factor App: http://12factor.net/
 Carnegie Mellon Paper: http://www.sei.cmu.edu/reports/06tn012.pdf
 Circuit Breaker: http://martinfowler.com/bliki/CircuitBreaker.html
 Release It! by Michael T. Nygard
Questions
syntaxerror@ustream.tv

More Related Content

What's hot

"What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a..."What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a...Fwdays
 
Scaling Application
Scaling ApplicationScaling Application
Scaling ApplicationAlaor Bianco
 
Building Cloud Apps using Azure SQL Database
Building Cloud Apps using Azure SQL DatabaseBuilding Cloud Apps using Azure SQL Database
Building Cloud Apps using Azure SQL DatabaseWinWire Technologies Inc
 
Smolen Alex Securing The Mvc Architecture Part Two
Smolen Alex Securing The Mvc Architecture Part TwoSmolen Alex Securing The Mvc Architecture Part Two
Smolen Alex Securing The Mvc Architecture Part Twoalsmola
 
Whitepaper : Building an Efficient Microservices Architecture
Whitepaper : Building an Efficient Microservices ArchitectureWhitepaper : Building an Efficient Microservices Architecture
Whitepaper : Building an Efficient Microservices ArchitectureNewt Global Consulting LLC
 
Azure AWS real time-interview questions part 9
Azure AWS real time-interview questions part 9Azure AWS real time-interview questions part 9
Azure AWS real time-interview questions part 9Malleswar Reddy
 
Designing distributed systems
Designing distributed systemsDesigning distributed systems
Designing distributed systemsMalisa Ncube
 
Elastic APM: Combinalo con tus logs y métricas para una visibilidad completa
Elastic APM: Combinalo con tus logs y métricas para una visibilidad completaElastic APM: Combinalo con tus logs y métricas para una visibilidad completa
Elastic APM: Combinalo con tus logs y métricas para una visibilidad completaElasticsearch
 
Elastic APM: Amplía tus logs y métricas para ver el panorama completo
Elastic APM: Amplía tus logs y métricas para ver el panorama completoElastic APM: Amplía tus logs y métricas para ver el panorama completo
Elastic APM: Amplía tus logs y métricas para ver el panorama completoElasticsearch
 
eSobi Site Initiation
eSobi Site InitiationeSobi Site Initiation
eSobi Site InitiationAllan Huang
 

What's hot (10)

"What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a..."What database can tell about application issues? What application can tell a...
"What database can tell about application issues? What application can tell a...
 
Scaling Application
Scaling ApplicationScaling Application
Scaling Application
 
Building Cloud Apps using Azure SQL Database
Building Cloud Apps using Azure SQL DatabaseBuilding Cloud Apps using Azure SQL Database
Building Cloud Apps using Azure SQL Database
 
Smolen Alex Securing The Mvc Architecture Part Two
Smolen Alex Securing The Mvc Architecture Part TwoSmolen Alex Securing The Mvc Architecture Part Two
Smolen Alex Securing The Mvc Architecture Part Two
 
Whitepaper : Building an Efficient Microservices Architecture
Whitepaper : Building an Efficient Microservices ArchitectureWhitepaper : Building an Efficient Microservices Architecture
Whitepaper : Building an Efficient Microservices Architecture
 
Azure AWS real time-interview questions part 9
Azure AWS real time-interview questions part 9Azure AWS real time-interview questions part 9
Azure AWS real time-interview questions part 9
 
Designing distributed systems
Designing distributed systemsDesigning distributed systems
Designing distributed systems
 
Elastic APM: Combinalo con tus logs y métricas para una visibilidad completa
Elastic APM: Combinalo con tus logs y métricas para una visibilidad completaElastic APM: Combinalo con tus logs y métricas para una visibilidad completa
Elastic APM: Combinalo con tus logs y métricas para una visibilidad completa
 
Elastic APM: Amplía tus logs y métricas para ver el panorama completo
Elastic APM: Amplía tus logs y métricas para ver el panorama completoElastic APM: Amplía tus logs y métricas para ver el panorama completo
Elastic APM: Amplía tus logs y métricas para ver el panorama completo
 
eSobi Site Initiation
eSobi Site InitiationeSobi Site Initiation
eSobi Site Initiation
 

Similar to Scalable Service Architectures

Scalable service architectures @ VDB16
Scalable service architectures @ VDB16Scalable service architectures @ VDB16
Scalable service architectures @ VDB16Zoltán Németh
 
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...MSDEVMTL
 
Microsoft Azure Cloud Basics Tutorial
Microsoft Azure Cloud Basics TutorialMicrosoft Azure Cloud Basics Tutorial
Microsoft Azure Cloud Basics TutorialIIMSE Edu
 
(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...
(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...
(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...Amazon Web Services
 
Muves3 Elastic Grid Java One2009 Final
Muves3 Elastic Grid Java One2009 FinalMuves3 Elastic Grid Java One2009 Final
Muves3 Elastic Grid Java One2009 FinalElastic Grid, LLC.
 
How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...Amazon Web Services
 
Eda on the azure services platform
Eda on the azure services platformEda on the azure services platform
Eda on the azure services platformYves Goeleven
 
A guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update ConferenceA guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update ConferenceEldert Grootenboer
 
CSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps sessionCSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps sessionTom Laszewski
 
Windows Azure - Uma Plataforma para o Desenvolvimento de Aplicações
Windows Azure - Uma Plataforma para o Desenvolvimento de AplicaçõesWindows Azure - Uma Plataforma para o Desenvolvimento de Aplicações
Windows Azure - Uma Plataforma para o Desenvolvimento de AplicaçõesComunidade NetPonto
 
Kluczowe elementy infrastruktury...
Kluczowe elementy infrastruktury...Kluczowe elementy infrastruktury...
Kluczowe elementy infrastruktury...Alicja Sieminska
 
20141021 AWS Cloud Taekwon - Startup Best Practices on AWS
20141021 AWS Cloud Taekwon - Startup Best Practices on AWS20141021 AWS Cloud Taekwon - Startup Best Practices on AWS
20141021 AWS Cloud Taekwon - Startup Best Practices on AWSAmazon Web Services Korea
 
Albara Abdalkhalig
Albara AbdalkhaligAlbara Abdalkhalig
Albara Abdalkhaligbrra51
 
Current trends in software engineering
Current trends in software engineeringCurrent trends in software engineering
Current trends in software engineeringbrra51
 

Similar to Scalable Service Architectures (20)

Scalable service architectures @ VDB16
Scalable service architectures @ VDB16Scalable service architectures @ VDB16
Scalable service architectures @ VDB16
 
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
 
Microsoft Azure Cloud Basics Tutorial
Microsoft Azure Cloud Basics TutorialMicrosoft Azure Cloud Basics Tutorial
Microsoft Azure Cloud Basics Tutorial
 
Azure and cloud design patterns
Azure and cloud design patternsAzure and cloud design patterns
Azure and cloud design patterns
 
(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...
(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...
(ENT210) Accelerating Business Innovation with DevOps on AWS | AWS re:Invent ...
 
Muves3 Elastic Grid Java One2009 Final
Muves3 Elastic Grid Java One2009 FinalMuves3 Elastic Grid Java One2009 Final
Muves3 Elastic Grid Java One2009 Final
 
Managing the cloud
Managing the cloudManaging the cloud
Managing the cloud
 
Sql Azure
Sql AzureSql Azure
Sql Azure
 
How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...How a National Transportation Software Provider Migrated a Mission-Critical T...
How a National Transportation Software Provider Migrated a Mission-Critical T...
 
Eda on the azure services platform
Eda on the azure services platformEda on the azure services platform
Eda on the azure services platform
 
A guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update ConferenceA guide through the Azure Messaging services - Update Conference
A guide through the Azure Messaging services - Update Conference
 
CSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps sessionCSC AWS re:Invent Enterprise DevOps session
CSC AWS re:Invent Enterprise DevOps session
 
Introduction To Cloud Computing
Introduction To Cloud ComputingIntroduction To Cloud Computing
Introduction To Cloud Computing
 
Microsoft: Invent with Purpose
Microsoft: Invent with PurposeMicrosoft: Invent with Purpose
Microsoft: Invent with Purpose
 
12-Factor App
12-Factor App12-Factor App
12-Factor App
 
Windows Azure - Uma Plataforma para o Desenvolvimento de Aplicações
Windows Azure - Uma Plataforma para o Desenvolvimento de AplicaçõesWindows Azure - Uma Plataforma para o Desenvolvimento de Aplicações
Windows Azure - Uma Plataforma para o Desenvolvimento de Aplicações
 
Kluczowe elementy infrastruktury...
Kluczowe elementy infrastruktury...Kluczowe elementy infrastruktury...
Kluczowe elementy infrastruktury...
 
20141021 AWS Cloud Taekwon - Startup Best Practices on AWS
20141021 AWS Cloud Taekwon - Startup Best Practices on AWS20141021 AWS Cloud Taekwon - Startup Best Practices on AWS
20141021 AWS Cloud Taekwon - Startup Best Practices on AWS
 
Albara Abdalkhalig
Albara AbdalkhaligAlbara Abdalkhalig
Albara Abdalkhalig
 
Current trends in software engineering
Current trends in software engineeringCurrent trends in software engineering
Current trends in software engineering
 

More from Zoltán Németh

Reveal The Secrets of Your Videos
Reveal The Secrets of Your VideosReveal The Secrets of Your Videos
Reveal The Secrets of Your VideosZoltán Németh
 
Voxxed Days Belgrade 2017 - How not to do DevOps
Voxxed Days Belgrade 2017 - How not to do DevOpsVoxxed Days Belgrade 2017 - How not to do DevOps
Voxxed Days Belgrade 2017 - How not to do DevOpsZoltán Németh
 
Content protection with UMS
Content protection with UMSContent protection with UMS
Content protection with UMSZoltán Németh
 
Implementing DevOps In Practice
Implementing DevOps In PracticeImplementing DevOps In Practice
Implementing DevOps In PracticeZoltán Németh
 
On-demand real time transcoding
On-demand real time transcoding On-demand real time transcoding
On-demand real time transcoding Zoltán Németh
 
DB séma kezelés Liquibase-el
DB séma kezelés Liquibase-elDB séma kezelés Liquibase-el
DB séma kezelés Liquibase-elZoltán Németh
 

More from Zoltán Németh (9)

Reveal The Secrets of Your Videos
Reveal The Secrets of Your VideosReveal The Secrets of Your Videos
Reveal The Secrets of Your Videos
 
Voxxed Days Belgrade 2017 - How not to do DevOps
Voxxed Days Belgrade 2017 - How not to do DevOpsVoxxed Days Belgrade 2017 - How not to do DevOps
Voxxed Days Belgrade 2017 - How not to do DevOps
 
Content protection with UMS
Content protection with UMSContent protection with UMS
Content protection with UMS
 
Implementing DevOps In Practice
Implementing DevOps In PracticeImplementing DevOps In Practice
Implementing DevOps In Practice
 
Building our own CDN
Building our own CDNBuilding our own CDN
Building our own CDN
 
Culture @ Velocity UK
Culture @ Velocity UKCulture @ Velocity UK
Culture @ Velocity UK
 
On-demand real time transcoding
On-demand real time transcoding On-demand real time transcoding
On-demand real time transcoding
 
DB séma kezelés Liquibase-el
DB séma kezelés Liquibase-elDB séma kezelés Liquibase-el
DB séma kezelés Liquibase-el
 
Daemons in PHP
Daemons in PHPDaemons in PHP
Daemons in PHP
 

Recently uploaded

How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?Linksys Velop Login
 
Topology of the Network class 8 .ppt pdf
Topology of the Network class 8 .ppt pdfTopology of the Network class 8 .ppt pdf
Topology of the Network class 8 .ppt pdfAnushkaTripathi61
 
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital PresenceCyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital PresencePC Doctors NET
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxChloeMeadows1
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appscristianmanaila2
 
Bug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's GuideBug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's GuideVarun Mithran
 
Pvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdfPvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdfPvtaan
 
Thank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirtsThank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirtsrahman018755
 
Statistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdfStatistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdfOndejSur
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkklolsDocherty
 
Premier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdfPremier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdfappinfoedgeca
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirtrahman018755
 
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.Tortogel
 
Reggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirtsrahman018755
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebJie Liau
 
The Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case StudyThe Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case StudyDamar Juniarto
 

Recently uploaded (16)

How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?How Do I Begin the Linksys Velop Setup Process?
How Do I Begin the Linksys Velop Setup Process?
 
Topology of the Network class 8 .ppt pdf
Topology of the Network class 8 .ppt pdfTopology of the Network class 8 .ppt pdf
Topology of the Network class 8 .ppt pdf
 
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital PresenceCyber Security Services Unveiled: Strategies to Secure Your Digital Presence
Cyber Security Services Unveiled: Strategies to Secure Your Digital Presence
 
Production 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptxProduction 2024 sunderland culture final - Copy.pptx
Production 2024 sunderland culture final - Copy.pptx
 
Development Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of appsDevelopment Lifecycle.pptx for the secure development of apps
Development Lifecycle.pptx for the secure development of apps
 
Bug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's GuideBug Bounty Blueprint : A Beginner's Guide
Bug Bounty Blueprint : A Beginner's Guide
 
Pvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdfPvtaan Social media marketing proposal.pdf
Pvtaan Social media marketing proposal.pdf
 
Thank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirtsThank You Luv I’ll Never Walk Alone Again T shirts
Thank You Luv I’ll Never Walk Alone Again T shirts
 
Statistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdfStatistical Analysis of DNS Latencies.pdf
Statistical Analysis of DNS Latencies.pdf
 
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkkaudience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
audience research (emma) 1.pptxkkkkkkkkkkkkkkkkk
 
Premier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdfPremier Mobile App Development Agency in USA.pdf
Premier Mobile App Development Agency in USA.pdf
 
I’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 ShirtI’ll See Y’All Motherfuckers In Game 7 Shirt
I’ll See Y’All Motherfuckers In Game 7 Shirt
 
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
TORTOGEL TELAH MENJADI SALAH SATU PLATFORM PERMAINAN PALING FAVORIT.
 
Reggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirtsReggie miller choke t shirts
Reggie miller choke t shirtsReggie miller choke t shirts
 
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWebiThome_CYBERSEC2024_Drive_Into_the_DarkWeb
iThome_CYBERSEC2024_Drive_Into_the_DarkWeb
 
The Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case StudyThe Use of AI in Indonesia Election 2024: A Case Study
The Use of AI in Indonesia Election 2024: A Case Study
 

Scalable Service Architectures

  • 1. Scalable Service Architectures Lessons learned Zoltán Németh Engineering Manager, Core Systems
  • 2. Agenda  Our scalability experience  What is Scalability?  Requirements in detail  Tips and tools  Extras, Closing remarks
  • 5.
  • 6. Defining scalability Scalability is the ability to handle increased workload by repeatedly applying a costeffective strategy for extending a system’s capacity. (CMU paper, 2006) How well a solution to some problem will work when the size of the problem increases. When the size decreases, the solution must fit. (dictionary.com and Theo Schlossnagle, 2006)
  • 7. Self-contained service  Explicitly declare and isolate dependencies  Isolation from the outside system  Static linking  Do not rely on system packages
  • 8. Disposability  Maximize robustness with fast startup and graceful shutdown  Disposable processes  Graceful shutdown on SIGTERM  Handling sudden death: robust queue backend
  • 9. Startup and Shutdown  Automate all the things  Chef  Docker  Gold image based deployment  Immutable  Handling tasks before shutdown
  • 10. Backing Services  Treat backing services as attached resources  No distinction between local and third party services  Easily swap out resources  Export services via port binding  Become the backing service for another app
  • 11. Processes, concurrency  Stateless processes (not even sticky sessions)  Process types by work type  We <3 linux process  Shared-nothing  adding concurrency is safe  Process distribution spanning machines
  • 12. Statelessness  Store everything in a datastore  Aggregate data  Chandra  Scalable datastores  Redis  Cassandra  Aerospike  Handling user sessions
  • 13. Monitoring  Application state and metrics  Dashboards  Alerting  Health  Remove failing nodes  Capacity  Act on trends
  • 14. Monitoring  Metrics collecting  Graphite, New Relic  Self-aware checks  Cluster state  Zookeeper, Consul  Scaling decision types  Capacity amount  Graph derivative  App requests
  • 15.
  • 16. Load Balance and Resource Allocation  Load Balance: distribute tasks  Utilize machines efficiently  VM compatible apps  Flexibility  Adapting to available resources
  • 17. Load Balance  DNS or API  App level balance  Uniform entry point or proxy  Balance decisions  Load  Zookeeper state  Resource policies
  • 18. Service Separation  Failure is inevitable  Protect from failing components  Cascading failure  Fail fast  Decoupling  Asynchronous operations  Message queues
  • 19. Service Separation  Rate limiting  Circuit Breaker pattern  Stop cascading failure, allow recovery  Hystrix  Fail fast, fail silent  Service decoupling
  • 20. Extras  Debugging features  Logs  Clojure / JS consoles  Runtime configuration via env  Scaling API  Integrating several cloud providers  Automatic start / stop
  • 21. Reading  Scalable Internet Architectures by Theo Schlossnagle  The 12-factor App: http://12factor.net/  Carnegie Mellon Paper: http://www.sei.cmu.edu/reports/06tn012.pdf  Circuit Breaker: http://martinfowler.com/bliki/CircuitBreaker.html  Release It! by Michael T. Nygard

Editor's Notes

  1. A bit of Ustream intro
  2. Definition Requirements coming from 12-factor, and some added by us Some more detail and tools on selected requirements
  3. 30 day viewer graph. Clear peaks -> need for scaling
  4. Quick description of the streaming stack, roles of components, how they require scaling - Transcontroller/transcoder scaling - UMS scaling
  5. Quick description of the streaming stack, roles of components, how they require scaling - Transcontroller/transcoder scaling - UMS scaling
  6. Carnegie Mellon University paper by Charles B. Weinstock, John B. Goodenough: On System Scalability LINFO: The Linux Information Project http://www.linfo.org/
  7. Example: calling imagemagick or curl from code – they might be there or might not be Bundle everything into the app instead
  8. Disposable process: they can be started or stopped at a moment’s notice For a web process, graceful shutdown is achieved by ceasing to listen on the service port (thereby refusing any new requests), allowing any current requests to finish, and then exiting. Implicit in this model is that HTTP requests are short (no more than a few seconds), or in the case of long polling, the client should seamlessly attempt to reconnect when the connection is lost. For a worker process, graceful shutdown is achieved by returning the current job to the work queue.
  9. Docker: build images from dockerfile, deploy from repository Tasks before shutdown: moving jobs, log collection, sleep
  10. A backing service is any service the app consumes over the network as part of its normal operation. Examples include datastores (such as MySQL or CouchDB), messaging/queueing systems (such as RabbitMQ or Beanstalkd), SMTP services for outbound email (such as Postfix), and caching systems (such as Memcached). Put a resource locator in the config only – environment variables Example: Easily swap out a local mysql to a remote service The app does not rely on runtime injection of a webserver into the execution environment to create a web-facing service. The web app exports HTTP as a service by binding to a port, and listening to requests coming in on that port. One app can become the backing service for another app, by providing the URL to the backing app as a resource handle in the config for the consuming app
  11. Handle diverse workloads by assigning each type of work to a process type. For example, HTTP requests may be handled by a web process, and long-running background tasks handled by a worker process An individual VM can only grow so large (vertical scale), so the application must also be able to span multiple processes running on multiple physical machines.
  12. Aggregate everything within the app and write it out in bulk – careful about write frequency, must not lose too many data on a crash Redis: scales reads, write problematic Cassandra: quick scaling questionable Aerospike: scales reads and writes, working together with their eng team User sessions: persistent connection, NIO+
  13. Alerting -> openduty Two important groups: Health vs capacity
  14. Report everything to graphite, constantly check graph trends automatically Apps are self-aware, they know their health App instances report into Zookeeper and thus know about each other Central logic can request resource based on capacity or graph, app can request based on self-check or zookeeper Zookeeper, Consul: miért, mik az előnyei
  15. load balancing distributes workloads across multiple computing resources Flexibility: can increase or decrease its own size, example: Threadpools Adapting to CPU, RAM, disk, network
  16. App level: transcontroller selects transcoder App level balance with proxy can be SPOF, careful Resource policies: even distribution, keep large chunks free for possible large tasks (transcoder use case), group requests together on some attribute (pro, etc)
  17. Failure inevitable because: large numbers, hw issues, independent network Decoupling: serving one request should not wait on others
  18. Hystrix by Netflix 2011/12 Circuit Breaker: Martin Fowler post from 2014 Service decoupling example: inserting layers between DB and UMS -> RGW. Then another layer between RGW and UMS -> Queue
  19. Logs: logs as stream / stdout (factor #9), collect / transport / process Scaling API: Other considerations: price, network line to the cloud provider, instance type (spot vs normal)