SlideShare a Scribd company logo

To Batch Or Not To Batch

A presentation with tips and tools on how to integrate batch and asynchronous operations in a generic ruby on rails application. Did this at rubyday.it 2011

1 of 46
Download to read offline
To Batch Or Not To Batch
         Luca Mearelli
        rubyday.it 2011
First and foremost, we believe that speed
is more than a feature. Speed is the most
important feature. If your application is
                            application is
slow, people won’t use it.
       people won’t use it.
Fred Wilson




                                       @lmea #rubyday
Not all the interesting features are fast




          Interacting with remote API
          Sending emails
          Media transcoding
          Large dataset handling




                                            @lmea #rubyday
Anatomy of an asynchronous action




        The app decides it needs to do a long operation
        The app asks the async system to do the
        operation and quickly returns the response
        The async system executes the operation out-
        of-band




                                                 @lmea #rubyday
Batch
Asynchronous jobs
Queues & workers




                    @lmea #rubyday
Batch




        @lmea #rubyday
Ad

Recommended

Controlling The Cloud With Python
Controlling The Cloud With PythonControlling The Cloud With Python
Controlling The Cloud With PythonLuca Mearelli
 
A Little Backbone For Your App
A Little Backbone For Your AppA Little Backbone For Your App
A Little Backbone For Your AppLuca Mearelli
 
And now you have two problems. Ruby regular expressions for fun and profit by...
And now you have two problems. Ruby regular expressions for fun and profit by...And now you have two problems. Ruby regular expressions for fun and profit by...
And now you have two problems. Ruby regular expressions for fun and profit by...Codemotion
 
OSCON Google App Engine Codelab - July 2010
OSCON Google App Engine Codelab - July 2010OSCON Google App Engine Codelab - July 2010
OSCON Google App Engine Codelab - July 2010ikailan
 
Why Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebWhy Task Queues - ComoRichWeb
Why Task Queues - ComoRichWebBryan Helmig
 
Django Celery - A distributed task queue
Django Celery - A distributed task queueDjango Celery - A distributed task queue
Django Celery - A distributed task queueAlex Eftimie
 
Europython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryEuropython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryMauro Rocco
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to CeleryIdan Gazit
 

More Related Content

What's hot

The worst Ruby codes I’ve seen in my life - RubyKaigi 2015
The worst Ruby codes I’ve seen in my life - RubyKaigi 2015The worst Ruby codes I’ve seen in my life - RubyKaigi 2015
The worst Ruby codes I’ve seen in my life - RubyKaigi 2015Fernando Hamasaki de Amorim
 
Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task QueueDuy Do
 
New Symfony Tips & Tricks (SymfonyCon Paris 2015)
New Symfony Tips & Tricks (SymfonyCon Paris 2015)New Symfony Tips & Tricks (SymfonyCon Paris 2015)
New Symfony Tips & Tricks (SymfonyCon Paris 2015)Javier Eguiluz
 
Advanced symfony Techniques
Advanced symfony TechniquesAdvanced symfony Techniques
Advanced symfony TechniquesKris Wallsmith
 
Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...
Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...
Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...King Foo
 
Symfony & Javascript. Combining the best of two worlds
Symfony & Javascript. Combining the best of two worldsSymfony & Javascript. Combining the best of two worlds
Symfony & Javascript. Combining the best of two worldsIgnacio Martín
 
The Coolest Symfony Components you’ve never heard of - DrupalCon 2017
The Coolest Symfony Components you’ve never heard of - DrupalCon 2017The Coolest Symfony Components you’ve never heard of - DrupalCon 2017
The Coolest Symfony Components you’ve never heard of - DrupalCon 2017Ryan Weaver
 
Datagrids with Symfony 2, Backbone and Backgrid
Datagrids with Symfony 2, Backbone and BackgridDatagrids with Symfony 2, Backbone and Backgrid
Datagrids with Symfony 2, Backbone and Backgrideugenio pombi
 
Building Cloud Castles
Building Cloud CastlesBuilding Cloud Castles
Building Cloud CastlesBen Scofield
 
Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...
Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...
Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...Ville Mattila
 
Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015Masahiro Nagano
 
With a Mighty Hammer
With a Mighty HammerWith a Mighty Hammer
With a Mighty HammerBen Scofield
 
A Functional Guide to Cat Herding with PHP Generators
A Functional Guide to Cat Herding with PHP GeneratorsA Functional Guide to Cat Herding with PHP Generators
A Functional Guide to Cat Herding with PHP GeneratorsMark Baker
 
Python RESTful webservices with Python: Flask and Django solutions
Python RESTful webservices with Python: Flask and Django solutionsPython RESTful webservices with Python: Flask and Django solutions
Python RESTful webservices with Python: Flask and Django solutionsSolution4Future
 
The road to Ember.js 2.0
The road to Ember.js 2.0The road to Ember.js 2.0
The road to Ember.js 2.0Codemotion
 
Effective Doctrine2: Performance Tips for Symfony2 Developers
Effective Doctrine2: Performance Tips for Symfony2 DevelopersEffective Doctrine2: Performance Tips for Symfony2 Developers
Effective Doctrine2: Performance Tips for Symfony2 DevelopersMarcin Chwedziak
 

What's hot (20)

Practical Celery
Practical CeleryPractical Celery
Practical Celery
 
The worst Ruby codes I’ve seen in my life - RubyKaigi 2015
The worst Ruby codes I’ve seen in my life - RubyKaigi 2015The worst Ruby codes I’ve seen in my life - RubyKaigi 2015
The worst Ruby codes I’ve seen in my life - RubyKaigi 2015
 
Celery - A Distributed Task Queue
Celery - A Distributed Task QueueCelery - A Distributed Task Queue
Celery - A Distributed Task Queue
 
New Symfony Tips & Tricks (SymfonyCon Paris 2015)
New Symfony Tips & Tricks (SymfonyCon Paris 2015)New Symfony Tips & Tricks (SymfonyCon Paris 2015)
New Symfony Tips & Tricks (SymfonyCon Paris 2015)
 
Advanced symfony Techniques
Advanced symfony TechniquesAdvanced symfony Techniques
Advanced symfony Techniques
 
Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...
Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...
Building Web Services with Zend Framework (PHP Benelux meeting 20100713 Vliss...
 
Symfony & Javascript. Combining the best of two worlds
Symfony & Javascript. Combining the best of two worldsSymfony & Javascript. Combining the best of two worlds
Symfony & Javascript. Combining the best of two worlds
 
The Coolest Symfony Components you’ve never heard of - DrupalCon 2017
The Coolest Symfony Components you’ve never heard of - DrupalCon 2017The Coolest Symfony Components you’ve never heard of - DrupalCon 2017
The Coolest Symfony Components you’ve never heard of - DrupalCon 2017
 
Django Celery
Django Celery Django Celery
Django Celery
 
Datagrids with Symfony 2, Backbone and Backgrid
Datagrids with Symfony 2, Backbone and BackgridDatagrids with Symfony 2, Backbone and Backgrid
Datagrids with Symfony 2, Backbone and Backgrid
 
Building Cloud Castles
Building Cloud CastlesBuilding Cloud Castles
Building Cloud Castles
 
Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...
Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...
Running a Scalable And Reliable Symfony2 Application in Cloud (Symfony Sweden...
 
Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015
 
With a Mighty Hammer
With a Mighty HammerWith a Mighty Hammer
With a Mighty Hammer
 
Developing apps using Perl
Developing apps using PerlDeveloping apps using Perl
Developing apps using Perl
 
A Functional Guide to Cat Herding with PHP Generators
A Functional Guide to Cat Herding with PHP GeneratorsA Functional Guide to Cat Herding with PHP Generators
A Functional Guide to Cat Herding with PHP Generators
 
Python RESTful webservices with Python: Flask and Django solutions
Python RESTful webservices with Python: Flask and Django solutionsPython RESTful webservices with Python: Flask and Django solutions
Python RESTful webservices with Python: Flask and Django solutions
 
The road to Ember.js 2.0
The road to Ember.js 2.0The road to Ember.js 2.0
The road to Ember.js 2.0
 
Symfony2 revealed
Symfony2 revealedSymfony2 revealed
Symfony2 revealed
 
Effective Doctrine2: Performance Tips for Symfony2 Developers
Effective Doctrine2: Performance Tips for Symfony2 DevelopersEffective Doctrine2: Performance Tips for Symfony2 Developers
Effective Doctrine2: Performance Tips for Symfony2 Developers
 

Similar to To Batch Or Not To Batch

Ruby on Rails - Introduction
Ruby on Rails - IntroductionRuby on Rails - Introduction
Ruby on Rails - IntroductionVagmi Mudumbai
 
Ruby/Rails
Ruby/RailsRuby/Rails
Ruby/Railsrstankov
 
TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011Lance Ball
 
Background Jobs - Com BackgrounDRb
Background Jobs - Com BackgrounDRbBackground Jobs - Com BackgrounDRb
Background Jobs - Com BackgrounDRbJuan Maiz
 
Lean Php Presentation
Lean Php PresentationLean Php Presentation
Lean Php PresentationAlan Pinstein
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby TeamArto Artnik
 
Rails web api 开发
Rails web api 开发Rails web api 开发
Rails web api 开发shaokun
 
Deferred Processing in Ruby - Philly rb - August 2011
Deferred Processing in Ruby - Philly rb - August 2011Deferred Processing in Ruby - Philly rb - August 2011
Deferred Processing in Ruby - Philly rb - August 2011rob_dimarco
 
Background Jobs with Resque
Background Jobs with ResqueBackground Jobs with Resque
Background Jobs with Resquehomanj
 
A tour on ruby and friends
A tour on ruby and friendsA tour on ruby and friends
A tour on ruby and friends旻琦 潘
 
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...bobmcwhirter
 
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQueryRemedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQueryTatsuhiko Miyagawa
 
Pourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirentPourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirentNicolas Ledez
 
Web aplikāciju izstrāde ar Ruby on Rails un Oracle DB
Web aplikāciju izstrāde ar Ruby on Rails un Oracle DBWeb aplikāciju izstrāde ar Ruby on Rails un Oracle DB
Web aplikāciju izstrāde ar Ruby on Rails un Oracle DBRaimonds Simanovskis
 

Similar to To Batch Or Not To Batch (20)

Ruby on Rails - Introduction
Ruby on Rails - IntroductionRuby on Rails - Introduction
Ruby on Rails - Introduction
 
Ruby/Rails
Ruby/RailsRuby/Rails
Ruby/Rails
 
TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011TorqueBox - Ruby Hoedown 2011
TorqueBox - Ruby Hoedown 2011
 
Background Jobs - Com BackgrounDRb
Background Jobs - Com BackgrounDRbBackground Jobs - Com BackgrounDRb
Background Jobs - Com BackgrounDRb
 
Lean Php Presentation
Lean Php PresentationLean Php Presentation
Lean Php Presentation
 
Supa fast Ruby + Rails
Supa fast Ruby + RailsSupa fast Ruby + Rails
Supa fast Ruby + Rails
 
Celery with python
Celery with pythonCelery with python
Celery with python
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby Team
 
Rails web api 开发
Rails web api 开发Rails web api 开发
Rails web api 开发
 
Wider than rails
Wider than railsWider than rails
Wider than rails
 
Deferred Processing in Ruby - Philly rb - August 2011
Deferred Processing in Ruby - Philly rb - August 2011Deferred Processing in Ruby - Philly rb - August 2011
Deferred Processing in Ruby - Philly rb - August 2011
 
Background Jobs with Resque
Background Jobs with ResqueBackground Jobs with Resque
Background Jobs with Resque
 
Php resque
Php resquePhp resque
Php resque
 
A tour on ruby and friends
A tour on ruby and friendsA tour on ruby and friends
A tour on ruby and friends
 
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...TorqueBox: The beauty of Ruby with the power of JBoss.  Presented at Devnexus...
TorqueBox: The beauty of Ruby with the power of JBoss. Presented at Devnexus...
 
Celery
CeleryCelery
Celery
 
Capistrano Overview
Capistrano OverviewCapistrano Overview
Capistrano Overview
 
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQueryRemedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
 
Pourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirentPourquoi ruby et rails déchirent
Pourquoi ruby et rails déchirent
 
Web aplikāciju izstrāde ar Ruby on Rails un Oracle DB
Web aplikāciju izstrāde ar Ruby on Rails un Oracle DBWeb aplikāciju izstrāde ar Ruby on Rails un Oracle DB
Web aplikāciju izstrāde ar Ruby on Rails un Oracle DB
 

More from Luca Mearelli

More from Luca Mearelli (9)

And Now You Have Two Problems
And Now You Have Two ProblemsAnd Now You Have Two Problems
And Now You Have Two Problems
 
The anatomy of an infographic
The anatomy of an infographicThe anatomy of an infographic
The anatomy of an infographic
 
L'altra meta del web
L'altra meta del webL'altra meta del web
L'altra meta del web
 
WorseSoftware
WorseSoftwareWorseSoftware
WorseSoftware
 
Open Web
Open WebOpen Web
Open Web
 
Capistrano2
Capistrano2Capistrano2
Capistrano2
 
Wikierp
WikierpWikierp
Wikierp
 
Introduzione a Ruby On Rails
Introduzione a Ruby On RailsIntroduzione a Ruby On Rails
Introduzione a Ruby On Rails
 
Integrating services with OAuth
Integrating services with OAuthIntegrating services with OAuth
Integrating services with OAuth
 

Recently uploaded

Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfSafe Software
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringMassimo Talia
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manualDomotica daVinci
 
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Adrian Sanabria
 
"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes", Volodymyr TsapFwdays
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologySafe Software
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stackSummit
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...ISPMAIndia
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementMimmo Squillace
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura RochniakFwdays
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotelPhilippines
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfDomotica daVinci
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfIsidro Navarro
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolProduct School
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVARobert McDermott
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxMaarten Balliauw
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellencePrecisely
 
M.Aathiraju Self Intro.docx-AD21001_____
M.Aathiraju Self Intro.docx-AD21001_____M.Aathiraju Self Intro.docx-AD21001_____
M.Aathiraju Self Intro.docx-AD21001_____Aathiraju
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?GleecusTechlabs1
 

Recently uploaded (20)

Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdfIntroducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
Introducing the New FME Community Webinar - Feb 21, 2024 (2).pdf
 
Dynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineeringDynamical systems simulation in Python for science and engineering
Dynamical systems simulation in Python for science and engineering
 
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
Zi-Stick UBS Dongle ZIgbee from  Aeotec manualZi-Stick UBS Dongle ZIgbee from  Aeotec manual
Zi-Stick UBS Dongle ZIgbee from Aeotec manual
 
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
 
"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap"Running Open-Source LLM models on Kubernetes",  Volodymyr Tsap
"Running Open-Source LLM models on Kubernetes", Volodymyr Tsap
 
Breaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI TechnologyBreaking Barriers & Leveraging the Latest Developments in AI Technology
Breaking Barriers & Leveraging the Latest Developments in AI Technology
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stack
 
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
AI MODELS USAGE IN FINTECH PRODUCTS: PM APPROACH & BEST PRACTICES by Kasthuri...
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvement
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak
 
Campotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company ProfileCampotel: Telecommunications Infra and Network Builder - Company Profile
Campotel: Telecommunications Infra and Network Builder - Company Profile
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdf
 
Artificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdfArtificial-Intelligence-in-Marketing-Data.pdf
Artificial-Intelligence-in-Marketing-Data.pdf
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
The Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product SchoolThe Future of Product, by Founder & CEO, Product School
The Future of Product, by Founder & CEO, Product School
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptx
 
Automate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center ExcellenceAutomate Your Master Data Processes for Shared Service Center Excellence
Automate Your Master Data Processes for Shared Service Center Excellence
 
M.Aathiraju Self Intro.docx-AD21001_____
M.Aathiraju Self Intro.docx-AD21001_____M.Aathiraju Self Intro.docx-AD21001_____
M.Aathiraju Self Intro.docx-AD21001_____
 
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
Unlocking the Cloud's True Potential: Why Multitenancy Is The Key?
 

To Batch Or Not To Batch

  • 1. To Batch Or Not To Batch Luca Mearelli rubyday.it 2011
  • 2. First and foremost, we believe that speed is more than a feature. Speed is the most important feature. If your application is application is slow, people won’t use it. people won’t use it. Fred Wilson @lmea #rubyday
  • 3. Not all the interesting features are fast Interacting with remote API Sending emails Media transcoding Large dataset handling @lmea #rubyday
  • 4. Anatomy of an asynchronous action The app decides it needs to do a long operation The app asks the async system to do the operation and quickly returns the response The async system executes the operation out- of-band @lmea #rubyday
  • 5. Batch Asynchronous jobs Queues & workers @lmea #rubyday
  • 6. Batch @lmea #rubyday
  • 7. Cron scheduled operations unrelated to the requests low frequency longer run time @lmea #rubyday
  • 8. Anatomy of a cron batch: the rake task namespace :export do task :items_xml => :environment do # read the env variables # make the export end end @lmea #rubyday
  • 9. Anatomy of a cron batch: the shell script #!/bin/sh # this goes in script/item_export_full.sh cd /usr/rails/MyApp/current export RAILS_ENV=production echo "Item Export Full started: `date`" rake export:items_xml XML_FOLDER='data/exports' echo "Item Export Full completed: `date`" @lmea #rubyday
  • 10. Anatomy of a cron batch: the crontab entry 0 0 1 * * /usr/rails/MyApp/current/script/item_export_full.sh >> /usr/rails/ MyApp/current/log/dump_item_export.log 2>&1 30 13 * * * cd /usr/rails/MyApp/current; ruby /usr/rails/MyApp/current/script/ runner -e production "Newsletter.deliver_daily" >> /usr/rails/MyApp/current/log/ newsletter_daily.log 2>&1 @lmea #rubyday
  • 11. Cron helpers Whenever https://github.com/javan/whenever Craken https://github.com/latimes/craken @lmea #rubyday
  • 12. Whenever: schedule.rb # adds ">> /path/to/file.log 2>&1" to all commands set :output, '/path/to/file.log' every 3.hours do rake "my:rake:task" end every 1.day, :at => '4:30 am' do runner "MyModel.task_to_run_at_four_thirty_in_the_morning" end every :hour do # Many shortcuts available: :hour, :day, :month, :year, :reboot command "/usr/bin/my_great_command", :output => {:error => 'error.log', :standard => 'cron.log'} end @lmea #rubyday
  • 13. Cracken: raketab 59 * * * * thing:to_do > /tmp/thing_to_do.log 2>&1 @daily solr:reindex > /tmp/solr_daily.log 2>&1 # also @yearly, @annually, @monthly, @weekly, @midnight, @hourly @lmea #rubyday
  • 14. Cracken: raketab.rb Raketab.new do |cron| cron.schedule 'thing:to_do > /tmp/thing_to_do.log 2>&1', :every => mon..fri cron.schedule 'first:five:days > /tmp/thing_to_do.log 2>&1', :days => [1,2,3,4,5] cron.schedule 'first:day:q1 > /tmp/thing_to_do.log 2>&1', :the => '1st', :in => [jan,feb,mar] cron.schedule 'first:day:q4 > /tmp/thing_to_do.log 2>&1', :the => '1st', :months => 'October,November,December' end @lmea #rubyday
  • 15. Queues & Workers un-scheduled operations responding to a request mid to high frequency mixed run time @lmea #rubyday
  • 16. Queues & Workers Delayed job https://github.com/collectiveidea/delayed_job Resque https://github.com/defunkt/resque @lmea #rubyday
  • 17. Delayed job Any object method can be a job Db backed queue Integer-based priority Lifecycle hooks (enqueue, before, after, ... ) @lmea #rubyday
  • 18. Delayed job: simple jobs # without delayed_job @user.notify!(@event) # with delayed_job @user.delay.notify!(@event) # always asyncronous method class Newsletter def deliver # long running method end handle_asynchronously :deliver end newsletter = Newsletter.new newsletter.deliver @lmea #rubyday
  • 19. Delayed job: handle_asyncronously handle_asynchronously :sync_method, :priority => 20 handle_asynchronously :in_the_future, :run_at => Proc.new { 5.minutes.from_now } handle_asynchronously :call_a_class_method, :run_at => Proc.new { when_to_run } handle_asynchronously :call_an_instance_method, :priority => Proc.new {|i| i.how_important } @lmea #rubyday
  • 20. Delayed job class NewsletterJob < Struct.new(:text, :emails) def perform emails.each { |e| NewsMailer.deliver_text_to_email(text, e) } end end Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...', User.find(:all).collect(&:email)) @lmea #rubyday
  • 21. Delayed job RAILS_ENV=production script/delayed_job -n 2 --min-priority 10 start RAILS_ENV=production script/delayed_job stop rake jobs:work @lmea #rubyday
  • 22. Delayed job: checking the job status The queue is for scheduled and running jobs Handle the status outside Delayed::Job object @lmea #rubyday
  • 23. Delayed job: checking the job status # Include this in your initializers somewhere class Queue < Delayed::Job def self.status(id) self.find_by_id(id).nil? ? "success" : (job.last_error.nil? ? "queued" : "failure") end end # Use this method in your poll method like so: def poll status = Queue.status(params[:id]) if status == "success" # Success, notify the user! elsif status == "failure" # Failure, notify the user! end end @lmea #rubyday
  • 24. Delayed job: checking the job status class AJob < Struct.new(:options) def perform do_something(options) end def success(job) # record success of job.id Rails.cache.write("status:#{job.id}", "success") end end # a helper def job_completed_with_success(job_id) Rails.cache.read("status:#{job_id}")=="success" end @lmea #rubyday
  • 25. Resque Redis-backed queues Queue/dequeue speed independent of list size Forking behaviour Built in front-end Multiple queues / no priorities @lmea #rubyday
  • 26. Resque: the job class Export @queue = :export_jobs def self.perform(dataset_id, kind = 'full') ds = Dataset.find(dataset_id) ds.create_export(kind) end end @lmea #rubyday
  • 27. Resque: enqueuing the job class Dataset def async_create_export(kind) Resque.enqueue(Export, self.id, kind) end end ds = Dataset.find(100) ds.async_create_export('full') @lmea #rubyday
  • 28. Resque: persisting the job # jobs are persisted as JSON, # so jobs should only take arguments that can be expressed as JSON { 'class': 'Export', 'args': [ 100, 'full' ] } # don't do this: Resque.enqueue(Export, self, kind) # do this: Resque.enqueue(Export, self.id, kind) @lmea #rubyday
  • 29. Resque: generic async methods # A simple async helper class Repository < ActiveRecord::Base # This will be called by a worker when a job needs to be processed def self.perform(id, method, *args) find(id).send(method, *args) end # We can pass this any Repository instance method that we want to # run later. def async(method, *args) Resque.enqueue(Repository, id, method, *args) end end # Now we can call any method and have it execute later: @repo.async(:update_disk_usage) @repo.async(:update_network_source_id, 34) @lmea #rubyday
  • 30. Resque: anatomy of a worker # a worker does this: start loop do if job = reserve job.process else sleep 5 end end shutdown @lmea #rubyday
  • 31. Resque: working the queues $ QUEUES=critical,high,low rake resque:work $ QUEUES=* rake resque:work $ PIDFILE=./resque.pid QUEUE=export_jobs rake environment resque:work task "resque:setup" => :environment do AppConfig.a_parameter = ... end @lmea #rubyday
  • 32. Resque: monit recipe # example monit monitoring recipe check process resque_worker_batch_01 with pidfile /app/current/tmp/pids/worker_01.pid start program = "/bin/bash -c 'cd /app/current; RAILS_ENV=production QUEUE=batch_queue nohup rake environment resque:work & > log/worker_01.log && echo $! > tmp/pids/worker_01.pid'" as uid deploy and gid deploy stop program = "/bin/bash -c 'cd /app/current && kill -s QUIT `cat tmp/pids/worker_01.pid` && rm -f tmp/pids/worker_01.pid; exit 0;'" if totalmem is greater than 1000 MB for 10 cycles then restart # eating up memory? group resque_workers @lmea #rubyday
  • 33. Resque: built-in monitoring @lmea #rubyday
  • 34. Resque plugins Resque-status https://github.com/quirkey/resque-status Resque-scheduler https://github.com/bvandenbos/resque-scheduler/ More at: https://github.com/defunkt/resque/wiki/plugins @lmea #rubyday
  • 35. Resque-status Simple trackable jobs for resque Job instances have a UUID Jobs can report their status while running @lmea #rubyday
  • 36. Resque-status # inheriting from JobWithStatus class ExportJob < Resque::JobWithStatus # perform is an instance method def perform limit = options['limit'].to_i || 1000 items = Item.limit(limit) total = items.count exported = [] items.each_with_index do |item, num| at(num, total, "At #{num} of #{total}") exported << item.to_csv end File.open(local_filename, 'w') { |f| f.write(exported.join("n")) } complete(:filename=>local_filename) end end @lmea #rubyday
  • 37. Resque-status job_id = SleepJob.create(:length => 100) status = Resque::Status.get(job_id) # the status object tell us: status.pct_complete #=> 0 status.status #=> 'queued' status.queued? #=> true status.working? #=> false status.time #=> Time object status.message #=> "Created at ..." Resque::Status.kill(job_id) @lmea #rubyday
  • 38. Resque-scheduler Queueing for future execution Scheduling jobs (like cron!) @lmea #rubyday
  • 39. Resque-scheduler # run a job in 5 days Resque.enqueue_in(5.days, SendFollowupEmail) # run SomeJob at a specific time Resque.enqueue_at(5.days.from_now, SomeJob) @lmea #rubyday
  • 40. Resque-scheduler namespace :resque do task :setup do require 'resque' require 'resque_scheduler' require 'resque/scheduler' Resque.redis = 'localhost:6379' # The schedule doesn't need to be stored in a YAML, it just needs to # be a hash. YAML is usually the easiest. Resque::Scheduler.schedule = YAML.load_file('your_resque_schedule.yml') # When dynamic is set to true, the scheduler process looks for # schedule changes and applies them on the fly. # Also if dynamic the Resque::Scheduler.set_schedule (and remove_schedule) # methods can be used to alter the schedule #Resque::Scheduler.dynamic = true end end $ rake resque:scheduler @lmea #rubyday
  • 41. Resque-scheduler: the yaml configuration queue_documents_for_indexing: cron: "0 0 * * *" class: QueueDocuments queue: high args: description: "This job queues all content for indexing in solr" export_items: cron: "30 6 * * 1" class: Export queue: low args: full description: "This job does a weekly export" @lmea #rubyday
  • 42. Other (commercial) SimpleWorker http://simpleworker.com SQS https://github.com/appoxy/aws/ http://rubygems.org/gems/right_aws http://sdruby.org/video/024_amazon_sqs.m4v @lmea #rubyday
  • 43. Other (historical) Beanstalkd and Stalker http://asciicasts.com/episodes/243-beanstalkd-and-stalker http://kr.github.com/beanstalkd/ https://github.com/han/stalker Backgroundjob (Bj) https://github.com/ahoward/bj BackgroundRb http://backgroundrb.rubyforge.org/ @lmea #rubyday
  • 44. Other (different approaches) Nanite http://www.slideshare.net/jendavis100/background-processing-with-nanite Cloud Crowd https://github.com/documentcloud/cloud-crowd/wiki/Getting-Started @lmea #rubyday
  • 45. Ciao! me@spazidigitali.com @lmea #rubyday
  • 46. http://www.flickr.com/photos/rkbcupcakes/3373909785/ http://www.flickr.com/photos/anjin/23460398 http://www.flickr.com/photos/vivacomopuder/3122401239 http://www.flickr.com/photos/pacdog/4968422200 http://www.flickr.com/photos/comedynose/3834416952 http://www.flickr.com/photos/rhysasplundh/5177851910/ http://www.flickr.com/photos/marypcb/104308457 http://www.flickr.com/photos/shutterhacks/4474421855 http://www.flickr.com/photos/kevinschoenmakersnl/5562839479 http://www.flickr.com/photos/triplexpresso/496995086 http://www.flickr.com/photos/saxonmoseley/24523450 http://www.flickr.com/photos/gadl/89650415 http://www.flickr.com/photos/matvey_andreyev/3656451273 http://www.flickr.com/photos/bryankennedy/1992770068 http://www.flickr.com/photos/27282406@N03/4134661728/ @lmea #rubyday