SlideShare a Scribd company logo
1 of 30
Download to read offline
Sylvain Zimmer / @sylvinus
PyParis 2017
DEVELOPER-FRIENDLY TASKQUEUES
WHAT WE LEARNED BUILDING MRQ
& WHAT YOU SHOULD ASK YOURSELF BEFORE CHOOSING ONE
/usr/bin/whoami
▸ (SpaceX nerd)
▸ Founder, dotConferences
▸ CTO Pricing Assistant
▸ Co-organizer Paris.py meetup
▸ User of Python taskqueues for 10+ years
▸ Main contributor of MRQ
4 years ago...
OMG !1!!!
TASKQUEUE FUNDAMENTALS
Credit: Adrien Di Pasquale
A typical job/task
def send_an_email(email_type, user):
html = template(email_type, user)
status = email.send(html, user["email"])
metrics.send("email_%s" % status, 1)
return status
KERNEL PANIC
Task properties
Re-entrant Idempotent Nullipotent< <
▸ Safe to interrupt
and then retry
▸ Safe to call
multiple times
▸ Result will be the
same
▸ Free of side-effects
def reentrant(a):
value = a + random()
db.insert(value)
def idempotent(key, value):
db.update(key, value)
def nullipotent(a):
return a ** 2
Other task properties & best practices
▸ Serializable args, serializable result
▸ Args validation / documentation
▸ Least args possible
▸ Canonical path vs. registration
▸ Concurrent safety
▸ Statuses
Coroutines vs. Threads vs. Processes
▸ IO-bound tasks vs. CPU-bound tasks
▸ Threads offer few benefits for a Python worker (GIL)
▸ Coroutines/Greenlets are ideal for IO-bound tasks
▸ Processes are required for CPU-bound tasks
▸ If you have heterogenous tasks, your TQ should support
both!
$ mrq-worker --greenlets 25 --processes 4
Performance: latency & throughput
APP
BROKER WORKER
RESULT STORE
OPS & TOOLING
Errors
▸ Exception handlers
▸ Timeouts
▸ Retry rules
▸ Sentry & friends
▸ gevent: test your tracebacks!
▸ Priorities
▸ Human process to manage failed tasks!
Task visibility
▸ Tasks by status, path, worker, ...
▸ Tracebacks & current stack
▸ Logs
▸ Timing info
▸ Cancel / Kill / Move tasks
▸ Progress
Memory leaks
▸ Workers = long-running processes
▸ gevent makes debugging harder
▸ Watch out for global variables or mutable class attributes!
▸ Python's ecosystem is surprisingly poor in this area
▸ guppy, objgraph can usually help
Misc tools
▸ Scheduler
▸ Command-line runner, e.g. mrq-run tasks.myTask {"a": 1}
▸ Autoscaling
▸ Profiler
CONSISTENCY
Consistency guarantees
▸ At least once vs. At most once vs. Exactly once
▸ Ordering
▸ Critical operations:
▸ Queueing
▸ Marking tasks as started
▸ Timeouts & retries
Types of brokers
▸ Specialized message queues (RabbitMQ, SNS, Kafka, ...)
▸ Performance, complexity, poor visibility
▸ In-memory data stores (Redis, ...)
▸ Performance, simplicity, harder to scale
▸ Regular databases (MongoDB, PostgreSQL, ...)
▸ Often enough for the job!
At the heart of the broker
▸ Atomic update from "queued" to "started"
▸ MRQ with MongoDB broker: find_one_and_update()
▸ MRQ with Redis broker: Pushback in a ZSET
ZSETs in Redis
▸ Sorted sets with O(log(N)) scalability
▸ set/get by key, order by key, lookups by key or value
▸ Very interesting properties for task queues: Unicity,
Ordering, Atomicity of updates, Performance
▸ MRQ's "Pushback" model:
▸ Queue with key=timestamp
▸ Unqueue by fetching key range & setting new keys in
the future
▸ After completion the task adjusts or removes the key
Consistency guarantees
▸ Must be thought of for the whole system, not just the
broker!
▸ Brokers can be misused or misconfigured
▸ The workers can drop tasks if they want to ;-)
▸ Consistency starts at queueing time!
TIME TO CHOOSE!
Think hard about what you need
▸ Will your taskqueue be the foundation of your
architecture, or is it just a side project?
▸ What performance do you need? (IO vs. CPU, latency, ...)
▸ What level of visibility and control do you need on queued
& running tasks?
▸ Can workers terminate abruptly? Lots of design
consequences!
▸ What language interop do you need?
And then all the usual questions...
▸ Is it supported by a lively community?
▸ License
▸ Documentation
▸ Future plans
Which one to pick?
▸ Celery: High performance, large community, very complex,
major upgrades painful
▸ RQ: Extremely simple to understand, low performance
▸ MRQ: Adjust task visibility vs. performance, simple to
understand, 1.0 soon
▸ Lots of other valid options! Just be sure to ask yourself the
right questions ;-)
BE GRATEFUL FOR
THE OSS YOU USE!
REMINDER
Hiring Pythonistas!
QUESTIONS?
THANKS!
Photo credits: https://www.flickr.com/photos/spacex/

More Related Content

What's hot

Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...
Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...
Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...Codemotion
 
Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...
Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...
Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...Codemotion
 
Functional Programming in JavaScript
Functional Programming in JavaScriptFunctional Programming in JavaScript
Functional Programming in JavaScriptTroy Miles
 
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...Codemotion
 
React Development with the MERN Stack
React Development with the MERN StackReact Development with the MERN Stack
React Development with the MERN StackTroy Miles
 
Gatling @ Scala.Io 2013
Gatling @ Scala.Io 2013Gatling @ Scala.Io 2013
Gatling @ Scala.Io 2013slandelle
 
[231] the simplicity of cluster apps with circuit
[231] the simplicity of cluster apps with circuit[231] the simplicity of cluster apps with circuit
[231] the simplicity of cluster apps with circuitNAVER D2
 
Asynchronous programming in ASP.NET
Asynchronous programming in ASP.NETAsynchronous programming in ASP.NET
Asynchronous programming in ASP.NETAlex Thissen
 
The Many Ways to Test Your React App
The Many Ways to Test Your React AppThe Many Ways to Test Your React App
The Many Ways to Test Your React AppAll Things Open
 
Basics of Node.js
Basics of Node.jsBasics of Node.js
Basics of Node.jsAlper Unal
 
Hadoop Demystified + Automation Smackdown! Austin JUG June 24 2014
Hadoop Demystified + Automation Smackdown!  Austin JUG June 24 2014Hadoop Demystified + Automation Smackdown!  Austin JUG June 24 2014
Hadoop Demystified + Automation Smackdown! Austin JUG June 24 2014datafundamentals
 
Scala Future & Promises
Scala Future & PromisesScala Future & Promises
Scala Future & PromisesKnoldus Inc.
 
Woo: Writing a fast web server
Woo: Writing a fast web serverWoo: Writing a fast web server
Woo: Writing a fast web serverfukamachi
 

What's hot (20)

Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...
Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...
Erik Wendel - Beyond JavaScript Frameworks: Writing Reliable Web Apps With El...
 
Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...
Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...
Carlo Sciolla - Above and beyond type systems with clojure.spec - Codemotion ...
 
Functional Programming in JavaScript
Functional Programming in JavaScriptFunctional Programming in JavaScript
Functional Programming in JavaScript
 
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...
Alberto Maria Angelo Paro - Isomorphic programming in Scala and WebDevelopmen...
 
Javantura v3 - ES6 – Future Is Now – Nenad Pečanac
Javantura v3 - ES6 – Future Is Now – Nenad PečanacJavantura v3 - ES6 – Future Is Now – Nenad Pečanac
Javantura v3 - ES6 – Future Is Now – Nenad Pečanac
 
Celery with python
Celery with pythonCelery with python
Celery with python
 
OHHttpStubs
OHHttpStubsOHHttpStubs
OHHttpStubs
 
Async Programming in C# 5
Async Programming in C# 5Async Programming in C# 5
Async Programming in C# 5
 
React Development with the MERN Stack
React Development with the MERN StackReact Development with the MERN Stack
React Development with the MERN Stack
 
Gatling @ Scala.Io 2013
Gatling @ Scala.Io 2013Gatling @ Scala.Io 2013
Gatling @ Scala.Io 2013
 
[231] the simplicity of cluster apps with circuit
[231] the simplicity of cluster apps with circuit[231] the simplicity of cluster apps with circuit
[231] the simplicity of cluster apps with circuit
 
Asynchronous programming in ASP.NET
Asynchronous programming in ASP.NETAsynchronous programming in ASP.NET
Asynchronous programming in ASP.NET
 
The Many Ways to Test Your React App
The Many Ways to Test Your React AppThe Many Ways to Test Your React App
The Many Ways to Test Your React App
 
Async/Await
Async/AwaitAsync/Await
Async/Await
 
Basics of Node.js
Basics of Node.jsBasics of Node.js
Basics of Node.js
 
Async await
Async awaitAsync await
Async await
 
Hadoop Demystified + Automation Smackdown! Austin JUG June 24 2014
Hadoop Demystified + Automation Smackdown!  Austin JUG June 24 2014Hadoop Demystified + Automation Smackdown!  Austin JUG June 24 2014
Hadoop Demystified + Automation Smackdown! Austin JUG June 24 2014
 
Scala Future & Promises
Scala Future & PromisesScala Future & Promises
Scala Future & Promises
 
Woo: Writing a fast web server
Woo: Writing a fast web serverWoo: Writing a fast web server
Woo: Writing a fast web server
 
Introduction to Node.JS
Introduction to Node.JSIntroduction to Node.JS
Introduction to Node.JS
 

Similar to Developer-friendly taskqueues: What you should ask yourself before choosing one

Modern day jvm controversies
Modern day jvm controversiesModern day jvm controversies
Modern day jvm controversiesVictorSzoltysek
 
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Brian Brazil
 
The Best Feature of Go – A 5 Year Retrospective
The Best Feature of Go – A 5 Year RetrospectiveThe Best Feature of Go – A 5 Year Retrospective
The Best Feature of Go – A 5 Year RetrospectiveTahir Hashmi
 
Node, can you even in CPU intensive operations?
Node, can you even in CPU intensive operations?Node, can you even in CPU intensive operations?
Node, can you even in CPU intensive operations?The Software House
 
DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.Vlad Fedosov
 
PowerShell - Be A Cool Blue Kid
PowerShell - Be A Cool Blue KidPowerShell - Be A Cool Blue Kid
PowerShell - Be A Cool Blue KidMatthew Johnson
 
Software Engineering Best Practices @ Nylas
Software Engineering Best Practices @ NylasSoftware Engineering Best Practices @ Nylas
Software Engineering Best Practices @ NylasBen Gotow
 
Human scaling on the front end
Human scaling on the front endHuman scaling on the front end
Human scaling on the front endRudy Rigot
 
Optimizing Performance - Clojure Remote - Nikola Peric
Optimizing Performance - Clojure Remote - Nikola PericOptimizing Performance - Clojure Remote - Nikola Peric
Optimizing Performance - Clojure Remote - Nikola PericNik Peric
 
Faster PHP apps using Queues and Workers
Faster PHP apps using Queues and WorkersFaster PHP apps using Queues and Workers
Faster PHP apps using Queues and WorkersRichard Baker
 
2014 11 20 Drupal 7 -> 8 test migratie
2014 11 20 Drupal 7 -> 8 test migratie2014 11 20 Drupal 7 -> 8 test migratie
2014 11 20 Drupal 7 -> 8 test migratiehcderaad
 
Nelson: Rigorous Deployment for a Functional World
Nelson: Rigorous Deployment for a Functional WorldNelson: Rigorous Deployment for a Functional World
Nelson: Rigorous Deployment for a Functional WorldTimothy Perrett
 
The "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/OpsThe "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/OpsErik Osterman
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsMike Brittain
 
Deep dive - Concourse CI/CD and Pipelines
Deep dive  - Concourse CI/CD and PipelinesDeep dive  - Concourse CI/CD and Pipelines
Deep dive - Concourse CI/CD and PipelinesSyed Imam
 
How to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringHow to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringAndrew Kirkpatrick
 
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)Tim Bunce
 

Similar to Developer-friendly taskqueues: What you should ask yourself before choosing one (20)

Intro to DevOps
Intro to DevOpsIntro to DevOps
Intro to DevOps
 
Modern day jvm controversies
Modern day jvm controversiesModern day jvm controversies
Modern day jvm controversies
 
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
Monitoring Hadoop with Prometheus (Hadoop User Group Ireland, December 2015)
 
The Best Feature of Go – A 5 Year Retrospective
The Best Feature of Go – A 5 Year RetrospectiveThe Best Feature of Go – A 5 Year Retrospective
The Best Feature of Go – A 5 Year Retrospective
 
Node, can you even in CPU intensive operations?
Node, can you even in CPU intensive operations?Node, can you even in CPU intensive operations?
Node, can you even in CPU intensive operations?
 
DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.DevOps Fest 2020. immutable infrastructure as code. True story.
DevOps Fest 2020. immutable infrastructure as code. True story.
 
PowerShell - Be A Cool Blue Kid
PowerShell - Be A Cool Blue KidPowerShell - Be A Cool Blue Kid
PowerShell - Be A Cool Blue Kid
 
Software Engineering Best Practices @ Nylas
Software Engineering Best Practices @ NylasSoftware Engineering Best Practices @ Nylas
Software Engineering Best Practices @ Nylas
 
Human scaling on the front end
Human scaling on the front endHuman scaling on the front end
Human scaling on the front end
 
To AWS with Ansible
To AWS with AnsibleTo AWS with Ansible
To AWS with Ansible
 
Optimizing Performance - Clojure Remote - Nikola Peric
Optimizing Performance - Clojure Remote - Nikola PericOptimizing Performance - Clojure Remote - Nikola Peric
Optimizing Performance - Clojure Remote - Nikola Peric
 
Nyt Prof 200910
Nyt Prof 200910Nyt Prof 200910
Nyt Prof 200910
 
Faster PHP apps using Queues and Workers
Faster PHP apps using Queues and WorkersFaster PHP apps using Queues and Workers
Faster PHP apps using Queues and Workers
 
2014 11 20 Drupal 7 -> 8 test migratie
2014 11 20 Drupal 7 -> 8 test migratie2014 11 20 Drupal 7 -> 8 test migratie
2014 11 20 Drupal 7 -> 8 test migratie
 
Nelson: Rigorous Deployment for a Functional World
Nelson: Rigorous Deployment for a Functional WorldNelson: Rigorous Deployment for a Functional World
Nelson: Rigorous Deployment for a Functional World
 
The "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/OpsThe "Holy Grail" of Dev/Ops
The "Holy Grail" of Dev/Ops
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details
 
Deep dive - Concourse CI/CD and Pipelines
Deep dive  - Concourse CI/CD and PipelinesDeep dive  - Concourse CI/CD and Pipelines
Deep dive - Concourse CI/CD and Pipelines
 
How to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringHow to get started with Site Reliability Engineering
How to get started with Site Reliability Engineering
 
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)
Devel::NYTProf v3 - 200908 (OUTDATED, see 201008)
 

More from Sylvain Zimmer

Ranking the Web with Spark
Ranking the Web with SparkRanking the Web with Spark
Ranking the Web with SparkSylvain Zimmer
 
The original vision of Nutch, 14 years later: Building an open source search ...
The original vision of Nutch, 14 years later: Building an open source search ...The original vision of Nutch, 14 years later: Building an open source search ...
The original vision of Nutch, 14 years later: Building an open source search ...Sylvain Zimmer
 
PyCon FR 2016 - Et si on recodait Google en Python ?
PyCon FR 2016 - Et si on recodait Google en Python ?PyCon FR 2016 - Et si on recodait Google en Python ?
PyCon FR 2016 - Et si on recodait Google en Python ?Sylvain Zimmer
 
[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013
[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013
[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013Sylvain Zimmer
 
140byt.es - The Dark Side of Javascript
140byt.es - The Dark Side of Javascript140byt.es - The Dark Side of Javascript
140byt.es - The Dark Side of JavascriptSylvain Zimmer
 
Joshfire Framework 0.9 Technical Overview
Joshfire Framework 0.9 Technical OverviewJoshfire Framework 0.9 Technical Overview
Joshfire Framework 0.9 Technical OverviewSylvain Zimmer
 
Javascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJSJavascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJSSylvain Zimmer
 
no.de quick presentation at #ParisJS 4
no.de quick presentation at #ParisJS 4no.de quick presentation at #ParisJS 4
no.de quick presentation at #ParisJS 4Sylvain Zimmer
 
Kinect + Javascript tech talk at #ParisJS Jan 2011
Kinect + Javascript tech talk at #ParisJS Jan 2011Kinect + Javascript tech talk at #ParisJS Jan 2011
Kinect + Javascript tech talk at #ParisJS Jan 2011Sylvain Zimmer
 
Web Crawling with NodeJS
Web Crawling with NodeJSWeb Crawling with NodeJS
Web Crawling with NodeJSSylvain Zimmer
 
Archicamp présentation
Archicamp présentationArchicamp présentation
Archicamp présentationSylvain Zimmer
 
Twisted presentation & Jamendo usecases
Twisted presentation & Jamendo usecasesTwisted presentation & Jamendo usecases
Twisted presentation & Jamendo usecasesSylvain Zimmer
 

More from Sylvain Zimmer (12)

Ranking the Web with Spark
Ranking the Web with SparkRanking the Web with Spark
Ranking the Web with Spark
 
The original vision of Nutch, 14 years later: Building an open source search ...
The original vision of Nutch, 14 years later: Building an open source search ...The original vision of Nutch, 14 years later: Building an open source search ...
The original vision of Nutch, 14 years later: Building an open source search ...
 
PyCon FR 2016 - Et si on recodait Google en Python ?
PyCon FR 2016 - Et si on recodait Google en Python ?PyCon FR 2016 - Et si on recodait Google en Python ?
PyCon FR 2016 - Et si on recodait Google en Python ?
 
[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013
[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013
[fr] Introduction et Live-code Backbone.js à DevoxxFR 2013
 
140byt.es - The Dark Side of Javascript
140byt.es - The Dark Side of Javascript140byt.es - The Dark Side of Javascript
140byt.es - The Dark Side of Javascript
 
Joshfire Framework 0.9 Technical Overview
Joshfire Framework 0.9 Technical OverviewJoshfire Framework 0.9 Technical Overview
Joshfire Framework 0.9 Technical Overview
 
Javascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJSJavascript Views, Client-side or Server-side with NodeJS
Javascript Views, Client-side or Server-side with NodeJS
 
no.de quick presentation at #ParisJS 4
no.de quick presentation at #ParisJS 4no.de quick presentation at #ParisJS 4
no.de quick presentation at #ParisJS 4
 
Kinect + Javascript tech talk at #ParisJS Jan 2011
Kinect + Javascript tech talk at #ParisJS Jan 2011Kinect + Javascript tech talk at #ParisJS Jan 2011
Kinect + Javascript tech talk at #ParisJS Jan 2011
 
Web Crawling with NodeJS
Web Crawling with NodeJSWeb Crawling with NodeJS
Web Crawling with NodeJS
 
Archicamp présentation
Archicamp présentationArchicamp présentation
Archicamp présentation
 
Twisted presentation & Jamendo usecases
Twisted presentation & Jamendo usecasesTwisted presentation & Jamendo usecases
Twisted presentation & Jamendo usecases
 

Recently uploaded

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxMasterG
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingScyllaDB
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewDianaGray10
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdfMuhammad Subhan
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTopCSSGallery
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctBrainSell Technologies
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuidePixlogix Infotech
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?Paolo Missier
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...ScyllaDB
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxFIDO Alliance
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهMohamed Sweelam
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxFIDO Alliance
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistandanishmna97
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard37
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityVictorSzoltysek
 

Recently uploaded (20)

Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
Event-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream ProcessingEvent-Driven Architecture Masterclass: Challenges in Stream Processing
Event-Driven Architecture Masterclass: Challenges in Stream Processing
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
“Iamnobody89757” Understanding the Mysterious of Digital Identity.pdf
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
ERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage IntacctERP Contender Series: Acumatica vs. Sage Intacct
ERP Contender Series: Acumatica vs. Sage Intacct
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
(Explainable) Data-Centric AI: what are you explaininhg, and to whom?
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptxHarnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
Harnessing Passkeys in the Battle Against AI-Powered Cyber Threats.pptx
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistan
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 

Developer-friendly taskqueues: What you should ask yourself before choosing one

  • 1. Sylvain Zimmer / @sylvinus PyParis 2017 DEVELOPER-FRIENDLY TASKQUEUES WHAT WE LEARNED BUILDING MRQ & WHAT YOU SHOULD ASK YOURSELF BEFORE CHOOSING ONE
  • 2. /usr/bin/whoami ▸ (SpaceX nerd) ▸ Founder, dotConferences ▸ CTO Pricing Assistant ▸ Co-organizer Paris.py meetup ▸ User of Python taskqueues for 10+ years ▸ Main contributor of MRQ
  • 5. Credit: Adrien Di Pasquale
  • 6. A typical job/task def send_an_email(email_type, user): html = template(email_type, user) status = email.send(html, user["email"]) metrics.send("email_%s" % status, 1) return status KERNEL PANIC
  • 7. Task properties Re-entrant Idempotent Nullipotent< < ▸ Safe to interrupt and then retry ▸ Safe to call multiple times ▸ Result will be the same ▸ Free of side-effects def reentrant(a): value = a + random() db.insert(value) def idempotent(key, value): db.update(key, value) def nullipotent(a): return a ** 2
  • 8. Other task properties & best practices ▸ Serializable args, serializable result ▸ Args validation / documentation ▸ Least args possible ▸ Canonical path vs. registration ▸ Concurrent safety ▸ Statuses
  • 9. Coroutines vs. Threads vs. Processes ▸ IO-bound tasks vs. CPU-bound tasks ▸ Threads offer few benefits for a Python worker (GIL) ▸ Coroutines/Greenlets are ideal for IO-bound tasks ▸ Processes are required for CPU-bound tasks ▸ If you have heterogenous tasks, your TQ should support both! $ mrq-worker --greenlets 25 --processes 4
  • 10. Performance: latency & throughput APP BROKER WORKER RESULT STORE
  • 12.
  • 13. Errors ▸ Exception handlers ▸ Timeouts ▸ Retry rules ▸ Sentry & friends ▸ gevent: test your tracebacks! ▸ Priorities ▸ Human process to manage failed tasks!
  • 14. Task visibility ▸ Tasks by status, path, worker, ... ▸ Tracebacks & current stack ▸ Logs ▸ Timing info ▸ Cancel / Kill / Move tasks ▸ Progress
  • 15. Memory leaks ▸ Workers = long-running processes ▸ gevent makes debugging harder ▸ Watch out for global variables or mutable class attributes! ▸ Python's ecosystem is surprisingly poor in this area ▸ guppy, objgraph can usually help
  • 16. Misc tools ▸ Scheduler ▸ Command-line runner, e.g. mrq-run tasks.myTask {"a": 1} ▸ Autoscaling ▸ Profiler
  • 18. Consistency guarantees ▸ At least once vs. At most once vs. Exactly once ▸ Ordering ▸ Critical operations: ▸ Queueing ▸ Marking tasks as started ▸ Timeouts & retries
  • 19. Types of brokers ▸ Specialized message queues (RabbitMQ, SNS, Kafka, ...) ▸ Performance, complexity, poor visibility ▸ In-memory data stores (Redis, ...) ▸ Performance, simplicity, harder to scale ▸ Regular databases (MongoDB, PostgreSQL, ...) ▸ Often enough for the job!
  • 20. At the heart of the broker ▸ Atomic update from "queued" to "started" ▸ MRQ with MongoDB broker: find_one_and_update() ▸ MRQ with Redis broker: Pushback in a ZSET
  • 21. ZSETs in Redis ▸ Sorted sets with O(log(N)) scalability ▸ set/get by key, order by key, lookups by key or value ▸ Very interesting properties for task queues: Unicity, Ordering, Atomicity of updates, Performance ▸ MRQ's "Pushback" model: ▸ Queue with key=timestamp ▸ Unqueue by fetching key range & setting new keys in the future ▸ After completion the task adjusts or removes the key
  • 22.
  • 23. Consistency guarantees ▸ Must be thought of for the whole system, not just the broker! ▸ Brokers can be misused or misconfigured ▸ The workers can drop tasks if they want to ;-) ▸ Consistency starts at queueing time!
  • 25. Think hard about what you need ▸ Will your taskqueue be the foundation of your architecture, or is it just a side project? ▸ What performance do you need? (IO vs. CPU, latency, ...) ▸ What level of visibility and control do you need on queued & running tasks? ▸ Can workers terminate abruptly? Lots of design consequences! ▸ What language interop do you need?
  • 26. And then all the usual questions... ▸ Is it supported by a lively community? ▸ License ▸ Documentation ▸ Future plans
  • 27. Which one to pick? ▸ Celery: High performance, large community, very complex, major upgrades painful ▸ RQ: Extremely simple to understand, low performance ▸ MRQ: Adjust task visibility vs. performance, simple to understand, 1.0 soon ▸ Lots of other valid options! Just be sure to ask yourself the right questions ;-)
  • 28. BE GRATEFUL FOR THE OSS YOU USE! REMINDER