SlideShare a Scribd company logo
Working with
Asynchronous Events
Jan Gregor Emge-Triebel (@jan0707)
Dennis Oehme (@dennisoehme)
Garden of Concepts GmbH
● Hanau / Munich
● In house CMS “.stone”
● In house CRM “Elvis”
● Various individual customer projects
● Symfony based development, supported by:
○ MongoDB, ElasticSearch, MySQL, RabbitMQ,
Redis, AngularJS, Twig, Jade, Gitlab, Docker,
Vagrant, Ansible
● Say hi @goc_mediastudio
Requests == Event
“Symfony is a request based
framework.”
Fabien Potencier, Symfony Live 2014
@fabpot
Requests
Request Controller Logic 1 Logic 2 Response
Request Controller
Logic 1
Logic 2
Response
Synchronous
Asynchronous
Why do we need asynchronous event processing ?
● Move long running logic
● Cut down reaction times
● Transfer load
● Microservices
● Exchange data/messages with other (sub-)systems
● Parallelization
● Scaling
Our journey
● Gearman
● (PHP) Resque
○ Demo
● RabbitMQ
Gearman
● Gearman Job Server (written in C)
● Runs as a Linux daemon
● Requires PECL extension for worker API
● Worker APIs exists for other Languages (C, Perl,...)
Gearman
http://gearman.org/examples/send-emails/
Gearman
● Latest PECL-Extension version is 1.1.2 (August 2013)
● Latest Server-Release (spring 2014)
● No more process is being made
● Gearman daemon and extension are often incompatible
(PHP) Resque
● Is a Resque port to php:
○ Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple
queues, and processing them later.
● Needs Redis(-Cluster)
● Redis is capable of atomic transactions (FIFO)
Demo
https://github.com/Jan0707/async-symfony
(PHP) Resque
● Hardly any commits since 2012
● Last commit to Master on 2015-05-12
● chrisboulton/php-resque & michelsalib/BCCResqueBundle must be presumed
dead ?
RabbitMQ
Or why this talk should be named:
“Working with Asynchronous Events in RabbitMQ”
RabbitMQ
● Message Broker written in Erlang
● AMQP (Advanced Message Queuing Protocol)
● Open, but binary network protocoll
● Routing, message distribution
● Transactions
● Plus, more nice features
○ TTL
○ DLX (Dead Letter eXchange)
○ PreFetching
Of publishers and consumers ...
Publisher Publish Consumer
Ex-
change
QueueRoutes Consumes
There is a Bundle for it
● https://github.com/php-amqplib/RabbitMqBundle
● Consumer and Producer as Symfony services
● Every queue Consumer runs as a Symfony command
● Configuration via app/config.yml
○ Better: Create app/config/rabbitmq.yml and import in app/config/config.yml
app/config/rabbitmq.yml
Symfony Producer
Symfony Consumer
Lessons Learned: Consumer
● PHP threads are not meant to be long running
○ New problems: memory leaks, ...
● Use supervisor to check and restart Consumers
● Open database connections, timeouts
○ Better quit when inactive
● Cap maximum runtime per thread
○ Automatically shutdown after processign x messages
● Memory limit: RabbitMQ-Bundle utilizes “soft-quota”
● Booting consumers takes rather long
● Prefetching (Warning: Order!)
Lessons Learned: Message
● Compact/short messages
● Queues eat up a lot of storage / RAM
● Common format such as JSON or XML
○ Do not serialize PHP objects or arrays
○ Ensure interoperability with other systems and languages
Lessons Learned: Message Transformation
● Use JMS/Serializer to convert objects to JSON
○ Deserialize JSON to PHP objects in consumer
○ Objects can and should be validated (with symfony forms and asserts)
○ Reload entities / documents to keep them up to date (doctrine/unit of work)
● Alternative approach: PHPs JsonSerializable Interface
Lessons Learned: Message Header
● Use message “Header” for content-type and version
○ Message formats can change
○ consumers must remain compatible
● Unqiue IDs make logging / debugging / live easier
● Other meta data:
○ timestamp
○ user / session
○ App id
○ ...
Tip: Custom Consumer-/Producer-Classes
● Message transformation
● Events
● Logging
Transactions
● Messages must be brokered to exactly one Consumer
● Consumer must explicitly answer a message with “acknowledge”
● If not, the message will be marked “rejected” and be requeued
○ Beware of requeuing:
■ Can lead to toxic messages
■ Can change the order of messages
○ Beware of final rejection:
■ Can lead to loss of data
Toxic Messages
● If a message is not explicitly being answered, it will be requeued.
● If that happens multiple times :
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
Dead Letter Exchange
● RabbitMQ comes with a Dead Letter Exchange (DLX)
● Every queue determines its own or shared DLX
● DLX is a separate queue, storing all rejected messages
○ (Reject, TTL, Prefetch)
app/config/rabbitmq.yml
Signal Handling
● Graceful Shutdown
● pcntl_signal / _dispatch
● SIGTERM
● FATAL ERROR abfangen
● Logging
● Toxic Message-Handling
Shutdown
Deployment
● Consumers need an up to date code base
● Consumers need a graceful shutdown
● Consumers must only be shutdown when idling
○ Otherwise : Loss of message / data
Other challenges
● Significant change of infrastructure
○ Choose your libraries and dependencies carefully!
● Moving load is not removing load
● Rethinking necessary
○ Display of status / intermediate result
○ Websockets or Push instead of Polling
● Interdependencies of messages
○ Especially when running parallel consumers!
● Performance (Doctrine ORM / ODM)
Questions ?
Thank you :)
Find this and other talks at: https://www.gardenofconcepts.
com/talks/
Feedback is always welcome
& please rate our talk !

More Related Content

What's hot

VXCON 2017
VXCON 2017VXCON 2017
VXCON 2017
Kelvin Chan
 
Event Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQEvent Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQ
Luke Luo
 
ZeroMQ with NodeJS
ZeroMQ with NodeJSZeroMQ with NodeJS
ZeroMQ with NodeJS
Fernando Sanabria
 
Rust Primer
Rust PrimerRust Primer
Rust Primer
Knoldus Inc.
 
Rabbit MQ
Rabbit MQRabbit MQ
Rabbit MQ
Tran Thanh Thi
 
Practical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profilingPractical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profiling
Lubomir Rintel
 
WebAssembly: A New World of Native Exploits on the Browser
WebAssembly: A New World of Native Exploits on the BrowserWebAssembly: A New World of Native Exploits on the Browser
WebAssembly: A New World of Native Exploits on the Browser
Priyanka Aash
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?
ScyllaDB
 
Leveraging zeromq for node.js
Leveraging zeromq for node.jsLeveraging zeromq for node.js
Leveraging zeromq for node.js
Ruben Tan
 
2013 - Brian Stanley - Memcached, Cached all the things
2013 - Brian Stanley - Memcached, Cached all the things2013 - Brian Stanley - Memcached, Cached all the things
2013 - Brian Stanley - Memcached, Cached all the things
PHP Conference Argentina
 
Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQ
pieterh
 
Distributed app development with nodejs and zeromq
Distributed app development with nodejs and zeromqDistributed app development with nodejs and zeromq
Distributed app development with nodejs and zeromq
Ruben Tan
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQ
fcrippa
 
Let's Talk Locks!
Let's Talk Locks!Let's Talk Locks!
Let's Talk Locks!
C4Media
 
The art of concurrent programming
The art of concurrent programmingThe art of concurrent programming
The art of concurrent programming
Iskren Chernev
 
Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...
Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...
Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...
.NET Crowd
 
PostgreSQL: present and near future
PostgreSQL: present and near futurePostgreSQL: present and near future
PostgreSQL: present and near future
NaN-tic
 
Vagrantfordevops
VagrantfordevopsVagrantfordevops
Vagrantfordevops
Deepanshu Gajbhiye
 
ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!
Pedro Januário
 
Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)
Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)
Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)
Anne Nicolas
 

What's hot (20)

VXCON 2017
VXCON 2017VXCON 2017
VXCON 2017
 
Event Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQEvent Driven with LibUV and ZeroMQ
Event Driven with LibUV and ZeroMQ
 
ZeroMQ with NodeJS
ZeroMQ with NodeJSZeroMQ with NodeJS
ZeroMQ with NodeJS
 
Rust Primer
Rust PrimerRust Primer
Rust Primer
 
Rabbit MQ
Rabbit MQRabbit MQ
Rabbit MQ
 
Practical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profilingPractical SystemTAP basics: Perl memory profiling
Practical SystemTAP basics: Perl memory profiling
 
WebAssembly: A New World of Native Exploits on the Browser
WebAssembly: A New World of Native Exploits on the BrowserWebAssembly: A New World of Native Exploits on the Browser
WebAssembly: A New World of Native Exploits on the Browser
 
Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?Rust Is Safe. But Is It Fast?
Rust Is Safe. But Is It Fast?
 
Leveraging zeromq for node.js
Leveraging zeromq for node.jsLeveraging zeromq for node.js
Leveraging zeromq for node.js
 
2013 - Brian Stanley - Memcached, Cached all the things
2013 - Brian Stanley - Memcached, Cached all the things2013 - Brian Stanley - Memcached, Cached all the things
2013 - Brian Stanley - Memcached, Cached all the things
 
Overview of ZeroMQ
Overview of ZeroMQOverview of ZeroMQ
Overview of ZeroMQ
 
Distributed app development with nodejs and zeromq
Distributed app development with nodejs and zeromqDistributed app development with nodejs and zeromq
Distributed app development with nodejs and zeromq
 
Europycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQEuropycon2011: Implementing distributed application using ZeroMQ
Europycon2011: Implementing distributed application using ZeroMQ
 
Let's Talk Locks!
Let's Talk Locks!Let's Talk Locks!
Let's Talk Locks!
 
The art of concurrent programming
The art of concurrent programmingThe art of concurrent programming
The art of concurrent programming
 
Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...
Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...
Tomas Urbonaitis "Introduction to asynchronous persistent messaging with NSer...
 
PostgreSQL: present and near future
PostgreSQL: present and near futurePostgreSQL: present and near future
PostgreSQL: present and near future
 
Vagrantfordevops
VagrantfordevopsVagrantfordevops
Vagrantfordevops
 
ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!ZeroMQ - Sockets on steroids!
ZeroMQ - Sockets on steroids!
 
Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)
Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)
Kernel Recipes 2016 - kernelci.org: 1.5 million kernel boots (and counting)
 

Viewers also liked

Telephony with OpenShift Twilio and MongoDB
Telephony with OpenShift Twilio and MongoDBTelephony with OpenShift Twilio and MongoDB
Telephony with OpenShift Twilio and MongoDB
Mark Atwood
 
Juc boston2014.pptx
Juc boston2014.pptxJuc boston2014.pptx
Juc boston2014.pptx
Brandon Mueller
 
Theres a rabbit on my symfony
Theres a rabbit on my symfonyTheres a rabbit on my symfony
Theres a rabbit on my symfony
Alvaro Videla
 
Asynchronous processing with PHP and Symfony2. Do it simple
Asynchronous processing with PHP and Symfony2. Do it simpleAsynchronous processing with PHP and Symfony2. Do it simple
Asynchronous processing with PHP and Symfony2. Do it simple
Kirill Chebunin
 
Scaling applications with RabbitMQ at SunshinePHP
Scaling applications with RabbitMQ   at SunshinePHPScaling applications with RabbitMQ   at SunshinePHP
Scaling applications with RabbitMQ at SunshinePHP
Alvaro Videla
 
Service Oriented Architecture for Distributed Identity Management System
Service Oriented Architecture for Distributed Identity Management SystemService Oriented Architecture for Distributed Identity Management System
Service Oriented Architecture for Distributed Identity Management System
adameq
 
Laravel - 系統全攻略
Laravel - 系統全攻略Laravel - 系統全攻略
Laravel - 系統全攻略
Vincent Chi
 
Scaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK MeetupScaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Kacper Gunia
 
PHP is the King, nodejs the prince and python the fool
PHP is the King, nodejs the prince and python the foolPHP is the King, nodejs the prince and python the fool
PHP is the King, nodejs the prince and python the fool
Alessandro Cinelli (cirpo)
 
Interoperability With RabbitMq
Interoperability With RabbitMqInteroperability With RabbitMq
Interoperability With RabbitMq
Alvaro Videla
 
PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試
PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試
PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試
亮亮 閃
 
Laravel - 系統全攻略(續)
Laravel - 系統全攻略(續)Laravel - 系統全攻略(續)
Laravel - 系統全攻略(續)
Vincent Chi
 
Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...
Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...
Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...
Alessandro Nadalin
 
RESTful API的设计与开发
RESTful API的设计与开发RESTful API的设计与开发
RESTful API的设计与开发
Ace Lee
 
PHPSpec - the only Design Tool you need - 4Developers
PHPSpec - the only Design Tool you need - 4DevelopersPHPSpec - the only Design Tool you need - 4Developers
PHPSpec - the only Design Tool you need - 4Developers
Kacper Gunia
 
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQL
Joe Stump
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Aaron Saray
 
Speed up your Symfony2 application and build awesome features with Redis
Speed up your Symfony2 application and build awesome features with RedisSpeed up your Symfony2 application and build awesome features with Redis
Speed up your Symfony2 application and build awesome features with Redis
Ricard Clau
 
KSDG BaaS Intro
KSDG BaaS IntroKSDG BaaS Intro
KSDG BaaS Intro
ericpi Bi
 
Write microservice in golang
Write microservice in golangWrite microservice in golang
Write microservice in golang
Bo-Yi Wu
 

Viewers also liked (20)

Telephony with OpenShift Twilio and MongoDB
Telephony with OpenShift Twilio and MongoDBTelephony with OpenShift Twilio and MongoDB
Telephony with OpenShift Twilio and MongoDB
 
Juc boston2014.pptx
Juc boston2014.pptxJuc boston2014.pptx
Juc boston2014.pptx
 
Theres a rabbit on my symfony
Theres a rabbit on my symfonyTheres a rabbit on my symfony
Theres a rabbit on my symfony
 
Asynchronous processing with PHP and Symfony2. Do it simple
Asynchronous processing with PHP and Symfony2. Do it simpleAsynchronous processing with PHP and Symfony2. Do it simple
Asynchronous processing with PHP and Symfony2. Do it simple
 
Scaling applications with RabbitMQ at SunshinePHP
Scaling applications with RabbitMQ   at SunshinePHPScaling applications with RabbitMQ   at SunshinePHP
Scaling applications with RabbitMQ at SunshinePHP
 
Service Oriented Architecture for Distributed Identity Management System
Service Oriented Architecture for Distributed Identity Management SystemService Oriented Architecture for Distributed Identity Management System
Service Oriented Architecture for Distributed Identity Management System
 
Laravel - 系統全攻略
Laravel - 系統全攻略Laravel - 系統全攻略
Laravel - 系統全攻略
 
Scaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK MeetupScaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
Scaling Symfony2 apps with RabbitMQ - Symfony UK Meetup
 
PHP is the King, nodejs the prince and python the fool
PHP is the King, nodejs the prince and python the foolPHP is the King, nodejs the prince and python the fool
PHP is the King, nodejs the prince and python the fool
 
Interoperability With RabbitMq
Interoperability With RabbitMqInteroperability With RabbitMq
Interoperability With RabbitMq
 
PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試
PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試
PHP也有day #27 - From apprentice to artisan 解耦合處理程序及單元測試
 
Laravel - 系統全攻略(續)
Laravel - 系統全攻略(續)Laravel - 系統全攻略(續)
Laravel - 系統全攻略(續)
 
Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...
Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...
Tips and Tricks for your Service Oriented Architecture @ CakeFest 2013 in San...
 
RESTful API的设计与开发
RESTful API的设计与开发RESTful API的设计与开发
RESTful API的设计与开发
 
PHPSpec - the only Design Tool you need - 4Developers
PHPSpec - the only Design Tool you need - 4DevelopersPHPSpec - the only Design Tool you need - 4Developers
PHPSpec - the only Design Tool you need - 4Developers
 
Services Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQLServices Oriented Architecture with PHP and MySQL
Services Oriented Architecture with PHP and MySQL
 
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
Enterprise PHP Architecture through Design Patterns and Modularization (Midwe...
 
Speed up your Symfony2 application and build awesome features with Redis
Speed up your Symfony2 application and build awesome features with RedisSpeed up your Symfony2 application and build awesome features with Redis
Speed up your Symfony2 application and build awesome features with Redis
 
KSDG BaaS Intro
KSDG BaaS IntroKSDG BaaS Intro
KSDG BaaS Intro
 
Write microservice in golang
Write microservice in golangWrite microservice in golang
Write microservice in golang
 

Similar to Working with Asynchronous Events

PHP at Density and Scale (Lone Star PHP 2014)
PHP at Density and Scale (Lone Star PHP 2014)PHP at Density and Scale (Lone Star PHP 2014)
PHP at Density and Scale (Lone Star PHP 2014)
David Timothy Strauss
 
WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...
WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...
WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...
WebCamp
 
AMQP with RabbitMQ
AMQP with RabbitMQAMQP with RabbitMQ
AMQP with RabbitMQ
Spyros Papageorgiou
 
MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ
MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ
MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ
Jitendra Bafna
 
Enlightenment as Standalone Wayland Compositor
Enlightenment as Standalone Wayland CompositorEnlightenment as Standalone Wayland Compositor
Enlightenment as Standalone Wayland Compositor
Samsung Open Source Group
 
WebCamp Ukraine 2016: Instant messenger with Python. Back-end development
WebCamp Ukraine 2016: Instant messenger with Python. Back-end developmentWebCamp Ukraine 2016: Instant messenger with Python. Back-end development
WebCamp Ukraine 2016: Instant messenger with Python. Back-end development
Viach Kakovskyi
 
What we've learned from running thousands of production RabbitMQ clusters - L...
What we've learned from running thousands of production RabbitMQ clusters - L...What we've learned from running thousands of production RabbitMQ clusters - L...
What we've learned from running thousands of production RabbitMQ clusters - L...
RabbitMQ Summit
 
Leveraging Android's Linux Heritage
Leveraging Android's Linux HeritageLeveraging Android's Linux Heritage
Leveraging Android's Linux Heritage
Opersys inc.
 
Leveraging Android's Linux Heritage at ELC-E 2011
Leveraging Android's Linux Heritage at ELC-E 2011Leveraging Android's Linux Heritage at ELC-E 2011
Leveraging Android's Linux Heritage at ELC-E 2011
Opersys inc.
 
Server fleet management using Camunda by Akhil Ahuja
Server fleet management using Camunda by Akhil AhujaServer fleet management using Camunda by Akhil Ahuja
Server fleet management using Camunda by Akhil Ahuja
camunda services GmbH
 
Play Framework
Play FrameworkPlay Framework
Play Framework
Eduard Tudenhoefner
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logs
Jeremy Cook
 
Patna_Meetup_MQ
Patna_Meetup_MQPatna_Meetup_MQ
Patna_Meetup_MQ
Om Prakash
 
Efficient Buffer Management
Efficient Buffer ManagementEfficient Buffer Management
Efficient Buffer Management
basisspace
 
10 Things Every Developer Using RabbitMQ Should Know
10 Things Every Developer Using RabbitMQ Should Know10 Things Every Developer Using RabbitMQ Should Know
10 Things Every Developer Using RabbitMQ Should Know
VMware Tanzu
 
Netty training
Netty trainingNetty training
Zing Me Real Time Web Chat Architect
Zing Me Real Time Web Chat ArchitectZing Me Real Time Web Chat Architect
Zing Me Real Time Web Chat Architect
Chau Thanh
 
Task migration using CRIU
Task migration using CRIUTask migration using CRIU
Task migration using CRIU
Rohit Jnagal
 
The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"
Knoldus Inc.
 
Event driven architectures with Kinesis
Event driven architectures with KinesisEvent driven architectures with Kinesis
Event driven architectures with Kinesis
Mark Harrison
 

Similar to Working with Asynchronous Events (20)

PHP at Density and Scale (Lone Star PHP 2014)
PHP at Density and Scale (Lone Star PHP 2014)PHP at Density and Scale (Lone Star PHP 2014)
PHP at Density and Scale (Lone Star PHP 2014)
 
WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...
WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...
WebCamp 2016: Python. Вячеслав Каковский: Real-time мессенджер на Python. Осо...
 
AMQP with RabbitMQ
AMQP with RabbitMQAMQP with RabbitMQ
AMQP with RabbitMQ
 
MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ
MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ
MuleSoft Surat Virtual Meetup#33 - Unleash the power of Anypoint MQ and DLQ
 
Enlightenment as Standalone Wayland Compositor
Enlightenment as Standalone Wayland CompositorEnlightenment as Standalone Wayland Compositor
Enlightenment as Standalone Wayland Compositor
 
WebCamp Ukraine 2016: Instant messenger with Python. Back-end development
WebCamp Ukraine 2016: Instant messenger with Python. Back-end developmentWebCamp Ukraine 2016: Instant messenger with Python. Back-end development
WebCamp Ukraine 2016: Instant messenger with Python. Back-end development
 
What we've learned from running thousands of production RabbitMQ clusters - L...
What we've learned from running thousands of production RabbitMQ clusters - L...What we've learned from running thousands of production RabbitMQ clusters - L...
What we've learned from running thousands of production RabbitMQ clusters - L...
 
Leveraging Android's Linux Heritage
Leveraging Android's Linux HeritageLeveraging Android's Linux Heritage
Leveraging Android's Linux Heritage
 
Leveraging Android's Linux Heritage at ELC-E 2011
Leveraging Android's Linux Heritage at ELC-E 2011Leveraging Android's Linux Heritage at ELC-E 2011
Leveraging Android's Linux Heritage at ELC-E 2011
 
Server fleet management using Camunda by Akhil Ahuja
Server fleet management using Camunda by Akhil AhujaServer fleet management using Camunda by Akhil Ahuja
Server fleet management using Camunda by Akhil Ahuja
 
Play Framework
Play FrameworkPlay Framework
Play Framework
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logs
 
Patna_Meetup_MQ
Patna_Meetup_MQPatna_Meetup_MQ
Patna_Meetup_MQ
 
Efficient Buffer Management
Efficient Buffer ManagementEfficient Buffer Management
Efficient Buffer Management
 
10 Things Every Developer Using RabbitMQ Should Know
10 Things Every Developer Using RabbitMQ Should Know10 Things Every Developer Using RabbitMQ Should Know
10 Things Every Developer Using RabbitMQ Should Know
 
Netty training
Netty trainingNetty training
Netty training
 
Zing Me Real Time Web Chat Architect
Zing Me Real Time Web Chat ArchitectZing Me Real Time Web Chat Architect
Zing Me Real Time Web Chat Architect
 
Task migration using CRIU
Task migration using CRIUTask migration using CRIU
Task migration using CRIU
 
The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"The Beam Vision for Portability: "Write once run anywhere"
The Beam Vision for Portability: "Write once run anywhere"
 
Event driven architectures with Kinesis
Event driven architectures with KinesisEvent driven architectures with Kinesis
Event driven architectures with Kinesis
 

Recently uploaded

Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Green Software Development
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
Ayan Halder
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Julian Hyde
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
Hornet Dynamics
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptxLORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
lorraineandreiamcidl
 
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdfRevolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
Undress Baby
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
What is Master Data Management by PiLog Group
What is Master Data Management by PiLog GroupWhat is Master Data Management by PiLog Group
What is Master Data Management by PiLog Group
aymanquadri279
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Neo4j
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Crescat
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
ICS
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
lorraineandreiamcidl
 
Empowering Growth with Best Software Development Company in Noida - Deuglo
Empowering Growth with Best Software  Development Company in Noida - DeugloEmpowering Growth with Best Software  Development Company in Noida - Deuglo
Empowering Growth with Best Software Development Company in Noida - Deuglo
Deuglo Infosystem Pvt Ltd
 
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeA Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
Aftab Hussain
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
kalichargn70th171
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
Philip Schwarz
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
Hironori Washizaki
 

Recently uploaded (20)

Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
GreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-JurisicGreenCode-A-VSCode-Plugin--Dario-Jurisic
GreenCode-A-VSCode-Plugin--Dario-Jurisic
 
Using Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional SafetyUsing Xen Hypervisor for Functional Safety
Using Xen Hypervisor for Functional Safety
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 
Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)Measures in SQL (SIGMOD 2024, Santiago, Chile)
Measures in SQL (SIGMOD 2024, Santiago, Chile)
 
E-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet DynamicsE-commerce Development Services- Hornet Dynamics
E-commerce Development Services- Hornet Dynamics
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptxLORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
 
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdfRevolutionizing Visual Effects Mastering AI Face Swaps.pdf
Revolutionizing Visual Effects Mastering AI Face Swaps.pdf
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
What is Master Data Management by PiLog Group
What is Master Data Management by PiLog GroupWhat is Master Data Management by PiLog Group
What is Master Data Management by PiLog Group
 
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissancesAtelier - Innover avec l’IA Générative et les graphes de connaissances
Atelier - Innover avec l’IA Générative et les graphes de connaissances
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
 
Webinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for EmbeddedWebinar On-Demand: Using Flutter for Embedded
Webinar On-Demand: Using Flutter for Embedded
 
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOMLORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM
 
Empowering Growth with Best Software Development Company in Noida - Deuglo
Empowering Growth with Best Software  Development Company in Noida - DeugloEmpowering Growth with Best Software  Development Company in Noida - Deuglo
Empowering Growth with Best Software Development Company in Noida - Deuglo
 
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of CodeA Study of Variable-Role-based Feature Enrichment in Neural Models of Code
A Study of Variable-Role-based Feature Enrichment in Neural Models of Code
 
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
 
Hand Rolled Applicative User Validation Code Kata
Hand Rolled Applicative User ValidationCode KataHand Rolled Applicative User ValidationCode Kata
Hand Rolled Applicative User Validation Code Kata
 
SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024SWEBOK and Education at FUSE Okinawa 2024
SWEBOK and Education at FUSE Okinawa 2024
 

Working with Asynchronous Events

  • 1. Working with Asynchronous Events Jan Gregor Emge-Triebel (@jan0707) Dennis Oehme (@dennisoehme)
  • 2. Garden of Concepts GmbH ● Hanau / Munich ● In house CMS “.stone” ● In house CRM “Elvis” ● Various individual customer projects ● Symfony based development, supported by: ○ MongoDB, ElasticSearch, MySQL, RabbitMQ, Redis, AngularJS, Twig, Jade, Gitlab, Docker, Vagrant, Ansible ● Say hi @goc_mediastudio
  • 3. Requests == Event “Symfony is a request based framework.” Fabien Potencier, Symfony Live 2014 @fabpot
  • 4. Requests Request Controller Logic 1 Logic 2 Response Request Controller Logic 1 Logic 2 Response Synchronous Asynchronous
  • 5. Why do we need asynchronous event processing ? ● Move long running logic ● Cut down reaction times ● Transfer load ● Microservices ● Exchange data/messages with other (sub-)systems ● Parallelization ● Scaling
  • 6. Our journey ● Gearman ● (PHP) Resque ○ Demo ● RabbitMQ
  • 7. Gearman ● Gearman Job Server (written in C) ● Runs as a Linux daemon ● Requires PECL extension for worker API ● Worker APIs exists for other Languages (C, Perl,...)
  • 9. Gearman ● Latest PECL-Extension version is 1.1.2 (August 2013) ● Latest Server-Release (spring 2014) ● No more process is being made ● Gearman daemon and extension are often incompatible
  • 10. (PHP) Resque ● Is a Resque port to php: ○ Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later. ● Needs Redis(-Cluster) ● Redis is capable of atomic transactions (FIFO)
  • 12. (PHP) Resque ● Hardly any commits since 2012 ● Last commit to Master on 2015-05-12 ● chrisboulton/php-resque & michelsalib/BCCResqueBundle must be presumed dead ?
  • 13. RabbitMQ Or why this talk should be named: “Working with Asynchronous Events in RabbitMQ”
  • 14. RabbitMQ ● Message Broker written in Erlang ● AMQP (Advanced Message Queuing Protocol) ● Open, but binary network protocoll ● Routing, message distribution ● Transactions ● Plus, more nice features ○ TTL ○ DLX (Dead Letter eXchange) ○ PreFetching
  • 15. Of publishers and consumers ... Publisher Publish Consumer Ex- change QueueRoutes Consumes
  • 16. There is a Bundle for it ● https://github.com/php-amqplib/RabbitMqBundle ● Consumer and Producer as Symfony services ● Every queue Consumer runs as a Symfony command ● Configuration via app/config.yml ○ Better: Create app/config/rabbitmq.yml and import in app/config/config.yml
  • 20. Lessons Learned: Consumer ● PHP threads are not meant to be long running ○ New problems: memory leaks, ... ● Use supervisor to check and restart Consumers ● Open database connections, timeouts ○ Better quit when inactive ● Cap maximum runtime per thread ○ Automatically shutdown after processign x messages ● Memory limit: RabbitMQ-Bundle utilizes “soft-quota” ● Booting consumers takes rather long ● Prefetching (Warning: Order!)
  • 21. Lessons Learned: Message ● Compact/short messages ● Queues eat up a lot of storage / RAM ● Common format such as JSON or XML ○ Do not serialize PHP objects or arrays ○ Ensure interoperability with other systems and languages
  • 22. Lessons Learned: Message Transformation ● Use JMS/Serializer to convert objects to JSON ○ Deserialize JSON to PHP objects in consumer ○ Objects can and should be validated (with symfony forms and asserts) ○ Reload entities / documents to keep them up to date (doctrine/unit of work) ● Alternative approach: PHPs JsonSerializable Interface
  • 23. Lessons Learned: Message Header ● Use message “Header” for content-type and version ○ Message formats can change ○ consumers must remain compatible ● Unqiue IDs make logging / debugging / live easier ● Other meta data: ○ timestamp ○ user / session ○ App id ○ ...
  • 24. Tip: Custom Consumer-/Producer-Classes ● Message transformation ● Events ● Logging
  • 25. Transactions ● Messages must be brokered to exactly one Consumer ● Consumer must explicitly answer a message with “acknowledge” ● If not, the message will be marked “rejected” and be requeued ○ Beware of requeuing: ■ Can lead to toxic messages ■ Can change the order of messages ○ Beware of final rejection: ■ Can lead to loss of data
  • 26. Toxic Messages ● If a message is not explicitly being answered, it will be requeued. ● If that happens multiple times : ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop ● Congratulations on your infinite loop
  • 27. Dead Letter Exchange ● RabbitMQ comes with a Dead Letter Exchange (DLX) ● Every queue determines its own or shared DLX ● DLX is a separate queue, storing all rejected messages ○ (Reject, TTL, Prefetch)
  • 29. Signal Handling ● Graceful Shutdown ● pcntl_signal / _dispatch ● SIGTERM ● FATAL ERROR abfangen ● Logging ● Toxic Message-Handling Shutdown
  • 30. Deployment ● Consumers need an up to date code base ● Consumers need a graceful shutdown ● Consumers must only be shutdown when idling ○ Otherwise : Loss of message / data
  • 31. Other challenges ● Significant change of infrastructure ○ Choose your libraries and dependencies carefully! ● Moving load is not removing load ● Rethinking necessary ○ Display of status / intermediate result ○ Websockets or Push instead of Polling ● Interdependencies of messages ○ Especially when running parallel consumers! ● Performance (Doctrine ORM / ODM)
  • 32. Questions ? Thank you :) Find this and other talks at: https://www.gardenofconcepts. com/talks/ Feedback is always welcome & please rate our talk !