Talk given at the PHP user group Frankfurt regarding asynchronous events in Symfony 2 and the RabbitMQ library. We demonstrate some ups and downs as well as best practices and lessons learned.

1. Working with
Asynchronous Events
Jan Gregor Emge-Triebel (@jan0707)
Dennis Oehme (@dennisoehme)

2. Garden of Concepts GmbH
● Hanau / Munich
● In house CMS “.stone”
● In house CRM “Elvis”
● Various individual customer projects
● Symfony based development, supported by:
○ MongoDB, ElasticSearch, MySQL, RabbitMQ,
Redis, AngularJS, Twig, Jade, Gitlab, Docker,
Vagrant, Ansible
● Say hi @goc_mediastudio

3. Requests == Event
“Symfony is a request based
framework.”
Fabien Potencier, Symfony Live 2014
@fabpot

4. Requests
Request Controller Logic 1 Logic 2 Response
Request Controller
Logic 1
Logic 2
Response
Synchronous
Asynchronous

5. Why do we need asynchronous event processing ?
● Move long running logic
● Cut down reaction times
● Transfer load
● Microservices
● Exchange data/messages with other (sub-)systems
● Parallelization
● Scaling

6. Our journey
● Gearman
● (PHP) Resque
○ Demo
● RabbitMQ

7. Gearman
● Gearman Job Server (written in C)
● Runs as a Linux daemon
● Requires PECL extension for worker API
● Worker APIs exists for other Languages (C, Perl,...)

8. Gearman
http://gearman.org/examples/send-emails/

9. Gearman
● Latest PECL-Extension version is 1.1.2 (August 2013)
● Latest Server-Release (spring 2014)
● No more process is being made
● Gearman daemon and extension are often incompatible

10. (PHP) Resque
● Is a Resque port to php:
○ Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple
queues, and processing them later.
● Needs Redis(-Cluster)
● Redis is capable of atomic transactions (FIFO)

11. Demo
https://github.com/Jan0707/async-symfony

12. (PHP) Resque
● Hardly any commits since 2012
● Last commit to Master on 2015-05-12
● chrisboulton/php-resque & michelsalib/BCCResqueBundle must be presumed
dead ?

13. RabbitMQ
Or why this talk should be named:
“Working with Asynchronous Events in RabbitMQ”

14. RabbitMQ
● Message Broker written in Erlang
● AMQP (Advanced Message Queuing Protocol)
● Open, but binary network protocoll
● Routing, message distribution
● Transactions
● Plus, more nice features
○ TTL
○ DLX (Dead Letter eXchange)
○ PreFetching

15. Of publishers and consumers ...
Publisher Publish Consumer
Ex-
change
QueueRoutes Consumes

16. There is a Bundle for it
● https://github.com/php-amqplib/RabbitMqBundle
● Consumer and Producer as Symfony services
● Every queue Consumer runs as a Symfony command
● Configuration via app/config.yml
○ Better: Create app/config/rabbitmq.yml and import in app/config/config.yml

17. app/config/rabbitmq.yml

18. Symfony Producer

19. Symfony Consumer

20. Lessons Learned: Consumer
● PHP threads are not meant to be long running
○ New problems: memory leaks, ...
● Use supervisor to check and restart Consumers
● Open database connections, timeouts
○ Better quit when inactive
● Cap maximum runtime per thread
○ Automatically shutdown after processign x messages
● Memory limit: RabbitMQ-Bundle utilizes “soft-quota”
● Booting consumers takes rather long
● Prefetching (Warning: Order!)

21. Lessons Learned: Message
● Compact/short messages
● Queues eat up a lot of storage / RAM
● Common format such as JSON or XML
○ Do not serialize PHP objects or arrays
○ Ensure interoperability with other systems and languages

22. Lessons Learned: Message Transformation
● Use JMS/Serializer to convert objects to JSON
○ Deserialize JSON to PHP objects in consumer
○ Objects can and should be validated (with symfony forms and asserts)
○ Reload entities / documents to keep them up to date (doctrine/unit of work)
● Alternative approach: PHPs JsonSerializable Interface

23. Lessons Learned: Message Header
● Use message “Header” for content-type and version
○ Message formats can change
○ consumers must remain compatible
● Unqiue IDs make logging / debugging / live easier
● Other meta data:
○ timestamp
○ user / session
○ App id
○ ...

24. Tip: Custom Consumer-/Producer-Classes
● Message transformation
● Events
● Logging

25. Transactions
● Messages must be brokered to exactly one Consumer
● Consumer must explicitly answer a message with “acknowledge”
● If not, the message will be marked “rejected” and be requeued
○ Beware of requeuing:
■ Can lead to toxic messages
■ Can change the order of messages
○ Beware of final rejection:
■ Can lead to loss of data

26. Toxic Messages
● If a message is not explicitly being answered, it will be requeued.
● If that happens multiple times :
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop
● Congratulations on your infinite loop

27. Dead Letter Exchange
● RabbitMQ comes with a Dead Letter Exchange (DLX)
● Every queue determines its own or shared DLX
● DLX is a separate queue, storing all rejected messages
○ (Reject, TTL, Prefetch)

28. app/config/rabbitmq.yml

29. Signal Handling
● Graceful Shutdown
● pcntl_signal / _dispatch
● SIGTERM
● FATAL ERROR abfangen
● Logging
● Toxic Message-Handling
Shutdown

30. Deployment
● Consumers need an up to date code base
● Consumers need a graceful shutdown
● Consumers must only be shutdown when idling
○ Otherwise : Loss of message / data

31. Other challenges
● Significant change of infrastructure
○ Choose your libraries and dependencies carefully!
● Moving load is not removing load
● Rethinking necessary
○ Display of status / intermediate result
○ Websockets or Push instead of Polling
● Interdependencies of messages
○ Especially when running parallel consumers!
● Performance (Doctrine ORM / ODM)

32. Questions ?
Thank you :)
Find this and other talks at: https://www.gardenofconcepts.
com/talks/
Feedback is always welcome
& please rate our talk !