Celery for internal API in SOA infrastructure

11,603 views

Published on

Slides from my presentation on pycon.ru on using Celery for internal API (pycon.ru, February, 2013)

Celery for internal API in SOA infrastructure

  1. 1. Celery for internal API in SOA infrastructure Roman Imankulov PyCon Russia, Feb 2013
  2. 2. 1. Internal API. Whats the dealSOA service architecture example
  3. 3. 1. Internal API. Whats the deal● DCOM: too Microsoftish● CORBA: too Enterprisy● SOAP: XML inside● RESTful: too sloppyCelery to the rescue?
  4. 4. Agenda1. Internal API. Whats the deal2. How to use Celery for internal API3. Organizing Celery-based API. Routing4. Celery-based API. Benefits, pitfalls and security notes
  5. 5. 2. How to use Celery for internal API Celery Worker
  6. 6. 2. How to use Celery for internal API Classic celery client imports task function
  7. 7. 2. How to use Celery for internal APIClassic celery client imports makes you:● install everything ○ slows down installation● import everything ○ python processes grow in size ○ circular dependencies ○ tight coupling● share everything ○ ... with outsource developers
  8. 8. 2. How to use Celery for internal API "Task implementation is not to be exposed without necessity" William of Ockham
  9. 9. 2. How to use Celery for internal API Low-level interface defines only task name
  10. 10. 2. How to use Celery for internal APIConvenient API requires some boilerplate code
  11. 11. 2. How to use Celery for internal APIMake it even more convenient withintrospection
  12. 12. 2. How to use Celery for internal APIIntrospection magic with celery-apihttp://github.com/imankulov/celery-api
  13. 13. 3. Celery-based API. Routing
  14. 14. 3. Celery-based API. RoutingCreating API endpoint1. Common package for all API endpoints2. Every worker imports celery instance3. Expose tasks for worker with CELERY_IMPORTS4. Launch worker with celery worker -A module_name -Q foo -n foo
  15. 15. 3. Celery-based API. RoutingSend tasks withadd.apply_async(queue=foo, args=(1, 1))
  16. 16. 3. Celery-based API. RoutingDefine routing rules
  17. 17. 3. Celery-based API. RoutingTask in the wire{ "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange":"foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
  18. 18. 3. Celery-based API. RoutingTask in the wire "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
  19. 19. 3. Celery-based API. RoutingAMQP for dummies
  20. 20. 3. Celery-based API. RoutingDefine all queues (API endpoints) you have
  21. 21. 4. Celery-based API. BenefitsHigh level protocol.Takes the responsibility for data serialization routing exception handling
  22. 22. 4. Celery-based API. BenefitsSupport for wide range of brokers.You can switch between brokers with no majorchanges in your codeBrokers:RabbitMQ Redis SQLAlchemyDjango MongoDB Amazon SQSCouchDB Beanstalk
  23. 23. 4. Celery-based API. BenefitsBuilt-in extra stuff which you were afraid to everdream of.asynchronous execution parallel executionasynchronous parallel executiondelayed execution throttlingautomatic retrying of failed taskslimiting the time of executionautoscaling API inspectionpublic key cryptography (message signing)
  24. 24. 4. Celery-based API. PitfallsPython-centric.Celery is written in Python and meant to beused with Python code exclusively
  25. 25. 4. Celery-based API. PitfallsIncreasing complexity.Celery codebase 46k LOCKombu codebase 19k LOCCompareFlask codebase 10k LOCWerkzeug codebase 28k LOCJinja2 codebase 16k LOC
  26. 26. 4. Celery-based API. Security noteNever ever expose Celery broker to theWeb!
  27. 27. 4. Celery-based API. Security note1. Try>>> redis_url = redis://:pycon2013@broker.imankulov.name>>> celery = Celery(broker=redis_url, result=redis_url)2. Get the contents of /etc/pycon_secret3. Exchange it for a secret prize ;)

×