Celery for internal API in  SOA infrastructure        Roman Imankulov     PyCon Russia, Feb 2013
1. Internal API. Whats the dealSOA service architecture example
1. Internal API. Whats the deal● DCOM: too Microsoftish● CORBA: too Enterprisy● SOAP: XML inside● RESTful: too sloppyCeler...
Agenda1. Internal API. Whats the deal2. How to use Celery for internal API3. Organizing Celery-based API. Routing4. Celery...
2. How to use Celery for internal API             Celery Worker
2. How to use Celery for internal API  Classic celery client imports task function
2. How to use Celery for internal APIClassic celery client imports makes you:● install everything   ○ slows down installat...
2. How to use Celery for internal API            "Task implementation is not to             be exposed without necessity" ...
2. How to use Celery for internal API  Low-level interface defines only task name
2. How to use Celery for internal APIConvenient API requires some boilerplate code
2. How to use Celery for internal APIMake it even more convenient withintrospection
2. How to use Celery for internal APIIntrospection magic with celery-apihttp://github.com/imankulov/celery-api
3. Celery-based API. Routing
3. Celery-based API. RoutingCreating API endpoint1. Common package for all API endpoints2. Every worker imports celery ins...
3. Celery-based API. RoutingSend tasks withadd.apply_async(queue=foo,                args=(1, 1))
3. Celery-based API. RoutingDefine routing rules
3. Celery-based API. RoutingTask in the wire{ "body": <base64 encoded string with task name and args>,  "properties": {   ...
3. Celery-based API. RoutingTask in the wire     "delivery_info": {       "priority": 0,       "routing_key": "foo",      ...
3. Celery-based API. RoutingAMQP for dummies
3. Celery-based API. RoutingDefine all queues (API endpoints) you have
4. Celery-based API. BenefitsHigh level protocol.Takes the responsibility for                   data serialization        ...
4. Celery-based API. BenefitsSupport for wide range of brokers.You can switch between brokers with no majorchanges in your...
4. Celery-based API. BenefitsBuilt-in extra stuff which you were afraid to everdream of.asynchronous execution        para...
4. Celery-based API. PitfallsPython-centric.Celery is written in Python and meant to beused with Python code exclusively
4. Celery-based API. PitfallsIncreasing complexity.Celery codebase          46k LOCKombu codebase           19k LOCCompare...
4. Celery-based API. Security noteNever ever expose Celery broker to theWeb!
4. Celery-based API. Security note1. Try>>> redis_url = redis://:pycon2013@broker.imankulov.name>>> celery = Celery(broker...
Upcoming SlideShare
Loading in...5
×

Celery for internal API in SOA infrastructure

7,667

Published on

Slides from my presentation on pycon.ru on using Celery for internal API (pycon.ru, February, 2013)

0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,667
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
46
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide

Celery for internal API in SOA infrastructure

  1. 1. Celery for internal API in SOA infrastructure Roman Imankulov PyCon Russia, Feb 2013
  2. 2. 1. Internal API. Whats the dealSOA service architecture example
  3. 3. 1. Internal API. Whats the deal● DCOM: too Microsoftish● CORBA: too Enterprisy● SOAP: XML inside● RESTful: too sloppyCelery to the rescue?
  4. 4. Agenda1. Internal API. Whats the deal2. How to use Celery for internal API3. Organizing Celery-based API. Routing4. Celery-based API. Benefits, pitfalls and security notes
  5. 5. 2. How to use Celery for internal API Celery Worker
  6. 6. 2. How to use Celery for internal API Classic celery client imports task function
  7. 7. 2. How to use Celery for internal APIClassic celery client imports makes you:● install everything ○ slows down installation● import everything ○ python processes grow in size ○ circular dependencies ○ tight coupling● share everything ○ ... with outsource developers
  8. 8. 2. How to use Celery for internal API "Task implementation is not to be exposed without necessity" William of Ockham
  9. 9. 2. How to use Celery for internal API Low-level interface defines only task name
  10. 10. 2. How to use Celery for internal APIConvenient API requires some boilerplate code
  11. 11. 2. How to use Celery for internal APIMake it even more convenient withintrospection
  12. 12. 2. How to use Celery for internal APIIntrospection magic with celery-apihttp://github.com/imankulov/celery-api
  13. 13. 3. Celery-based API. Routing
  14. 14. 3. Celery-based API. RoutingCreating API endpoint1. Common package for all API endpoints2. Every worker imports celery instance3. Expose tasks for worker with CELERY_IMPORTS4. Launch worker with celery worker -A module_name -Q foo -n foo
  15. 15. 3. Celery-based API. RoutingSend tasks withadd.apply_async(queue=foo, args=(1, 1))
  16. 16. 3. Celery-based API. RoutingDefine routing rules
  17. 17. 3. Celery-based API. RoutingTask in the wire{ "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange":"foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
  18. 18. 3. Celery-based API. RoutingTask in the wire "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
  19. 19. 3. Celery-based API. RoutingAMQP for dummies
  20. 20. 3. Celery-based API. RoutingDefine all queues (API endpoints) you have
  21. 21. 4. Celery-based API. BenefitsHigh level protocol.Takes the responsibility for data serialization routing exception handling
  22. 22. 4. Celery-based API. BenefitsSupport for wide range of brokers.You can switch between brokers with no majorchanges in your codeBrokers:RabbitMQ Redis SQLAlchemyDjango MongoDB Amazon SQSCouchDB Beanstalk
  23. 23. 4. Celery-based API. BenefitsBuilt-in extra stuff which you were afraid to everdream of.asynchronous execution parallel executionasynchronous parallel executiondelayed execution throttlingautomatic retrying of failed taskslimiting the time of executionautoscaling API inspectionpublic key cryptography (message signing)
  24. 24. 4. Celery-based API. PitfallsPython-centric.Celery is written in Python and meant to beused with Python code exclusively
  25. 25. 4. Celery-based API. PitfallsIncreasing complexity.Celery codebase 46k LOCKombu codebase 19k LOCCompareFlask codebase 10k LOCWerkzeug codebase 28k LOCJinja2 codebase 16k LOC
  26. 26. 4. Celery-based API. Security noteNever ever expose Celery broker to theWeb!
  27. 27. 4. Celery-based API. Security note1. Try>>> redis_url = redis://:pycon2013@broker.imankulov.name>>> celery = Celery(broker=redis_url, result=redis_url)2. Get the contents of /etc/pycon_secret3. Exchange it for a secret prize ;)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×