Your SlideShare is downloading. ×
Celery for internal API in SOA infrastructure
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Celery for internal API in SOA infrastructure

6,614
views

Published on

Slides from my presentation on pycon.ru on using Celery for internal API (pycon.ru, February, 2013)

Slides from my presentation on pycon.ru on using Celery for internal API (pycon.ru, February, 2013)


0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
6,614
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
39
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Celery for internal API in SOA infrastructure Roman Imankulov PyCon Russia, Feb 2013
  • 2. 1. Internal API. Whats the dealSOA service architecture example
  • 3. 1. Internal API. Whats the deal● DCOM: too Microsoftish● CORBA: too Enterprisy● SOAP: XML inside● RESTful: too sloppyCelery to the rescue?
  • 4. Agenda1. Internal API. Whats the deal2. How to use Celery for internal API3. Organizing Celery-based API. Routing4. Celery-based API. Benefits, pitfalls and security notes
  • 5. 2. How to use Celery for internal API Celery Worker
  • 6. 2. How to use Celery for internal API Classic celery client imports task function
  • 7. 2. How to use Celery for internal APIClassic celery client imports makes you:● install everything ○ slows down installation● import everything ○ python processes grow in size ○ circular dependencies ○ tight coupling● share everything ○ ... with outsource developers
  • 8. 2. How to use Celery for internal API "Task implementation is not to be exposed without necessity" William of Ockham
  • 9. 2. How to use Celery for internal API Low-level interface defines only task name
  • 10. 2. How to use Celery for internal APIConvenient API requires some boilerplate code
  • 11. 2. How to use Celery for internal APIMake it even more convenient withintrospection
  • 12. 2. How to use Celery for internal APIIntrospection magic with celery-apihttp://github.com/imankulov/celery-api
  • 13. 3. Celery-based API. Routing
  • 14. 3. Celery-based API. RoutingCreating API endpoint1. Common package for all API endpoints2. Every worker imports celery instance3. Expose tasks for worker with CELERY_IMPORTS4. Launch worker with celery worker -A module_name -Q foo -n foo
  • 15. 3. Celery-based API. RoutingSend tasks withadd.apply_async(queue=foo, args=(1, 1))
  • 16. 3. Celery-based API. RoutingDefine routing rules
  • 17. 3. Celery-based API. RoutingTask in the wire{ "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange":"foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
  • 18. 3. Celery-based API. RoutingTask in the wire "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
  • 19. 3. Celery-based API. RoutingAMQP for dummies
  • 20. 3. Celery-based API. RoutingDefine all queues (API endpoints) you have
  • 21. 4. Celery-based API. BenefitsHigh level protocol.Takes the responsibility for data serialization routing exception handling
  • 22. 4. Celery-based API. BenefitsSupport for wide range of brokers.You can switch between brokers with no majorchanges in your codeBrokers:RabbitMQ Redis SQLAlchemyDjango MongoDB Amazon SQSCouchDB Beanstalk
  • 23. 4. Celery-based API. BenefitsBuilt-in extra stuff which you were afraid to everdream of.asynchronous execution parallel executionasynchronous parallel executiondelayed execution throttlingautomatic retrying of failed taskslimiting the time of executionautoscaling API inspectionpublic key cryptography (message signing)
  • 24. 4. Celery-based API. PitfallsPython-centric.Celery is written in Python and meant to beused with Python code exclusively
  • 25. 4. Celery-based API. PitfallsIncreasing complexity.Celery codebase 46k LOCKombu codebase 19k LOCCompareFlask codebase 10k LOCWerkzeug codebase 28k LOCJinja2 codebase 16k LOC
  • 26. 4. Celery-based API. Security noteNever ever expose Celery broker to theWeb!
  • 27. 4. Celery-based API. Security note1. Try>>> redis_url = redis://:pycon2013@broker.imankulov.name>>> celery = Celery(broker=redis_url, result=redis_url)2. Get the contents of /etc/pycon_secret3. Exchange it for a secret prize ;)