• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Celery for internal API in SOA infrastructure
 

Celery for internal API in SOA infrastructure

on

  • 4,543 views

Slides from my presentation on pycon.ru on using Celery for internal API (pycon.ru, February, 2013)

Slides from my presentation on pycon.ru on using Celery for internal API (pycon.ru, February, 2013)

Statistics

Views

Total Views
4,543
Views on SlideShare
1,221
Embed Views
3,322

Actions

Likes
3
Downloads
20
Comments
0

5 Embeds 3,322

http://www.imankulov.name 3245
http://localhost 74
http://192.168.1.253 1
http://www.google.com 1
http://webcache.googleusercontent.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Celery for internal API in SOA infrastructure Celery for internal API in SOA infrastructure Presentation Transcript

    • Celery for internal API in SOA infrastructure Roman Imankulov PyCon Russia, Feb 2013
    • 1. Internal API. Whats the dealSOA service architecture example
    • 1. Internal API. Whats the deal● DCOM: too Microsoftish● CORBA: too Enterprisy● SOAP: XML inside● RESTful: too sloppyCelery to the rescue?
    • Agenda1. Internal API. Whats the deal2. How to use Celery for internal API3. Organizing Celery-based API. Routing4. Celery-based API. Benefits, pitfalls and security notes
    • 2. How to use Celery for internal API Celery Worker
    • 2. How to use Celery for internal API Classic celery client imports task function
    • 2. How to use Celery for internal APIClassic celery client imports makes you:● install everything ○ slows down installation● import everything ○ python processes grow in size ○ circular dependencies ○ tight coupling● share everything ○ ... with outsource developers
    • 2. How to use Celery for internal API "Task implementation is not to be exposed without necessity" William of Ockham
    • 2. How to use Celery for internal API Low-level interface defines only task name
    • 2. How to use Celery for internal APIConvenient API requires some boilerplate code
    • 2. How to use Celery for internal APIMake it even more convenient withintrospection
    • 2. How to use Celery for internal APIIntrospection magic with celery-apihttp://github.com/imankulov/celery-api
    • 3. Celery-based API. Routing
    • 3. Celery-based API. RoutingCreating API endpoint1. Common package for all API endpoints2. Every worker imports celery instance3. Expose tasks for worker with CELERY_IMPORTS4. Launch worker with celery worker -A module_name -Q foo -n foo
    • 3. Celery-based API. RoutingSend tasks withadd.apply_async(queue=foo, args=(1, 1))
    • 3. Celery-based API. RoutingDefine routing rules
    • 3. Celery-based API. RoutingTask in the wire{ "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange":"foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
    • 3. Celery-based API. RoutingTask in the wire "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
    • 3. Celery-based API. RoutingAMQP for dummies
    • 3. Celery-based API. RoutingDefine all queues (API endpoints) you have
    • 4. Celery-based API. BenefitsHigh level protocol.Takes the responsibility for data serialization routing exception handling
    • 4. Celery-based API. BenefitsSupport for wide range of brokers.You can switch between brokers with no majorchanges in your codeBrokers:RabbitMQ Redis SQLAlchemyDjango MongoDB Amazon SQSCouchDB Beanstalk
    • 4. Celery-based API. BenefitsBuilt-in extra stuff which you were afraid to everdream of.asynchronous execution parallel executionasynchronous parallel executiondelayed execution throttlingautomatic retrying of failed taskslimiting the time of executionautoscaling API inspectionpublic key cryptography (message signing)
    • 4. Celery-based API. PitfallsPython-centric.Celery is written in Python and meant to beused with Python code exclusively
    • 4. Celery-based API. PitfallsIncreasing complexity.Celery codebase 46k LOCKombu codebase 19k LOCCompareFlask codebase 10k LOCWerkzeug codebase 28k LOCJinja2 codebase 16k LOC
    • 4. Celery-based API. Security noteNever ever expose Celery broker to theWeb!
    • 4. Celery-based API. Security note1. Try>>> redis_url = redis://:pycon2013@broker.imankulov.name>>> celery = Celery(broker=redis_url, result=redis_url)2. Get the contents of /etc/pycon_secret3. Exchange it for a secret prize ;)