Celery in the Django

2,489 views
2,339 views

Published on

Shortly introduce Celery and some pitfalls of using Celery.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,489
On SlideShare
0
From Embeds
0
Number of Embeds
189
Actions
Shares
0
Downloads
36
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Description很多人都是因為追著蟒蛇誤入叢林,叢林是危機重重的,如何在危險的叢林裏野地求生,找尋自己的一條生路呢?在此介紹你【芹菜】,芹菜是一個好菜,它健康,它機能性強,除了幫助你在叢林裏生存,自從我吃了它,我每次考試都考100分。
  • We have a system that need to process 1 billion tasks per day.
  • Celery in the Django

    1. 1. 叢林裡的小芹菜Celery in the DjangoBy Walter Liu
    2. 2. About me Learn Python since 2004 Was a Chief Technical Director in Softstar. Now an Architect in Trend Micro Light Celery user for several months.
    3. 3. Celery is X Distributed Asynchronous Task Queue
    4. 4. Have you met? Deliver thousands to millions e-mails to your members? Many computation heavy queries at the same time that blocks your web site or DB? Query slow or too many connection to external API? (like Twitter/Facebook) Long run tasks that occupied your web server resources?
    5. 5. Why Celery? POC without Celery  Add a “@celery.task” and the tasks become distributed tasks. Multi-queue, routing, all configurable. Many utility functionality to help task execution  Group/Chord/Chain/Callback  Retry/timeout  Etc…..
    6. 6. from celery import Celery@celery.task( max_retries = 3, default_retry_delay=2 * 60, # retry in 2 minutes. rate_limit = 20, time_limit = 10, # hard time limit soft_time_limit = 5, ignore_result = True, acks_late = True, # make sure this task will be done. )def send_twitter_status(oauth, tweet): try: twitter = Twitter(oauth) twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError), exc: raise send_twitter_status.retry(exc=exc)
    7. 7. Understand the personality ofyour celery What happen if the program throw an exception? What happen if the program hard/soft time limit? What happen if the hard/soft time limit reached in your group/chord function? Many others.
    8. 8. Our pitfall/trouble cases Many, too many small tasks  For example 1 billion per day. Large task argument (workaround => store task at cache)  Send file directly Better not wait another task in a task. Using Redis as AMQP and no HA. Hard time limit reached in Chord function. Multi-process and multi-thread problem in Windows Logging. Queue management
    9. 9. Some internet complain aboutCelery Only Python API change and break his program  2.5.x => 3.0.x Pylint
    10. 10. Some similar things Gearman Huey Advanced Python Scheduler
    11. 11. Recruiting – we’re using Front-end: not very limited. Backend  Python  Django  MongoDB  Redis/Memcache  Celery

    ×