Your SlideShare is downloading. ×
0
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Celery in the Django
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Celery in the Django

2,155

Published on

Shortly introduce Celery and some pitfalls of using Celery.

Shortly introduce Celery and some pitfalls of using Celery.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,155
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
33
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Description很多人都是因為追著蟒蛇誤入叢林,叢林是危機重重的,如何在危險的叢林裏野地求生,找尋自己的一條生路呢?在此介紹你【芹菜】,芹菜是一個好菜,它健康,它機能性強,除了幫助你在叢林裏生存,自從我吃了它,我每次考試都考100分。
  • We have a system that need to process 1 billion tasks per day.
  • Transcript

    1. 叢林裡的小芹菜Celery in the DjangoBy Walter Liu
    2. About me Learn Python since 2004 Was a Chief Technical Director in Softstar. Now an Architect in Trend Micro Light Celery user for several months.
    3. Celery is X Distributed Asynchronous Task Queue
    4. Have you met? Deliver thousands to millions e-mails to your members? Many computation heavy queries at the same time that blocks your web site or DB? Query slow or too many connection to external API? (like Twitter/Facebook) Long run tasks that occupied your web server resources?
    5. Why Celery? POC without Celery  Add a “@celery.task” and the tasks become distributed tasks. Multi-queue, routing, all configurable. Many utility functionality to help task execution  Group/Chord/Chain/Callback  Retry/timeout  Etc…..
    6. from celery import Celery@celery.task( max_retries = 3, default_retry_delay=2 * 60, # retry in 2 minutes. rate_limit = 20, time_limit = 10, # hard time limit soft_time_limit = 5, ignore_result = True, acks_late = True, # make sure this task will be done. )def send_twitter_status(oauth, tweet): try: twitter = Twitter(oauth) twitter.update_status(tweet) except (Twitter.FailWhaleError, Twitter.LoginError), exc: raise send_twitter_status.retry(exc=exc)
    7. Understand the personality ofyour celery What happen if the program throw an exception? What happen if the program hard/soft time limit? What happen if the hard/soft time limit reached in your group/chord function? Many others.
    8. Our pitfall/trouble cases Many, too many small tasks  For example 1 billion per day. Large task argument (workaround => store task at cache)  Send file directly Better not wait another task in a task. Using Redis as AMQP and no HA. Hard time limit reached in Chord function. Multi-process and multi-thread problem in Windows Logging. Queue management
    9. Some internet complain aboutCelery Only Python API change and break his program  2.5.x => 3.0.x Pylint
    10. Some similar things Gearman Huey Advanced Python Scheduler
    11. Recruiting – we’re using Front-end: not very limited. Backend  Python  Django  MongoDB  Redis/Memcache  Celery

    ×