This document discusses using Django for large-scale backend services and processing. It describes two use cases: 1) an automated data integration service and 2) a social media analysis service. For both use cases, Django was able to handle large and complex processing loads robustly through techniques like message queues, workflows, and cloud computing. Key frameworks discussed include Celery, Suds, django-xmpp, and Google Chart API.

1. Large-scale processing using Django Mashing clouds, queues & workflows PyWeb-IL 8 th meeting Udi h Bauman (@dibau_naum_h) Tikal Knowledge (http://tikalk.com)

2. Agenda Web apps vs. Back-end Services

3. Addressing Scalability

4. Experience with Django

5. Use-case 1: automated data integration service

6. Use-case 2: social media analysis service

7. Recommendations

8. Links

9. Web apps vs. Back-end Services Common conception is that a Web framework is just for Web sites

10. Web back-ends become thinner - just services

11. Applications become service providers, usually over HTTP

12. All reasons for using Django for almost any back-end offering services

13. Web apps vs. Back-end Services How are back-end services different? Usually have behaviors not triggered by client requests

14. Usually involve long processing

15. May involve continuous communications, & not just request-response

16. Reliability & high-availability are usually more important with non-human users

17. Lots of communication with other back-ends

18. Addressing the needs of back-end services Message Queues abstract invocation & enable reliable distributed processing

19. Workflow Engines manage long processing

20. Continuous communication (e.g., TCP-based) is possible, can be abstracted with XMPP

21. Clouds & auto-scaling enable high-availability

22. Can use SOAP/REST for protocols against other back-ends

23. Experience with Django No matter how heavy & large the task & load were – it just worked.

24. Even when processing took days to complete, Django was 100% robust

25. Had no issues with Performance