Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Does Django scale? An introduction to Scalability

424 views

Published on

Introductory talk about Scalability and Django. Main topics:
- Does Django Scale?
- What is Scalability?
- What is Django?
- How to scale Django?

Talk given at Python Barcelona (25th April 2019):
https://www.meetup.com/python-185/events/260642585/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Does Django scale? An introduction to Scalability

  1. 1. “Does Django scale?” An introduction to Scalability Python Barcelona David Arcos
  2. 2. Hi! I’m David ● I’m a backend developer specialized in distributed systems, scalability and security ● >10 years building SaaS platforms in different sectors ● CTO at Lead Ratings – SaaS solution that uses Machine-Learning algorithms to predict business metrics (conversion, billing, churn...) ● Associate professor at ESADE – Teaching “Cloud Computing” ● PyBCN meetup assistant since 2008, organizer since 2013
  3. 3. 1) Does Django Scale? Questions
  4. 4. https://stackoverflow.com/questions/886221/does-django-scale
  5. 5. https://docs.djangoproject.com/en/dev/faq/general/#does-django-scale
  6. 6. 1) Does Django Scale? YES 2) What is Scalability? Questions
  7. 7. Scalability “The potential of a system to be enlarged to accommodate a growing amount of work”
  8. 8. “A Word on Scalability” (2006) “A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added” - Werner Vogels (Amazon CTO) https://www.allthingsdistributed.com/2006/03/a_word_on_scalability.html
  9. 9. Will it scale? An algorithm/protocol/system does scale, if it is suitably efficient and practical when applied to large situations. In example: – a large input data set – a large number of outputs or users – a large number of participating nodes If it fails when increasing, it doesn’t scale.
  10. 10. Performance != Scalability ● Performance is a quality metric: – “The time it takes to execute one request” ● Scalability: – The ability to maintain that performance, under increasing load – Or: increase performance when adding resources
  11. 11. Where is the state? ● Scaling is much easier if: – The app servers are stateless – The database keeps the state ● So: – Vertical scalability on (stateful) db servers – Horizontal scalability on (stateless) app servers
  12. 12. Vertical / horizontal scaling
  13. 13. Vertical scaling ● Scale up/down: add/remove resources to a single node – Adding more power (faster CPU, more CPUs, more memory, a bigger disk…) to an existing machine ● Very common on databases (stateful systems) ● Easier… when possible
  14. 14. Vertical scaling… but... ● But: – There are hardware limits ● RDS current limit of 488 GiB of memory (db.r4.16xlarge) – Diminishing returns ● At some point, the level of benefits gained is less than the amount of money invested. – Involves downtime ● Keep a hot spare! ● RDS Multi A-Z has a “standby” replica with automatic failover
  15. 15. Horizontal scaling ● Scale out/in: add/remove nodes to a system – Adding new computers to a cluster ● Very common on (stateless) web servers ● High availability – doesn’t matter if some nodes crash ● Cheaper (HW is cheap, VMs are cheap)
  16. 16. Horizontal scaling.. but... ● But: – Architecture must support it ● Keep the state in external db! – Management overhead ● Automate it! ● Sysadmin operations must be treated as software + config
  17. 17. 1) Does Django Scale? YES 2) What is Scalability? 3) What is Django? Questions
  18. 18. “The web framework for perfectionists with deadlines” https://www.djangoproject.com
  19. 19. Who uses Django? ● Instagram ● Pinterest ● Spotify ● Disqus ● Bitbucket ● Mozilla Firefox ● Dropbox ● Open Stack ● NASA ● The Washington Post ● National Geographic ● The Onion ● Eventbrite ● Prezi ● Reddit Gifts ● Open Knowledge Foundation ● MacArthur Foundation
  20. 20. DRF: Django REST framework https://www.django-rest-framework.org “A powerful and flexible toolkit for building Web APIs”
  21. 21. http://www.celeryproject.org Celery: Distributed Task Queue
  22. 22. Other Python web frameworks
  23. 23. 1) Does Django Scale? YES 2) What is Scalability? 3) What is Django? 4) How to scale Django? Questions
  24. 24. How to Scale Django? ● Measure first, find bottlenecks – Rinse and repeat ● Focus on database performance – Indexes, queries, replicas... ● Cache all the things! – Excellent cache documentation ● Run slow tasks in async workers – (when possible)
  25. 25. “High Performance Django” book https://highperformancedjango.com
  26. 26. 1) Does Django Scale? YES 2) What is Scalability? 3) What is Django? 4) How to scale Django? Questions
  27. 27. Thanks for attending! At Lead Ratings we are looking for: – Python Developers – Data Engineers – Data Scientists
  28. 28. Questions?

×