SlideShare a Scribd company logo
1 of 35
Download to read offline
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Efficient Django
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Abstract
Tips and best practices for avoiding scalability
issues and performance bottlenecks in Django
● 1) Basic concepts: the theory
● 2) Measuring: how to find bottlenecks
● 3) Tips and tricks
● 4) Conclusion (yes, it scales!)
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Hi!
● I'm David Arcos
● Python/Django developer since 2008
● Co-organizer at Python Barcelona
● CTO at Lead Ratings
David Arcos - @DZPMEfficient Django – #EuroPython 2016
●
“We improve your sales conversions, using
predictive algorithms to rate the leads”
●
Prediction API, “Machine Learning as a Service”
●
http://lead-ratings.com
David Arcos - @DZPMEfficient Django – #EuroPython 2016
1) Basic concepts
David Arcos - @DZPMEfficient Django – #EuroPython 2016
The Pareto Principle
"For many events, roughly 80% of the effects
come from 20% of the causes"
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Prioritize and focus
Focus on the few tasks that will have the most impact
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Basic scalability
“Potential to be enlarged to handle a growing
amount of work”
●
Stateless app servers
– Load balance them, scale horizontally
●
Keep the state on the database(s)
– This is the difficult part! Each system is different
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Database performance
●
Do less requests:
– Less reads
– Less writes
●
Do faster requests:
– Indexed fields
– De-normalize
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Templates
●
Cache them
●
Jinja2 is a bit faster than the default engine
– but cache them anyways
●
You can do fragment caching (for blocks)
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Cache
●
Generic approach: cache at each stack level
●
The cache documentation is excellent
●
Beware of the cache invalidation!
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Cache
●
Generic approach: cache at each stack level
●
The cache documentation is excellent
●
Beware of the cache invalidation!
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Bottlenecks
●
Where is your bottleneck?
●
CPU bound or I/O bound?
– CPU? Run heavy calculations in async workers
– Memory? Compress objects before caching
– Database? Read from db replicas
●
How to find it?
David Arcos - @DZPMEfficient Django – #EuroPython 2016
2) Measuring
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Can't improve what you don't measure
●
Measure your system to find bottlenecks
●
Optimize those bottlenecks
●
Verify the improvements
●
Rinse and repeat!
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Monitoring
●
System: load, CPU, memory...
●
Database: q/s, response time, size
●
Cache: q/s, hit rate
●
Queue: length
●
Custom: metrics for your app
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Profiling
●
The cProfile module provides profiling of
Python programs by collecting data:
– Number of calls, running time, time per call...
David Arcos - @DZPMEfficient Django – #EuroPython 2016
timeit
●
The timeit module is a simple way to time
execution time of small bits of Python code:
David Arcos - @DZPMEfficient Django – #EuroPython 2016
ipdb
●
Like pdb, but for ipython
– tab completion, syntax highlighting, better
tracebacks, better introspection…
●
Use ipdb.set_trace() to add a breakpoint and
jump in with the debugger
David Arcos - @DZPMEfficient Django – #EuroPython 2016
django-debug-toolbar
●
Display debug information about the current
request/response
●
Panels, very modular
David Arcos - @DZPMEfficient Django – #EuroPython 2016
django-debug-toolbar-line-profiler
●
A toolbar panel for profiling
Django Debug Panel
●
Chrome extension
●
For AJAX requests and non-HTML responses
David Arcos - @DZPMEfficient Django – #EuroPython 2016
3) Tips and tricks
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Add db indexes
●
Single (db_index) or multiple (index_together)
●
Be sure to profile and measure!
– Sometimes it’s not obvious (i.e., admin)
– Huge difference, i.e. from 15s to 3 ms (3.5M rows)
●
But: uses more space, slower writes
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Do bulk operations
●
Will greatly reduce the number of SQL queries:
– Model.objects.bulk_create()
– qs.update() <- maybe with F() expressions
– qs.delete()
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Get related objects
●
Return FK fields in same query:
– qs.select_related()
●
Return M2M fields, extra query:
– qs.prefetch_related()
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Slow admin?
●
Use list_select_related
●
Overwrite get_queryset() with prefetch_related
●
Is ordering using an index? Same for search_fields
●
readonly_fields will avoid FK/M2M queries
●
Use the raw_id_fields widget (or better:
django-salmonella)
●
Extend admin/filter.html to show filters as <select>
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Cachalot
●
Caches your Django ORM queries and
automatically invalidates them
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Queues and workers
●
Do slow stuff later
●
Some operations can be queued, and executed
asynchronously in workers
●
Use Celery
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Cached sessions
●
Use SESSION_ENGINE to set cached sessions:
– Non-persistent: don’t hit the DB
– Persistent: don’t hit the DB… so often
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Persistent connections
●
Use CONN_MAX_AGE to set the lifetime of a
database connection (persistence)
David Arcos - @DZPMEfficient Django – #EuroPython 2016
UUIDs
●
Use UUID for Primary Keys (instead of
incremental IDs)
– Guaranteed uniqueness, avoid collisions
– UUIDs are well-indexed
●
Easier db sharding
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Slow tests?
●
Skip migrations: --keepdb
●
Run in parallel: --parallel
●
Disable unused middlewares, installed_apps,
password hashers, logging, etc…
●
Use mocking whenever possible
David Arcos - @DZPMEfficient Django – #EuroPython 2016
4) Conclusions
●
Measure first
●
Optimize only the bottleneck
●
Go for the low-hanging fruit
●
Measure again
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Good resources
●
The official Django documentation
●
Book: “High Performance Django”
●
Blog: “Instagram Engineering”
●
“Latency Numbers Every Programmer Should Know”
David Arcos - @DZPMEfficient Django – #EuroPython 2016
Thanks for attending!
- Get the slides at http://slideshare.net/DZPM
- We are looking for engineers and data scientists!

More Related Content

What's hot

PSR-7 - Middleware - Zend Expressive
PSR-7 - Middleware - Zend ExpressivePSR-7 - Middleware - Zend Expressive
PSR-7 - Middleware - Zend ExpressiveMilad Arabi
 
The Django Web Application Framework 2
The Django Web Application Framework 2The Django Web Application Framework 2
The Django Web Application Framework 2fishwarter
 
Introduction to Django REST Framework, an easy way to build REST framework in...
Introduction to Django REST Framework, an easy way to build REST framework in...Introduction to Django REST Framework, an easy way to build REST framework in...
Introduction to Django REST Framework, an easy way to build REST framework in...Zhe Li
 
The Evil Tester's Guide to HTTP proxies Tutorial
The Evil Tester's Guide to HTTP proxies TutorialThe Evil Tester's Guide to HTTP proxies Tutorial
The Evil Tester's Guide to HTTP proxies TutorialAlan Richardson
 
Behat Workshop at WeLovePHP
Behat Workshop at WeLovePHPBehat Workshop at WeLovePHP
Behat Workshop at WeLovePHPMarcos Quesada
 
API Design & Security in django
API Design & Security in djangoAPI Design & Security in django
API Design & Security in djangoTareque Hossain
 
Drupal and contribution (2010 - 2011 / 2)
Drupal and contribution (2010 - 2011 / 2)Drupal and contribution (2010 - 2011 / 2)
Drupal and contribution (2010 - 2011 / 2)Peter Arato
 
Zend expressive workshop
Zend expressive workshopZend expressive workshop
Zend expressive workshopAdam Culp
 
WordPress automation and CI
WordPress automation and CIWordPress automation and CI
WordPress automation and CIRan Bar-Zik
 
GDD HTML5, Flash, and the Battle for Faster Cat Videos
GDD HTML5, Flash, and the Battle for Faster Cat VideosGDD HTML5, Flash, and the Battle for Faster Cat Videos
GDD HTML5, Flash, and the Battle for Faster Cat VideosGreg Schechter
 
Testing nightwatch, by David Torroija
Testing nightwatch, by David TorroijaTesting nightwatch, by David Torroija
Testing nightwatch, by David TorroijaDavid Torroija
 
10 things you should know about django
10 things you should know about django10 things you should know about django
10 things you should know about djangoAdieu
 
High Performance JavaScript 2011
High Performance JavaScript 2011High Performance JavaScript 2011
High Performance JavaScript 2011Nicholas Zakas
 
Buildr - build like you code
Buildr -  build like you codeBuildr -  build like you code
Buildr - build like you codeIzzet Mustafaiev
 
Django Article V0
Django Article V0Django Article V0
Django Article V0Udi Bauman
 
Automated testing in Drupal
Automated testing in DrupalAutomated testing in Drupal
Automated testing in DrupalArtem Berdishev
 

What's hot (20)

PSR-7 - Middleware - Zend Expressive
PSR-7 - Middleware - Zend ExpressivePSR-7 - Middleware - Zend Expressive
PSR-7 - Middleware - Zend Expressive
 
The Django Web Application Framework 2
The Django Web Application Framework 2The Django Web Application Framework 2
The Django Web Application Framework 2
 
Django
DjangoDjango
Django
 
Introduction to Django REST Framework, an easy way to build REST framework in...
Introduction to Django REST Framework, an easy way to build REST framework in...Introduction to Django REST Framework, an easy way to build REST framework in...
Introduction to Django REST Framework, an easy way to build REST framework in...
 
The Evil Tester's Guide to HTTP proxies Tutorial
The Evil Tester's Guide to HTTP proxies TutorialThe Evil Tester's Guide to HTTP proxies Tutorial
The Evil Tester's Guide to HTTP proxies Tutorial
 
Behat Workshop at WeLovePHP
Behat Workshop at WeLovePHPBehat Workshop at WeLovePHP
Behat Workshop at WeLovePHP
 
Introduction to django
Introduction to djangoIntroduction to django
Introduction to django
 
API Design & Security in django
API Design & Security in djangoAPI Design & Security in django
API Design & Security in django
 
Drupal and contribution (2010 - 2011 / 2)
Drupal and contribution (2010 - 2011 / 2)Drupal and contribution (2010 - 2011 / 2)
Drupal and contribution (2010 - 2011 / 2)
 
Zend expressive workshop
Zend expressive workshopZend expressive workshop
Zend expressive workshop
 
WordPress automation and CI
WordPress automation and CIWordPress automation and CI
WordPress automation and CI
 
GDD HTML5, Flash, and the Battle for Faster Cat Videos
GDD HTML5, Flash, and the Battle for Faster Cat VideosGDD HTML5, Flash, and the Battle for Faster Cat Videos
GDD HTML5, Flash, and the Battle for Faster Cat Videos
 
Testing nightwatch, by David Torroija
Testing nightwatch, by David TorroijaTesting nightwatch, by David Torroija
Testing nightwatch, by David Torroija
 
Python/Django Training
Python/Django TrainingPython/Django Training
Python/Django Training
 
10 things you should know about django
10 things you should know about django10 things you should know about django
10 things you should know about django
 
Go at Skroutz
Go at SkroutzGo at Skroutz
Go at Skroutz
 
High Performance JavaScript 2011
High Performance JavaScript 2011High Performance JavaScript 2011
High Performance JavaScript 2011
 
Buildr - build like you code
Buildr -  build like you codeBuildr -  build like you code
Buildr - build like you code
 
Django Article V0
Django Article V0Django Article V0
Django Article V0
 
Automated testing in Drupal
Automated testing in DrupalAutomated testing in Drupal
Automated testing in Drupal
 

Similar to Efficient Django

Serverless for High Performance Computing
Serverless for High Performance ComputingServerless for High Performance Computing
Serverless for High Performance ComputingLuciano Mammino
 
Creating a reasonable project boilerplate
Creating a reasonable project boilerplateCreating a reasonable project boilerplate
Creating a reasonable project boilerplateStanislav Petrov
 
Scalable Django Architecture
Scalable Django ArchitectureScalable Django Architecture
Scalable Django ArchitectureRami Sayar
 
NE Scala 2016 roundup
NE Scala 2016 roundupNE Scala 2016 roundup
NE Scala 2016 roundupHung Lin
 
Serverless for High Performance Computing
Serverless for High Performance ComputingServerless for High Performance Computing
Serverless for High Performance ComputingLuciano Mammino
 
Angular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - LinagoraAngular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - LinagoraLINAGORA
 
There is something about serverless
There is something about serverlessThere is something about serverless
There is something about serverlessgjdevos
 
Python Django Intro V0.1
Python Django Intro V0.1Python Django Intro V0.1
Python Django Intro V0.1Udi Bauman
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
Serverless? How (not) to develop, deploy and operate serverless applications.
Serverless? How (not) to develop, deploy and operate serverless applications.Serverless? How (not) to develop, deploy and operate serverless applications.
Serverless? How (not) to develop, deploy and operate serverless applications.gjdevos
 
Apache Spark Performance Observations
Apache Spark Performance ObservationsApache Spark Performance Observations
Apache Spark Performance ObservationsAdam Roberts
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSamuel Lampa
 
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
 Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F... Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...Databricks
 
Raising ux bar with offline first design
Raising ux bar with offline first designRaising ux bar with offline first design
Raising ux bar with offline first designKyrylo Reznykov
 
Intro To Django
Intro To DjangoIntro To Django
Intro To DjangoUdi Bauman
 
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGEko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGPablo Garbossa
 

Similar to Efficient Django (20)

Serverless for High Performance Computing
Serverless for High Performance ComputingServerless for High Performance Computing
Serverless for High Performance Computing
 
Free django
Free djangoFree django
Free django
 
Creating a reasonable project boilerplate
Creating a reasonable project boilerplateCreating a reasonable project boilerplate
Creating a reasonable project boilerplate
 
Scalable Django Architecture
Scalable Django ArchitectureScalable Django Architecture
Scalable Django Architecture
 
Software maintenance PyConPL 2016
Software maintenance PyConPL 2016Software maintenance PyConPL 2016
Software maintenance PyConPL 2016
 
NE Scala 2016 roundup
NE Scala 2016 roundupNE Scala 2016 roundup
NE Scala 2016 roundup
 
Serverless for High Performance Computing
Serverless for High Performance ComputingServerless for High Performance Computing
Serverless for High Performance Computing
 
Angular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - LinagoraAngular (v2 and up) - Morning to understand - Linagora
Angular (v2 and up) - Morning to understand - Linagora
 
django
djangodjango
django
 
There is something about serverless
There is something about serverlessThere is something about serverless
There is something about serverless
 
Python Django Intro V0.1
Python Django Intro V0.1Python Django Intro V0.1
Python Django Intro V0.1
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
Serverless? How (not) to develop, deploy and operate serverless applications.
Serverless? How (not) to develop, deploy and operate serverless applications.Serverless? How (not) to develop, deploy and operate serverless applications.
Serverless? How (not) to develop, deploy and operate serverless applications.
 
Apache Spark Performance Observations
Apache Spark Performance ObservationsApache Spark Performance Observations
Apache Spark Performance Observations
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programming
 
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
 Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F... Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
 
Raising ux bar with offline first design
Raising ux bar with offline first designRaising ux bar with offline first design
Raising ux bar with offline first design
 
Fuzzing - Part 2
Fuzzing - Part 2Fuzzing - Part 2
Fuzzing - Part 2
 
Intro To Django
Intro To DjangoIntro To Django
Intro To Django
 
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORINGEko10 workshop - OPEN SOURCE DATABASE MONITORING
Eko10 workshop - OPEN SOURCE DATABASE MONITORING
 

Recently uploaded

The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfFIDO Alliance
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...FIDO Alliance
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfFIDO Alliance
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Patrick Viafore
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераMark Opanasiuk
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityScyllaDB
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoUXDXConf
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101vincent683379
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekCzechDreamin
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...FIDO Alliance
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomCzechDreamin
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftshyamraj55
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka DoktorováCzechDreamin
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfFIDO Alliance
 

Recently uploaded (20)

The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджераIntro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, Ocado
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 

Efficient Django

  • 1. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Efficient Django
  • 2. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Abstract Tips and best practices for avoiding scalability issues and performance bottlenecks in Django ● 1) Basic concepts: the theory ● 2) Measuring: how to find bottlenecks ● 3) Tips and tricks ● 4) Conclusion (yes, it scales!)
  • 3. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Hi! ● I'm David Arcos ● Python/Django developer since 2008 ● Co-organizer at Python Barcelona ● CTO at Lead Ratings
  • 4. David Arcos - @DZPMEfficient Django – #EuroPython 2016 ● “We improve your sales conversions, using predictive algorithms to rate the leads” ● Prediction API, “Machine Learning as a Service” ● http://lead-ratings.com
  • 5. David Arcos - @DZPMEfficient Django – #EuroPython 2016 1) Basic concepts
  • 6. David Arcos - @DZPMEfficient Django – #EuroPython 2016 The Pareto Principle "For many events, roughly 80% of the effects come from 20% of the causes"
  • 7. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Prioritize and focus Focus on the few tasks that will have the most impact
  • 8. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Basic scalability “Potential to be enlarged to handle a growing amount of work” ● Stateless app servers – Load balance them, scale horizontally ● Keep the state on the database(s) – This is the difficult part! Each system is different
  • 9. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Database performance ● Do less requests: – Less reads – Less writes ● Do faster requests: – Indexed fields – De-normalize
  • 10. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Templates ● Cache them ● Jinja2 is a bit faster than the default engine – but cache them anyways ● You can do fragment caching (for blocks)
  • 11. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Cache ● Generic approach: cache at each stack level ● The cache documentation is excellent ● Beware of the cache invalidation!
  • 12. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Cache ● Generic approach: cache at each stack level ● The cache documentation is excellent ● Beware of the cache invalidation!
  • 13. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Bottlenecks ● Where is your bottleneck? ● CPU bound or I/O bound? – CPU? Run heavy calculations in async workers – Memory? Compress objects before caching – Database? Read from db replicas ● How to find it?
  • 14. David Arcos - @DZPMEfficient Django – #EuroPython 2016 2) Measuring
  • 15. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Can't improve what you don't measure ● Measure your system to find bottlenecks ● Optimize those bottlenecks ● Verify the improvements ● Rinse and repeat!
  • 16. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Monitoring ● System: load, CPU, memory... ● Database: q/s, response time, size ● Cache: q/s, hit rate ● Queue: length ● Custom: metrics for your app
  • 17. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Profiling ● The cProfile module provides profiling of Python programs by collecting data: – Number of calls, running time, time per call...
  • 18. David Arcos - @DZPMEfficient Django – #EuroPython 2016 timeit ● The timeit module is a simple way to time execution time of small bits of Python code:
  • 19. David Arcos - @DZPMEfficient Django – #EuroPython 2016 ipdb ● Like pdb, but for ipython – tab completion, syntax highlighting, better tracebacks, better introspection… ● Use ipdb.set_trace() to add a breakpoint and jump in with the debugger
  • 20. David Arcos - @DZPMEfficient Django – #EuroPython 2016 django-debug-toolbar ● Display debug information about the current request/response ● Panels, very modular
  • 21. David Arcos - @DZPMEfficient Django – #EuroPython 2016 django-debug-toolbar-line-profiler ● A toolbar panel for profiling Django Debug Panel ● Chrome extension ● For AJAX requests and non-HTML responses
  • 22. David Arcos - @DZPMEfficient Django – #EuroPython 2016 3) Tips and tricks
  • 23. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Add db indexes ● Single (db_index) or multiple (index_together) ● Be sure to profile and measure! – Sometimes it’s not obvious (i.e., admin) – Huge difference, i.e. from 15s to 3 ms (3.5M rows) ● But: uses more space, slower writes
  • 24. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Do bulk operations ● Will greatly reduce the number of SQL queries: – Model.objects.bulk_create() – qs.update() <- maybe with F() expressions – qs.delete()
  • 25. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Get related objects ● Return FK fields in same query: – qs.select_related() ● Return M2M fields, extra query: – qs.prefetch_related()
  • 26. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Slow admin? ● Use list_select_related ● Overwrite get_queryset() with prefetch_related ● Is ordering using an index? Same for search_fields ● readonly_fields will avoid FK/M2M queries ● Use the raw_id_fields widget (or better: django-salmonella) ● Extend admin/filter.html to show filters as <select>
  • 27. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Cachalot ● Caches your Django ORM queries and automatically invalidates them
  • 28. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Queues and workers ● Do slow stuff later ● Some operations can be queued, and executed asynchronously in workers ● Use Celery
  • 29. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Cached sessions ● Use SESSION_ENGINE to set cached sessions: – Non-persistent: don’t hit the DB – Persistent: don’t hit the DB… so often
  • 30. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Persistent connections ● Use CONN_MAX_AGE to set the lifetime of a database connection (persistence)
  • 31. David Arcos - @DZPMEfficient Django – #EuroPython 2016 UUIDs ● Use UUID for Primary Keys (instead of incremental IDs) – Guaranteed uniqueness, avoid collisions – UUIDs are well-indexed ● Easier db sharding
  • 32. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Slow tests? ● Skip migrations: --keepdb ● Run in parallel: --parallel ● Disable unused middlewares, installed_apps, password hashers, logging, etc… ● Use mocking whenever possible
  • 33. David Arcos - @DZPMEfficient Django – #EuroPython 2016 4) Conclusions ● Measure first ● Optimize only the bottleneck ● Go for the low-hanging fruit ● Measure again
  • 34. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Good resources ● The official Django documentation ● Book: “High Performance Django” ● Blog: “Instagram Engineering” ● “Latency Numbers Every Programmer Should Know”
  • 35. David Arcos - @DZPMEfficient Django – #EuroPython 2016 Thanks for attending! - Get the slides at http://slideshare.net/DZPM - We are looking for engineers and data scientists!