SlideShare a Scribd company logo
1 of 27
Download to read offline
Faster Programming in
Python/Django
Subit Raj Pokharel
@ rajsubit,
rajsubit
First Code then optimize
Response Time Limits
● 0.1 seconds
○ limit for having the user feel that the system is reacting
instantaneously
● 1 seconds
○ limit for the user's flow of thought to stay uninterrupted
● 10 seconds
○ limit for keeping the user's attention focused on the dialogue
http://www.nngroup.com/articles/response-times-3-important-limits/
Time consuming function
New Relic
http://newrelic.com/
def find_recent_blog_post():
...
url = requests.get("http://blog.flipkarma.com/feed")
soup = BeautifulSoup.BeautifulSoup(url.text)
recent_post = soup.findAll('item')[:2]
...
A simple example
Analysis with Line Profiler
Line # Hits Time Per Hit % Time Line Contents
==============================================================
70 1 1112952 1112952.0 73.4 url = requests.get("http://blog.flipkarma.com/feed")
71 1 103695 103695.0 6.8 soup = BeautifulSoup.BeautifulSoup(url.text)
72 1 2186 2186.0 0.1 recent_post = soup.findAll('item')[:2]
https://github.com/rkern/line_profiler
$ pip install line_profiler
Optimization with Memcached
# Original code
url = requests.get("http://blog.flipkarma.com/feed")
soup = BeautifulSoup.BeautifulSoup(url.text)
recent_post = soup.findAll('item')[:2]
# Using Memcached
key = "flipkarma_blog_post"
cache_time = 10000
result = cache.get(key)
if result:
recent_post = result
else:
url = requests.get("http://blog.flipkarma.com/feed")
soup = BeautifulSoup.BeautifulSoup(url.text)
recent_post = soup.findAll('item')[:2]
cache.set(key, recent_post, cache_time)
http://memcached.org/
Line # Hits Time Per Hit % Time Line Contents
==============================================================
65 1 3 3.0 0.0 cache_time = 10000 # time to live in seconds
66 1 17973 17973.0 1.3 result = cache.get(cache_key)
67 1 4 4.0 0.0 if result:
68 1 31 31.0 0.0 parameters["result"] = result
69 1 3 3.0 0.0 if not result:
74 url = requests.get("http://blog.flipkarma.com/feed")
soup = BeautifulSoup.BeautifulSoup(url.text)
recent_post = soup.findAll('item')[:2]
cache.set(cache_key, recent_post, cache_time)
Result
New Relic
● analyze the application
● help understand the stories of application
data
● real-time business insights
http://newrelic.com/
Profiling
Python Tools:
● cProfile
● profile
● hotshot
Django Debug Toolbar
Line Profiler
Django Debug Toolbar
SQL Query
Query Time
Reducing Query
Query Time
Some Tips
If it's fast enough, don't optimise it.
Find the slowest step first.
Make the slowest operation faster
● Python functions have an overhead. Inline if possible
● List comprehensions are faster than for or if
Reduce the number of hits.
Perform each operation as rarely as
possible.
● Cache the result if speed is more important than memory
● Move slower operations inside if conditions
● assign operation to variable only if necessary
Reducing Queries
● Use of Class Based Views
● proper use of query sets
● select_related() and prefetch_related()
Change the algorithm.
This has the second-biggest impact on speed
The largest impact comes from
eliminating code.
Functionality is an asset. Code is a
liability.
Thank You

More Related Content

Similar to Faster python/django programming

Garelic: Google Analytics as App Performance monitoring
Garelic: Google Analytics as App Performance monitoringGarelic: Google Analytics as App Performance monitoring
Garelic: Google Analytics as App Performance monitoring
Jano Suchal
 
Search Lucene
Search LuceneSearch Lucene
Search Lucene
Jeremy Coates
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails Applications
Serge Smetana
 
Scrum Primer
Scrum PrimerScrum Primer
Scrum Primer
davelucey
 

Similar to Faster python/django programming (20)

Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
Faster Python Programs Through Optimization by Dr.-Ing Mike MullerFaster Python Programs Through Optimization by Dr.-Ing Mike Muller
Faster Python Programs Through Optimization by Dr.-Ing Mike Muller
 
PyGrunn2013 High Performance Web Applications with TurboGears
PyGrunn2013  High Performance Web Applications with TurboGearsPyGrunn2013  High Performance Web Applications with TurboGears
PyGrunn2013 High Performance Web Applications with TurboGears
 
Angular - Improve Runtime performance 2019
Angular - Improve Runtime performance 2019Angular - Improve Runtime performance 2019
Angular - Improve Runtime performance 2019
 
Creating a Smooth Development Workflow for High-Quality Modular Open-Source P...
Creating a Smooth Development Workflow for High-Quality Modular Open-Source P...Creating a Smooth Development Workflow for High-Quality Modular Open-Source P...
Creating a Smooth Development Workflow for High-Quality Modular Open-Source P...
 
Garelic: Google Analytics as App Performance monitoring
Garelic: Google Analytics as App Performance monitoringGarelic: Google Analytics as App Performance monitoring
Garelic: Google Analytics as App Performance monitoring
 
Journey through high performance django application
Journey through high performance django applicationJourney through high performance django application
Journey through high performance django application
 
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
 
Search Lucene
Search LuceneSearch Lucene
Search Lucene
 
"Swoole: double troubles in c", Alexandr Vronskiy
"Swoole: double troubles in c", Alexandr Vronskiy"Swoole: double troubles in c", Alexandr Vronskiy
"Swoole: double troubles in c", Alexandr Vronskiy
 
PyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web ApplicationsPyCon AU 2012 - Debugging Live Python Web Applications
PyCon AU 2012 - Debugging Live Python Web Applications
 
Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails Applications
 
Become a GC Hero
Become a GC HeroBecome a GC Hero
Become a GC Hero
 
Salesforce Apex Hours :- Hyper batch
Salesforce Apex Hours :- Hyper batchSalesforce Apex Hours :- Hyper batch
Salesforce Apex Hours :- Hyper batch
 
Design & Develop Batch Applications in Java/JEE
Design & Develop Batch Applications in Java/JEEDesign & Develop Batch Applications in Java/JEE
Design & Develop Batch Applications in Java/JEE
 
Scrum Primer
Scrum PrimerScrum Primer
Scrum Primer
 
EuroPython 2013 - Python3 TurboGears Training
EuroPython 2013 - Python3 TurboGears TrainingEuroPython 2013 - Python3 TurboGears Training
EuroPython 2013 - Python3 TurboGears Training
 
Building source code level profiler for C++.pdf
Building source code level profiler for C++.pdfBuilding source code level profiler for C++.pdf
Building source code level profiler for C++.pdf
 
Day 7 - Make it Fast
Day 7 - Make it FastDay 7 - Make it Fast
Day 7 - Make it Fast
 
jBPM5 Community Training Module 4: jBPM5 APIs Overview + Hands On
jBPM5 Community Training Module 4: jBPM5 APIs Overview + Hands OnjBPM5 Community Training Module 4: jBPM5 APIs Overview + Hands On
jBPM5 Community Training Module 4: jBPM5 APIs Overview + Hands On
 

Recently uploaded

Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 

Recently uploaded (20)

FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 

Faster python/django programming

  • 1. Faster Programming in Python/Django Subit Raj Pokharel @ rajsubit, rajsubit
  • 2. First Code then optimize
  • 3. Response Time Limits ● 0.1 seconds ○ limit for having the user feel that the system is reacting instantaneously ● 1 seconds ○ limit for the user's flow of thought to stay uninterrupted ● 10 seconds ○ limit for keeping the user's attention focused on the dialogue http://www.nngroup.com/articles/response-times-3-important-limits/
  • 4. Time consuming function New Relic http://newrelic.com/
  • 5. def find_recent_blog_post(): ... url = requests.get("http://blog.flipkarma.com/feed") soup = BeautifulSoup.BeautifulSoup(url.text) recent_post = soup.findAll('item')[:2] ... A simple example
  • 6. Analysis with Line Profiler Line # Hits Time Per Hit % Time Line Contents ============================================================== 70 1 1112952 1112952.0 73.4 url = requests.get("http://blog.flipkarma.com/feed") 71 1 103695 103695.0 6.8 soup = BeautifulSoup.BeautifulSoup(url.text) 72 1 2186 2186.0 0.1 recent_post = soup.findAll('item')[:2] https://github.com/rkern/line_profiler $ pip install line_profiler
  • 7. Optimization with Memcached # Original code url = requests.get("http://blog.flipkarma.com/feed") soup = BeautifulSoup.BeautifulSoup(url.text) recent_post = soup.findAll('item')[:2] # Using Memcached key = "flipkarma_blog_post" cache_time = 10000 result = cache.get(key) if result: recent_post = result else: url = requests.get("http://blog.flipkarma.com/feed") soup = BeautifulSoup.BeautifulSoup(url.text) recent_post = soup.findAll('item')[:2] cache.set(key, recent_post, cache_time) http://memcached.org/
  • 8. Line # Hits Time Per Hit % Time Line Contents ============================================================== 65 1 3 3.0 0.0 cache_time = 10000 # time to live in seconds 66 1 17973 17973.0 1.3 result = cache.get(cache_key) 67 1 4 4.0 0.0 if result: 68 1 31 31.0 0.0 parameters["result"] = result 69 1 3 3.0 0.0 if not result: 74 url = requests.get("http://blog.flipkarma.com/feed") soup = BeautifulSoup.BeautifulSoup(url.text) recent_post = soup.findAll('item')[:2] cache.set(cache_key, recent_post, cache_time) Result
  • 9. New Relic ● analyze the application ● help understand the stories of application data ● real-time business insights http://newrelic.com/
  • 10.
  • 11. Profiling Python Tools: ● cProfile ● profile ● hotshot Django Debug Toolbar Line Profiler
  • 18. If it's fast enough, don't optimise it.
  • 19. Find the slowest step first.
  • 20. Make the slowest operation faster ● Python functions have an overhead. Inline if possible ● List comprehensions are faster than for or if
  • 21. Reduce the number of hits.
  • 22. Perform each operation as rarely as possible. ● Cache the result if speed is more important than memory ● Move slower operations inside if conditions ● assign operation to variable only if necessary
  • 23. Reducing Queries ● Use of Class Based Views ● proper use of query sets ● select_related() and prefetch_related()
  • 24. Change the algorithm. This has the second-biggest impact on speed
  • 25. The largest impact comes from eliminating code.
  • 26. Functionality is an asset. Code is a liability.