This document discusses techniques for improving the performance of Django projects handling high traffic volumes. It identifies common areas of concern like database performance, web server configuration, caching, and template rendering. It provides examples of optimizing ORM queries, implementing object caching with invalidation plans, profiling code to identify bottlenecks, and leveraging tools like Varnish, memcached, and database query profiling. The key lessons are to develop a caching strategy, use profiling to optimize problem areas, and consider alternative web server software or configurations to improve performance.
Quando uma aplicação começa a ficar grande e complexa, fazer buscas nos seus models torna-se uma tarefa complicada. Efetuar as buscas diretamente no banco de dados é um processo lento, ineficiente e que permite pouca ou nenhuma maleabilidade sobre a forma com que a busca é feita. Surge então o ElasticSearch, uma engine de busca utilizada por empresas como Github, Twitter e 4square para indexar e buscar literalmente milhões de documentos em tempo real. Nessa palestra, explicarei quando, como e porque utilizar o ElasticSearch para facilmente indexar e efetuar buscas complexas nos seus models.
Quando uma aplicação começa a ficar grande e complexa, fazer buscas nos seus models torna-se uma tarefa complicada. Efetuar as buscas diretamente no banco de dados é um processo lento, ineficiente e que permite pouca ou nenhuma maleabilidade sobre a forma com que a busca é feita. Surge então o ElasticSearch, uma engine de busca utilizada por empresas como Github, Twitter e 4square para indexar e buscar literalmente milhões de documentos em tempo real. Nessa palestra, explicarei quando, como e porque utilizar o ElasticSearch para facilmente indexar e efetuar buscas complexas nos seus models.
Beyond PHP - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just writing PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
This was a talk given at HTML5DevConf SF in 2015.
Ever wanted to write your own Browserify or Babel? Maybe have an idea for something new? This talk will get you started understanding how to use a JavaScript AST to transform and generate new code.
Cache Money is a new model caching framework for Active Record that transparently stores, synchronizes and expires model instances in memcached, through Active Record callbacks. It's a framework that provides almost all the pieces of transparently turning an application without model caching into one with model caching at the flip of a switch. In this presentation I'll show what the framework delivers, its limitation and what problems to watch out for when putting it in as drop-in intermediary for your database connection.
As present at FluentConf 2014 on March 11th, 2014.
AngularJS is one of the most popular, and powerful, JavaScript frameworks for building rich client-side applications. AngularJS is both simultaneously both simple to use and extremely full featured. With AngularJS a little goes a long way, but to make the most of it, you need to know what you’re doing.
In this workshop we will build a complex application to help exercise all of the salient points of the AngularJS framework.
Topics covered include, ngResource, directives, fitlers, routing, templates, controllers, testing, and more.
Code can be found at: https://github.com/markbates/fluent-2014
Meet Magento Sweden - Magento 2 Layout and Code Compilation for PerformanceIvan Chepurnyi
It is a pity, but I have to admit, that Magento 2 has issues in layout generation process. You'll learn during the talk about the benefits of compiling XML structures into PHP code, and even use the compilation of PHP code into PHP code to speed up your modules complex logic.
A look at how HTML5 aims to plug the holes that Flash has been filling in browsers for the last decade, looking at both HTML5 and non-HTML5 JavaScript APIs.
For Flash Brighton in Feb 2010.
Django, Framework Python para desenvolvimento webMayron Cachina
Esse foi o slide de um mini curso que apresentei no FLISOL 2008 que ocorreu no CEFET-RN no dia 26/04/08, infelizmente aconteceram várias coisas e n deu tempo para eu fazer um slide legal, deixei de mostrar muitas coisas, além de n poder mostrar a versão mais nova (0.97), foi instalado no pc a 0.96.1
Visite http://cachina.wordpress.com/2008/04/29/flisol-minicurso-django/ para ver a resenha :D
Beyond PHP - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just writing PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
This was a talk given at HTML5DevConf SF in 2015.
Ever wanted to write your own Browserify or Babel? Maybe have an idea for something new? This talk will get you started understanding how to use a JavaScript AST to transform and generate new code.
Cache Money is a new model caching framework for Active Record that transparently stores, synchronizes and expires model instances in memcached, through Active Record callbacks. It's a framework that provides almost all the pieces of transparently turning an application without model caching into one with model caching at the flip of a switch. In this presentation I'll show what the framework delivers, its limitation and what problems to watch out for when putting it in as drop-in intermediary for your database connection.
As present at FluentConf 2014 on March 11th, 2014.
AngularJS is one of the most popular, and powerful, JavaScript frameworks for building rich client-side applications. AngularJS is both simultaneously both simple to use and extremely full featured. With AngularJS a little goes a long way, but to make the most of it, you need to know what you’re doing.
In this workshop we will build a complex application to help exercise all of the salient points of the AngularJS framework.
Topics covered include, ngResource, directives, fitlers, routing, templates, controllers, testing, and more.
Code can be found at: https://github.com/markbates/fluent-2014
Meet Magento Sweden - Magento 2 Layout and Code Compilation for PerformanceIvan Chepurnyi
It is a pity, but I have to admit, that Magento 2 has issues in layout generation process. You'll learn during the talk about the benefits of compiling XML structures into PHP code, and even use the compilation of PHP code into PHP code to speed up your modules complex logic.
A look at how HTML5 aims to plug the holes that Flash has been filling in browsers for the last decade, looking at both HTML5 and non-HTML5 JavaScript APIs.
For Flash Brighton in Feb 2010.
Django, Framework Python para desenvolvimento webMayron Cachina
Esse foi o slide de um mini curso que apresentei no FLISOL 2008 que ocorreu no CEFET-RN no dia 26/04/08, infelizmente aconteceram várias coisas e n deu tempo para eu fazer um slide legal, deixei de mostrar muitas coisas, além de n poder mostrar a versão mais nova (0.97), foi instalado no pc a 0.96.1
Visite http://cachina.wordpress.com/2008/04/29/flisol-minicurso-django/ para ver a resenha :D
A small presentation by me at the Barcamp Vienna in Sept. 2007 where I tried to give a short overview about some of the features I really love about Django :D
PHP classes in mumbai, Introduction to PHP/MYSQL..
best PHP/MYSQL classes in mumbai with job assistance.
our features are:
expert guidance by IT industry professionals
lowest fees of 5000
practical exposure to handle projects
well equiped lab
after course resume writing guidance
For more Visit: http://vibranttechnologies.co.in/php-classes-in-mumbai.html or http://phptraining.vibranttechnologies.co.in
This presentation was prepared for a Webcast where John Yerhot, Engine Yard US Support Lead, and Chris Kelly, Technical Evangelist at New Relic discussed how you can scale and improve the performance of your Ruby web apps. They shared detailed guidance on issues like:
Caching strategies
Slow database queries
Background processing
Profiling Ruby applications
Picking the right Ruby web server
Sharding data
Attendees will learn how to:
Gain visibility on site performance
Improve scalability and uptime
Find and fix key bottlenecks
See the on-demand replay:
http://pages.engineyard.com/6TipsforImprovingRubyApplicationPerformance.html
Behind the curtain - How Django handles a requestDaniel Hepper
Does Django sometimes feel like Magic to you? In this presentation, we will peek behind the curtain to understand the wizardry that turns an HTTP request into a response.
Despite rumors to the contrary, Django does not actually involve any magic - it's just code! We will dive deep into this code and trace the exact steps that turn an HTTP request into a response.
On this exciting journey, you'll get a high-level overview of the different components that make up Django and dissect elegant code constructs.
Originally presented at DjangoCon Europe 2018 in Heidelberg, Germany.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
2. Curse
• Peak daily traffic of approx. 15m pages, 150m hits.
• Average monthly traffic 120m pages, 6m uniques.
• Python, MySQL, Squid, memcached, mod_python, lighty.
• Most developers came strictly from PHP (myself included).
• 12 web servers, 4 database servers, 2 squid caches.
3. iBegin
• Massive amounts of data, 100m+ rows.
• Python, PHP, MySQL, mod_wsgi.
• Small team of developers.
• Complex database partitioning/synchronization tasks.
• Attempting to not branch off of Django.
4. Areas of Concern
• Database (ORM)
• Webserver (Resources, Handling Millions of Reqs)
• Caching (Invalidation, Cache Dump)
• Template Rendering (Logic Separation)
• Profiling
6. How We Did It
• “Primary” web servers serving Django using mod_python.
• Media servers using Django on lighttpd.
• Static served using additional instances of lighttpd.
• Load balancers passing requests to multiple Squids.
• Squids passing requests to multiple web servers.
7. Lessons Learned
• Don’t be afraid to experiment. You’re not limited to a one.
• mod_wsgi is a huge step forward from mod_python.
• Serving static files using different software can help.
• Send proper HTTP headers where they are needed.
• Use services like S3, Akamai, Limelight, etc..
9. Database (ORM)
• Won’t make your queries efficient. Make your own indexes.
• select_related() can be good, as well as bad.
• Inherited ordering (Meta: ordering) will get you.
• Hundreds of queries on a page is never a good thing.
• Know when to not use the ORM.
10. Handling JOINs
class Category(models.Model):
name = models.CharField()
created_by = models.ForeignKey(User)
class Poll(models.Model):
name = models.CharField()
category = models.ForeignKey(Category)
created_by = models.ForeignKey(User)
# We need to output a page listing all Poll's with
# their name and category's name.
def a_bad_example(request):
# We have just caused Poll to JOIN with User and Category,
# which will also JOIN with User a second time.
my_polls = Poll.objects.all().select_related()
return render_to_response('polls.html', locals(), request)
def a_good_example(request):
# Use select_related explicitly in each case.
poll = Poll.objects.all().select_related('category')
return render_to_response('polls.html', locals(), request)
11. Template Rendering
• Sandboxed engines are typically slower by nature.
• Keep logic in views and template tags.
• Be aware of performance in loops, and groupby (regroup).
• Loaded templates can be cached to avoid disk reads.
• Switching template engines is easy, but may not give you
any worthwhile performance gain.
13. Caching
• Two flavors of caching: object cache and browser cache.
• Django provides built-in support for both.
• Invalidation is a headache without a well thought out plan.
• Caching isn’t a solution for slow loading pages or improper indexes.
• Use a reverse proxy in between the browser and your web servers:
Squid, Varnish, Nginx, etc..
14. Cache With a Plan
• Build your pages to use proper cache headers.
• Create a plan for object cache expiration, and invalidation.
• For typical web apps you can serve the same cached page
for both anonymous and authenticated users.
• Contain commonly used querysets in managers for
transparent caching and invalidation.
15. Cache Commonly Used Items
def my_context_processor(request):
# We access object_list every time we use our context processors so
# it makes sense to cache this, no?
cache_key = ‘mymodel:all’
object_list = cache.get(cache_key)
if object_list is None:
object_list = MyModel.objects.all()
cache.set(cache_key, object_list)
return {‘object_list’: object_list}
# Now that we are caching the object list we are going to want to invalidate it
class MyModel(models.Model):
name = models.CharField()
def save(self, *args, **kwargs):
super(MyModel, self).save(*args, **kwargs)
# save it before you update the cache
cache.set(‘mymodel:all’, MyModel.objects.all())
16. Profiling Code
• Finding the bottleneck can be time consuming.
• Tools exist to help identify common problematic areas.
– cProfile/Profile Python modules.
– PDB (Python Debugger)
17. Profiling Code With cProfile
import sys
try: import cProfile as profile
except ImportError: import profile
try: from cStringIO import StringIO
except ImportError: import StringIO
from django.conf import settings
class ProfilerMiddleware(object):
def can(self, request):
return settings.DEBUG and 'prof' in request.GET and (not settings.INTERNAL_IPS or request.META['REMOTE_ADDR'] in
settings.INTERNAL_IPS)
def process_view(self, request, callback, callback_args, callback_kwargs):
if self.can(request):
self.profiler = profile.Profile()
args = (request,) + callback_args
return self.profiler.runcall(callback, *args, **callback_kwargs)
def process_response(self, request, response):
if self.can(request):
self.profiler.create_stats()
out = StringIO()
old_stdout, sys.stdout = sys.stdout, out
self.profiler.print_stats(1)
sys.stdout = old_stdout
response.content = '<pre>%s</pre>' % out.getvalue()
return response
21. Summary
• Database efficiency is the typical problem in web apps.
• Develop and deploy a caching plan early on.
• Use profiling tools to find your problematic areas. Don’t pre-
optimize unless there is good reason.
• Find someone who knows more than me to configure your
server software.