SlideShare a Scribd company logo
Monkeying Around
  At New Relic
      Graham Dumpleton
  DjangoCon September 2011
Web interface (1)
Web interface (1)




THROUGHPUT
Web interface (1)




       RESPONSE TIME
Web interface (1)




        BREAKDOWN
Web interface (1)



EXCEPTIONS
Web interface (1)



SLOW TRANSACTIONS
Web interface (2)
Web interface (3)
Web interface (4)
Web transactions

✴When they occurred.
✴How long they ran for.
✴What handled the request.
✴What work was done.
✴Were there any errors.
What work was done?


✴Functions called.
✴Databases operations.
✴External service requests.
✴Memcache operations.
Trace class (1)

import time

class Trace(object):
    def __init__(self):
        self.start = 0
    def enter(self):
        self.start = time.time()
    def exit(self):
        self.duration = time.time() - self.start
        print self.duration
Timing code block

def function():
    time.sleep(0.1)

trace = Trace()

trace.enter()
for i in range(10):
    function()
trace.exit()
Stop on exceptions


def function():
    time.sleep(0.1)
    raise MissingThisTalk('@pydanny')

trace = Trace()

trace.enter()
try:
     for i in range(10):
         function()
finally:
     trace.exit()
Catch exception (1)

def function():
    time.sleep(0.1)
    raise NotVeryHappy('@pydanny')

trace = Trace()

trace.enter()
try:
   for i in range(10):
         function()
except:
     print sys.exc_info()
     raise
finally:
     trace.exit()
Trace class (2)

import time

class Trace(object):
    def __init__(self):
        self.start = 0
    def enter(self):
        self.start = time.time()
    def exit(self,
            exc=None, value=None, tb=None):
        self.duration = time.time() - self.start
        print exc, value, tb
        print self.duration
Catch exception (2)

trace = Trace()

success = True
trace.enter()
try:
     function()
except:
     success = False
     trace.exit(*sys.exc_info())
     raise
finally:
     if success:
         trace.exit(None, None, None)
Hang on!


✴Doesn’t this pattern look familiar.
Hang on!


✴Doesn’t this pattern look familiar.
✴Isn’t this just a context manager?
Context managers


trace = Trace()

with trace:
    function()
Trace class (3)

import time

class Trace(object):
    def __init__(self):
        self.start = 0
    def __enter__(self):
        self.start = time.time()
    def __exit__(self,
            exc=None, value=None, tb=None):
        self.duration = time.time() - self.start
        print exc, value, tb
        print self.duration
Catch exception (3)

trace = Trace()

success = True
trace.__enter__()
try:
     function()
except:
     success = False
     if not trace.__exit__(*sys.exc_info()):
         raise
finally:
     if success:
         trace.__exit__(None, None, None)
Decorating functions (1)


def decorator(f):
    def wrapper(*args, **kwargs):
        with Trace():
            return f(*args, **kwargs)
    return wrapper

@decorator
def function():
    time.sleep(1.0)

function()
Decorator issues (1)


def decorator(f):
    def wrapper(*args, **kwargs):
        with Trace():
            return f(*args, **kwargs)
    return wrapper

@decorator
@csrf_exempt
def handler():
    ...
Decorator issues (1)


def decorator(f):
    def wrapper(*args, **kwargs):
        with Trace():
            return f(*args, **kwargs)
    return wrapper

@decorator
@csrf_exempt
def handler():
    ...

    Why is my view expecting CSRF token?
Decorating functions (2)


import functools

def decorator(f):
    @functools.wraps(f)
    def wrapper(*args, **kwargs):
        with Trace():
            return f(*args, **kwargs)
    return wrapper

@decorator
def function():
    time.sleep(1.0)

function()
Decorator issues (2)


class Object(object):
    @decorator
    @staticmethod
    def function():
        time.sleep(1.0)

o = Object()

o.function()
Decorator issues (2)


class Object(object):
    @decorator
    @staticmethod
    def function():
        time.sleep(1.0)

o = Object()

o.function()


  AttributeError: 'staticmethod' object
      has no attribute '__module__'
Descriptor issues

def available_attrs(f):
    # http://bugs.python.org/issue3445.
    return tuple(a for a in
            functools.WRAPPER_ASSIGNMENTS
            if hasattr(f, a))

def decorator(f):
    @functools.wraps(f, available_attrs(f))
    def wrapper(*args, **kwargs):
        with Trace():
            return f(*args, **kwargs)
    return wrapper
Descriptor issues

def available_attrs(f):
    # http://bugs.python.org/issue3445.
    return tuple(a for a in
            functools.WRAPPER_ASSIGNMENTS
            if hasattr(f, a))

def decorator(f):
    @functools.wraps(f, available_attrs(f))
    def wrapper(*args, **kwargs):
        with Trace():
            return f(*args, **kwargs)
    return wrapper

    TypeError: 'staticmethod' object is
                  not callable
Decorating functions (2)

class Wrapper(object):
    def __init__(self, func):
        functools.update_wrapper(self, func,
                available_attrs(func))
        self.func = func

    def __get__(self, instance, klass):
        if instance is None: return self
        desc = self.func.__get__(instance, klass)
        return self.__class__(desc)

    def __call__(self, *args, **kwargs):
        with Trace():
            return self.func(*args, **kwargs)

def decorator(func):
    return Wrapper(func)
WSGI application (1)


@decorator
def application(environ, start_response):
    status = '200 OK'
    start_response(status,
            [('Content-type', 'text/plain'),])

    time.sleep(1.0)
    return ['Is @pydanny here yet?']
WSGI application (2)


@decorator
def application(environ, start_response):
    status = '200 OK'
    start_response(status,
            [('Content-type', 'text/plain'),])

    time.sleep(1.0)
    yield 'Maybe @audreyr is watching.'
WSGI middleware (1)

class Wrapper(object):

   ...

   def __call__(self, environ, start_response):
       trace = Trace()
       trace.__enter__()

         try:
             result = self.f(environ, start_response)
         except:
             trace.__exit__(*sys.exc_info())
             raise

         return Iterable(trace, result)
WSGI middleware (2)

class Iterable(object):

   def __init__(self, trace, generator):
       self.trace = trace
       self.generator = generator

   def __iter__(self):
       for item in self.generator:
           yield item

   ...
WSGI middleware (3)

class Iterable(object):

   ...

   def close(self):
       try:
            if hasattr(self.generator, 'close'):
                self.generator.close()
       except:
            self.trace.__exit__(*sys.exc_info())
            raise
       else:
            self.trace.__exit__(None, None, None)
Agent initialisation (1)


import newrelic.agent
newrelic.agent.initialize('/etc/newrelic.ini')

@newrelic.agent.wsgi_application()
def application(environ, start_response):
    ...
Agent initialisation (2)


import newrelic.agent
newrelic.agent.initialize('/etc/newrelic.ini')

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
PEP 369 / Post import hooks




@imp.when_imported('django.core.handlers.wsgi')
def instrument(module):
    method = module.WSGIHandler.__call__
    module.WSGIHandler.__call__ = 
        WSGIApplicationWrapper(method)
DIY post import hooks (1)


class ImportHookFinder:
    def __init__(self):
        self._skip = {}

   def find_module(self, fullname, path=None):
       if not fullname in _import_hooks: return None
       if fullname in self._skip: return None
       self._skip[fullname] = True
       try:
            __import__(fullname)
       finally:
            del self._skip[fullname]
       return ImportHookLoader()
DIY post import hooks (2)


class ImportHookLoader:

    def load_module(self, fullname):
        module = sys.modules[fullname]
        _notify_import_hooks(fullname, module)
        return module

sys.meta_path.insert(0, ImportHookFinder())
Instrumentation


from newrelic.api.web_transaction 
    import wrap_wsgi_application

def instrument(module):
    wrap_wsgi_application(module,
            'WSGIHandler.__call__')
WSGI applications (1)


✴Context manager
 ✴Trace -> WebTransaction
WSGI applications (1)


✴Context manager
 ✴Trace -> WebTransaction
✴Wrapper/Descriptor
 ✴Wrapper -> WSGIApplicationWrapper
WSGI applications (2)


✴Decorator
 ✴decorator -> wsgi_application()
WSGI applications (2)


✴Decorator
 ✴decorator -> wsgi_application()
✴Monkey patching
 ✴wrap_wsgi_application()
Function trace (1)

✴Context manager
 ✴Trace -> FunctionTrace
Function trace (1)

✴Context manager
 ✴Trace -> FunctionTrace
✴Wrapper/Descriptor
 ✴Wrapper -> FunctionTraceWrapper
Function Trace (2)

✴Decorator
 ✴decorator -> function_trace()
Function Trace (2)

✴Decorator
 ✴decorator -> function_trace()
✴Monkey patching
 ✴wrap_function_trace()
Same again


✴DatabaseTrace, etc
✴ExternalTrace, etc
✴MemcacheTrace, etc
Database Trace (1)


from newrelic.api.database_trace 
    import wrap_database_trace

def instrument(module):
    def _sql(cursor, sql, params=()):
        return sql

    wrap_database_trace(module,
            'Cursor.execute',
            sql=_sql)
Database Trace (2)

class DatabaseTraceWrapper(object):

    def __init__(self, func, sql):
       functools.update_wrapper(self, func,
                available_attrs(func))

        if type(func) == tuples.TupleType:
            self.instance, func = func
        else:
            self.instance = None

        self.func = func
        self.sql = sql
Database Trace (3)


class DatabaseTraceWrapper(object):

   def __get__(self, instance, klass):
        if instance is None: return self
        desc = self.func.__get__(instance, klass)
        return self.__class__((instance, desc))
Database Trace (4)
class DatabaseTraceWrapper(object):

    def __call__(self, *args, **kwargs):

        if not isinstance(self.sql, basestring):
            if (self.instance and
                    inspect.ismethod(self.func)):
                sql = self.sql(self.instance,
                        *args, **kwargs)
            else:
                sql = self.sql(*args, **kargs)
        else:
            sql = self.sql

        with DatabaseTrace(sql):
            return self.func(*args, **kwargs)
But it fails!!!


✴Not all Python DBAPI modules are Python
But it fails!!!


✴Not all Python DBAPI modules are Python
✴Can’t wrap methods of Python C objects
Cursor wrapper (1)


def instrument(module):
    module.connect = ConnectionFactory(
            module.connect)
Cursor wrapper (2)

class ConnectionWrapper(object):
    def __init__(self, connection):
        self.__connection = connection
   def __getattr__(self, name):
         return getattr(self.__connection, name)
    def cursor(self, *args, **kwargs):
        return CursorWrapper(
                self.__connection.cursor(
                *args, **kwargs))

class ConnectionFactory(object):
    def __init__(self, connect):
         self.__connect = connect
    def __call__(self, *args, **kwargs):
         return ConnectionWrapper(
                 self.__connect(*args, **kwargs))
Cursor wrapper (3)

def _sql(sql, params=()):
    return sql

class CursorWrapper(object):
    def __init__(self, cursor):
        self.__cursor = cursor

   def execute(self, *args, **kwargs):
       return DatabaseTraceWrapper(
           self.__cursor.execute,
           _sql)(*args, **kwargs)

   def __getattr__(self, name):
       return getattr(self.__cursor, name)
Enough already

✴Too much information to cover.
✴Is never enough time for it all.
Enough already

✴Too much information to cover.
✴Is never enough time for it all.
✴So so come and talk to us later.
Enough already

✴Too much information to cover.
✴Is never enough time for it all.
✴So so come and talk to us later.
✴Get a demonstration of New Relic.
Enough already

✴Too much information to cover.
✴Is never enough time for it all.
✴So so come and talk to us later.
✴Get a demonstration of New Relic.
✴Drinks and nibbles Wednesday evening.
One final thing
Questions?

http://newrelic.com/djangocon
 python-beta@newrelic.com


    Graham Dumpleton
   graham@newrelic.com

Graham.Dumpleton@gmail.com
    @GrahamDumpleton

More Related Content

What's hot

Con-FESS 2015 - Having Fun With Javassist
Con-FESS 2015 - Having Fun With JavassistCon-FESS 2015 - Having Fun With Javassist
Con-FESS 2015 - Having Fun With JavassistAnton Arhipov
 
Advanced Debugging Using Java Bytecodes
Advanced Debugging Using Java BytecodesAdvanced Debugging Using Java Bytecodes
Advanced Debugging Using Java BytecodesGanesh Samarthyam
 
Riga Dev Day 2016 - Having fun with Javassist
Riga Dev Day 2016 - Having fun with JavassistRiga Dev Day 2016 - Having fun with Javassist
Riga Dev Day 2016 - Having fun with JavassistAnton Arhipov
 
Py.test
Py.testPy.test
Py.testsoasme
 
Gevent what's the point
Gevent what's the pointGevent what's the point
Gevent what's the pointseanmcq
 
Concurrent Programming in Java
Concurrent Programming in JavaConcurrent Programming in Java
Concurrent Programming in JavaRuben Inoto Soto
 
Test Driven Development With Python
Test Driven Development With PythonTest Driven Development With Python
Test Driven Development With PythonSiddhi
 
Europython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryEuropython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryMauro Rocco
 
Async programming and python
Async programming and pythonAsync programming and python
Async programming and pythonChetan Giridhar
 
Concurrency Utilities in Java 8
Concurrency Utilities in Java 8Concurrency Utilities in Java 8
Concurrency Utilities in Java 8Martin Toshev
 
Spock: Test Well and Prosper
Spock: Test Well and ProsperSpock: Test Well and Prosper
Spock: Test Well and ProsperKen Kousen
 
The Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFuturesThe Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFuturesHaim Yadid
 
Threads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSThreads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSTechWell
 
Google Guava & EMF @ GTUG Nantes
Google Guava & EMF @ GTUG NantesGoogle Guava & EMF @ GTUG Nantes
Google Guava & EMF @ GTUG Nantesmikaelbarbero
 
Spock Testing Framework - The Next Generation
Spock Testing Framework - The Next GenerationSpock Testing Framework - The Next Generation
Spock Testing Framework - The Next GenerationBTI360
 

What's hot (20)

Sailing with Java 8 Streams
Sailing with Java 8 StreamsSailing with Java 8 Streams
Sailing with Java 8 Streams
 
Con-FESS 2015 - Having Fun With Javassist
Con-FESS 2015 - Having Fun With JavassistCon-FESS 2015 - Having Fun With Javassist
Con-FESS 2015 - Having Fun With Javassist
 
Advanced Debugging Using Java Bytecodes
Advanced Debugging Using Java BytecodesAdvanced Debugging Using Java Bytecodes
Advanced Debugging Using Java Bytecodes
 
Riga Dev Day 2016 - Having fun with Javassist
Riga Dev Day 2016 - Having fun with JavassistRiga Dev Day 2016 - Having fun with Javassist
Riga Dev Day 2016 - Having fun with Javassist
 
Dynamic Python
Dynamic PythonDynamic Python
Dynamic Python
 
Java Concurrency by Example
Java Concurrency by ExampleJava Concurrency by Example
Java Concurrency by Example
 
Py.test
Py.testPy.test
Py.test
 
Gevent what's the point
Gevent what's the pointGevent what's the point
Gevent what's the point
 
Introduce Django
Introduce DjangoIntroduce Django
Introduce Django
 
Concurrent Programming in Java
Concurrent Programming in JavaConcurrent Programming in Java
Concurrent Programming in Java
 
Unit testing
Unit testingUnit testing
Unit testing
 
Test Driven Development With Python
Test Driven Development With PythonTest Driven Development With Python
Test Driven Development With Python
 
Europython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & CeleryEuropython 2011 - Playing tasks with Django & Celery
Europython 2011 - Playing tasks with Django & Celery
 
Async programming and python
Async programming and pythonAsync programming and python
Async programming and python
 
Concurrency Utilities in Java 8
Concurrency Utilities in Java 8Concurrency Utilities in Java 8
Concurrency Utilities in Java 8
 
Spock: Test Well and Prosper
Spock: Test Well and ProsperSpock: Test Well and Prosper
Spock: Test Well and Prosper
 
The Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFuturesThe Future of Futures - A Talk About Java 8 CompletableFutures
The Future of Futures - A Talk About Java 8 CompletableFutures
 
Threads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOSThreads, Queues, and More: Async Programming in iOS
Threads, Queues, and More: Async Programming in iOS
 
Google Guava & EMF @ GTUG Nantes
Google Guava & EMF @ GTUG NantesGoogle Guava & EMF @ GTUG Nantes
Google Guava & EMF @ GTUG Nantes
 
Spock Testing Framework - The Next Generation
Spock Testing Framework - The Next GenerationSpock Testing Framework - The Next Generation
Spock Testing Framework - The Next Generation
 

Similar to DjangoCon US 2011 - Monkeying around at New Relic

Testing My Patience
Testing My PatienceTesting My Patience
Testing My PatienceAdam Lowry
 
Python magicmethods
Python magicmethodsPython magicmethods
Python magicmethodsdreampuf
 
Funkcija, objekt, python
Funkcija, objekt, pythonFunkcija, objekt, python
Funkcija, objekt, pythonRobert Lujo
 
Is java8 a true functional programming language
Is java8 a true functional programming languageIs java8 a true functional programming language
Is java8 a true functional programming languageSQLI
 
Is java8a truefunctionallanguage
Is java8a truefunctionallanguageIs java8a truefunctionallanguage
Is java8a truefunctionallanguageSamir Chekkal
 
Ten useful JavaScript tips & best practices
Ten useful JavaScript tips & best practicesTen useful JavaScript tips & best practices
Ten useful JavaScript tips & best practicesAnkit Rastogi
 
JavaScript Growing Up
JavaScript Growing UpJavaScript Growing Up
JavaScript Growing UpDavid Padbury
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to CeleryIdan Gazit
 
Python decorators (中文)
Python decorators (中文)Python decorators (中文)
Python decorators (中文)Yiwei Chen
 
Functional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with PythonFunctional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with PythonCarlos V.
 
Pyimproved again
Pyimproved againPyimproved again
Pyimproved againrik0
 
(map Clojure everyday-tasks)
(map Clojure everyday-tasks)(map Clojure everyday-tasks)
(map Clojure everyday-tasks)Jacek Laskowski
 
Jggug 2010 330 Grails 1.3 観察
Jggug 2010 330 Grails 1.3 観察Jggug 2010 330 Grails 1.3 観察
Jggug 2010 330 Grails 1.3 観察Tsuyoshi Yamamoto
 

Similar to DjangoCon US 2011 - Monkeying around at New Relic (20)

Practical Celery
Practical CeleryPractical Celery
Practical Celery
 
Testing My Patience
Testing My PatienceTesting My Patience
Testing My Patience
 
Python magicmethods
Python magicmethodsPython magicmethods
Python magicmethods
 
Django Celery
Django Celery Django Celery
Django Celery
 
Funkcija, objekt, python
Funkcija, objekt, pythonFunkcija, objekt, python
Funkcija, objekt, python
 
DevOps with Fabric
DevOps with FabricDevOps with Fabric
DevOps with Fabric
 
How to fake_properly
How to fake_properlyHow to fake_properly
How to fake_properly
 
Is java8 a true functional programming language
Is java8 a true functional programming languageIs java8 a true functional programming language
Is java8 a true functional programming language
 
Is java8a truefunctionallanguage
Is java8a truefunctionallanguageIs java8a truefunctionallanguage
Is java8a truefunctionallanguage
 
Ten useful JavaScript tips & best practices
Ten useful JavaScript tips & best practicesTen useful JavaScript tips & best practices
Ten useful JavaScript tips & best practices
 
PythonOOP
PythonOOPPythonOOP
PythonOOP
 
JavaScript Growing Up
JavaScript Growing UpJavaScript Growing Up
JavaScript Growing Up
 
An Introduction to Celery
An Introduction to CeleryAn Introduction to Celery
An Introduction to Celery
 
Django Good Practices
Django Good PracticesDjango Good Practices
Django Good Practices
 
Python decorators (中文)
Python decorators (中文)Python decorators (中文)
Python decorators (中文)
 
Functional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with PythonFunctional Programming inside OOP? It’s possible with Python
Functional Programming inside OOP? It’s possible with Python
 
обзор Python
обзор Pythonобзор Python
обзор Python
 
Pyimproved again
Pyimproved againPyimproved again
Pyimproved again
 
(map Clojure everyday-tasks)
(map Clojure everyday-tasks)(map Clojure everyday-tasks)
(map Clojure everyday-tasks)
 
Jggug 2010 330 Grails 1.3 観察
Jggug 2010 330 Grails 1.3 観察Jggug 2010 330 Grails 1.3 観察
Jggug 2010 330 Grails 1.3 観察
 

More from Graham Dumpleton

Implementing a decorator for thread synchronisation.
Implementing a decorator for thread synchronisation.Implementing a decorator for thread synchronisation.
Implementing a decorator for thread synchronisation.Graham Dumpleton
 
Data analytics in the cloud with Jupyter notebooks.
Data analytics in the cloud with Jupyter notebooks.Data analytics in the cloud with Jupyter notebooks.
Data analytics in the cloud with Jupyter notebooks.Graham Dumpleton
 
“warpdrive”, making Python web application deployment magically easy.
“warpdrive”, making Python web application deployment magically easy.“warpdrive”, making Python web application deployment magically easy.
“warpdrive”, making Python web application deployment magically easy.Graham Dumpleton
 
OpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaSOpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaSGraham Dumpleton
 
Automated Image Builds in OpenShift and Kubernetes
Automated Image Builds in OpenShift and KubernetesAutomated Image Builds in OpenShift and Kubernetes
Automated Image Builds in OpenShift and KubernetesGraham Dumpleton
 
PyCon HK 2015 - Monitoring the performance of python web applications
PyCon HK 2015 -  Monitoring the performance of python web applicationsPyCon HK 2015 -  Monitoring the performance of python web applications
PyCon HK 2015 - Monitoring the performance of python web applicationsGraham Dumpleton
 
PyCon AU 2015 - Using benchmarks to understand how wsgi servers work
PyCon AU 2015  - Using benchmarks to understand how wsgi servers workPyCon AU 2015  - Using benchmarks to understand how wsgi servers work
PyCon AU 2015 - Using benchmarks to understand how wsgi servers workGraham Dumpleton
 
PyCon NZ 2013 - Advanced Methods For Creating Decorators
PyCon NZ 2013 - Advanced Methods For Creating DecoratorsPyCon NZ 2013 - Advanced Methods For Creating Decorators
PyCon NZ 2013 - Advanced Methods For Creating DecoratorsGraham Dumpleton
 
PyCon US 2013 Making Apache suck less for hosting Python web applications
PyCon US 2013 Making Apache suck less for hosting Python web applicationsPyCon US 2013 Making Apache suck less for hosting Python web applications
PyCon US 2013 Making Apache suck less for hosting Python web applicationsGraham Dumpleton
 
PyCon AU 2010 - Getting Started With Apache/mod_wsgi.
PyCon AU 2010 - Getting Started With Apache/mod_wsgi.PyCon AU 2010 - Getting Started With Apache/mod_wsgi.
PyCon AU 2010 - Getting Started With Apache/mod_wsgi.Graham Dumpleton
 
PyCon US 2012 - State of WSGI 2
PyCon US 2012 - State of WSGI 2PyCon US 2012 - State of WSGI 2
PyCon US 2012 - State of WSGI 2Graham Dumpleton
 
PyCon US 2012 - Web Server Bottlenecks and Performance Tuning
PyCon US 2012 - Web Server Bottlenecks and Performance TuningPyCon US 2012 - Web Server Bottlenecks and Performance Tuning
PyCon US 2012 - Web Server Bottlenecks and Performance TuningGraham Dumpleton
 

More from Graham Dumpleton (13)

Implementing a decorator for thread synchronisation.
Implementing a decorator for thread synchronisation.Implementing a decorator for thread synchronisation.
Implementing a decorator for thread synchronisation.
 
Not Tom Eastman
Not Tom EastmanNot Tom Eastman
Not Tom Eastman
 
Data analytics in the cloud with Jupyter notebooks.
Data analytics in the cloud with Jupyter notebooks.Data analytics in the cloud with Jupyter notebooks.
Data analytics in the cloud with Jupyter notebooks.
 
“warpdrive”, making Python web application deployment magically easy.
“warpdrive”, making Python web application deployment magically easy.“warpdrive”, making Python web application deployment magically easy.
“warpdrive”, making Python web application deployment magically easy.
 
OpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaSOpenShift, Docker, Kubernetes: The next generation of PaaS
OpenShift, Docker, Kubernetes: The next generation of PaaS
 
Automated Image Builds in OpenShift and Kubernetes
Automated Image Builds in OpenShift and KubernetesAutomated Image Builds in OpenShift and Kubernetes
Automated Image Builds in OpenShift and Kubernetes
 
PyCon HK 2015 - Monitoring the performance of python web applications
PyCon HK 2015 -  Monitoring the performance of python web applicationsPyCon HK 2015 -  Monitoring the performance of python web applications
PyCon HK 2015 - Monitoring the performance of python web applications
 
PyCon AU 2015 - Using benchmarks to understand how wsgi servers work
PyCon AU 2015  - Using benchmarks to understand how wsgi servers workPyCon AU 2015  - Using benchmarks to understand how wsgi servers work
PyCon AU 2015 - Using benchmarks to understand how wsgi servers work
 
PyCon NZ 2013 - Advanced Methods For Creating Decorators
PyCon NZ 2013 - Advanced Methods For Creating DecoratorsPyCon NZ 2013 - Advanced Methods For Creating Decorators
PyCon NZ 2013 - Advanced Methods For Creating Decorators
 
PyCon US 2013 Making Apache suck less for hosting Python web applications
PyCon US 2013 Making Apache suck less for hosting Python web applicationsPyCon US 2013 Making Apache suck less for hosting Python web applications
PyCon US 2013 Making Apache suck less for hosting Python web applications
 
PyCon AU 2010 - Getting Started With Apache/mod_wsgi.
PyCon AU 2010 - Getting Started With Apache/mod_wsgi.PyCon AU 2010 - Getting Started With Apache/mod_wsgi.
PyCon AU 2010 - Getting Started With Apache/mod_wsgi.
 
PyCon US 2012 - State of WSGI 2
PyCon US 2012 - State of WSGI 2PyCon US 2012 - State of WSGI 2
PyCon US 2012 - State of WSGI 2
 
PyCon US 2012 - Web Server Bottlenecks and Performance Tuning
PyCon US 2012 - Web Server Bottlenecks and Performance TuningPyCon US 2012 - Web Server Bottlenecks and Performance Tuning
PyCon US 2012 - Web Server Bottlenecks and Performance Tuning
 

Recently uploaded

Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxEasyPrinterHelp
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeCzechDreamin
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoTAnalytics
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...CzechDreamin
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaCzechDreamin
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka DoktorováCzechDreamin
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxJennifer Lim
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?Mark Billinghurst
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastUXDXConf
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024Stephanie Beckett
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Julian Hyde
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekCzechDreamin
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyJohn Staveley
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfFIDO Alliance
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty SecureFemke de Vroome
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCzechDreamin
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsStefano
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...FIDO Alliance
 

Recently uploaded (20)

Buy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptxBuy Epson EcoTank L3210 Colour Printer Online.pptx
Buy Epson EcoTank L3210 Colour Printer Online.pptx
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024IoT Analytics Company Presentation May 2024
IoT Analytics Company Presentation May 2024
 
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
 
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at ComcastDesigning for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxUnpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
 
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)
 
AI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří KarpíšekAI revolution and Salesforce, Jiří Karpíšek
AI revolution and Salesforce, Jiří Karpíšek
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCustom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder
 
PLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. StartupsPLAI - Acceleration Program for Generative A.I. Startups
PLAI - Acceleration Program for Generative A.I. Startups
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 

DjangoCon US 2011 - Monkeying around at New Relic

  • 1. Monkeying Around At New Relic Graham Dumpleton DjangoCon September 2011
  • 4. Web interface (1) RESPONSE TIME
  • 5. Web interface (1) BREAKDOWN
  • 7. Web interface (1) SLOW TRANSACTIONS
  • 11. Web transactions ✴When they occurred. ✴How long they ran for. ✴What handled the request. ✴What work was done. ✴Were there any errors.
  • 12. What work was done? ✴Functions called. ✴Databases operations. ✴External service requests. ✴Memcache operations.
  • 13. Trace class (1) import time class Trace(object): def __init__(self): self.start = 0 def enter(self): self.start = time.time() def exit(self): self.duration = time.time() - self.start print self.duration
  • 14. Timing code block def function(): time.sleep(0.1) trace = Trace() trace.enter() for i in range(10): function() trace.exit()
  • 15. Stop on exceptions def function(): time.sleep(0.1) raise MissingThisTalk('@pydanny') trace = Trace() trace.enter() try: for i in range(10): function() finally: trace.exit()
  • 16. Catch exception (1) def function(): time.sleep(0.1) raise NotVeryHappy('@pydanny') trace = Trace() trace.enter() try: for i in range(10): function() except: print sys.exc_info() raise finally: trace.exit()
  • 17. Trace class (2) import time class Trace(object): def __init__(self): self.start = 0 def enter(self): self.start = time.time() def exit(self, exc=None, value=None, tb=None): self.duration = time.time() - self.start print exc, value, tb print self.duration
  • 18. Catch exception (2) trace = Trace() success = True trace.enter() try: function() except: success = False trace.exit(*sys.exc_info()) raise finally: if success: trace.exit(None, None, None)
  • 19. Hang on! ✴Doesn’t this pattern look familiar.
  • 20. Hang on! ✴Doesn’t this pattern look familiar. ✴Isn’t this just a context manager?
  • 21. Context managers trace = Trace() with trace: function()
  • 22. Trace class (3) import time class Trace(object): def __init__(self): self.start = 0 def __enter__(self): self.start = time.time() def __exit__(self, exc=None, value=None, tb=None): self.duration = time.time() - self.start print exc, value, tb print self.duration
  • 23. Catch exception (3) trace = Trace() success = True trace.__enter__() try: function() except: success = False if not trace.__exit__(*sys.exc_info()): raise finally: if success: trace.__exit__(None, None, None)
  • 24. Decorating functions (1) def decorator(f): def wrapper(*args, **kwargs): with Trace(): return f(*args, **kwargs) return wrapper @decorator def function(): time.sleep(1.0) function()
  • 25. Decorator issues (1) def decorator(f): def wrapper(*args, **kwargs): with Trace(): return f(*args, **kwargs) return wrapper @decorator @csrf_exempt def handler(): ...
  • 26. Decorator issues (1) def decorator(f): def wrapper(*args, **kwargs): with Trace(): return f(*args, **kwargs) return wrapper @decorator @csrf_exempt def handler(): ... Why is my view expecting CSRF token?
  • 27. Decorating functions (2) import functools def decorator(f): @functools.wraps(f) def wrapper(*args, **kwargs): with Trace(): return f(*args, **kwargs) return wrapper @decorator def function(): time.sleep(1.0) function()
  • 28. Decorator issues (2) class Object(object): @decorator @staticmethod def function(): time.sleep(1.0) o = Object() o.function()
  • 29. Decorator issues (2) class Object(object): @decorator @staticmethod def function(): time.sleep(1.0) o = Object() o.function() AttributeError: 'staticmethod' object has no attribute '__module__'
  • 30. Descriptor issues def available_attrs(f): # http://bugs.python.org/issue3445. return tuple(a for a in functools.WRAPPER_ASSIGNMENTS if hasattr(f, a)) def decorator(f): @functools.wraps(f, available_attrs(f)) def wrapper(*args, **kwargs): with Trace(): return f(*args, **kwargs) return wrapper
  • 31. Descriptor issues def available_attrs(f): # http://bugs.python.org/issue3445. return tuple(a for a in functools.WRAPPER_ASSIGNMENTS if hasattr(f, a)) def decorator(f): @functools.wraps(f, available_attrs(f)) def wrapper(*args, **kwargs): with Trace(): return f(*args, **kwargs) return wrapper TypeError: 'staticmethod' object is not callable
  • 32. Decorating functions (2) class Wrapper(object): def __init__(self, func): functools.update_wrapper(self, func, available_attrs(func)) self.func = func def __get__(self, instance, klass): if instance is None: return self desc = self.func.__get__(instance, klass) return self.__class__(desc) def __call__(self, *args, **kwargs): with Trace(): return self.func(*args, **kwargs) def decorator(func): return Wrapper(func)
  • 33. WSGI application (1) @decorator def application(environ, start_response): status = '200 OK' start_response(status, [('Content-type', 'text/plain'),]) time.sleep(1.0) return ['Is @pydanny here yet?']
  • 34. WSGI application (2) @decorator def application(environ, start_response): status = '200 OK' start_response(status, [('Content-type', 'text/plain'),]) time.sleep(1.0) yield 'Maybe @audreyr is watching.'
  • 35. WSGI middleware (1) class Wrapper(object): ... def __call__(self, environ, start_response): trace = Trace() trace.__enter__() try: result = self.f(environ, start_response) except: trace.__exit__(*sys.exc_info()) raise return Iterable(trace, result)
  • 36. WSGI middleware (2) class Iterable(object): def __init__(self, trace, generator): self.trace = trace self.generator = generator def __iter__(self): for item in self.generator: yield item ...
  • 37. WSGI middleware (3) class Iterable(object): ... def close(self): try: if hasattr(self.generator, 'close'): self.generator.close() except: self.trace.__exit__(*sys.exc_info()) raise else: self.trace.__exit__(None, None, None)
  • 38. Agent initialisation (1) import newrelic.agent newrelic.agent.initialize('/etc/newrelic.ini') @newrelic.agent.wsgi_application() def application(environ, start_response): ...
  • 39. Agent initialisation (2) import newrelic.agent newrelic.agent.initialize('/etc/newrelic.ini') import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()
  • 40. PEP 369 / Post import hooks @imp.when_imported('django.core.handlers.wsgi') def instrument(module): method = module.WSGIHandler.__call__ module.WSGIHandler.__call__ = WSGIApplicationWrapper(method)
  • 41. DIY post import hooks (1) class ImportHookFinder: def __init__(self): self._skip = {} def find_module(self, fullname, path=None): if not fullname in _import_hooks: return None if fullname in self._skip: return None self._skip[fullname] = True try: __import__(fullname) finally: del self._skip[fullname] return ImportHookLoader()
  • 42. DIY post import hooks (2) class ImportHookLoader: def load_module(self, fullname): module = sys.modules[fullname] _notify_import_hooks(fullname, module) return module sys.meta_path.insert(0, ImportHookFinder())
  • 43. Instrumentation from newrelic.api.web_transaction import wrap_wsgi_application def instrument(module): wrap_wsgi_application(module, 'WSGIHandler.__call__')
  • 44. WSGI applications (1) ✴Context manager ✴Trace -> WebTransaction
  • 45. WSGI applications (1) ✴Context manager ✴Trace -> WebTransaction ✴Wrapper/Descriptor ✴Wrapper -> WSGIApplicationWrapper
  • 46. WSGI applications (2) ✴Decorator ✴decorator -> wsgi_application()
  • 47. WSGI applications (2) ✴Decorator ✴decorator -> wsgi_application() ✴Monkey patching ✴wrap_wsgi_application()
  • 48. Function trace (1) ✴Context manager ✴Trace -> FunctionTrace
  • 49. Function trace (1) ✴Context manager ✴Trace -> FunctionTrace ✴Wrapper/Descriptor ✴Wrapper -> FunctionTraceWrapper
  • 50. Function Trace (2) ✴Decorator ✴decorator -> function_trace()
  • 51. Function Trace (2) ✴Decorator ✴decorator -> function_trace() ✴Monkey patching ✴wrap_function_trace()
  • 53. Database Trace (1) from newrelic.api.database_trace import wrap_database_trace def instrument(module): def _sql(cursor, sql, params=()): return sql wrap_database_trace(module, 'Cursor.execute', sql=_sql)
  • 54. Database Trace (2) class DatabaseTraceWrapper(object): def __init__(self, func, sql): functools.update_wrapper(self, func, available_attrs(func)) if type(func) == tuples.TupleType: self.instance, func = func else: self.instance = None self.func = func self.sql = sql
  • 55. Database Trace (3) class DatabaseTraceWrapper(object): def __get__(self, instance, klass): if instance is None: return self desc = self.func.__get__(instance, klass) return self.__class__((instance, desc))
  • 56. Database Trace (4) class DatabaseTraceWrapper(object): def __call__(self, *args, **kwargs): if not isinstance(self.sql, basestring): if (self.instance and inspect.ismethod(self.func)): sql = self.sql(self.instance, *args, **kwargs) else: sql = self.sql(*args, **kargs) else: sql = self.sql with DatabaseTrace(sql): return self.func(*args, **kwargs)
  • 57. But it fails!!! ✴Not all Python DBAPI modules are Python
  • 58. But it fails!!! ✴Not all Python DBAPI modules are Python ✴Can’t wrap methods of Python C objects
  • 59. Cursor wrapper (1) def instrument(module): module.connect = ConnectionFactory( module.connect)
  • 60. Cursor wrapper (2) class ConnectionWrapper(object): def __init__(self, connection): self.__connection = connection def __getattr__(self, name): return getattr(self.__connection, name) def cursor(self, *args, **kwargs): return CursorWrapper( self.__connection.cursor( *args, **kwargs)) class ConnectionFactory(object): def __init__(self, connect): self.__connect = connect def __call__(self, *args, **kwargs): return ConnectionWrapper( self.__connect(*args, **kwargs))
  • 61. Cursor wrapper (3) def _sql(sql, params=()): return sql class CursorWrapper(object): def __init__(self, cursor): self.__cursor = cursor def execute(self, *args, **kwargs): return DatabaseTraceWrapper( self.__cursor.execute, _sql)(*args, **kwargs) def __getattr__(self, name): return getattr(self.__cursor, name)
  • 62. Enough already ✴Too much information to cover. ✴Is never enough time for it all.
  • 63. Enough already ✴Too much information to cover. ✴Is never enough time for it all. ✴So so come and talk to us later.
  • 64. Enough already ✴Too much information to cover. ✴Is never enough time for it all. ✴So so come and talk to us later. ✴Get a demonstration of New Relic.
  • 65. Enough already ✴Too much information to cover. ✴Is never enough time for it all. ✴So so come and talk to us later. ✴Get a demonstration of New Relic. ✴Drinks and nibbles Wednesday evening.
  • 67. Questions? http://newrelic.com/djangocon python-beta@newrelic.com Graham Dumpleton graham@newrelic.com Graham.Dumpleton@gmail.com @GrahamDumpleton

Editor's Notes

  1. As most would know, I am the author of mod_wsgi. Some time back I started adding instrumentation to mod_wsgi to track time spent in handling requests and number of concurrent requests, so could use it to tune Apache/mod_wsgi. I gave up though, as no good way to display it. Was thus very excited when introduced to New Relic as allowed me to do what I had wanted to do. Decided I had just had to work for them and now I am.\n
  2. The key missing ingredient here I was needing was the web interface that New Relic provides for their existing agents. It can show you what is going on in your web application. It actually has the ability to show much more data than I was intending to capture. More importantly, they position there product as being for your production web application, not only for development or staging. So what are some of the things the New Relic web interface can display?\n
  3. We have throughput.\n
  4. We have response time as recorded by the application.\n
  5. We can see a breakdown of how much time is spent doing different things as part of the request. Queueing time. Database. Memcache. Externals. As well as a rollup of time spent in the rest of the application.\n
  6. We get a record of what exceptions occurred within the application.\n
  7. And we get a sampling of requests which were regarded as being slow.\n
  8. There is the ability to drill down into web transactions and what parts of the application they were handled by.\n
  9. And for the slow web transactions, you can drill down again and get a targeted breakdown of where time was spent for that specific request.\n
  10. You also have a breakdown of your database queries. Which occur the most, which are the slowest, the type of query and so on. This is only a part of the functionality available through the web interface though. There is actually a great deal more than this, but this talk is meant to be about the techniques we used to capture this information in Python web applications so we will move on to that.\n
  11. Summarising what I pointed out in the web interface, the main things we are therefore interested in tracking for web transactions are (ABOVE).\n
  12. Drilling down within a web transaction we are interested in (ABOVE). I will not be able to cover all the work we had to do to capture all this information, but will try and cover some of the more interesting aspects.\n
  13. The key thing we are interested in and which everything else rests on is how long things take. For that we can create a simple trace class which is triggered on entry to a block of code and thence on exit. The duration along with details about what was being traced would then be collected up and stored for later processing.\n
  14. The code block where we wish to record time taken is thence just wrapped with the calls to enter and exit. Problem with this simplistic approach though is if an exception occurs within the code block. As the code is written, if an exception does occur the exit call will be skipped. We therefore have to scope the code block with try/finally.\n
  15. In some cases though we are interested not only in how long something took, we are also interested in what exceptions may have occurred. This is important for Django because Django swallows exceptions and converts them to a 500 error response. They will therefore not propagate up to the top level and we will want to remember them.\n
  16. What we will want to do therefore is add an except clause and also capture the exception details. Having this information within the exception clause where we are wrapping the code block isn't terribly useful though. We really want to pass that off to our trace object instead.\n
  17. What we can do here is overload the use of the exit function and have it accept any exception details with None being used for case where no exception occurred.\n
  18. Our wrapper code is then modified to call exit in both the except and finally clauses. Because finally will be called for successful case and when exception occurs, we do need to protect ourselves to ensure that the exit method isn't called twice. \n
  19. But wait, doesn't this all look sort of familiar. (NEXT) Haven't we just reinvented the wheel and ended up with the pattern which underlies how context managers work.\n
  20. But wait, doesn't this all look sort of familiar. (NEXT) Haven't we just reinvented the wheel and ended up with the pattern which underlies how context managers work.\n
  21. For those not familiar with context managers it is the name used to refer to objects used in conjunction with the Python 'with' statement. For this language construct, the interpreter itself will ensure that enter and exit methods are called for us automatically, including the variation for when the exception occurs.\n
  22. We will need to just make a slight change to our trace object, and that is to rename the enter and exit methods to match the dunderscore names which a context manager must use.\n
  23. One other thing to note about the 'with' statement and context managers is that it does provide for an additional feature that we aren't actually making use of. That is that the exit function of a context manager can choose to swallow the exception by returning a true value. The implementation of 'with' is therefore more like that shown here. (ABOVE).\n
  24. Although you could trace specific blocks of code, usually though we are more interested in tracing time spent in a function as a whole. For this we can introduce a decorator and apply that to the function to be traced. Unfortunately decorators aren't necessarily that straight forward and there are various gotchas you have to be aware of.\n
  25. The first one is stacking of decorators. Take for example the situation of the 'csrf_exempt' decorator in Django. If we wrap our decorator around it, then it doesn't work out how we want it. (NEXT) The problem in this case is that the 'csrf_exempt' decorator ends up ineffectual and no longer does what it is meant to.\n
  26. The reason for this is that is that 'csrf_exempt' decorator adds an attribute to the wrapped function which Django then checks for to determine if CSRF checks are required or not. By wrapping the 'csrf_exempt' decorator we have hidden that attribute. We need therefore to use 'functools.wraps()' in the implementation of the decorator, which copies the attributes up to the level of our decorator.\n
  27. Now, because our decorator is to be used to track time taken, we want it to work in any situation. And for class methods it does work, but we end up with further problems when we start stacking decorators commonly used with methods, in this case with the 'staticmethod' decorator. (NEXT) Here the problem is 'functools.wraps()' which tries to access the '__module__' attribute which doesn't exist for 'staticmethod' decorator objects.\n
  28. This one is actually a bug or shortcoming in Python which is fixed in Python 3.1 or 3.2, with 'wraps' not actually checking whether the attributes it is trying to copy actually exists. So, we need to workaround that and only copy certain attributes if they exist. (NEXT) Fix that and we still have a problem though in that a 'staticmethod' decorator object doesn't appear to be callable.\n
  29. The issue here is that the presence of a '__call__' method isn't the sole arbiter that an object is callable. The use of nested functions to implement a decorator doesn't take that into consideration. We instead need to deal with the case where the object being wrapped may implement a method descriptor via '__get__()' to return the callable object. To do that we need to use a class object for the wrapper which daisy chains '__get__()' from the wrapped object.\n
  30. So we think we now have a way of wrapping functions that will work, what about when we now apply that to a WSGI application. What is going to happen? For the simple case where a list is returned as iterable, it appears to work, but in reality it doesn't count time taken to write the response back to the client, which we don't want to leave out.\n
  31. The case of where 'yield' is being used is even worse. The timing will stop immediately as the function will return a generator object before any code of the function is actually executed. That is, code in the function only gets executed when there is an attempt to retrieve the first item from the generator. So, we came up with this way of wrapping functions only to find it doesn't work for the most important case we have of a WSGI application itself. What do we do?\n
  32. The only choice in this situation is to create a WSGI middleware which is aware of the special way that an iterable/generator can be returned by the WSGI application. We cannot use the 'with' statement. Our wrapper instead needs to explicitly call the enter method before invoking the application. If an error occurs, then we will call exit. Otherwise we return the result from the application in a special iterable object.\n
  33. This iterable object overrides '__iter__' and proxies that through to the iterable/generator that was originally returned from the wrapped WSGI application.\n
  34. The important bit though is that our iterable wrapper provides a 'close()' method. The 'close()' method is special and is a method which when present on the object returned is required to be called by a WSGI adapter or WSGI middleware when the complete response is consumed or the request otherwise ends. In our case we call any 'close()' method on wrapped object and call the exit method on our trace object as appropriate signalling the end of timing.\n
  35. With this problem solved, how then do things look for actually initialising and wrapping a WSGI application when using our actual Python agent? For that we have our special 'wsgi_application()' decorator which would be applied to the WSGI application in the WSGI script file or module. The actual initialisation is done via an explicit initialisation call and passing in the agent configuration file. This in place we will get basic web transaction metrics.\n
  36. What about Django? For Django we still have the initialisation call, but for the case of the WSGI application entry point, although we could have wrapped it, it actually isn't necessary to do so. Only the initilisation is required. So, although I have gone and explained the mechanics of wrapping stuff, we don't want you to have to do that. We want things to be automatic so that the only thing that needs to be done is the initialisation. But how can we do that?\n
  37. Python has this great PEP, PEP 369 for post import hooks which can help us out here. It defines a mechanism whereby one can trigger code to be executed when a module is imported, allowing us to go in and monkey patch the module before the code doing the import gets it. That way, we can transparently make this change and others and you don't need to. One small problem. PEP 369 has never been adopted into the Python core. You have to implement your own.\n
  38. To implement our own we need to hook into the Python import loader. To do that we need to implement two parts. First is the finder. When these are registered they are called to try and resolve the target of a module import. What we do in ours is to call back into '__import__' again being mindful to protect ourselves second time through to avoid a loop. If that works we return with the second part, which is our loader.\n
  39. The loader simply grabs the module out of 'sys.modules' from the prior '__import__' we did in our finder. It then calls any of the import hooks we have registered against that module to trigger any instrumentation.\n
  40. Our instrumentation function for 'django.core.handlers.wsgi' thus being as shown (ABOVE). You will note though that we have a new helper function called 'wrap_wsgi_application()' which eliminates the need for us to explicitly call '__getattr__' and '__setattr__'. It is just given the module and the object path of what we want to wrap with a WSGI application wrapper and does it thus simplifying it further.\n
  41. So although the intent is that you should never need to use it, we still have used a layer of objects ourselves. This starts with the context manager which can be used with the 'with' statement directly if need be. (NEXT) A wrapper object which in this case has special knowledge of how a WSGI application works.\n
  42. So although the intent is that you should never need to use it, we still have used a layer of objects ourselves. This starts with the context manager which can be used with the 'with' statement directly if need be. (NEXT) A wrapper object which in this case has special knowledge of how a WSGI application works.\n
  43. Plus also have our decorator function (NEXT) and a high level wrapping function which can bring that altogether. As is obvious, you could use all these directly yourself and manually modify your code to instrument it, but we don't want you having to do that. Thus we use our import hook mechanism to trigger instrumentation we supply for Django and it goes in and does everything. All you need is to add the initialisation in an appropriate place.\n
  44. Plus also have our decorator function (NEXT) and a high level wrapping function which can bring that altogether. As is obvious, you could use all these directly yourself and manually modify your code to instrument it, but we don't want you having to do that. Thus we use our import hook mechanism to trigger instrumentation we supply for Django and it goes in and does everything. All you need is to add the initialisation in an appropriate place.\n
  45. Similarly to when wrapping the WSGI application itself to time the request as a whole. We also have these layers of objects for function tracing. The context manager (NEXT) and the wrapper object.\n
  46. Similarly to when wrapping the WSGI application itself to time the request as a whole. We also have these layers of objects for function tracing. The context manager (NEXT) and the wrapper object.\n
  47. Plus the decorator (NEXT) and high level wrap function.\n
  48. Plus the decorator (NEXT) and high level wrap function.\n
  49. Same again for tracing functions when recording database queries, external service requests and memcache operations. As with wrapping WSGI applications, we have lots of built in instrumentation. For Django we will wrap function traces around middleware, template rendering and other operations of interest. We also wrap DBAPI compliant database client modules. Everything is automatic and you don't need to change the code yourself.\n
  50. The next problem is that in most trace object types we need to capture additional detail. For database queries we want to capture what the SQL was. Our high level wrap function for this can accept the SQL as a string, which would be static, or be supplied a function which can use to derive it from the arguments to the called function.\n
  51. The trick here is our wrapper objects are implementing this dual role of being a wrapper, but also a method descriptor in the case of a bound method. We thus have to modify our wrapper to take either an actual function, or a instance and a bound method.\n
  52. When the '__get__()' method is called to create the bound method, it will supply both the instance and the now bound method.\n
  53. Why is this important? Well things get a bit messy at this point. Where a function is supplied to derive what the SQL is from the arguments, for the generalised case we really want to know what the instance was as well and if you take only the arguments being passed to the bound method you don't get that. Thus we have to explicitly insert the instance object so our function deriving the SQL from the arguments can use it if need be.\n
  54. So, we do everything right but it still fails in some cases. Turns out not all database client modules are pure Python code. (NEXT) And you cannot monkey patch methods of Python objects implemented in C. Well, you can't do it from Python code. You can do it from C code by being evil and modify the Python C object '__dict__', but that is shared across sub interpreters which would cause problems for mod_wsgi.\n
  55. So, we do everything right but it still fails in some cases. Turns out not all database client modules are pure Python code. (NEXT) And you cannot monkey patch methods of Python objects implemented in C. Well, you can't do it from Python code. You can do it from C code by being evil and modify the Python C object '__dict__', but that is shared across sub interpreters which would cause problems for mod_wsgi.\n
  56. What we have to do in this case is start from the top and start wrapping objects from the only place we can change things, which is the module instance itself. We therefore wrap the DBAPI 'connect()' function and replace it with a factory object.\n
  57. That factory object intercepts the call and wraps the result with a connection object wrapper. The connection object wrapper in turn intercepts the 'cursor()' call and wraps the result with our own cursor object wrapper. \n
  58. Finally we can then intercept the 'execute()' method in which we can capture the SQL. Now remember how we used 'functools.wraps()' before. We can’t use that here as the ‘description’ attribute of a Cursor object is updated after ‘execute()’ is called. The 'functools.wraps()' function only makes a copy at the time of its call and doesn't reflect live changes. We therefore must use ‘__getattr__()’ instead.\n
  59. Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
  60. Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
  61. Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
  62. Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
  63. Too much stuff. Too little time. (NEXT) Come and talk to us outside where we will be in one of the side rooms. (NEXT) We will be happy to give you a demonstration. (NEXT) Come to drinks and nibbles on Wednesday evening. Meet some others from New Relic coming up from the office. Also look out for Diane from ActiveState, one of our first partners who will be offering up a special deal for Stackato users.\n
  64. One final thing. Specially for DjangoCon, we have moved up our public beta just so you could all get in on the action, and even perhaps go as far as trying it out while we are here to help you to get it going. We will be more than happy to then see what sort of data you are getting and help you to understand it. Also will be interested in any feedback. \n
  65. If we have still have time we can do some questions, but if not then you can get me later. While doing questions here is the link for signing up for the beta and the email address if you want to send us questions that way instead.\n