USE              IMPROVE                   EVANGELIZE


Observability Matters:
How DTrace Helped Twitter
Adam Leventhal
Br...
USE   IMPROVE   EVANGELIZE




This Talk
    Twitter performance meltdown
●

    Introduction to DTrace
●

    DTrace on t...
USE     IMPROVE   EVANGELIZE




What Is Twitter?
    Social networking/RSS/SMS
●

    Ruby on Rails application
●

    Ho...
USE     IMPROVE   EVANGELIZE




Twitter's Problem
    Started with a few users
●

    Didn't scale with their success
●

...
USE    IMPROVE   EVANGELIZE




Enter DTrace
    Luckily: Twitter had DTrace
●
        Solaris 10, Mac OS X 10.5, FreeBSD*...
USE   IMPROVE   EVANGELIZE




DEMO

                       6
USE   IMPROVE   EVANGELIZE




The Investigation
    High latencies under load
●

    What could cause these latencies?
●
...
USE   IMPROVE   EVANGELIZE




DEMO

                       8
USE   IMPROVE   EVANGELIZE




Investigation: Steps
1. Started with a broad look at the system
 using familiar tools
     ...
USE   IMPROVE   EVANGELIZE




Investigation: So Far
    What we knew:
●

        High application latency (from somewhere...
USE   IMPROVE   EVANGELIZE




DEMO

                       11
USE   IMPROVE   EVANGELIZE




Investigation: Steps
4. Function call counts
5. ustack() for memcpy()
6. Time spent in back...
USE   IMPROVE   EVANGELIZE




DEMO

                       13
USE   IMPROVE   EVANGELIZE




Investigation: Steps
7. Located source of backtrace() calls in
  Ruby code

    What we lea...
USE   IMPROVE   EVANGELIZE




Results: No Exceptions
    Rather than blithely calling the method:
●
    @string = (str.st...
USE   IMPROVE   EVANGELIZE




Ruby Provider
    Joyent built DTrace-enabled Ruby
●

    Trace function entry and return
●...
USE   IMPROVE   EVANGELIZE




DEMO

                       17
USE   IMPROVE   EVANGELIZE




Results: CRC32 in C
    Precision optimization
●

    Rewrite the CRC32 computation in C
●
...
USE   IMPROVE   EVANGELIZE




DTrace Your Application
    Get your application on an OS with
●
    DTrace
    Start with ...
USE    IMPROVE   EVANGELIZE




    Q&A (and links)
    DTrace home page
–
         http://www.opensolaris.org/os/communit...
USE      IMPROVE   EVANGELIZE


Observability Matters:
How DTrace Helped Twitter
Adam Leventhal
http://blogs.sun.com/ahl

...
Upcoming SlideShare
Loading in...5
×

Os Leventhal

1,150
-1

Published on

Published in: Business, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,150
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Os Leventhal

  1. 1. USE IMPROVE EVANGELIZE Observability Matters: How DTrace Helped Twitter Adam Leventhal Brendan Gregg FishWorks Sun Microsystems a version of this presentation with the demo materials will be available shortly after the conference at http://blogs.sun.com/ahl
  2. 2. USE IMPROVE EVANGELIZE This Talk Twitter performance meltdown ● Introduction to DTrace ● DTrace on the case ● Solutions and results ● DTrace for your application ● Q&A ● 2
  3. 3. USE IMPROVE EVANGELIZE What Is Twitter? Social networking/RSS/SMS ● Ruby on Rails application ● Horizontally scaled ● Centralized MySQL backend ● 3
  4. 4. USE IMPROVE EVANGELIZE Twitter's Problem Started with a few users ● Didn't scale with their success ● High latencies could make it unusable ● Many possible suspects: ● The OS: kernel, libraries, etc. – Ruby, MySQL, Apache – The Twitter application itself – 4
  5. 5. USE IMPROVE EVANGELIZE Enter DTrace Luckily: Twitter had DTrace ● Solaris 10, Mac OS X 10.5, FreeBSD* – Systemic observability ● Ruby, Java, JavaScript, C/C++, kernel ... – Concise answers to arbitrary questions ● Designed for production systems ● Architected to always be safe – No overhead when not in use – Also great for developers ● 5
  6. 6. USE IMPROVE EVANGELIZE DEMO 6
  7. 7. USE IMPROVE EVANGELIZE The Investigation High latencies under load ● What could cause these latencies? ● on-CPU time: slow/numerous functions, ... – off-CPU time: I/O, lock contention, ... – 7
  8. 8. USE IMPROVE EVANGELIZE DEMO 8
  9. 9. USE IMPROVE EVANGELIZE Investigation: Steps 1. Started with a broad look at the system using familiar tools mpstat, ... ● 2. DTrace profiling to sample processes 3. DTrace profiling to sample user stacks 9
  10. 10. USE IMPROVE EVANGELIZE Investigation: So Far What we knew: ● High application latency (from somewhere) – What we learned: ● Latency may be due to high CPU load – CPU load is due to some Ruby component: – Twitter application ● Standard Ruby libraries ● Interpreter itself ● 10
  11. 11. USE IMPROVE EVANGELIZE DEMO 11
  12. 12. USE IMPROVE EVANGELIZE Investigation: Steps 4. Function call counts 5. ustack() for memcpy() 6. Time spent in backtrace() What we learned: ● 40% of CPU time spent in backtrace() – 12
  13. 13. USE IMPROVE EVANGELIZE DEMO 13
  14. 14. USE IMPROVE EVANGELIZE Investigation: Steps 7. Located source of backtrace() calls in Ruby code What we learned: ● Several instances of code like this: – @string = (str.string rescue str) 14
  15. 15. USE IMPROVE EVANGELIZE Results: No Exceptions Rather than blithely calling the method: ● @string = (str.string rescue str) ... check first: ● @string = str.responds_to?(:string) ? str.string : str Result: 30% drop in CPU utilization ● 15
  16. 16. USE IMPROVE EVANGELIZE Ruby Provider Joyent built DTrace-enabled Ruby ● Trace function entry and return ● Probes for line execution, memory ● allocation, etc. 16
  17. 17. USE IMPROVE EVANGELIZE DEMO 17
  18. 18. USE IMPROVE EVANGELIZE Results: CRC32 in C Precision optimization ● Rewrite the CRC32 computation in C ● Estimated result: 15% drop in CPU ● utilization 18
  19. 19. USE IMPROVE EVANGELIZE DTrace Your Application Get your application on an OS with ● DTrace Start with the tools you know and dive ● deeper with DTrace If you can do it today or tomorrow, find ● Adam and Brendan: we'll help you investigate DTrace sees all ● 19
  20. 20. USE IMPROVE EVANGELIZE Q&A (and links) DTrace home page – http://www.opensolaris.org/os/community/dtrace ● DTrace-enabled Ruby – http://dtrace.joyent.com ● Getting started with DTrace – http://blogs.sun.com/ahl/entry/dtrace_boot_camp ● Exception problem: – http://dev.rubyonrails.org/ticket/8159 ● http://dev.rubyonrails.org/changeset/6571 ● Joyent's blog post on this subject – http://joyeur.com/2007/04/24/solaris-dtrace-and-rails 20 ●
  21. 21. USE IMPROVE EVANGELIZE Observability Matters: How DTrace Helped Twitter Adam Leventhal http://blogs.sun.com/ahl Brendan Gregg http://blogs.sun.com/brendan

×