SlideShare a Scribd company logo
1 of 45
Download to read offline
Collaboration Summit 2013




LTTng-UST: Efficient System-Wide
      User-Space Tracing


                  christian.babeux@efficios.com 
                                       @c_bab   1
whoami

   Christian Babeux, Software Developer, EfficiOS,

       Background in embedded and ASIC tools,

       Active contributor to the LTTng projects:
    ●   lttng-tools & lttng-ust,
    ●   CI infra, Website, Twitter.

       AUR package maintainer for Arch Linux.

                                                    2
Content

    Overview of LTTng 2.x and UST,

    User-space instrumentation sources,

    Trace format standardisation efforts,

    Tales from a user-space tracer,

     Recent features & future work.
                                             3
Overview of LTTng 2.x



                       4
Overview of LTTng 2.x
   Unified user interface, API, kernel and user
space tracers,


   Trace output in a unified format,


   Low overhead,


  Shipped in distros: Ubuntu, Debian, Suse,
Fedora, Linaro, Wind River, etc.
                                                   5
Project overview




                  

                       6
Tracers




●   lttng-modules: kernel tracer module, compatible
    with kernels from 2.6.38* to 3.9,
●   lttng-ust: user-space tracer, in-process library.
* Kernel tracing is now possible on 2.6.32 to 2.6.37 by backport of
3 Linux Kernel patches [1].

                                                                      7
Utilities


     
●   lttng-tools: cli utilities and daemons for trace control,
           –   lttng: cli utility for tracing control,
           –   lttng-sessiond: tracing registry daemon,
           –   lttng-consumerd: consume trace data,
           –   lttng-relayd: network streaming daemon.

                                                                8
Viewers


     
●   babeltrace: cli text viewer, trace converter,
    plugin system,
●   lttngtop: ncurse top-like viewer,
●   Eclipse lttng plugin: front-end for lttng, collect,
    visualize and analyze traces, highly extensible.
                                                          9
Overview of LTTng-UST



                       10
LTTng-UST – Features
      Pure userspace implementation,
        –   Shared memory map between apps and trace
            consumers,
        –   Portable to other OS: BSDs, Cygwin
            (experimental).


      Optimized for low-overhead, high-throughput
[2],
        –   Generic kernel ringbuffer ported to userspace,
        –   Efficient concurrent data structures for trace
            control.
                                                             11
LTTng-UST – Features (cont.)

    Dynamically enabled, statically defined
instrumentation,


    Per user tracing and system-wide tracing,
      –   Tracing group for system-wide tracing.


   Traces recoverable even after application crash.


                                                   12
LTTng-UST – How does it work?

    Users instrument their applications with
 static tracepoints,
    liblttng-ust, in-process library, dynamically
 linked with application,
    Session setup, etc.,
    Run app, collect traces,
    Post analysis with viewers.


                                                     13
Tracing session - Setup




           Session setup     $ lttng create
User-space event enabling    $ lttng enable-event -u -a
             Start tracing   $ lttng start



                                                          14
Tracing session - A wild app appears




●   Listener thread spawned via
    constructor (GCC extension),
●   App registration,
●   Send SHM and wait fd.




                                       15
Tracing session – App. execution & teardown




●   App running,
●   Events written to ringbuffer,
●   Notification of data
    availability via pipe,
●   App unregistered via
    destructor.




                                          16
User-space instrumentation
         sources



                            17
Tracepoints - Declaration
TRACEPOINT_EVENT(
    /* Provider name */
    ust_tests_hello,

    /* Tracepoint name */
    tptest,

    /* Type, variable name */
    TP_ARGS(int, anint,
            long *, values,
            float, floatarg),

    /* Type, field name, expression */
    TP_FIELDS(ctf_integer(int, intfield, anint),
              ctf_array(long, arrfield1, values, 3),
              ctf_float(float, floatfield, floatarg))
)

                                                        18
Tracepoints - Invocation
void function(void)
{
    int i = 0;
    long vals[3] = { 0x42, 0xCC, 0xC001CAFE };
    float flt = M_PI;

    [...]
    tracepoint(ust_tests_hello,
               tptest,
               i,
               &vals,
               flt);
    [...]
}
                                                 19
SystemTAP SDT Providers

 Integration result of Collaboration Summit 2011
discussions,



 Compatibility with SystemTAP SDT,
     –   Users can use SystemTAP with tracepoint()
         instrumented code.



                                                     20
Uprobes

   Kernel patchset merged in 3.5,



   LTTng integration:
     – Initial lttng-modules patchset proposed [4],
     – Need usability improvement
     – Interface not exported by kernel



                                                      21
Trace format standardisation efforts




                                       22
Trace format standardisation efforts




Source: xkcd.com/927



                                       23
Trace format standardisation efforts
   Joking aside: We need a common open format,
   Collaboration: Multicore Association, Ericsson,
   Goals of the Common Trace Format (CTF):
     –   Common format for SW and HW traces,
     –   Portable,
     –   Compact,
   Tools based on CTF:
     –   LTTng 2.x, Babeltrace, Eclipse LTTng plugin
     –   GDB (save trace to CTF) [3],
     –   Javeltrace                                    24
Common Trace Format




    Self-described, packet-based format.
                                            25
Common Trace Format – More info.

   “Interoperability Between Tracing Tools with the
 Common Trace Format”,
       –   Mathieu Desnoyers at Linux Plumbers 2012 [5]


    Common Trace Format (CTF) Specification [6],


    Common Trace Format compliance testsuite [7].


                                                          26
Tales from a user-space
         tracer



                         27
Non-intrusive handling of SIGPIPE

    Ringbuffer delivery notification use a pipe,
      –   Traced applications can receive SIGPIPE if
          consumer end dies abruptly.


    Suppress SIGPIPE only in our lib without
 affecting signal handling for the rest of the
 process [8].



                                                       28
TLS & constructors
●   Thread Local Storage (TLS) variable storage in
    dynamically libs. allocated when first used [9],

●   Rely on internal glibc mutex to protect against
    dynamic linker,

●   Same mutex is held while running ctor/dtor,



                                                       29
TLS & constructors (cont.)




                             30
TLS & constructors (cont.)


●   Take mutex within constructors while TLS fixup
    performed,
                    Deadlock!

●   Workaround: Force TLS fixup within lib ctor.




                                                     31
Tracing of apps closing all fds




                                  32
Close all the things

●   When daemonizing, some apps close all
    available fds,




                                            33
Tracing of apps closing all fds

●   When daemonizing, some apps close all
    available fds,


           No communication == No tracing.

●   Fix: None for the moment.




                                              34
Recent features & future
         work



                          35
Recent features
    2.1 (Basse Messe)
          Network streaming over TCP,
           ●   Introduce lttng-relayd, receive traces from
               remote consumers.


          Filtering before data collection,
           ●   C-like syntax, bytecode interpreter.
           ●   UST only for the moment.



              Session daemon health monitoring API.
                                                             36
Network streaming over TCP




                             37
Filtering (1)




 Filter:
 “(intfield > 42 && intfield <= 44) || longfield == 1”



                                                         38
Filtering (2)




 “(intfield > 42 && intfield <= 44) || longfield == 1”
                                                         39
Recent features (cont.)

    2.2 (Cuda, Currently in RC)
          Per-uid buffers in UST,


          Context filtering,
           ●   '$ctx.procname == “demo*”',
           ●   '$ctx.vpid > 9000'.


          Trace file size limits,

                                             40
Future work

    Flight recorder mode tracing (2.3),



    Trace data extracted on core dump (2.3),



     Java tracing.

                                                41
Future work (cont.)

   Tracer triggers actions on specific events & filters


   Compressed, encrypted streaming and storage,


 LTTng accepted in Google Summer of Code [10].
     –   Dynamic instrumentation support in UST,
     –   Android port.


                                                      42
Conclusion




  Usability of user space tracing in production




                                                   43
Questions ?




   ?
                 www.efficios.com




               lttng.org
               lttng-dev@lists.lttng.org
               @lttng_project           44
References
●   [1] – Userspace tracing in small footprint devices – Jason Wessel
●   [2] – lttng-modules README -
●   [3] – [lttng-dev] [lttng-modules PATCH] Add uprobes support – Yannick Brosseau
●   [4] – [PATCH v3 00/15] CTF Support – Yao Qi
●   [5] -
    “Interoperability Between Tracing Tools with the Common Trace Format” - Mathieu Desnoyers
    , Linux Plumbers 2012
●   [6] - Common Trace Format (CTF) Specification
●   [7] - Common Trace Format compliance testsuite
●   [8] – LTTng-UST – 2C44F5B9 - Fix UST SIGPIPE handling
●   [9] – ELF Handling for Thread-Local Storage – Ulrich Drepper (page 8)
●   [10] – LTTng GSoC 2013 Ideas list




                                                                                         45

More Related Content

What's hot

Mpls by vidhu
Mpls by vidhuMpls by vidhu
Mpls by vidhu
CU
 

What's hot (20)

Internet standard routing protocols
Internet standard routing protocolsInternet standard routing protocols
Internet standard routing protocols
 
Оверлейные сети ЦОД Технологии VXLAN и EVPN
Оверлейные сети ЦОД Технологии VXLAN и EVPN Оверлейные сети ЦОД Технологии VXLAN и EVPN
Оверлейные сети ЦОД Технологии VXLAN и EVPN
 
FD.io Vector Packet Processing (VPP)
FD.io Vector Packet Processing (VPP)FD.io Vector Packet Processing (VPP)
FD.io Vector Packet Processing (VPP)
 
Snabbflow: A Scalable IPFIX exporter
Snabbflow: A Scalable IPFIX exporterSnabbflow: A Scalable IPFIX exporter
Snabbflow: A Scalable IPFIX exporter
 
Implementing IPv6 Segment Routing in the Linux kernel
Implementing IPv6 Segment Routing in the Linux kernelImplementing IPv6 Segment Routing in the Linux kernel
Implementing IPv6 Segment Routing in the Linux kernel
 
Build a High Available NFS Cluster Based on CephFS - Shangzhong Zhu
Build a High Available NFS Cluster Based on CephFS - Shangzhong ZhuBuild a High Available NFS Cluster Based on CephFS - Shangzhong Zhu
Build a High Available NFS Cluster Based on CephFS - Shangzhong Zhu
 
Ifupdown2: Network Interface Manager
Ifupdown2: Network Interface ManagerIfupdown2: Network Interface Manager
Ifupdown2: Network Interface Manager
 
Mpls by vidhu
Mpls by vidhuMpls by vidhu
Mpls by vidhu
 
1 typhoon, rafale, gripen
1 typhoon, rafale, gripen1 typhoon, rafale, gripen
1 typhoon, rafale, gripen
 
EIGRP Routing Protocols
EIGRP Routing ProtocolsEIGRP Routing Protocols
EIGRP Routing Protocols
 
Ceph Performance and Sizing Guide
Ceph Performance and Sizing GuideCeph Performance and Sizing Guide
Ceph Performance and Sizing Guide
 
Vxlan deep dive session rev0.5 final
Vxlan deep dive session rev0.5   finalVxlan deep dive session rev0.5   final
Vxlan deep dive session rev0.5 final
 
Seastore: Next Generation Backing Store for Ceph
Seastore: Next Generation Backing Store for CephSeastore: Next Generation Backing Store for Ceph
Seastore: Next Generation Backing Store for Ceph
 
Remotely Piloted Aircraft System
Remotely Piloted Aircraft SystemRemotely Piloted Aircraft System
Remotely Piloted Aircraft System
 
EIGRP (Enhanced Interior Gateway Routing Protocol)
EIGRP (Enhanced Interior Gateway Routing Protocol)EIGRP (Enhanced Interior Gateway Routing Protocol)
EIGRP (Enhanced Interior Gateway Routing Protocol)
 
VXLAN Practice Guide
VXLAN Practice GuideVXLAN Practice Guide
VXLAN Practice Guide
 
BGP Overview
BGP OverviewBGP Overview
BGP Overview
 
Service Function Chaining with SRv6
Service Function Chaining with SRv6Service Function Chaining with SRv6
Service Function Chaining with SRv6
 
Lab6.4.1
Lab6.4.1Lab6.4.1
Lab6.4.1
 
6. sokerit
6. sokerit6. sokerit
6. sokerit
 

Similar to LTTng-UST: Efficient System-Wide User-Space Tracing

Lcu14 101- coresight overview
Lcu14 101- coresight overviewLcu14 101- coresight overview
Lcu14 101- coresight overview
Linaro
 
下午1 intel yang, elton_mee_go-arch-update-final
下午1 intel yang, elton_mee_go-arch-update-final下午1 intel yang, elton_mee_go-arch-update-final
下午1 intel yang, elton_mee_go-arch-update-final
csdnmobile
 

Similar to LTTng-UST: Efficient System-Wide User-Space Tracing (20)

Bridging the gap between hardware and software tracing
Bridging the gap between hardware and software tracingBridging the gap between hardware and software tracing
Bridging the gap between hardware and software tracing
 
Automotive Grade Linux and systemd
Automotive Grade Linux and systemdAutomotive Grade Linux and systemd
Automotive Grade Linux and systemd
 
Deploy STM32 family on Zephyr - SFO17-102
Deploy STM32 family on Zephyr - SFO17-102Deploy STM32 family on Zephyr - SFO17-102
Deploy STM32 family on Zephyr - SFO17-102
 
OpenTelemetry For Architects
OpenTelemetry For ArchitectsOpenTelemetry For Architects
OpenTelemetry For Architects
 
Tos tutorial
Tos tutorialTos tutorial
Tos tutorial
 
Introduction to FreeRTOS
Introduction to FreeRTOSIntroduction to FreeRTOS
Introduction to FreeRTOS
 
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
Cotopaxi - IoT testing toolkit (Black Hat Asia 2019 Arsenal)
 
Heterogeneous multiprocessing on androd and i.mx7
Heterogeneous multiprocessing on androd and i.mx7Heterogeneous multiprocessing on androd and i.mx7
Heterogeneous multiprocessing on androd and i.mx7
 
Pristine glif 2015
Pristine glif 2015Pristine glif 2015
Pristine glif 2015
 
Implementing Observability for Kubernetes.pdf
Implementing Observability for Kubernetes.pdfImplementing Observability for Kubernetes.pdf
Implementing Observability for Kubernetes.pdf
 
LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1
 
Advanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTngAdvanced Tracing features using GDB and LTTng
Advanced Tracing features using GDB and LTTng
 
AMIT SRIVASTAVA
AMIT SRIVASTAVAAMIT SRIVASTAVA
AMIT SRIVASTAVA
 
Lcu14 101- coresight overview
Lcu14 101- coresight overviewLcu14 101- coresight overview
Lcu14 101- coresight overview
 
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaPrometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
 
Provenance for Data Munging Environments
Provenance for Data Munging EnvironmentsProvenance for Data Munging Environments
Provenance for Data Munging Environments
 
下午1 intel yang, elton_mee_go-arch-update-final
下午1 intel yang, elton_mee_go-arch-update-final下午1 intel yang, elton_mee_go-arch-update-final
下午1 intel yang, elton_mee_go-arch-update-final
 
Onnc intro
Onnc introOnnc intro
Onnc intro
 
Tizen RT: A Lightweight RTOS Platform for Low-End IoT Devices
Tizen RT: A Lightweight RTOS Platform for Low-End IoT DevicesTizen RT: A Lightweight RTOS Platform for Low-End IoT Devices
Tizen RT: A Lightweight RTOS Platform for Low-End IoT Devices
 
D4 Project Presentation
D4 Project PresentationD4 Project Presentation
D4 Project Presentation
 

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 

Recently uploaded (20)

Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Tech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdfTech Trends Report 2024 Future Today Institute.pdf
Tech Trends Report 2024 Future Today Institute.pdf
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 

LTTng-UST: Efficient System-Wide User-Space Tracing

  • 1. Collaboration Summit 2013 LTTng-UST: Efficient System-Wide User-Space Tracing christian.babeux@efficios.com  @c_bab  1
  • 2. whoami  Christian Babeux, Software Developer, EfficiOS,  Background in embedded and ASIC tools,  Active contributor to the LTTng projects: ● lttng-tools & lttng-ust, ● CI infra, Website, Twitter.  AUR package maintainer for Arch Linux. 2
  • 3. Content  Overview of LTTng 2.x and UST,  User-space instrumentation sources,  Trace format standardisation efforts,  Tales from a user-space tracer,  Recent features & future work. 3
  • 4. Overview of LTTng 2.x  4
  • 5. Overview of LTTng 2.x  Unified user interface, API, kernel and user space tracers,  Trace output in a unified format,  Low overhead,  Shipped in distros: Ubuntu, Debian, Suse, Fedora, Linaro, Wind River, etc. 5
  • 6. Project overview   6
  • 7. Tracers ● lttng-modules: kernel tracer module, compatible with kernels from 2.6.38* to 3.9, ● lttng-ust: user-space tracer, in-process library. * Kernel tracing is now possible on 2.6.32 to 2.6.37 by backport of 3 Linux Kernel patches [1]. 7
  • 8. Utilities  ● lttng-tools: cli utilities and daemons for trace control, – lttng: cli utility for tracing control, – lttng-sessiond: tracing registry daemon, – lttng-consumerd: consume trace data, – lttng-relayd: network streaming daemon. 8
  • 9. Viewers  ● babeltrace: cli text viewer, trace converter, plugin system, ● lttngtop: ncurse top-like viewer, ● Eclipse lttng plugin: front-end for lttng, collect, visualize and analyze traces, highly extensible. 9
  • 11. LTTng-UST – Features  Pure userspace implementation, – Shared memory map between apps and trace consumers, – Portable to other OS: BSDs, Cygwin (experimental).  Optimized for low-overhead, high-throughput [2], – Generic kernel ringbuffer ported to userspace, – Efficient concurrent data structures for trace control. 11
  • 12. LTTng-UST – Features (cont.)  Dynamically enabled, statically defined instrumentation,  Per user tracing and system-wide tracing, – Tracing group for system-wide tracing.  Traces recoverable even after application crash. 12
  • 13. LTTng-UST – How does it work?  Users instrument their applications with static tracepoints,  liblttng-ust, in-process library, dynamically linked with application,  Session setup, etc.,  Run app, collect traces,  Post analysis with viewers. 13
  • 14. Tracing session - Setup Session setup $ lttng create User-space event enabling $ lttng enable-event -u -a Start tracing $ lttng start 14
  • 15. Tracing session - A wild app appears ● Listener thread spawned via constructor (GCC extension), ● App registration, ● Send SHM and wait fd. 15
  • 16. Tracing session – App. execution & teardown ● App running, ● Events written to ringbuffer, ● Notification of data availability via pipe, ● App unregistered via destructor. 16
  • 17. User-space instrumentation sources  17
  • 18. Tracepoints - Declaration TRACEPOINT_EVENT( /* Provider name */ ust_tests_hello, /* Tracepoint name */ tptest, /* Type, variable name */ TP_ARGS(int, anint, long *, values, float, floatarg), /* Type, field name, expression */ TP_FIELDS(ctf_integer(int, intfield, anint), ctf_array(long, arrfield1, values, 3), ctf_float(float, floatfield, floatarg)) ) 18
  • 19. Tracepoints - Invocation void function(void) { int i = 0; long vals[3] = { 0x42, 0xCC, 0xC001CAFE }; float flt = M_PI; [...] tracepoint(ust_tests_hello, tptest, i, &vals, flt); [...] } 19
  • 20. SystemTAP SDT Providers  Integration result of Collaboration Summit 2011 discussions,  Compatibility with SystemTAP SDT, – Users can use SystemTAP with tracepoint() instrumented code. 20
  • 21. Uprobes  Kernel patchset merged in 3.5,  LTTng integration: – Initial lttng-modules patchset proposed [4], – Need usability improvement – Interface not exported by kernel 21
  • 23. Trace format standardisation efforts Source: xkcd.com/927 23
  • 24. Trace format standardisation efforts  Joking aside: We need a common open format,  Collaboration: Multicore Association, Ericsson,  Goals of the Common Trace Format (CTF): – Common format for SW and HW traces, – Portable, – Compact,  Tools based on CTF: – LTTng 2.x, Babeltrace, Eclipse LTTng plugin – GDB (save trace to CTF) [3], – Javeltrace 24
  • 25. Common Trace Format  Self-described, packet-based format. 25
  • 26. Common Trace Format – More info.  “Interoperability Between Tracing Tools with the Common Trace Format”, – Mathieu Desnoyers at Linux Plumbers 2012 [5]  Common Trace Format (CTF) Specification [6],  Common Trace Format compliance testsuite [7]. 26
  • 27. Tales from a user-space tracer  27
  • 28. Non-intrusive handling of SIGPIPE  Ringbuffer delivery notification use a pipe, – Traced applications can receive SIGPIPE if consumer end dies abruptly.  Suppress SIGPIPE only in our lib without affecting signal handling for the rest of the process [8]. 28
  • 29. TLS & constructors ● Thread Local Storage (TLS) variable storage in dynamically libs. allocated when first used [9], ● Rely on internal glibc mutex to protect against dynamic linker, ● Same mutex is held while running ctor/dtor, 29
  • 30. TLS & constructors (cont.) 30
  • 31. TLS & constructors (cont.) ● Take mutex within constructors while TLS fixup performed,  Deadlock! ● Workaround: Force TLS fixup within lib ctor. 31
  • 32. Tracing of apps closing all fds 32
  • 33. Close all the things ● When daemonizing, some apps close all available fds, 33
  • 34. Tracing of apps closing all fds ● When daemonizing, some apps close all available fds,  No communication == No tracing. ● Fix: None for the moment. 34
  • 35. Recent features & future work  35
  • 36. Recent features  2.1 (Basse Messe)  Network streaming over TCP, ● Introduce lttng-relayd, receive traces from remote consumers.  Filtering before data collection, ● C-like syntax, bytecode interpreter. ● UST only for the moment.  Session daemon health monitoring API. 36
  • 38. Filtering (1) Filter: “(intfield > 42 && intfield <= 44) || longfield == 1” 38
  • 39. Filtering (2) “(intfield > 42 && intfield <= 44) || longfield == 1” 39
  • 40. Recent features (cont.)  2.2 (Cuda, Currently in RC)  Per-uid buffers in UST,  Context filtering, ● '$ctx.procname == “demo*”', ● '$ctx.vpid > 9000'.  Trace file size limits, 40
  • 41. Future work  Flight recorder mode tracing (2.3),  Trace data extracted on core dump (2.3),  Java tracing. 41
  • 42. Future work (cont.)  Tracer triggers actions on specific events & filters  Compressed, encrypted streaming and storage,  LTTng accepted in Google Summer of Code [10]. – Dynamic instrumentation support in UST, – Android port. 42
  • 43. Conclusion  Usability of user space tracing in production 43
  • 44. Questions ? ?  www.efficios.com  lttng.org  lttng-dev@lists.lttng.org  @lttng_project 44
  • 45. References ● [1] – Userspace tracing in small footprint devices – Jason Wessel ● [2] – lttng-modules README - ● [3] – [lttng-dev] [lttng-modules PATCH] Add uprobes support – Yannick Brosseau ● [4] – [PATCH v3 00/15] CTF Support – Yao Qi ● [5] - “Interoperability Between Tracing Tools with the Common Trace Format” - Mathieu Desnoyers , Linux Plumbers 2012 ● [6] - Common Trace Format (CTF) Specification ● [7] - Common Trace Format compliance testsuite ● [8] – LTTng-UST – 2C44F5B9 - Fix UST SIGPIPE handling ● [9] – ELF Handling for Thread-Local Storage – Ulrich Drepper (page 8) ● [10] – LTTng GSoC 2013 Ideas list 45