Bridging the gap between hardware and software tracing
Upcoming SlideShare
Loading in...5
×
 

Bridging the gap between hardware and software tracing

on

  • 559 views

For a numbers of years, silicon vendors have been providing hardware tracing facilities to embedded developers. By using these, developers can resolve performance and latency issues more quickly, ...

For a numbers of years, silicon vendors have been providing hardware tracing facilities to embedded developers. By using these, developers can resolve performance and latency issues more quickly, resulting in shorter time to market. In this talk, we will cover the hardware based tracing facilities offered by various manufacturers and see how they differ from their software counterparts with respect to their instrumentation capabilities, transport mechanisms, output formats, etc. We will also show how joint hardware and software tracing can be used by developers to gain deeper insights in their applications’ behaviour. Finally, we will outline the on-going work within the Linux Trace Toolkit next generation (LTTng) project to enhance hardware tracing support and tracing data visualization.

Statistics

Views

Total Views
559
Views on SlideShare
551
Embed Views
8

Actions

Likes
1
Downloads
13
Comments
0

1 Embed 8

http://www.linkedin.com 8

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Bridging the gap between hardware and software tracing Bridging the gap between hardware and software tracing Presentation Transcript

  • Embedded Linux Conference Europe 2013 Bridging the gap between hardware and software tracing christian.babeux@efficios.com  1 @c_bab 
  • Source: arm.com [1] 2
  • ARM Cortex A9 Source: arm.com [1] 3
  • ARM Cortex A9 PTM Program Trace Macrocell Source: arm.com [1] 4
  • whoami  Christian Babeux, Software Developer, EfficiOS,  Background in embedded and ASIC tools,  Active contributor to the LTTng projects: ● ●  lttng-tools, lttng-ust, babeltrace, CI infra, Website, Twitter. AUR package maintainer for Arch Linux. 5
  • Content  What is hardware tracing?,  Why is it useful?,  ARM Coresight & ETM,  Freescale QorIQ & Nexus tracing,  LTTng & hardware tracing. 6
  • What is hardware tracing?  7
  • What is hardware tracing?  Hardware component(s) used to traces instructions and data movement of a processing device  Real-time observation Low intrusiveness 8
  • What is hardware tracing? Cont. ● External trace 9
  • What is hardware tracing? Cont. ● Pros – – ● Can accommodate high data bandwidth Minimal impact on system performance – Trace port not always available – Custom hardware needed Cons 10
  • What is hardware tracing? Cont. ● Self-hosted trace 11
  • What is hardware tracing? Cont. ● Pros – – ● Self-contained, facilities can be used by host OS No need for special hardware – Limited internal buffer space – Might impact system performance Cons 12
  • Hardware tracing support ● ARM – – Program Trace Flow Macrocell (PTM), – ● Embedded Trace Macrocell (ETM), System Trace Macrocell (STM) PowerPC – – ● Freescale QorIQ with Nexus tracing Branch History Rolling Buffer (BHRB) – Intel Processor Tracing (PT) – Last Branch Record (LBR), Branch Trace Store (BTS) Intel 13
  • Hardware tracing vs. software tracing ● Software tracing: – – Can be intrusive – Can be slow – ● Static instrumentation or dynamic code patching Tracepoint level granularity Hardware tracing: – Tracing done on hardware – Instrumentation not required – Instruction level granularity 14
  • Why hardware tracing is useful  15
  • Why hardware tracing is useful  Profiling – Very fine granularity profiling  Performance measurement  Code coverage  Monitoring – Statistics on application currently running? 16
  • Why hardware tracing is useful cont.  Snapshot on crash/anomaly – Trace overwrites old data until anomaly detected  Event trigger trace  Hardware-assisted software tracing – Use hardware facility (ringbuffer) 17
  • ARM Coresight & ETM  18
  • ARM Coresight ● Coresight – – Trace and debug a complete SoC – ● Collection of hardware components Open architecture Trace source – Processing elements (CPU, DSP, etc.) – Buses – System trace (generated from software) 19
  • ARM ETM ● ETM – Monitor the core internal bus – Instructions + data trace – Hardware filters and triggers – Trace stream compression – Traces can be saved in internal buffer (ETB) or shared system memory 20
  • ARM Coresight Source: arm.com [2] 21
  • State of Coresight & ETM in Linux ● ETM tracer implementation available in Linux – ● Coresight support status – ● Seems to work only on specific hardware Framework patchset proposed by Pratik Pratel [3] Trace decoder availability? 22
  • Interested about Coresight/ETM? “Hardware Trace in the Kernel” BoF by Pawel Moll from ARM Today, 4:30 PM in Pentland 23
  • Freescale QorIQ & Nexus tracing  24
  • Freescale QorIQ ● PowerPC based platform targeted for high-performance communications usage – Multiple e500mc processors, – DPAA support (packet processing offloading), – Support the Nexus debugging & tracing standard. Source: radiolocman 25 [4]
  • Nexus standard ● ● ● ISO standard for debugging embedded systems (IEEEISTO-5001-2003), Designed for low pin count, standard set of connectors (JTAG or Debug port), Multiple level of Nexus “compliance”. 26
  • Nexus standard level ● Level 1: – – ● Run time control only (run, stop, breakpoints, etc.) Tracing not supported Level 2: – ● Level 3: – ● Ownership and program trace Data write trace & memory read/write on the fly Level 4: – Memory substitution – Trace triggering via a watchpoint 27
  • Nexus standard format ● Packet based output format, – Standard defines public messages, vendors can define extensions (TCODES) – Fixed packet size per message, last packet can be of variable length – Message can have an optional timestamp 28
  • Example decoded Nexus message Message # 328 TCODE : 2 Ownership Trace Message SRC ID : 0 Core0 / CPU0 (Clst0:Core0:Thread0) PID INDEX : 0x02 - Sync PID PID VALUE : 0x01e01bf59c - LPIDR, MSR[GS], PID/NPIDR (ref:DC1[OTS]), MSR[PR] TIMESTAMP : 1140061 (0x11655d) Message # 329 TCODE : 27 Resource Full Message SRC ID : 0 Core0 / CPU0 (Clst0:Core0:Thread0) RCODE : 0x8 - Timestamp counter RDATA : 0x00ffffff TIMESTAMP : 0 (0x0) 29
  • State of Nexus in Linux ● Nexus qoriq-debug kernel module, – Available in Freescale QorIQ SDK Yocto Layer – Implements a debugfs with memory mapped access to Nexus control register – cat /sys/kernel/debug/npc/trace_buffer > trace 30
  • Nexus debugfs root@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg [...] drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu0 drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu1 drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu2 drwxr-xr-x 2 root root 0 Aug 6 20:22 cpu3 drwxr-xr-x 2 root root 0 Aug 6 20:22 ddr1 drwxr-xr-x 2 root root 0 Aug 6 20:22 dpaa [...] drwxr-xr-x 2 root root 0 Aug 6 20:22 npc drwxr-xr-x 2 root root 0 Aug 6 20:22 nxc [...] root@p3041ds:~# ls -al /sys/kernel/debug/qoriq-dbg/cpu0 [...] -rw-rw-rw- 1 root root 0 Aug 6 20:22 dc1 -rw-rw-rw- 1 root root 0 Aug 6 20:22 dc2 -rw-rw-rw- 1 root root 0 Aug 6 20:22 dc4 --w--w--w- 1 root root 0 Aug 6 20:22 ddam [...] 31
  • State of Nexus in Linux ● Nexus decoder availability – ● Released as part of the babeltrace project Open-source software to reconstruct program flow not yet developed – Integrate such functionality in an IDE ? perf? 32
  • LTTng & Hardware tracing  33
  • Project overview   34
  • Tracers ● ● lttng-modules: kernel tracer module, compatible with kernels from 2.6.38* to 3.11, lttng-ust: user-space tracer, in-process library. * Kernel tracing is now possible on 2.6.32 to 2.6.37 by backport of 3 Linux Kernel patches [1]. 35
  • Utilities  ● lttng-tools: cli utilities and daemons for trace control, – lttng: cli utility for tracing control, – lttng-sessiond: tracing registry daemon, – lttng-consumerd: consume trace data, – lttng-relayd: network streaming daemon. 36
  • Viewers  ● ● ● babeltrace: cli text viewer, trace converter, plugin system, lttngtop: ncurse top-like viewer, Eclipse lttng plugin: front-end for lttng, collect, visualize and analyze traces, highly extensible. 37
  • Hardware tracing support ● Initial attempt to support hardware tracing: – – ● Babeltrace Nexus to CTF converter [5] Goal: Leverage existing traces visualizer Encountered issues: – Traces are not self-contained, need sideband information – Internal trace buffer size limitation – Synchronization with other traces can be tricky 38
  • Hardware tracing support demo DEMO 39
  • Future work ● ● ● Decoder/Converter for ARM ETM? Control hardware tracing facilities with the lttngtools command-line? Custom views for hardware traces in Eclipse plugin 40
  • Conclusion ● Availability and usefulness of hardware tracing ● Initial support for self-hosted hardware tracing ● Common abstraction for hardware tracing in the Linux kernel? 41
  • Questions ? ?  www.efficios.com  lttng.org  lttng-dev@lists.lttng.org  @lttng_project 42
  • References [1] - http://www.arm.com/images/processor/Cortex-A9-osprey.jpg [2] - http://www.arm.com/images/CoreSight_Diagram_Tiny.jpg [3] - https://lkml.org/lkml/2012/12/19/331 [4] - http://www.rlocman.ru/i/Image/2010/06/24/2.jpg [5] - https://github.com/cbab/babeltrace/tree/nexus 43