- Undo Ltd is a company that developed UndoDB, a reversible debugger that allows debugging programs by rewinding execution. - UndoDB records non-deterministic events during execution to create an event log, allowing replay of execution in either direction. - When the event log is full, it can either rewind to an earlier point, increase the log size, or switch to circular mode.

1. Copyright Undo Ltd, 2014
Introduction to UndoDB
Greg Law
Co-founder and CEO, Undo Ltd

2. Copyright Undo Ltd, 2014
Undo Overview
•Founded in 2005, with 25 man years development in debug technology Founder : Greg Law, Ph.D , Acorn, Solarflare, Nexwave,etc Founder: Julian Smith, Ph.D, Acorn, E-14, Broadcom, etc
•Well funded with $1.25 million raised in round 3.
−Backed financially by Cambridge Angels and Jaan Tallinn, co-founder Skype
−Currently 11 full time staff
•Market Momentum
−Revenue and headcount doubled in 2012 and 2013.
−OEMs with ARM (DS5 v5.16) and Rogue Wave (Totalview)
−Selected as Gartner Cool Vendor in Application Development for 2014.
−Customers include : NASA, Lawrence Livermore Labs, Cadence Design Systems, Mentor Graphics, Synopsys Inc.

3. Copyright Undo Ltd, 2014
Debugging Involves Backwards Reasoning
“Reason back from the state of the crashed program to determine what could have caused this. Debugging involves backwards reasoning, like solving murder mysteries. Something impossible occurred, and the only solid information is that it really did occur. So we must think backwards from the result to discover the reasons.”
Kernighan & Pike's The Practice of Programming

4. Copyright Undo Ltd, 2014
Debugging Involves Backwards Reasoning
“Reason back from the state of the crashed program to determine what could have caused this. Debugging involves backwards reasoning, like solving murder mysteries. Something impossible occurred, and the only solid information is that it really did occur. So we must think backwards from the result to discover the reasons.” Kernighan & Pike's The Practice of Programming

5. Copyright Undo Ltd, 2014
Debugging Involves Backwards Reasoning
“Reason back from the state of the crashed program to determine what could have caused this. Debugging involves backwards reasoning, like solving murder mysteries. Something impossible occurred, and the only solid information is that it really did occur. So we must think backwards from the result to discover the reasons.”
Kernighan & Pike's The Practice of Programming
•Regular debuggers give you pause/continue

6. Copyright Undo Ltd, 2014
Debugging Involves Backwards Reasoning
“Reason back from the state of the crashed program to determine what could have caused this. Debugging involves backwards reasoning, like solving murder mysteries. Something impossible occurred, and the only solid information is that it really did occur. So we must think backwards from the result to discover the reasons.”
Kernighan & Pike's The Practice of Programming
•Regular debuggers give you pause/continue
•UndoDB gives you a rewind button, etc
•CCTV for your code

7. Copyright Undo Ltd, 2014
Reversible Debugging for Real Code
A holy-grail of tools research for decades.
UndoDB is the first and only reversible debugger that is effective on real-world, complex software
•Used on many of the world’s most complex software projects Scientific supercomputing clusters (NASA Ames, LLNL, etc) EDA, Enterprise (several fortune 500 customers)
High performance, scalable, for compiled code on Linux
"I found the idea of the product amazing and a boon to my productivity... I have already been able to fix a deadlock that was driving me crazy for a week in only ten minutes!”

8. Copyright Undo Ltd, 2014
Performance is key
Patented checkpoint + replay approach
•Exploit the natural determinism of computers
•Reduces time/space overheads by many orders of magnitude
•Highly optimised JIT binary translation
Benchmark
•Time to ‘gzip’ a 16MB* file (gzip is mostly CPU bound, with some IO) * GDB times extrapolated from 16K file
Native
UndoDB
GDB “process record”
Time
1.49s
2.61s (1.75x)
21 hours (>50,000x)
Space
N/A
17.8MB
63GB

9. Copyright Undo Ltd, 2014
The Event Log
All non-deterministic events (syscalls, signals, thread switches) stored in the Event Log
−In record mode: operations executed as normal, and results stored in event log
−In replay mode: operations not executed, but results synthesized from the contents of the event log
Replay is therefore completely deterministic
No interaction with outside world in replay mode

10. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs

11. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs

12. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs

13. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs

14. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs

15. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs

16. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs

17. Copyright Undo Ltd, 2014
undodb-gdb: error: UndoDB's event log is full, so no more history can be recorded.
The event log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:

18. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:
Go back in time
reverse-next

19. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:
Go back in time
reverse-next

20. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:
Go back in time, or
Increase its size
undodb-set-max-event-log-size 768M

21. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:
Go back in time, or
Increase its size, or
Select circular mode
undodb-set-event-log-mode circular

22. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:
Go back in time, or
Increase its size, or
Select circular mode
undodb-set-event-log-mode circular; continue

23. Copyright Undo Ltd, 2014
The event log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:
Go back in time, or
Increase its size, or
Select circular mode
undodb-gdb: error: UndoDB's event log is full...

24. Copyright Undo Ltd, 2014
The Event Log
Event log is a fixed size in memory
Fills up as your program runs
When it’s full:
Go back in time, or
Increase its size, or
Select circular mode

25. Copyright Undo Ltd, 2014
Snapshots

26. Copyright Undo Ltd, 2014
Snapshots

27. Copyright Undo Ltd, 2014
Snapshots

28. Copyright Undo Ltd, 2014
Snapshots

29. Copyright Undo Ltd, 2014
Threads

30. Copyright Undo Ltd, 2014
Threads

31. Copyright Undo Ltd, 2014
Threads
Scheduling is serialized, but preemptive
−Many race conditions can be reproduced
•But not all – e.g. missing or incorrect memory barrier
−Bigger impact on performance
Record: OS scheduler selects which thread runs next
Replay: UndoDB forces scheduling to mirror record

32. Copyright Undo Ltd, 2014
UndoDB Save/Load
Collaborate
Save a recording and send it to a colleague.
Load the recording onto another machine
Defer and prioritise
Capture now, debug later  Save a recording for analysis later.
Capture sporadic/intermittent bugs and save them for later.
Protect against losing your data
Resilient to power or system failures -- never reproduce a bug a second time.

33. Copyright Undo Ltd, 2014
Record In-The-Field Failures
Undo Flight Recorder is a new product to record failures at the customer site
A library you link into your code and ship to your customers
By default, library is dormant. C API allows your program to enable recording

34. Copyright Undo Ltd, 2014
Flight Recorder
Your Executable
Ship to customer

35. Copyright Undo Ltd, 2014
Flight Recorder
Your Executable
Ship to customer

36. Copyright Undo Ltd, 2014
Flight Recorder
Your Executable
undofr.so
Ship to customer

37. Copyright Undo Ltd, 2014
Flight Recorder
Your
Executable
undofr.so
Ship to customer
Back to you – load into UndoDB recording

38. Copyright Undo Ltd, 2014
Good Kinds of Bug for UndoDB
1.Long run times
2.Frequently-called functions
3.Intermittent bugs
4.Stack corruption
5.Memory leaks
6.Real-time network protocols
7.Race conditions
8.Data structure corruption
9.Dynamic code

39. Copyright Undo Ltd, 2014
Good Kinds of Bug for UndoDB
1.Long run times
2.Frequently-called functions
3.Intermittent bugs
4.Stack corruption
5.Memory leaks
6.Real-time network protocols
7.Race conditions
8.Data structure corruption
9.Dynamic code

40. Copyright Undo Ltd, 2014
Good Kinds of Bug for UndoDB
1.Long run times
2.Frequently-called functions
3.Intermittent bugs
4.Stack corruption
5.Memory leaks
6.Real-time network protocols
7.Race conditions
8.Data structure corruption
9.Dynamic code
Demo #1
(square roots)

41. Copyright Undo Ltd, 2014
Good Bugs for UndoDB
1.Long run times
2.Frequently-called functions
3.Intermittent bugs
4.Stack corruption
5.Memory leaks
6.Real-time network protocols
7.Race conditions
8.Data structure corruption
9.Dynamic code
Demo #2
(stack smash)

42. Copyright Undo Ltd, 2014
Features and benefits
Works on unmodified Linux x86 and ARM systems and programs
•No kernel patches or modules, or special hardware requirements
•No recompilation or special libraries for program being debugged
•No restrictions on application to debug: threads, signals, shared memory, etc
Greatest value is on the bugs where:
•Effects manifest themselves a long time after underlying bug
•Non-deterministic “once in a blue moon” failures
•Large, unfamiliar codebases (“how did that happen?”)
Full-featured debugger
•Attach, watchpoints, etc.
•Fits seamlessly into your existing workflow (gdb / Eclipse / Emacs, etc)

43. Copyright Undo Ltd, 2014
Competition Time
Get the code at: http://undo-software.com/bugmug
Can you find the bug?
#bugmug @undosoft
support@undo-software.com

44. Copyright Undo Ltd, 2014
Support
Questions or problems: support@undo-software.com
If you suspect a bug in UndoDB:
−Run with --undodb-asserts-1 on the command line, or set UNDODB_debug_level_internal=2 env var to enable internal logging/extra checking
−On UndoDB crash, files a la undodb_log.1234 will be created; send these as attachments to Undo
−Alternatively, use maint-undodb-dump-internal-log command to dump the logs immediately

45. Copyright Undo Ltd, 2014
Part II UndoDB master class
Tips and tricks for better debugging

46. Copyright Undo Ltd, 2014
Timeline
“What time is it?”, or “Goto” a particular time:
undodb-get-n
undodb-goto-n
undodb-show-event-log-extent
Precise control of timeline:
undodb-set-bookmark and
undodb-goto-bookmark
Tip: Use these commands to binary chop

47. Copyright Undo Ltd, 2014
Record Mode and Replay Mode
When you first run, you’re in Record Mode
If you step forwards, you stay in Record Mode
Issue a reverse operation puts you in Replay Mode

48. Copyright Undo Ltd, 2014
Record/replay continued
Replaying forwards to the end of the event log takes you back into record mode (debugger will stop)
Use undodb-goto-record-mode to jump straight to record mode
−As with undodb-goto-n, will not hit breakpoints
State (registers, memory) can be modified in record mode, but not in replay mode
User issued function-calls (e.g. ‘call’ command from gdb prompt) can be issued from replay mode or record mode (see below)

49. Copyright Undo Ltd, 2014
The Event Log
All “non-deterministic” inputs are recorded
−System-calls
−Thread-switches
−Signals
−Non-deterministic instructions
−Shared memory reads (incl. DMA)
List events
maint-undodb-show-events b,e
maint-undodb-show-eventstats b,e

50. Copyright Undo Ltd, 2014
More on the Event Log
Increase the event-log size dynamically
undodb-set-max-event-log-size
Stop when the event log gets full (default)
undodb-event-log-mode straight
Record forever with a circular event log
undodb-event-log-mode circular

51. Copyright Undo Ltd, 2014
Performance tuning
Start recording later:
−Start with --undodb-defer-recording option
−Then when you’re ready: undodb-enable-record
−Note: you cannot go back to record recording was enabled
Overview of performance metrics
−maint-undodb-show-exec-summary
−Make sure your instrumentation heap is big enough:
--undodb-instr-heapsize 512M
−Look out for executing code in writable memory
approx 2x slower than executing for read-only memory

52. Copyright Undo Ltd, 2014
Snapshots
List existing snapshots
maint-undodb-show-snapshots
Beware memory consumption
e.g. --undodb-snapshots 5
Note that reverse opterations may be slower
Hint to create a snapshot soon
maint-undodb-snapshot
(will create a snapshot at next basic-block boundary)

53. Copyright Undo Ltd, 2014
More tips and tricks
No symbols/no backtrace?
reverse-stepi or reverse-finish
undodb-goto-n -1000
Set default command-line options
~/.undodb-gdb.defaults
Calling functions in debuggee/inferior
−Side-effects are discarded after function is run
Function is run in a temporary process
−Use our `infcall’ gdb patch to make this bullet-proof
e.g. break foo.c:42 if !memcmp( a, b, 32)

54. Copyright Undo Ltd, 2014
More tips and tricks
Autotrace feature
Attach to an executable by name rather than pid
--undodb-autotrace /path/to/myexe cmd
cmd is the command as executed; if it or a child process exec’s /path/to/myexec then undodb will attach to that
The path must be complete path or a glob
Work differently
−Don’t try to anticipate where to put breakpoints; run until the bug happens, then “think backwards”