TimelapseInteractive record/replay for the web   Brian Burg      Andrew J. Ko         Michael D. Ernst           Computer ...
Reproducing program behavior is       essential in software development   Demonstration:          interacting with a progr...
Task coupling can be hazardous   Example: debugging a color picker widget
Barriers to reproducing behaviorDemonstration• Tedious, time-consuming, and manual• Difficult to explain repro. steps (e.g...
Simplifying by decouplingDecoupling behavior demonstration from investigation  simplifies reproduction and enables new pra...
Solution: deterministic record/replay                                 User-space librariesOld idea: achieve identical     ...
Timelapse: record/replay for the web    Timelapse is an infrastructure and developer tool to   record, replay, and investi...
Demonstrating and Investigating              (with Timelapse)       1.   Begin recording       2.   Demonstrate behavior  ...
Recreating a specific execution Executions differ because of sources of nondeterminism.To eliminate nondeterminism,  • Ide...
Programs and InputsAn execution is determined by a program and its inputs.                                                ...
Programs and InputsAn execution is determined by a program and its inputs.                                                ...
Programs and InputsAn execution is determined by a program and its inputs.                                                ...
Interpreters and InputsIn an interpreter session, the program is the interpreter itself.                                  ...
Web browsers are interpretersWeb browsers interpret JavaScript, CSS, HTML, and user input.      Input            Web Inter...
Web browsers are interpretersExecutions can be reproduced by capturing and reusing inputs.                          Inputs...
Record/replay goalsPrecise deterministic record/replay  • Must capture and reuse explicit and implicit inputs  • Able to d...
Record/replay technique: proxying    During normal execution, Date.now() returns the current time.                        ...
Record/replay technique: proxying       During recording, the return value of Date.now() is saved.                        ...
Record/replay technique: proxying        On replay, the logged return value of Date.now() is used.                        ...
Shim: the thing in the middle   Shims are used to implement deterministic record/replay.The hard part of implementing reco...
Background: browser architectures     Web interpreters are embeddable and cross-platform.              User input         ...
Shim placement in the browser stack Shims are placed at the API boundaries of the web interpreter.Embedders               ...
Comparison: shim placementFor record/replay, VM only needs shim between OS/hardware
Why not place shims elsewhere?At the OS or VM level (VMWare Replay) Low overhead and exact replay Can’t integrate with d...
Interpreter inputs by source         User: mouse, keyboard, scroll, resize         Network: images, scripts, HTML, AJAX   ...
“Pull” vs. “Push” inputsPULL FROM                       PUSHresource data,                    TO                          ...
Comparison: “Pull” vs. “Push”           RTDSC,       interrupts             etc
Mechanics of execution replayInjecting “push” inputs causes dispatch of DOM event(s).                     1. mousedow     ...
Comparison: replay timingVMs and Timelapse share notions of counting and injecting.Problem: when to inject “push” inputs t...
Challenge: finding and taming nondeterminism   Taming nondeterminism is easier with principled design.Video games explicit...
Challenge: dealing with real web platformsWeb interpreters are complex, many-limbed, and fast-changing.• >1 MLOC (mainly C...
Lesson: consider many, implement oneFor research, only address essential architectural complexity. It’s fine to choose one...
Interacting with a recordingControlling video on YouTube   The Omniscient Debugger for Java
Interacting with Whyline [Ko & Myers]Whyline for Java: program output   Whyline for Java: program output
Interacting with TimelapseTimelapse Panel inside Web Inspector   Application being replayed
Timeline visualizationInfoVis mantra: overview, zoom and filter, details-on-demand.                     Time ruler        ...
Timeline visualizationInfoVis mantra: overview, zoom and filter, details-on-demand.                                       ...
Table of program inputsInfoVis mantra: overview, zoom and filter, details-on-demand.                         Inputs by id ...
Recording, replaying, and modality• Recording can be started/stopped from any Inspector panel• Replay is controlled from t...
Slider control and modality• Timeline and table are linked by sliders and filters.• Dragging slider seeks replay to the dr...
How do developers use Timelapse?  Actual use cases? Good affordances? Workflow integration?Formative user study• Subjects:...
Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
Study Results No significant effect on task completion time. Why? • Timelapse didn’t affect types of information sought • ...
Future Work“Canned” executions can bring research ideas into practice.Dynamic analysison-demand race detection, code cover...
Future Work“Canned” executions can bring research ideas into practice.Dynamic analysison-demand race detection, code cover...
On-demand dynamic analysisDynamic analyses generate data at runtime with instrumentation.      Instrumented   Generated   ...
On-demand dynamic analysisRecord/replay enables on-demand dynamic analysis tools. Normal     Timelapse   Developer   Instr...
ConclusionTimelapse decouples demonstration and investigationusing ideas from virtual machine record/replay work.The Timel...
CALL FOR BACKUP
Network: a real proxy, not a shim   An HTTP proxy records and replays network traffic.This is a hack.In the short-term, ea...
Replay fidelity and completenessDivergence detection supports piecewise implementation.Web interpreters expose a large and...
Performance characteristicsPerformance hasn’t been a priority, but is adequate anyway.Record and replay slowdowns are negl...
Embedding and platform APIs  Abstraction layers separate web interpreters from platforms/embedders.                  EMBED...
Embedding and platform APIs  Abstraction layers separate web interpreters from platforms/embedders.                  EMBED...
Embedding and platform APIs Shims sit between the web interpreter and abstraction layers.               EMBEDDING API     ...
Embedding and platform APIs Shims sit between the web interpreter and abstraction layers.               EMBEDDING API     ...
Shim behavior when recording    EMBEDDING API                                 PLATFORM API                       Start    ...
Shim behavior when recording       EMBEDDING API                                    PLATFORM API[user mouse press]-> (IPC ...
Shim behavior when replaying    EMBEDDING API                                 PLATFORM API                      Start     ...
Shim behavior when replaying      EMBEDDING API                                   PLATFORM API DeterminismController::disp...
Upcoming SlideShare
Loading in …5
×

Timelapse: interactive record/replay for the web

2,395 views

Published on

These are the slides for a series of work-in-progress talks given about Timelapse in June 2012. Please send comments and questions to @brrian on twitter, or find my email on the Timelapse project page:

http://cs.washington.edu/homes/burg/timelapse/

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,395
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
22
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Show the color picker widget.A user has reported that moving one of the RGB sliders up and down repeatedly can cause the others to change values.Try to reproduce this.Once it’s been reproduced, try to find the event handler for mousemoveSet a breakpoint there, then try to reproduce.Oh, it won’t work. I guess I’ll have to simulate.. Or insert logging.How would I describe this bug any more concisely than the user? Test it?
  • Demo1: create a repro of the bug with two distinct repro interactions w/ the slider.Demo2: Real-time playback, past the second interaction.Seek to the beginning of second interaction.See where we are in the repro.Disable timer category, scope to just this interaction.Find a breakpoint.Link scope to breakpoints.Use breakpoint to binary search until see the change.Step into
  • TODO Need better connection of goals to strategies
  • For Timelapse, the browser and platform need to appear deterministic. For VMWare, only the platform needs to appear deterministic.
  • I will say that the boxes are things that must execute at the same time. Rerunning an analysis on new data means re-demonstrating the behavior.
  • With a pipeline approach, one recording is sufficient for any number of different sets of instrumentation, as needed.
  • This slide might be backup, or combined with fidelity slide, or need inspiration.
  • Timelapse: interactive record/replay for the web

    1. 1. TimelapseInteractive record/replay for the web Brian Burg Andrew J. Ko Michael D. Ernst Computer Science and Engineering University of Washington
    2. 2. Reproducing program behavior is essential in software development Demonstration: interacting with a program to achieve a behavior Investigation: interacting with developer tools to understand behaviorThe phases of demonstrating and investigating are inseparable.
    3. 3. Task coupling can be hazardous Example: debugging a color picker widget
    4. 4. Barriers to reproducing behaviorDemonstration• Tedious, time-consuming, and manual• Difficult to explain repro. steps (e.g., in a bug report)Investigation• Repeated demonstration to investigate same behavior• The behavior being investigated is not held constant• Interleaving the two phases incurs task switching penalties
    5. 5. Simplifying by decouplingDecoupling behavior demonstration from investigation simplifies reproduction and enables new practices. Behavior could be demonstrated, and later, investigated, by different people. Behavior could be reproduced precisely, automatically, quickly, and easily. The means of investigation could change independently without re-demonstration. http://dryicons.com, http://www.getfirebug.com/
    6. 6. Solution: deterministic record/replay User-space librariesOld idea: achieve identical Jockey (C), Mugshot (JavaScript)execution by controllingnondeterminism. Application-specific Video game enginesRecent idea: modern Operating systems dOS, Determinatorbrowsers are virtual machines. Virtual machines ReVirt, LEAP, VMWare WorkstationNew idea: adapt virtualmachine record/replaytechniques to web browsers Hardware DMP, RCDC, Calvin
    7. 7. Timelapse: record/replay for the web Timelapse is an infrastructure and developer tool to record, replay, and investigate web application behavior.Contributions• Adaptation of VM record/replay techniques to the web• Infrastructure and tool for interactive record/replay• Design concepts for visualizing, interacting with recordings
    8. 8. Demonstrating and Investigating (with Timelapse) 1. Begin recording 2. Demonstrate behavior 3. End recording 4. Investigate as needed Example: color picker, revisited
    9. 9. Recreating a specific execution Executions differ because of sources of nondeterminism.To eliminate nondeterminism, • Identify what is/isn’t deterministic • Decide what sources to capture, and how • Figure out how to replay the recorded nondeterminismTimelapse adapts definitions and techniques from VM research.
    10. 10. Programs and InputsAn execution is determined by a program and its inputs. main.js function foo(a,b) { return a + b; } function bar(c) { return c * Math.random(); } foo(argv[1], argv[2]); main.js bar(argv[1]); bar(argv[3]); main.js
    11. 11. Programs and InputsAn execution is determined by a program and its inputs. main.js function foo(a,b) { return a + b; } function bar(c) { return c * Math.random(); } foo(argv[1], argv[2]); main.js bar(argv[1]); bar(argv[3]); main.js
    12. 12. Programs and InputsAn execution is determined by a program and its inputs. main.js function foo(a,b) { return a + b; } function bar(c) { return c * Math.random(); } foo(argv[1], argv[2]); bar(argv[1]); bar(argv[3]); Sources of nondeterminism that affect the course of execution are implicit inputs to the program.
    13. 13. Interpreters and InputsIn an interpreter session, the program is the interpreter itself. The session could be reproduced>> load(“defs.js”); REPL session exactly by reusing inputs.>> foo(1,2);>> foo(3,4);>> bar(5); 1. load defs.js interpreter>> bar(5); inputs 2. CALL foo 1, 2 3. CALL foo 3, 4 defs.js 4. CALL bar, 5>> foo(a,b) { return a + b; } 5. RET Math.random, 0.453863>> bar(c) { return c * Math.random(); } 6. CALL bar, 5 7. RET Math.random, 0.986364
    14. 14. Web browsers are interpretersWeb browsers interpret JavaScript, CSS, HTML, and user input. Input Web Interpreter Output
    15. 15. Web browsers are interpretersExecutions can be reproduced by capturing and reusing inputs. Inputs Log Input Web Interpreter Output
    16. 16. Record/replay goalsPrecise deterministic record/replay • Must capture and reuse explicit and implicit inputs • Able to detect divergence of replay from recordingSelf-contained, sharable recordings • Must not depend on any machine-specific context • Replay does not require external communicationIntegration with existing architecture and tools • Record/replay must have low performance overhead • Controllable and compatible with existing developer tools
    17. 17. Record/replay technique: proxying During normal execution, Date.now() returns the current time. /* file: Source/wtf/DateMath.h */oo(a,b) { return a + b; } inline double jsCurrentTime() {(c) { return “now:”+ Date.now(); } return floor(WTF::currentTimeMS()); }
    18. 18. Record/replay technique: proxying During recording, the return value of Date.now() is saved. /* file: Source/wtf/DateMath.h */oo(a,b) { return a + b; } inline double jsCurrentTime() {(c) { return “now:”+ Date.now(); } return floor(WTF::currentTimeMS()); } Inputs log
    19. 19. Record/replay technique: proxying On replay, the logged return value of Date.now() is used. /* file: Source/wtf/DateMath.h */oo(a,b) { return a + b; } inline double jsCurrentTime() {(c) { return “now:”+ Date.now(); } return floor(WTF::currentTimeMS()); } Inputs log
    20. 20. Shim: the thing in the middle Shims are used to implement deterministic record/replay.The hard part of implementing record/replay is designing and placing shims.
    21. 21. Background: browser architectures Web interpreters are embeddable and cross-platform. User input Date.now, setTimeout Callbacks, visual timer fire outputEmbedders Web Interpreter Platforms (WebKit, Gecko)
    22. 22. Shim placement in the browser stack Shims are placed at the API boundaries of the web interpreter.Embedders Web Interpreter Platforms (WebKit, Gecko)
    23. 23. Comparison: shim placementFor record/replay, VM only needs shim between OS/hardware
    24. 24. Why not place shims elsewhere?At the OS or VM level (VMWare Replay) Low overhead and exact replay Can’t integrate with developer toolsAt the JS library level (Mugshot: Mickens et al., 2010) Cross-platform Slower, lower fidelity, can’t integrate with toolsAt the level of DOM events only (WaRR: Adrica et al., 2011) Most DOM events are deterministic and/or have no effect Skipping pre-dispatch browser code causes strange behavior
    25. 25. Interpreter inputs by source User: mouse, keyboard, scroll, resize Network: images, scripts, HTML, AJAX Commands: navigate, open, close page Internal nondeterminism: Animations, transitions, multimedia, async. actions (load, error, popstate) Functions: Date.now, Math.random Caching: resources, cookies Timers: timer interrupt schedule
    26. 26. “Pull” vs. “Push” inputsPULL FROM PUSHresource data, TO mouse, keyboard,client API hooks, scroll, navigate,input handling network events Animation, transition, load, error, media eventstimestamp,random, cookies, timer callbackspersistent state
    27. 27. Comparison: “Pull” vs. “Push” RTDSC, interrupts etc
    28. 28. Mechanics of execution replayInjecting “push” inputs causes dispatch of DOM event(s). 1. mousedow function onClicked(..) n { ... = Date.now(); 2. mouseup } 3. click 4. focus 5. submit DOM Event DIspatch JavaScript execution
    29. 29. Comparison: replay timingVMs and Timelapse share notions of counting and injecting.Problem: when to inject “push” inputs to cause same execution VM Browser record/replay record/replay• Count branches, • Count DOM event instructions dispatches• Inject hardware • Inject “push” inputs interrupts
    30. 30. Challenge: finding and taming nondeterminism Taming nondeterminism is easier with principled design.Video games explicitly design to arecord/replay-like architecture.Taming nondeterminism withshims relies on well-defined APIs.Taming nondeterminism post-hocrequires refactoring and hacks. Networked games like Starcraft 2 rely on record/replay facilities.
    31. 31. Challenge: dealing with real web platformsWeb interpreters are complex, many-limbed, and fast-changing.• >1 MLOC (mainly C++/ JavaScript)• Hundreds of committers, and 500- 1000 commits per week• 6 independent build systems• Supports many OS, CPU, GUI toolkits, graphics engines, toggleable features, embedders http://hawanja.deviantart.com/
    32. 32. Lesson: consider many, implement oneFor research, only address essential architectural complexity. It’s fine to choose one reasonable configuration. But, don’t ignore essential complexity. Platform-specific examples: • input method editor handling • rendering and animation • resource caching, cookies Embedder-specific examples: • JavaScript engine • Embedding API usage http://commons.wikimedia.org
    33. 33. Interacting with a recordingControlling video on YouTube The Omniscient Debugger for Java
    34. 34. Interacting with Whyline [Ko & Myers]Whyline for Java: program output Whyline for Java: program output
    35. 35. Interacting with TimelapseTimelapse Panel inside Web Inspector Application being replayed
    36. 36. Timeline visualizationInfoVis mantra: overview, zoom and filter, details-on-demand. Time ruler Program inputs Filter inputs over time, by by type type
    37. 37. Timeline visualizationInfoVis mantra: overview, zoom and filter, details-on-demand. Filter inputs by time span
    38. 38. Table of program inputsInfoVis mantra: overview, zoom and filter, details-on-demand. Inputs by id Input types and time by color Table is adjustable Panel- Global record specific controls controls
    39. 39. Recording, replaying, and modality• Recording can be started/stopped from any Inspector panel• Replay is controlled from the Timelapse panel only• Modality displayed next to record/replay controls
    40. 40. Slider control and modality• Timeline and table are linked by sliders and filters.• Dragging slider seeks replay to the dropped position. Sliders show same position Dragging induces drop previews
    41. 41. How do developers use Timelapse? Actual use cases? Good affordances? Workflow integration?Formative user study• Subjects: 9+3 professional web developers• Tasks: create a test case, fix a bug• Time: 45 min/task, + tutorial & exit interview• How: within-subjects design, Camtasia, Timelapse
    42. 42. Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
    43. 43. Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
    44. 44. Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
    45. 45. Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
    46. 46. Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
    47. 47. Important use casesPlay, watch and pauseVisual binary searchStepping through inputsExecution anchoringRefining repro. steps
    48. 48. Study Results No significant effect on task completion time. Why? • Timelapse didn’t affect types of information sought • “Program inputs” not related to existing concepts • New strategies only possible with tight integrationInteractive record/replay mainly* accelerates existing strategies.
    49. 49. Future Work“Canned” executions can bring research ideas into practice.Dynamic analysison-demand race detection, code coverage, optimizationsProgram synthesiscreate automated tests from recorded interactionsCorpus analysistrain models, test type systems, or evaluate analyses
    50. 50. Future Work“Canned” executions can bring research ideas into practice.Dynamic analysison-demand race detection, code coverage, optimizationsProgram synthesiscreate automated tests from recorded interactionsCorpus analysistrain models, test type systems, or evaluate analyses
    51. 51. On-demand dynamic analysisDynamic analyses generate data at runtime with instrumentation. Instrumented Generated Dynamic Analysis execution data analysis Output
    52. 52. On-demand dynamic analysisRecord/replay enables on-demand dynamic analysis tools. Normal Timelapse Developer Instrumented Data andexecution recording tools execution analysis
    53. 53. ConclusionTimelapse decouples demonstration and investigationusing ideas from virtual machine record/replay work.The Timelapse tool supports interactive inspection.Timelapse’s infrastructure is useful for other purposes.The Timelapse research prototype is an open-source forkof WebKit, and is freely available from the project website. https://bitbucket.org/burg/timelapse/
    54. 54. CALL FOR BACKUP
    55. 55. Network: a real proxy, not a shim An HTTP proxy records and replays network traffic.This is a hack.In the short-term, easier than attempting to recreateplatform-dependent network streams/handlesLong-term solution:A resource loader implementation with built-inrecord/replay primitives, or a higher-level network API
    56. 56. Replay fidelity and completenessDivergence detection supports piecewise implementation.Web interpreters expose a large and ever-changing API.Timelapse doesn’t tame all sources of nondeterminism.Excepting untamed sources, the DOM tree and JavaScript heapare identical for all recorded and replayed executions.Divergence is automatically detected by comparing DOM events.
    57. 57. Performance characteristicsPerformance hasn’t been a priority, but is adequate anyway.Record and replay slowdowns are negligible becauseshims are placed along existing architectural boundaries.Timelapse is ideal for CPU- and user-bound programs.Network-bound applications are slower, due to prototypeshortcuts (disabling caching and pipelining).Replay/seek speed could be improved dramatically bycheckpointing and clipping unnecessary layout/renders.
    58. 58. Embedding and platform APIs Abstraction layers separate web interpreters from platforms/embedders. EMBEDDING API PLATFORM APIEmbedders Web Interpreter Platforms (WebKit, Gecko)
    59. 59. Embedding and platform APIs Abstraction layers separate web interpreters from platforms/embedders. EMBEDDING API PLATFORM APIEmbedders Web Interpreter Platforms (WebKit, Gecko)
    60. 60. Embedding and platform APIs Shims sit between the web interpreter and abstraction layers. EMBEDDING API PLATFORM APIEmbedders Web Interpreter Platforms (WebKit, Gecko)
    61. 61. Embedding and platform APIs Shims sit between the web interpreter and abstraction layers. EMBEDDING API PLATFORM APIEmbedders Web Interpreter Platforms (WebKit, Gecko)
    62. 62. Shim behavior when recording EMBEDDING API PLATFORM API Start recording!
    63. 63. Shim behavior when recording EMBEDDING API PLATFORM API[user mouse press]-> (IPC communication)-> WebKit::WebPage::mouseEvent()-> WebCore::InputProxy::handleMousePress()---> WebCore::DeterminismController::capture()-> WebCore::EventHandler::handleMousePress()-> WebCore::Node::dispatchMouseEvent()
    64. 64. Shim behavior when replaying EMBEDDING API PLATFORM API Start replaying!
    65. 65. Shim behavior when replaying EMBEDDING API PLATFORM API DeterminismController::dispatchNextAction() -> MousePressAction::dispatch() -> InputProxy::handleMousePress(true) -> EventHandler::handleMousePress() -> Node::dispatchMouseEvent() -> WebKit::WebPage::mouseEvent() -> InputProxy::handleMousePress(false)

    ×