Performance Metrics in a Day with Selenium
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Performance Metrics in a Day with Selenium

on

  • 12,047 views

#seconf

#seconf

Statistics

Views

Total Views
12,047
Views on SlideShare
12,044
Embed Views
3

Actions

Likes
14
Downloads
178
Comments
0

3 Embeds 3

http://www.slashdocs.com 1
http://www.docseek.net 1
http://www.docshut.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Welcome to our talk about getting performance metrics in a day
  • Outline the talk, what is the takeawayRun your standard unit tests (Junit, minitest, ….) but put in performance testing as well
  • Becoming the standard mechanism to store/dump performance data
  • Lots of text on this slide, we’ll keep it more entetaining laterOverall page load time= The obvious starting point and usually the easiest to acquire
  • Setting your own timings – obvious, but is not always very accurate if you are in a sequence of steps and capture page transitions and other delays Very coarse, incorporates latency/setup/teardown delays in the times reported, e.g. it sometimes takes a second or 2 for an IE window to become responsiveWeb Timings == Very simple at the moment, but gives very accurate overall stats, like initial connection time, DOM ready, and total request->response times.Browser plugins == Accurate picture of what the browser is doing, but not so straightforward to incorporate into your code.Proxy == Gives you very good timings for http traffic but can’t give you timings like render time or DOM ready events.Network traffic capture tools like WinPCAP, or wireshark
  • Doesn't seem to work currently in FF4Will have to parse the HAR format if you just want the basic timing out of it
  • Going to go over:- How to grab metrics using a proxy How using a proxy can be beneficial for testing in general
  • (Fiddler refers to thehttp transaction as 'sessions')
  • Documentation spread out over blogs, fiddler google groups, and fiddler extensions pageAPI is little difficult

Performance Metrics in a Day with Selenium Presentation Transcript

  • 1. Performance Metrics In A Day
    Shane Hender
    Mark Watson
  • 2. Agenda
    Goal is to show how you can enhance your automation tests to gather performance metrics
    Keep the short presentation as practical as possible with enough code snippets to give you a starting point
  • 3. One more thing…
    HTTP Archive (HAR)
    What?
    UTF8 Text based JSON encoding to represent performance data
    Why?
    Many libraries to convert from JSON to objects
    More tools allowing import/export/view HAR
    Becoming the standard format
  • 4. Snippet of HAR format
    {
    "log": {
    "version": "1.1",
    "creator": {
    "name": "Firebug",
    "version": "1.7.0"
    },
    "browser": {
    "name": "Firefox",
    "version": "4.0"
    },
    "pages": [
    {
    "startedDateTime": "2011-03-31T16:56:50.455-07:00",
    "id": "page_62901",
    "title": "Google",
    "pageTimings": {
    "onContentLoad": 431,
    "onLoad": 3148
    }
    }
    ],
    "entries": [
    {
    "pageref": "page_62901",
    "startedDateTime": "2011-03-31T16:56:50.455-07:00",
    "time": 250,
    "request": {
    "method": "GET",
    "url": "http://www.google.com/",
    "httpVersion": "HTTP/1.1",
    "cookies": [
    {
    "name": "PREF",
    "value": "ID"
    },
  • 5. HAR Viewers
    http://www.softwareishard.com/har/viewer/
    Fiddler UI can import HAR files
    ShowSlowwebapp can be used to archive and view HAR files
  • 6. Which Metrics?
    Overall page load time
    DOM loading/interactive/complete, browser 1st render, …
    Per-item timings
    DNS, SSL connect, time to first byte, total time to download bytes, …
    Headers, status codes, and content
    For a more detailed picture of the reasons for the page performance
    Can help with debugging performance, e.g. were items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering
  • 7. Methods for gathering metrics
    Setting your own timings in test code
    Using the new ‘Navigation.Timings’ or Web Timings standard built into browsers
    Using browser plugins that report back the timings to you, e.g. WebPageTest
    Per HTTP request logs via Selenium’s built-in proxy
    Routing the browser traffic through a local proxy and gathering the statistics from there.
    Network traffic capture
  • 8. Method 1: Own Timings
    WebDriver driver = newChromeDriver();
    Date start = new Date();
    driver.get("http://www.webmetrics.com");
    Date end = new Date();
  • 9. Method 2: Web Timings
    Currently Chrome and IE9 supported, coming soon for Firefox
    http://w3c-test.org/webperf/specs/NavigationTiming/
    WebDriver driver = newChromeDriver();
    // A "base url", used by selenium to resolve relative URLs
    StringbaseUrl = "http://www.webmetrics.com";
    driver.get(baseUrl);
    JavascriptExecutorjs = (JavascriptExecutor)driver;
    LongloadEventEnd= (Long) js.executeScript("return
    window.performance.timing.loadEventEnd");
    LongnavigationStart= (Long) js.executeScript("returnwindow.performance.timing.navigationStart");
    Long answerToAllProblems =loadEventEnd - navigationStart;
  • 10. Method 2: Web Timings
    Could also modify this by using your own JavaScript loaded into the page to set timings
    Also would work in combination with the already set Web Timings
    E.g. Set a Date().getTime() variable when some dynamically loaded content is loaded into the DOM
    LongperceivedLoadTime = (Long) js.executeScript("return
    elementInsertTime –
    window.performance.timing.navigationStart");
  • 11. Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....
  • 12. Method 3: Browser Plugins
    Typically can give compute time metrics
    Javascript execution time
    CPU usage
    Render times
    IE8 - WebPageTest
    CPU usage
    Video capture
    Firefox - Firebug Net Panel + NetExport
    https://github.com/lightbody/browsermob-page-perf
    https://github.com/AutomatedTester/AutomatedTester.PagePerf.git
    DynaTrace browser plugin (FireFox/IE on Windows)
    Good breakdown of stats (Javascript execution times, CSS times, render, 'first impression' time).
    http://ajax.dynatrace.com/ajax/en/
    Chrome – some export from the Developer Tools Network Panel?
  • 13. FirefoxProfilep=newFirefoxProfile();
    try{
    p.addExtension(newFile("c:/firebug-1.6.0.xpi"));
    p.addExtension(newFile("c:/netExport-0.8b9.xpi"));
    p.addExtension(newFile("c:/firestarter-0.1.a5.xpi"));
    }catch(IOExceptione){
    thrownewRuntimeException(“Failed to load extensions:",e);
    }
    p.setPreference("extensions.firebug.netexport.autoExportActive",true);
    //p.setPreference("extensions.firebug.netexport.defaultLogDir", "c:/");
    p.setPreference("extensions.firebug.onByDefault",true);
    p.setPreference("extensions.firebug.defaultPanelName","net");
    p.setPreference("extensions.firebug.net.enableSites",true);
    p.setPreference("extensions.firebug.previousPlacement",1);
    driver=newFirefoxDriver(p);
    Method 3: Example - Firebug + NetExport
  • 14. Method 4: Example - Firebug + NetExport
  • 15. Method 4: Use a Proxy
    Use built in Selenium 1 proxy
    Use a 3rd party proxy library
    Many available, few capture metrics in a convenient way
    Two good ones:
    BrowserMob Proxy
    Fiddler
  • 16. Method 4: Selenium built-in Proxy
    Use API Selenium.captureNetworkTraffic()
    Selenium selenium = newDefaultSelenium("localhost", 4444,
    "*firefox", "http://www.webmetrics.com"); selenium.start("captureNetworkTraffic=true");
    selenium.open("http://www.webmetrics.com");
    Stringjson = selenium.captureNetworkTraffic("json");
    selenium.stop();
  • 17. Method 4: Selenium built-in Proxy
    Not in HTTP Archive (HAR) format, but does report:
    HTTP status code, URL and HTTP method
    Request & response headers
    Overall request->response timing
    Bytes downloaded
    Works with Chrome and Firefox
    Doesn’t work for WebDriver tests.
    Still work with Selenium 2 RC, but only if send jobs via Selenese API.
    Not much control over the proxy
  • 18. Method 4: Advantages of using a Proxy
    Testing benefits beyond performance monitoring
    Blacklisting/whitelisting URLs
    Comparing page load times of site with/without external content
    Not hitting analytics/advertising content
    Simulating external content being offline
    URL rewrites
    Redirect production URLs back to staging/test environment
    Pretend to be production as far as the browser is concerned (e.g. for cookies)
    Make sure ad/metrics requests are being made
  • 19. Method 4: Advantages of using a Proxy
    Header changes
    Set the user agent manually to test different browser behavior
    Auto authentication
    Wait until all content is downloaded
    HTTP idle for X seconds
    Limit bandwidth
  • 20. Method 4: BrowserMob Proxy
    Open source cross platform proxy written in Java.
    HTTP Archive support
    Friendly API
    Source code available:
    https://github.com/lightbody/browsermob-proxy
  • 21. Method 4: BrowserMob Proxy
    Export HAR Sample
    import org.browsermob.proxy.ProxyServer;
    ProxyServer proxy = new ProxyServer(9090);
    proxy.start();
    Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090");
    DesiredCapabilities caps = new DesiredCapabilities();
    caps.setCapability("proxy", mobProxy);
    FirefoxDriver driver = new FirefoxDriver(caps);
  • 22. Method 4: BrowserMob Proxy
    Run test, denoting pages in HAR
    proxy.newHar("Yahoo");
    driver.get("http://yahoo.com");
    proxy.endPage();
    proxy.newPage("CNN");
    driver.get("http://cnn.com");
    proxy.endPage();
  • 23. Method 4: BrowserMob Proxy
    Write out HTTP Archive file
    proxy.getHar().writeTo(newFile("test.har"));
  • 24. Method 4: BrowserMob Proxy
    • Blacklist/Whitelist URLs
    proxy.blacklistRequests(regex, responseCode)
    proxy.whitelistRequests(regex, responseCode)
    Redirecting URLs
    proxy.rewriteUrl(regex, replace)
    • Limit Bandwidth
    proxy.setDownstreamKbps(kbps)
    proxy.setUpstreamKbps(kbps)
  • 25. Method 4: BrowserMob Proxy
    When to use
    Cross platform
    Java
    Can set the browser proxy
  • 26. Method 4: Proxy - FiddlerCore
    Fiddler is an application for viewing HTTP traffic on Windows.
    Works with Chrome, FireFox, Internet Explorer.
    Fiddler Application is built on FiddlerCore
    .NET library
    Allows extensive programmatic control over the proxy
    Captures HTTP timings
    Allows request/responses to be intercepted and modified
    Configures itself as default Windows proxy (supports proxy chaining)
    Can decrypt SSL traffic
  • 27. To start the proxy and register it as the default system wide proxy.
    Each HTTP transaction can then be recorded as follows:
    Method 4: Proxy - FiddlerCore
    // Initialize the proxy
    Fiddler.FiddlerApplication.Startup(
    8877, FiddlerCoreStartupFlags.Default);
    var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.SessionoS)
    {
    items.Add(oS);
    };
  • 28. Method 4: Proxy - FiddlerCore
    Run Selenium test as normal
    As each item is downloaded it will be added to the ’items’ var in previous slide.
    string baseUrl = "http://www.webmetrics.com";
    varwebDriver =
    newOpenQA.Selenium.Firefox.FirefoxDriver();
    var selenium =
    newSelenium.WebDriverBackedSelenium
    (webDriver, baseUrl);
    selenium.Start();
    selenium.Open(baseUrl);
    selenium.WaitForPageToLoad("30000");
    selenium.Stop();
  • 29. Method 4: Proxy (Fiddler -> HAR)
    Using the HAR Exporter to convert the sessions to HAR
    Add FiddlerCore-BasicFormats.dll to the project references and load the assembly:
    https://www.fiddler2.com/dl/FiddlerCore-BasicFormats.zip
    String exePath = Assembly.GetExecutingAssembly().Location;
    String path = Path.Combine(
    Path.GetDirectoryName(exePath),
    @"FiddlerCore-BasicFormats.dll");
    FiddlerApplication.oTranscoders.ImportTranscoders(path);
  • 30. Method 4: Proxy (Fiddler -> HAR)
    Finally, export the HAR to a file:
    varoExportOptions = new Dictionary<string, object>();
    string filename = @"output.har”;
    oExportOptions.Add("Filename", filename);
    Fiddler.FiddlerApplication.DoExport(
    "HTTPArchive v1.2", sessions, oExportOptions, null);
  • 31. Method 4: Proxy Bonus Features
    Modifying HTTP requests
    Fiddler.FiddlerApplication.BeforeRequest +=
    delegate(Fiddler.SessionoS)
    {
    // Ignore requests made to the main site.
    RegexmatchUrl = newRegex("webmetrics.com”);
    if (matchUrl.IsMatch(oS.fullUrl)) {
    oS.utilCreateResponseAndBypassServer();
    oS.responseCode = 200;
    }
    };
    • In the same way responses from the server can be modified
    Fiddler.FiddlerApplication.BeforeResponse
  • 32. Method 4: Proxy – FiddlerCore
    When to use
    Windows Only
    Cross browser
    .NET
  • 33. Method 5: Wire HTTP capture
    Captured network traffic can be converted to HAR
    On Unix/OSX use tcpdump
    tcpdump -i en0 -n -s 0 -wtraffic.pcap
    On Windows use WinPCap
    winpcap -i 0 -n -s 0 -wtraffic.pcap
    Then use pcap2har to do the conversion:
    main.pytraffic.pcap http-requests.har
    Pcap2har
    https://github.com/andrewf/pcap2har
  • 34. Method 5: Wire HTTP capture
    Captures allHTTP traffic
    Timings based on actual network traffic
    No interference from proxy
    No extra network delays/context switches talking to the proxy
    Browsers sometimes behave differently when talking to a proxy
  • 35. Summary
    Discussed 5 methods for recording performance metrics
    Use timers around selenium commands
    Use WebTimings JavaScript API
    Firebug & NetExport
    Proxy
    Use the built-in Selenium proxy
    Use a 3rd party proxy
    Sniff network traffic
  • 36. Links
    BrowserMob proxy
    https://github.com/lightbody/browsermob-proxy
    Fiddler Cookbook
    http://www.fiddler2.com/fiddler/dev/ScriptSamples.asp
    Examples from this talk
    https://github.com/watsonmw/selenium-pageloadmetrics