Your SlideShare is downloading. ×

Performance Metrics in a Day with Selenium

13,454

Published on

#seconf

#seconf

Published in: Technology
0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
13,454
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
201
Comments
0
Likes
20
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Welcome to our talk about getting performance metrics in a day
  • Outline the talk, what is the takeawayRun your standard unit tests (Junit, minitest, ….) but put in performance testing as well
  • Becoming the standard mechanism to store/dump performance data
  • Lots of text on this slide, we’ll keep it more entetaining laterOverall page load time= The obvious starting point and usually the easiest to acquire
  • Setting your own timings – obvious, but is not always very accurate if you are in a sequence of steps and capture page transitions and other delays Very coarse, incorporates latency/setup/teardown delays in the times reported, e.g. it sometimes takes a second or 2 for an IE window to become responsiveWeb Timings == Very simple at the moment, but gives very accurate overall stats, like initial connection time, DOM ready, and total request->response times.Browser plugins == Accurate picture of what the browser is doing, but not so straightforward to incorporate into your code.Proxy == Gives you very good timings for http traffic but can’t give you timings like render time or DOM ready events.Network traffic capture tools like WinPCAP, or wireshark
  • Doesn't seem to work currently in FF4Will have to parse the HAR format if you just want the basic timing out of it
  • Going to go over:- How to grab metrics using a proxy How using a proxy can be beneficial for testing in general
  • (Fiddler refers to thehttp transaction as 'sessions')
  • Documentation spread out over blogs, fiddler google groups, and fiddler extensions pageAPI is little difficult
  • Transcript

    • 1. Performance Metrics In A Day
      Shane Hender
      Mark Watson
    • 2. Agenda
      Goal is to show how you can enhance your automation tests to gather performance metrics
      Keep the short presentation as practical as possible with enough code snippets to give you a starting point
    • 3. One more thing…
      HTTP Archive (HAR)
      What?
      UTF8 Text based JSON encoding to represent performance data
      Why?
      Many libraries to convert from JSON to objects
      More tools allowing import/export/view HAR
      Becoming the standard format
    • 4. Snippet of HAR format
      {
      "log": {
      "version": "1.1",
      "creator": {
      "name": "Firebug",
      "version": "1.7.0"
      },
      "browser": {
      "name": "Firefox",
      "version": "4.0"
      },
      "pages": [
      {
      "startedDateTime": "2011-03-31T16:56:50.455-07:00",
      "id": "page_62901",
      "title": "Google",
      "pageTimings": {
      "onContentLoad": 431,
      "onLoad": 3148
      }
      }
      ],
      "entries": [
      {
      "pageref": "page_62901",
      "startedDateTime": "2011-03-31T16:56:50.455-07:00",
      "time": 250,
      "request": {
      "method": "GET",
      "url": "http://www.google.com/",
      "httpVersion": "HTTP/1.1",
      "cookies": [
      {
      "name": "PREF",
      "value": "ID"
      },
    • 5. HAR Viewers
      http://www.softwareishard.com/har/viewer/
      Fiddler UI can import HAR files
      ShowSlowwebapp can be used to archive and view HAR files
    • 6. Which Metrics?
      Overall page load time
      DOM loading/interactive/complete, browser 1st render, …
      Per-item timings
      DNS, SSL connect, time to first byte, total time to download bytes, …
      Headers, status codes, and content
      For a more detailed picture of the reasons for the page performance
      Can help with debugging performance, e.g. were items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering
    • 7. Methods for gathering metrics
      Setting your own timings in test code
      Using the new ‘Navigation.Timings’ or Web Timings standard built into browsers
      Using browser plugins that report back the timings to you, e.g. WebPageTest
      Per HTTP request logs via Selenium’s built-in proxy
      Routing the browser traffic through a local proxy and gathering the statistics from there.
      Network traffic capture
    • 8. Method 1: Own Timings
      WebDriver driver = newChromeDriver();
      Date start = new Date();
      driver.get("http://www.webmetrics.com");
      Date end = new Date();
    • 9. Method 2: Web Timings
      Currently Chrome and IE9 supported, coming soon for Firefox
      http://w3c-test.org/webperf/specs/NavigationTiming/
      WebDriver driver = newChromeDriver();
      // A "base url", used by selenium to resolve relative URLs
      StringbaseUrl = "http://www.webmetrics.com";
      driver.get(baseUrl);
      JavascriptExecutorjs = (JavascriptExecutor)driver;
      LongloadEventEnd= (Long) js.executeScript("return
      window.performance.timing.loadEventEnd");
      LongnavigationStart= (Long) js.executeScript("returnwindow.performance.timing.navigationStart");
      Long answerToAllProblems =loadEventEnd - navigationStart;
    • 10. Method 2: Web Timings
      Could also modify this by using your own JavaScript loaded into the page to set timings
      Also would work in combination with the already set Web Timings
      E.g. Set a Date().getTime() variable when some dynamically loaded content is loaded into the DOM
      LongperceivedLoadTime = (Long) js.executeScript("return
      elementInsertTime –
      window.performance.timing.navigationStart");
    • 11. Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....
    • 12. Method 3: Browser Plugins
      Typically can give compute time metrics
      Javascript execution time
      CPU usage
      Render times
      IE8 - WebPageTest
      CPU usage
      Video capture
      Firefox - Firebug Net Panel + NetExport
      https://github.com/lightbody/browsermob-page-perf
      https://github.com/AutomatedTester/AutomatedTester.PagePerf.git
      DynaTrace browser plugin (FireFox/IE on Windows)
      Good breakdown of stats (Javascript execution times, CSS times, render, 'first impression' time).
      http://ajax.dynatrace.com/ajax/en/
      Chrome – some export from the Developer Tools Network Panel?
    • 13. FirefoxProfilep=newFirefoxProfile();
      try{
      p.addExtension(newFile("c:/firebug-1.6.0.xpi"));
      p.addExtension(newFile("c:/netExport-0.8b9.xpi"));
      p.addExtension(newFile("c:/firestarter-0.1.a5.xpi"));
      }catch(IOExceptione){
      thrownewRuntimeException(“Failed to load extensions:",e);
      }
      p.setPreference("extensions.firebug.netexport.autoExportActive",true);
      //p.setPreference("extensions.firebug.netexport.defaultLogDir", "c:/");
      p.setPreference("extensions.firebug.onByDefault",true);
      p.setPreference("extensions.firebug.defaultPanelName","net");
      p.setPreference("extensions.firebug.net.enableSites",true);
      p.setPreference("extensions.firebug.previousPlacement",1);
      driver=newFirefoxDriver(p);
      Method 3: Example - Firebug + NetExport
    • 14. Method 4: Example - Firebug + NetExport
    • 15. Method 4: Use a Proxy
      Use built in Selenium 1 proxy
      Use a 3rd party proxy library
      Many available, few capture metrics in a convenient way
      Two good ones:
      BrowserMob Proxy
      Fiddler
    • 16. Method 4: Selenium built-in Proxy
      Use API Selenium.captureNetworkTraffic()
      Selenium selenium = newDefaultSelenium("localhost", 4444,
      "*firefox", "http://www.webmetrics.com"); selenium.start("captureNetworkTraffic=true");
      selenium.open("http://www.webmetrics.com");
      Stringjson = selenium.captureNetworkTraffic("json");
      selenium.stop();
    • 17. Method 4: Selenium built-in Proxy
      Not in HTTP Archive (HAR) format, but does report:
      HTTP status code, URL and HTTP method
      Request & response headers
      Overall request->response timing
      Bytes downloaded
      Works with Chrome and Firefox
      Doesn’t work for WebDriver tests.
      Still work with Selenium 2 RC, but only if send jobs via Selenese API.
      Not much control over the proxy
    • 18. Method 4: Advantages of using a Proxy
      Testing benefits beyond performance monitoring
      Blacklisting/whitelisting URLs
      Comparing page load times of site with/without external content
      Not hitting analytics/advertising content
      Simulating external content being offline
      URL rewrites
      Redirect production URLs back to staging/test environment
      Pretend to be production as far as the browser is concerned (e.g. for cookies)
      Make sure ad/metrics requests are being made
    • 19. Method 4: Advantages of using a Proxy
      Header changes
      Set the user agent manually to test different browser behavior
      Auto authentication
      Wait until all content is downloaded
      HTTP idle for X seconds
      Limit bandwidth
    • 20. Method 4: BrowserMob Proxy
      Open source cross platform proxy written in Java.
      HTTP Archive support
      Friendly API
      Source code available:
      https://github.com/lightbody/browsermob-proxy
    • 21. Method 4: BrowserMob Proxy
      Export HAR Sample
      import org.browsermob.proxy.ProxyServer;
      ProxyServer proxy = new ProxyServer(9090);
      proxy.start();
      Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090");
      DesiredCapabilities caps = new DesiredCapabilities();
      caps.setCapability("proxy", mobProxy);
      FirefoxDriver driver = new FirefoxDriver(caps);
    • 22. Method 4: BrowserMob Proxy
      Run test, denoting pages in HAR
      proxy.newHar("Yahoo");
      driver.get("http://yahoo.com");
      proxy.endPage();
      proxy.newPage("CNN");
      driver.get("http://cnn.com");
      proxy.endPage();
    • 23. Method 4: BrowserMob Proxy
      Write out HTTP Archive file
      proxy.getHar().writeTo(newFile("test.har"));
    • 24. Method 4: BrowserMob Proxy
      • Blacklist/Whitelist URLs
      proxy.blacklistRequests(regex, responseCode)
      proxy.whitelistRequests(regex, responseCode)
      Redirecting URLs
      proxy.rewriteUrl(regex, replace)
      • Limit Bandwidth
      proxy.setDownstreamKbps(kbps)
      proxy.setUpstreamKbps(kbps)
    • 25. Method 4: BrowserMob Proxy
      When to use
      Cross platform
      Java
      Can set the browser proxy
    • 26. Method 4: Proxy - FiddlerCore
      Fiddler is an application for viewing HTTP traffic on Windows.
      Works with Chrome, FireFox, Internet Explorer.
      Fiddler Application is built on FiddlerCore
      .NET library
      Allows extensive programmatic control over the proxy
      Captures HTTP timings
      Allows request/responses to be intercepted and modified
      Configures itself as default Windows proxy (supports proxy chaining)
      Can decrypt SSL traffic
    • 27. To start the proxy and register it as the default system wide proxy.
      Each HTTP transaction can then be recorded as follows:
      Method 4: Proxy - FiddlerCore
      // Initialize the proxy
      Fiddler.FiddlerApplication.Startup(
      8877, FiddlerCoreStartupFlags.Default);
      var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.SessionoS)
      {
      items.Add(oS);
      };
    • 28. Method 4: Proxy - FiddlerCore
      Run Selenium test as normal
      As each item is downloaded it will be added to the ’items’ var in previous slide.
      string baseUrl = "http://www.webmetrics.com";
      varwebDriver =
      newOpenQA.Selenium.Firefox.FirefoxDriver();
      var selenium =
      newSelenium.WebDriverBackedSelenium
      (webDriver, baseUrl);
      selenium.Start();
      selenium.Open(baseUrl);
      selenium.WaitForPageToLoad("30000");
      selenium.Stop();
    • 29. Method 4: Proxy (Fiddler -> HAR)
      Using the HAR Exporter to convert the sessions to HAR
      Add FiddlerCore-BasicFormats.dll to the project references and load the assembly:
      https://www.fiddler2.com/dl/FiddlerCore-BasicFormats.zip
      String exePath = Assembly.GetExecutingAssembly().Location;
      String path = Path.Combine(
      Path.GetDirectoryName(exePath),
      @"FiddlerCore-BasicFormats.dll");
      FiddlerApplication.oTranscoders.ImportTranscoders(path);
    • 30. Method 4: Proxy (Fiddler -> HAR)
      Finally, export the HAR to a file:
      varoExportOptions = new Dictionary<string, object>();
      string filename = @"output.har”;
      oExportOptions.Add("Filename", filename);
      Fiddler.FiddlerApplication.DoExport(
      "HTTPArchive v1.2", sessions, oExportOptions, null);
    • 31. Method 4: Proxy Bonus Features
      Modifying HTTP requests
      Fiddler.FiddlerApplication.BeforeRequest +=
      delegate(Fiddler.SessionoS)
      {
      // Ignore requests made to the main site.
      RegexmatchUrl = newRegex("webmetrics.com”);
      if (matchUrl.IsMatch(oS.fullUrl)) {
      oS.utilCreateResponseAndBypassServer();
      oS.responseCode = 200;
      }
      };
      • In the same way responses from the server can be modified
      Fiddler.FiddlerApplication.BeforeResponse
    • 32. Method 4: Proxy – FiddlerCore
      When to use
      Windows Only
      Cross browser
      .NET
    • 33. Method 5: Wire HTTP capture
      Captured network traffic can be converted to HAR
      On Unix/OSX use tcpdump
      tcpdump -i en0 -n -s 0 -wtraffic.pcap
      On Windows use WinPCap
      winpcap -i 0 -n -s 0 -wtraffic.pcap
      Then use pcap2har to do the conversion:
      main.pytraffic.pcap http-requests.har
      Pcap2har
      https://github.com/andrewf/pcap2har
    • 34. Method 5: Wire HTTP capture
      Captures allHTTP traffic
      Timings based on actual network traffic
      No interference from proxy
      No extra network delays/context switches talking to the proxy
      Browsers sometimes behave differently when talking to a proxy
    • 35. Summary
      Discussed 5 methods for recording performance metrics
      Use timers around selenium commands
      Use WebTimings JavaScript API
      Firebug & NetExport
      Proxy
      Use the built-in Selenium proxy
      Use a 3rd party proxy
      Sniff network traffic
    • 36. Links
      BrowserMob proxy
      https://github.com/lightbody/browsermob-proxy
      Fiddler Cookbook
      http://www.fiddler2.com/fiddler/dev/ScriptSamples.asp
      Examples from this talk
      https://github.com/watsonmw/selenium-pageloadmetrics

    ×