Performance Metrics In A Day<br />Shane Hender<br />Mark Watson<br />
Agenda<br />Goal is to show how you can enhance your automation tests to gather performance metrics<br />Keep the short pr...
One more thing…<br />HTTP Archive (HAR)<br />What?<br />UTF8 Text based JSON encoding to represent performance data<br />W...
Snippet of HAR format<br />{<br />"log": {<br />"version": "1.1",<br />     "creator": {<br />       "name": "Firebug",<br...
HAR Viewers<br />http://www.softwareishard.com/har/viewer/<br />Fiddler UI can import HAR files<br />ShowSlowwebapp can be...
Which Metrics?<br />Overall page load time<br />DOM loading/interactive/complete, browser 1st render, …<br />Per-item timi...
Methods for gathering metrics<br />Setting your own timings in test code<br />Using the new ‘Navigation.Timings’ or Web Ti...
Method 1: Own Timings<br />WebDriver driver = newChromeDriver();<br />Date start = new Date();<br />driver.get("http://www...
Method 2: Web Timings<br />Currently Chrome and IE9 supported, coming soon for Firefox<br />http://w3c-test.org/webperf/sp...
Method 2: Web Timings<br />Could also modify this by using your own JavaScript loaded into the page to set timings<br />Al...
Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....<br />
Method 3: Browser Plugins<br />Typically can give compute time metrics<br />Javascript execution time<br />CPU usage<br />...
FirefoxProfilep=newFirefoxProfile();<br />try{<br />p.addExtension(newFile("c:/firebug-1.6.0.xpi"));<br />p.addExtension(n...
Method 4: Example - Firebug + NetExport<br />
Method 4: Use a Proxy<br />Use built in Selenium 1 proxy<br />Use a 3rd party proxy library <br />Many available, few capt...
Method 4: Selenium built-in Proxy<br />Use API Selenium.captureNetworkTraffic()<br />Selenium selenium = newDefaultSeleniu...
Method 4: Selenium built-in Proxy<br />Not in HTTP Archive (HAR) format, but does report:<br />HTTP status code, URL and H...
Method 4: Advantages of using a Proxy<br />Testing benefits beyond performance monitoring<br />Blacklisting/whitelisting U...
Method 4: Advantages of using a Proxy<br />Header changes<br />Set the user agent manually to test different browser behav...
Method 4: BrowserMob Proxy<br />Open source cross platform proxy written in Java.<br />HTTP Archive support<br />Friendly ...
Method 4: BrowserMob Proxy<br />Export HAR Sample<br />import org.browsermob.proxy.ProxyServer;<br />ProxyServer proxy = n...
Method 4: BrowserMob Proxy<br />Run test, denoting pages in HAR<br />proxy.newHar("Yahoo");<br />driver.get("http://yahoo....
Method 4: BrowserMob Proxy<br />Write out HTTP Archive file<br />proxy.getHar().writeTo(newFile("test.har"));<br />
Method 4: BrowserMob Proxy<br /><ul><li>Blacklist/Whitelist URLs</li></ul>proxy.blacklistRequests(regex, responseCode)<br ...
Method 4: BrowserMob Proxy<br />When to use<br />Cross platform<br />Java<br />Can set the browser proxy<br />
Method 4: Proxy - FiddlerCore<br />Fiddler is an application for viewing HTTP traffic on Windows.<br />Works with Chrome, ...
To start the proxy and register it as the default system wide proxy.<br />Each HTTP transaction can then be recorded as fo...
Method 4: Proxy - FiddlerCore<br />Run Selenium test as normal<br />As each item is downloaded it will be added to the ’it...
Method 4: Proxy (Fiddler -> HAR)<br />Using the HAR Exporter to convert the sessions to HAR<br />Add FiddlerCore-BasicForm...
Method 4: Proxy (Fiddler -> HAR)<br />Finally, export the HAR to a file:<br />varoExportOptions = new Dictionary<string, o...
Method 4: Proxy Bonus Features<br />Modifying HTTP requests<br />Fiddler.FiddlerApplication.BeforeRequest +=<br />delegate...
Method 4: Proxy – FiddlerCore<br />When to use<br />Windows Only<br />Cross browser<br />.NET<br />
Method 5: Wire HTTP capture<br />Captured network traffic can be converted to HAR<br />On Unix/OSX use tcpdump<br />tcpdum...
Method 5: Wire HTTP capture<br />Captures allHTTP traffic<br />Timings based on actual network traffic<br />No interferenc...
Summary<br />Discussed 5 methods for recording performance metrics<br />Use timers around selenium commands<br />Use WebTi...
Links<br />BrowserMob proxy<br />https://github.com/lightbody/browsermob-proxy<br />Fiddler Cookbook<br />http://www.fiddl...
Upcoming SlideShare
Loading in...5
×

Performance Metrics in a Day with Selenium

14,497

Published on

#seconf

Published in: Technology
0 Comments
21 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
14,497
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
211
Comments
0
Likes
21
Embeds 0
No embeds

No notes for slide
  • Welcome to our talk about getting performance metrics in a day
  • Outline the talk, what is the takeawayRun your standard unit tests (Junit, minitest, ….) but put in performance testing as well
  • Becoming the standard mechanism to store/dump performance data
  • Lots of text on this slide, we’ll keep it more entetaining laterOverall page load time= The obvious starting point and usually the easiest to acquire
  • Setting your own timings – obvious, but is not always very accurate if you are in a sequence of steps and capture page transitions and other delays Very coarse, incorporates latency/setup/teardown delays in the times reported, e.g. it sometimes takes a second or 2 for an IE window to become responsiveWeb Timings == Very simple at the moment, but gives very accurate overall stats, like initial connection time, DOM ready, and total request-&gt;response times.Browser plugins == Accurate picture of what the browser is doing, but not so straightforward to incorporate into your code.Proxy == Gives you very good timings for http traffic but can’t give you timings like render time or DOM ready events.Network traffic capture tools like WinPCAP, or wireshark
  • Doesn&apos;t seem to work currently in FF4Will have to parse the HAR format if you just want the basic timing out of it
  • Going to go over:- How to grab metrics using a proxy How using a proxy can be beneficial for testing in general
  • (Fiddler refers to thehttp transaction as &apos;sessions&apos;)
  • Documentation spread out over blogs, fiddler google groups, and fiddler extensions pageAPI is little difficult
  • Transcript of "Performance Metrics in a Day with Selenium"

    1. 1. Performance Metrics In A Day<br />Shane Hender<br />Mark Watson<br />
    2. 2. Agenda<br />Goal is to show how you can enhance your automation tests to gather performance metrics<br />Keep the short presentation as practical as possible with enough code snippets to give you a starting point<br />
    3. 3. One more thing…<br />HTTP Archive (HAR)<br />What?<br />UTF8 Text based JSON encoding to represent performance data<br />Why?<br />Many libraries to convert from JSON to objects<br />More tools allowing import/export/view HAR<br />Becoming the standard format<br />
    4. 4. Snippet of HAR format<br />{<br />"log": {<br />"version": "1.1",<br /> "creator": {<br /> "name": "Firebug",<br /> "version": "1.7.0"<br /> },<br /> "browser": {<br /> "name": "Firefox",<br /> "version": "4.0"<br /> },<br /> "pages": [<br /> {<br /> "startedDateTime": "2011-03-31T16:56:50.455-07:00",<br /> "id": "page_62901",<br /> "title": "Google",<br /> "pageTimings": {<br /> "onContentLoad": 431,<br /> "onLoad": 3148<br /> }<br /> }<br /> ],<br />"entries": [<br /> {<br /> "pageref": "page_62901",<br /> "startedDateTime": "2011-03-31T16:56:50.455-07:00",<br /> "time": 250,<br /> "request": {<br /> "method": "GET",<br /> "url": "http://www.google.com/",<br /> "httpVersion": "HTTP/1.1",<br /> "cookies": [<br /> {<br /> "name": "PREF",<br /> "value": "ID"<br /> },<br />
    5. 5. HAR Viewers<br />http://www.softwareishard.com/har/viewer/<br />Fiddler UI can import HAR files<br />ShowSlowwebapp can be used to archive and view HAR files<br />
    6. 6. Which Metrics?<br />Overall page load time<br />DOM loading/interactive/complete, browser 1st render, …<br />Per-item timings<br />DNS, SSL connect, time to first byte, total time to download bytes, …<br />Headers, status codes, and content<br />For a more detailed picture of the reasons for the page performance<br />Can help with debugging performance, e.g. were items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering<br />
    7. 7. Methods for gathering metrics<br />Setting your own timings in test code<br />Using the new ‘Navigation.Timings’ or Web Timings standard built into browsers <br />Using browser plugins that report back the timings to you, e.g. WebPageTest<br />Per HTTP request logs via Selenium’s built-in proxy <br />Routing the browser traffic through a local proxy and gathering the statistics from there.<br />Network traffic capture<br />
    8. 8. Method 1: Own Timings<br />WebDriver driver = newChromeDriver();<br />Date start = new Date();<br />driver.get("http://www.webmetrics.com");<br />Date end = new Date();<br />
    9. 9. Method 2: Web Timings<br />Currently Chrome and IE9 supported, coming soon for Firefox<br />http://w3c-test.org/webperf/specs/NavigationTiming/<br />WebDriver driver = newChromeDriver();<br />// A "base url", used by selenium to resolve relative URLs <br />StringbaseUrl = "http://www.webmetrics.com"; <br />driver.get(baseUrl); <br />JavascriptExecutorjs = (JavascriptExecutor)driver; <br />LongloadEventEnd= (Long) js.executeScript("return<br />window.performance.timing.loadEventEnd");<br />LongnavigationStart= (Long) js.executeScript("returnwindow.performance.timing.navigationStart");<br />Long answerToAllProblems =loadEventEnd - navigationStart;<br />
    10. 10. Method 2: Web Timings<br />Could also modify this by using your own JavaScript loaded into the page to set timings<br />Also would work in combination with the already set Web Timings<br />E.g. Set a Date().getTime() variable when some dynamically loaded content is loaded into the DOM<br />LongperceivedLoadTime = (Long) js.executeScript("return<br />elementInsertTime – <br />window.performance.timing.navigationStart");<br />
    11. 11. Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....<br />
    12. 12. Method 3: Browser Plugins<br />Typically can give compute time metrics<br />Javascript execution time<br />CPU usage<br />Render times<br />IE8 - WebPageTest<br />CPU usage<br />Video capture<br />Firefox - Firebug Net Panel + NetExport<br />https://github.com/lightbody/browsermob-page-perf<br />https://github.com/AutomatedTester/AutomatedTester.PagePerf.git<br />DynaTrace browser plugin (FireFox/IE on Windows)<br />Good breakdown of stats (Javascript execution times, CSS times, render, 'first impression' time).<br />http://ajax.dynatrace.com/ajax/en/<br />Chrome – some export from the Developer Tools Network Panel?<br />
    13. 13. FirefoxProfilep=newFirefoxProfile();<br />try{<br />p.addExtension(newFile("c:/firebug-1.6.0.xpi"));<br />p.addExtension(newFile("c:/netExport-0.8b9.xpi"));<br />p.addExtension(newFile("c:/firestarter-0.1.a5.xpi"));<br />}catch(IOExceptione){<br />thrownewRuntimeException(“Failed to load extensions:",e);<br />}<br />p.setPreference("extensions.firebug.netexport.autoExportActive",true);<br />//p.setPreference("extensions.firebug.netexport.defaultLogDir", "c:/");<br />p.setPreference("extensions.firebug.onByDefault",true);<br />p.setPreference("extensions.firebug.defaultPanelName","net");<br />p.setPreference("extensions.firebug.net.enableSites",true);<br />p.setPreference("extensions.firebug.previousPlacement",1);<br />driver=newFirefoxDriver(p);<br />Method 3: Example - Firebug + NetExport<br />
    14. 14. Method 4: Example - Firebug + NetExport<br />
    15. 15. Method 4: Use a Proxy<br />Use built in Selenium 1 proxy<br />Use a 3rd party proxy library <br />Many available, few capture metrics in a convenient way<br />Two good ones:<br />BrowserMob Proxy<br />Fiddler<br />
    16. 16. Method 4: Selenium built-in Proxy<br />Use API Selenium.captureNetworkTraffic()<br />Selenium selenium = newDefaultSelenium("localhost", 4444,<br />"*firefox", "http://www.webmetrics.com"); selenium.start("captureNetworkTraffic=true");<br />selenium.open("http://www.webmetrics.com"); <br />Stringjson = selenium.captureNetworkTraffic("json"); <br />selenium.stop();<br />
    17. 17. Method 4: Selenium built-in Proxy<br />Not in HTTP Archive (HAR) format, but does report:<br />HTTP status code, URL and HTTP method<br />Request & response headers<br />Overall request->response timing<br />Bytes downloaded<br />Works with Chrome and Firefox<br />Doesn’t work for WebDriver tests.<br />Still work with Selenium 2 RC, but only if send jobs via Selenese API.<br />Not much control over the proxy<br />
    18. 18. Method 4: Advantages of using a Proxy<br />Testing benefits beyond performance monitoring<br />Blacklisting/whitelisting URLs<br />Comparing page load times of site with/without external content<br />Not hitting analytics/advertising content<br />Simulating external content being offline<br />URL rewrites<br />Redirect production URLs back to staging/test environment<br />Pretend to be production as far as the browser is concerned (e.g. for cookies)<br />Make sure ad/metrics requests are being made<br />
    19. 19. Method 4: Advantages of using a Proxy<br />Header changes<br />Set the user agent manually to test different browser behavior<br />Auto authentication<br />Wait until all content is downloaded<br />HTTP idle for X seconds<br />Limit bandwidth<br />
    20. 20. Method 4: BrowserMob Proxy<br />Open source cross platform proxy written in Java.<br />HTTP Archive support<br />Friendly API<br />Source code available:<br />https://github.com/lightbody/browsermob-proxy<br />
    21. 21. Method 4: BrowserMob Proxy<br />Export HAR Sample<br />import org.browsermob.proxy.ProxyServer;<br />ProxyServer proxy = new ProxyServer(9090);<br />proxy.start();<br />Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090");<br />DesiredCapabilities caps = new DesiredCapabilities();<br />caps.setCapability("proxy", mobProxy);<br />FirefoxDriver driver = new FirefoxDriver(caps);<br />
    22. 22. Method 4: BrowserMob Proxy<br />Run test, denoting pages in HAR<br />proxy.newHar("Yahoo");<br />driver.get("http://yahoo.com");<br />proxy.endPage();<br />proxy.newPage("CNN");<br />driver.get("http://cnn.com");<br />proxy.endPage();<br />
    23. 23. Method 4: BrowserMob Proxy<br />Write out HTTP Archive file<br />proxy.getHar().writeTo(newFile("test.har"));<br />
    24. 24. Method 4: BrowserMob Proxy<br /><ul><li>Blacklist/Whitelist URLs</li></ul>proxy.blacklistRequests(regex, responseCode)<br />proxy.whitelistRequests(regex, responseCode)<br />Redirecting URLs<br />proxy.rewriteUrl(regex, replace)<br /><ul><li>Limit Bandwidth</li></ul>proxy.setDownstreamKbps(kbps)<br />proxy.setUpstreamKbps(kbps)<br />
    25. 25. Method 4: BrowserMob Proxy<br />When to use<br />Cross platform<br />Java<br />Can set the browser proxy<br />
    26. 26. Method 4: Proxy - FiddlerCore<br />Fiddler is an application for viewing HTTP traffic on Windows.<br />Works with Chrome, FireFox, Internet Explorer.<br />Fiddler Application is built on FiddlerCore<br />.NET library<br />Allows extensive programmatic control over the proxy<br />Captures HTTP timings<br />Allows request/responses to be intercepted and modified<br />Configures itself as default Windows proxy (supports proxy chaining)<br />Can decrypt SSL traffic<br />
    27. 27. To start the proxy and register it as the default system wide proxy.<br />Each HTTP transaction can then be recorded as follows:<br />Method 4: Proxy - FiddlerCore<br />// Initialize the proxy<br />Fiddler.FiddlerApplication.Startup(<br /> 8877, FiddlerCoreStartupFlags.Default);<br />var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.SessionoS)<br />{<br />items.Add(oS);<br />};<br />
    28. 28. Method 4: Proxy - FiddlerCore<br />Run Selenium test as normal<br />As each item is downloaded it will be added to the ’items’ var in previous slide.<br />string baseUrl = "http://www.webmetrics.com";<br />varwebDriver =<br />newOpenQA.Selenium.Firefox.FirefoxDriver();<br />var selenium = <br />newSelenium.WebDriverBackedSelenium<br /> (webDriver, baseUrl);<br />selenium.Start();<br />selenium.Open(baseUrl);<br />selenium.WaitForPageToLoad("30000");<br />selenium.Stop();<br />
    29. 29. Method 4: Proxy (Fiddler -> HAR)<br />Using the HAR Exporter to convert the sessions to HAR<br />Add FiddlerCore-BasicFormats.dll to the project references and load the assembly:<br />https://www.fiddler2.com/dl/FiddlerCore-BasicFormats.zip<br />String exePath = Assembly.GetExecutingAssembly().Location;<br />String path = Path.Combine(<br />Path.GetDirectoryName(exePath),<br /> @"FiddlerCore-BasicFormats.dll");<br />FiddlerApplication.oTranscoders.ImportTranscoders(path);<br />
    30. 30. Method 4: Proxy (Fiddler -> HAR)<br />Finally, export the HAR to a file:<br />varoExportOptions = new Dictionary<string, object>();<br />string filename = @"output.har”;<br />oExportOptions.Add("Filename", filename);<br />Fiddler.FiddlerApplication.DoExport(<br />"HTTPArchive v1.2", sessions, oExportOptions, null);<br />
    31. 31. Method 4: Proxy Bonus Features<br />Modifying HTTP requests<br />Fiddler.FiddlerApplication.BeforeRequest +=<br />delegate(Fiddler.SessionoS)<br />{<br />// Ignore requests made to the main site.<br />RegexmatchUrl = newRegex("webmetrics.com”);<br />if (matchUrl.IsMatch(oS.fullUrl)) {<br />oS.utilCreateResponseAndBypassServer();<br />oS.responseCode = 200;<br /> }<br />};<br /><ul><li>In the same way responses from the server can be modified</li></ul>Fiddler.FiddlerApplication.BeforeResponse<br />
    32. 32. Method 4: Proxy – FiddlerCore<br />When to use<br />Windows Only<br />Cross browser<br />.NET<br />
    33. 33. Method 5: Wire HTTP capture<br />Captured network traffic can be converted to HAR<br />On Unix/OSX use tcpdump<br />tcpdump -i en0 -n -s 0 -wtraffic.pcap<br />On Windows use WinPCap<br />winpcap -i 0 -n -s 0 -wtraffic.pcap<br />Then use pcap2har to do the conversion:<br />main.pytraffic.pcap http-requests.har<br />Pcap2har<br />https://github.com/andrewf/pcap2har<br />
    34. 34. Method 5: Wire HTTP capture<br />Captures allHTTP traffic<br />Timings based on actual network traffic<br />No interference from proxy<br />No extra network delays/context switches talking to the proxy<br />Browsers sometimes behave differently when talking to a proxy<br />
    35. 35. Summary<br />Discussed 5 methods for recording performance metrics<br />Use timers around selenium commands<br />Use WebTimings JavaScript API<br />Firebug & NetExport<br />Proxy<br />Use the built-in Selenium proxy<br />Use a 3rd party proxy<br />Sniff network traffic<br />
    36. 36. Links<br />BrowserMob proxy<br />https://github.com/lightbody/browsermob-proxy<br />Fiddler Cookbook<br />http://www.fiddler2.com/fiddler/dev/ScriptSamples.asp<br />Examples from this talk<br />https://github.com/watsonmw/selenium-pageloadmetrics<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×