Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Performance Metrics In A Day<br />Shane Hender<br />Mark Watson<br />
Agenda<br />Goal is to show how you can enhance your automation tests to gather performance metrics<br />Keep the short pr...
One more thing…<br />HTTP Archive (HAR)<br />What?<br />UTF8 Text based JSON encoding to represent performance data<br />W...
Snippet of HAR format<br />{<br />"log": {<br />"version": "1.1",<br />     "creator": {<br />       "name": "Firebug",<br...
HAR Viewers<br />http://www.softwareishard.com/har/viewer/<br />Fiddler UI can import HAR files<br />ShowSlowwebapp can be...
Which Metrics?<br />Overall page load time<br />DOM loading/interactive/complete, browser 1st render, …<br />Per-item timi...
Methods for gathering metrics<br />Setting your own timings in test code<br />Using the new ‘Navigation.Timings’ or Web Ti...
Method 1: Own Timings<br />WebDriver driver = newChromeDriver();<br />Date start = new Date();<br />driver.get("http://www...
Method 2: Web Timings<br />Currently Chrome and IE9 supported, coming soon for Firefox<br />http://w3c-test.org/webperf/sp...
Method 2: Web Timings<br />Could also modify this by using your own JavaScript loaded into the page to set timings<br />Al...
Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....<br />
Method 3: Browser Plugins<br />Typically can give compute time metrics<br />Javascript execution time<br />CPU usage<br />...
FirefoxProfilep=newFirefoxProfile();<br />try{<br />p.addExtension(newFile("c:/firebug-1.6.0.xpi"));<br />p.addExtension(n...
Method 4: Example - Firebug + NetExport<br />
Method 4: Use a Proxy<br />Use built in Selenium 1 proxy<br />Use a 3rd party proxy library <br />Many available, few capt...
Method 4: Selenium built-in Proxy<br />Use API Selenium.captureNetworkTraffic()<br />Selenium selenium = newDefaultSeleniu...
Method 4: Selenium built-in Proxy<br />Not in HTTP Archive (HAR) format, but does report:<br />HTTP status code, URL and H...
Method 4: Advantages of using a Proxy<br />Testing benefits beyond performance monitoring<br />Blacklisting/whitelisting U...
Method 4: Advantages of using a Proxy<br />Header changes<br />Set the user agent manually to test different browser behav...
Method 4: BrowserMob Proxy<br />Open source cross platform proxy written in Java.<br />HTTP Archive support<br />Friendly ...
Method 4: BrowserMob Proxy<br />Export HAR Sample<br />import org.browsermob.proxy.ProxyServer;<br />ProxyServer proxy = n...
Method 4: BrowserMob Proxy<br />Run test, denoting pages in HAR<br />proxy.newHar("Yahoo");<br />driver.get("http://yahoo....
Method 4: BrowserMob Proxy<br />Write out HTTP Archive file<br />proxy.getHar().writeTo(newFile("test.har"));<br />
Method 4: BrowserMob Proxy<br /><ul><li>Blacklist/Whitelist URLs</li></ul>proxy.blacklistRequests(regex, responseCode)<br ...
Method 4: BrowserMob Proxy<br />When to use<br />Cross platform<br />Java<br />Can set the browser proxy<br />
Method 4: Proxy - FiddlerCore<br />Fiddler is an application for viewing HTTP traffic on Windows.<br />Works with Chrome, ...
To start the proxy and register it as the default system wide proxy.<br />Each HTTP transaction can then be recorded as fo...
Method 4: Proxy - FiddlerCore<br />Run Selenium test as normal<br />As each item is downloaded it will be added to the ’it...
Method 4: Proxy (Fiddler -> HAR)<br />Using the HAR Exporter to convert the sessions to HAR<br />Add FiddlerCore-BasicForm...
Method 4: Proxy (Fiddler -> HAR)<br />Finally, export the HAR to a file:<br />varoExportOptions = new Dictionary<string, o...
Method 4: Proxy Bonus Features<br />Modifying HTTP requests<br />Fiddler.FiddlerApplication.BeforeRequest +=<br />delegate...
Method 4: Proxy – FiddlerCore<br />When to use<br />Windows Only<br />Cross browser<br />.NET<br />
Method 5: Wire HTTP capture<br />Captured network traffic can be converted to HAR<br />On Unix/OSX use tcpdump<br />tcpdum...
Method 5: Wire HTTP capture<br />Captures allHTTP traffic<br />Timings based on actual network traffic<br />No interferenc...
Summary<br />Discussed 5 methods for recording performance metrics<br />Use timers around selenium commands<br />Use WebTi...
Links<br />BrowserMob proxy<br />https://github.com/lightbody/browsermob-proxy<br />Fiddler Cookbook<br />http://www.fiddl...
Upcoming SlideShare
Loading in …5
×

Performance Metrics in a Day with Selenium

21,010 views

Published on

#seconf

Published in: Technology
  • Be the first to comment

Performance Metrics in a Day with Selenium

  1. 1. Performance Metrics In A Day<br />Shane Hender<br />Mark Watson<br />
  2. 2. Agenda<br />Goal is to show how you can enhance your automation tests to gather performance metrics<br />Keep the short presentation as practical as possible with enough code snippets to give you a starting point<br />
  3. 3. One more thing…<br />HTTP Archive (HAR)<br />What?<br />UTF8 Text based JSON encoding to represent performance data<br />Why?<br />Many libraries to convert from JSON to objects<br />More tools allowing import/export/view HAR<br />Becoming the standard format<br />
  4. 4. Snippet of HAR format<br />{<br />"log": {<br />"version": "1.1",<br /> "creator": {<br /> "name": "Firebug",<br /> "version": "1.7.0"<br /> },<br /> "browser": {<br /> "name": "Firefox",<br /> "version": "4.0"<br /> },<br /> "pages": [<br /> {<br /> "startedDateTime": "2011-03-31T16:56:50.455-07:00",<br /> "id": "page_62901",<br /> "title": "Google",<br /> "pageTimings": {<br /> "onContentLoad": 431,<br /> "onLoad": 3148<br /> }<br /> }<br /> ],<br />"entries": [<br /> {<br /> "pageref": "page_62901",<br /> "startedDateTime": "2011-03-31T16:56:50.455-07:00",<br /> "time": 250,<br /> "request": {<br /> "method": "GET",<br /> "url": "http://www.google.com/",<br /> "httpVersion": "HTTP/1.1",<br /> "cookies": [<br /> {<br /> "name": "PREF",<br /> "value": "ID"<br /> },<br />
  5. 5. HAR Viewers<br />http://www.softwareishard.com/har/viewer/<br />Fiddler UI can import HAR files<br />ShowSlowwebapp can be used to archive and view HAR files<br />
  6. 6. Which Metrics?<br />Overall page load time<br />DOM loading/interactive/complete, browser 1st render, …<br />Per-item timings<br />DNS, SSL connect, time to first byte, total time to download bytes, …<br />Headers, status codes, and content<br />For a more detailed picture of the reasons for the page performance<br />Can help with debugging performance, e.g. were items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering<br />
  7. 7. Methods for gathering metrics<br />Setting your own timings in test code<br />Using the new ‘Navigation.Timings’ or Web Timings standard built into browsers <br />Using browser plugins that report back the timings to you, e.g. WebPageTest<br />Per HTTP request logs via Selenium’s built-in proxy <br />Routing the browser traffic through a local proxy and gathering the statistics from there.<br />Network traffic capture<br />
  8. 8. Method 1: Own Timings<br />WebDriver driver = newChromeDriver();<br />Date start = new Date();<br />driver.get("http://www.webmetrics.com");<br />Date end = new Date();<br />
  9. 9. Method 2: Web Timings<br />Currently Chrome and IE9 supported, coming soon for Firefox<br />http://w3c-test.org/webperf/specs/NavigationTiming/<br />WebDriver driver = newChromeDriver();<br />// A "base url", used by selenium to resolve relative URLs <br />StringbaseUrl = "http://www.webmetrics.com"; <br />driver.get(baseUrl); <br />JavascriptExecutorjs = (JavascriptExecutor)driver; <br />LongloadEventEnd= (Long) js.executeScript("return<br />window.performance.timing.loadEventEnd");<br />LongnavigationStart= (Long) js.executeScript("returnwindow.performance.timing.navigationStart");<br />Long answerToAllProblems =loadEventEnd - navigationStart;<br />
  10. 10. Method 2: Web Timings<br />Could also modify this by using your own JavaScript loaded into the page to set timings<br />Also would work in combination with the already set Web Timings<br />E.g. Set a Date().getTime() variable when some dynamically loaded content is loaded into the DOM<br />LongperceivedLoadTime = (Long) js.executeScript("return<br />elementInsertTime – <br />window.performance.timing.navigationStart");<br />
  11. 11. Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....<br />
  12. 12. Method 3: Browser Plugins<br />Typically can give compute time metrics<br />Javascript execution time<br />CPU usage<br />Render times<br />IE8 - WebPageTest<br />CPU usage<br />Video capture<br />Firefox - Firebug Net Panel + NetExport<br />https://github.com/lightbody/browsermob-page-perf<br />https://github.com/AutomatedTester/AutomatedTester.PagePerf.git<br />DynaTrace browser plugin (FireFox/IE on Windows)<br />Good breakdown of stats (Javascript execution times, CSS times, render, 'first impression' time).<br />http://ajax.dynatrace.com/ajax/en/<br />Chrome – some export from the Developer Tools Network Panel?<br />
  13. 13. FirefoxProfilep=newFirefoxProfile();<br />try{<br />p.addExtension(newFile("c:/firebug-1.6.0.xpi"));<br />p.addExtension(newFile("c:/netExport-0.8b9.xpi"));<br />p.addExtension(newFile("c:/firestarter-0.1.a5.xpi"));<br />}catch(IOExceptione){<br />thrownewRuntimeException(“Failed to load extensions:",e);<br />}<br />p.setPreference("extensions.firebug.netexport.autoExportActive",true);<br />//p.setPreference("extensions.firebug.netexport.defaultLogDir", "c:/");<br />p.setPreference("extensions.firebug.onByDefault",true);<br />p.setPreference("extensions.firebug.defaultPanelName","net");<br />p.setPreference("extensions.firebug.net.enableSites",true);<br />p.setPreference("extensions.firebug.previousPlacement",1);<br />driver=newFirefoxDriver(p);<br />Method 3: Example - Firebug + NetExport<br />
  14. 14. Method 4: Example - Firebug + NetExport<br />
  15. 15. Method 4: Use a Proxy<br />Use built in Selenium 1 proxy<br />Use a 3rd party proxy library <br />Many available, few capture metrics in a convenient way<br />Two good ones:<br />BrowserMob Proxy<br />Fiddler<br />
  16. 16. Method 4: Selenium built-in Proxy<br />Use API Selenium.captureNetworkTraffic()<br />Selenium selenium = newDefaultSelenium("localhost", 4444,<br />"*firefox", "http://www.webmetrics.com"); selenium.start("captureNetworkTraffic=true");<br />selenium.open("http://www.webmetrics.com"); <br />Stringjson = selenium.captureNetworkTraffic("json"); <br />selenium.stop();<br />
  17. 17. Method 4: Selenium built-in Proxy<br />Not in HTTP Archive (HAR) format, but does report:<br />HTTP status code, URL and HTTP method<br />Request & response headers<br />Overall request->response timing<br />Bytes downloaded<br />Works with Chrome and Firefox<br />Doesn’t work for WebDriver tests.<br />Still work with Selenium 2 RC, but only if send jobs via Selenese API.<br />Not much control over the proxy<br />
  18. 18. Method 4: Advantages of using a Proxy<br />Testing benefits beyond performance monitoring<br />Blacklisting/whitelisting URLs<br />Comparing page load times of site with/without external content<br />Not hitting analytics/advertising content<br />Simulating external content being offline<br />URL rewrites<br />Redirect production URLs back to staging/test environment<br />Pretend to be production as far as the browser is concerned (e.g. for cookies)<br />Make sure ad/metrics requests are being made<br />
  19. 19. Method 4: Advantages of using a Proxy<br />Header changes<br />Set the user agent manually to test different browser behavior<br />Auto authentication<br />Wait until all content is downloaded<br />HTTP idle for X seconds<br />Limit bandwidth<br />
  20. 20. Method 4: BrowserMob Proxy<br />Open source cross platform proxy written in Java.<br />HTTP Archive support<br />Friendly API<br />Source code available:<br />https://github.com/lightbody/browsermob-proxy<br />
  21. 21. Method 4: BrowserMob Proxy<br />Export HAR Sample<br />import org.browsermob.proxy.ProxyServer;<br />ProxyServer proxy = new ProxyServer(9090);<br />proxy.start();<br />Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090");<br />DesiredCapabilities caps = new DesiredCapabilities();<br />caps.setCapability("proxy", mobProxy);<br />FirefoxDriver driver = new FirefoxDriver(caps);<br />
  22. 22. Method 4: BrowserMob Proxy<br />Run test, denoting pages in HAR<br />proxy.newHar("Yahoo");<br />driver.get("http://yahoo.com");<br />proxy.endPage();<br />proxy.newPage("CNN");<br />driver.get("http://cnn.com");<br />proxy.endPage();<br />
  23. 23. Method 4: BrowserMob Proxy<br />Write out HTTP Archive file<br />proxy.getHar().writeTo(newFile("test.har"));<br />
  24. 24. Method 4: BrowserMob Proxy<br /><ul><li>Blacklist/Whitelist URLs</li></ul>proxy.blacklistRequests(regex, responseCode)<br />proxy.whitelistRequests(regex, responseCode)<br />Redirecting URLs<br />proxy.rewriteUrl(regex, replace)<br /><ul><li>Limit Bandwidth</li></ul>proxy.setDownstreamKbps(kbps)<br />proxy.setUpstreamKbps(kbps)<br />
  25. 25. Method 4: BrowserMob Proxy<br />When to use<br />Cross platform<br />Java<br />Can set the browser proxy<br />
  26. 26. Method 4: Proxy - FiddlerCore<br />Fiddler is an application for viewing HTTP traffic on Windows.<br />Works with Chrome, FireFox, Internet Explorer.<br />Fiddler Application is built on FiddlerCore<br />.NET library<br />Allows extensive programmatic control over the proxy<br />Captures HTTP timings<br />Allows request/responses to be intercepted and modified<br />Configures itself as default Windows proxy (supports proxy chaining)<br />Can decrypt SSL traffic<br />
  27. 27. To start the proxy and register it as the default system wide proxy.<br />Each HTTP transaction can then be recorded as follows:<br />Method 4: Proxy - FiddlerCore<br />// Initialize the proxy<br />Fiddler.FiddlerApplication.Startup(<br /> 8877, FiddlerCoreStartupFlags.Default);<br />var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.SessionoS)<br />{<br />items.Add(oS);<br />};<br />
  28. 28. Method 4: Proxy - FiddlerCore<br />Run Selenium test as normal<br />As each item is downloaded it will be added to the ’items’ var in previous slide.<br />string baseUrl = "http://www.webmetrics.com";<br />varwebDriver =<br />newOpenQA.Selenium.Firefox.FirefoxDriver();<br />var selenium = <br />newSelenium.WebDriverBackedSelenium<br /> (webDriver, baseUrl);<br />selenium.Start();<br />selenium.Open(baseUrl);<br />selenium.WaitForPageToLoad("30000");<br />selenium.Stop();<br />
  29. 29. Method 4: Proxy (Fiddler -> HAR)<br />Using the HAR Exporter to convert the sessions to HAR<br />Add FiddlerCore-BasicFormats.dll to the project references and load the assembly:<br />https://www.fiddler2.com/dl/FiddlerCore-BasicFormats.zip<br />String exePath = Assembly.GetExecutingAssembly().Location;<br />String path = Path.Combine(<br />Path.GetDirectoryName(exePath),<br /> @"FiddlerCore-BasicFormats.dll");<br />FiddlerApplication.oTranscoders.ImportTranscoders(path);<br />
  30. 30. Method 4: Proxy (Fiddler -> HAR)<br />Finally, export the HAR to a file:<br />varoExportOptions = new Dictionary<string, object>();<br />string filename = @"output.har”;<br />oExportOptions.Add("Filename", filename);<br />Fiddler.FiddlerApplication.DoExport(<br />"HTTPArchive v1.2", sessions, oExportOptions, null);<br />
  31. 31. Method 4: Proxy Bonus Features<br />Modifying HTTP requests<br />Fiddler.FiddlerApplication.BeforeRequest +=<br />delegate(Fiddler.SessionoS)<br />{<br />// Ignore requests made to the main site.<br />RegexmatchUrl = newRegex("webmetrics.com”);<br />if (matchUrl.IsMatch(oS.fullUrl)) {<br />oS.utilCreateResponseAndBypassServer();<br />oS.responseCode = 200;<br /> }<br />};<br /><ul><li>In the same way responses from the server can be modified</li></ul>Fiddler.FiddlerApplication.BeforeResponse<br />
  32. 32. Method 4: Proxy – FiddlerCore<br />When to use<br />Windows Only<br />Cross browser<br />.NET<br />
  33. 33. Method 5: Wire HTTP capture<br />Captured network traffic can be converted to HAR<br />On Unix/OSX use tcpdump<br />tcpdump -i en0 -n -s 0 -wtraffic.pcap<br />On Windows use WinPCap<br />winpcap -i 0 -n -s 0 -wtraffic.pcap<br />Then use pcap2har to do the conversion:<br />main.pytraffic.pcap http-requests.har<br />Pcap2har<br />https://github.com/andrewf/pcap2har<br />
  34. 34. Method 5: Wire HTTP capture<br />Captures allHTTP traffic<br />Timings based on actual network traffic<br />No interference from proxy<br />No extra network delays/context switches talking to the proxy<br />Browsers sometimes behave differently when talking to a proxy<br />
  35. 35. Summary<br />Discussed 5 methods for recording performance metrics<br />Use timers around selenium commands<br />Use WebTimings JavaScript API<br />Firebug & NetExport<br />Proxy<br />Use the built-in Selenium proxy<br />Use a 3rd party proxy<br />Sniff network traffic<br />
  36. 36. Links<br />BrowserMob proxy<br />https://github.com/lightbody/browsermob-proxy<br />Fiddler Cookbook<br />http://www.fiddler2.com/fiddler/dev/ScriptSamples.asp<br />Examples from this talk<br />https://github.com/watsonmw/selenium-pageloadmetrics<br />

×