Testing, Performance Analysis, and jQuery 1.4
Upcoming SlideShare
Loading in...5
×
 

Testing, Performance Analysis, and jQuery 1.4

on

  • 14,016 views

This is the talk that I gave at JSConf.eu 2009, then modified slightly and given again at the December Bayjax meetup (the parts on jQuery and HTML 5 in IE were added).

This is the talk that I gave at JSConf.eu 2009, then modified slightly and given again at the December Bayjax meetup (the parts on jQuery and HTML 5 in IE were added).

Statistics

Views

Total Views
14,016
Views on SlideShare
13,899
Embed Views
117

Actions

Likes
27
Downloads
438
Comments
2

12 Embeds 117

http://www.slideshare.net 60
http://docs.intra.mixi.co.jp 29
http://blog.espol.edu.ec 12
http://lanyrd.com 7
http://127.0.0.1 2
http://safe.tumblr.com 1
https://vuws.uws.edu.au 1
http://www.brijj.com 1
http://static.slidesharecdn.com 1
http://www.blogger.com 1
http://actx.angellearning.com 1
https://learn-future.csuchico.edu 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Very nice to see statistical methods applied to performance analysis in web development context!
    Are you sure you want to
    Your message goes here
    Processing…
  • Full video and transcript of this talk: http://developer.yahoo.com/yui/theater/video.php?v=resig-testing
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Testing, Performance Analysis, and jQuery 1.4 Testing, Performance Analysis, and jQuery 1.4 Presentation Transcript

  • Testing, Performance Analysis, and jQuery 1.4 John Resig http://ejohn.org/ - http://twitter.com/jeresig
  • Why Test JavaScript? ✦ Cross-browser issues. ✦ The possibility for causing an unforeseen problem is simply too great.
  • JavaScript Testing Isn’t The Same As Desktop or Server-Side Testing
  • What should I use?
  • Looong Tail 300 225 150 75 0 Other
  • Basic Components ✦ Writing and understanding a JavaScript test suite is easy. ✦ Test Suite ✦ Tests ✦ Assertions ✦ Async Tests ✦ Test Runner
  • Assertions  <html>   <head>     <title>Test Suite</title>     <script>     function assert( value, desc ) {       var li = document.createElement("li");       li.className = value ? "pass" : "fail";       li.appendChild( document.createTextNode( desc ) );       document.getElementById("results").appendChild( li );     }          window.onload = function(){       assert( true, "The test suite is running." );     };     </script>     <style>       #results li.pass { color: green; }       #results li.fail { color: red; }     </style>   </head>   <body>     <ul id="results"></ul>   </body>   </html> 
  • Tests      test("A test.", function(){         assert( true, "First assertion completed" );         assert( true, "Second assertion completed" );         assert( true, "Third assertion completed" );       });              test("Another test.", function(){         assert( true, "First test completed" );         assert( false, "Second test failed" );         assert( true, "Third assertion completed" );       }); 
  • Tests    var results;          function assert( value, desc ) {       var li = document.createElement("li");       li.className = value ? "pass" : "fail";       li.appendChild( document.createTextNode( desc ) );       results.appendChild( li );       if ( !value ) {         li.parentNode.parentNode.className = "fail";       }       return li;     }          function test(name, fn){       results = document.getElementById("results");       results = assert( true, name ).appendChild(         document.createElement("ul") );       fn();     } 
  • Async Tests  test("Async Test #1", function(){     pause();     setTimeout(function(){       assert( true, "First test completed" );       resume();     }, 1000);   });      test("Async Test #2", function(){     pause();     setTimeout(function(){       assert( true, "Second test completed" );       resume();     }, 1000);   }); 
  • Async Tests  (function(){     var queue = [], paused = false;          this.test = function(fn){       queue.push( fn );       runTest();     };          this.pause = function(){       paused = true;     };          this.resume = function(){       paused = false;       setTimeout(runTest, 1);     };          function runTest(){       if ( !paused && queue.length ) {         queue.shift()();         if ( !paused ) {           resume();         }       }     }   })(); 
  • People don’t test. :-( 900 675 450 225 0 None
  • Popular Test Frameworks 300 225 150 75 0 QUnit JSUnit Selenium YUITest FireUnit Screw.Unit JSSpec
  • Unit Testing ✦ Break code into logical chucks for testing. ✦ Focus on one method at a time. ✦ Good for testing APIs. ✦ Popular Frameworks: ✦ QUnit ✦ JSUnit ✦ YUITest ✦ FireUnit
  • JSUnit ✦ One of the oldest JavaScript testing frameworks. ✦ A port of JUnit to JavaScript, circa 2001. ✦ Code feels very 2001 (frames!) ✦ http://www.jsunit.net/
  • JSUnit        function coreSuite() {             var result = new top.jsUnitTestSuite();             result.addTestPage("tests/jsUnitAssertionTests.html");             result.addTestPage("tests/jsUnitSetUpTearDownTests.html");             result.addTestPage("tests/jsUnitRestoredHTMLDivTests.html");             result.addTestPage("tests/jsUnitFrameworkUtilityTests.html");             result.addTestPage("tests/jsUnitOnLoadTests.html");             result.addTestPage("tests/jsUnitUtilityTests.html");             return result;         }                  function serverSuite() {             var result = new top.jsUnitTestSuite();             result.addTestPage("tests/server/jsUnitVersionCheckTests.html");             result.addTestPage("tests/server/jsUnitServerAjaxTests.html");             return result;         }                  function librariesSuite() {             var result = new top.jsUnitTestSuite();             result.addTestPage("tests/jsUnitMockTimeoutTest.html");             return result;         }                  function suite() {             var newsuite = new top.jsUnitTestSuite();             newsuite.addTestSuite(coreSuite());             newsuite.addTestSuite(serverSuite());             newsuite.addTestSuite(librariesSuite());             return newsuite;         }
  • JSUnit  function testAssertNotUndefined() {       assertNotUndefined("1 should not be undefined", 1);       assertNotUndefined(1);   }      function testAssertNaN() {       assertNaN("a string should not be a number", "string");       assertNaN("string");   }      function testAssertNotNaN() {       assertNotNaN("1 should not be not a number", 1);       assertNotNaN(1);   }      function testFail() {       var excep = null;       try {           fail("Failure message");       } catch (e) {           excep = e;       }       assertJsUnitException("fail(string) should throw a JsUnitException", excep);   }      function testTooFewArguments() {       var excep = null;       try {           assert();       } catch (e1) {           excep = e1;       }       assertNonJsUnitException("Calling an assertion function with too  few arguments should throw an exception", excep);   }
  • JSUnit
  • YUITest (2 & 3) ✦ Testing framework built and developed by Yahoo (released Oct 2008). ✦ Completely overhauled to go with YUI v3. ✦ Features: ✦ Supports async tests. ✦ Has good event simulation. ✦ v2: http://developer.yahoo.com/yui/ examples/yuitest/ ✦ v3: http://developer.yahoo.com/yui/3/test/
  • YUITest 2   YAHOO.example.yuitest.ArrayTestCase = new YAHOO.tool.TestCase({       name : "Array Tests",              setUp : function () {           this.data = [0,1,2,3,4]       },              tearDown : function () {           delete this.data;       },                testPop : function () {           var Assert = YAHOO.util.Assert;                    var value = this.data.pop();                      Assert.areEqual(4, this.data.length);           Assert.areEqual(4, value);                   },                      testPush : function () {           var Assert = YAHOO.util.Assert;                      this.data.push(5);                      Assert.areEqual(6, this.data.length);           Assert.areEqual(5, this.data[5]);                   }  }); 
  • YUITest 2
  • YUITest 3  Y.example.test.DataTestCase = new Y.Test.Case({       name : "Data Tests",              setUp : function () {           this.data = {               name: "test",               year: 2007,               beta: true           };       },              tearDown : function () {           delete this.data;       },              testName : function () {           var Assert = Y.Assert;                      Assert.isObject(this.data);           Assert.isString(this.data.name);           Assert.areEqual("test", this.data.name);                   },              testYear : function () {           var Assert = Y.Assert;                      Assert.isObject(this.data);           Assert.isNumber(this.data.year);           Assert.areEqual(2007, this.data.year);                   }  }); 
  • YUITest 3
  • QUnit ✦ Unit Testing framework built for jQuery. ✦ Features: ✦ Supports asynchronous testing. ✦ Can break code into modules. ✦ Supports test timeouts. ✦ No dependencies. ✦ Painfully simple. ✦ http://docs.jquery.com/QUnit
  • QUnit Style  test("a basic test example", function() {     ok( true, "this test is fine" );     var value = "hello";     equals( "hello", value, "We expect value to be hello" );   });      module("Module A");      test("first test within module", function() {     ok( true, "all pass" );   });      test("second test within module", function() {     ok( true, "all pass" );   });      module("Module B");      test("some other test", function() {     expect(2);     equals( true, false, "failing test" );     equals( true, true, "passing test" );   }); 
  • QUnit
  • FireUnit ✦ Unit testing extension for Firebug ✦ fireunit.ok( true, “...” ); ✦ http://fireunit.org/
  • Standardization ✦ CommonJS: A unified cross-platform API for JavaScript. ✦ (Including the server-side!) ✦ Working to standardize a simple testing API. ✦ http://wiki.commonjs.org/wiki/CommonJS
  • Server-Side ✦ Ignore the browser! Simulate it on the server-side. ✦ Almost always uses Java + Rhino to construct a browser. ✦ Some frameworks: ✦ Crosscheck ✦ Env.js ✦ Blueridge
  • Server-Side ✦ Crosscheck ✦ Pure Java, even simulates browser bugs. ✦ http://www.thefrontside.net/crosscheck ✦ Env.js ✦ Pure JavaScript, focuses on standards support. ✦ http://github.com/thatcher/env-js/tree/ master ✦ Blueridge ✦ Env.js + Screw.Unit + Rhino ✦ http://github.com/relevance/blue-ridge/
  • Env.js $ java -jar build/js.jar Rhino 1.6 release 6 2007 06 28 js> load('build/runtest/env.js'); js> window.location = 'test/index.html'; test/index.html js> load('dist/jquery.js'); // Add pretty printing to jQuery objects: js> jQuery.fn.toString = DOMNodeList.prototype.toString; js> $('span').remove(); [ <span#å°åŒ—Taibei>, <span#å°åŒ—>, <span#utf8class1>, <span#utf8class2>, <span#foo:bar>, <span#test.foo[5]bar> ] // Yes - UTF-8 is supported in DOM documents! js> $('span') [ ] js> $('div').append('<span><b>hello!</b> world</span>'); [ <div#main>, <div#foo> ] js> $('span') [ <span>, <span> ] js> $('span').text() hello! worldhello! world
  • Browser Launching ✦ Automates the process of opening browser windows, running tests, and getting results. ✦ Frequently require a specific framework. ✦ Popular frameworks: ✦ WebDriver http://code.google.com/p/ webdriver/ (Java) ✦ Waitr http://wtr.rubyforge.org/ (Ruby) ✦ JsTestDriver http://code.google.com/p/ js-test-driver/ (Java) ✦ Selenium RC http://seleniumhq.org/ projects/remote-control/ (Java)
  • Browser Launching
  • Distributed ✦ Selenium Grid ✦ Push Selenium tests out to many machines (that you manage), simultaneously. ✦ Collect and store the results. ✦ http://selenium-grid.seleniumhq.org/ ✦ TestSwarm ✦ Push tests to a distributed swarm of clients. ✦ Results viewable on the server. ✦ http://testswarm.com/
  • The Scaling Problem ✦ The Problem: ✦ jQuery has 6 test suites ✦ Run in 15 browsers ✦ (Not even including multiple platforms or mobile browsers!) ✦ All need to be run for every commit, patch, and plugin. ✦ JavaScript testing doesn’t scale well.
  • Distributed Testing ✦ Hub server ✦ Clients connect and help run tests ✦ A simple JavaScript client that can be run in all browsers ✦ Including mobile browsers! ✦ TestSwarm
  • FF 3.5 FF 3.5 FF 3.5 IE 6 IE 6 FF 3 IE 6 Op 9 FF 3 IE 7 TestSwarm IE 7 Test Suite Test Suite Test Suite
  • TestSwarm.com ✦ Incentives for top testers (t-shirts, books) ✦ Will be opening for alpha testing very soon ✦ Help your favorite JavaScript library become better tested! ✦ http://testswarm.com
  • Accurately Measuring JavaScript
  • Major Cases ✦ Same Code, Different Platforms ✦ Compare V8 vs. SpiderMonkey vs. JavaScriptCore ✦ Different Code, Same Platform ✦ Compare CSS Selector Engines ✦ A/B testing a piece of code
  • Same Code, Different Platform ✦ A number of suites analyzing JS perf: ✦ SunSpider (from WebKit) ✦ V8 Benchmark (from V8/Chrome) ✦ Dromaeo (from Mozilla) ✦ Statistical accuracy and reproducibility is paramount.
  • SunSpider ✦ All tests were highly balanced. ✦ Provide some level of statistical accuracy. ✦ +/- 5ms (for example) ✦ Tests are run by loading an iframe with the test 5 times. ✦ getTime() is run before/after each test. ✦ Entire suite must be trashed in order to upgrade/fix a test.
  • Error Rate? ✦ How do we get it? What does it mean? ✦ It’s how confident we are that we arrived at the result we want in the number of runs that we’ve done.
  • Normal Distribution ✦ First: Assume that the results are coming back in a normal distribution. ✦ The “bell curve”
  • Confidence ✦ Next: We need a confidence level. ✦ T-Distribution works well here. http://en.wikipedia.org/wiki/Student%27s_t-distribution
  • Error Rate ✦ 5 runs ✦ (each run is potentially 1000s of individual test runs) ✦ 95% Confidence (t-distribution = 2.776) ✦ Standard Errors Mean = ✦ (std_dev / sqrt(runs)) * 2.776 ✦ Error = (err_mean / mean) * 100 ✦ This way you can get results like: ✦ 123ms +/- 5ms
  • V8 Benchmark ✦ Tests are run, potentially, 1000s of times. ✦ Also provides an error rate. ✦ (Use a geometric mean to arrive at a result.)
  • Small Time Accuracy ✦ Small time: ✦ 1ms, 1ms, 1ms, 1ms, 3ms ✦ huge error! ✦ Large time: ✦ 1234ms, 1234ms, 1234ms, 1234ms, 1238ms ✦ tiny error! ✦ Tests that run faster need to be run more times. ✦ Running more times = less potential for weird results. http://ejohn.org/blog/javascript-benchmark-quality/
  • Runs/Second ✦ var start = (new Date).getTime(); while (time < 1000) { runTest(); time = (new Date).getTime() - start; } ✦ More test runs, more statistical accuracy. ✦ V8 & Dromaeo-style suites handle this. ✦ (Problem: getTime() is being run on every loop - it should be run less frequently in order to influence the numbers less.)
  • Runs/Second ✦ You are now measuring tests/second rather than seconds per test. ✦ You run tests as many times in one second as you can. ✦ Then you do that multiple times (5?) ✦ THEN you analyze the final numbers: ✦ 1234run/s, 1230runs/s, 1240runs/s, ...
  • Harmonic Mean ✦ A way to average rates ✦ Which is what we have! runs/second ✦ For example: ✦ 1234run/s, 1230runs/s, 1240runs/s, 1236runs/ms, 1232runs/s ✦ 5 / ( (1/1234) + (1/1230) + (1/1240) + (1/1236) + (1/1232) ) = ✦ 1234.39runs/s! http://en.wikipedia.org/wiki/Harmonic_mean
  • Dromaeo ✦ All individual tests are versioned ✦ Makes it easy to update or fix a bug in a test ✦ Can only run tests of specific versions against each other ✦ Uses V8’s style of running tests. ✦ Also has DOM and framework tests. ✦ ...and hooks for doing Shark profiling.
  • Bug Fixes ✦ Tests will, inevitably, have bugs that need to be fixed. ✦ Fixing a bug changes the result quality. ✦ Tests need to be versioned so that changes can be made. ✦ You look at Test v1 vs. Test v1 results. ✦ Not Test v2 vs. Test v1. ✦ Tip: Just use the last revision control commit # for the test file.
  • Different Code, Same Platform ✦ Most solutions here are very poor. ✦ Run the test very few times, use getTime(). ✦ Highly inaccurate results, massive error.
  • Garbage Collection ✦ Browsers periodically run garbage collectors to clean up old objects no longer referenced. ✦ This can take a long time and spike your test results. ✦ Example: ✦ 10ms, 13ms, 11ms, 12ms, 486ms, 12ms, ... ✦ When comparing engine to engine, this doesn’t matter. ✦ Comparing code vs. code, it does.
  • Mean, Median, Mode? ✦ Mode! ✦ Run your tests a large number of times. ✦ What is the ‘mode’ (the result that occurs most frequently) ✦ Example: ✦ 10, 11, 11, 12, 12, 12, 13, 14 ✦ Mode = 12ms. ✦ Less accurate than mean, but gives you a more-consistent result. ✦ DON’T DISCARD “BAD” RESULTS!
  • getTime() Accuracy http://ejohn.org/blog/accuracy-of-javascript-time/
  • 15ms intervals ONLY! Error Rate of 50-750%!
  • IE in Wine ✦ Running Internet Explorer in Wine (on Linux) gives fine-grained timer results ✦ Down to the millisecond! ✦ You can also run IE, in Wine, on OS X: ✦ ies4osx ✦ Huge Caveat: It gives you fine-grained time, but that doesn’t mean it’s accurate.
  • Different Code, Same Platform ✦ How can we get good numbers? ✦ We have to go straight to the source: Use the tools the browsers provide.
  • Firebug Profiler
  • Safari 4 Profiler
  • IE 8 Profiler
  • IE 6-8: DynaTrace Ajax
  • IE 6-8: DynaTrace Ajax
  • Shark Profiling ✦ Extremely low-level ✦ Watch for all internal function calls ✦ See what gets called the most. ✦ Dromaeo includes shark profiling hooks.
  • Interesting Problems Tackled in jQuery 1.4
  • Tackled in 1.4 ✦ Complexity reduction ✦ Bubbling change, submit, focus, blur ✦ Required script loading
  • Questions ✦ Contact: ✦ John Resig ✦ http://ejohn.org/ ✦ http://twitter.com/jeresig