Testing, Performance Analysis, and jQuery 1.4

Testing, Performance Analysis, and jQuery 1.4 John Resig http://ejohn.org/ - http://twitter.com/jeresig

Why Test JavaScript? ✦ Cross-browser issues. ✦ The possibility for causing an unforeseen problem is simply too great.

JavaScript Testing Isn’t The Same As Desktop or Server-Side Testing

What should I use?

Looong Tail 300 225 150 75 0 Other

Basic Components ✦ Writing and understanding a JavaScript test suite is easy. ✦ Test Suite ✦ Tests ✦ Assertions ✦ Async Tests ✦ Test Runner

Assertions <html> <head> <title>Test Suite</title> <script> function assert( value, desc ) { var li = document.createElement("li"); li.className = value ? "pass" : "fail"; li.appendChild( document.createTextNode( desc ) ); document.getElementById("results").appendChild( li ); } window.onload = function(){ assert( true, "The test suite is running." ); }; </script> <style> #results li.pass { color: green; } #results li.fail { color: red; } </style> </head> <body> <ul id="results"></ul> </body> </html>

Tests test("A test.", function(){ assert( true, "First assertion completed" ); assert( true, "Second assertion completed" ); assert( true, "Third assertion completed" ); }); test("Another test.", function(){ assert( true, "First test completed" ); assert( false, "Second test failed" ); assert( true, "Third assertion completed" ); });

Tests var results; function assert( value, desc ) { var li = document.createElement("li"); li.className = value ? "pass" : "fail"; li.appendChild( document.createTextNode( desc ) ); results.appendChild( li ); if ( !value ) { li.parentNode.parentNode.className = "fail"; } return li; } function test(name, fn){ results = document.getElementById("results"); results = assert( true, name ).appendChild( document.createElement("ul") ); fn(); }

Async Tests test("Async Test #1", function(){ pause(); setTimeout(function(){ assert( true, "First test completed" ); resume(); }, 1000); }); test("Async Test #2", function(){ pause(); setTimeout(function(){ assert( true, "Second test completed" ); resume(); }, 1000); });

Async Tests (function(){ var queue = [], paused = false; this.test = function(fn){ queue.push( fn ); runTest(); }; this.pause = function(){ paused = true; }; this.resume = function(){ paused = false; setTimeout(runTest, 1); }; function runTest(){ if ( !paused && queue.length ) { queue.shift()(); if ( !paused ) { resume(); } } } })();

People don’t test. :-( 900 675 450 225 0 None

Popular Test Frameworks 300 225 150 75 0 QUnit JSUnit Selenium YUITest FireUnit Screw.Unit JSSpec

Unit Testing ✦ Break code into logical chucks for testing. ✦ Focus on one method at a time. ✦ Good for testing APIs. ✦ Popular Frameworks: ✦ QUnit ✦ JSUnit ✦ YUITest ✦ FireUnit

JSUnit ✦ One of the oldest JavaScript testing frameworks. ✦ A port of JUnit to JavaScript, circa 2001. ✦ Code feels very 2001 (frames!) ✦ http://www.jsunit.net/

JSUnit function coreSuite() { var result = new top.jsUnitTestSuite(); result.addTestPage("tests/jsUnitAssertionTests.html"); result.addTestPage("tests/jsUnitSetUpTearDownTests.html"); result.addTestPage("tests/jsUnitRestoredHTMLDivTests.html"); result.addTestPage("tests/jsUnitFrameworkUtilityTests.html"); result.addTestPage("tests/jsUnitOnLoadTests.html"); result.addTestPage("tests/jsUnitUtilityTests.html"); return result; } function serverSuite() { var result = new top.jsUnitTestSuite(); result.addTestPage("tests/server/jsUnitVersionCheckTests.html"); result.addTestPage("tests/server/jsUnitServerAjaxTests.html"); return result; } function librariesSuite() { var result = new top.jsUnitTestSuite(); result.addTestPage("tests/jsUnitMockTimeoutTest.html"); return result; } function suite() { var newsuite = new top.jsUnitTestSuite(); newsuite.addTestSuite(coreSuite()); newsuite.addTestSuite(serverSuite()); newsuite.addTestSuite(librariesSuite()); return newsuite; }

JSUnit function testAssertNotUndefined() { assertNotUndefined("1 should not be undefined", 1); assertNotUndefined(1); } function testAssertNaN() { assertNaN("a string should not be a number", "string"); assertNaN("string"); } function testAssertNotNaN() { assertNotNaN("1 should not be not a number", 1); assertNotNaN(1); } function testFail() { var excep = null; try { fail("Failure message"); } catch (e) { excep = e; } assertJsUnitException("fail(string) should throw a JsUnitException", excep); } function testTooFewArguments() { var excep = null; try { assert(); } catch (e1) { excep = e1; } assertNonJsUnitException("Calling an assertion function with too few arguments should throw an exception", excep); }

JSUnit

YUITest (2 & 3) ✦ Testing framework built and developed by Yahoo (released Oct 2008). ✦ Completely overhauled to go with YUI v3. ✦ Features: ✦ Supports async tests. ✦ Has good event simulation. ✦ v2: http://developer.yahoo.com/yui/ examples/yuitest/ ✦ v3: http://developer.yahoo.com/yui/3/test/

YUITest 2 YAHOO.example.yuitest.ArrayTestCase = new YAHOO.tool.TestCase({ name : "Array Tests", setUp : function () { this.data = [0,1,2,3,4] }, tearDown : function () { delete this.data; }, testPop : function () { var Assert = YAHOO.util.Assert; var value = this.data.pop(); Assert.areEqual(4, this.data.length); Assert.areEqual(4, value); }, testPush : function () { var Assert = YAHOO.util.Assert; this.data.push(5); Assert.areEqual(6, this.data.length); Assert.areEqual(5, this.data[5]); } });

YUITest 2

YUITest 3 Y.example.test.DataTestCase = new Y.Test.Case({ name : "Data Tests", setUp : function () { this.data = { name: "test", year: 2007, beta: true }; }, tearDown : function () { delete this.data; }, testName : function () { var Assert = Y.Assert; Assert.isObject(this.data); Assert.isString(this.data.name); Assert.areEqual("test", this.data.name); }, testYear : function () { var Assert = Y.Assert; Assert.isObject(this.data); Assert.isNumber(this.data.year); Assert.areEqual(2007, this.data.year); } });

YUITest 3

QUnit ✦ Unit Testing framework built for jQuery. ✦ Features: ✦ Supports asynchronous testing. ✦ Can break code into modules. ✦ Supports test timeouts. ✦ No dependencies. ✦ Painfully simple. ✦ http://docs.jquery.com/QUnit

QUnit Style test("a basic test example", function() { ok( true, "this test is fine" ); var value = "hello"; equals( "hello", value, "We expect value to be hello" ); }); module("Module A"); test("first test within module", function() { ok( true, "all pass" ); }); test("second test within module", function() { ok( true, "all pass" ); }); module("Module B"); test("some other test", function() { expect(2); equals( true, false, "failing test" ); equals( true, true, "passing test" ); });

FireUnit ✦ Unit testing extension for Firebug ✦ ﬁreunit.ok( true, “...” ); ✦ http://ﬁreunit.org/

Standardization ✦ CommonJS: A uniﬁed cross-platform API for JavaScript. ✦ (Including the server-side!) ✦ Working to standardize a simple testing API. ✦ http://wiki.commonjs.org/wiki/CommonJS

Server-Side ✦ Ignore the browser! Simulate it on the server-side. ✦ Almost always uses Java + Rhino to construct a browser. ✦ Some frameworks: ✦ Crosscheck ✦ Env.js ✦ Blueridge

Server-Side ✦ Crosscheck ✦ Pure Java, even simulates browser bugs. ✦ http://www.thefrontside.net/crosscheck ✦ Env.js ✦ Pure JavaScript, focuses on standards support. ✦ http://github.com/thatcher/env-js/tree/ master ✦ Blueridge ✦ Env.js + Screw.Unit + Rhino ✦ http://github.com/relevance/blue-ridge/

Env.js $ java -jar build/js.jar Rhino 1.6 release 6 2007 06 28 js> load('build/runtest/env.js'); js> window.location = 'test/index.html'; test/index.html js> load('dist/jquery.js'); // Add pretty printing to jQuery objects: js> jQuery.fn.toString = DOMNodeList.prototype.toString; js> $('span').remove(); [ <span#å°åŒ—Taibei>, <span#å°åŒ—>, <span#utf8class1>, <span#utf8class2>, <span#foo:bar>, <span#test.foo[5]bar> ] // Yes - UTF-8 is supported in DOM documents! js> $('span') [ ] js> $('div').append('<span><b>hello!</b> world</span>'); [ <div#main>, <div#foo> ] js> $('span') [ <span>, <span> ] js> $('span').text() hello! worldhello! world

Browser Launching ✦ Automates the process of opening browser windows, running tests, and getting results. ✦ Frequently require a speciﬁc framework. ✦ Popular frameworks: ✦ WebDriver http://code.google.com/p/ webdriver/ (Java) ✦ Waitr http://wtr.rubyforge.org/ (Ruby) ✦ JsTestDriver http://code.google.com/p/ js-test-driver/ (Java) ✦ Selenium RC http://seleniumhq.org/ projects/remote-control/ (Java)

Browser Launching

Distributed ✦ Selenium Grid ✦ Push Selenium tests out to many machines (that you manage), simultaneously. ✦ Collect and store the results. ✦ http://selenium-grid.seleniumhq.org/ ✦ TestSwarm ✦ Push tests to a distributed swarm of clients. ✦ Results viewable on the server. ✦ http://testswarm.com/

The Scaling Problem ✦ The Problem: ✦ jQuery has 6 test suites ✦ Run in 15 browsers ✦ (Not even including multiple platforms or mobile browsers!) ✦ All need to be run for every commit, patch, and plugin. ✦ JavaScript testing doesn’t scale well.

Distributed Testing ✦ Hub server ✦ Clients connect and help run tests ✦ A simple JavaScript client that can be run in all browsers ✦ Including mobile browsers! ✦ TestSwarm

FF 3.5 FF 3.5 FF 3.5 IE 6 IE 6 FF 3 IE 6 Op 9 FF 3 IE 7 TestSwarm IE 7 Test Suite Test Suite Test Suite

TestSwarm.com ✦ Incentives for top testers (t-shirts, books) ✦ Will be opening for alpha testing very soon ✦ Help your favorite JavaScript library become better tested! ✦ http://testswarm.com

Accurately Measuring JavaScript

Major Cases ✦ Same Code, Diﬀerent Platforms ✦ Compare V8 vs. SpiderMonkey vs. JavaScriptCore ✦ Diﬀerent Code, Same Platform ✦ Compare CSS Selector Engines ✦ A/B testing a piece of code

Same Code, Diﬀerent Platform ✦ A number of suites analyzing JS perf: ✦ SunSpider (from WebKit) ✦ V8 Benchmark (from V8/Chrome) ✦ Dromaeo (from Mozilla) ✦ Statistical accuracy and reproducibility is paramount.

SunSpider ✦ All tests were highly balanced. ✦ Provide some level of statistical accuracy. ✦ +/- 5ms (for example) ✦ Tests are run by loading an iframe with the test 5 times. ✦ getTime() is run before/after each test. ✦ Entire suite must be trashed in order to upgrade/ﬁx a test.

Error Rate? ✦ How do we get it? What does it mean? ✦ It’s how conﬁdent we are that we arrived at the result we want in the number of runs that we’ve done.

Normal Distribution ✦ First: Assume that the results are coming back in a normal distribution. ✦ The “bell curve”

Conﬁdence ✦ Next: We need a conﬁdence level. ✦ T-Distribution works well here. http://en.wikipedia.org/wiki/Student%27s_t-distribution

Error Rate ✦ 5 runs ✦ (each run is potentially 1000s of individual test runs) ✦ 95% Conﬁdence (t-distribution = 2.776) ✦ Standard Errors Mean = ✦ (std_dev / sqrt(runs)) * 2.776 ✦ Error = (err_mean / mean) * 100 ✦ This way you can get results like: ✦ 123ms +/- 5ms

V8 Benchmark ✦ Tests are run, potentially, 1000s of times. ✦ Also provides an error rate. ✦ (Use a geometric mean to arrive at a result.)

Small Time Accuracy ✦ Small time: ✦ 1ms, 1ms, 1ms, 1ms, 3ms ✦ huge error! ✦ Large time: ✦ 1234ms, 1234ms, 1234ms, 1234ms, 1238ms ✦ tiny error! ✦ Tests that run faster need to be run more times. ✦ Running more times = less potential for weird results. http://ejohn.org/blog/javascript-benchmark-quality/

Runs/Second ✦ var start = (new Date).getTime(); while (time < 1000) { runTest(); time = (new Date).getTime() - start; } ✦ More test runs, more statistical accuracy. ✦ V8 & Dromaeo-style suites handle this. ✦ (Problem: getTime() is being run on every loop - it should be run less frequently in order to inﬂuence the numbers less.)

Runs/Second ✦ You are now measuring tests/second rather than seconds per test. ✦ You run tests as many times in one second as you can. ✦ Then you do that multiple times (5?) ✦ THEN you analyze the ﬁnal numbers: ✦ 1234run/s, 1230runs/s, 1240runs/s, ...

Harmonic Mean ✦ A way to average rates ✦ Which is what we have! runs/second ✦ For example: ✦ 1234run/s, 1230runs/s, 1240runs/s, 1236runs/ms, 1232runs/s ✦ 5 / ( (1/1234) + (1/1230) + (1/1240) + (1/1236) + (1/1232) ) = ✦ 1234.39runs/s! http://en.wikipedia.org/wiki/Harmonic_mean

Dromaeo ✦ All individual tests are versioned ✦ Makes it easy to update or fix a bug in a test ✦ Can only run tests of specific versions against each other ✦ Uses V8’s style of running tests. ✦ Also has DOM and framework tests. ✦ ...and hooks for doing Shark profiling.

Bug Fixes ✦ Tests will, inevitably, have bugs that need to be ﬁxed. ✦ Fixing a bug changes the result quality. ✦ Tests need to be versioned so that changes can be made. ✦ You look at Test v1 vs. Test v1 results. ✦ Not Test v2 vs. Test v1. ✦ Tip: Just use the last revision control commit # for the test ﬁle.

Diﬀerent Code, Same Platform ✦ Most solutions here are very poor. ✦ Run the test very few times, use getTime(). ✦ Highly inaccurate results, massive error.

Garbage Collection ✦ Browsers periodically run garbage collectors to clean up old objects no longer referenced. ✦ This can take a long time and spike your test results. ✦ Example: ✦ 10ms, 13ms, 11ms, 12ms, 486ms, 12ms, ... ✦ When comparing engine to engine, this doesn’t matter. ✦ Comparing code vs. code, it does.

Mean, Median, Mode? ✦ Mode! ✦ Run your tests a large number of times. ✦ What is the ‘mode’ (the result that occurs most frequently) ✦ Example: ✦ 10, 11, 11, 12, 12, 12, 13, 14 ✦ Mode = 12ms. ✦ Less accurate than mean, but gives you a more-consistent result. ✦ DON’T DISCARD “BAD” RESULTS!

getTime() Accuracy http://ejohn.org/blog/accuracy-of-javascript-time/

15ms intervals ONLY! Error Rate of 50-750%!

IE in Wine ✦ Running Internet Explorer in Wine (on Linux) gives ﬁne-grained timer results ✦ Down to the millisecond! ✦ You can also run IE, in Wine, on OS X: ✦ ies4osx ✦ Huge Caveat: It gives you ﬁne-grained time, but that doesn’t mean it’s accurate.

Diﬀerent Code, Same Platform ✦ How can we get good numbers? ✦ We have to go straight to the source: Use the tools the browsers provide.

Firebug Proﬁler

Safari 4 Proﬁler

IE 8 Proﬁler

IE 6-8: DynaTrace Ajax

Shark Proﬁling ✦ Extremely low-level ✦ Watch for all internal function calls ✦ See what gets called the most. ✦ Dromaeo includes shark proﬁling hooks.

Interesting Problems Tackled in jQuery 1.4

Tackled in 1.4 ✦ Complexity reduction ✦ Bubbling change, submit, focus, blur ✦ Required script loading

Questions ✦ Contact: ✦ John Resig ✦ http://ejohn.org/ ✦ http://twitter.com/jeresig

http://www.slideshare.net	60
http://docs.intra.mixi.co.jp	23
http://blog.espol.edu.ec	8
http://lanyrd.com	4
http://safe.tumblr.com	1
http://actx.angellearning.com	1
http://www.blogger.com	1
http://static.slidesharecdn.com	1
http://www.brijj.com	1

Testing, Performance Analysis, and jQuery 1.4

by jeresig

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 100

Statistics

Testing, Performance Analysis, and jQuery 1.4 Presentation Transcript