Testing, Performance Analysis, and jQuery 1.4

Testing, Performance
Analysis, and jQuery 1.4
John Resig
http://ejohn.org/ - http://twitter.com/jeresig

Why Test JavaScript?
✦ Cross-browser issues.
✦ The possibility for causing an unforeseen
problem is simply too great.

JavaScript Testing Isn’t
The Same As Desktop
or Server-Side Testing

Looong Tail

300

225

150

75

0

Other

Basic Components
✦ Writing and understanding a JavaScript
test suite is easy.
✦ Test Suite
✦ Tests
✦ Assertions
✦ Async Tests

✦ Test Runner

Assertions
<html>
<head>
   <title>Test Suite</title>
   <script>
   function assert( value, desc ) {
     var li = document.createElement("li");
     li.className = value ? "pass" : "fail";
     li.appendChild( document.createTextNode( desc ) );
     document.getElementById("results").appendChild( li );
   }

   window.onload = function(){
     assert( true, "The test suite is running." );
   };
   </script>
   <style>
     #results li.pass { color: green; }
     #results li.fail { color: red; }
   </style>
</head>
<body>
   <ul id="results"></ul>
</body>
</html>

Tests
     test("A test.", function(){
       assert( true, "First assertion completed" );
       assert( true, "Second assertion completed" );
       assert( true, "Third assertion completed" );
     });

     test("Another test.", function(){
       assert( true, "First test completed" );
       assert( false, "Second test failed" );
       assert( true, "Third assertion completed" );
     });

Tests
   var results;

   function assert( value, desc ) {
     var li = document.createElement("li");
     li.className = value ? "pass" : "fail";
     li.appendChild( document.createTextNode( desc ) );
     results.appendChild( li );
     if ( !value ) {
       li.parentNode.parentNode.className = "fail";
     }
     return li;
   }

   function test(name, fn){
     results = document.getElementById("results");
     results = assert( true, name ).appendChild(
       document.createElement("ul") );
     fn();
   }

Async Tests
test("Async Test #1", function(){
   pause();
   setTimeout(function(){
     assert( true, "First test completed" );
     resume();
   }, 1000);
});

test("Async Test #2", function(){
   pause();
   setTimeout(function(){
     assert( true, "Second test completed" );
     resume();
   }, 1000);
});

Async Tests
(function(){
   var queue = [], paused = false;

   this.test = function(fn){
     queue.push( fn );
     runTest();
   };

   this.pause = function(){
     paused = true;
   };

   this.resume = function(){
     paused = false;
     setTimeout(runTest, 1);
   };

   function runTest(){
     if ( !paused && queue.length ) {
       queue.shift()();
       if ( !paused ) {
         resume();
       }
     }
   }
})();

People don’t test. :-(

900

675

450

225

0

None

Popular Test Frameworks

300

225

150

75

0
QUnit JSUnit Selenium YUITest FireUnit Screw.Unit JSSpec

Unit Testing
✦ Break code into logical chucks for testing.
✦ Focus on one method at a time.
✦ Good for testing APIs.
✦ Popular Frameworks:
✦ QUnit
✦ JSUnit
✦ YUITest
✦ FireUnit

JSUnit
✦ One of the oldest JavaScript testing
frameworks.
✦ A port of JUnit to JavaScript, circa 2001.
✦ Code feels very 2001 (frames!)
✦ http://www.jsunit.net/

JSUnit
       function coreSuite() {
           var result = new top.jsUnitTestSuite();
           result.addTestPage("tests/jsUnitAssertionTests.html");
           result.addTestPage("tests/jsUnitSetUpTearDownTests.html");
           result.addTestPage("tests/jsUnitRestoredHTMLDivTests.html");
           result.addTestPage("tests/jsUnitFrameworkUtilityTests.html");
           result.addTestPage("tests/jsUnitOnLoadTests.html");
           result.addTestPage("tests/jsUnitUtilityTests.html");
           return result;
       }

       function serverSuite() {
           result.addTestPage("tests/server/jsUnitVersionCheckTests.html");
           result.addTestPage("tests/server/jsUnitServerAjaxTests.html");
           return result;
       }

       function librariesSuite() {
           result.addTestPage("tests/jsUnitMockTimeoutTest.html");
           return result;
       }

       function suite() {
           var newsuite = new top.jsUnitTestSuite();
           newsuite.addTestSuite(coreSuite());
           newsuite.addTestSuite(serverSuite());
           newsuite.addTestSuite(librariesSuite());
           return newsuite;
       }

JSUnit
function testAssertNotUndefined() {
     assertNotUndefined("1 should not be undefined", 1);
     assertNotUndefined(1);
}

function testAssertNaN() {
     assertNaN("a string should not be a number", "string");
     assertNaN("string");
}

function testAssertNotNaN() {
     assertNotNaN("1 should not be not a number", 1);
     assertNotNaN(1);
}

function testFail() {
     var excep = null;
     try {
         fail("Failure message");
     } catch (e) {
         excep = e;
     }
     assertJsUnitException("fail(string) should throw a JsUnitException", excep);
}

function testTooFewArguments() {
     var excep = null;
     try {
         assert();
     } catch (e1) {
         excep = e1;
     }
     assertNonJsUnitException("Calling an assertion function with too
few arguments should throw an exception", excep);
}

YUITest (2 & 3)
✦ Testing framework built and developed by
Yahoo (released Oct 2008).
✦ Completely overhauled to go with YUI v3.
✦ Features:
✦ Supports async tests.
✦ Has good event simulation.

✦ v2: http://developer.yahoo.com/yui/
examples/yuitest/
✦ v3: http://developer.yahoo.com/yui/3/test/

YUITest 2
  YAHOO.example.yuitest.ArrayTestCase = new YAHOO.tool.TestCase({
     name : "Array Tests",

     setUp : function () {
         this.data = [0,1,2,3,4]
     },

     tearDown : function () {
         delete this.data;
     },

     testPop : function () {
         var Assert = YAHOO.util.Assert;
         var value = this.data.pop();

         Assert.areEqual(4, this.data.length);
         Assert.areEqual(4, value);
     },

     testPush : function () {
         var Assert = YAHOO.util.Assert;

         this.data.push(5);

         Assert.areEqual(6, this.data.length);
         Assert.areEqual(5, this.data[5]);
     }
});

YUITest 3
Y.example.test.DataTestCase = new Y.Test.Case({
     name : "Data Tests",

     setUp : function () {
         this.data = {
             name: "test",
             year: 2007,
             beta: true
         };
     },

     tearDown : function () {
         delete this.data;
     },

     testName : function () {
         var Assert = Y.Assert;

         Assert.isObject(this.data);
         Assert.isString(this.data.name);
         Assert.areEqual("test", this.data.name);
     },

     testYear : function () {
         var Assert = Y.Assert;

         Assert.isObject(this.data);
         Assert.isNumber(this.data.year);
         Assert.areEqual(2007, this.data.year);
     }
});

QUnit
✦ Unit Testing framework built for jQuery.
✦ Features:
✦ Supports asynchronous testing.
✦ Can break code into modules.
✦ Supports test timeouts.
✦ No dependencies.
✦ Painfully simple.

✦ http://docs.jquery.com/QUnit

QUnit Style
test("a basic test example", function() {
   ok( true, "this test is fine" );
   var value = "hello";
   equals( "hello", value, "We expect value to be hello" );
});

module("Module A");

test("first test within module", function() {
   ok( true, "all pass" );
});

test("second test within module", function() {
   ok( true, "all pass" );
});

module("Module B");

test("some other test", function() {
   expect(2);
   equals( true, false, "failing test" );
   equals( true, true, "passing test" );
});

FireUnit
✦ Unit testing extension for Firebug
✦ ﬁreunit.ok( true, “...” );
✦ http://ﬁreunit.org/

Standardization
✦ CommonJS: A uniﬁed cross-platform API
for JavaScript.
✦ (Including the server-side!)
✦ Working to standardize a simple testing
API.
✦ http://wiki.commonjs.org/wiki/CommonJS

Server-Side
✦ Ignore the browser! Simulate it on the
server-side.
✦ Almost always uses Java + Rhino to
construct a browser.
✦ Some frameworks:
✦ Crosscheck
✦ Env.js
✦ Blueridge

Server-Side
✦ Crosscheck
✦ Pure Java, even simulates browser bugs.
✦ http://www.thefrontside.net/crosscheck

✦ Env.js
✦ Pure JavaScript, focuses on standards
support.
✦ http://github.com/thatcher/env-js/tree/
master
✦ Blueridge
✦ Env.js + Screw.Unit + Rhino
✦ http://github.com/relevance/blue-ridge/

Env.js
$ java -jar build/js.jar
Rhino 1.6 release 6 2007 06 28
js> load('build/runtest/env.js');

js> window.location = 'test/index.html';
test/index.html

js> load('dist/jquery.js');

// Add pretty printing to jQuery objects:
js> jQuery.fn.toString = DOMNodeList.prototype.toString;

js> $('span').remove();
[ <span#å°åŒ—Taibei>, <span#å°åŒ—>, <span#utf8class1>,
<span#utf8class2>, <span#foo:bar>, <span#test.foo[5]bar> ]

// Yes - UTF-8 is supported in DOM documents!
js> $('span')
[ ]

js> $('div').append('<span><b>hello!</b> world</span>');
[ <div#main>, <div#foo> ]

js> $('span')
[ <span>, <span> ]

js> $('span').text()
hello! worldhello! world

Browser Launching
✦ Automates the process of opening browser
windows, running tests, and getting results.
✦ Frequently require a speciﬁc framework.
✦ Popular frameworks:
✦ WebDriver http://code.google.com/p/
webdriver/ (Java)
✦ Waitr http://wtr.rubyforge.org/ (Ruby)
✦ JsTestDriver http://code.google.com/p/
js-test-driver/ (Java)
✦ Selenium RC http://seleniumhq.org/
projects/remote-control/ (Java)

Distributed
✦ Selenium Grid
✦ Push Selenium tests out to many
machines (that you manage),
simultaneously.
✦ Collect and store the results.
✦ http://selenium-grid.seleniumhq.org/

✦ TestSwarm
✦ Push tests to a distributed swarm of
clients.
✦ Results viewable on the server.
✦ http://testswarm.com/

The Scaling Problem
✦ The Problem:
✦ jQuery has 6 test suites
✦ Run in 15 browsers
✦ (Not even including multiple platforms
or mobile browsers!)
✦ All need to be run for every commit,
patch, and plugin.
✦ JavaScript testing doesn’t scale well.

Distributed Testing
✦ Hub server
✦ Clients connect and help run tests
✦ A simple JavaScript client that can be run
in all browsers
✦ Including mobile browsers!

✦ TestSwarm

FF 3.5 FF 3.5 FF 3.5
IE 6
IE 6
FF 3 IE 6
Op 9
FF 3

IE 7

TestSwarm
IE 7

Test Suite Test Suite Test Suite

TestSwarm.com
✦ Incentives for top testers (t-shirts, books)
✦ Will be opening for alpha testing very soon
✦ Help your favorite JavaScript library
become better tested!
✦ http://testswarm.com

Accurately Measuring
JavaScript

Major Cases
✦ Same Code, Diﬀerent Platforms
✦ Compare V8 vs. SpiderMonkey vs.
JavaScriptCore
✦ Diﬀerent Code, Same Platform
✦ Compare CSS Selector Engines
✦ A/B testing a piece of code

Same Code, Diﬀerent Platform
✦ A number of suites analyzing JS perf:
✦ SunSpider (from WebKit)
✦ V8 Benchmark (from V8/Chrome)
✦ Dromaeo (from Mozilla)

✦ Statistical accuracy and reproducibility is
paramount.

SunSpider
✦ All tests were highly balanced.
✦ Provide some level of statistical accuracy.
✦ +/- 5ms (for example)

✦ Tests are run by loading an iframe with the
test 5 times.
✦ getTime() is run before/after each test.

✦ Entire suite must be trashed in order to
upgrade/ﬁx a test.

Error Rate?
✦ How do we get it? What does it mean?
✦ It’s how conﬁdent we are that we arrived
at the result we want in the number of
runs that we’ve done.

Normal Distribution
✦ First: Assume that the results are coming
back in a normal distribution.
✦ The “bell curve”

Conﬁdence
✦ Next: We need a conﬁdence level.
✦ T-Distribution works well here.

http://en.wikipedia.org/wiki/Student%27s_t-distribution

Error Rate
✦ 5 runs
✦ (each run is potentially 1000s of
individual test runs)
✦ 95% Conﬁdence (t-distribution = 2.776)
✦ Standard Errors Mean =
✦ (std_dev / sqrt(runs)) * 2.776

✦ Error = (err_mean / mean) * 100
✦ This way you can get results like:
✦ 123ms +/- 5ms

V8 Benchmark
✦ Tests are run, potentially, 1000s of times.
✦ Also provides an error rate.
✦ (Use a geometric mean to arrive at a
result.)

Small Time Accuracy
✦ Small time:
✦ 1ms, 1ms, 1ms, 1ms, 3ms
✦ huge error!

✦ Large time:
✦ 1234ms, 1234ms, 1234ms, 1234ms, 1238ms
✦ tiny error!

✦ Tests that run faster need to be run more
times.
✦ Running more times = less potential for
weird results.
http://ejohn.org/blog/javascript-benchmark-quality/

Runs/Second
✦ var start = (new Date).getTime();
while (time < 1000) {
runTest();
time = (new Date).getTime() - start;
}
✦ More test runs, more statistical accuracy.
✦ V8 & Dromaeo-style suites handle this.
✦ (Problem: getTime() is being run on every
loop - it should be run less frequently in
order to inﬂuence the numbers less.)

Runs/Second
✦ You are now measuring tests/second rather
than seconds per test.
✦ You run tests as many times in one second
as you can.
✦ Then you do that multiple times (5?)
✦ THEN you analyze the ﬁnal numbers:
✦ 1234run/s, 1230runs/s, 1240runs/s, ...

Harmonic Mean
✦ A way to average rates
✦ Which is what we have! runs/second

✦ For example:
✦ 1234run/s, 1230runs/s, 1240runs/s,
1236runs/ms, 1232runs/s
✦ 5 / ( (1/1234) + (1/1230) + (1/1240) + (1/1236)
+ (1/1232) ) =
✦ 1234.39runs/s!
http://en.wikipedia.org/wiki/Harmonic_mean

Dromaeo
✦ All individual tests are versioned
✦ Makes it easy to update or fix a bug in a
test
✦ Can only run tests of specific versions
against each other
✦ Uses V8’s style of running tests.
✦ Also has DOM and framework tests.
✦ ...and hooks for doing Shark profiling.

Bug Fixes
✦ Tests will, inevitably, have bugs that need
to be ﬁxed.
✦ Fixing a bug changes the result quality.

✦ Tests need to be versioned so that changes
can be made.
✦ You look at Test v1 vs. Test v1 results.
✦ Not Test v2 vs. Test v1.

✦ Tip: Just use the last revision control
commit # for the test ﬁle.

Diﬀerent Code, Same Platform
✦ Most solutions here are very poor.
✦ Run the test very few times, use getTime().
✦ Highly inaccurate results, massive error.

Garbage Collection
✦ Browsers periodically run garbage
collectors to clean up old objects no longer
referenced.
✦ This can take a long time and spike your
test results.
✦ Example:
✦ 10ms, 13ms, 11ms, 12ms, 486ms, 12ms, ...

✦ When comparing engine to engine, this
doesn’t matter.
✦ Comparing code vs. code, it does.

Mean, Median, Mode?
✦ Mode!
✦ Run your tests a large number of times.
✦ What is the ‘mode’ (the result that
occurs most frequently)
✦ Example:
✦ 10, 11, 11, 12, 12, 12, 13, 14
✦ Mode = 12ms.

✦ Less accurate than mean, but gives you a
more-consistent result.
✦ DON’T DISCARD “BAD” RESULTS!

getTime() Accuracy

http://ejohn.org/blog/accuracy-of-javascript-time/

15ms intervals ONLY!

Error Rate of 50-750%!

IE in Wine
✦ Running Internet Explorer in Wine (on
Linux) gives ﬁne-grained timer results
✦ Down to the millisecond!

✦ You can also run IE, in Wine, on OS X:
✦ ies4osx

✦ Huge Caveat: It gives you ﬁne-grained
time, but that doesn’t mean it’s accurate.

Diﬀerent Code, Same Platform
✦ How can we get good numbers?
✦ We have to go straight to the source: Use
the tools the browsers provide.

Shark Proﬁling
✦ Extremely low-level
✦ Watch for all internal function calls
✦ See what gets called the most.

✦ Dromaeo includes shark proﬁling hooks.

Interesting Problems
Tackled in jQuery 1.4

Tackled in 1.4
✦ Complexity reduction
✦ Bubbling change, submit, focus, blur
✦ Required script loading

Questions
✦ Contact:
✦ John Resig
✦ http://ejohn.org/
✦ http://twitter.com/jeresig

Testing, Performance Analysis, and jQuery 1.4

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to Testing, Performance Analysis, and jQuery 1.4

Similar to Testing, Performance Analysis, and jQuery 1.4 (20)

More from jeresig

More from jeresig (20)

Recently uploaded

Recently uploaded (20)

Testing, Performance Analysis, and jQuery 1.4