Automated
Frontend Testing
NeilCrosby
(Image from “The Korean Robot” - http://www.flickr.com/photos/zebrapares/1344995547/)

Who am I?
@NeilCrosby
European Frontend Architect for Search
at Yahoo!
I help look after frontend code quality.

What’s this talk about?
• Automatically and regularly testing frontend
code against known standards.
• Why? What? How? Where?

Why?
What? How?
Where?

Why test?
• We all want to write code that works.
• Testing improves reliability.
• Repeatedly testing minimises regressions.
• Automatic testing can pick up “dumb-ass”
mistakes.

That sounds lovely
• Normally that doesn’t happen though.
• Normally what happens is...

1. We build things.
They adhere to our
standards.

2. We go to the pub.

3. Later, we add more
code.

4. Our code stops
adhering to our
standards.

5. Things become less
easy to maintain.

6. Bugs creep in.

7. We go to the pub in
despair.

8. Repeat.

So...
• Test automatically and often.
• By adhering to a known standard, and
constantly testing against it, it becomes
easier to spot problems with the code that
we're writing.

And that saves money
Fewer bugs
==
Less time fixing bugs
==
More time developing new features.

Why?
What? How?
Where?

What’s normally tested
Backend Frontend In-Browser
(API, functions) (HTML, CSS, JS) (Functional)
Validators, JsLint,
*Unit Selenium
YSlow etc.
Sometimes
Automatic via CI Manual
automatic via CI
Adhoc, when
Well defined Fairly well defined
remembered

Why not use Selenium?
• Selenium tests code after it’s been
interpreted by the browser.
• Good for functional testing, not so good for
testing discrete units of code.

Discrete Units?
• I’m looking at testing the quality of our
HTML, CSS and JavaScript.
• It should all adhere to a known standard.

So we’re testing what?
• The code that leaves the server.
• Before it is interpreted by the browser.

What should we test?
• Currently I’m testing against:
• W3C HTML Validator validator.w3.org
• W3C CSS Validator jigsaw.w3.org/css-validator
• JsLint jslint.com

Validation is a dirty word
• People don’t like the word.
• They always ask “what if?”.

Custom DTD
• Custom DTDs can help you create your
own internal standard.

Custom DTDs are Easy
<!ENTITY % HTML.strict SYSTEM \"strict.dtd\">
%HTML.strict;
<!ATTLIST OL
%attrs; -- %coreattrs, %i18n, %events --
start NUMBER #IMPLIED -- starting sequence number --
>

Why?
What? How?
Where?

How do I do this testing?
• I’ve been writing a test suite!
• http://github.com/NeilCrosby/frontend-
test-suite (CC Attribution-Share Alike)

Currently in Phase 1
• Tests can easily be run against known units
of HTML, CSS and JavaScript.
• I’m using phpunit as the framework.

HTML Testing
• By default tests against HTML 4.01 Strict.
• Test against custom DTDs.
• Module and full page testing.
• Test files, URLs or strings.

CSS Testing
• By default tests against CSS 2.1.
• Known exceptions to the standards can be
allowed to pass validation.
• Test files, URLs or strings.

JavaScript Testing
• Test against any configuration JsLint allows.
• Test files, URLs or strings.

Add to your CI
• Run these tests in your hourly builds.
• Run them as a pre-commit hook.

Be a good citizen
• You’ll be running these tests frequently.
• Set up your own local versions of the
HTML Validator, CSS Validator and JsLint.
• Instructions available on the web.

Running these tests
• If you need to test a simple site, it’s easy -
pass a config object to
TheCodeTrainEasyFrontendTestSuite.
• Otherwise, write custom TestCase
extensions (still not hard).

Including the Suite
require_once(
'TheCodeTrainEasyFrontendTestSuite.php'
);

Creating the Tests
class SomeTestSuite extends
TheCodeTrainEasyFrontendTestSuite {
public static function suite() {
return parent::suite(array(
‘html’ => array( ... ),
‘css’ => array( ... ),
‘js’ => array( ... ),
));
}
}

Creating the Tests
class SomeTestSuite extends
TheCodeTrainEasyFrontendTestSuite {
public static function suite() {
return parent::suite(array(
‘html’ => array( ... ),
‘css’ => array( ... ),
‘js’ => array( ... ),
));
}
}

Creating the Tests
class SomeTestSuite extends
TheCodeTrainEasyFrontendTestSuite {
public static function suite() {
return parent::suite(array(
‘html’ => array( ... ),
‘css’ => array( ... ),
‘js’ => array( ... ),
));
}
}

Creating the Tests
class SomeTestSuite extends
TheCodeTrainEasyFrontendTestSuite {
public static function suite() {
return parent::suite(array(
‘html’ => array( ... ),
‘css’ => array( ... ),
‘js’ => array( ... ),
));
}
}

The ‘html/css/js’ arrays
‘html’ => array(
‘validator’ => ‘http://yourvalidator’,
‘tests’ => array(
‘http://someurl.to/test’,
‘file://some/file/to/test’,
‘<p>Some string to test</p>’,
),
‘options’ => array( ... )
)

The ‘html/css/js’ arrays
‘html’ => array(
‘validator’ => ‘http://yourvalidator’,
‘tests’ => array(
‘http://someurl.to/test’,
‘file://some/file/to/test’,
‘<p>Some string to test</p>’,
),
‘options’ => array( ... )
)

The ‘html/css/js’ arrays
‘html’ => array(
‘validator’ => ‘http://yourvalidator’,
‘tests’ => array(
‘http://someurl.to/test’,
‘file://some/file/to/test’,
‘<p>Some string to test</p>’,
),
‘options’ => array( ... )
)

The ‘html/css/js’ arrays
‘html’ => array(
‘validator’ => ‘http://yourvalidator’,
‘tests’ => array(
‘http://someurl.to/test’,
‘file://some/file/to/test’,
‘<p>Some string to test</p>’,
),
‘options’ => array( ... )
)

The Options Array
‘options’ => array(
‘doctype_override’ =>
‘<!DOCTYPE HTML SYSTEM
\"http://site/doctype.dtd\">’,
‘document_section’ =>
TheCodeTrainHtmlValidator::HTML_CHUNK,
‘document_section_position’ =>
TheCodeTrainHtmlValidator::POSITION_HEAD
)

The Options Array
‘options’ => array(
‘doctype_override’ =>
‘<!DOCTYPE HTML SYSTEM
\"http://site/doctype.dtd\">’,
‘document_section’ =>
TheCodeTrainHtmlValidator::HTML_CHUNK,
‘document_section_position’ =>
TheCodeTrainHtmlValidator::POSITION_HEAD
)

The Options Array
‘options’ => array(
‘doctype_override’ =>
‘<!DOCTYPE HTML SYSTEM
\"http://site/doctype.dtd\">’,
‘document_section’ =>
TheCodeTrainHtmlValidator::HTML_CHUNK,
‘document_section_position’ =>
TheCodeTrainHtmlValidator::POSITION_HEAD
)

The Options Array
‘options’ => array(
‘doctype_override’ =>
‘<!DOCTYPE HTML SYSTEM
\"http://site/doctype.dtd\">’,
‘document_section’ =>
TheCodeTrainHtmlValidator::HTML_CHUNK,
‘document_section_position’ =>
TheCodeTrainHtmlValidator::POSITION_HEAD
)

Options?
• Custom doctypes, full page or module,
position on page.
• Options can also be given for individual
tests.
array(
‘file://some/file/to/test’,
array( your_options )
)

Running these tests
> phpunit SomeTestSuite
PHPUnit 3.3.1 by Sebastian Bergmann.
.........
Time: 7 seconds
OK (9 tests, 9 assertions)

Some failures
....F.....
Failed asserting that
Array
(
[0] => Array
(
[line] => 7
[errortype] =>
[error] => syntax of attribute value does not conform to declared value
[original_line] => <label for=\"\">Some label</label>
)
)
is false.

Some failures
.....F....
Failed asserting that
Array
(
[0] => Array
(
[line] => 1
[errortype] => parse-error
[error] => Value Error : display (http://www.w3.org/TR/CSS21/
visuren.html#propdef-display) inlin is not a display value :
[original_line] => p
)
)
is false.

Why?
What? How?
Where?

Where’s this used?
• Yahoo! - SearchMonkey
• TheCodeTrain.co.uk’s WordPress theme.

It’s found problems
• Security holes.
• Form usability issues.
• Other suspect HTML.

The Future!

Phase 2 - The Future
• Test for things a simple DTD check can’t
pick up on.
• CodeSniffer.
• Whitelisting sections of code - e.g. adverts.
• Basic accessibility testing - working with the
experts at Yahoo!.

Available Online
• This talk: http://icanhaz.com/aft
• The Code (please fork and contribute):
http://github.com/NeilCrosby/frontend-
test-suite
• Twitter: automated frontend testing,
@NeilCrosby
• My blog hub: http://neilcrosby.com