Testing Ember Apps

Testing Ember Apps Jo Liss https://twitter.com/jo_lissHi, I’m Jo, I’m on the http://solitr.com/blogCapybara and Konachacore teams, and I’m anEmber contributor. This talk will be more an architectural over view than a tutorial. I’ll assume Rails in some places, but the concepts apply to other backends as well. Slides licensed under CC BY 3.0.

This presentation has 2 parts:Part 1: Full-Stack Integration Tests Capybara similar: PhantomJS

Part 1: Full-Stack Integration Tests CapybaraPart 2: Client-Side Integration Tests Konacha similar: QUnit, Mocha, Jasmine I like to test apps with a combination of Capybara and Konacha tests. I noticed that having a good test suite makes me *much* more productive.

Part 1: Full-Stack Integration Tests Capybara It’s not very hard to get Capybara working right. So I want to focus on high-level architecture instead. I think that’s sometimes under- appreciated.

CapybaraQuick refresher: “Selenium for Rails” You can plug in different Q: Who has used Capybara to drivers, like WebKit. All test a JavaScript app? of the following applies -------> to any driver. Q: Who ran into problems with either performance, or flickering tests (brittleness)?

CapybaraThese pains are architectural. These pains are because of the architecture.

CapybaraThese pains are architectural. Performance :-( Brittleness :-( There are probably limits So let’s talk about why to how fast you can this happens. make Capybara tests. And you can hunt down flickering tests (and you should), but they will still pop up occasionally.

So you have a test suitewritten in Ruby, and ittalks to a DB backend. Test suite DB

Firefox + So when you have a Capybara test, Capybara will automatically spin up aSelenium Firefox for you, and your test suite will send requests to that Firefox. Test suite But Selenium commands are mostly non-blocking, i.e. they return before they finish. So when you have a DB sequence of commands, even with Ajax, Capybara does some intelligent polling to make them appear synchronous. This works transparent 99% of the time, and it makes for very readable test code, but very occasionally, you’ll end up with a race condition, and then you have to understand what’s going on underneath. So let me double that line to indicate that it’s async.

So who does the Firefox makeFirefox + HTTP requests to? Clearly, Ruby is busy with the test suite.Selenium Capybara actually spins up another ser ver thread. Test suite DB

Firefox + Selenium Server Test suiteSo now Firefox has aserver to load the pages DBfrom.If there’s Ajax requests,this whole thing becomesasynchronous. So that’sanother source of raceconditions. Let me doublethat line...

Firefox + Selenium Server Test suiteAnd of course the servertalks to the database -- DBanother line...

Firefox + Selenium Server Test suiteWhich means you also This is clearly fundamentally quite complexhave to be careful not tohave the t wo Ruby DB and error-prone.threads interfere with I’m not trying to convince you that Capybaraeach other while they is a bad tool. But I think it’s actually useful toaccess the database. understand this architecture. By the way, just to be clear, that’s not Selenium’s fault. If you drop in WebKit or PhantomJS into Capybara, it suffers from the same issues.

Firefox + Selenium At the moment, this kind of architecture is the only solution to give us a full-stack integration test, database to DOM. So it’s actually very useful.Server If you don’t use Rails, I’d definitely Test suite recommend finding a comparable solution, or hand-rolling one. DB

CapybaraPowerful but clunky So one solid strategy ...

... is limiting yourself to exercising one happy path for Capybara each model, to make sure that stuff in the DB ends up in the DOM, and vice versa.Powerful but clunkyStrategy: Only test every model once DB-to-DOM, read and write

And then we can use pure client- side integration testing to get Capybara more fine-grained testing.Powerful but clunkyStrategy: Only test every model once DB-to-DOM, read and writeMove ﬁner-grained tests to client side

Part 2: Client-Side Integration Tests with Konacha The idea is to limit the architectural complexity. No backend server, no DB, test runs directly in the browser, not in a separate process. This makes it faster, and extremely reliable.

KonachaRails gem packaging Konacha is prettyMocha.js testing framework + simple, so if you’re not on Rails, you can easily hand-roll an equivalent test setup.Chai.js assertion lib Let me show you what Mocha and Chai do...

Mocha + Chaidescribe todo list, -> it can add items, -> Mocha is a testing framework, like QUnit or Jasmine, or RSpec on Ruby. But Mocha doesn’t do assertions, so you typically combine it with the Chai assertion library.

Mocha + Chaidescribe todo list -> it can add items, -> ... $(.todo).should .contain Buy milk

Interlude: Konacha Video Before I go on, let me show you what we’re trying to achieve.http://www.youtube.com/watch?v=heK78M6Ql9Q So we basically just hit Cmd+R on Konacha’s development ser ver. Konacha automatically runs your tests in an iframe, and loads your application.css. That’s also really easy to do yourself if you don’t use Rails with Konacha. This is also really useful to ramp up people to Ember, because you can visually see what your test does. And if a test fails, you can just click into the iframe and interact with the application.

KonachaIn the browser (Cmd+R) for devrake task for CI CI task through Capybara

Why is Konacha fast?No page loads You don’t have to serve and parse the entire stack of assets for every test case.

Why is Konacha fast?No page loads100% synchronous No polling, no waiting.

Why is Konacha fast?No page loads100% synchronousNo stack Most expensive thing is the DOM.

Unit vs. IntegrationWhat you just saw I call “client-sideintegration tests”. They don’t involve thebackend, but on the frontend, they exercisethe entire Ember app.

Unit vs. IntegrationLots of simple layers ==> integrationtests win. On Ember, I avoid unit tests. The reason is, the individual layers of an Ember app are very simple. What I care about is whether they play together. Unit tests tend to be very “weak”, i.e. they’ll just keep passing even when your app Okay, bread’n’butter breaks. time. As a rule of thumb, I tend to do zero unit testing for Ember apps.

Ember setupNo “ofﬁcial” support for testing.So we wing it. A bunch of pieces are missing to make this easy like with Rails. (Notably, good fixture handling, package manager, ...) Here’s some practical tips:

There’s no repeated appinstantiation yet, so Ilike to start up the Starting the appapplication at thebeginning of the entiretest suite.

Starting the app# Parse time: We call App.deferReadiness, and advance readiness itApp.deferReadiness() once you want the app to run.# Global before all:before -> UPDATE Feb 26 2013: App.advanceReadiness() Instead you can now Enable Ember.testing *before* parsing your app code, and kick it off with “before -> App.initialize()”; no more advanceReadiness.

Router We tell the router not to mess with the URL.App.Router.reopen location: none

RouterApp.Router.reopen location: nonebeforeEach: -> App.router.transitionTo(...) Before each test, UPDATE Feb 26 2013: transition back to the root state. You can now use “beforeEach -> This is a pretty hackish App.reset()”. way to reset app state, but for now it’s surprisingly reliable.

Reset storebeforeEach -> App.store.destroy() if App.store App.store = DS.Store.create()

Konacha: Keep bodyKonacha.reset = function() { } Konacha by default cleans the <body> element before each test. But our app keeps running bet ween tests, so we disable this by nuking Konacha.reset.

Runloop & setTimeout Ember automatically schedules runloops with setTimeout. But setTimeout isEmber.testing = true the enemy of deterministic tests. So we disable automatic runloop creation by enabling the Ember.testing flag. You want to put this before loading (parsing) your app modules.

Runloop & setTimeoutEmber.testing = trueIt’s OK to have Em.run everywhere:Em.run => foo.set ...Em.run => $(...).click() Most actions have their effects deferred to the end of the runloop. In test code you need the effects immediately, so you wrap things in Ember.run. Looks funny, but it’s nothing to worry about. :-)

AnimationsjQuery.fx.off = trueNothing like this for D3. :-( This is not Ember-related, but you don’t want animations, because they are asynchronous. For D3,, this may require support in your production code. :-( Probably the trickiest thing is data fixtures:

Model Fixtures1. Client-side ﬁxtures. You have a choice whether you write them in JavaScript or2. Server-side ﬁxtures. generate them from the server side, and it’s kind of a trade off.

(1) Client-Side FixturesFixtureAdapter (immature)App.TodoList.FIXTURES = [ { ... }, { ... } For client-side fixtures,] there is a FixtureAdapter in Ember. It still needs some love, but we can probably get it there pretty soon. And basically you just define an array of fixtures for every model.

(1) Client-Side Fixtures:-) Easy to set up

(1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend But you don’t know if your fixtures actually represent the reality of your backend.

(1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend:-( Fragile You can even have bugs in your fixtures, where you don’t set up bidirectional belongsTo and hasMany relationships properly.

(1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend:-( Fragile:-( Server-side computed attributes JavaScript is still not very powerful. Oftentimes it’s So the alternative to all easier in practice to implement computed properties on the this is... backend side and ser ve them out as read-only attributes. In one backend I worked on, half of the properties were DB columns, and half were just methods on the Rails models. Trying to keep these properties manually updated in your fixture data is obviously painful.

(2) Server-Side Fixtures rake test:fixtures1. Write ﬁxtures to DB2. Generate JSON to ﬁxtures.js

(2) Server-Side Fixtures rake test:fixtures1. Write ﬁxtures to DB2. Generate JSON to ﬁxtures.js Load through RESTAdapterLoad that data in fixtures.js before every testcase, perhaps using your RESTAdapter so youtranslate the JSON correctly.

(2) Server-Side Fixtures :-) Covers models, serializers, adapter with no speed penalty

(2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain Compact definitions, esp. w/ FactoryGirl; stuff doesn’t go out of sync.

(2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain :-( Usability Generated fixtures file can go stale and you have to regenerate. It’s not bad, just bothersome.

(2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain :-( Usability :-( Complex to set up Work to set up. You end up with some custom code, and it ties tightly into backend. I personally think it’s generally worth it, but it also depends on your specific app.

Fixture WoesGlobal ﬁxtures :-( Both of these techniques mean that you have the same fixture set for the entire test suite. want FactoryGirl Ideally we’d build something like FactoryGirl, but I don’t think we’re quite there yet.

Bonus: JS-driven?Capybara but in JavaScript? One thing that sometimes comes up is, can we have a full-stack integration test written in JavaScript?

Firefox + So instead of having the Selenium Test suite written in Ruby...Server Test suite DB

Firefox w/ JS tests ... we push them into the browser, and so we avoid the t wo-threaded problem.Server I think that would be really awesome -- and the truth is, we just don’t have the tooling yet to make that work easily. Perhaps the biggest hurdle DB is that we don’t have a way to reset the database, add fixtures, and perhaps query database records from JavaScript. But it is probably a good direction to move towards

Q &ANotes from the talk, thanks to @cgcardona:Q: With Konacha do you see opportunity for TTD?A: Not quite TDD (hard with visual stuff), but continuous.When working with Konacha I wrote tests as I went.Q: Can you test views in isolation?A: It’s really tricky to instantiate a view in isolation, Emberwants a whole app… It’s too ‘unit testy’. It might bepossible.

Q &AComment: We use ‘Rosie’ (https://github.com/bkeepers/rosie) for Javascript factories.Comment: Have you tried VCR for server side things? It’s aRuby library that will record request responses so you canplay them back later. We run VCR against a live productionserver and generate response data that the tests use.

Q &AQ: A lot of bugs come from asynchronicity. Have you triedto test that speciﬁcally?A: No I haven’t. My hope is that a lot of these bugsdisappear with ember-data. Mocha allows for asynchronoustests where you set a callback for when your test iscomplete.

Thanks for listening!Johttps://twitter.com/jo_lisshttp://solitr.com/blog

http://lanyrd.com	865
https://twitter.com	257
http://staging.twylah.com	13
http://twylah.local	12
http://www.twylah.com	9
http://web-server-2	5
http://x-web-64-1	2
http://ec2-23-20-140-51.compute-1.amazonaws.com	2
http://www.mefeedia.com	1
http://www.diffbot.com&_=1361129865970 HTTP	1
http://web-server-1	1
http://web-server-5	1

Testing Ember Apps

by jo_liss on Feb 15, 2013

Accessibility

Categories

Upload Details

Usage Rights

12 Embeds 1,169

Statistics