Your SlideShare is downloading. ×
0
Testing Ember Apps                                          Jo Liss                         https://twitter.com/jo_lissHi,...
This presentation has 2 parts:Part 1: Full-Stack Integration Tests                                  Capybara   similar: Ph...
Part 1: Full-Stack Integration Tests              CapybaraPart 2: Client-Side Integration Tests               Konacha   si...
Part 1: Full-Stack Integration Tests              Capybara                            It’s not very hard to get           ...
CapybaraQuick refresher:                            “Selenium for Rails”                   You can plug in different   Q: ...
CapybaraThese pains are architectural.                  These pains are because of the                  architecture.
CapybaraThese pains are architectural.       Performance :-(        Brittleness :-(         There are probably limits     ...
So you have a test suitewritten in Ruby, and ittalks to a DB backend.                                Test suite           ...
Firefox +           So when you have a                    Capybara test, Capybara                    will automatically sp...
So who does the Firefox makeFirefox +   HTTP requests to? Clearly, Ruby is            busy with the test suite.Selenium   ...
Firefox +                            Selenium     Server                             Test suiteSo now Firefox has aserver ...
Firefox +                           Selenium               Server                  Test suiteAnd of course the servertalks...
Firefox +                            Selenium            Server                          Test suiteWhich means you also   ...
Firefox +                  Selenium         At the moment, this kind of architecture is         the only solution to give ...
CapybaraPowerful but clunky                      So one solid strategy ...
... is limiting yourself to                           exercising one happy path for         Capybara                      ...
And then we can use pure client-                             side integration testing to get          Capybara            ...
Part 2: Client-Side Integration Tests                             with Konacha       The idea is to limit the       archit...
KonachaRails gem packaging                                               Konacha is prettyMocha.js testing framework +    ...
Mocha + Chaidescribe todo list, ->  it can add items, ->             Mocha is a testing             framework, like QUnit ...
Mocha + Chaidescribe todo list ->  it can add items, ->    ...    $(.todo).should      .contain Buy milk
Interlude: Konacha Video Before I go on, let me show you what we’re trying to achieve.http://www.youtube.com/watch?v=heK78...
KonachaIn the browser (Cmd+R) for devrake task for CI   CI task through Capybara
Why is Konacha fast?No page loads   You don’t have to serve and parse the entire                stack of assets for every ...
Why is Konacha fast?No page loads100% synchronous   No polling, no waiting.
Why is Konacha fast?No page loads100% synchronousNo stack   Most expensive thing is the DOM.
Unit vs. IntegrationWhat you just saw I call “client-sideintegration tests”. They don’t involve thebackend, but on the fro...
Unit vs. IntegrationLots of simple layers ==> integrationtests win. On Ember, I avoid unit tests. The reason is, the indiv...
Ember setupNo “official” support for testing.So we wing it.                A bunch of pieces are                           ...
There’s no repeated appinstantiation yet, so Ilike to start up the                          Starting the appapplication at...
Starting the app# Parse time:              We call                           App.deferReadiness, and                      ...
Router                     We tell the router not to                     mess with the URL.App.Router.reopen  location: none
RouterApp.Router.reopen  location: nonebeforeEach: -> App.router.transitionTo(...)       Before each test,          UPDATE...
Reset storebeforeEach ->  App.store.destroy() if App.store  App.store = DS.Store.create()
Konacha: Keep bodyKonacha.reset = function() { }                 Konacha by default                 cleans the <body>     ...
Runloop & setTimeout                       Ember automatically                       schedules runloops with              ...
Runloop & setTimeoutEmber.testing = trueIt’s OK to have Em.run everywhere:Em.run => foo.set ...Em.run => $(...).click()   ...
AnimationsjQuery.fx.off = trueNothing like this for D3. :-(              This is not Ember-related,                       ...
Model Fixtures1. Client-side fixtures.   You have a choice                          whether you write them                 ...
(1) Client-Side FixturesFixtureAdapter (immature)App.TodoList.FIXTURES = [    { ... }, { ... }                     For cli...
(1) Client-Side Fixtures:-) Easy   to set up
(1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend       But you don’t know if       your fixtures actually...
(1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend:-( Fragile   You can even have bugs              in your...
(1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend:-( Fragile:-( Server-side computed attributes JavaScript...
(2) Server-Side Fixtures  rake test:fixtures1. Write fixtures to DB2. Generate JSON to fixtures.js
(2) Server-Side Fixtures     rake test:fixtures1. Write fixtures to DB2. Generate JSON to fixtures.js     Load through RESTA...
(2) Server-Side Fixtures :-) Covers models, serializers, adapter               with no speed penalty
(2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain   Compact definitions, esp. w/ Facto...
(2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain :-( Usability   Generated fixtures f...
(2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain :-( Usability :-( Complex to set up ...
Fixture WoesGlobal fixtures :-(   Both of these techniques                     mean that you have the same                 ...
Bonus: JS-driven?Capybara but in JavaScript?                      One thing that                      sometimes comes up i...
Firefox +                       So instead of having the         Selenium      Test suite written in                      ...
Firefox w/           JS tests                      ... we push them into the                      browser, and so we avoid...
Q &ANotes from the talk, thanks to @cgcardona:Q: With Konacha do you see opportunity for TTD?A: Not quite TDD (hard with v...
Q &AComment: We use ‘Rosie’ (https://github.com/bkeepers/rosie) for Javascript factories.Comment: Have you tried VCR for s...
Q &AQ: A lot of bugs come from asynchronicity. Have you triedto test that specifically?A: No I haven’t. My hope is that a l...
Thanks for listening!Johttps://twitter.com/jo_lisshttp://solitr.com/blog
Upcoming SlideShare
Loading in...5
×

Testing Ember Apps

12,237

Published on

Testing Ember Apps: Full-stack integration testing with Capybara (Selenium & friends); client-side integration testing with Konacha (Mocha & friends).

Published in: Technology

Transcript of "Testing Ember Apps"

  1. 1. Testing Ember Apps Jo Liss https://twitter.com/jo_lissHi, I’m Jo, I’m on the http://solitr.com/blogCapybara and Konachacore teams, and I’m anEmber contributor. This talk will be more an architectural over view than a tutorial. I’ll assume Rails in some places, but the concepts apply to other backends as well. Slides licensed under CC BY 3.0.
  2. 2. This presentation has 2 parts:Part 1: Full-Stack Integration Tests Capybara similar: PhantomJS
  3. 3. Part 1: Full-Stack Integration Tests CapybaraPart 2: Client-Side Integration Tests Konacha similar: QUnit, Mocha, Jasmine I like to test apps with a combination of Capybara and Konacha tests. I noticed that having a good test suite makes me *much* more productive.
  4. 4. Part 1: Full-Stack Integration Tests Capybara It’s not very hard to get Capybara working right. So I want to focus on high-level architecture instead. I think that’s sometimes under- appreciated.
  5. 5. CapybaraQuick refresher: “Selenium for Rails” You can plug in different Q: Who has used Capybara to drivers, like WebKit. All test a JavaScript app? of the following applies -------> to any driver. Q: Who ran into problems with either performance, or flickering tests (brittleness)?
  6. 6. CapybaraThese pains are architectural. These pains are because of the architecture.
  7. 7. CapybaraThese pains are architectural. Performance :-( Brittleness :-( There are probably limits So let’s talk about why to how fast you can this happens. make Capybara tests. And you can hunt down flickering tests (and you should), but they will still pop up occasionally.
  8. 8. So you have a test suitewritten in Ruby, and ittalks to a DB backend. Test suite DB
  9. 9. Firefox + So when you have a Capybara test, Capybara will automatically spin up aSelenium Firefox for you, and your test suite will send requests to that Firefox. Test suite But Selenium commands are mostly non-blocking, i.e. they return before they finish. So when you have a DB sequence of commands, even with Ajax, Capybara does some intelligent polling to make them appear synchronous. This works transparent 99% of the time, and it makes for very readable test code, but very occasionally, you’ll end up with a race condition, and then you have to understand what’s going on underneath. So let me double that line to indicate that it’s async.
  10. 10. So who does the Firefox makeFirefox + HTTP requests to? Clearly, Ruby is busy with the test suite.Selenium Capybara actually spins up another ser ver thread. Test suite DB
  11. 11. Firefox + Selenium Server Test suiteSo now Firefox has aserver to load the pages DBfrom.If there’s Ajax requests,this whole thing becomesasynchronous. So that’sanother source of raceconditions. Let me doublethat line...
  12. 12. Firefox + Selenium Server Test suiteAnd of course the servertalks to the database -- DBanother line...
  13. 13. Firefox + Selenium Server Test suiteWhich means you also This is clearly fundamentally quite complexhave to be careful not tohave the t wo Ruby DB and error-prone.threads interfere with I’m not trying to convince you that Capybaraeach other while they is a bad tool. But I think it’s actually useful toaccess the database. understand this architecture. By the way, just to be clear, that’s not Selenium’s fault. If you drop in WebKit or PhantomJS into Capybara, it suffers from the same issues.
  14. 14. Firefox + Selenium At the moment, this kind of architecture is the only solution to give us a full-stack integration test, database to DOM. So it’s actually very useful.Server If you don’t use Rails, I’d definitely Test suite recommend finding a comparable solution, or hand-rolling one. DB
  15. 15. CapybaraPowerful but clunky So one solid strategy ...
  16. 16. ... is limiting yourself to exercising one happy path for Capybara each model, to make sure that stuff in the DB ends up in the DOM, and vice versa.Powerful but clunkyStrategy: Only test every model once DB-to-DOM, read and write
  17. 17. And then we can use pure client- side integration testing to get Capybara more fine-grained testing.Powerful but clunkyStrategy: Only test every model once DB-to-DOM, read and writeMove finer-grained tests to client side
  18. 18. Part 2: Client-Side Integration Tests with Konacha The idea is to limit the architectural complexity. No backend server, no DB, test runs directly in the browser, not in a separate process. This makes it faster, and extremely reliable.
  19. 19. KonachaRails gem packaging Konacha is prettyMocha.js testing framework + simple, so if you’re not on Rails, you can easily hand-roll an equivalent test setup.Chai.js assertion lib Let me show you what Mocha and Chai do...
  20. 20. Mocha + Chaidescribe todo list, -> it can add items, -> Mocha is a testing framework, like QUnit or Jasmine, or RSpec on Ruby. But Mocha doesn’t do assertions, so you typically combine it with the Chai assertion library.
  21. 21. Mocha + Chaidescribe todo list -> it can add items, -> ... $(.todo).should .contain Buy milk
  22. 22. Interlude: Konacha Video Before I go on, let me show you what we’re trying to achieve.http://www.youtube.com/watch?v=heK78M6Ql9Q So we basically just hit Cmd+R on Konacha’s development ser ver. Konacha automatically runs your tests in an iframe, and loads your application.css. That’s also really easy to do yourself if you don’t use Rails with Konacha. This is also really useful to ramp up people to Ember, because you can visually see what your test does. And if a test fails, you can just click into the iframe and interact with the application.
  23. 23. KonachaIn the browser (Cmd+R) for devrake task for CI CI task through Capybara
  24. 24. Why is Konacha fast?No page loads You don’t have to serve and parse the entire stack of assets for every test case.
  25. 25. Why is Konacha fast?No page loads100% synchronous No polling, no waiting.
  26. 26. Why is Konacha fast?No page loads100% synchronousNo stack Most expensive thing is the DOM.
  27. 27. Unit vs. IntegrationWhat you just saw I call “client-sideintegration tests”. They don’t involve thebackend, but on the frontend, they exercisethe entire Ember app.
  28. 28. Unit vs. IntegrationLots of simple layers ==> integrationtests win. On Ember, I avoid unit tests. The reason is, the individual layers of an Ember app are very simple. What I care about is whether they play together. Unit tests tend to be very “weak”, i.e. they’ll just keep passing even when your app Okay, bread’n’butter breaks. time. As a rule of thumb, I tend to do zero unit testing for Ember apps.
  29. 29. Ember setupNo “official” support for testing.So we wing it. A bunch of pieces are missing to make this easy like with Rails. (Notably, good fixture handling, package manager, ...) Here’s some practical tips:
  30. 30. There’s no repeated appinstantiation yet, so Ilike to start up the Starting the appapplication at thebeginning of the entiretest suite.
  31. 31. Starting the app# Parse time: We call App.deferReadiness, and advance readiness itApp.deferReadiness() once you want the app to run.# Global before all:before -> UPDATE Feb 26 2013: App.advanceReadiness() Instead you can now Enable Ember.testing *before* parsing your app code, and kick it off with “before -> App.initialize()”; no more advanceReadiness.
  32. 32. Router We tell the router not to mess with the URL.App.Router.reopen location: none
  33. 33. RouterApp.Router.reopen location: nonebeforeEach: -> App.router.transitionTo(...) Before each test, UPDATE Feb 26 2013: transition back to the root state. You can now use “beforeEach -> This is a pretty hackish App.reset()”. way to reset app state, but for now it’s surprisingly reliable.
  34. 34. Reset storebeforeEach -> App.store.destroy() if App.store App.store = DS.Store.create()
  35. 35. Konacha: Keep bodyKonacha.reset = function() { } Konacha by default cleans the <body> element before each test. But our app keeps running bet ween tests, so we disable this by nuking Konacha.reset.
  36. 36. Runloop & setTimeout Ember automatically schedules runloops with setTimeout. But setTimeout isEmber.testing = true the enemy of deterministic tests. So we disable automatic runloop creation by enabling the Ember.testing flag. You want to put this before loading (parsing) your app modules.
  37. 37. Runloop & setTimeoutEmber.testing = trueIt’s OK to have Em.run everywhere:Em.run => foo.set ...Em.run => $(...).click() Most actions have their effects deferred to the end of the runloop. In test code you need the effects immediately, so you wrap things in Ember.run. Looks funny, but it’s nothing to worry about. :-)
  38. 38. AnimationsjQuery.fx.off = trueNothing like this for D3. :-( This is not Ember-related, but you don’t want animations, because they are asynchronous. For D3,, this may require support in your production code. :-( Probably the trickiest thing is data fixtures:
  39. 39. Model Fixtures1. Client-side fixtures. You have a choice whether you write them in JavaScript or2. Server-side fixtures. generate them from the server side, and it’s kind of a trade off.
  40. 40. (1) Client-Side FixturesFixtureAdapter (immature)App.TodoList.FIXTURES = [ { ... }, { ... } For client-side fixtures,] there is a FixtureAdapter in Ember. It still needs some love, but we can probably get it there pretty soon. And basically you just define an array of fixtures for every model.
  41. 41. (1) Client-Side Fixtures:-) Easy to set up
  42. 42. (1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend But you don’t know if your fixtures actually represent the reality of your backend.
  43. 43. (1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend:-( Fragile You can even have bugs in your fixtures, where you don’t set up bidirectional belongsTo and hasMany relationships properly.
  44. 44. (1) Client-Side Fixtures:-) Easy:-( Goes out of sync with backend:-( Fragile:-( Server-side computed attributes JavaScript is still not very powerful. Oftentimes it’s So the alternative to all easier in practice to implement computed properties on the this is... backend side and ser ve them out as read-only attributes. In one backend I worked on, half of the properties were DB columns, and half were just methods on the Rails models. Trying to keep these properties manually updated in your fixture data is obviously painful.
  45. 45. (2) Server-Side Fixtures rake test:fixtures1. Write fixtures to DB2. Generate JSON to fixtures.js
  46. 46. (2) Server-Side Fixtures rake test:fixtures1. Write fixtures to DB2. Generate JSON to fixtures.js Load through RESTAdapterLoad that data in fixtures.js before every testcase, perhaps using your RESTAdapter so youtranslate the JSON correctly.
  47. 47. (2) Server-Side Fixtures :-) Covers models, serializers, adapter with no speed penalty
  48. 48. (2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain Compact definitions, esp. w/ FactoryGirl; stuff doesn’t go out of sync.
  49. 49. (2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain :-( Usability Generated fixtures file can go stale and you have to regenerate. It’s not bad, just bothersome.
  50. 50. (2) Server-Side Fixtures :-) Covers models, serializers, adapter :-) Easy to maintain :-( Usability :-( Complex to set up Work to set up. You end up with some custom code, and it ties tightly into backend. I personally think it’s generally worth it, but it also depends on your specific app.
  51. 51. Fixture WoesGlobal fixtures :-( Both of these techniques mean that you have the same fixture set for the entire test suite. want FactoryGirl Ideally we’d build something like FactoryGirl, but I don’t think we’re quite there yet.
  52. 52. Bonus: JS-driven?Capybara but in JavaScript? One thing that sometimes comes up is, can we have a full-stack integration test written in JavaScript?
  53. 53. Firefox + So instead of having the Selenium Test suite written in Ruby...Server Test suite DB
  54. 54. Firefox w/ JS tests ... we push them into the browser, and so we avoid the t wo-threaded problem.Server I think that would be really awesome -- and the truth is, we just don’t have the tooling yet to make that work easily. Perhaps the biggest hurdle DB is that we don’t have a way to reset the database, add fixtures, and perhaps query database records from JavaScript. But it is probably a good direction to move towards
  55. 55. Q &ANotes from the talk, thanks to @cgcardona:Q: With Konacha do you see opportunity for TTD?A: Not quite TDD (hard with visual stuff), but continuous.When working with Konacha I wrote tests as I went.Q: Can you test views in isolation?A: It’s really tricky to instantiate a view in isolation, Emberwants a whole app… It’s too ‘unit testy’. It might bepossible.
  56. 56. Q &AComment: We use ‘Rosie’ (https://github.com/bkeepers/rosie) for Javascript factories.Comment: Have you tried VCR for server side things? It’s aRuby library that will record request responses so you canplay them back later. We run VCR against a live productionserver and generate response data that the tests use.
  57. 57. Q &AQ: A lot of bugs come from asynchronicity. Have you triedto test that specifically?A: No I haven’t. My hope is that a lot of these bugsdisappear with ember-data. Mocha allows for asynchronoustests where you set a callback for when your test iscomplete.
  58. 58. Thanks for listening!Johttps://twitter.com/jo_lisshttp://solitr.com/blog
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×