Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Extreme Testing at kaChing: From Commit to Production in 5 Minutes

3,207 views

Published on



At kaChing (www.kaching.com), we are on a 5-minute commit-to-production cycle. We have adopted continuous deployment as a way of life and as the natural next step to continuous integration.

In this talk, I will present how we achieved the core of our extreme iteration cycles: test-driven development or how to automate quality assurance. We will start at a very high level and look at the two fundamental aspects of software: transformations, which are stateless data operations, and interactions, which deal with state (such as a database, or an e-mail server). With this background we will delve into practical matters and survey kaChing's testing infrastructure by motivating each category of tests with different kind of problems often encountered. Finally, we will look at software patterns that lend themselves to testing and achieving separation of concerns allowing unparalleled software composability.

This talk will focus on Java and the JVM even though the discussion will be largely applicable.

Check out http://eng.kaching.com/search/label/tests for the latest from our company's blog.

Published in: Technology
  • Be the first to comment

Extreme Testing at kaChing: From Commit to Production in 5 Minutes

  1. 1. Extreme Testing at kaChing<br />From Commit to Production in 5 minutes<br />Pascal-Louis Perez – @pascallouis<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  2. 2. Engineering at kaChing<br />TDD from day one<br />Full regression suite runs in less than 3 minutes<br />Deploy to production 30+ times a day<br />People have written and launched new features during interview process<br />Dedicated engineers to build testing frameworks<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  3. 3.
  4. 4. Testable Code Because…<br />It allows quick iterations<br />It makes it easier to scale the team<br />It is more cost effective than debugging<br />It obsoletes the need for functional QA<br />It facilitates continuous refactoring, allowing the code to get better with age<br />It attracts the right kind of engineers<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  5. 5. High Level Classification<br />Pascal-Louis Perez, kaChing Group Inc.<br />Correctness<br />Availability<br />λ<br />π<br />
  6. 6. High Level Classification<br />Pascal-Louis Perez, kaChing Group Inc.<br />Correctness<br />Availability<br /><ul><li> Bad: incorrect but available system
  7. 7. Bad: correct but unavailable system</li></ul>λ<br />π<br />
  8. 8. High Level Classification<br />Pascal-Louis Perez, kaChing Group Inc.<br />Correctness<br />Availability<br /><ul><li> Transformations (λ)
  9. 9. Interactions (π): send an e-mail, store data in a db, RPC call
  10. 10. In π implicit global mutable state</li></ul>λ<br />π<br />
  11. 11. High Level Classification<br />Pascal-Louis Perez, kaChing Group Inc.<br />Correctness<br />Availability<br />λ<br /><ul><li> Simplest to test
  12. 12. Input produces an output: just assert
  13. 13. (Not to be confused with implementation complexity!)</li></ul>π<br />
  14. 14. High Level Classification<br />Pascal-Louis Perez, kaChing Group Inc.<br />Correctness<br />Availability<br />λ<br /><ul><li> Transformations can only be CPU bound
  15. 15. Timing tests, benchmarking, …</li></ul>π<br />
  16. 16. High Level Classification<br />Pascal-Louis Perez, kaChing Group Inc.<br />Correctness<br />Availability<br />λ<br />π<br /><ul><li> Key idea: fake the world
  17. 17. Mocks, stubs, fakes</li></li></ul><li>High Level Classification<br />Pascal-Louis Perez, kaChing Group Inc.<br />Correctness<br />Availability<br />λ<br />π<br /><ul><li> Work in small batches
  18. 18. End-to-end tests, regression test, performance lab</li></ul>(Find yourself a great architect!)<br />
  19. 19. Think Outside the Box<br />Pascal-Louis Perez, kaChing Group Inc.<br />A<br />A<br />A<br />A<br />C<br />C<br />C<br />C<br />λ<br />λ<br />λ<br />λ<br />π<br />π<br />π<br />π<br />
  20. 20. Repository (or Client) Pattern<br />Pascal-Louis Perez, kaChing Group Inc.<br />@ImplementedBy(DbUserRepo.class)interfaceUserRepository {<br /> User get(Id<User> id);}<br />UserRepository repo = ...;<br />User pascal = repo.get(Id.<User> of(8));<br />A<br />A<br />C<br />C<br />λ<br />λ<br />π<br />π<br />classDbUserRepoimplementsUserRepository { // implementation}<br />
  21. 21. Components<br />Software composability is key to test-driven development<br />Testable software abstracts the control flow; As such is functional by nature<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  22. 22. Examples of Simple Tests<br />Marshalling tests<br />Dealing with time<br />Equivalence tester<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  23. 23. Marshalling tests<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br /> Fail fast<br />Json.Object o =<br />TwoLattes.createMarshaller(Trade.class).marshall(...);<br />assertSameJson(<br />object(<br />string("id"), number(4),<br />string("portfolioId"), NULL,<br />string("trade"), string("interest"),<br />string("date"), string("2008-07-06"),<br />string("toDate"), string("2008-06-06"),<br />string("fromDate"), string("2008-07-05"),<br />string("longInterest"), number(56.43),<br />string("shortInterest"), number(-0.81)),<br /> o);<br />λ<br />π<br />
  24. 24. Marshalling tests<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />Json.Object o =<br />TwoLattes.createMarshaller(Trade.class).marshall(...);<br />assertSameJson(<br />object(<br />string("id"), number(4),<br />string("portfolioId"), NULL,<br />string("trade"), string("interest"),<br />string("date"), string("2008-07-06"),<br />string("toDate"), string("2008-06-06"),<br />string("fromDate"), string("2008-07-05"),<br />string("longInterest"), number(56.43),<br />string("shortInterest"), number(-0.81)),<br /> o);<br /> Dedicated assertions<br />λ<br />π<br />
  25. 25. Marshalling tests<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br /> Descriptive expectation<br />Json.Object o =<br />TwoLattes.createMarshaller(Trade.class).marshall(...);<br />assertSameJson(<br />object(<br />string("id"), number(4),<br />string("portfolioId"), NULL,<br />string("trade"), string("interest"),<br />string("date"), string("2008-07-06"),<br />string("toDate"), string("2008-06-06"),<br />string("fromDate"), string("2008-07-05"),<br />string("longInterest"), number(56.43),<br />string("shortInterest"), number(-0.81)),<br /> o);<br />λ<br />π<br />
  26. 26. Dealing with time<br />Use clocks<br />Provider<DateTime><br />Provider<LocalDate><br />Pass in now or today in business logic<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />λ<br />π<br />
  27. 27. Time Logic<br />Pascal-Louis Perez, kaChing Group Inc.<br /> Time logic is easy to test<br />@VisibleForTesting<br /><T> T provideCurrentOrHistorical(<br />DateTime date, Provider<T> current, Provider<T> historical) {<br />DateTimelastTradeDate = getLastTradeDate();<br />booleanisCurrent = date == null || lastTradeDate == null<br /> || date.compareTo(lastTradeDate) > 0;<br />returnisCurrent ? current.get() : historical.get();<br />}<br />
  28. 28. Time Logic<br />Pascal-Louis Perez, kaChing Group Inc.<br /> Higher-order functions<br />@VisibleForTesting<br /><T> T provideCurrentOrHistorical(<br />DateTime date, Provider<T> current, Provider<T> historical) {<br />DateTimelastTradeDate = getLastTradeDate();<br />booleanisCurrent = date == null || lastTradeDate == null<br /> || date.compareTo(lastTradeDate) > 0;<br />returnisCurrent ? current.get() : historical.get();<br />}<br />
  29. 29. Equivalence Tester<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />Tests Reflexivity Symmetry Transitivity<br /> hashCode consistency compareTo consistency<br /> proper null handling<br />EquivalenceTester.check(<br />asList(<br />newIsin("abCdEFgHiJK3“),<br />newIsin("ABCDEFGHIJK3“),<br />newIsin(“abCDefGHIJK3")),<br />asList(<br />newIsin("AU0000XVGZA3"),<br />newIsin("AU0000XVGZA3")));<br />λ<br />π<br />
  30. 30. Rationale for Database Testing<br />Accessing the database correctly<br />O/R properly configured and used<br />Hand written queries<br />Optimistic concurrency<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  31. 31. JDBC<br />Pascal-Louis Perez, kaChing Group Inc.<br /> Global static state<br />DriverManager.registerDriver(neworg.gjt.mm.mysql.Driver());<br /> Connection conn = DriverManager.getConnection("jdbc:...");<br />PreparedStatementst = conn.prepareStatement("select ...");<br />st.setString(1, "...");<br />ResultSetrs = st.executeQuery();<br />while (rs.next()) {<br />// work<br /> }<br />rs.close();<br />st.close();<br />conn.close();<br />
  32. 32. JDBC<br />Pascal-Louis Perez, kaChing Group Inc.<br />DriverManager.registerDriver(neworg.gjt.mm.mysql.Driver());<br /> Connection conn = DriverManager.getConnection("jdbc:...");<br />PreparedStatementst = conn.prepareStatement("select ...");<br />st.setString(1, "...");<br />ResultSetrs = st.executeQuery();<br />while (rs.next()) {<br />// work<br /> }<br />rs.close();<br />st.close();<br />conn.close();<br /> Low level, stateful API, wide scoping<br />
  33. 33. JDBC<br />Pascal-Louis Perez, kaChing Group Inc.<br />DriverManager.registerDriver(neworg.gjt.mm.mysql.Driver());<br /> Connection conn = DriverManager.getConnection("jdbc:...");<br />PreparedStatementst = conn.prepareStatement("select ...");<br />st.setString(1, "...");<br />ResultSetrs = st.executeQuery();<br />while (rs.next()) {<br />// work<br /> }<br />rs.close();<br />st.close();<br />conn.close();<br /> Resources hell<br />
  34. 34. Hibernate<br />Pascal-Louis Perez, kaChing Group Inc.<br /> Global state<br />SessionFactory factory = ...;<br />Session session = factory.openSession();<br />Transaction tx = session.beginTransaction();List<User> users = session .createQuery("from User where ...") .list();<br />for (User user : users) {<br />// work<br /> }<br />tx.commit();<br />session.close();<br />
  35. 35. Hibernate<br />Pascal-Louis Perez, kaChing Group Inc.<br /> High-level API<br />SessionFactory factory = ...;<br />Session session = factory.openSession();<br />Transaction tx = session.beginTransaction();List<User> users = session .createQuery("from User where ...") .list();<br />for (User user : users) {<br />// work<br /> }<br />tx.commit();<br />session.close();<br />
  36. 36. Hibernate<br />Pascal-Louis Perez, kaChing Group Inc.<br />SessionFactory factory = ...;<br />Session session = factory.openSession();<br />Transaction tx = session.beginTransaction();List<User> users = session .createQuery("from User where ...") .list();<br />for (User user : users) {<br />// work<br /> }<br />tx.commit();<br />session.close();<br /> Resources hell<br />
  37. 37. Transacter<br />Pascal-Louis Perez, kaChing Group Inc.<br /> Global state<br />Transactertransacter = ...;<br />transacter.execute(newWithSession() {<br />publicvoid run(Session session) { List<User> users = ...;<br />for (User user : users) {<br />// work<br /> }<br /> }<br />});<br />
  38. 38. Transacter<br />Pascal-Louis Perez, kaChing Group Inc.<br /> High-level API, no resources<br />Transactertransacter = ...;<br />transacter.execute(newWithSession() {<br />publicvoid run(Session session) { List<User> users = ...;<br />for (User user : users) {<br />// work<br /> }<br /> }<br />});<br />
  39. 39. Transacter<br />Pascal-Louis Perez, kaChing Group Inc.<br />Transactertransacter = ...;<br />transacter.execute(newWithSession() {<br />publicvoid run(Session session) { List<User> users = ...;<br />for (User user : users) {<br />// work<br /> }<br /> }<br />});<br /> Lexical scoping<br />
  40. 40. Database Testing is Hard<br />Complex to set up<br />Schema installation<br />Requires data to be loaded<br />Developer must have proper environment<br />Pascal-Louis Perez, kaChing Group Inc.<br />: PersistenceTestRunnerü<br />: O/R Mapping ü<br />: Fixtures, DbUnitü<br />: in-memory DBs (HSQLDB, Connector/MXJ) ü<br />
  41. 41. PersistenceTestRunner<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br /> Declarative setup<br />@RunWith(PersistenceTestRunner.class)<br />@PersistentEntities({Item.class, Note.class})<br />publicclassExamplePersistentTest {<br />@Test<br />publicvoid example1(Transactertransacter) {<br />// work<br /> }<br />@Test<br />publicvoid example2() {<br />// work<br /> }<br />}<br />λ<br />π<br />
  42. 42. PersistenceTestRunner<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />@RunWith(PersistenceTestRunner.class)<br />@PersistentEntities({Item.class, Note.class})<br />publicclassExamplePersistentTest {<br />@Test<br />publicvoid example1(Transactertransacter) {<br />// work<br /> }<br />@Test<br />publicvoid example2() {<br />// work<br /> }<br />}<br /> Lexical scoping<br />λ<br />π<br />
  43. 43. PersistenceTestRunner<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />@RunWith(PersistenceTestRunner.class)<br />@PersistentEntities({Item.class, Note.class})<br />publicclassExamplePersistentTest {<br />@Test<br />publicvoid example1(Transactertransacter) {<br />// work<br /> }<br />@Test<br />publicvoid example2() {<br />// work<br /> }<br />}<br />λ<br /> RuntimeException<br />π<br />
  44. 44. PersistenceTestRunner<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />@RunWith(PersistenceTestRunner.class)<br />@PersistentEntities({Item.class, Note.class})<br />publicclassExamplePersistentTest {<br />@Test<br />publicvoid example1(Transactertransacter) {<br />// work<br /> }<br />@Test<br />publicvoid example2() {<br />// work<br /> }<br />}<br />λ<br />π<br /> Non-persistent tests run as usual<br />
  45. 45. Testing Interactions (I/O, …)<br />Implicit global state<br />Many cases<br />HTTP GET: succeed, fail to connect, connection closed before headers are read, …<br />Sending e-mail: succeed, connection lost after HELO, …<br />Sequencing is crucial<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />λ<br />π<br />
  46. 46. Testing the Transacter (success)<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br /> Mocking<br />finalWithSessionwithSession = mockery.mock(WithSession.class);<br />mockery.checking(new Expectations() {{<br /> one(sessionFactory).openSession();<br /> will(returnValue(session));<br />inSequence(execution);<br /> one(session).connection(); will(...); inSequence(...);<br /> one(connection).setReadOnly(with(equal(false))); ...<br /> one(connection).setAutoCommit(with(equal(false))); ...<br /> one(session).beginTransaction(); inSequence(...); ...<br />}});<br />transacter.execute(withSession);<br />mockery.assertIsSatisfied();<br />λ<br />π<br />
  47. 47. Testing the Transacter (success)<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />finalWithSessionwithSession = mockery.mock(WithSession.class);<br />mockery.checking(new Expectations() {{<br /> one(sessionFactory).openSession();<br /> will(returnValue(session));<br />inSequence(execution);<br /> one(session).connection(); will(...); inSequence(...);<br /> one(connection).setReadOnly(with(equal(false))); ...<br /> one(connection).setAutoCommit(with(equal(false))); ...<br /> one(session).beginTransaction(); inSequence(...); ...<br />}});<br />transacter.execute(withSession);<br />mockery.assertIsSatisfied();<br /> Interactions are scripted<br />λ<br />π<br />
  48. 48. Testing the Transacter (failure)<br />Pascal-Louis Perez, kaChing Group Inc.<br /> Failure is controlled<br />C<br />mockery.checking(new Expectations() {{<br /> one(sessionFactory).openSession(); ...<br /> one(connection).setReadOnly(with(equal(false)));<br /> will(throwException(exception));<br />inSequence(execution);<br /> one(session).close(); inSequence(execution);<br />}});<br />try {<br />transacter.execute(withSession); fail();<br />} catch (RuntimeException e) {<br />assertTrue(e == exception);<br />}mockery.assertIsSatisfied();<br />A<br />λ<br />π<br />
  49. 49. Testing the Transacter (failure)<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />mockery.checking(new Expectations() {{<br /> one(sessionFactory).openSession(); ...<br /> one(connection).setReadOnly(with(equal(false)));<br /> will(throwException(exception));<br />inSequence(execution);<br /> one(session).close(); inSequence(execution);<br />}});<br />try {<br />transacter.execute(withSession); fail();<br />} catch (RuntimeException e) {<br />assertTrue(e == exception);<br />}mockery.assertIsSatisfied();<br />A<br /> Exact same object (correct bubbling)<br />λ<br />π<br />
  50. 50. Testing the Transacter (failure)<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />mockery.checking(new Expectations() {{<br /> one(sessionFactory).openSession(); ...<br /> one(connection).setReadOnly(with(equal(false)));<br /> will(throwException(exception));<br />inSequence(execution);<br /> one(session).close(); inSequence(execution);<br />}});<br />try {<br />transacter.execute(withSession); fail();<br />} catch (RuntimeException e) {<br />assertTrue(e == exception);<br />}mockery.assertIsSatisfied();<br />A<br /> Unreachable program point<br />λ<br />π<br />
  51. 51. Meta Tests<br />Cross domain tests to sweep the codebase, exactly like static analysis<br />Catch common mistakes<br />Distracted developers<br />Bad APIs<br />New hires learning conventions<br />Pascal-Louis Perez, kaChing Group Inc.<br />
  52. 52. Dependency Test<br />Declare software dependencies<br />Controls layering of application<br />Simpler and more flexible than different compilation units<br />Can guarantee that certain APIs are not used directly<br />Patched Jdepend to parse annotations<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />λ<br />π<br />
  53. 53. Dependency Test<br />Pascal-Louis Perez, kaChing Group Inc.<br />dependencies.forPackages(<br />"com.kaching.*"<br /> ). <br />check("com.kaching.supermap").mayDependOn(<br />"com.kaching.platform.guice",<br />"org.apache.commons.lang",<br />"voldemort.*",<br />"org.kohsuke.args4j"<br /> ).<br />assertIsVerified(jDepend.getPackages());<br />
  54. 54. Bad Code Test<br />Enforce conventions<br />Gotchas, bad APIs<br />E.g. System.out.println<br />E.g. java.net.URL<br />Code reviews<br />Design patterns, custom APIs, etc.<br />Enforce conventions: automate this!<br />Pascal-Louis Perez, kaChing Group Inc.<br />C<br />A<br />λ<br />π<br />
  55. 55. Bad Code Test<br />Pascal-Louis Perez, kaChing Group Inc.<br />@RunWith(BadCodeSnippetsRunner.class)<br />@CodeSnippets({<br />@Check(paths = "src/com/kaching", snippets = {<br />// no uses of java.net.URL<br />@Snippet(value = "javanetURL", exceptions = {<br />"src/com/kaching/...",<br />"src/com/kaching/..."<br /> }),<br />// never call default super constructor<br />@Snippet("super")<br />})})class MyBadCodeSnippetsTest {<br />
  56. 56. Key Takeaways<br />Invest in your tests, and testing frameworks<br />Design patterns to separate concerns<br />Components<br />TDD pushes towards functional techniques<br />We’re recruiting jobs@kaching.com<br />More http://eng.kaching.com<br />Pascal-Louis Perez, kaChing Group Inc.<br />

×