Extreme Testing at kaChing

2,551 views

Published on

At kaChing (www.kaching.com), we are on a 5-minute commit-to-production cycle. We have adopted continuous deployment as a way of life and as the natural next step to continuous integration.

In this talk, I will present how we achieved this extreme iteration cycle. We will start at a very high level and look at the two fundamental aspects of software: transformations, which are stateless data operations, and interactions, which deal with state (such as a database, or an e-mail server). With this background we will delve into practical matters and survey kaChing's testing infrastructure by motivating each category of tests with different kind of problems often encountered. Finally, we will look at software patterns that lend themselves to testing and achieving separation of concerns allowing unparalleled software composability.

(This talk will focus on a Java environment even though the discussion will be largely applicable.)

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,551
On SlideShare
0
From Embeds
0
Number of Embeds
686
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Extreme Testing at kaChing

  1. 1. Extreme Testing at kaChing  From Commit to Production in 5 minutes    Pascal‐Louis Perez –               .com  Pascal‐Louis Perez, kaChing Group Inc. 
  2. 2. kaChing: the Investing Talent Marketplace  • Marketplace for qualified managers  – Young professionals  – RIAs  – Hedge Fund managers  • Customers can mirror “kaChing Geniuses”  – Your own brokerage account  – Real time reporting; Analytics; Drift alerts  • Unparalleled Transparency  – See everything: holdings, risk metrics, trades, …  – Clear fee structure: management fee. No hidden fees,  exit fees, …  Pascal‐Louis Perez, kaChing Group Inc. 
  3. 3.   Pascal‐Louis Perez, kaChing Group Inc. 
  4. 4. Our System in 30 Seconds  Pascal‐Louis Perez, kaChing Group Inc. 
  5. 5. Testing Matters  Architectural Soundness  Delighting Customers  Pascal‐Louis Perez, kaChing Group Inc. 
  6. 6. Testable Code Because…  • It allows quick iterations  • It makes it easier to scale the team  • It is more cost effective than debugging  • It obsoletes the need for functional QA  • It facilitates continuous refactoring, allowing  the code to get better with age  • It attracts the right kind of engineers  Pascal‐Louis Perez, kaChing Group Inc. 
  7. 7. High Level Classification  Correctness  Availability  λ  π  Pascal‐Louis Perez, kaChing Group Inc. 
  8. 8. High Level Classification  Correctness  Availability  λ  • Bad: incorrect  but available system  • Bad: correct but unavailable system  π  Pascal‐Louis Perez, kaChing Group Inc. 
  9. 9. High Level Classification  Correctness  Availability  • Transformations (λ)  λ  • Interactions (π): send an e‐mail,  store data in a db, RPC call  • In π implicit global mutable state  π  Pascal‐Louis Perez, kaChing Group Inc. 
  10. 10. High Level Classification  Correctness  Availability  λ  • Simplest to test  • Input produces an output: just assert  π  • (Not  to be confused with  implementation complexity!)  Pascal‐Louis Perez, kaChing Group Inc. 
  11. 11. High Level Classification  Correctness  Availability  λ  • Transformations can only be CPU  π  bound  • Timing tests, benchmarking, …  Pascal‐Louis Perez, kaChing Group Inc. 
  12. 12. High Level Classification  Correctness  Availability  λ  π  • Key idea: fake the world  • Mocks, stubs, fakes  Pascal‐Louis Perez, kaChing Group Inc. 
  13. 13. High Level Classification  Correctness  Availability  λ  π  • Work in small batches  • End‐to‐end tests, regression test,  performance lab  (Find yourself a great architect!)  Pascal‐Louis Perez, kaChing Group Inc. 
  14. 14. Think Outside the Box  C  A  λ  π  A  C  A  C  λ  λ  π  π  C  A  λ  π  Pascal‐Louis Perez, kaChing Group Inc. 
  15. 15. Repository (or Client) Pattern  C  A  @ImplementedBy(DbUserRepo.class) λ  interface UserRepository { π  User get(Id<User> id); } UserRepository repo = ...; User pascal = repo.get(Id.<User> of(8)); C  A  class DbUserRepo λ  implements UserRepository { π  // implementation } Pascal‐Louis Perez, kaChing Group Inc. 
  16. 16. Components  • Software composability is key to test‐driven  development  • Testable software abstracts the control flow;  As such is functional by nature  Pascal‐Louis Perez, kaChing Group Inc. 
  17. 17. Examples of Simple Tests  • Marshalling tests  • Dealing with time  • Equivalence tester  Pascal‐Louis Perez, kaChing Group Inc. 
  18. 18. C  A  λ  Marshalling tests  π  Json.Object o =  Fail fast  TwoLattes.createMarshaller(Trade.class).marshall(...); assertSameJson( object( string("id"), number(4), string("portfolioId"), NULL, string("trade"), string("interest"), string("date"), string("2008-07-06"), string("toDate"), string("2008-06-06"), string("fromDate"), string("2008-07-05"), string("longInterest"), number(56.43), string("shortInterest"), number(-0.81)), o);  Pascal‐Louis Perez, kaChing Group Inc. 
  19. 19. C  A  λ  Marshalling tests  π  Json.Object o = TwoLattes.createMarshaller(Trade.class).marshall(...); assertSameJson(  Dedicated assertions  object( string("id"), number(4), string("portfolioId"), NULL, string("trade"), string("interest"), string("date"), string("2008-07-06"), string("toDate"), string("2008-06-06"), string("fromDate"), string("2008-07-05"), string("longInterest"), number(56.43), string("shortInterest"), number(-0.81)), o);  Pascal‐Louis Perez, kaChing Group Inc. 
  20. 20. C  A  λ  Marshalling tests  π  Json.Object o = TwoLattes.createMarshaller(Trade.class).marshall(...); assertSameJson(  Descriptive expectation  object( string("id"), number(4), string("portfolioId"), NULL, string("trade"), string("interest"), string("date"), string("2008-07-06"), string("toDate"), string("2008-06-06"), string("fromDate"), string("2008-07-05"), string("longInterest"), number(56.43), string("shortInterest"), number(-0.81)), o);  Pascal‐Louis Perez, kaChing Group Inc. 
  21. 21. C  A  λ  Dealing with time  π  • Use clocks  – Provider<DateTime>  – Provider<LocalDate>  • Pass in now or today in business logic  Pascal‐Louis Perez, kaChing Group Inc. 
  22. 22. Time Logic  @VisibleForTesting <T> T provideCurrentOrHistorical( DateTime date, Provider<T> current, Provider<T> historical) { DateTime lastTradeDate = getLastTradeDate();  Time logic is easy  boolean isCurrent = date == null      to test  || lastTradeDate == null || date.compareTo(lastTradeDate) > 0; return isCurrent ? current.get() : historical.get(); }  Pascal‐Louis Perez, kaChing Group Inc. 
  23. 23. Time Logic  @VisibleForTesting <T> T provideCurrentOrHistorical(  Higher‐order functions  DateTime date, Provider<T> current, Provider<T> historical) { DateTime lastTradeDate = getLastTradeDate(); boolean isCurrent = date == null || lastTradeDate == null || date.compareTo(lastTradeDate) > 0; return isCurrent ? current.get() : historical.get(); }  Pascal‐Louis Perez, kaChing Group Inc. 
  24. 24. C  A  λ  Equivalence Tester  π  EquivalenceTester.check( Tests  asList(  Reflexivity  new Isin("abCdEFgHiJK3“),  Symmetry  new Isin("ABCDEFGHIJK3“),  Transitivity  new Isin(“abCDefGHIJK3")),   hashCode consistency  asList( new Isin("AU0000XVGZA3"), new Isin("AU0000XVGZA3")));  Pascal‐Louis Perez, kaChing Group Inc. 
  25. 25. Rationale for Database Testing  • Accessing the database correctly  • O/R properly configured and used  • Hand written queries  • Optimistic concurrency  Pascal‐Louis Perez, kaChing Group Inc. 
  26. 26. JDBC   Global static state  DriverManager.registerDriver(new org.gjt.mm.mysql.Driver()); Connection conn = DriverManager.getConnection("jdbc:..."); PreparedStatement st = conn.prepareStatement("select ..."); st.setString(1, "..."); ResultSet rs = st.executeQuery(); while (rs.next()) { // work } rs.close(); st.close(); conn.close();  Pascal‐Louis Perez, kaChing Group Inc. 
  27. 27. JDBC  DriverManager.registerDriver(new org.gjt.mm.mysql.Driver()); Connection conn = DriverManager.getConnection("jdbc:..."); PreparedStatement st = conn.prepareStatement("select ..."); st.setString(1, "..."); ResultSet rs = st.executeQuery(); while (rs.next()) { // work } rs.close();  Low level, stateful API, wide scoping  st.close(); conn.close();  Pascal‐Louis Perez, kaChing Group Inc. 
  28. 28. JDBC  DriverManager.registerDriver(new org.gjt.mm.mysql.Driver()); Connection conn = DriverManager.getConnection("jdbc:..."); PreparedStatement st = conn.prepareStatement("select ..."); st.setString(1, "..."); ResultSet rs = st.executeQuery(); while (rs.next()) { // work }  Resources hell  rs.close(); st.close(); conn.close();  Pascal‐Louis Perez, kaChing Group Inc. 
  29. 29. Hibernate   Global state  SessionFactory factory = ...; Session session = factory.openSession(); Transaction tx = session.beginTransaction(); List<User> users = session .createQuery("from User where ...") .list(); for (User user : users) { // work } tx.commit(); session.close();  Pascal‐Louis Perez, kaChing Group Inc. 
  30. 30. Hibernate  SessionFactory factory = ...;  High‐level API  Session session = factory.openSession(); Transaction tx = session.beginTransaction(); List<User> users = session .createQuery("from User where ...") .list(); for (User user : users) { // work } tx.commit(); session.close();  Pascal‐Louis Perez, kaChing Group Inc. 
  31. 31. Hibernate  SessionFactory factory = ...; Session session = factory.openSession(); Transaction tx = session.beginTransaction(); List<User> users = session .createQuery("from User where ...") .list(); for (User user : users) { // work }  Resources hell  tx.commit(); session.close();  Pascal‐Louis Perez, kaChing Group Inc. 
  32. 32. Transacter   Global state  Transacter transacter = ...; transacter.execute(new WithSession() { public void run(Session session) { List<User> users = ...; for (User user : users) { // work } } });  Pascal‐Louis Perez, kaChing Group Inc. 
  33. 33. Transacter  Transacter transacter = ...;  High‐level API, no resources  transacter.execute(new WithSession() { public void run(Session session) { List<User> users = ...; for (User user : users) { // work } } });  Pascal‐Louis Perez, kaChing Group Inc. 
  34. 34. Transacter  Transacter transacter = ...; transacter.execute(new WithSession() { public void run(Session session) {  Lexical scoping  List<User> users = ...; for (User user : users) { // work } } });  Pascal‐Louis Perez, kaChing Group Inc. 
  35. 35. Database Testing is Hard  • Complex to set up : PersistenceTestRunner  • Schema installation : O/R Mapping  • Requires data to be loaded : Fixtures, DbUnit  • Developer must have proper environment  : in‐memory DBs (HSQLDB, Connector/MXJ)  Pascal‐Louis Perez, kaChing Group Inc. 
  36. 36. C  A  λ  PersistenceTestRunner  π  @RunWith(PersistenceTestRunner.class)  Declarative setup  @PersistentEntities({Item.class, Note.class}) public class ExamplePersistentTest { @Test public void example1(Transacter transacter) { // work } @Test public void example2() { // work } } Pascal‐Louis Perez, kaChing Group Inc. 
  37. 37. C  A  λ  PersistenceTestRunner  π  @RunWith(PersistenceTestRunner.class) @PersistentEntities({Item.class, Note.class}) public class ExamplePersistentTest {  Lexical scoping  @Test public void example1(Transacter transacter) { // work } @Test public void example2() { // work } } Pascal‐Louis Perez, kaChing Group Inc. 
  38. 38. C  A  λ  PersistenceTestRunner  π  @RunWith(PersistenceTestRunner.class) @PersistentEntities({Item.class, Note.class}) public class ExamplePersistentTest { @Test  RuntimeException  public void example1(Transacter transacter) { // work } @Test public void example2() { // work } } Pascal‐Louis Perez, kaChing Group Inc. 
  39. 39. C  A  λ  PersistenceTestRunner  π  @RunWith(PersistenceTestRunner.class) @PersistentEntities({Item.class, Note.class}) public class ExamplePersistentTest { @Test public void example1(Transacter transacter) { // work }  Non‐persistent tests run as usual  @Test public void example2() { // work } } Pascal‐Louis Perez, kaChing Group Inc. 
  40. 40. C  A  λ  Testing Interactions (I/O, …)  π  • Implicit global state  • Many cases  – HTTP GET: succeed, fail to connect, connection  closed before headers are read, …  – Sending e‐mail: succeed, connection lost after  HELO, …  • Sequencing is crucial  Pascal‐Louis Perez, kaChing Group Inc. 
  41. 41. C  A  λ  Testing the Transacter (success)  π   Mocking  final WithSession withSession = mockery.mock(WithSession.class); mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); will(returnValue(session)); inSequence(execution); one(session).connection(); will(...); inSequence(...); one(connection).setReadOnly(with(equal(false))); ... one(connection).setAutoCommit(with(equal(false))); ... one(session).beginTransaction(); inSequence(...); ... }}); transacter.execute(withSession); mockery.assertIsSatisfied(); Pascal‐Louis Perez, kaChing Group Inc. 
  42. 42. C  A  λ  Testing the Transacter (success)  π  final WithSession withSession = mockery.mock(WithSession.class); mockery.checking(new Expectations() {{ one(sessionFactory).openSession();  Interactions are scripted  will(returnValue(session)); inSequence(execution); one(session).connection(); will(...); inSequence(...); one(connection).setReadOnly(with(equal(false))); ... one(connection).setAutoCommit(with(equal(false))); ... one(session).beginTransaction(); inSequence(...); ... }}); transacter.execute(withSession); mockery.assertIsSatisfied(); Pascal‐Louis Perez, kaChing Group Inc. 
  43. 43. C  A  λ  Testing the Transacter (failure)  π  mockery.checking(new Expectations() {{ one(sessionFactory).openSession();  Failure is controlled  ... one(connection).setReadOnly(with(equal(false))); will(throwException(exception)); inSequence(execution); one(session).close(); inSequence(execution); }}); try { transacter.execute(withSession); fail(); } catch (RuntimeException e) { assertTrue(e == exception); } mockery.assertIsSatisfied(); Pascal‐Louis Perez, kaChing Group Inc. 
  44. 44. C  A  λ  Testing the Transacter (failure)  π  mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); ... one(connection).setReadOnly(with(equal(false))); will(throwException(exception)); inSequence(execution); one(session).close(); inSequence(execution); }}); try { transacter.execute(withSession); fail(); } catch (RuntimeException e) { assertTrue(e == exception);  Exact  same object (correct bubbling)  } mockery.assertIsSatisfied(); Pascal‐Louis Perez, kaChing Group Inc. 
  45. 45. C  A  λ  Testing the Transacter (failure)  π  mockery.checking(new Expectations() {{ one(sessionFactory).openSession(); ... one(connection).setReadOnly(with(equal(false))); will(throwException(exception)); inSequence(execution); one(session).close(); inSequence(execution); }});  Unreachable program point  try { transacter.execute(withSession); fail(); } catch (RuntimeException e) { assertTrue(e == exception); } mockery.assertIsSatisfied(); Pascal‐Louis Perez, kaChing Group Inc. 
  46. 46. Meta Tests  • Cross domain tests to sweep the codebase,  exactly like static analysis  • Catch common mistakes  – Distracted developers  – Bad APIs  – New hires learning conventions    Pascal‐Louis Perez, kaChing Group Inc. 
  47. 47. C  A  λ  Dependency Test  π  • Declare software dependencies  • Controls layering of application  • Simpler and more flexible than different  compilation units  • Can guarantee that certain APIs are not used  directly  • Patched Jdepend to parse annotations  Pascal‐Louis Perez, kaChing Group Inc. 
  48. 48. Dependency Test  dependencies.forPackages( "com.kaching.*" ). check("com.kaching.supermap").mayDependOn( "com.kaching.platform.guice", "org.apache.commons.lang", "voldemort.*", "org.kohsuke.args4j" ). assertIsVerified(jDepend.getPackages()); Pascal‐Louis Perez, kaChing Group Inc. 
  49. 49. C  A  λ  Bad Code Test  π  • Enforce conventions  • Gotchas, bad APIs  – E.g. System.out.println  – E.g. java.net.URL  • Code reviews  – Design patterns, custom APIs, etc.  – Enforce conventions: automate this!  Pascal‐Louis Perez, kaChing Group Inc. 
  50. 50. Bad Code Test  @RunWith(BadCodeSnippetsRunner.class) @CodeSnippets({ @Check(paths = "src/com/kaching", snippets = { // no uses of java.net.URL @Snippet(value = "bjava.net.URLb", exceptions = { "src/com/kaching/...", "src/com/kaching/..." }), // never call default super constructor @Snippet("super()") })}) class MyBadCodeSnippetsTest { Pascal‐Louis Perez, kaChing Group Inc. 
  51. 51. Key Takeaways  • Invest in your tests, and testing frameworks  • Design patterns to separate concerns  • Components  • TDD pushes towards functional techniques    • We’re recruiting jobs@kaching.com  • More http://eng.kaching.com  Pascal‐Louis Perez, kaChing Group Inc. 

×