Successfully reported this slideshow.

Testing sync engine

0

Share

1 of 62
1 of 62

Testing sync engine

0

Share

Download to read offline

Small presentation I've made for Wire team about testing process in iOS Sync Engine Team. You can read more in depth here - http://www.objc.io/issues/15-testing/xctest/

Small presentation I've made for Wire team about testing process in iOS Sync Engine Team. You can read more in depth here - http://www.objc.io/issues/15-testing/xctest/

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

Testing sync engine

  1. 1. Testing Sync Engine
  2. 2. Testing (iOS) Sync Engine
  3. 3. Agenda • Overall tests stats • Our test strategies • MockTransportSession • Workflow • Problems • Future plans
  4. 4. Overall tests stats
  5. 5. • 3356 tests on OSX Overall tests stats
  6. 6. • 3356 tests on OSX • 3415 tests on iOS (+ apns tests) Overall tests stats
  7. 7. • 3356 tests on OSX • 3415 tests on iOS (+ apns tests) • ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
  8. 8. • 3356 tests on OSX • 3415 tests on iOS (+ apns tests) • ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
  9. 9. • Unit tests ~ 80% Overall tests stats
  10. 10. • Unit tests ~ 80% • Integration tests ~ 20% Overall tests stats
  11. 11. • Unit tests ~ 80% • Integration tests ~ 20% • Performance tests - 18 Overall tests stats
  12. 12. • Unit tests ~ 80% • Integration tests ~ 20% • Performance tests - 18 • Tests of tests Overall tests stats
  13. 13. • Unit tests ~ 80% • Integration tests ~ 20% • Performance tests - 18 • Tests of tests • Sorry, no UI tests Overall tests stats
  14. 14. Our test strategies
  15. 15. What we DON’T do: Our test strategies
  16. 16. What we DON’T do: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Our test strategies
  17. 17. What we DON’T do: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Why? • Stubs are not that flexible. Our test strategies
  18. 18. What we DON’T do: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Why? • Stubs are not that flexible. • It’s still slow. Our test strategies
  19. 19. What we really do: • Test against fake backend Our test strategies
  20. 20. What we really do: • Test against fake backend Advantages • Fast • Flexible • Independent • Testable Our test strategies
  21. 21. What we really do: • Test against fake backend Advantages • Fast • Flexible • Independent • Testable Disadvantages • Need to write (and test!) code • Need to be in sync with real backend (can be improved) Our test strategies
  22. 22. MockTransportSession
  23. 23. WTF is THAT? MockTransportSession
  24. 24. WTF is THAT? ZMTransportSession ZMTransportSessi onErrorCode ZMBackgroundAc tivity ZMURLSessionCa ncelTimer ZMURLSessionSw itch ZMURLSession ZMTransportCodec ZMAccessToken ZMUserAgent ZMReachability ZMTransportData ZMTransportResp onse ZMTaskIdentifierM ap ZMExponentialBa ckoff ZMAccessTokenH andler ZMTransportRequ estScheduler ZMNetworkSocket ZMDataBuffer ZMWebSocketFra me ZMWebSocketHan dshake ZMWebSocket ZMPushChannelC onnection ZMTransportPush Channel ZMStreamPairThre ad MockTransportSession
  25. 25. WTF is THAT? ZMTransportSession MockTransportSession
  26. 26. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession MockTransportSession
  27. 27. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession • Sends http requests to scheduler MockTransportSession
  28. 28. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession • Sends http requests to scheduler • Handles authentication MockTransportSession
  29. 29. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession • Sends http requests to scheduler • Handles authentication • Opens/closes web socket connection MockTransportSession
  30. 30. WTF is THAT? MockTransportSession - is a mock of ZMTransportSession MockTransportSession
  31. 31. WTF is THAT? MockTransportSession - is a mock of ZMTransportSession MockTransportSession - acts as a fake backend MockTransportSession
  32. 32. WTF is THAT? • Make fake requests MockTransportSession
  33. 33. WTF is THAT? • Make fake requests • Records requests and responds to them MockTransportSession
  34. 34. WTF is THAT? • Make fake requests • Records requests and responds to them • Manages internal database MockTransportSession
  35. 35. WTF is THAT? • Make fake requests • Records requests and responds to them • Manages internal database • Can simulate remote changes. MockTransportSession
  36. 36. WTF is THAT? • Make fake requests • Records requests and responds to them • Manages internal database • Can simulate remote changes. • Can send push channel (web socket) events MockTransportSession
  37. 37. • Before each test we insert objects in internal database (in memory) MockTransportSession
  38. 38. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession MockTransportSession
  39. 39. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) MockTransportSession
  40. 40. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) • Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) MockTransportSession
  41. 41. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) • Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) • Creates and sends response. Optionally can delegate it to other object (i.e. test case). MockTransportSession
  42. 42. Workflow
  43. 43. • Finished feature - run all the tests Workflow
  44. 44. • Finished feature - run all the tests • Create pull request Workflow
  45. 45. • Finished feature - run all the tests • Create pull request • Review, merge Workflow
  46. 46. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally Workflow
  47. 47. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) Workflow
  48. 48. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop Workflow
  49. 49. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop • Each bot for each platform runs all the tests Workflow
  50. 50. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop • Each bot for each platform runs all the tests • Bots archive and upload binary to the cloud Workflow
  51. 51. Problems
  52. 52. Solved: • Tests isolation (mostly database state) Problems
  53. 53. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Problems
  54. 54. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: Problems
  55. 55. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time U don’t like me? =( Problems
  56. 56. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time • Sync with real backend Problems
  57. 57. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time • Sync with real backend • Flaky tests Problems
  58. 58. Future plans
  59. 59. • Split to smaller frameworks - to improve run time Future plans
  60. 60. • Split to smaller frameworks - to improve run time • Setup Buildasaur - to run tests on Github pull requests Future plans
  61. 61. • Split to smaller frameworks - to improve run time • Setup Buildasaur - to run tests on Github pull requests • Record/replay tests for MockTransportSession - to keep it in sync Future plans
  62. 62. That’s it!

×