Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Testing Sync Engine
Testing (iOS) Sync Engine
Agenda
• Overall tests stats
• Our test strategies
• MockTransportSession
• Workflow
• Problems
• Future plans
Overall tests stats
• 3356 tests on OSX
Overall tests stats
• 3356 tests on OSX
• 3415 tests on iOS (+ apns tests)
Overall tests stats
• 3356 tests on OSX
• 3415 tests on iOS (+ apns tests)
• ~ 7.5 minutes to build and run all the tests on
one platform (bui...
• 3356 tests on OSX
• 3415 tests on iOS (+ apns tests)
• ~ 7.5 minutes to build and run all the tests on
one platform (bui...
• Unit tests ~ 80%
Overall tests stats
• Unit tests ~ 80%
• Integration tests ~ 20%
Overall tests stats
• Unit tests ~ 80%
• Integration tests ~ 20%
• Performance tests - 18
Overall tests stats
• Unit tests ~ 80%
• Integration tests ~ 20%
• Performance tests - 18
• Tests of tests
Overall tests stats
• Unit tests ~ 80%
• Integration tests ~ 20%
• Performance tests - 18
• Tests of tests
• Sorry, no UI tests
Overall tests ...
Our test strategies
What we DON’T do:
Our test strategies
What we DON’T do:
• test against http:// (real backend)
• test using http requests stubs (like
OHHTTPStubs)
Our test strat...
What we DON’T do:
• test against http:// (real backend)
• test using http requests stubs (like
OHHTTPStubs)
Why?
• Stubs a...
What we DON’T do:
• test against http:// (real backend)
• test using http requests stubs (like
OHHTTPStubs)
Why?
• Stubs a...
What we really do:
• Test against fake backend
Our test strategies
What we really do:
• Test against fake backend
Advantages
• Fast
• Flexible
• Independent
• Testable
Our test strategies
What we really do:
• Test against fake backend
Advantages
• Fast
• Flexible
• Independent
• Testable
Disadvantages
• Need ...
MockTransportSession
WTF is THAT?
MockTransportSession
WTF is THAT?
ZMTransportSession
ZMTransportSessi
onErrorCode
ZMBackgroundAc
tivity
ZMURLSessionCa
ncelTimer
ZMURLSessionSw...
WTF is THAT?
ZMTransportSession
MockTransportSession
WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
MockTransportSession
WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
• Sends http requests to scheduler
MockTrans...
WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
• Sends http requests to scheduler
• Handles...
WTF is THAT?
ZMTransportSession - is a delegate of a delegate
of NSURLSession
• Sends http requests to scheduler
• Handles...
WTF is THAT?
MockTransportSession - is a mock of
ZMTransportSession
MockTransportSession
WTF is THAT?
MockTransportSession - is a mock of
ZMTransportSession
MockTransportSession - acts as a fake backend
MockTran...
WTF is THAT?
• Make fake requests
MockTransportSession
WTF is THAT?
• Make fake requests
• Records requests and responds to them
MockTransportSession
WTF is THAT?
• Make fake requests
• Records requests and responds to them
• Manages internal database
MockTransportSession
WTF is THAT?
• Make fake requests
• Records requests and responds to them
• Manages internal database
• Can simulate remot...
WTF is THAT?
• Make fake requests
• Records requests and responds to them
• Manages internal database
• Can simulate remot...
• Before each test we insert objects in internal
database (in memory)
MockTransportSession
• Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
...
• Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
...
• Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
...
• Before each test we insert objects in internal
database (in memory)
• Test code make request using MockTransportSession
...
Workflow
• Finished feature - run all the tests
Workflow
• Finished feature - run all the tests
• Create pull request
Workflow
• Finished feature - run all the tests
• Create pull request
• Review, merge
Workflow
• Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
Workf...
• Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pus...
• Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pus...
• Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pus...
• Finished feature - run all the tests
• Create pull request
• Review, merge
• Release script runs all tests locally
• Pus...
Problems
Solved:
• Tests isolation (mostly database state)
Problems
Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Problems
Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
Problems
Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
• Reduce tests running time...
Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
• Reduce tests running time...
Solved:
• Tests isolation (mostly database state)
• Handling asynchronous code
Not yet solved:
• Reduce tests running time...
Future plans
• Split to smaller frameworks - to improve run time
Future plans
• Split to smaller frameworks - to improve run time
• Setup Buildasaur - to run tests on Github pull
requests
Future plans
• Split to smaller frameworks - to improve run time
• Setup Buildasaur - to run tests on Github pull
requests
• Record/rep...
That’s it!
Upcoming SlideShare
Loading in …5
×

Testing sync engine

229 views

Published on

Small presentation I've made for Wire team about testing process in iOS Sync Engine Team. You can read more in depth here - http://www.objc.io/issues/15-testing/xctest/

Published in: Mobile
  • Be the first to comment

  • Be the first to like this

Testing sync engine

  1. 1. Testing Sync Engine
  2. 2. Testing (iOS) Sync Engine
  3. 3. Agenda • Overall tests stats • Our test strategies • MockTransportSession • Workflow • Problems • Future plans
  4. 4. Overall tests stats
  5. 5. • 3356 tests on OSX Overall tests stats
  6. 6. • 3356 tests on OSX • 3415 tests on iOS (+ apns tests) Overall tests stats
  7. 7. • 3356 tests on OSX • 3415 tests on iOS (+ apns tests) • ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
  8. 8. • 3356 tests on OSX • 3415 tests on iOS (+ apns tests) • ~ 7.5 minutes to build and run all the tests on one platform (build ~ 40s, tests ~ 7 min) Overall tests stats
  9. 9. • Unit tests ~ 80% Overall tests stats
  10. 10. • Unit tests ~ 80% • Integration tests ~ 20% Overall tests stats
  11. 11. • Unit tests ~ 80% • Integration tests ~ 20% • Performance tests - 18 Overall tests stats
  12. 12. • Unit tests ~ 80% • Integration tests ~ 20% • Performance tests - 18 • Tests of tests Overall tests stats
  13. 13. • Unit tests ~ 80% • Integration tests ~ 20% • Performance tests - 18 • Tests of tests • Sorry, no UI tests Overall tests stats
  14. 14. Our test strategies
  15. 15. What we DON’T do: Our test strategies
  16. 16. What we DON’T do: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Our test strategies
  17. 17. What we DON’T do: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Why? • Stubs are not that flexible. Our test strategies
  18. 18. What we DON’T do: • test against http:// (real backend) • test using http requests stubs (like OHHTTPStubs) Why? • Stubs are not that flexible. • It’s still slow. Our test strategies
  19. 19. What we really do: • Test against fake backend Our test strategies
  20. 20. What we really do: • Test against fake backend Advantages • Fast • Flexible • Independent • Testable Our test strategies
  21. 21. What we really do: • Test against fake backend Advantages • Fast • Flexible • Independent • Testable Disadvantages • Need to write (and test!) code • Need to be in sync with real backend (can be improved) Our test strategies
  22. 22. MockTransportSession
  23. 23. WTF is THAT? MockTransportSession
  24. 24. WTF is THAT? ZMTransportSession ZMTransportSessi onErrorCode ZMBackgroundAc tivity ZMURLSessionCa ncelTimer ZMURLSessionSw itch ZMURLSession ZMTransportCodec ZMAccessToken ZMUserAgent ZMReachability ZMTransportData ZMTransportResp onse ZMTaskIdentifierM ap ZMExponentialBa ckoff ZMAccessTokenH andler ZMTransportRequ estScheduler ZMNetworkSocket ZMDataBuffer ZMWebSocketFra me ZMWebSocketHan dshake ZMWebSocket ZMPushChannelC onnection ZMTransportPush Channel ZMStreamPairThre ad MockTransportSession
  25. 25. WTF is THAT? ZMTransportSession MockTransportSession
  26. 26. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession MockTransportSession
  27. 27. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession • Sends http requests to scheduler MockTransportSession
  28. 28. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession • Sends http requests to scheduler • Handles authentication MockTransportSession
  29. 29. WTF is THAT? ZMTransportSession - is a delegate of a delegate of NSURLSession • Sends http requests to scheduler • Handles authentication • Opens/closes web socket connection MockTransportSession
  30. 30. WTF is THAT? MockTransportSession - is a mock of ZMTransportSession MockTransportSession
  31. 31. WTF is THAT? MockTransportSession - is a mock of ZMTransportSession MockTransportSession - acts as a fake backend MockTransportSession
  32. 32. WTF is THAT? • Make fake requests MockTransportSession
  33. 33. WTF is THAT? • Make fake requests • Records requests and responds to them MockTransportSession
  34. 34. WTF is THAT? • Make fake requests • Records requests and responds to them • Manages internal database MockTransportSession
  35. 35. WTF is THAT? • Make fake requests • Records requests and responds to them • Manages internal database • Can simulate remote changes. MockTransportSession
  36. 36. WTF is THAT? • Make fake requests • Records requests and responds to them • Manages internal database • Can simulate remote changes. • Can send push channel (web socket) events MockTransportSession
  37. 37. • Before each test we insert objects in internal database (in memory) MockTransportSession
  38. 38. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession MockTransportSession
  39. 39. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) MockTransportSession
  40. 40. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) • Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) MockTransportSession
  41. 41. • Before each test we insert objects in internal database (in memory) • Test code make request using MockTransportSession • It records and process incoming request (filtering by path and lots of ifs) • Performs some CRUD action on database depending on request (add message, create conversation, fetch conversation) • Creates and sends response. Optionally can delegate it to other object (i.e. test case). MockTransportSession
  42. 42. Workflow
  43. 43. • Finished feature - run all the tests Workflow
  44. 44. • Finished feature - run all the tests • Create pull request Workflow
  45. 45. • Finished feature - run all the tests • Create pull request • Review, merge Workflow
  46. 46. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally Workflow
  47. 47. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) Workflow
  48. 48. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop Workflow
  49. 49. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop • Each bot for each platform runs all the tests Workflow
  50. 50. • Finished feature - run all the tests • Create pull request • Review, merge • Release script runs all tests locally • Pushes to remote branches (master/develop) • Xcode bots make integration on each push to master/ develop • Each bot for each platform runs all the tests • Bots archive and upload binary to the cloud Workflow
  51. 51. Problems
  52. 52. Solved: • Tests isolation (mostly database state) Problems
  53. 53. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Problems
  54. 54. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: Problems
  55. 55. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time U don’t like me? =( Problems
  56. 56. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time • Sync with real backend Problems
  57. 57. Solved: • Tests isolation (mostly database state) • Handling asynchronous code Not yet solved: • Reduce tests running time • Sync with real backend • Flaky tests Problems
  58. 58. Future plans
  59. 59. • Split to smaller frameworks - to improve run time Future plans
  60. 60. • Split to smaller frameworks - to improve run time • Setup Buildasaur - to run tests on Github pull requests Future plans
  61. 61. • Split to smaller frameworks - to improve run time • Setup Buildasaur - to run tests on Github pull requests • Record/replay tests for MockTransportSession - to keep it in sync Future plans
  62. 62. That’s it!

×