There are so many testing frameworks available to us that we sometimes overlook a completely valid, and sometimes preferable option: writing our own. The drivers team at MongoDB focused over the last year on conforming to common APIs and algorithms but we needed a way to validate our consistency. We therefore ended up building our own testing DSL, REST service, and individual test frameworks. Using these common tests and the Ruby driver's test suite as examples, this talk will demonstrate when existing test frameworks aren't the best choice and show how you can build your own.

1. DIY
(Do-it-yourself)
Testing

2. Ruby Engineer on the
drivers team.
Based in Berlin.
Maintain the gems:
bson
mongo
mongoid
bson_ext
mongo_kerberos
@EmStolfo

3. MongoDB Drivers
C
C++
C#
Java
Node.js
Perl
PHP
Python
Ruby
Scala

4. MongoDB Drivers
Interface to the server.
10 official drivers available.
Many started as OS projects.
History

5. No specifications.
Divergent codebases
and features.
MongoDB Drivers
Reality

6. Server “interface” was not consistent.
MongoDB Drivers
Reality

7. Organizations can experience 1 interface.
Specifications guide design
and provide documentation.
Rewrites with collective knowledge.
MongoDB Drivers
Revamp

8. Server “interface” is a single product.
MongoDB Drivers
Revamp

9. Specifications
Equally valuable to the drivers team
as to the community.

10. Server Discovery and
Monitoring Spec
Logic required to make an application
using MongoDB highly available.

11. How to validate
compliance?

12. Requirements for
validating compliance
• Unit tests.
• Integration tests.
• Tests define as “data”.
• Language-agnostic.
• Reproducible scenarios.

13. DIY Testing
Our custom tests: Unit / Integration
Build your own test framework.

14. define
Format - YAML
Tests
Scenarios
Our approach:
“DIY” testing

15. benefits
Format:YAML
• Describes data.
• Can translate to actions.
• Most languages can use a YAML parsing
library.
• Driver authors write a reusable harness.
• Changes in the spec can be
communicated via additional YAML tests
or changes to existing ones.

16. Test requirements
• Unit tests.
• Integration tests.
• Defined as “data”.
• Language-agnostic.
• Reproducible scenarios.

17. Unit test - YAML
single/direct_connection_standalone.yml

18. Unit test - Test object
spec/support/server_discovery_and_monitoring.rb

19. Unit test - spec
spec/mongo/server_discovery_and_monitoring_spec.rb

20. Unit test - YAML
rs/discover_primary.yml

21. Test requirements
• Unit tests.
• Integration tests.
• Defined as “data”.
• Language-agnostic.
• Reproducible scenarios.
Mongo Orchestration

22. Mongo
Orchestration

23. Define clusters using JSON.
Manipulate clusters via RESTful API.
Mongo Orchestration
HTTP server providing REST interface to manage
multiple MongoDB processes on the same machine.
Implemented in python.

24. What about Automation!?
• Is intended for testing.
• Starts processes on one machine.
• Allows fault injection.
• Does not have a UI.
• Has no protection against downtime.
• Has one agent, so no resiliency.
• Does not handle operation tasks.
Mongo Orchestration..
!

25. Create different cluster topologies.
Single
Mongo Orchestration
Replica Set
Sharded Cluster

26. Setup

27. Single server
config options

28. Replica Set
config options

29. Sharded Cluster
config options

30. Define different installations
in a config file
mongo-orchestration.config

31. Set up a Replica Set
http://bit.ly/mo-usage

32. Benefits of MO
• Reproducible test scenarios.
• Abstracts differing configuration
options across MongoDB versions.
• Uniform interface regardless of OS.
• Ability to define different locations
of MongoDB installations for multi-
version testing.

33. Multi-version testing
Multi-topology testing
SSL
Authentication
Used internally
Drivers integration tests

34. No need for cluster managers
test/tools/mongo_config.rb
(cluster manager in 1.x ruby driver)

35. Jenkins testing

36. Integration tests

37. Integration tests
Test driver behavior.
Input: cluster state
Output: driver behavior
Mongo Orchestration

38. Test harness
(Ruby driver example)
PR #584 [WIP] Integration tests using Mongo Orchestration

39. YAML integration test
rs/connection/primary-not-available.yml (I)

40. YAML integration test
rs/connection/primary-not-available.yml (II)

41. Harness structure
1. Resource (MO cluster)
2. Specification (parsed YAML)
3. Test (can be run)
PR #584 [WIP] Integration tests using Mongo Orchestration

42. Resource
spec/support/mongo_orchestration/resource.rb

43. Specification
spec/support/mongo_orchestration/spec.rb

44. Test
spec/support/mongo_orchestration/operation/client_operation.rb

45. Restarted primary integration test
rs/connection/primary-restarted.yml (I)

46. Restarted primary integration test
rs/connection/primary-restarted.yml (II)

47. Restarted primary integration test
rs/connection/primary-restarted.yml (III)

48. How to build your own
testing framework

49. Ask yourself
1. Unit or integration tests?
2. Need to be language-agnostic?
3. Directly related to specs or
documentation?

50. Test framework entities
Resource
Wrapper around
the external component.

51. Test framework entities
Specification
The parsed YAML.

52. Test framework entities
State manipulator
Processor of phases to get
to a certain state.

53. Test framework entities
Tests
Can be executed.
Result is ‘pass’ or ‘fail’.

54. Make the test runner
as generic as possible.

55. Install Mongo Orchestration

57. Resources
http://bit.ly/ruby-kaigi-diy-testing
@EmStolfo