Get your database under control                 Automated testing with                 MongoDB and OffScaleOmer GertelCTO ...
Managingdatabases iscomplicated
Many people             are involvedDevelopers     QA   DBAs    IT
Production        MultipleDev      environments     Integration             QA
Multiplecopies and versions
We Help Companies…          Iterate Faster &      Release Better ProductsBy…      Shortening product cycles        with DB...
Copy dataWrite codeWrite unit tests                        Production        Dev                          Integration Clon...
Copy data                                    DataGroveWrite codeWrite unit tests                    Git for databases     ...
For developers, by developers  $   datagrove     tag -n=new_tag  $   datagrove     list  $   datagrove     load -n=old_tag...
Automated Testing• Integration tests• Build servers and continuous integration• Migrations  –Some MongoDB specific issues
Automated Tests•   Isolated•   Repeatable•   Researchable•   Fast
Automated Tests• For each test:  – Populate database  – Run test  – Drop database
What happens if a test breaks?•   Does it affect other tests?•   Does it stop the build cycle?•   How do you discover what...
Automated Tests• For each test:  – Load dataset  – Run test  – If test fails:     • Tag dataset at end of run
Schema Migrations
Schema Migrations
Schema MigrationsObject Structure & Index Changes• Migrate ALL the objects  – Test before you run• Support older structure...
Version management that doesn’t suck Make MongoDB a first class citizen of the development cycle
Existing MongoDB Support  For single node environments      $ datagrove load -      n=tag_name                      Mongod...
Existing MongoDB Support  For single node environments      $ datagrove tag -      n=tag_name                             ...
Upcoming MongoDB Cluster Support Conceptual architecture for taking a snapshot                                       $ dat...
Automated testing withOffScale & MongoDB• Data is a first class  citizen in your  environment• Testing is important to  ke...
Get your database under control                         Now available at                 http://off-scale.com/
Automated testing with OffScale and MongoDB
Automated testing with OffScale and MongoDB
Upcoming SlideShare
Loading in …5
×

Automated testing with OffScale and MongoDB

4,801
-1

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
4,801
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 20-30% of time, talked to many companies, copying and refreshing data for many purposes during the life cycle of the product (not just the production system)
  • Tales of BigCo & BigIT.“With DataGrove the R&D can do it himself in minutes instead in days”This also plays on the self-service on premise service-like feature of our product. Liberation of programmers from DBAs.
  • Tales of littleCo. Can’t reproduce. Have to constantly copy and refresh.
  • And then, we just had multiple versions, as we branched the code. So each developer had their own flavor of data that they liked.But all that just made us want to scream…
  • So if now there’s a lot of management and copying of data around, we turn the product development cycle into something more flexible. We bring the paradigms from source control into the database world, and can make your mongoDB a first class citizen in your development process just like code and other resources that are managed.
  • Introducing DataGrove, Git for databases. We do this by providing a snapshot mechanism that has been optimized to allow multiple users to quickly tag, restore and share the state of the database between team members and environments.
  • By developers, for developers.We try and make the experience as seamless as possible, so, for example, when you tag, we take care of whatever needs to be done to the database to enforce that the tag is consistant. With mongodb, this will be fsync + write lock for a second. There’s still a lot of work to be done – such as helping with the configuration of the instance, etc.Easy to integrate, should be able to connect with hooks to existing source control with a few simple lines of code. Also have API. We will provide connectors to most popular source control and build servers, but will be even happier if these will be owned and managed by the community.Our beta is available on our website, url at the end of presentation. Allows for a single server installation (tag & restore, without sharing) akin a time machine for databass. Supports popular OS and DBs Linux, OS X, mysql, postgresql and a single node mongodb (excited to announce).And it’s FREE, so we would love to hear back from you about how to improve.
  • Now, let’s talk about automated testing, because I know for a fact that when it comes to the database, there’s a tendency to act too boldly. Either because it’s too complicated to get a working copy of the data they need to test, or that it takes too much time to develop scripts to solve the issue of automated tests against a database.So they write unit tests. But in a NoSQL environment, when the database can get very large and varied (which is worse with NoSQL), testing the code alone is not enough.
  • Having unit tests to test the code is good, but not enough. There are several layers of testing, and the more automation you can have, the more efficient you will be in releasing new features and products, while maintaining a good cycle time and product quality, and protecting you from human error.
  • Automated testing with OffScale and MongoDB

    1. 1. Get your database under control Automated testing with MongoDB and OffScaleOmer GertelCTO @ OffScale
    2. 2. Managingdatabases iscomplicated
    3. 3. Many people are involvedDevelopers QA DBAs IT
    4. 4. Production MultipleDev environments Integration QA
    5. 5. Multiplecopies and versions
    6. 6. We Help Companies… Iterate Faster & Release Better ProductsBy… Shortening product cycles with DB management automation
    7. 7. Copy dataWrite codeWrite unit tests Production Dev Integration Clone Schema Clone schema Copy data Copy data Test the application Run integration QA
    8. 8. Copy data DataGroveWrite codeWrite unit tests Git for databases Production Staging DevClone Schema Clone schemaCopy data Copy dataTest the application Run integration QA
    9. 9. For developers, by developers $ datagrove tag -n=new_tag $ datagrove list $ datagrove load -n=old_tag $ datagrove unload • Works automagically • Easy to integrate with existing tools • Your personal database time machine for FREE
    10. 10. Automated Testing• Integration tests• Build servers and continuous integration• Migrations –Some MongoDB specific issues
    11. 11. Automated Tests• Isolated• Repeatable• Researchable• Fast
    12. 12. Automated Tests• For each test: – Populate database – Run test – Drop database
    13. 13. What happens if a test breaks?• Does it affect other tests?• Does it stop the build cycle?• How do you discover what went wrong?• Is the bug related to the data?• How can you reproduce the bug?• What happens when you push a fix?
    14. 14. Automated Tests• For each test: – Load dataset – Run test – If test fails: • Tag dataset at end of run
    15. 15. Schema Migrations
    16. 16. Schema Migrations
    17. 17. Schema MigrationsObject Structure & Index Changes• Migrate ALL the objects – Test before you run• Support older structures in code – Test code against old objects – Have datasets ready tagged by version
    18. 18. Version management that doesn’t suck Make MongoDB a first class citizen of the development cycle
    19. 19. Existing MongoDB Support For single node environments $ datagrove load - n=tag_name Mongod 1 Create virtual mount point: /media/offscale (1-10 sec) /media/offscale 2 mongod --dpath= /media/offscale path-to-store-path-to-old-data changes (optional)
    20. 20. Existing MongoDB Support For single node environments $ datagrove tag - n=tag_name {fsync:1,lock:1} 1 (using Java driver) Mongod 2 Tag point in time in virtual drive (1-10 sec) /media/offscale 3 db.$cmd.sys.unlock.findOne() path-to-store-path-to-old-data changes (optional)
    21. 21. Upcoming MongoDB Cluster Support Conceptual architecture for taking a snapshot $ datagrove tag-n=t Primary Primary SlavePrimary Cfg 1 Slave Slave Slave Cfg 2 Slave Pause shard Replica Set Cfg 3 1 Slave balancing Replica Set Configuration Issue datagrove tag for 2 Replica Set cluster configuration server 3 Issue datagrove tag,One node managed under One or more nodes one shard at a timeOffScale, per replica set, managed by OffScale for Resume shardallows for backup, recovery backup and recovery of 4 balancingand creating new nodes configuration servers
    22. 22. Automated testing withOffScale & MongoDB• Data is a first class citizen in your environment• Testing is important to keep your code working• Iterate faster & release better products
    23. 23. Get your database under control Now available at http://off-scale.com/
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×