Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Testing regression
Similar to Testing regression (20)
Recently uploaded
Recently uploaded (20)
Testing regression
- 1. Preventing Plan Regressions During Testing Richie Lee
- 3. Question: Why do bugs make their way into Production? Answer: Because we were too busy testing for bugs that aren’t there… Pop Quiz
- 4. Find More Bugs with This Simple Trick… • Bugs are most likely in two places: • In New Features • In Old Features
- 5. Finding Bugs • New Features are the most likely place to find bugs as test coverage is theoretically zero. • New features that interface with old features may create/uncover bugs previously undiscovered • Need to test new features, and then verify old features work and work well.
- 6. Regression Testing • Ensuring that any change has not negatively impacted the functional and non functional aspects of the application; that it works, and that it works well.
- 8. Database Test Driven Development Frameworks • tSQLt • SSDT Unit Tests • DBFit • MSTest
- 9. Testing Challenge • Verify that Query Plans Matched • Test large databases (in terms of data volume) on a build server ideally using only LocalDB
- 10. Testing Challenge • Query Optimizer uses the following to generate a query plan: • The database metadata • The hardware environment • The database session state
- 11. Test Database To Test The Test Framework • Scenario One: Verify that a certain index on table is searched using an index seek and not a scan • Scenario Two: Make a Change and Verify that there are no negative changes to the query plan • Scenario Three: Use DBCC OPTIMIZER_WHATIF to mock the hardware environment • Scenario Four: Verify that the same query plan is used when changing the compatibility level from 2012 to 2014.
- 12. Future Changes • Remove XPath
- 13. ?
Editor's Notes
- Ask if there are developers in the room? If there are any DBA’s in the room? Ask if there are any testers in the room? If there are they may want to look away now.
- Started off as a tester, and I found the job pretty tough and thankless. Didn’t sit comfortably with me as the job is to find fault with other peoples work, which in all fairness I couldn’t do. Then I realised that developers are really bad at writing tests, so testers are a necessary evil. One of my quests is to help developers write better tests to catch bugs.
- I promise you: this will be the only time that I will ask for audience participation… There are bugs found in test that were fixed, and so we were finding bugs, just not this bug. If I knew where the bugs were, you’d have to question my integrity. None of us are Neo…. So we need to get good at guessing where bugs are going to be. I’m not going to delve into practices like Boundary Value Analysis, or Edge Case/Corner Case Testing. Because we cannot test 100% all scenarios within the timeframe we have for testing, we have to make an educated guess on where to focus my testing to maximise my chances of finding bugs.
- Right, so like all clickbait, this is pretty misleading and not very helpful…. So lets expand upon this.
- This makes sense: 0 test coverage means that there is more likely of uncovering bugs there. Presuming old features have had tests run on them, these tests may be redundant/require changing/require re-running to verify that the tests passed as expected.
- And so as a product grows, unless the duration of our sprints are increasing to accommodate the amount of regression testing required, we need to automate testing. Clearly, unit testing is the easiest to automate.
- The testing triangle… and cloud….. A majority of bugs can be uncovered at the point of unit testing: we are able to test the code paths and scenarios quicker and so are more likely to find bugs that will cause a bigger impact. hence it being at the bottom of the triangle not because it is least important but because it takes up a greater surface area than subsequent areas. Story of the pink triangles and colour of the lines…. About UI.
- tSQLt: Good: assertions of expected against actual; mocking objects (strips constraints to help isolate testing on single stored procedures), easy to add to builds. Bad: runs within transactions, so can cause issues with nested transactions SSDT Unit Tests Good: assertion testing, checksum checking, pre and post unit test setup for data, easy to add to build. Bad: missing mocking features that makes tSQLt easy to use DBFit Good: uses Fit, a popular test framework, can alter/modify with own fixtures/stored procedures, readable tests, can be expanded to integration testing. Bad: No one has ever heard of it. MSTest Good: writing your own framework means you can do whatever you wish. Which is good if you’re testing something out there. Bad: some assembly required.
- My decision was to go down the MSTest route, as I was testing the demos written for SQL Server Training. This included lock escalation, clustered columnstore, in memory oltp, the evils of user defined functions, truncate vs delete, forced parameterisation…. So none of the frameworks available really accommodated this kind of testing. Oh… localdb. Noble ideal but victim to Microsoft marketing.
- Instead of mocking the data, if we mocked the database using statistics then we can generate a query plan that closely resembles that of production in our unit testing. Hardware we will come to later Database session state is defined on the connection string; so this must match that used in production. Talk abut how ideally you’d use same code used in production but sometimes not possible as test runs across a single connection whereas the standard is to connect and disconnect per command. Plan is to execute showplan, generate the query plan, and use xpath to assert that the operator expected is the actual operator returned.
- My decision was to go down the MSTest route, as I was testing the demos written for SQL Server Training. This included lock escalation, clustered columnstore, in memory oltp, the evils of user defined functions, truncate vs delete, forced parameterisation…. So none of the frameworks available really accommodated this kind of testing. Use DBCC OPTIMIZER_WHATIF to mock the hardware environment
- Xpath is the bane of testing because it is time consuming to write and can be flaky. Also we are severely limiting what we can test here: we could be testing just 2/3% of all plan operators out of a large query plan. Also finding xpaths is time consuming. Although there is the argument that a change in structure of the query plan will cause the xpath to fail so we are sort of black box testing the rest of the query plan, a branch could change and not affect the overall structure of the query plan we are testing and so we may still have plan regression. So one idea is to remove strip out everything except the plan operators then maybe create a hash and compare hashes.