Confidential Production Environment Data to Improve Testing Test based on established facts instead of assumptions Rev PA1 2011-10-26 1
Introduction According to many voices the future of testing is to move test into a production environment  Defects are no longer cheaper to find early in development, but just as cheap to fix in production, according to some  Testing in production finds problems at a scale that most groups can’t afford to duplicate with a test environment  Without being overly dramatic, one can assume that data from a production environment will be an important input to testing in the future
Rapid Deployment & Product EnvironmentFeedback Deployment of software has become more and more rapid Companies move towards more and more agile development  Multiple releases of software ensures that customer feedback for one release can be used to influence testing of future releases A framework for gathering this information is needed, and it is important to understand what product environment feedback is needed to improve testing
Production Environment Information If a framework is in place to collect data from a production environment, the important question is what information is actually needed to improve testing This information varies dependant on the software, and it is a question everyone who is releasing software will have to ask themselves in the future What information is important to extract from the production environment?
Production Environment InformationExample Resource usage What different variants and Software configurations degrade over of the time software are used Production Environment Information Stability in Performance production in production environment environment Most used applications and services
How to use the information The information gained can be used in a multitude of ways If you know what applications and services the customer uses most, you can focus your testing to those areas If you find degrades over time, you know to focus on that area Performance and stability issues in specific areas can give input to additional testing in those areas If you know which variants and configurations of the software is used most, then you can focus on using those variants during testing The actions taken depends on the information, which as we said previously in turn depends on the software
Conclusion Of course having a good production environment feedback structure in place will replace some testing, but the real gain will be in improving the testing that is still being executed It is important to identify the information needed from the production environment early, even before the feedback structure is in place – this will give important insight into what gain the production environment information will yield to your test execution Getting this product environment feedback structure in place will be key for almost all software development in the future
Reference The Liberation of Testinghttp://marlenacompton.com/?p=2514 Testing services in productionhttp://www.pnsqc.org/2011-conference/technical-paper-abstracts-and-bios-2#T-11 Test is Deadhttp://googletesting.blogspot.com/2011/06/test-is-dead.html Software G Forces: The Effects of Accelerationhttp://www.youtube.com/watch?v=KIkUWG5ACFYhttp://www.slideshare.net/KentBeck/software-g-forces Has Agile Peaked?http://drdobbs.com/architecture-and-design/207600615