Deployment Testing in an Internet World


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Deployment Testing in an Internet World

  1. 1. Deployment Testing in an Internet World Michael Cookson CSQA, Manager, DTT Mary Wood, Senior QA Analyst, DTT April 21, 2010
  2. 2. Deployment Testing <ul><li>AGENDA </li></ul><ul><ul><li>Brief overview of MRO Software </li></ul></ul><ul><ul><li>The Challenge </li></ul></ul><ul><ul><li>Specific Issues </li></ul></ul><ul><ul><li>Creation and Role of Deployment Test Team </li></ul></ul><ul><ul><li>Next Steps </li></ul></ul>If at any time, there are questions, please shout them out!
  3. 3. Deployment Testing <ul><li>MRO Software, Inc. </li></ul><ul><ul><li>MRO = Maintenance, Repairs and Operations </li></ul></ul><ul><ul><li>Office in London Ontario is the operations center for the hosted production site </li></ul></ul><ul><ul><li>We host a series of Internet applications, allowing for the searching and purchasing of operational materials </li></ul></ul><ul><ul><li>Includes workflow, Purchase Order integration to client distribution systems, real time pricing and availability specific to purchasing organizations, order status and integration to different ERP systems for control of inventory </li></ul></ul>
  4. 4. Deployment Testing <ul><li>Mike Cookson started with MRO Software in November 1999 to become the test manager </li></ul><ul><li>Mary Wood joined MRO Software in June 1997, and moved through a series of positions before becoming the initial member of the test team in 1998 </li></ul><ul><li>Initially this new test team was responsible for the functional testing of the product and was part of the development organization </li></ul><ul><li>Later, in 2000, the test team was moved to the operations area, but maintained its functional testing focus </li></ul><ul><li>This created some conflict, as we will discuss </li></ul>
  5. 5. The Challenge
  6. 6. Deployment Testing <ul><li>The Internet – Client Perspective </li></ul><ul><ul><li>Internet environment demands 7x24 availability </li></ul></ul><ul><ul><li>Clients are increasingly dependant on the availability of the system to support their business processes </li></ul></ul><ul><ul><li>In order to be successful in this marketplace, need to be able to provide the highest availability </li></ul></ul><ul><ul><li>Many contracts will have a Service Level Agreement (SLA) attached with penalties for not hitting targets </li></ul></ul>
  7. 7. Deployment Testing <ul><li>The Internet – MRO Software Perspective </li></ul><ul><ul><li>We host applications that our clients access </li></ul></ul><ul><ul><li>Multiple teams within MRO Software provide changes to the hosted environment </li></ul></ul><ul><ul><li>These changes usually arrive independently of one another, but could affect other components </li></ul></ul><ul><ul><li>Many of these changes are attached to potential revenue contracts </li></ul></ul><ul><ul><li>The change needs to be implemented in a timely manner, with delivery commitments already made to clients </li></ul></ul><ul><ul><li>Many clients connections can only be tested in the production environment because they do not have test back end systems and firewall issues </li></ul></ul>
  8. 8. Deployment Testing Bottom Line: Change is needed for revenue; Change can cause loss of revenue!
  9. 9. Deployment Testing <ul><li>The Challenge: </li></ul>Develop a system to ensure the timely implementation of change into the hosted production environment, while at the same time ensuring the change has minimal negative impact in terms of availability of the systems and client connections.
  10. 10. Deployment Testing <ul><li>What does that mean? </li></ul><ul><ul><li>Most companies perform the deployment testing tasks we are about to describe in one form or another </li></ul></ul><ul><ul><li>The question is whether these tasks need to be formalized or more focus put on them </li></ul></ul><ul><ul><li>This presentation will describe the role of the Deployment Test Team at MRO Software and why we have organized it this way </li></ul></ul>
  11. 11. Deployment Testing <ul><li>MRO Software was organized with a traditional development organization and functional testing organization </li></ul><ul><li>This organization left some ‘gaps’ in terms of coverage </li></ul><ul><li>Organizationally, we were not ready to fill these gaps </li></ul><ul><ul><li>For example: </li></ul></ul><ul><ul><ul><li>Who is responsible for coordinating changes being made by multiple teams? </li></ul></ul></ul><ul><ul><ul><li>Who is responsible for ensuring the changes are made in the change windows? – 6x24 needed by clients currently and shrinking </li></ul></ul></ul><ul><ul><ul><li>Who is ensuring changes integrate with the client systems? – not always possible in the development environment </li></ul></ul></ul><ul><ul><ul><li>Who is responsible for ensuring the dependancies for environments are taken into account? </li></ul></ul></ul>
  12. 12. Deployment Testing <ul><li>There were many other issues that we were trying to deal with </li></ul><ul><li>These issues were summarized into four broad categories </li></ul><ul><ul><li>Change Management </li></ul></ul><ul><ul><li>Implementation Management </li></ul></ul><ul><ul><li>Client Management (at a tactical level) </li></ul></ul><ul><ul><li>Test Environment Management </li></ul></ul><ul><li>These issues were beginning to have an impact on the reputation and ability of MRO Software to deliver a quality product to the hosted environment in a reasonable time frame </li></ul>
  13. 13. Deployment Testing <ul><li>The Test Team at MRO Software had a functional testing focus, although part of the operations group and so functional and operational issues had to be considered simultaneously at each deploy </li></ul><ul><li>This was different from all other parts of the organization where the test team was part of the development organization </li></ul><ul><li>The Internet world demanded more than just functional testing, but there was no room in the organization for anything but functional testing </li></ul><ul><li>The Customer’s business was directly affected by the success or failure of the code deploy </li></ul>
  14. 14. Deployment Testing <ul><li>A number of pressure points were converging </li></ul><ul><li>Technical and operational issues were mounting </li></ul><ul><li>Organizationally we needed to bridge the gap between functional testing and operational issues </li></ul>
  15. 15. Deployment Testing <ul><li>The evolution of the Deployment Testing Team was in direct response to the pressure points </li></ul><ul><li>Under the Operations umbrella the Functional Test Team was beginning to take ownership of some of the processes </li></ul><ul><ul><li>For example Change Management and Implementation Management </li></ul></ul><ul><li>A completely different mind set and conflicting set of priorities were becoming evident </li></ul><ul><li>Operational testing tasks and functional testing tasks had different goals, objectives and accountabilities </li></ul><ul><li>Staff was having difficulties switching between these </li></ul>
  16. 16. The Solution
  17. 17. Deployment Testing <ul><li>The conclusion was that the functional team had to be split and the responsibilities divided </li></ul><ul><li>Functional testing would return to the Development organization </li></ul><ul><li>Deployment Test Team was created in the Operations area to assume ownership of the Operational processes </li></ul>
  18. 18. Deployment Testing The Mission of the Deployment Test Team is to manage all changes to the production hosted environment to ensure minimal negative impact to our customers.   This will be accomplished through the application of a formal, structured approach to every change to the hosted environment in order to provide the benefits of the change in a timely manner.   The Deployment Test Team will employ: Risk Assessment Process to identify the appropriate verification and validation procedures; Change Management Process to co-ordinate varied resources to ensure a smooth introduction into the production environment and ensure that all impacted areas have the necessary information regarding the change; Manual and automated tools and test plans to ensure the change has been properly implemented.   The goal of the Deployment Test Team is not necessarily to apply perfect change, but rather, to ensure that every change to the production environment is applied perfectly.
  19. 19. Deployment Testing Deployment testing, quite simply, is the actions taken to ensure a deliverable or change is properly implemented into a production environment. The word ‘Testing’ in this case, does not adequately cover the myriad of tasks necessary to ensure the changes make it to production successfully.
  20. 20. Deployment Testing <ul><li>So what are the items that the DTT are responsible for? </li></ul><ul><ul><li>Implementation Management </li></ul></ul><ul><ul><li>Change Management </li></ul></ul><ul><ul><li>Verification of Deployments </li></ul></ul><ul><ul><li>Ongoing Client communication and coordination </li></ul></ul><ul><ul><li>Maintenance of a mirror test environment </li></ul></ul>
  21. 21. Deployment Testing <ul><li>Implementation Planning owned by DTT </li></ul><ul><ul><li>For every change, what is needed to successfully implement? </li></ul></ul><ul><ul><li>What dependencies are there? </li></ul></ul><ul><ul><li>How can we test the plan? </li></ul></ul><ul><ul><li>How can this affect the clients with connections to our software? </li></ul></ul><ul><ul><li>The detail needed in the implementation plan is dependent on the risk of the change </li></ul></ul><ul><ul><li>We have developed a very simple risk model to help us identify low, medium and high risk projects and based on that, develop the implementation plan </li></ul></ul><ul><ul><li>Includes back out planning and verification of the plan </li></ul></ul>
  22. 22. Deployment Testing <ul><li>Change Management Process owned by DTT </li></ul><ul><ul><li>What changes are expected? </li></ul></ul><ul><ul><li>What is the time frame for these changes? </li></ul></ul><ul><ul><li>What else is planned for that change window? </li></ul></ul><ul><ul><li>Are all the impacted areas aware of this change? </li></ul></ul><ul><ul><li>Do clients need to be notified – who and when? </li></ul></ul><ul><ul><li>Ensures all change is identified before the date of implementation </li></ul></ul><ul><ul><li>If multiple changes needed the same window, ensures the different teams are involved for cross dependencies </li></ul></ul>
  23. 23. Deployment Testing <ul><li>Verification of the change in the production environment is owned by the DTT </li></ul><ul><ul><li>Once a change is implemented, how do we know the change was applied correctly? </li></ul></ul><ul><ul><li>What are the indicators to prove all the changes arrived in production correctly? </li></ul></ul><ul><ul><li>What regression tests should be done to ensure client connectivity and critical functions have not been impacted after the change? </li></ul></ul><ul><ul><li>Deploying the code is only half the battle </li></ul></ul><ul><ul><li>Need to ensure all clients are connecting normally, and the application is also functioning </li></ul></ul><ul><ul><li>If things aren’t working, then need to invoke the back out plan and then verify the back out worked </li></ul></ul>
  24. 24. Deployment Testing <ul><li>Ongoing Client Communication and Coordination </li></ul><ul><ul><li>What impact does this change have on the client? </li></ul></ul><ul><ul><li>Is there a risk of disrupting their connections to the software? </li></ul></ul><ul><ul><li>How can we verify these changes? </li></ul></ul><ul><ul><li>What change window do we have based on the client’s business needs? </li></ul></ul><ul><ul><li>Involves continual communication with our clients and with the teams that are working with the clients to ensure their needs are understood </li></ul></ul><ul><ul><li>The operations group owns the relationship with the client at a tactical level, the DTT is one tool used to maintain and enhance this relationship </li></ul></ul>
  25. 25. Deployment Testing <ul><li>Maintenance of a Mirror test environment </li></ul><ul><ul><li>To ensure deployments happen appropriately, need to have a practice site </li></ul></ul><ul><ul><li>This mirror site needs to be as close to the live environment as possible </li></ul></ul><ul><ul><li>It is not a place to perform functional testing – by the time the code is applied here, it is functionally correct </li></ul></ul><ul><ul><li>Used to test deploys, backouts, connect to client back ends and perform transactions </li></ul></ul>
  26. 26. Deployment Testing <ul><li>What is the DTT not responsible for? </li></ul><ul><ul><li>The functional testing of the change </li></ul></ul><ul><ul><li>DTT assumes the change has been tested and is ready to be put into the production environment </li></ul></ul><ul><ul><li>The actual deployment of the change – done by another operations team </li></ul></ul><ul><ul><li>Work closely with the team performing the deployment to ensure their needs are included in the planning </li></ul></ul><ul><ul><li>The Deployment Test Team is not doing any work that another team is responsible for. We do not redo or recheck previously done work. </li></ul></ul>
  27. 27. Deployment Testing <ul><li>Current Status </li></ul><ul><ul><li>Change Process has been defined and implemented </li></ul></ul><ul><ul><li>Risk Assessment has been created and is used for each change </li></ul></ul><ul><ul><li>Checklist of key tests for each client has been created and is used to check each change, based on the change and the risk </li></ul></ul><ul><ul><li>Automation begun on these key tests </li></ul></ul><ul><ul><li>Mirror environment in place for key applications and plans in place to create mirror environments for the others </li></ul></ul><ul><ul><li>Skills inventory completed, and self assessments done. </li></ul></ul>
  28. 28. Deployment Testing <ul><li>The splitting of the test team allowed the DTT to focus on the operational issues </li></ul><ul><li>Due to the way MRO Software splits the development and operational responsibilities, needed to ensure the operations tasks had the focus required </li></ul><ul><li>Development needs to focus on the creation and enhancement of new products </li></ul><ul><li>Operations needs to focus on the client relationship on a day to day basis and the availability of the hosted production applications </li></ul><ul><li>The DTT team can focus on these tasks without the issues of functional testing and project deadlines </li></ul>
  29. 29. Deployment Testing <ul><li>Key Success Factors for the DTT </li></ul><ul><ul><li>Marketing – make sure all areas aware of the role of the DTT </li></ul></ul><ul><ul><li>Ensure all change is properly planned for </li></ul></ul><ul><ul><li>All change implemented within the change window </li></ul></ul><ul><ul><ul><li>Need to perform all the tasks necessary, but no more – implies a very solid risk assessment process </li></ul></ul></ul><ul><ul><ul><li>Need to automate as much of the verification as possible to save time </li></ul></ul></ul><ul><ul><li>External clients are not negatively impacted by change </li></ul></ul><ul><ul><ul><li>All connections available when change is done </li></ul></ul></ul><ul><ul><ul><li>We minimize impact to world wide clients for down time due to change </li></ul></ul></ul><ul><ul><ul><li>Ensure we will not need to perform an emergency back out of upgrades because something was missed in the deployment </li></ul></ul></ul>
  30. 30. What About You?
  31. 31. Deployment Testing <ul><li>Does your organization need focus on these responsibilities? </li></ul><ul><li>If, around the water cooler, you hear these questions, it is possible you want to put some focus on these tasks </li></ul><ul><ul><li>The conversion is still going? That’s over 70 hours! </li></ul></ul><ul><ul><li>Why didn’t the change go in? This project has been in the works for months now! </li></ul></ul><ul><ul><li>It worked in development. </li></ul></ul><ul><ul><li>Why is production running a different version of the database? Of course it won’t work </li></ul></ul>
  32. 32. Deployment Testing <ul><ul><li>Why wasn’t our change tested with the other change? Everything has to go at the same time for this to work. </li></ul></ul><ul><ul><li>Why are these two changes scheduled for the same night? They have to be implemented separately. </li></ul></ul><ul><ul><li>This is an emergency. How do we get our change in there now?!?! </li></ul></ul><ul><ul><li>Our change went in and is working. Of course, it broke everything else, but that is not my problem. </li></ul></ul>
  33. 33. Deployment Testing <ul><li>If you are hearing conversations like this, then it is probable that some focus needs to be put on the tasks we have discussed </li></ul><ul><li>How you implement will depend on your organizational structure and how responsibilities are divided </li></ul><ul><li>The creation of the DTT is working at MRO. At your place, you may need a different structure to work, or maybe, the DTT will work for you as well… </li></ul>
  34. 34. Next Steps @ MRO
  35. 35. Deployment Testing <ul><li>Next steps for the DTT </li></ul><ul><ul><li>Need to implement a tool to better track Changes </li></ul></ul><ul><ul><li>Finish the Mirror Environment for all applications </li></ul></ul><ul><ul><li>Continue to build the automated test suite </li></ul></ul><ul><ul><li>Enhance and refine the checklist for verification procedures </li></ul></ul><ul><ul><li>Close the gap between the identified skills needed to be successful and our current skill set and level </li></ul></ul><ul><ul><li>Continue to build the visibility of the DTT at MRO Software </li></ul></ul>
  36. 36. Deployment Testing <ul><li>Next Steps for the DTT </li></ul><ul><ul><li>Identify the measures for the Deployment Test Team so we can: </li></ul></ul><ul><ul><ul><li>Continually improve the Change Process </li></ul></ul></ul><ul><ul><ul><li>Continually improve Implementation Process </li></ul></ul></ul><ul><ul><ul><li>Continually improve the Risk Assessment Process </li></ul></ul></ul><ul><ul><ul><li>Continually improve our client relationships </li></ul></ul></ul><ul><ul><ul><li>Continually improve the effectiveness of our test environments </li></ul></ul></ul>
  37. 37. Deployment Testing