Your SlideShare is downloading. ×
Using DevOps to Improve Software Quality in the Cloud
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Using DevOps to Improve Software Quality in the Cloud

145
views

Published on

DevOps is gaining popularity as a way to quickly and successfully deploy new software. With all the emphasis on deployment, software quality can sometimes be overlooked. In order to understand how …

DevOps is gaining popularity as a way to quickly and successfully deploy new software. With all the emphasis on deployment, software quality can sometimes be overlooked. In order to understand how DevOps and software testing mesh, Glenn Buckholz demonstrates a fully implemented continuous integration/continuous delivery (CI/CD) stack. After describing the internals of how CI/CD works, Glenn identifies the touch points in the stack that are important for testing organizations. With the now accelerated ability to deliver software, the testing groups need to know how this technology works and what to do with it because swarms of manual testers will not be able to keep up. Glenn demonstrates where and how to use automated testing, how to collect and make sense of the massive amount of test results that can be generated from CI/CD, and how to usefully apply manual testing.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
145
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. T21 Test Automation 5/8/2014 3:00:00 PM Using DevOps to Improve Software Quality in the Cloud Presented by: Glenn Buckholz Coveros Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  • 2. GlennBuckholz Coveros Currently, Glenn Buckholz leads CI and deployment automations efforts at Coveros. Using his fifteen years of industry experience, Glenn brings success to his customers. Beginning his career as a consultant implementing automated test frameworks, he introduced the concept of change management to many, many projects. Glenn then decided to become a part of honest society and settled down at the Public Company Accounting Oversight Board as their full time enterprise change manager. After several years, he ventured back into the real world at Coveros, specializing in implementing agile practices, implementing CI, and engineering configuration management instead of simply documenting it.
  • 3. 4/26/2014 1 © Copyright 2014 Coveros, Inc. All rights reserved. Using DevOps to Improve Software Quality in the Cloud STAR EAST 2014 – 08 May 2014 Glenn Buckholz glenn.buckholz@coveros.com © Copyright 2014 Coveros, Inc. All rights reserved. Agenda What did I do? What is it good for? Demo Architecture Workflow Orchestration Cost vs Velocity Data Analysis What is Coveros doing with it for customers?
  • 4. 4/26/2014 2 © Copyright 2014 Coveros, Inc. All rights reserved. What did I do? Combined several technologies that Coveros is familiar with plus Puppet to demonstrate a full implementation of CI (Continuous Integration) This could be titled “Everything I wanted to do on Forge.mil but am not allowed.” Specifically, I cobbled together: Jenkins Puppet Puppet master EC2 Linux CURL ANT © Copyright 2014 Coveros, Inc. All rights reserved. What is it good for? Fail Faster Used with permission from the people at Extra Credits. http://www.youtube.com/extracredits
  • 5. 4/26/2014 3 © Copyright 2014 Coveros, Inc. All rights reserved. Demo © Copyright 2014 Coveros, Inc. All rights reserved. Major Points How is this DevOps? Releases are engineered, the systems engineer is simply pushing the button. If the list of manual installation instructions is more than one paragraph there is more engineering work to do. Code is the documentation. Operations people now govern the access, developers govern the how. New testing concerns. Deployment code. OS templates. Developers now have the job of convincing operations to run their deployment code.
  • 6. 4/26/2014 4 © Copyright 2014 Coveros, Inc. All rights reserved. Architecture © Copyright 2014 Coveros, Inc. All rights reserved. Workflow Developer commits code. Jenkins detects code change. Proper RPMs are built and distributed to repositories. Jenkins contacts EC2 and spins up blank instances. Puppet is delivered to the blank instances. Jenkins works with the Puppet Master for orchestration. Jenkins confirms that the application is running. Jenkins fires off automated tests.
  • 7. 4/26/2014 5 © Copyright 2014 Coveros, Inc. All rights reserved. Code Commit – Different Approaches Something has to trigger the testing. A code commit is typically a significant enough change to require some measure of retesting. Unit test are the minimum criteria to the publishing a change. Even if the code is bad the system must still run. By convention, code commits should be complete and significant. Each commit now carries a cost. This should be enforced by training. For larger projects with large volumes of © Copyright 2014 Coveros, Inc. All rights reserved. RPMs Packaging To move quickly code must be delivered in a consistent format. Copying files is too unpredictable and error- prone. Permissions Location Clean up of old file Dependency checking Any of the following are good choices RPM – Centos, RHEL Emerge/Portage - Gentoo Deb – Debian, Ubuntu
  • 8. 4/26/2014 6 © Copyright 2014 Coveros, Inc. All rights reserved. Cloud Provider The main issue is elasticity. Number of servers will vary. Tests will be running in parallel. Need a way to catalog all the machines Assign metadata to VMs Ability to group VMs and test results by deployment Automatically mapping code commits to cloud deployments is necessary for proper bookkeeping. © Copyright 2014 Coveros, Inc. All rights reserved. OS templates Each OS must originate with a standard template. All change must be managed by Puppet or some other deployment language If the OS is not standard too much time will be spent engineering around OS peculiarities CM must be mastered Every change to a standard OS must be recorded. Every change must be programmed, no hand- jamming All deployments are now hands-off You can almost get Continuous Delivery for free. OS Customizations for only two reasons Integrate into the Jenkins/Puppet Integrate into the cloud provider from a standard
  • 9. 4/26/2014 7 © Copyright 2014 Coveros, Inc. All rights reserved. OS Template Testing Must Be Automated RHEL, CENTOS, Windows change every month, Manual checks must be limited to keep up. Security STIG - Manual OpenSCAP – STIG Automated OWASP top 10 - Automated Organizational Standards Centralized Login Standard directories etc © Copyright 2014 Coveros, Inc. All rights reserved. Deployment Testing This is a new area since deployments are now code. No traditional tools to test this type of functionality Currently methods are mostly manual. Small automated tests must be constructed to help with orchestration. It either worked or it didn't Manually sifting through logs to troubleshoot. Who does this? Unfortunately, either a sysadmin or a
  • 10. 4/26/2014 8 © Copyright 2014 Coveros, Inc. All rights reserved. Automated Testing Automated testing must be an established practice. You cannot perform the volume of tests required for CI efficiently without some sort of automated check to separate the good changes from the bad changes. The cost and practice must be understood within the development and testing organization. Testing resources must be able to handle max capacity. Tests must be able to run in parallel against s © Copyright 2014 Coveros, Inc. All rights reserved. Role of Manual Testing Manual Functional Testing DOES NOT GO AWAY! Automated tests are not as effective as a person because they are so specific. Manual Testing is now a precious resource that can only be applied in a limited fashion. Time is what limits Manual testing 1-2 weeks of testing while code is written. 2-3 weeks of total testing effort. Testing effort is divided. Automation of new functionality testing. Spot manual testing. Manual free form regression. Targeted regression based on automated results.
  • 11. 4/26/2014 9 © Copyright 2014 Coveros, Inc. All rights reserved. Disposition of the Systems Test reporting and the cloud provider must be linked through a dashboard. Manual testers must be able to verify failures. Testers must have access to the broken running system Access will allow reproducibility and verification Developers must be able to identify broken VMs Access to the app to see the unexpected behavior © Copyright 2014 Coveros, Inc. All rights reserved. Orchestration This is the combination of architectural components and services in a meaningful fashion to produce a running system. For this example: Database App server Database initialization App server initialization Coordinated starting of all the services
  • 12. 4/26/2014 10 © Copyright 2014 Coveros, Inc. All rights reserved. Costs vs Velocity Velocity comes with parallelism. Tests running in parallel Code being tested in a pipelined fashion. Cost Number of testing machines and infrastructure. Number of parallel instances x machines per instances Storage – keeping failed instances around Budget and Team size Even with infinite money if you have a team of 5, the 400th VM is likely only giving very little marginal utility. When money is the constraint, developer and tester habits must best utilize the limited VMs wisely. © Copyright 2014 Coveros, Inc. All rights reserved. Data Analysis In order to move quickly the right people must get the right data. Test Results Quick access to the failed test Quick access a screen shot of the failure Ability to assign errors to the proper developers Integrated bug tracking. Server Analysis Developers need to zero in on failed servers quickly Developers need to link failures to their code quickly Developers need to separate system/OS failures
  • 13. 4/26/2014 11 © Copyright 2014 Coveros, Inc. All rights reserved. Dashboard (Data Analysis Visualization) Centralization of results. Sonar Cube Custom pages Views for Management, Testers, and Developers. Traceability Features all have GUIDs GUIDs linked to code commits Unit tests linked to GUIDs Automated tests are linked to feature GUIDs VMs linked to list of new GUIDs © Copyright 2014 Coveros, Inc. All rights reserved. What is Coveros Doing with it for Customers? Training and Process improvement Being the exemplar on a project to move this practice to other projects. Showing the change in the cost to test a feature within an organization Implementing pieces of the CI to show marginal improvements. Using the gains marginal gains to sell the whole process.
  • 14. 4/26/2014 12 © Copyright 2014 Coveros, Inc. All rights reserved. Level of Effort The demonstration needs to be customized for each environment. Very few projects that I have dealt with are “the same”. While this demo can be a guide, I don't believe it can be generalized to a solution, each environment must be customized. This implementation required research into the APIs and inner workings of several complicated software suites and services. The engineering effort is commensurate with the complexity of the architecture. © Copyright 2014 Coveros, Inc. All rights reserved. Why Link All These Technologies Together? The industry is moving this way in various forms. AWS templates Service Mesh JuJu Hadoop Flexiant Even though the products are varied, they all share general themes. Inventory Relational Mapping OS Templates Virtual Network Capabilities If you know one you can have a feel for them all.
  • 15. 4/26/2014 13 © Copyright 2014 Coveros, Inc. All rights reserved. Thoughts? Questions? Thank you for your time.