Automation Culture: Essential to Agile Success


Published on

For organizations developing large-scale applications, transitioning to agile is challenging enough. If your organization has not yet adopted an automation culture, brace yourself for a big surprise because automation is essential to agile success. From the safety nets provided by automated unit and acceptance tests to the automation of build, build verification, and deployment processes, the iterative nature of agile demands a culture of automation across your engineering organization. Geoff Meyer shares lessons learned in adopting a test automation culture as the Dell Enterprise Systems Group simultaneously adopted Scrum and agile processes across its entire software product portfolio. Learn to address the practical challenges of establishing an automation culture at the outset by ensuring that your organizational makeover incorporates changes to your hiring, staffing, and training practices. Find out how you can apply automation beyond the Scrum team in areas including continuous integration, scale and stress testing, and performance testing.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Automation Culture: Essential to Agile Success

  1. 1. W16 Agile Testing 5/1/2013 3:00:00 PM Automation Culture: Essential to Agile Success Presented by: Geoff Meyer Dell, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Geoff Meyer A test architect in the Dell Enterprise Solutions Group, Geoff Meyer has more than twenty-seven years of experience as a software developer, manager, test architect, and business analyst. Geoff co-chairs the Agile Steering committee within Dell Enterprise Solutions Group which guides the software development practices of more than 600 development, test, and UX engineers across three Global Design Centers. He is an active member of the Agile Austin community.
  3. 3. 4/16/2013 Automation Culture: Essential to Agile Success STAR East 2012 Geoff Meyer, 24 April 2013 Last updated: February 28, 2013 Session Objectives • • Challenges automating in a HW-dependent environment • NOT Real-world examples − from a large organization perspective • IS Practical Solutions to What, When and What Not to Automate • IS Challenges to expect when adopting Test automation during your transition to Agile For Organizational Leaders driving Strategy • A Tools Discussion • Focused on how to develop Test Automation 1
  4. 4. 4/16/2013 Introductions 3 Geoff Meyer • Dell Inc, 1998 – present – Test Architect › Agile Test & Automation Strategy – Agile Steering Committee Co-Chair – Global Projects › 15 Scrum teams › 9 scrum teams › 7 Scrum teams • NCR Corp. 1984 – 1998 – SW developer, Project Lead, SW Manager • B.S. Computer Science, San Diego State University • Masters Engineering Management - NTU 4 2
  5. 5. 4/16/2013 Agenda • Why Automation is essential to Agile • The Dell Landscape and Agile @ Dell – Organizational Context Matters • Challenges and Common Pitfalls • The Automation Landscape • Foundations of an Automation Culture • Care and Feeding of the Automation Culture 5 The Need, The Challenge, The Pitfalls 6 3
  6. 6. 4/16/2013 Why is Automation So Important in Agile? • Near-term – Ensures that you don’t break what you just built – Provides safety net for developers & rapid feedback to new changes – Continuous Integration and use of Build Verification Test (BVT) • Long-term – Maximizes velocity of Scrum team – Creates capacity for Exploratory and ad-hoc Testing – Enables activities that can’t be done cost-effectively by humans And if you don’t… Projects rapidly build up an unmanageable regression suite Differences with Automation in Waterfall? In Waterfall… • Automated tests are derived from the backlog of completed features – In Agile, Automation can be incorporated in the requirement • Testing and automation is performed after Development is complete • Focus is on first-time discovery of defects and optimizing your test coverage vs. Agile… where automation provides immediate feedback to defect introduction 8 4
  7. 7. 4/16/2013 The Challenge Adopting Agile… …In an organization that doesn’t have an Automation Culture “Apply Test Automation in the context of your organizational automation needs” --- Bob Galen, iControl The Context @ Dell 10 5
  8. 8. 4/16/2013 Dell Enterprise Solutions Group Global Design and Development using Agile Scrum Silicon Valley Design Center Noida Design Center Austin Design Center Bangalore Design Center 11 Dell Enterprise Solutions Software Products • Server Systems Management • Converged Infrastructure Systems Management • Private Cloud Server Management • Console Plug-ins (i.e. for SCCM, vCenter…) Common Product Characteristics: • Large hardware support test matrix • Software is installed the Data Center • Enterprise update cycles ~6 months • Products must function even as underlying FW, BIOS, and Drivers are upgraded - SUSTAINING 12 6
  9. 9. 4/16/2013 Agile @ Dell Roles/Responsibilities Product Management Epic Prioritization & Business Value Represents Customer - Profiles/Roles Requirement Acceptance Product Owner User Story Development Customer rep. for Scrum team Story Acceptance Product Owner Proxy 1 per Scrum Scrum Master Facilitate, Remove Scrum impediments 1 per Scrum Development Architecture, Design, Development, Unit Test 4-5 per Scrum Test Test Design, Automation Development 2:1 InfoDev User-facing text, error messages, online help 1 per 3 scrum team UI UI Design, Development, Unit Test 1 per Scrum Scrum teams 13 Agile @ Dell with Adaptations Feature Complete Release Exit Code Freeze Release Plan Sprints 1 1 2 2 1 2 3 3 … 3 N-1 N-1 N N N-1 … N Hardening Stability Extended Sprint Test Software System Test PRP Define Plan Develop Launch 14 7
  10. 10. 4/16/2013 Pitfalls encountered at • Development didn’t historically automate unit tests • Build teams were staffed with non-Build practitioners • Minimal guidance beyond “Go forth and automate” • Architecture(s) not optimized for Automatability • Automation was interpreted by many as “Automate the UI” • Insufficient SW engineering background across Test teams 15 Which Resulted In… • Automation not keeping up within the sprint • Feature Devotion • Multiple automation tools & licenses • Test scripts not designed for re-use 16 8
  11. 11. 4/16/2013 The Automation Landscape 17 Opportunities for Automation • It’s not just about Test cases • Can provide efficiencies to: –Test Preparation, Setup and Configuration –Test Content Development and Execution › Continuous Integration & Automated BVT › Acceptance Testing › Test Matrix coverage –Non-Functional Testing › Longevity, Scale and Performance Characterization 18 9
  12. 12. 4/16/2013 Test Preparation • Setup & Configuration – OS Provisioning and Configuration • Deployment • Test case staging • Environment Cleanup/Baseline • Virtualization-based test environments 19 Test Content Development Whole team ownership of QA & Automation • UI automation on Customer Usage workflows • Automated CI (UT, Build, BVT) • Web-services (or CLI) Test Automation for Functional Testing • In-Sprint, Automated Acceptance Tests • Unit Test Automation 1 CAUTION: Application Architecture can be an enabler or inhibitor 1 - Agile Testing: A Practical Guide for Testers and Agile teams 20 10
  13. 13. 4/16/2013 Application Architecture Matters • Does it facilitate testing at the API/Services level? • Does the Business/Error Logic reside below the services level? • Does the UI architecture support test automation other than record-playback? For Large organizations: – Standardize UI architectures across the product portfolio › HTML5, Flex, Silverlight… – Standardize Service architectures across the product portfolio › SOAP, REST, CLI, API 21 Prioritize and Identify what NOT to Automate Prioritize High ROI 1. BVT Candidate Acceptance Test – Test of core functionality that executes in a short duration 2. Core Functionality to be run in Nightly Regression 3. Sustaining Test Candidate Test – Functional test that verifies the application can withstand subsystem changes What NOT to Automate • Everything • GUI – Except for High-value customer usage flows once the UI is stable • Tests that would only find low severity bugs 22 11
  14. 14. 4/16/2013 Non-Functional Testing • Performance Characterization 2 • Longevity • Stress • Scale • Concurrency Often times analysis of these areas are simply too cost-prohibitive to be done manually 2 - 23 Foundations of an Automation Culture 24 12
  15. 15. 4/16/2013 Keys To A Culture Transition • Development vs. Test • A different “School” of Test3 • Evolve from Functional Responsibility …to Whole Product Ownership 3 – Scott Barber “Approaches to Software Testing: An Introduction “ 25 Establishing an Automation Culture • Identify • Near and long-term automation focus areas • Inventory the culture and skillset of organization • Establish – Tooling and Infrastructure Standardization – Develop your ‘community’ • Develop – Workforce transition plan • Organize – Embedded vs. Specialized Automation team – Re-align project staffing • Operationalize 26 13
  16. 16. 4/16/2013 Identify Focus Areas for Automation 4 4 - 27 Establish • Tooling and Infrastructure –Tooling Standards and Frameworks –Team to develop/manage home-grown tools –Lead tool evaluations to meet new architectures & technologies Multiple automation Tools/FW’s may be needed to automate at layers of testing: • • • Unit level Middle tier UI-driven • Community: Internal and External 28 14
  17. 17. 4/16/2013 Develop & Organize • Adjust Hiring Practices • Re-align Project staffing • Embedded Automation vs. Specialized Automation team • Training –New-hire and ongoing 29 Confidential Operationalize • Whole team commitment in words and actions • Include Test Automation in Acceptance Criteria –Unit –Acceptance • Continuous Integration and BVT • Establish Metrics and Governance 30 15
  18. 18. 4/16/2013 Care and Feeding of the Automation Culture 31 Maintaining the Automation Culture • Operationalize Automation – Include Test Automation in the Acceptance Criteria • Encourage Community Practice and Participation – External - Industry User groups & Conferences – Internal - Brown-bag sessions, Showcases, Mini-conference • For Large organizations: – Overcome Skillset deficiencies by adjusting staffing strategy – Automation Leadership team, Automation Architect(s) – Continually monitor the alignment of Arch, Dev and Test – Develop reward systems 32 16
  19. 19. 4/16/2013 Recognizing a Successful Automation Culture • Automation is a shared responsibility • Automatability is a key architectural consideration • Continuous Integration & BVT • Teams are staffed to include an automation skillset • Automation is operationalized Resources • Agile Manifesto • Articles: – - SmartBear – Lisa Crispin – ObjectId=17793&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban • Books: – Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell – How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo – Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory • Presentations: – Approaches to Software Testing: An Introduction – Scott Barber – Agile Testing: Challenges Beyond the Easy Contexts – Bob Galen 34 17
  20. 20. 4/16/2013 Questions? 35 Backup 36 18