Flintstones or Jetsons? Jump Start Your Virtual Test Lab


Published on

The power of virtualization has made it easy and inexpensive to create multiple environments for testing. How you implement your virtualization strategy can boost not only the savings on physical gear and availability of test environments but also your testing productivity. Sharing his experience working through the evolution of Verisign’s virtual test lab, David Silk examines how a well-implemented virtual lab can push your testing productivity to new levels. Learn about the key practices to get a virtual test lab working like an advanced Jetson’s-style machine while avoiding the Flintstone's dinosaur approach. See how Verisign’s approach focuses on the whole environment—not just one virtual machine at a time. Learn where to start and how to build a virtual test lab that leverages the technology, ensures repeatability, and saves test engineers time and effort. Don’t be a Flintstone!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Flintstones or Jetsons? Jump Start Your Virtual Test Lab

  1. 1. T9 Test Automation 5/2/2013 11:15:00 AM Flintstones or Jetsons? Jump Start Your Virtual Test Lab Presented by: David Silk Verisign, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
  2. 2. David Silk A passionate technologist with more than fifteen years of software development experience, David Silk is a systems assurance manager focused on building virtual lab infrastructure automation and test automation frameworks. David has been an engineer and manager in both configuration management and systems assurance. For the past ten years at Verisign, a technology leader in Internet infrastructure, he has played a lead role in implementing an automated system using virtualization to deliver software with greatly improved efficiency and increased savings on hardware investments. David’s zeal for automated solutions to drive software development practices has played an integral role in his success.
  3. 3. 4/16/2013 Flintstones or Jetsons? Jumpstart your Virtual Test Lab David Silk, Verisign Inc. StarEast 2013 * Flintstones and Jetsons are © Warner Bros. Animation & Cartoon Network Studios, originally created by Hanna-Barbera Productions, Inc. Introductions • Background • Developer, QA • Config Mgt • Software/Systems Ops • Engineering Manager • Systems Assurance • Test Automation • Virtual Lab Management • Shared Infrastructure Dev • Tech guy 2 Verisign Public 1
  4. 4. 4/16/2013 Introductions 3 Verisign Public Expectations 4 2
  5. 5. 4/16/2013 Break down barriers, automate manual steps, accelerate your process. 5 The Quality Virtualization Evolution Virtualize Test Execution Application Deployment 6 Automate Environments Inventory Accountability Verisign Public 3
  6. 6. 4/16/2013 Why Flintstones and Jetsons? Flintstones… Clunky, manual, slow. Jetsons Sleek, modern, cool. 7 Verisign Public The Scenario • Consider this: • A new release is coming, we need a test environment, but • We won’t finish the previous release until 2 weeks after the first test cycle of the new release will start • Our other environment is for production support, we can’t touch it • It takes at least a week to create and shake out a new environment 1 • Historically, we built time in our schedules (days! weeks!!!) for the new environment to be created and shaken out. • Just like software builds are consistent, repeatable events • Test environments should be created by consistent, repeatable events. 8 2 3 Verisign Public 4
  7. 7. 4/16/2013 The Quality Virtualization Evolution Virtualize Test Execution Application Deployment 9 Automate Environments Inventory Accountability Verisign Public Virtualization Technologies • The advent of virtualization: • combined with high-powered servers with multi-core processors, has made test environments much more agile • Functional test environments don’t need to scale like production – perfect fit for virtualization. • We can run many virtual servers on a single physical server, providing scalability of environments to support more parallel development and test efforts. • Time and cost for new environments becomes bearable for any project. • BUT do I need a team of highly trained experts to manage the virtualization stuff? • Not really. Its an operating system running cool software. We are already pretty good at handling that stuff! 10 Verisign Public 5
  8. 8. 4/16/2013 Virtualization Technologies (continued) • So, where do you start? • Pick a Virtualization technology vendor: • VMWare – this is what we use. • Openstack – Open Source Software • Microsoft, KVM, Xen and others • Pick one that fits best with your environment. • We use a lot of servers running RedHat Enterprise Linux, Solaris, FreeBSD operating systems. VMWare fits well. • Next Steps: Hardware • Pick a server with • Multiple multi-core processors – today 4 x hex-core procs • Lots of RAM – 128 Gb+ • Storage – can get tricky, a shared storage system is best • Using local server storage can work, make sure it uses RAID 11 Verisign Public Takeaway #1 Virtualization and cloud computing are here and have tremendous value. Embrace the technology. Don’t plan a big roll out with a long lead time – make it agile and get started in small increments today. 12 Verisign Public 6
  9. 9. 4/16/2013 The Quality Virtualization Evolution Virtualize Test Execution Application Deployment 13 Automate Environments Inventory Accountability Verisign Public Automation is key to making virtualization manageable. 14 7
  10. 10. 4/16/2013 Getting Started • Backstory: • • • • • • • We started small. Manually installed operating systems. Requested IP addresses from network team. Manually installed a bunch of dependencies. Firewall rules. We started to grow major support effort. We needed to automate to keep up. • The Virtual QA Lab (VQL) was born. • Dedicated focus on managing the virtual environments 15 Verisign Public Automation Checklist for a Virtual Machine • Automating the creation of a VM: Make use of the hypervisor command line interface Script out the creation process Create a VM container in the hypervisor Includes virtual disk, memory, # of CPUs to assign Use a boot ISO to start the VM and initiate the OS installation Assign an IP address Use a reserved pool of IP addresses Install software dependencies and infrastructure Monitoring, backup, antivirus, & other runtime dependencies E.g. Nagios, Java, Weblogic Setup user accounts Automate the creation of the required user accounts for the system 16 Verisign Public 8
  11. 11. 4/16/2013 Advanced Automation Tools • CMDB – authoritative inventory of all systems • PXE booting – pre-execution environment • Install an OS over the network • Cobbler - http://cobbler.github.com/ • Account Management • Centralized solution for managing credentials, e.g. Kerberos, LDAP, AD • Software package repository • Enables automation to quickly deploy. E.g. Yum on Linux • Your IT or operations teams might already have solutions for these – collaborate! 17 Verisign Public Creating Whole Ecosystems • Software systems have many moving parts • Various application servers, Databases, third party apps • The benefits of Virtualization • VM templates/cloning – start from the same state every time. • Test Beds • A combination of multiple unique components in an application environment all necessary to run the app end-to-end. • Application Env Profiles • Combination of VM templates that make up a whole test bed or env. • Databases - big data wrinkles 18 Verisign Public 9
  12. 12. 4/16/2013 Making it work @ Verisign • Virtualization standard • VMWare an early standard • Unique environment profiles • QA environment churn • Established standards: • OS images existed, TADDs • No end-to-end automation solution to start with • VMs were still experimental when we started • Needed to bring together a system to bridge our evolving systems. • A Verisign Formation • An automation platform developed internally • Pluggable architecture, made VM automation a snap! 19 Verisign Public Overview of Automation @ Verisign Verisign Formation: VM Automation System VMWare ESXi: Hypervisor ServiceNow: CMDB Jenkins/ TeamCity/ Bamboo: CI Tool Cobbler: OS Provisioning 20 Vipor: Orchestration Tool (Fabric) Puppet: Infrastructure automation Jira: Ticketing System Tranquility: Account Mgt Yum: RPM Repository Verisign Public 10
  13. 13. 4/16/2013 Formation Snippet • Developed @ Verisign • Dedicated automation team • Drives the automation via a workflow that is easily customized to any environment. • Adapter system allows us to integrate with other systems. • You can use OSS or Freeware tools to get started. 21 Verisign Public Takeaway #2 Automated, repeatable processes enable QA analysts and engineers to focus more on the quality of the application and how it meets the business needs. Focus efforts to automate the environment setup steps that have to be done every time before testing the application can begin. 22 Verisign Public 11
  14. 14. 4/16/2013 The Quality Virtualization Evolution Virtualize Test Execution Application Deployment 23 Automate Environments Inventory Accountability Verisign Public Knowledge is power. Metrics are a catalyst. 24 12
  15. 15. 4/16/2013 Accountability is important • VMs are easily created, easily decommissioned. • Encourage recycling! • Be good stewards of your assets. • Keep an inventory online • Assign owners for every VM created • Quotas for each team/owner • Limit the amount of resources each group can consume • Everyone has a budget of virtual disk/memory/CPUs • Poll machines for idle time, last login • Send reports to managers • Automation should update the inventory • Have a stakeholders meeting! • State of the Virtualization Address 25 Verisign Public Generate Metrics about your inventory 26 Verisign Public 13
  16. 16. 4/16/2013 Tracking Change Activity • Jira tracks all requests • Automation reads from Jira, updates tickets with progress. • Some Virtual QA Lab requests are for updates, modifications. • Automating all one-off tasks may not be appropriate. • Each request is evaluated, automation is constantly evolving. 27 Verisign Public Takeaway #3 Be a good steward of the investment you made in the virtualized environment with an accurate inventory. Use your existing toolsets to track changes and be accountable to getting the most out of your investment. 28 Verisign Public 14
  17. 17. 4/16/2013 The Quality Virtualization Evolution Virtualize Test Execution Application Deployment 29 Automate Environments Inventory Accountability Verisign Public Software can be complex, but deployments shouldn’t be. 30 15
  18. 18. 4/16/2013 Orchestrating Complex Deployments • Challenges of software systems deployments • Multiple dependent steps, specific order of execution • Configuration information – system, server & application • Lots of tools in the market, commercial & open source • Verisign deployments • Large databases, application servers, gateways • Network silos & multiple sites • Tight timelines for deployments • DevOps & Continuous Delivery 31 Verisign Public Orchestration Scenarios • Two scenarios: • Upgrade an existing application • Many dependent steps, order is important. • Most new software releases are upgrades. • Build a server from the ground up • Can be bare metal (physical) or virtual • New OS, all necessary tools plus your application • Both approaches have valid use cases • Live deployments must be upgraded. • Backup and failover strategies keep you running • If a server or device fails, a new one must replace it • Business Continuity Planning • Solving this problem can contribute to your BCP strategy! 32 Verisign Public 16
  19. 19. 4/16/2013 Takeaway #4 Get both Dev and IT Ops working together to make software deployments a consistent, repeatable process. Embracing DevOps concepts of continuous delivery will help application environment management even if you don’t plan to push rapid updates to production. 33 Verisign Public The Quality Virtualization Evolution Virtualize Test Execution Application Deployment 34 Automate Environments Inventory Accountability Verisign Public 17
  20. 20. 4/16/2013 Are your tests always running? If not, why not? 35 Test Automation & the Build Server • Continuous Integration • Build often, test often. • Dev teams and QA teams work towards the same goal – fewer bugs. • You don’t have to be “agile” to do this! Test Develop Deploy Build • Test Automation • Trigger regression tests with every build • For every release candidate, automated execution validates the app before anyone starts testing. • Tune your tests to be efficient! 36 Verisign Public 18
  21. 21. 4/16/2013 Takeaway #5 Integrate your test automation in the build system for maximum effect. Integrating your test automation into the build process provides regular validation and it exposes development to the test results. 37 Verisign Public Final thoughts • 640K ought to be enough for anybody. • Microsoft Chairman Bill Gates,1981 • If you tried to read every document on the web, then for each day's effort you would be a year further behind in your goal. • Anonymous • There might be new technology, but technological progress itself was nothing new - and over the years it had not destroyed jobs, but created them. • Margaret Thatcher 38 Verisign Public 19
  22. 22. 4/16/2013 Q&A 39 Thank You © 2011 VeriSign, Inc. All rights reserved. VERISIGN and other trademarks, service marks, and designs are registered or unregistered trademarks of VeriSign, Inc. and its subsidiaries in the United States and in foreign countries. All other trademarks are property of their respective owners. 20