Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hybrid Automation for the Hybrid Cloud

1,388 views

Published on

- Complete Automation Lifecycle Management (ALM) solution;
- Providing Automated Testing as a service in the Cloud;
- "Pay As You Go" flexible billing (no expensive tool licensing costs);
- Distributed continuous build integration with Application Under Test;
- Shorter market deployment times through quicker Automated Testing;
- Shorter environment deployment times through Automated Health Check;
- Better quality testing freeing specialist test resource to concentrate on non-repetitive test tasks;
- Easy interface for non-technical testers to create their own user stories;
- Capability Maturity Model Integration (CMMI) through Business Process Modelling (BPM) comprising of Business Level Keywords (BLK);
- Introduction of Full Agile automation testing and First day automation.

  • Be the first to comment

  • Be the first to like this

Hybrid Automation for the Hybrid Cloud

  1. 1. Hybrid Keyword Data DrivenHybrid Keyword Data Driven Automation FrameworksAutomation Frameworks Test Automation in the CloudTest Automation in the Cloud Real world case studyReal world case study Jonathon Lee WrightJonathon Lee Wright jlwright@iee.orgjlwright@iee.org www.automation.org.ukwww.automation.org.uk www.linkedin.comwww.linkedin.com/in/automation/in/automation 11th November 201011th November 2010
  2. 2. • Complete “Automation Lifecycle Management” (ALM) solution; • Providing “Automated Testing as a service” in the Cloud; • “Pay As You Go” flexible billing (no expensive tool licensing costs); • Distributed continuous build integration with “Application Under Test”; • Shorter market deployment times through quicker Automated Testing; • Shorter environment deployment times through Automated Health Check; • Better quality testing freeing specialist test resource to concentrate on non-repetitive test tasks; • Easy interface for non-technical testers to create their own “user stories”; • Capability Maturity Model Integration (CMMI) through Business Process Modelling (BPM) comprising of Business Level Keywords (BLK); • Introduction of “Full” Agile automation testing and “First” day automation. What value did this case study deliver?
  3. 3. What approach? • Manual Approach; • Traditional Automated Approach: • Hybrid Keyword Data Driven Framework Approach. Vendor Tool (Record & Playback, Adapt Scripts) Limited requirements Limited output
  4. 4. Framework Approach Hybrid utilising the best technologies and resourcesHybrid Keyword Data Driven (Modularity) Keyword business level keywords represent user stories Data dynamic use of sanitised business data Driven (Modularity) modularity of test procedures to provide reliable processing of the framework components
  5. 5. Hybrid Approach .NET 4.0 Development (Visual Studio 2010) Cloud (Windows Azure/SQL/AppFabric) Automation (Ranorex/Coded UI APIs)) ●●●●● Hybrid Vendor Tool (Ranorex IDE)●○○○○Traditional ToolsModularityDataKeywordBPTBPMApproach ○- Limited Support ●- Full Support Single Technology Stack (SPOF) Tool Vendor Dependency C# Project incompatible with Visual Studio Technology Blocks (Custom Flex Objects/Classes) Limited non-technical tester interface Vendor Support (maintenance subscription) Object Repository (object-based Record / Replay) ConstraintsAdvantages Traditional (Vendor Tool) In-house framework development & maintenance effort. Additional skills / knowledge resources Azure Platform Subscription Object Repository (limited Record / Replay support) Multiple Technology Stack (VS2010 / C# / VB.NET / ASP.NET / TFS Windows Azure / (WCF) Web Services / DLL/ XML/ Azure SQL/AppFabric Coded UI / Ranorex / AIR / Flex / XPATH ) Integration with development (Visual Studio 2010) Integration with deployment (TeamCity v5.1) Integration with testing (Quality Center v11) ConstraintsAdvantages Hybrid (.NET Development)
  6. 6. Return on Investment 1 2 3 4 5 6 7 Number of Excecutions InvestmentinEffort Market Configuration Testing + Health Check Market Configuration Testing Traditional Manual MCT = 3 MCT+ HC = 3.4 Traditional = 3.7 Initial Investment Leverage Hybrid (Framework) Vendor Tool Rn = Aa / Am = (Va / n + n*Da) / (Vm / n + n*Dm)
  7. 7. Automation Goals Reusable modularity of test procedures & functions Manageable effective test design, execution and traceability Maintainable to reduce the amount of test maintenance effort Robust providing object/event/error handing & recovery Flexible independent of test environment and be scalable Measurable customisable reporting & test execution progress
  8. 8. Business Process Testing (Process) Login.Process (Data Stage) Test Management Tool Team Foundation Server User Stories Business Level Keywords Business Process Modelling (BPM) Business Process Testing (BPT) Business Level Data Modularity Keyword Data Driven Hybrid Login.Process (Keyword Stage) Action Set Object Username Data #Username Parameters Data #Username Value Shop4Test Object Repository Object Username Class EditBox Framework Action(s) Objects(s) Data Object Handler Data Handler Event Handler
  9. 9. Business Process Modelling (Example) LoginFrm LoginChangePassword Frm A0 A1 A2 MyShopHome Frm B0 EditProfileFrmB3 ShopHome ShopLogin CreateNewReport CreateNewReport Frm C0 ReportType Frm C1 ProductGroups Frm C2 RunReport Frm C7 TimePeriods Frm C3 ProfileFrmC4 C8 C9 Geographies Frm C5 C10 ReportSche duleFrm C12 TermsConditions Frm D0 B1 B2 ShopperPan elsFrm C6 C11 CurrentReports Frm D0 CurrentReports CompletedReports Frm R0 CompletedReports Flex Reports Excel F0 E0 MyGroupFrm RecurringReports Frm S0 RecurringReports MyGroups G0 ModifyGroupFrmG1 OrderDetailsFrmR1 Index - Complete - In Complete - Supported - Out of Scope
  10. 10. Business Level Keywords (Testing) Translates
  11. 11. Automation Lifecycle Management (Process)
  12. 12. Technology Stack
  13. 13. Framework Architecture (Cloud Testing)
  14. 14. Framework Architecture (Deployment) Shared Persistent Message Queue Server Customer Facing Web Services Per Customer Database Resources Customer Deployment Per Customer On demand VM Test Lab Environment Rule Store Schedule History Real time progress Test runner Dashboard Control and Reporting Dashboard Scheduler Scheduler User Story Editor/Workbench Results Technology Adapter - Ranorex / Coded UI / Watir / Selenium etc... Generic Data Services Logging Results Recorder Cloud Resources Billing Credentials Asset Loader Build Integration Quality Centre or other BPM Integration Asset Store Results Recorder Results Recorder Scheduler System Management - Cloud/Application Tools Configuration Monitoring Test Criteria - User Data, Load, Geo., Technology etc... User Data Store Test Adapter - Accessability / Load / Regression / Penetration / Browser/ Platform / Smoke SEO etc...
  15. 15. Framework Architecture (High Level)
  16. 16. Automation Lifecycle Management (ALM)
  17. 17. Requirements - Business Process Data AUT Database Quality Center Test Automation Database Mock Live Data (Sanitised) Team Foundation Server
  18. 18. Test Planning - MasterDriver (Excel) Object Repository Test Repository
  19. 19. Test Planning - XML Schema
  20. 20. Test Execution - Testing in the Cloud WebServices(WCF) <<RealtimeTestResults>> <<RealtimeErrorReporting>> <<CheckTestSchedule>>
  21. 21. Test Execution - Automation Scheduler Selectable list of user case stories (Test Sets) Available machines Global Stop Buttons (Stops all tests on all running machines) Global Pause button Global Play button
  22. 22. Web Services (WCF) XML
  23. 23. Communication - Message Definition (WCF)
  24. 24. Windows Service (Lifecycle)
  25. 25. Event / Object Handling Supported events: button.click text.parmsetcheck text.validate link.click reporttype.selecttreeitem reporttype.opentreenode list.parmsetcheck checkbox.select tree.validate tree.parmsetcheck radio.parmsetcheck checkbox.parmsetcheck checkbox.validate
  26. 26. Business Level Keywords (Process) Business Level Keywords represent a number of logical test steps that make up a business process i.e. “Login.Process” User Stories are represented by a number of Business Level Keywords: Each User Story represents client Market Configuration Testing mappings: LoginFrm.Logo.Image LoginFrm.UserName.Label LoginFrm.Password.Label LoginFrm.LoginBoxHeader.Label LoginFrm.LoginButton.Label LoginFrm.ChangePWButton.Label LoginFrm.ForgotPWLink.Label LoginFrm.ServiceDeskTel.Label LoginFrm.ServiceDeskMail.Label LoginFrm.Copyright.Label A B C D E F G H I J F G H I J A B C D E
  27. 27. Market Configuration Testing (Process) Process the first test step configuration data value located in the “Test Repository” i.e. #dh.MyServices_Login_UserNameLabel Using the data value to confirm that the “Market Requirement” exists for the target market (<Client> + <WorkStream>) Once confirmed the configuration value can be extracted from the “Market Specification” by selecting the target market (<Language>) Test Step 1 – Using the “Configuration” value to verify that market configuration is correct: UserNameLabel = “User Name:” Result = FAIL
  28. 28. Object Recognition (Smart Identification) Get the test step object name from the Test Repository (MasterDriver) i.e. LoginFrm.Password.Input Using the object name extract the object class & associated value from the Object Repository (MasterParameters) i.e. Html.EditBox & “password” Using the object class extract the primary & secondary class tag identifiers from the Class Definition (MasterParameters) i.e. “InnerText” & “Type” Attempt #1 – Using the “WebElement” Class Describe the Object Physical Description: WebElement Object= [@InnerText = “password”] Attempt #2 – Using the “InputTag” Class Describe the Object Physical Description: InputTag Object= [@Type = “password”] Physical Object Physical Description
  29. 29. Object Recognition (Fuzzy Logic) FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']"; IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>(); Get all the screen objects on the flex page. Retrieve all the Buttons on the page. For ( int I = 1; >= flexbuttons.Count; i++) { // get the xpath for each element. RxPath me = flexButtons.ElementAt<Ranorex.Button>(i).GetPath(); // convert the xpath to a string string[] str = me.ToString().split(‘/’); // convert xpath to a split of strings string[] str1 = xPath.Split(‘/’); // compare both string paths until a common tree route is found (this will be your button) CompareTreePaths( str, str1 ) } When you have identified a tree path, you can then use it in the way described in the MasterDriver file (e.g. flexButton.Click).
  30. 30. Unified Test Reporting (UTR)
  31. 31. Unified Test Reporting – Real-time Log TestRunnerInput (XML) Output (WebService) .NET Object WCF Error/Event Handling Logging (Debug) Results (XML)
  32. 32. Process Improvement Agile Planning – Team Foundation Server linked to Task Board (SfTS v2); Coding Standards – Code reviews (internal/external to team), coding tools, unit test standards, agile hints, logging etc etc; Testing - Internal peer reviews, unit tests developed, exploratory testing for each story as a task, Integration test story in each sprint; Wiki – Knowledge capture including online framework documentation (screenshot) Technical Notes/Debt - on project wiki (intranet SharePoint).
  33. 33. Key Achievements • Centralised - management of test assets to provide open connectivity as a generic source of test scenario data; • Dynamic - generation of sanitised input test data validated against business rules along with an object repository based on a combination of descriptive programming containing XPATH regular expressions; • Legacy - support for static data types plus full/partial fall back support for manual test execution; • Unified reporting - using funnel virtualisation techniques to analyse the quality of builds, assisted by diagnostic data adapters; • Generic reusable - test components, modules and libraries provided reliable processing of the entire test automation effort.
  34. 34. SSUMMARYUMMARY •• HybridHybrid –– utilising the best technologies and resources to do the job;utilising the best technologies and resources to do the job; •• KeywordKeyword –– creating simple and robust test scenarios written in businesscreating simple and robust test scenarios written in business level keywords combined with natural language;level keywords combined with natural language; •• DataData –– effective use of dynamic business data to provide an input sourceffective use of dynamic business data to provide an input sourcee •• DrivenDriven –– processes the test component actions, objects and eventsprocesses the test component actions, objects and events seamlessly;seamlessly; •• AutomationAutomation –– that is collaborative, distributed and scalable;that is collaborative, distributed and scalable; •• FrameworksFrameworks –– independent of application or environment under test;independent of application or environment under test;

×