WSO2 Test Automation Framework : Approach and Adoption


Published on

Published in: Technology, Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

WSO2 Test Automation Framework : Approach and Adoption

  1. 1. WSO2 Test AutomationFrameworkApproach and AdoptionByKrishantha SamaraweeraTechnical Lead, Test AutomationDharshana WarusavitharanaSoftware Engineer, Test Automation
  2. 2. About WSO2● Providing the only complete open source componentized cloudplatform● Dedicated to removing all the stumbling blocks to enterprise agility● Enabling you to focus on business logic and business value● Recognized by leading analyst firms as visionaries and leaders● Gartner cites WSO2 as visionaries in all 3 categories of applicationinfrastructure● Forrester places WSO2 in top 2 for API Management● Global corporation with offices in USA, UK & Sri Lanka● 200+ employees and growing● Business model of selling comprehensive support &maintenance for our products
  3. 3. 150+ globally positioned support customers
  4. 4. Agenda● Requirements and early efforts of Automation in WSO2.● Approach and Adoption.● Framework overview.● Adopting scenarios and flexibility.● Architecture overview.● Automation Framework Structure.● Anticipating challenges of automation.● Demos○ Admin service client API base integration test.○ Selenium integration test.○ Test with tooling support (Jmeter).
  5. 5. Requirements Of Automation At WSO2● Should be easy to adopt and manage.● The automated test suites are subject to run on○ Integration○ Platform○ Private or public cloud● Environments on User / Tenant mode.● Single test case should be able torun seamlessly on all above executions.● Should be able to run with nightly buildsystems such as Bamboo or Jenkins
  6. 6. Requirements Of Automation At WSO2● Should be able to run existing Jmeter test scripts.● Should support UI Automation.● All tests should be available as deliverable artifact - PlatformAutomated Test Suite.● Open source and release under apache License.● Coverage and Reporting.
  7. 7. History Of Automation In WSO2● Developed first UI test framework in 2009 based on Selenium.● Separate framework for testing Data Services.● Developed component test framework in 2010● Developed Integration test framework in 2011 - AKA CTF● Various ruby, perl, ant and shell scripts to automate producttesting with different web application containers and databases.● System Test Framework in 2012● Framework unification -Test Automation Framework = Integration test framework +System test framework in 2012 August
  8. 8. Approach to Test Automation At WSO2Test Automationapproach/designTestAutomationRequirementsUnderstandingtestingEnvironmentsUnderstandingSystem UnderTest
  9. 9. What we will Automate at WSO2● Platform scenarios.● Integration scenarios.● UI scenarios.● Real world use cases.● Performance and Security scenarios.● Patches provided.
  10. 10. Overview of Automation FrameworkTest Automation Framework is to make automation● Eazy● Organized● Relevant● Optimized
  11. 11. Technology OutlineTest Framework Build SystemsCode CoverageTooling SupportExecution Environment
  12. 12. In Automation WSO2 is focusing on the following executionscenarios.● Integration tests - Which verifies the product build at build level.○ User mode which allows run product as a standalone user.○ Tenant mode Which allows product run as a cloud baseddeployment.● Platform tests - Which verifies the scenarios on platforms whichcontains more than one WSO2 product or integration with differentplatforms.● Cloud based deployment test (Stratos based tests) - Whichexecutes the tests based targeting cloud deployments.Adopting Scenarios
  13. 13. Flexibilities Provided By Automation Framework● Manage several products inside single test.● Manage users in several product domains.● Deployment of artifacts for different products under differentusers.● Automatic authentication.● Configured admin service clients as test oriented API.● Retrieving test environment configurations easily.● Running same test in both local and Stratos environments.● Reporting TestNg and surefire reports for all your tests.● Tooling support of Selenium and Jmeter.
  14. 14. Architecture Overview● Depends on TestNg Listeners.● Platform wide single framework● Context providing interfaces.● Admin Service based Automation API.● Selenium page object repository and element mapper.● Automation framework Utils.● Coverage and Reporting.
  15. 15. Automation Framework Overall Architectureuser.csv /tenant.csvautomation.propertiesinstrument.txtResultAutomation Framework CoreTestDashboardMavenTestManagersTest SuiteSurefireTestNg AutomationFramework APIAutomationFramework UtillsUserPopulatorCoverage(emmaServermanagerCustomReporterContextProvider
  16. 16. Framework CoreCorePlatformExecutionManagerPlatformSuiteManagerPlatformTestManagerPlatformAnnotationTransferManagerPlatformInvokedMethodManagerPlatformReportManagerEnvironmentContextFrameworkAnnotationsArtifact ManagementUser ManagementServer ManagementCoverageCustom Error Reporting
  17. 17. Execution Template - TestNgExecution Started@BeforeSuite@AfterSuiteExecution Closed@BeforeTest@AfterTest@BeforeClass@BeforeGroups@BeforeMethod@AfterMethod@AfterGroups@AfterClassReference:-
  18. 18. TestNg Listeners● Listeners are interfaces allows to modify TestNGsbehavior.● Listeners are binned to a testNg execution.● Listeners implement the interface org.testng.ITestListener○ IExecutionListener■ Triggers at start and end of any execution.■ ISuiteListener.○ ITestListner■ Triggers at suite start and end.■ Triggers at Test start, Finish, Failure, Skip and partial failure.● Reporters implement the interface org.testng.IReporter○ Notified when all the suites have been run by TestNG.○ The IReporter instance receives a summary of entire test run
  19. 19. Usage of Listeners● PlatformExecutionManager (IExecutionListener)○ onExecutionStart()■ Emma instrumentations■ Server start○ onExecutionFinish()■ Server Shutdown■ Emma report generation● PlatformSuiteManager (ISuiteListener)○ Set environment properties ex- Key Store Paths○ Populate Users.
  20. 20. Usage of Listeners contd..● PlatformReportManager (IReporter)○ Generate TestNg Report○ Generate surefire report○ Export data for Dashboard● PlatformInvokedMethodManager (InvokedMethodListener)○ Artifact deployment in platform scenarios.● PlatformAnnotationTransferManager (IAnnotationTransformer)○ Annotation-Platform-user type based test selection.
  21. 21. Context providing interfaces● Execution Based context -● Platform Based Context -● User Based Context - user.csv / tenant.csv● Coverage Based Context - instrumentation.txt
  22. 22. Automation Propertiesstratos.test=false#execution.environment = integration|platform|stratosexecution.environment=integration#execution.mode = user|tenant|allexecution.mode=userport.enable=truecarbon.web.context.enable=falseservice.deployment.delay=30000cluster.enable=trueproduct.list=ESBbuilder.enable=truecoverage.enable=true● To configure Test frameworkfor the environment andconfigure package details● Contains all information about The environment.
  23. 23. Context provider Architectureuser.csv /tenant.csvautomation.propertiesinstrument.txtEnvironmentContextContextProviderGroupContextProviderNodeReaderUrlGeneratorNodeContextProviderResultGlobal ContextFrameworkContextProviderUserInfoReaderGlobalContextTestContextInitiator
  24. 24. Server Management● Needs to configure the server and start the server with in thetest case.● Most of the cases it might be○ Single serve with single instance○ Single server with multiple instance○ Multiple product servers in a clustered environment.● In this module framework is managing○ Extract and configure the server.○ Add offsets and deploy custom modules needs for the test.
  25. 25. Test Execution Modes● Handled by a custom annotation introduced at test class and method level●●●● Annotations are based on the execution environment and the user type of the test● Tests are skipped at the "PlatformAnnotationTransferManager"● integration_all, integration_user, integration_tenant, platform_all, platform_user,platform_tenant, stratos, all@SetEnvironment(executionEnvironments = {ExecutionEnvironment.integration_all})public void createGroup() throws Exception{ }#execution.environment = integration|platform|stratosexecution.environment=integration#execution.mode = user|tenant|allexecution.mode=user
  26. 26. Automation Framework API● Test Friendly api wrapping service stubs.○ Depends on service stubs.○ Provides unified approach for perform verify and assertof each admin service.● Encapsulates the complexity of changing all available testsin a case of a stub change.● Maintain Page object classes for Selenium Automation.● Using UI maps to store locators required for test scripts.● Updates with the relevant release version.
  27. 27. Automation Framework Utils● Includes all utility classes that can play supportive role inside atest.○ Ex :-■ Axis2 Client.■ Wire message monitor.■ Custom server startup scenarios (Axis2, Tomcat,Qpid).■ Concurrency test scenarios.● Provides common methodology for all development teams tomaintain a set of supportive classes without changing the Core.● Reusability of utility methods in integration and platform tests.
  28. 28. Reporting● Includes TestNg reports ,SureFire report, And EmmaCoverage report● Directories containing separateresults for each suite ( Directoryname is as same as the suitename).● Surefire reports.● index.html (Dashboard forexecuted test).|-- reports| |-- BPSStructuredSuite| | |-- classes.html| | |-- FlowClient.html| | |--| | |-- FlowClient.xml| | |-- ForEachClient.html| | |--| | |-- ForEachClient.xml| |-- index.html| |-- junitreports| | |-- TEST-org.wso2.automation.common.test.bps.mgts.BpelStructAct_FlowClient.xml| | |-- TEST-org.wso2.automation.common.test.bps.mgtst.BpelStructAct_forEachClient.xml
  29. 29. Platform Automated Test Suite● Distribution contain product integration and platform test jars.● ANT based test jar executor - using TestNg ant task● Ability to execute tests on different environments.● Provide options to run test suites, packages and individualclasses.● TestNg reports.● Generate mail with test results.● Preview build available at
  30. 30. Future Challenges● Getting others to believe on automated tests.● Improve coverage on Integration test scenarios.● Adopting framework demanding requirement changes.● Adopting framework with complex deployment scenarios.
  31. 31. Project Structure of Automated Tests
  32. 32. Coverage StatisticsProduct Name LOC Coverage Class CoverageESB 38% 66%G-Reg 46% 65%DSS 49% 73%CEP 57% 74%MB 47% 90%AS 20% 36%API-Mgt 21% 42%● Included only integration test coverage.● Also guarded by loads of dependency and component levelunit tests.● UI test coverage not included.
  33. 33. Write First Integration Test : Demo 1Scenario To be Automated:Deploy a webapp on WSO2 application server and invoke itonce deployed successfully. Then remove the webapp at theend of test execution.
  34. 34. Write First Selenium Test : Demo 2Scenario To be Automated:Deploy a webapp through admin console as a war file and invokeonce deployed successfully.
  35. 35. Write Fist Jmeter Test : Demo 3Scenario To be Automated:Deploy Simple Axis2Service on WSO2 Application Server andinvoke it. Then remove the service at the end of test execution.
  36. 36. Questions?
  37. 37. Engage with WSO2● Helping you get the most out of your deployments● From project evaluation and inception to developmentand going into production, WSO2 is your partner inensuring 100% project success