• Like

Thanks for flagging this SlideShare!

Oops! An error has occurred.

WSO2 Test Automation Framework : Approach and Adoption



Published in Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. WSO2 Test AutomationFrameworkApproach and AdoptionByKrishantha SamaraweeraTechnical Lead, Test AutomationDharshana WarusavitharanaSoftware Engineer, Test Automation
  • 2. About WSO2● Providing the only complete open source componentized cloudplatform● Dedicated to removing all the stumbling blocks to enterprise agility● Enabling you to focus on business logic and business value● Recognized by leading analyst firms as visionaries and leaders● Gartner cites WSO2 as visionaries in all 3 categories of applicationinfrastructure● Forrester places WSO2 in top 2 for API Management● Global corporation with offices in USA, UK & Sri Lanka● 200+ employees and growing● Business model of selling comprehensive support &maintenance for our products
  • 3. 150+ globally positioned support customers
  • 4. Agenda● Requirements and early efforts of Automation in WSO2.● Approach and Adoption.● Framework overview.● Adopting scenarios and flexibility.● Architecture overview.● Automation Framework Structure.● Anticipating challenges of automation.● Demos○ Admin service client API base integration test.○ Selenium integration test.○ Test with tooling support (Jmeter).
  • 5. Requirements Of Automation At WSO2● Should be easy to adopt and manage.● The automated test suites are subject to run on○ Integration○ Platform○ Private or public cloud● Environments on User / Tenant mode.● Single test case should be able torun seamlessly on all above executions.● Should be able to run with nightly buildsystems such as Bamboo or Jenkins
  • 6. Requirements Of Automation At WSO2● Should be able to run existing Jmeter test scripts.● Should support UI Automation.● All tests should be available as deliverable artifact - PlatformAutomated Test Suite.● Open source and release under apache License.● Coverage and Reporting.
  • 7. History Of Automation In WSO2● Developed first UI test framework in 2009 based on Selenium.● Separate framework for testing Data Services.● Developed component test framework in 2010● Developed Integration test framework in 2011 - AKA CTF● Various ruby, perl, ant and shell scripts to automate producttesting with different web application containers and databases.● System Test Framework in 2012● Framework unification -Test Automation Framework = Integration test framework +System test framework in 2012 August
  • 8. Approach to Test Automation At WSO2Test Automationapproach/designTestAutomationRequirementsUnderstandingtestingEnvironmentsUnderstandingSystem UnderTest
  • 9. What we will Automate at WSO2● Platform scenarios.● Integration scenarios.● UI scenarios.● Real world use cases.● Performance and Security scenarios.● Patches provided.
  • 10. Overview of Automation FrameworkTest Automation Framework is to make automation● Eazy● Organized● Relevant● Optimized
  • 11. Technology OutlineTest Framework Build SystemsCode CoverageTooling SupportExecution Environment
  • 12. In Automation WSO2 is focusing on the following executionscenarios.● Integration tests - Which verifies the product build at build level.○ User mode which allows run product as a standalone user.○ Tenant mode Which allows product run as a cloud baseddeployment.● Platform tests - Which verifies the scenarios on platforms whichcontains more than one WSO2 product or integration with differentplatforms.● Cloud based deployment test (Stratos based tests) - Whichexecutes the tests based targeting cloud deployments.Adopting Scenarios
  • 13. Flexibilities Provided By Automation Framework● Manage several products inside single test.● Manage users in several product domains.● Deployment of artifacts for different products under differentusers.● Automatic authentication.● Configured admin service clients as test oriented API.● Retrieving test environment configurations easily.● Running same test in both local and Stratos environments.● Reporting TestNg and surefire reports for all your tests.● Tooling support of Selenium and Jmeter.
  • 14. Architecture Overview● Depends on TestNg Listeners.● Platform wide single framework● Context providing interfaces.● Admin Service based Automation API.● Selenium page object repository and element mapper.● Automation framework Utils.● Coverage and Reporting.
  • 15. Automation Framework Overall Architectureuser.csv /tenant.csvautomation.propertiesinstrument.txtResultAutomation Framework CoreTestDashboardMavenTestManagersTest SuiteSurefireTestNg AutomationFramework APIAutomationFramework UtillsUserPopulatorCoverage(emmaServermanagerCustomReporterContextProvider
  • 16. Framework CoreCorePlatformExecutionManagerPlatformSuiteManagerPlatformTestManagerPlatformAnnotationTransferManagerPlatformInvokedMethodManagerPlatformReportManagerEnvironmentContextFrameworkAnnotationsArtifact ManagementUser ManagementServer ManagementCoverageCustom Error Reporting
  • 17. Execution Template - TestNgExecution Started@BeforeSuite@AfterSuiteExecution Closed@BeforeTest@AfterTest@BeforeClass@BeforeGroups@BeforeMethod@AfterMethod@AfterGroups@AfterClassReference:- http://www.ibm.com/developerworks/java/library/j-testng/
  • 18. TestNg Listeners● Listeners are interfaces allows to modify TestNGsbehavior.● Listeners are binned to a testNg execution.● Listeners implement the interface org.testng.ITestListener○ IExecutionListener■ Triggers at start and end of any execution.■ ISuiteListener.○ ITestListner■ Triggers at suite start and end.■ Triggers at Test start, Finish, Failure, Skip and partial failure.● Reporters implement the interface org.testng.IReporter○ Notified when all the suites have been run by TestNG.○ The IReporter instance receives a summary of entire test run
  • 19. Usage of Listeners● PlatformExecutionManager (IExecutionListener)○ onExecutionStart()■ Emma instrumentations■ Server start○ onExecutionFinish()■ Server Shutdown■ Emma report generation● PlatformSuiteManager (ISuiteListener)○ Set environment properties ex- Key Store Paths○ Populate Users.
  • 20. Usage of Listeners contd..● PlatformReportManager (IReporter)○ Generate TestNg Report○ Generate surefire report○ Export data for Dashboard● PlatformInvokedMethodManager (InvokedMethodListener)○ Artifact deployment in platform scenarios.● PlatformAnnotationTransferManager (IAnnotationTransformer)○ Annotation-Platform-user type based test selection.
  • 21. Context providing interfaces● Execution Based context - automation.properties● Platform Based Context - automation.properties● User Based Context - user.csv / tenant.csv● Coverage Based Context - instrumentation.txt
  • 22. Automation Propertiesstratos.test=false#execution.environment = integration|platform|stratosexecution.environment=integration#execution.mode = user|tenant|allexecution.mode=userport.enable=truecarbon.web.context.enable=falseservice.deployment.delay=30000cluster.enable=trueproduct.list=ESBbuilder.enable=truecoverage.enable=true● To configure Test frameworkfor the environment andconfigure package details● Contains all information about The environment.
  • 23. Context provider Architectureuser.csv /tenant.csvautomation.propertiesinstrument.txtEnvironmentContextContextProviderGroupContextProviderNodeReaderUrlGeneratorNodeContextProviderResultGlobal ContextFrameworkContextProviderUserInfoReaderGlobalContextTestContextInitiator
  • 24. Server Management● Needs to configure the server and start the server with in thetest case.● Most of the cases it might be○ Single serve with single instance○ Single server with multiple instance○ Multiple product servers in a clustered environment.● In this module framework is managing○ Extract and configure the server.○ Add offsets and deploy custom modules needs for the test.
  • 25. Test Execution Modes● Handled by a custom annotation introduced at test class and method level●●●● Annotations are based on the execution environment and the user type of the test● Tests are skipped at the "PlatformAnnotationTransferManager"● integration_all, integration_user, integration_tenant, platform_all, platform_user,platform_tenant, stratos, all@SetEnvironment(executionEnvironments = {ExecutionEnvironment.integration_all})public void createGroup() throws Exception{ }#execution.environment = integration|platform|stratosexecution.environment=integration#execution.mode = user|tenant|allexecution.mode=user
  • 26. Automation Framework API● Test Friendly api wrapping service stubs.○ Depends on service stubs.○ Provides unified approach for perform verify and assertof each admin service.● Encapsulates the complexity of changing all available testsin a case of a stub change.● Maintain Page object classes for Selenium Automation.● Using UI maps to store locators required for test scripts.● Updates with the relevant release version.
  • 27. Automation Framework Utils● Includes all utility classes that can play supportive role inside atest.○ Ex :-■ Axis2 Client.■ Wire message monitor.■ Custom server startup scenarios (Axis2, Tomcat,Qpid).■ Concurrency test scenarios.● Provides common methodology for all development teams tomaintain a set of supportive classes without changing the Core.● Reusability of utility methods in integration and platform tests.
  • 28. Reporting● Includes TestNg reports ,SureFire report, And EmmaCoverage report● Directories containing separateresults for each suite ( Directoryname is as same as the suitename).● Surefire reports.● index.html (Dashboard forexecuted test).|-- reports| |-- BPSStructuredSuite| | |-- classes.html| | |-- FlowClient.html| | |-- FlowClient.properties| | |-- FlowClient.xml| | |-- ForEachClient.html| | |-- ForEachClient.properties| | |-- ForEachClient.xml| |-- index.html| |-- junitreports| | |-- TEST-org.wso2.automation.common.test.bps.mgts.BpelStructAct_FlowClient.xml| | |-- TEST-org.wso2.automation.common.test.bps.mgtst.BpelStructAct_forEachClient.xml
  • 29. Platform Automated Test Suite● Distribution contain product integration and platform test jars.● ANT based test jar executor - using TestNg ant task● Ability to execute tests on different environments.● Provide options to run test suites, packages and individualclasses.● TestNg reports.● Generate mail with test results.● Preview build available at https://svn.wso2.org/repos/wso2/people/krishantha/wso2pats-1.1.2.zip
  • 30. Future Challenges● Getting others to believe on automated tests.● Improve coverage on Integration test scenarios.● Adopting framework demanding requirement changes.● Adopting framework with complex deployment scenarios.
  • 31. Project Structure of Automated Tests
  • 32. Coverage StatisticsProduct Name LOC Coverage Class CoverageESB 38% 66%G-Reg 46% 65%DSS 49% 73%CEP 57% 74%MB 47% 90%AS 20% 36%API-Mgt 21% 42%● Included only integration test coverage.● Also guarded by loads of dependency and component levelunit tests.● UI test coverage not included.
  • 33. Write First Integration Test : Demo 1Scenario To be Automated:Deploy a webapp on WSO2 application server and invoke itonce deployed successfully. Then remove the webapp at theend of test execution.
  • 34. Write First Selenium Test : Demo 2Scenario To be Automated:Deploy a webapp through admin console as a war file and invokeonce deployed successfully.
  • 35. Write Fist Jmeter Test : Demo 3Scenario To be Automated:Deploy Simple Axis2Service on WSO2 Application Server andinvoke it. Then remove the service at the end of test execution.
  • 36. Questions?
  • 37. Engage with WSO2● Helping you get the most out of your deployments● From project evaluation and inception to developmentand going into production, WSO2 is your partner inensuring 100% project success