• Like

WSO2 Test Automation Framework : Approach and Adoption

  • 1,465 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,465
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
48
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. WSO2 Test AutomationFrameworkApproach and AdoptionByKrishantha SamaraweeraTechnical Lead, Test AutomationDharshana WarusavitharanaSoftware Engineer, Test Automation
  • 2. About WSO2● Providing the only complete open source componentized cloudplatform● Dedicated to removing all the stumbling blocks to enterprise agility● Enabling you to focus on business logic and business value● Recognized by leading analyst firms as visionaries and leaders● Gartner cites WSO2 as visionaries in all 3 categories of applicationinfrastructure● Forrester places WSO2 in top 2 for API Management● Global corporation with offices in USA, UK & Sri Lanka● 200+ employees and growing● Business model of selling comprehensive support &maintenance for our products
  • 3. 150+ globally positioned support customers
  • 4. Agenda● Requirements and early efforts of Automation in WSO2.● Approach and Adoption.● Framework overview.● Adopting scenarios and flexibility.● Architecture overview.● Automation Framework Structure.● Anticipating challenges of automation.● Demos○ Admin service client API base integration test.○ Selenium integration test.○ Test with tooling support (Jmeter).
  • 5. Requirements Of Automation At WSO2● Should be easy to adopt and manage.● The automated test suites are subject to run on○ Integration○ Platform○ Private or public cloud● Environments on User / Tenant mode.● Single test case should be able torun seamlessly on all above executions.● Should be able to run with nightly buildsystems such as Bamboo or Jenkins
  • 6. Requirements Of Automation At WSO2● Should be able to run existing Jmeter test scripts.● Should support UI Automation.● All tests should be available as deliverable artifact - PlatformAutomated Test Suite.● Open source and release under apache License.● Coverage and Reporting.
  • 7. History Of Automation In WSO2● Developed first UI test framework in 2009 based on Selenium.● Separate framework for testing Data Services.● Developed component test framework in 2010● Developed Integration test framework in 2011 - AKA CTF● Various ruby, perl, ant and shell scripts to automate producttesting with different web application containers and databases.● System Test Framework in 2012● Framework unification -Test Automation Framework = Integration test framework +System test framework in 2012 August
  • 8. Approach to Test Automation At WSO2Test Automationapproach/designTestAutomationRequirementsUnderstandingtestingEnvironmentsUnderstandingSystem UnderTest
  • 9. What we will Automate at WSO2● Platform scenarios.● Integration scenarios.● UI scenarios.● Real world use cases.● Performance and Security scenarios.● Patches provided.
  • 10. Overview of Automation FrameworkTest Automation Framework is to make automation● Eazy● Organized● Relevant● Optimized
  • 11. Technology OutlineTest Framework Build SystemsCode CoverageTooling SupportExecution Environment
  • 12. In Automation WSO2 is focusing on the following executionscenarios.● Integration tests - Which verifies the product build at build level.○ User mode which allows run product as a standalone user.○ Tenant mode Which allows product run as a cloud baseddeployment.● Platform tests - Which verifies the scenarios on platforms whichcontains more than one WSO2 product or integration with differentplatforms.● Cloud based deployment test (Stratos based tests) - Whichexecutes the tests based targeting cloud deployments.Adopting Scenarios
  • 13. Flexibilities Provided By Automation Framework● Manage several products inside single test.● Manage users in several product domains.● Deployment of artifacts for different products under differentusers.● Automatic authentication.● Configured admin service clients as test oriented API.● Retrieving test environment configurations easily.● Running same test in both local and Stratos environments.● Reporting TestNg and surefire reports for all your tests.● Tooling support of Selenium and Jmeter.
  • 14. Architecture Overview● Depends on TestNg Listeners.● Platform wide single framework● Context providing interfaces.● Admin Service based Automation API.● Selenium page object repository and element mapper.● Automation framework Utils.● Coverage and Reporting.
  • 15. Automation Framework Overall Architectureuser.csv /tenant.csvautomation.propertiesinstrument.txtResultAutomation Framework CoreTestDashboardMavenTestManagersTest SuiteSurefireTestNg AutomationFramework APIAutomationFramework UtillsUserPopulatorCoverage(emmaServermanagerCustomReporterContextProvider
  • 16. Framework CoreCorePlatformExecutionManagerPlatformSuiteManagerPlatformTestManagerPlatformAnnotationTransferManagerPlatformInvokedMethodManagerPlatformReportManagerEnvironmentContextFrameworkAnnotationsArtifact ManagementUser ManagementServer ManagementCoverageCustom Error Reporting
  • 17. Execution Template - TestNgExecution Started@BeforeSuite@AfterSuiteExecution Closed@BeforeTest@AfterTest@BeforeClass@BeforeGroups@BeforeMethod@AfterMethod@AfterGroups@AfterClassReference:- http://www.ibm.com/developerworks/java/library/j-testng/
  • 18. TestNg Listeners● Listeners are interfaces allows to modify TestNGsbehavior.● Listeners are binned to a testNg execution.● Listeners implement the interface org.testng.ITestListener○ IExecutionListener■ Triggers at start and end of any execution.■ ISuiteListener.○ ITestListner■ Triggers at suite start and end.■ Triggers at Test start, Finish, Failure, Skip and partial failure.● Reporters implement the interface org.testng.IReporter○ Notified when all the suites have been run by TestNG.○ The IReporter instance receives a summary of entire test run
  • 19. Usage of Listeners● PlatformExecutionManager (IExecutionListener)○ onExecutionStart()■ Emma instrumentations■ Server start○ onExecutionFinish()■ Server Shutdown■ Emma report generation● PlatformSuiteManager (ISuiteListener)○ Set environment properties ex- Key Store Paths○ Populate Users.
  • 20. Usage of Listeners contd..● PlatformReportManager (IReporter)○ Generate TestNg Report○ Generate surefire report○ Export data for Dashboard● PlatformInvokedMethodManager (InvokedMethodListener)○ Artifact deployment in platform scenarios.● PlatformAnnotationTransferManager (IAnnotationTransformer)○ Annotation-Platform-user type based test selection.
  • 21. Context providing interfaces● Execution Based context - automation.properties● Platform Based Context - automation.properties● User Based Context - user.csv / tenant.csv● Coverage Based Context - instrumentation.txt
  • 22. Automation Propertiesstratos.test=false#execution.environment = integration|platform|stratosexecution.environment=integration#execution.mode = user|tenant|allexecution.mode=userport.enable=truecarbon.web.context.enable=falseservice.deployment.delay=30000cluster.enable=trueproduct.list=ESBbuilder.enable=truecoverage.enable=true● To configure Test frameworkfor the environment andconfigure package details● Contains all information about The environment.
  • 23. Context provider Architectureuser.csv /tenant.csvautomation.propertiesinstrument.txtEnvironmentContextContextProviderGroupContextProviderNodeReaderUrlGeneratorNodeContextProviderResultGlobal ContextFrameworkContextProviderUserInfoReaderGlobalContextTestContextInitiator
  • 24. Server Management● Needs to configure the server and start the server with in thetest case.● Most of the cases it might be○ Single serve with single instance○ Single server with multiple instance○ Multiple product servers in a clustered environment.● In this module framework is managing○ Extract and configure the server.○ Add offsets and deploy custom modules needs for the test.
  • 25. Test Execution Modes● Handled by a custom annotation introduced at test class and method level●●●● Annotations are based on the execution environment and the user type of the test● Tests are skipped at the "PlatformAnnotationTransferManager"● integration_all, integration_user, integration_tenant, platform_all, platform_user,platform_tenant, stratos, all@SetEnvironment(executionEnvironments = {ExecutionEnvironment.integration_all})public void createGroup() throws Exception{ }#execution.environment = integration|platform|stratosexecution.environment=integration#execution.mode = user|tenant|allexecution.mode=user
  • 26. Automation Framework API● Test Friendly api wrapping service stubs.○ Depends on service stubs.○ Provides unified approach for perform verify and assertof each admin service.● Encapsulates the complexity of changing all available testsin a case of a stub change.● Maintain Page object classes for Selenium Automation.● Using UI maps to store locators required for test scripts.● Updates with the relevant release version.
  • 27. Automation Framework Utils● Includes all utility classes that can play supportive role inside atest.○ Ex :-■ Axis2 Client.■ Wire message monitor.■ Custom server startup scenarios (Axis2, Tomcat,Qpid).■ Concurrency test scenarios.● Provides common methodology for all development teams tomaintain a set of supportive classes without changing the Core.● Reusability of utility methods in integration and platform tests.
  • 28. Reporting● Includes TestNg reports ,SureFire report, And EmmaCoverage report● Directories containing separateresults for each suite ( Directoryname is as same as the suitename).● Surefire reports.● index.html (Dashboard forexecuted test).|-- reports| |-- BPSStructuredSuite| | |-- classes.html| | |-- FlowClient.html| | |-- FlowClient.properties| | |-- FlowClient.xml| | |-- ForEachClient.html| | |-- ForEachClient.properties| | |-- ForEachClient.xml| |-- index.html| |-- junitreports| | |-- TEST-org.wso2.automation.common.test.bps.mgts.BpelStructAct_FlowClient.xml| | |-- TEST-org.wso2.automation.common.test.bps.mgtst.BpelStructAct_forEachClient.xml
  • 29. Platform Automated Test Suite● Distribution contain product integration and platform test jars.● ANT based test jar executor - using TestNg ant task● Ability to execute tests on different environments.● Provide options to run test suites, packages and individualclasses.● TestNg reports.● Generate mail with test results.● Preview build available at https://svn.wso2.org/repos/wso2/people/krishantha/wso2pats-1.1.2.zip
  • 30. Future Challenges● Getting others to believe on automated tests.● Improve coverage on Integration test scenarios.● Adopting framework demanding requirement changes.● Adopting framework with complex deployment scenarios.
  • 31. Project Structure of Automated Tests
  • 32. Coverage StatisticsProduct Name LOC Coverage Class CoverageESB 38% 66%G-Reg 46% 65%DSS 49% 73%CEP 57% 74%MB 47% 90%AS 20% 36%API-Mgt 21% 42%● Included only integration test coverage.● Also guarded by loads of dependency and component levelunit tests.● UI test coverage not included.
  • 33. Write First Integration Test : Demo 1Scenario To be Automated:Deploy a webapp on WSO2 application server and invoke itonce deployed successfully. Then remove the webapp at theend of test execution.
  • 34. Write First Selenium Test : Demo 2Scenario To be Automated:Deploy a webapp through admin console as a war file and invokeonce deployed successfully.
  • 35. Write Fist Jmeter Test : Demo 3Scenario To be Automated:Deploy Simple Axis2Service on WSO2 Application Server andinvoke it. Then remove the service at the end of test execution.
  • 36. Questions?
  • 37. Engage with WSO2● Helping you get the most out of your deployments● From project evaluation and inception to developmentand going into production, WSO2 is your partner inensuring 100% project success