Your SlideShare is downloading. ×
  • Like

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Testing in an Open Source Middleware Platform Space The WSO2 Way.

  • 708 views
Published

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
708
On SlideShare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
20
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Testing in an Open-source MiddlewarePlatform Space: The WSO2 Way Iranga Muthuthanthri Technical Lead -QA
  • 2. About WSO2• Providing the only complete open source componentized cloud platform o Dedicated to removing all the stumbling blocks to enterprise agility o Enabling you to focus on business logic and business value• Recognized by leading analyst firms as visionaries and leaders o Gartner cites WSO2 as visionaries in all 3 categories of application infrastructure o Forrester places WSO2 in top 2 for API Management• Global corporation with offices in USA, UK & Sri Lanka o 200+ employees and growing• Business model of selling comprehensive support & maintenance for our products
  • 3. About WSO2
  • 4. Agenda• Engineering process at WSO2.• Product & Release Management.• Testing Process at WSO2.• Patch Verification Process.
  • 5. TestamentThe Problem:• eBay wanted to move away from a home-grown middleware solution to a commercial ESB• Required very fast response times (< 5ms)• Benchmarked 10 ESBs• Including Proprietary, Open Source, Hardware Solutions• WSO2 was the only ESB to support their performance and manageability requirements• 2-3x faster than the other software based solutions on eBay’s workload.The Solution:• 1.5 billion API calls during peak times (prior to 2011 Holiday season & Black Friday)• No downtime experienced during 2010 & 2011 holiday peaks!
  • 6. WSO2 Engineering
  • 7. Engineering ProcessDoing the "right" thing in the "collaborative" way to makepeople "happy".
  • 8. People Centric Process•• People centric process.• User-engineer engagement model.• Promotes constant communication in push mode give and receive feedback.• Build products users actually use and want.• Get the "buy-in" from engineers to follow the process instead of pushing down a model.
  • 9. Influenced by Apache Way Communit y Openness
  • 10. Passion & Commitment
  • 11. Elements (in) Process•• Product management.• Milestone model.• Release model.• Support model.
  • 12. Product Management Relationship
  • 13. Milestone Model
  • 14. Release Cycle
  • 15. Done-Done
  • 16. Support Model•• No Support Engineers.• Engineering runs Support.• SLA is Now!.• Engineers understand the real usage of their creation.• Engineers gets to know customers problems and pain.• Learning hub to understand platform aspects. Makes the next release cycle "Better".
  • 17. ProductOne ProductOne Code BaseOne Build CommandOne Team
  • 18. WSO2 Platform
  • 19. WSO2 Testing Principles•• Quality is #1 priority of Engineering practice.• Engineering owns and is responsible for quality.• Everyone in Engineering does testing.• Collaborative Test methodology.
  • 20. Agile Testing PrinciplesBased on Agile principles• Test early test often.• Face to Face communication (Hackathon mode).• Responsive to change.• Frequent feedback.
  • 21. SOA-Middleware Testing Challenges•• Complexity of testing due to "Headless" state.• Difficulty in defect isolation.• Distribution nature of production deployment environments.• Multiple configuration points for component integrations.
  • 22. SOA Middleware Test Practice Data Server Oracle/MySQ Service L/MSSQL ...Client(Browser/ ESBApp) Registry Service Identity LDAP/AD.. Service
  • 23. Types of Tests•• Unit Tests.• Integration Tests.• End to End Tests.• Performance Tests.• Security Tests.
  • 24. Integration Level Phase•• Objective is to validate the integration scenarios of heterogeneous WSO2 products.E.g: The ESB - Message Broker integration scenario.
  • 25. End to End Tests•• Platform aspects of testing.• Objective is to validate that the platform provides an application developer to build and launch the application in an enterprise context.E.g:Authorize a request to a deployed web service in app serverthrough the entitlement mediator defined in the ESB referring tothe entitlement policy defined at the identity server.
  • 26. Performance Tests• Performance Tests to Measure: o Throughput = requests served per second (data rate /sec). o Latency = How long each transaction takes ( time between requests).
  • 27. Reliability Tests•• Endurance tests: o Long running tests : Tests running for a significant periods of time (days,weeks).• Volume tests: o System loaded with incremental # of services, artifacts.
  • 28. Security TestsSecurity tests for the following threats are executed. Threat Types of tests Spoofing Authentication Tampering Signature Privilege Authorization Information Disclosure Encryption
  • 29. Cloud TestingTest the Cloud native features for• Multi Tenancy.• Elasticity.• Self Service.• Billing & Metering.
  • 30. "Dogfood" Tests•• Dogfood testing is about using own products for real usage.• Internal Patch Management Tool system developed using WSO2 G-reg product.• Usage introduced Product enhancements suggested by Engineers.
  • 31. Team
  • 32. Team•• Recognized as Team of the Year - 2010.• Technically skilled team.• 6+ years experienced SOA Testers.• WSO2 Product experts.• Apache Committers.• Authors and reviewers of published books on SOA .
  • 33. Tools
  • 34. Test Cycles
  • 35. The release life-cycle... • Run integration tests • Commit automated test cases Download builds • Fix regression issues of automated tests • Commit quality code Tester (new code, bug fixes)Automation Engineer Report Jira Release candidate Developer Fix Jira Report Jira Users (Community / Customers)
  • 36. Release Calendar
  • 37. Tester...Phase 1 - Test Planning• For new products - prepare detailed test plans for all features.• For mature products - add test cases for new features.E.g.:-https://docs.google.com/a/wso2.com/spreadsheet/ccc?key=0AvhVRxNoCg0KdHRVMm1pM2xLUDREajdHRVlQekJRcUE#gid=20
  • 38. Sources for Designing Test Plans Support Issues QSPs & Client engagements POCs Architecture & Design Reviews Samples Competitive Products
  • 39. Tester...Phase 2 - Generate Artifacts• Create artifacts for each test scenario - webapps, axis2 services - synapse configurations - configurable governance artifacts - Jmeter scripts (to create APIs, publish, subscribe & invoke in bulk) - scripts for endurance testing E.g.: - https://svn.wso2.org/repos/wso2/trunk/commons/qa/qa-artifacts
  • 40. Tester...Phase 3 (cycle 1) - Test Execution• Setup product platforms in test lab(with clustering/caching/mounting).• Prioritize & execute selected set of test cases on platform setups.• Report all L1s, L2s, L3s, etc (issues related to UI, features, samples & documentation).• Update test progress in the test plan.
  • 41. Developer...Cycle 2•Fix L1s, L2s, etc.•Continue functional & non-functional testing.•Run integration tests.
  • 42. Tester...Phase 3 (cycle 2) - Test Execution• Verify all fixed issues in new build.• Execute test cases and report more issues.Phase 3 (cycle n) - Test Execution• Do final round of testing (smoke test) on final RC (Release Candidate).• If no more L1s are found, release the product.
  • 43. Automation Engineer...• Automates - UI & other features (security, throttling, etc). - certain platform scenarios. - volume test scenarios. - certain performance test scenarios.
  • 44. Automation Engineer...• Code coverage for some products Product LOC coverage Class coverage ESB 38% 66% G-Reg 46% 65% DSS 49% 73% CEP 57% 74% MB 47% 90%
  • 45. Support Patch Verification Process Reports JIRACustomer Release the patch to the customer • Receives the JIRA Private • Post the first response SVN Signs the patch if successful Checkout the patch & Commits the patch verify the fix Simulate customerSupport setup in test labEngineer Provides information on Tester the issue Add integration tests for the patch Public SVN Developer / Commits the fix for future releases Automation Engineer
  • 46. Future... , Support Team
  • 47. Q & A.
  • 48. Engage with WSO2.
  • 49. Engage with WSO2.