Exploring the fault finding
capabilities of online testing in
              OSGi

Michaela Greiler    Hans-Gerhard Gross  ...
Overview
• Service-Oriented Architectures
      – SOA Challenges
• Online Testing in OSGi
      – A Case Study
      – See...
Service-Oriented Architectures


  A software architecture that aims at reusability
    and interoperability by exposing f...
SOA Runtime Evolution: Challenge I
Stakeholder Separation

                                      •    Communication
      ...
SOA Runtime Evolution: Challenge II
Service Integration

                                         • Black-boxes
          ...
SOA Runtime Evolution: Challenge III
Service Versioning and Migration


                                                  ...
SOA Runtime Evolution: Challenge IV
             Service Binding and Reconfiguration




                        Technical...
Case Study Design
• Goal: assess the capabilities of our online testing framework to find typical
  reconfiguration faults...
Case Study Overview




                  Technical University Delft – OSGi
14-12-2009
                         Users’Foru...
Built-in Testing
• Associate the tests to the component
• Isolate effects of testing operations
• Benefits
      –      Si...
Built-in Online Testing




                   Technical University Delft – OSGi
14-12-2009
                          User...
Online Tests
• Discovery Tests:
      – Can I find all required services?
      – Are those services the services expected...
Test Process




              Technical University Delft – OSGi
14-12-2009
                     Users’Forum 2009
SOA Faults
Fault Classes   Type/ Form
Publishing      Service Description
                Service Deployment
             ...
Faults, Forms and Tests
Fault Classes   Type/ Form                      Found by
Publishing      Service Description      ...
Example: Faulty Reconfiguration




                Technical University Delft – OSGi
14-12-2009
                       Us...
Lessons Learned
• Online versus Offline Testing
      – Adequate test environment
      – Current state of the production ...
General Technical Issues
      •      Extended Lifecycle Support
      •      Central Auditing Authority
      •      Addi...
Summary
• Online built-in testing framework that
  provides test isolation
• A case study demonstrating the capabilities o...
Thank you!
                       Michaela Greiler
                    m.s.greiler@tudelft.nl
                   Technical...
Upcoming SlideShare
Loading in …5
×

Exploring the fault finding capabilities of online testing in OSGi

738 views
681 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
738
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Exploring the fault finding capabilities of online testing in OSGi

  1. 1. Exploring the fault finding capabilities of online testing in OSGi Michaela Greiler Hans-Gerhard Gross Arie van Deursen Technical University Delft Software Engineering Research Group
  2. 2. Overview • Service-Oriented Architectures – SOA Challenges • Online Testing in OSGi – A Case Study – Seeded Faults – Fault Finding Capabilities Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  3. 3. Service-Oriented Architectures A software architecture that aims at reusability and interoperability by exposing functionality as loosely coupled services, that can be composed to form business processes. Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  4. 4. SOA Runtime Evolution: Challenge I Stakeholder Separation • Communication • Ownership • Responsibilities • Influence Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  5. 5. SOA Runtime Evolution: Challenge II Service Integration • Black-boxes • Test environment • (Runtime)-Testability Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  6. 6. SOA Runtime Evolution: Challenge III Service Versioning and Migration • Notification • Update Strategy • Regression Testing Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  7. 7. SOA Runtime Evolution: Challenge IV Service Binding and Reconfiguration Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  8. 8. Case Study Design • Goal: assess the capabilities of our online testing framework to find typical reconfiguration faults • Approach: 1. extend original system with online testing infrastructure 2. reconfiguration scenario, were a number of faults have been seeded • Key Metrics and Execution Environment: – Apache Felix 1.8 – OVTS system: 6 key bundles, 1 utility bundle, 2176 LOCs, 41 classes. 349 LOCs dedicated to testing; 39 test cases consisting of 290 LOCs. • Units of Analysis: – level of provided test isolation – detection rate for the seeded faults – how have they been found, or why couldn’t they be detected? – evaluation of the support of the execution environment to prevent inconsistencies in the overall system Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  9. 9. Case Study Overview Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  10. 10. Built-in Testing • Associate the tests to the component • Isolate effects of testing operations • Benefits – Simplify the management of the tests – Can add tests at runtime – Components’ functionality is separated from the tests – At anytime of the components can be re-validated, whenever needed (eg. after a modification of the system) • Adoption effort – Explicit definition of the tests – Testing framework has to be linked with the system platform Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  11. 11. Built-in Online Testing Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  12. 12. Online Tests • Discovery Tests: – Can I find all required services? – Are those services the services expected? • Binding Tests: – Can I bind required services? – Do they conform syntactically (service descriptions) to my expectation? • Execution Tests: – Test invocations looking for: • expectations about sequential constraints, i.e., the protocol, • simple input-output relations • non-functional requirements (e.g., timing) Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  13. 13. Test Process Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  14. 14. SOA Faults Fault Classes Type/ Form Publishing Service Description Service Deployment Publish Discover Discovery No Service Found Wrong Service Found Timed Out Composition No Valid Composition Composition Faulty Execute Composition Timed Out Binding Binding Denied Bound to Wrong Service Bind Timed Out Execution Service Crashed Incorrect Results Timed Out “A Fault Taxonomy for Service-Oriented Architecture” by Bruning et al. 2007 Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  15. 15. Faults, Forms and Tests Fault Classes Type/ Form Found by Publishing Service Description Binding Test Service Deployment Binding Test Discovery No Service Found Discovery Test Wrong Service Found Discovery Test Timed Out Time Test Composition No Valid Composition Composition Test Composition Faulty Composition Test Timed Out Time Test Binding Binding Denied Composition Test Bound to Wrong Service X Timed Out Time Test Execution Service Crashed Discovery Test Incorrect Results X (unit tests) Timed Out Time Test Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  16. 16. Example: Faulty Reconfiguration Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  17. 17. Lessons Learned • Online versus Offline Testing – Adequate test environment – Current state of the production system • Shortcomings OSGi Framework – Support for overall consistency management • Service - Description Mismatch • Declaration of Dependencies • Export/Import of Packages (e.g. wiring) – Middleware versus Application Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  18. 18. General Technical Issues • Extended Lifecycle Support • Central Auditing Authority • Additional Information • State Transfer • Automated Test Generation • (Runtime)-Testability • Performance Overhead Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  19. 19. Summary • Online built-in testing framework that provides test isolation • A case study demonstrating the capabilities of this online testing framework to detect faults during system reconfiguration that could not be addressed in the test environment • OSGi platform should support consistency management Technical University Delft – OSGi 14-12-2009 Users’Forum 2009
  20. 20. Thank you! Michaela Greiler m.s.greiler@tudelft.nl Technical University Delft Software Engineering Research Group Technical University Delft – OSGi 14-12-2009 Users’Forum 2009

×