Test Confessions: What Eclipsers   think and do about Testing   Michaela Greiler and Arie van Deursen      Delft Universit...
A Study to identify…     …what Eclipsers think and do about …            Testing
When it comes to testing…           When and why do you adopt a given                    testing approach? How are you goi...
Today’s Objectives•   See test practices in Eclipse•   Understand test adoption criteria•   Derive actionable items•   Fos...
Study• Goal: to get a deep understanding of the  testing practices and the testing culture within  the Eclipse community.•...
Grounded TheorySystematic procedure to discover theory from (qualitative) data  S. Adolph, W. Hall, Ph. Kruchten. Using Gr...
Grounded TheoryAnalysis of  – More than 200 testing articles specific 4 Eclipse  – Participation & Analysis of @Eclipse ev...
Theory      Category            Emerging Theory       Concept        Code                          Constant Comparison    ...
Requirements                                                    System Test     Analysis          High Level              ...
Tendency: No Separate Test          Teams
Focus on Unit Testing
Integration    Tests
Integration & System Testing
“Try to get to a level that you write unit tests,             always, whenever you can.““And write your code in such a way...
Who does automated unit testing?
Who does automated integration testing?
Who does automated cross-feature      integration testing?
Who does automated testing for different      Eclipse platform versions?
Who does automated combinatorial testing     for OS, versions and plug-ins?
Testing Plug-in IntegrationCross-Plug-In Integration?                         Combinatorial Testing?           Versioning?
Minimal AutomatedIntegration & System Testing       Why?
• Technical Prerequisites                   • Integration in DailyTest Methodology     Practice Adoption Model    • Applic...
Test Adoption Modelexplains               explains Used             Unused  Test              TestPractices         Practi...
Technical Prerequisites• Technical Knowledge  Ready available knowledge, existence of  books, blogs, training, events…• Ap...
Integration in Daily Practice• Test Execution Time  Need of human involvement, duration of test  execution, interruption o...
Application Knowledge• Design Documents  Developer-centered activity, how formal,  detail level…• End User Requirements  W...
Ownership• Controllability  Eclipse platform code, control over code  evolution, influence on other plug-ins…• Responsibil...
Automated                          Automated                                          Integration &                       ...
Compensation Strategy
ManualTesting?
“These tests that I do are very simple manual tests, the real tests are coming from the users, that are  doing all kind of...
But, how to involve the users?
Prerequisite for User Involvement                            Openness  Open Communication    Release Management        Ext...
User                   DeveloperCommunity                                                        Community                ...
Implications: Actionable Items•   For individuals•   For teams•   For tool developers•   For organizations•   For the Ecli...
How can the IDE support integration testing?
Product                                SolutionWhat should the process of the tool user look like?
How to leverage and organize user involvement?
Which technology can support user involvement?
Discussion:     Building a Testing Community?• What should the process of the tool user look  like?• What should the proce...
Your Opinion!1. Participate @Survey  www.surveymonkey.com/s/EclipseStudy2. Come and discuss @BOF  “Testing with, at and fo...
EclipseCon: Test Confessions - What Eclipsers think and do about testing
Upcoming SlideShare
Loading in …5
×

EclipseCon: Test Confessions - What Eclipsers think and do about testing

1,334 views

Published on

Slides of EclipseCon presentation: Test Confessions - What Eclipsers think and do about testing

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,334
On SlideShare
0
From Embeds
0
Number of Embeds
229
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

EclipseCon: Test Confessions - What Eclipsers think and do about testing

  1. 1. Test Confessions: What Eclipsers think and do about Testing Michaela Greiler and Arie van Deursen Delft University of Technology
  2. 2. A Study to identify… …what Eclipsers think and do about … Testing
  3. 3. When it comes to testing… When and why do you adopt a given testing approach? How are you going How do you test to test your GUI? different versions?How do you test the When do you integration of need integration multiple plug-ins? tests?
  4. 4. Today’s Objectives• See test practices in Eclipse• Understand test adoption criteria• Derive actionable items• Foster an open discussion on testing• Learning from each other
  5. 5. Study• Goal: to get a deep understanding of the testing practices and the testing culture within the Eclipse community.• Interviews with 25 experienced developers or testers representing well-known open and closed source projects.• Design followed Grounded Theory.
  6. 6. Grounded TheorySystematic procedure to discover theory from (qualitative) data S. Adolph, W. Hall, Ph. Kruchten. Using Grounded theory to study the experience of software development. Empirical Software Engineering, 2011. B. Glaser and J. Holton. Remodeling grounded theory. Forum Qualitative Res., 2004.
  7. 7. Grounded TheoryAnalysis of – More than 200 testing articles specific 4 Eclipse – Participation & Analysis of @Eclipse events – Interviews with 25 experienced professionals – More than 30 hours – EMF, SOA Platform, xText, Mylyn, Usus, EclEmma, GMF, CDO, IMP, Jubula…
  8. 8. Theory Category Emerging Theory Concept Code Constant Comparison Incident(Interviews, articles…)
  9. 9. Requirements System Test Analysis High Level Integration Design Test Detail Unit Design Test ImplementationTest Scope: collection of components to be verified
  10. 10. Tendency: No Separate Test Teams
  11. 11. Focus on Unit Testing
  12. 12. Integration Tests
  13. 13. Integration & System Testing
  14. 14. “Try to get to a level that you write unit tests, always, whenever you can.““And write your code in such a way that the structure and the classes can be tested with unit tests.” “And then, at max. you use one integration or PDE tests to probe the code.”“Ultimately, unit test are our best friends, and everything else is already difficult.” –P14
  15. 15. Who does automated unit testing?
  16. 16. Who does automated integration testing?
  17. 17. Who does automated cross-feature integration testing?
  18. 18. Who does automated testing for different Eclipse platform versions?
  19. 19. Who does automated combinatorial testing for OS, versions and plug-ins?
  20. 20. Testing Plug-in IntegrationCross-Plug-In Integration? Combinatorial Testing? Versioning?
  21. 21. Minimal AutomatedIntegration & System Testing Why?
  22. 22. • Technical Prerequisites • Integration in DailyTest Methodology Practice Adoption Model • Application Knowledge • Ownership
  23. 23. Test Adoption Modelexplains explains Used Unused Test TestPractices Practices
  24. 24. Technical Prerequisites• Technical Knowledge Ready available knowledge, existence of books, blogs, training, events…• Application Testability Mocking, models, devices, code coupling, working with frameworks…• Tooling & Test Infrastructure Set-up test environment, tool maturity, usability…
  25. 25. Integration in Daily Practice• Test Execution Time Need of human involvement, duration of test execution, interruption of other tasks…• Test Design Time Availability of time, duration of test design, integration in daily tasks…• Perception of Importance Recognizable benefits, attitude…• Motivation Clarity of responsibilities, identification with responsibility…
  26. 26. Application Knowledge• Design Documents Developer-centered activity, how formal, detail level…• End User Requirements Well-known, clear end user requirements, where do you get them from, determinable from one plug-in….
  27. 27. Ownership• Controllability Eclipse platform code, control over code evolution, influence on other plug-ins…• Responsibility Overall quality, feel responsible, support scenarios/use cases…
  28. 28. Automated Automated Integration & Unit Testing System TestingTechnical Prerequisites GoalTechnical Knowledge AvailableApplication Testability ManageableTooling & TestInfrastructure UsableIntegration in Daily PracticeTest Execution Time ShortTest Design Time AvailableImportance HighMotivation HighApplication KnowledgeDesign Documents ExistingEnd UserRequirements KnownOwnershipControllability HighResponsibility High
  29. 29. Compensation Strategy
  30. 30. ManualTesting?
  31. 31. “These tests that I do are very simple manual tests, the real tests are coming from the users, that are doing all kind of different things with *x+.“—P9
  32. 32. But, how to involve the users?
  33. 33. Prerequisite for User Involvement Openness Open Communication Release Management Extensibility Feedback Manual Testing Automated TestingApplication Knowledge Alpha & Beta Tester Downstream Projects & Release Train
  34. 34. User DeveloperCommunity Community Plug-In Characteristic Open Communication Extensibility Release Management Openness Openness Manual Automated Integration & System Integration & System Test Testing Testing TestInvolvement Involvement
  35. 35. Implications: Actionable Items• For individuals• For teams• For tool developers• For organizations• For the Eclipse community• For the research community
  36. 36. How can the IDE support integration testing?
  37. 37. Product SolutionWhat should the process of the tool user look like?
  38. 38. How to leverage and organize user involvement?
  39. 39. Which technology can support user involvement?
  40. 40. Discussion: Building a Testing Community?• What should the process of the tool user look like?• What should the process of the tool user look like?• How to leverage and organize user involvement?• Which technology can support community involvement?
  41. 41. Your Opinion!1. Participate @Survey www.surveymonkey.com/s/EclipseStudy2. Come and discuss @BOF “Testing with, at and for Eclipse”- Wednesday, 20:30, Alameda Make a chance to win a solar car

×