EclipseCon: Test Confessions - What Eclipsers think and do about testing
Test Confessions: What Eclipsers think and do about Testing Michaela Greiler and Arie van Deursen Delft University of Technology
A Study to identify… …what Eclipsers think and do about … Testing
When it comes to testing… When and why do you adopt a given testing approach? How are you going How do you test to test your GUI? different versions?How do you test the When do you integration of need integration multiple plug-ins? tests?
Today’s Objectives• See test practices in Eclipse• Understand test adoption criteria• Derive actionable items• Foster an open discussion on testing• Learning from each other
Study• Goal: to get a deep understanding of the testing practices and the testing culture within the Eclipse community.• Interviews with 25 experienced developers or testers representing well-known open and closed source projects.• Design followed Grounded Theory.
Grounded TheorySystematic procedure to discover theory from (qualitative) data S. Adolph, W. Hall, Ph. Kruchten. Using Grounded theory to study the experience of software development. Empirical Software Engineering, 2011. B. Glaser and J. Holton. Remodeling grounded theory. Forum Qualitative Res., 2004.
Grounded TheoryAnalysis of – More than 200 testing articles specific 4 Eclipse – Participation & Analysis of @Eclipse events – Interviews with 25 experienced professionals – More than 30 hours – EMF, SOA Platform, xText, Mylyn, Usus, EclEmma, GMF, CDO, IMP, Jubula…
Theory Category Emerging Theory Concept Code Constant Comparison Incident(Interviews, articles…)
Requirements System Test Analysis High Level Integration Design Test Detail Unit Design Test ImplementationTest Scope: collection of components to be verified
“Try to get to a level that you write unit tests, always, whenever you can.““And write your code in such a way that the structure and the classes can be tested with unit tests.” “And then, at max. you use one integration or PDE tests to probe the code.”“Ultimately, unit test are our best friends, and everything else is already difficult.” –P14
Minimal AutomatedIntegration & System Testing Why?
• Technical Prerequisites • Integration in DailyTest Methodology Practice Adoption Model • Application Knowledge • Ownership
Test Adoption Modelexplains explains Used Unused Test TestPractices Practices
Technical Prerequisites• Technical Knowledge Ready available knowledge, existence of books, blogs, training, events…• Application Testability Mocking, models, devices, code coupling, working with frameworks…• Tooling & Test Infrastructure Set-up test environment, tool maturity, usability…
Integration in Daily Practice• Test Execution Time Need of human involvement, duration of test execution, interruption of other tasks…• Test Design Time Availability of time, duration of test design, integration in daily tasks…• Perception of Importance Recognizable benefits, attitude…• Motivation Clarity of responsibilities, identification with responsibility…
Application Knowledge• Design Documents Developer-centered activity, how formal, detail level…• End User Requirements Well-known, clear end user requirements, where do you get them from, determinable from one plug-in….
Ownership• Controllability Eclipse platform code, control over code evolution, influence on other plug-ins…• Responsibility Overall quality, feel responsible, support scenarios/use cases…
Automated Automated Integration & Unit Testing System TestingTechnical Prerequisites GoalTechnical Knowledge AvailableApplication Testability ManageableTooling & TestInfrastructure UsableIntegration in Daily PracticeTest Execution Time ShortTest Design Time AvailableImportance HighMotivation HighApplication KnowledgeDesign Documents ExistingEnd UserRequirements KnownOwnershipControllability HighResponsibility High
Prerequisite for User Involvement Openness Open Communication Release Management Extensibility Feedback Manual Testing Automated TestingApplication Knowledge Alpha & Beta Tester Downstream Projects & Release Train
User DeveloperCommunity Community Plug-In Characteristic Open Communication Extensibility Release Management Openness Openness Manual Automated Integration & System Integration & System Test Testing Testing TestInvolvement Involvement
Implications: Actionable Items• For individuals• For teams• For tool developers• For organizations• For the Eclipse community• For the research community
Product SolutionWhat should the process of the tool user look like?
How to leverage and organize user involvement?
Which technology can support user involvement?
Discussion: Building a Testing Community?• What should the process of the tool user look like?• What should the process of the tool user look like?• How to leverage and organize user involvement?• Which technology can support community involvement?
Your Opinion!1. Participate @Survey www.surveymonkey.com/s/EclipseStudy2. Come and discuss @BOF “Testing with, at and for Eclipse”- Wednesday, 20:30, Alameda Make a chance to win a solar car