Analysing Dependencies andProviding Assurance in the World of      Plug-In Based Systems                  Shivanshu Singh ...
Study of Plugin Testing Practices in the Eclipse EcosystemM. Greiler, A. van Deursen and M. Storey.Test Confessions: A Stu...
Motivation• Plugin based systems are pretty mainstream• Deployed for serious commercial and non commercial use
Motivation• Myriad of plug-in combinations, versions• So many participants• Explosion of combinations• Need to make plugin...
the unseen and the unknown
Memory         Drivers
Memory         Drivers
And its growing ….
Compatibility Problems
“The number one reason people give us for not upgradingto the latest version of WordPress is fear that their pluginswon‟t ...
Point of InterestMissing dependencies not realized until eclipse isstarted.NoClassDefFound errors when trying to executeso...
Eclipse Plugin
So how does this really work?
plugin            plugin       pluginExtensionPoint                plugin
The PluginMANIFEST.MFContains the configuration,dependency informationPlugin.xmlContains plugin extensions /extension poin...
The Plugin                                Manifest-Version: 1.0                                Export-Package: com.shivans...
The Plugin                                <?xml version="1.0" encoding="UTF-8"?>                                <?eclipse ...
The Plugin                                • That implements the plugin                                    – Extends anothe...
Feature• Packs plugins together   – e.g. Ui plugin and Some parser plugin , collectively     offering a new functionality•...
FEATURE                                                                                                                   ...
The Eclipse Study
The Eclipse Study    Study of Plugin Testing Practices in the Eclipse Ecosystem    Objective: to increase understanding of...
The Eclipse Study   Interview of 25 senior practitioners   (most paid to develop software)   survey among 161 Eclipse deve...
Questions• Which testing practices are prevalent in the testing of plug-in-based  systems?• Does the plug-in architecture ...
Finding:Lack of dedicated test effort                                          10                                         ...
Friends with Unit Testing “[..]Ultimately, unit tests are our best friends, and everything elseis already difficult”• unit...
…. But they know that its not enough“We try to encapsulate the logic as much aspossible to be able to test with unit tests...
Questions• Which testing practices are prevalent in the testing of plug-in-based  systems?• Does the plug-in architecture ...
• Unit Testing with Eclipse PDE (Plugin Dev. Env.)    – JUnit tests using a special test runner that launches      another...
• What about testing plugin Extensions?Many different views   Some do it partially, some don‟tOne participant said:“I am n...
• What about Plugin Integration Testing?• Third Party Integration Testing?none of the projects report to have automated te...
Does this mean thatcross-product integration             No Ideaproblems do not occur?                 20%                ...
Integration / Compatibility Testing    • Largely treated optional.    • Integration testing is mainly performed manually, ...
Integration / Compatibility Testing               3%                                No compatibility tests, AT            ...
Testing Dependencies?    Version Ranges:    “A lot of people put version ranges in their bundle dependencies, and    they ...
Testing Dependencies?    • 63% do not test for plugin version dependencies    • Rest either don‟t or do it only for a very...
Testing Dependencies?     “We use a very old version of the main plug-in we depend on.     Sometimes we update, but there ...
Questions• Which testing practices are prevalent in the testing of plug-in-based  systems?• Does the plug-in architecture ...
Why not test dependencies?     • The responsibility for this kind of testing is less       clear.        – Offloading the ...
Why not test dependencies?     • Inconsistent dependency specification practices       and conventions        – Feature?  ...
Why not test dependencies?     • Unavailability of plug-in testing knowledge         Or Infrastructure     “Why [testing] ...
Why not test dependencies?      In short:      • Insufficient plug in testing knowledge and support      • Lack of ownersh...
Questions• Which testing practices are prevalent in the testing of plug-in-based  systems?• Does the plug-in architecture ...
Self Hosting   Or “eating your own dog food”   Warren Harrison. Eating your own dog food. IEEE Softw., 23:5–7, May 2006.  ...
Community Involvement  “The tests that I perform are very simple manual tests, the real tests are  coming from the users, ...
Observations Summary • Eclipse community is lost in testing [plugins]     Lack of time, resources, combinatorial complexit...
Making It Better
Community Involvement     • Make the community involvement more       attractive       – Reward participation       – Test...
Compatibility Repository                Central repository of                compatibility information                – Ea...
Plugin Analysis    Analyse plugins to see if there are any unstated    dependencies that exist    – Which in turn can be r...
Ongoing Work Analysis of features plugins over version ranges         – to see if there are any unstated dependencies that...
Further Study   Study other plugin-in based ecosystems   • mechanisms (interfacing, testing)
Plugin testing beyond unit testing is:“annoying” “distracting” “painful”Lack oftime, resources, infrastructure, combinator...
Plugin testing beyond unit testing is:“annoying” “distracting” “painful”                                                 C...
Analysing Dependencies and Providing Assurance  in the world of Plug In based systems (Eclipse Ecosystem)
Analysing Dependencies and Providing Assurance  in the world of Plug In based systems (Eclipse Ecosystem)
Upcoming SlideShare
Loading in …5
×

Analysing Dependencies and Providing Assurance in the world of Plug In based systems (Eclipse Ecosystem)

569 views

Published on

Plug In based systems are quite mainstream these days and form some of the most popular software systems used everyday. Eclipse, Android, Wordpress and Firefox are some examples. With so many plugins, numerous vendors, a big range of versions of these plugins and combinations thereof that can possibly exist, its easy to imagine all sorts of problems this might cause: Problems of testing the various combinations and checking dependencies among the various plugins and their versions, to list a few. In this presentation, we take a look at the concerns that exist when it comes to testing plugin based systems, especially in the context of the Eclipse ecosystem and what can be done about these issues with possible directions in which work can be done to ease some of the problems that exist.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
569
On SlideShare
0
From Embeds
0
Number of Embeds
65
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Known vs UnknownCentralized vs Decentralized
  • Consolidate Introduction slides into one.
  • Eclipse Menu item plugin -&gt; nothing happens on click.
  • Consolidate Introduction slides into one.
  • Consolidate Introduction slides into one.
  • Analysing Dependencies and Providing Assurance in the world of Plug In based systems (Eclipse Ecosystem)

    1. 1. Analysing Dependencies andProviding Assurance in the World of Plug-In Based Systems Shivanshu Singh sks@cmu.edu Institute for Software Research, School of Computer Science, Carnegie Mellon University
    2. 2. Study of Plugin Testing Practices in the Eclipse EcosystemM. Greiler, A. van Deursen and M. Storey.Test Confessions: A Study of Testing Practices for Plug-In Systems (ICSE 2012)What Eclipsers think and do about testing: A grounded theory. Technical Report SERG-2011-010, Delft University of Technology, 2011.
    3. 3. Motivation• Plugin based systems are pretty mainstream• Deployed for serious commercial and non commercial use
    4. 4. Motivation• Myriad of plug-in combinations, versions• So many participants• Explosion of combinations• Need to make plugin based systems more reliable
    5. 5. the unseen and the unknown
    6. 6. Memory Drivers
    7. 7. Memory Drivers
    8. 8. And its growing ….
    9. 9. Compatibility Problems
    10. 10. “The number one reason people give us for not upgradingto the latest version of WordPress is fear that their pluginswon‟t be compatible.”http://wordpress.org/news/2009/10/plugin-compatibility-beta“Thanks, but I think we have given up on Eclipse andBugzilla integration.”https://bugs.eclipse.org/bugs/show_bug.cgi?id=268207
    11. 11. Point of InterestMissing dependencies not realized until eclipse isstarted.NoClassDefFound errors when trying to executesomething provided by a plugin“How do I debug the Eclipse Plugin error „Pluginwas unable to load class‟?”http://stackoverflow.com/questions/648024/how-do-i-debug-the-eclipse-plugin-error-plugin-was-unable-to-load-classhttp://stackoverflow.com/questions/9128132/eclipse-plugin-install-errorhttp://androidforums.com/developer-101/543787-error-installing-adt-plugin-in-eclipse.html#post4343892http://www.eclipsezone.com/eclipse/forums/t93278.rhtmlhttp://stackoverflow.com/questions/1509511/eclipse-plugin-noclassdeffounderror-on-org-eclipse-jdt-ui-javauihttp://stackoverflow.com/questions/11514883/conflicting-dependencies-while-installing-adt-plugin-for-eclipse-helios
    12. 12. Eclipse Plugin
    13. 13. So how does this really work?
    14. 14. plugin plugin pluginExtensionPoint plugin
    15. 15. The PluginMANIFEST.MFContains the configuration,dependency informationPlugin.xmlContains plugin extensions /extension points related infoCode
    16. 16. The Plugin Manifest-Version: 1.0 Export-Package: com.shivanshusingh.eclipse.plugin.subMenuDefinition Require-Bundle: org.eclipse.ui, org.eclipse.core.runtime,MANIFEST.MF com.venukb.plugin.base;bundle-version=“[3.0.0,4.0.0)” com.shivanshu.eclipse.plugin.extensionplugin,Contains the configuration, Bundle-Version: 1.0.0.201210111514dependency information Bundle-Activator: com.shivanshusingh.eclipse.plugin.helloworld.Activator Bundle-ManifestVersion: 2 Import-Package: org.eclipse.e4.core.di.annotations Bundle-SymbolicName: com.shivanshusingh.eclipse.plugin.helloworld; singleton:=truePlugin.xml Bundle-RequiredExecutionEnvironment: JavaSE-1.6 Bundle-ClassPath: .,libs/sdkuilib.jar,libs/ninepatch.jar,libs/freemarker-Contains plugin extensions / 2.3.19.jar,libs/rule_api.jar,libs/assetstudio.jar,libs/lint_api.jar,libs/lint_checks.jar,libs/loextension points related info mbok-ast-0.2.jar,libs/asm-4.0.jar,libs/asm-tree-4.0.jar,libs/asm-analysis- 4.0.jar,libs/propertysheet.jar,libs/ant-glob.jar,libs/swtmenubar.jar,libs/manifmerger.jarCode
    17. 17. The Plugin <?xml version="1.0" encoding="UTF-8"?> <?eclipse version="3.4"?> <plugin> <extension-point id="subMenu" name="SampleSubMenu” schema="schema/subMenu.exsd"/>MANIFEST.MF <extension point="org.eclipse.ui.commands">Contains the configuration, <category name="Sample Category"dependency information id="com.shivanshusingh.eclipse.plugin.helloworld.commands.category”> .... .... </extension>Plugin.xml <runtime>Contains plugin extensions / <library name="lib/jaxbbuilder.jar"> <export name="*"/>extension points related info </library> .... </runtime> <requires>Code <import plugin="org.eclipse.ui"/> <import plugin="org.eclipse.ui.console" version="3.4.0"/> .... </requires> </plugin>
    18. 18. The Plugin • That implements the plugin – Extends another plugin‟s extension pointsMANIFEST.MF – Extension Point code (to use possible extensionsContains the provided by others)configuration, dependency – Exports more new codeinformation – Uses some exported public codePlugin.xml • Public code (exported packages)Contains plugin extensions / – can be accessed by other dependent pluginsextension points related info – Significant because: • this dependency may not have been explicitly specifiedCode • the contents of the exported packages (that other plugins might be using) might change over various versions
    19. 19. Feature• Packs plugins together – e.g. Ui plugin and Some parser plugin , collectively offering a new functionality• Can depend upon – Other features – Other plugins ( Feature.xml )• [Eclipse] Available generally from an update site
    20. 20. FEATURE p P l M L l u A U li g N G ib l i II i/ b n F N /j b . E ja / x S feature.xml m jr a T p r a .P l l rl LM PLUGIN PLUGIN M u UA li F g GN ib l i II i/ b n F N / bj . E ja / x S m jr a PLUGIN T r a l . r M F lib/jar lib/jar plugin.xml plugin.xml plugin.xml lib/jar lib/jar lib/jar lib/jar l i MANIFEST.MF l l b MANIFEST.MF i MANIFEST.MF i / b b j / / a j j r a a r r p M p M p M p p P P l A l A l A l l P l P l LM l LM l u N l l i u N P ll l u N l l u u L L i UA li UA li g I i i g I L p Mii g I i i g N ib g N i U M b p b U i b U b b G l G ib l F b i F lb A i F b i i G A / l b /P l G / II i/ b II i/ b n E P / / ll n E G u N/ l l n E / / n n I u j N j /L i I j NF / bj NF / bj . S L j l ii . S I gjj Ij i i . S j j . E . E N g a I a ib N aU b b N a a ja / ja / x T U a x T ia F b x T a x x F r i b a rG / r S m jr a S jr a m . G r r b // m . n ErrI / / j m . r r T m T l n E / j r M . S l a l r a . S I jj ja l M N j a l M . r . r F N x T a a F x T a F r M M ar m . r r m . r r F F l M l M F FFEATURE P L U G I N l i b plugin.xml / j a l r i b / j a l r i MANIFEST.MF b / PLUGIN j a r feature.xml P L U G plugin.xml lib/jar lib/jar l I MANIFEST.MF i N b l / i j b a plugin.xml / r j a l r i b / j a l r i MANIFEST.MF b / j a rPLUGIN lib/jar plugin.xml lib/jar MANIFEST.MF lib/jar PLUGIN lib/jar plugin.xml lib/jar l MANIFEST.MF i b / j a rP PL LU U PpG G L Pp l PI I U l p L u lN M i Ll U l N Gg lb l u M A Uu G i l Ii i M i g A lb l N G i Nn l/ b g A lb Ii N i/ i I Ii b . ij / N i/ Nn I l b / b F x ba Nn I l b . ij / plugin.xml plugin.xml / E jr . ij / F ba j m / a F a x E jr j S x b / a l j E jr m a l a T r / S r l a m S a l j r i r . j T i r l T r a b M a . b . r / F r M / M F j j F a a l l P r p i r L MANIFEST.MF i l b MANIFEST.MF U l b u / M i / Pp Gg lb j A j Ll Ii i/ a l N a Uu Nn l b M I r r Gg i lb . ij / A F a Ii i/ x b jr N E Nn l b m / a I S . ij / l j r F a T x b jr a E . m / a r S M l j r T F a . r M F
    21. 21. The Eclipse Study
    22. 22. The Eclipse Study Study of Plugin Testing Practices in the Eclipse Ecosystem Objective: to increase understanding of what testers and developers think and do when it comes to testing plug-in based systems. M. Greiler, A. van Deursen and M. Storey. Test Confessions: A Study of Testing Practices for Plug-In Systems (ICSE 2012) What Eclipsers think and do about testing: A grounded theory. Technical Report SERG-2011-010, Delft University of Technology, 2011.
    23. 23. The Eclipse Study Interview of 25 senior practitioners (most paid to develop software) survey among 161 Eclipse developers / testers @ EclipseCon Grounded Theory
    24. 24. Questions• Which testing practices are prevalent in the testing of plug-in-based systems?• Does the plug-in architecture lead to specific test approaches? What about plug-in specific integration challenges ? (e.g. versioning)• What are the barriers for plug in testing?• Additional compensation strategies used to support the testing of plug-ins?
    25. 25. Finding:Lack of dedicated test effort 10 15 0“Tester and developer, that’s one person. From our view, it does notmake sense to have a dedicated test team, which has no idea aboutwhat the software does and can only write some tests”
    26. 26. Friends with Unit Testing “[..]Ultimately, unit tests are our best friends, and everything elseis already difficult”• unit testing reported being the main automated test effort, rest is manual• popular, probably because in the majority of the projects, developers are responsible for testing.
    27. 27. …. But they know that its not enough“We try to encapsulate the logic as much aspossible to be able to test with unit tests.What cannot be encapsulated is not tested.”
    28. 28. Questions• Which testing practices are prevalent in the testing of plug-in-based systems?• Does the plug-in architecture lead to specific test approaches? What about plug-in specific integration challenges ? (e.g. versioning)• What are the barriers for plug in testing?• Additional compensation strategies used to support the testing of plug-ins?
    29. 29. • Unit Testing with Eclipse PDE (Plugin Dev. Env.) – JUnit tests using a special test runner that launches another Eclipse instance in a separate VM – Limited use only – Nothing more than Unit testing again“Our PDE tests do not really look at the integration of two components.”Participants use PDE tests for automated unit testing
    30. 30. • What about testing plugin Extensions?Many different views Some do it partially, some don‟tOne participant said:“I am not sure if there is a need to test if extensions correctly supportthe extension point, because it is mostly a registration thing.”BUT Bugs are there!“[….] The small glue code where the registry gets the extension, that’snot tested, because it is just hard to test that. And for these untestedglue code parts we had the most bugs.”
    31. 31. • What about Plugin Integration Testing?• Third Party Integration Testing?none of the projects report to have automated tests toensure product compatibility.Expected: „must play nicely with each other‟
    32. 32. Does this mean thatcross-product integration No Ideaproblems do not occur? 20% 28% Think „This should not happen‟ Are very well aware – some even pointed to the actual bug reports ** https://bugs.eclipse.org/bugs/Bug Identifier: 280598 and 213988 52%
    33. 33. Integration / Compatibility Testing • Largely treated optional. • Integration testing is mainly performed manually, or in a bug-driven way • Some others do it to a limited extent : with „some‟ other plugins “We handle problems between several plug-ins in a bug-driven way. If there is a bug we write a test, but we do not think ahead which problems could there be.”
    34. 34. Integration / Compatibility Testing 3% No compatibility tests, AT ALL 54% 43% Ad hoc, if at all Some systematic way
    35. 35. Testing Dependencies? Version Ranges: “A lot of people put version ranges in their bundle dependencies, and they say we can run with 3.3 up to version 4.0. But I am willing to bet that 99% of the people do not test that their stuff works, they might assert it, but I do not believe that they test.” In reality, many participants report that they test with one fixed version for each dependency
    36. 36. Testing Dependencies? • 63% do not test for plugin version dependencies • Rest either don‟t or do it only for a very specific and very small set of plugin versions most class not found errors are probably because of this Require-Bundle: org.eclipse.ui, org.eclipse.core.runtime, Specific Versions / ranges com.venukb.plugin.base;bundle-version=“[3.3.0,4.0.0)”, Are you sure? com.shivanshu.eclipse.plugin.extensionplugin, Is any version alright?
    37. 37. Testing Dependencies? “We use a very old version of the main plug-in we depend on. Sometimes we update, but there is always the risk that it will break something and then you have to do extensive [manual] testing.” .. So just leave it. A general feeling: Too many versions, too many plugins to deal with, just can‟t get to testing them all
    38. 38. Questions• Which testing practices are prevalent in the testing of plug-in-based systems?• Does the plug-in architecture lead to specific test approaches? What about plug-in specific integration challenges ? (e.g. versioning)• What are the barriers for plug in testing?• Additional compensation strategies used to support the testing of plug-ins?
    39. 39. Why not test dependencies? • The responsibility for this kind of testing is less clear. – Offloading the technical debt. • Most projects restrict their official support for compatibility with third-party plug-ins and the Eclipse platform itself !
    40. 40. Why not test dependencies? • Inconsistent dependency specification practices and conventions – Feature? – Plugin? – Manifest?
    41. 41. Why not test dependencies? • Unavailability of plug-in testing knowledge Or Infrastructure “Why [testing] is so difficult? For Web projects, you find good templates. For Eclipse, you don’t. [...] Especially for testing plug- ins, we would need some best practices.”
    42. 42. Why not test dependencies? In short: • Insufficient plug in testing knowledge and support • Lack of ownership and responsibility No time, money or resources Any testing beyond unit testing is: “annoying” “distracting” “painful”
    43. 43. Questions• Which testing practices are prevalent in the testing of plug-in-based systems?• Does the plug-in architecture lead to specific test approaches? What about plug-in specific integration challenges ? (e.g. versioning)• What are the barriers for plug in testing?• Additional compensation strategies used to support the testing of plug-ins?
    44. 44. Self Hosting Or “eating your own dog food” Warren Harrison. Eating your own dog food. IEEE Softw., 23:5–7, May 2006. …. But it goes only so far …. What about dependency testing for other plugins‟ versions, features?
    45. 45. Community Involvement “The tests that I perform are very simple manual tests, the real tests are coming from the users, who are doing all kind of different Things” To the community …. to cope with combinatorial testing Every installation of the plugin out there is like a test in itself .. .. !
    46. 46. Observations Summary • Eclipse community is lost in testing [plugins] Lack of time, resources, combinatorial complexity problems • Lack of responsibility for testing plugin compatibility – Moving beyond unit testing is seemed very difficult – Lack of Plugin testing knowledge – .. and tool support – Dependency specification practices Loose (and sometimes too rigid) interface specifications • Community involvement to the rescue
    47. 47. Making It Better
    48. 48. Community Involvement • Make the community involvement more attractive – Reward participation – Test days (Mozilla) – Bug Bounties (Facebook, Google, PayPal ….) • Still .. questionable – Bounties in a collaborative / plug-in, open source context?
    49. 49. Compatibility Repository Central repository of compatibility information – Ease the problem of arbitrary or very rigid specification of dependencies • Where versions dependencies are not specified, or are incorrect, or can be eased .
    50. 50. Plugin Analysis Analyse plugins to see if there are any unstated dependencies that exist – Which in turn can be recorded in the central repository Apply clever sampling techniques, to test the ranges of plugin versions to determine compatibility – Again to say speed up the time to build the central repository and then improve upon that later • Crowdsourcing compatibility checks  maintain it in a repository e.g. the Wordpress Compatibility Checker (Beta) http://wordpress.org/news/2009/10/plugin-compatibility-beta/
    51. 51. Ongoing Work Analysis of features plugins over version ranges – to see if there are any unstated dependencies that exist • Parse Java bytecode – extract public methods, types and such information – extract all outbound calls • (INVOKEVIRTUAL, INVOKESTATIC etc.) – Record these dependencies in a central repository. • Beyond what comes with MANIFEST, plugin.xml etc. Plans: • Make this available as a service – give us your plugin / feature jar and we will tell you what you need Or – give us the plugin / feature name, version and we will tell you what it needs • As an eclipse plugin ..for PDE? That uses this service Side Effects: (positive) Study API Deprecation in Java Ecosystems – frequency of usage and ripple effects over time. (Romain Robbes, Mircea Lungu, David Röthlisberger, “How Do Developers React to API Deprecation? The Case of a Smalltalk Ecosystem”, ESEC/FSE ‟12, November 10–17 2012, Research Triangle Park, North Carolina, United States)
    52. 52. Further Study Study other plugin-in based ecosystems • mechanisms (interfacing, testing)
    53. 53. Plugin testing beyond unit testing is:“annoying” “distracting” “painful”Lack oftime, resources, infrastructure, combinatorialcomplexityLoose (or too rigid) interface / dependencyspecificationsToo many plugins, too many versions
    54. 54. Plugin testing beyond unit testing is:“annoying” “distracting” “painful” Central repository of compatibility informationLack oftime, resources, infrastructure, combinatorialcomplexity Plugin Analysis for dependenciesLoose (or too rigid) interface / dependencyspecifications Apply sampling techniques, test the ranges of plugin versionsToo many plugins, too many versions Crowdsourcing compatibility checks Study other plugin-in based ecosystems

    ×