Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Solr's missing plugin ecosystem

518 views

Published on

My talk at Lucene/Solr Revolution 2017, Las Vegas

The improved plugin system being proposed in this talk utilizes PF4J to add bundle packaging (zip/jar), plugin discovery (repositories), one-line install/upgrade and automatic version compatibility checks. Think of it as Homebrew or Apt-Get for Solr :) The hope is that this will encourage hundreds of new plugins being created and thus give Solr developers a sense of community and a new “stage” to perform on.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Solr's missing plugin ecosystem

  1. 1. Solr’s missing plugin ecosystem Jan Høydahl Cominvent AS
  2. 2. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Recap • Q & A All images used are licensed for commercial re-use
  3. 3. 3 • Jan Høydahl • Norway, 45 years • Cominvent AS • Lucene/Solr committer & PMC Telecom Fast ESP Solr Freelancer 1995 2000 2007 2017 17 years of Search
  4. 4. 4
  5. 5. 5 «The Monolith»
  6. 6. «Technology ecosystems are product platforms defined by core components made by the platform owner and complemented by applications made by autonomous companies in the periphery» -ft.com/lexicon Ecosystem
  7. 7. 7 The Solr Ecosystem developer user isv
  8. 8. Agenda • Intro • Solr plugins today • What is a plugin • Loading, distribution • User poll • The improved plugin system • Technical overview • Recap • Q & A
  9. 9. Java classes pre-defined by 
 Solr for extension
 ZIP/JAR providing a
 certain feature Plugin - two levels
  10. 10. • Examples: • QueryParser • UpdateProcessor • SearchComponent • ResponseWriter • TokenFilter • … 10 What is a Solr plugin?
  11. 11. dataimport 11 Solr contribs • Optional features bundled as separate JARs • Released with Solr in the dist and contrib folders • May contain one or more Java-level plugins velocity
  12. 12. 12 How are plugins loaded? CLASSPATH SOLR_HOME/lib WEB-INF/lib <lib dir="${solr.install.dir}/contrib/velocity/lib" regex=".*.jar" />
 <lib dir="${solr.install.dir}/dist/" regex="solr-velocity-d.*.jar" />
 <requestHandler name="/browse" class="solr.SearchHandler" … /> Class Loader ResourceLoader
  13. 13. 13
  14. 14. 14 Plugin Distribution http://solr.cool
  15. 15. 15
  16. 16. 16 Who are you?
  17. 17. 17 Pluginusage
  18. 18. 18 Rate the following statements Solr plugins are easy to discover Solr plugins are easy to install Solr plugins are easy to configure Solr plugins are easy to upgrade It is easy to match plugin to Solr version I find a plugin for all my needs It is easy to develop my own plugin It is easy to distribute my own plugins
  19. 19. 19 Top-3 priority features
  20. 20. 20 Questions to plugin developers We need a Solr Plugin SDK All plugins belong in Apache GIT Github is a better home for Solr plugins Commercial plugins in the official repo? The current plugin types are sufficient Other languages than Java
  21. 21. Agenda • Intro • Solr plugins today • The improved plugin system • POC • The end user perspective • Deverloper perspective • Solr product perspective • Technical overview • Recap • Q & A
  22. 22. 23 Proof of concept • Integrated PF4J in Solr • Design document • SOLR-10665 with patch
 and a download to try • CLI (bin/solr plugin) • Plugin repositories
  23. 23. 24 Design goals • User friendly and familiar • Establish a plugin ecosystem • Spark further modularisation • Extensible and customisable
  24. 24. 25 The end user perspective Discover plugin Install plugin Stay up to dateDownload Solr «Users download a very slim core without any contribs, huge analysers or esoteric query parsers. A few megs…»
  25. 25. 26 Downloading Solr • 65% size reduction
 140 —> 50 Mb* • From 1min —> 20sec • Escape the «kitchen sink» • Room for more shaving! Core Contribs Test-fw Kuromoji HDFS *After converting contribs etc to plugins
  26. 26. 27 The end user experience Discover plugin Install plugin Stay up to dateDownload Solr «A new plugins tab in the Admin UI lists available official plugins as well as community plugins, installable with a single click.»
  27. 27. 28 Plugin Discovery
  28. 28. 29 PluginInstall
  29. 29. 30 The end user experience Stay up to dateDiscover plugin Install pluginDownload Solr «You will with a single command upgrade to the latest version that is compatible with your Solr version»
  30. 30. 31 CLI Usage - upgrade
  31. 31. 32 Developer perspective App Store
 advantage Simpler
 documentation Safe
 upgrades Free/
 Premium Visibility Distribution bin/solr plugin 
 install foo Version match SHA checksum PGP signature Paid plugins Try before buy
  32. 32. 33 product/project perspective Kitchen sink? Attract developers Modularity Licensing issues Smaller core Faster downloads Sense of
 community New ext. points Break out features JIRA patch -> plugin GPL dependencies User decides
  33. 33. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Choice of plugin framework • The new plugin bundle • Plugin repositories • Challenges • Recap • Q & A
  34. 34. 35 Choice of plugin framework Heavy weight, intrusive Jigsaw solves a different problem Not yet released Lightweight and customisable Provides version, dependencies & repository
  35. 35. 36 PF4J (Plugin Framework for Java) «PF4J is an open source, lightweight plugin framework for Java, with minimal dependencies and very extensible» www.pf4j.org
  36. 36. 37 The plugin bundle • Defined by PF4J
 • Simply «a JAR with a MANIFEST» • Or a ZIP with MANIFEST & dependencies • Lives in SOLR_HOME/plugins Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: decebal Build-Jdk: 1.6.0_17 Plugin-Class: com.example.MyPlugin Plugin-Dependencies: x, y, z Plugin-Id: my-plugin Plugin-Provider: John Doe Plugin-Version: 0.0.1 MANIFEST.mf <dependency org="org.pf4j" name="pf4j" rev="..." />
  37. 37. 38 Example contrib --> plugin ZIP analysis-extras-8.0.0.zip
  38. 38. 39
  39. 39. 40 Bootstrapping
  40. 40. Bootstrapping, class loading SolrResourceLoader ClassLoader - lib/ - WEB-INF/lib/ - <lib/> before
  41. 41. Bootstrapping, class loading SolrResourceLoader ClassLoader - lib/ - WEB-INF/lib/ - <lib/> PluginBundleClassLoader PluginBundleManager P1 P2 P3 1 2 findClass() (parent first) parent (parent last) after
  42. 42. 43 Plugin repositories • - • Any static website or folder • Each repo defines a plugins.json file <dependency org="org.pf4j" name="pf4j-update" rev="..." /> Solr version requirement
  43. 43. 44 Supported repositories dist.apache.org
 +mirrors solr-plugins/free
 solr-plugins/paid local solr/plugins • Official Apache
 • Community moderated
 • User added repos
 • Local file shares
 • Manually installed some.site org/repo /mnt/solr/repo
  44. 44. 45 The Apache dist repo dist.apache.org
 +mirrors • Custom PluginUpdateRepository • (1) closest mirror for exact version • (2) apache.org/dist • (3) archive.apache.org/dist • Validates SHA sum and PGP sig
  45. 45. 46 Needs more research • Multi version of dependency jars
 • Distributed install across cluster
 • Utilize runtimeLib / blob store?
 • Plugin upgrade requires restart
 • How to let plugins add UI menus
  46. 46. 47 Future possibilities • Collection declares plugin dependency
 <dependency>my-plugin</dependency>
 • Slim-JAR + 
 • Auto load more components using SPI
 • Plugin SDK with samples pexels.com/photo/blackboard-business-chalkboard-concept-355988/CC0
  47. 47. 48 Next steps • Break up POC into smaller JIRAs • Commit core support to master • Split out a few contribs as plugins • Release 7.x with plugin support • Split out more features as plugins • Currency FieldType • Excel response writer • Kuromoji japanese analyzer • Test-framework? pixabay.com/p-768642/Publicdomain
  48. 48. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Recap • Q & A
  49. 49. 50 Recap • Already modular • «Kitchen sink» • Hard to locate 3rd party plugins • integration introduces: • a well defined plugin bundle • tooling to work with plugins • an «App Store» approach to distribution • A larger ecosystem!
  50. 50. 51 Jan Høydahl Thank You! @cominvent janhoy SOLR-10665
  51. 51. Thank You
  52. 52. Extra slides
  53. 53. 54 How pluggable are we - really?
  54. 54. 55 Do you use Solr plugins today?
  55. 55. 56 01 Preferred packaging format
  56. 56. 57 Top-3 priority features (contd) Not prioritised by anyone: • A "plugin store" to purchase commercial Solr plugins • Plugins must be based on OSGI • Plugins must be Jigsaw modules
  57. 57. 58 «All plugins that I used are custom,because [there] doesn’t existanything that fit the needs, or I didn'tfind the plugin at research time» «From a plugin-developer point of view the greatest pain point is keeping up with changes in Lucene, occasionally even interface changes between bug fix releases» «Worst experience is to always reinvent the wheel for plugins after API change in Lucene/Solr»
  58. 58. 59 Plugin lifecycle CREATED The manager has loaded the ZIP/JAR DISABLED A plugin that will not be started. May e.g. be do to Solr<—>Plugin version mismatch STARTED Active plugin state, contributes extensions STOPPED A plugin can be explicitly stopped, or implicitly during upgrading (unload old, load new)
  59. 59. 60 Implicit loading? • Lucene Analysis plugins loaded with SPI • Enables use of name, e.g synonym instead of class name org.apache.lucene.analysis.synonym.SynonymFilterFactory • Not exposed by Solr :-( • Some UpdateProcessors are pre-initialized (7.0) • Reference by well-defined name • Statically defined, not for 3rd party URPs • All others need to be explicitly configured

×