Solr's missing plugin ecosystem

1. Solr’s missing plugin ecosystem Jan Høydahl Cominvent AS

2. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Recap • Q & A All images used are licensed for commercial re-use

3. 3 • Jan Høydahl • Norway, 45 years • Cominvent AS • Lucene/Solr committer & PMC Telecom Fast ESP Solr Freelancer 1995 2000 2007 2017 17 years of Search

5. 5 «The Monolith»

6. «Technology ecosystems are product platforms deﬁned by core components made by the platform owner and complemented by applications made by autonomous companies in the periphery» -ft.com/lexicon Ecosystem

7. 7 The Solr Ecosystem developer user isv

8. Agenda • Intro • Solr plugins today • What is a plugin • Loading, distribution • User poll • The improved plugin system • Technical overview • Recap • Q & A

9. Java classes pre-deﬁned by   Solr for extension  ZIP/JAR providing a  certain feature Plugin - two levels

10. • Examples: • QueryParser • UpdateProcessor • SearchComponent • ResponseWriter • TokenFilter • … 10 What is a Solr plugin?

11. dataimport 11 Solr contribs • Optional features bundled as separate JARs • Released with Solr in the dist and contrib folders • May contain one or more Java-level plugins velocity

12. 12 How are plugins loaded? CLASSPATH SOLR_HOME/lib WEB-INF/lib <lib dir="${solr.install.dir}/contrib/velocity/lib" regex=".*.jar" />  <lib dir="${solr.install.dir}/dist/" regex="solr-velocity-d.*.jar" />  <requestHandler name="/browse" class="solr.SearchHandler" … /> Class Loader ResourceLoader

13. 13

14. 14 Plugin Distribution http://solr.cool

15. 15

16. 16 Who are you?

17. 17 Pluginusage

18. 18 Rate the following statements Solr plugins are easy to discover Solr plugins are easy to install Solr plugins are easy to configure Solr plugins are easy to upgrade It is easy to match plugin to Solr version I find a plugin for all my needs It is easy to develop my own plugin It is easy to distribute my own plugins

19. 19 Top-3 priority features

20. 20 Questions to plugin developers We need a Solr Plugin SDK All plugins belong in Apache GIT Github is a better home for Solr plugins Commercial plugins in the official repo? The current plugin types are sufficient Other languages than Java

21. Agenda • Intro • Solr plugins today • The improved plugin system • POC • The end user perspective • Deverloper perspective • Solr product perspective • Technical overview • Recap • Q & A

22. 23 Proof of concept • Integrated PF4J in Solr • Design document • SOLR-10665 with patch  and a download to try • CLI (bin/solr plugin) • Plugin repositories

23. 24 Design goals • User friendly and familiar • Establish a plugin ecosystem • Spark further modularisation • Extensible and customisable

24. 25 The end user perspective Discover plugin Install plugin Stay up to dateDownload Solr «Users download a very slim core without any contribs, huge analysers or esoteric query parsers. A few megs…»

25. 26 Downloading Solr • 65% size reduction  140 —> 50 Mb* • From 1min —> 20sec • Escape the «kitchen sink» • Room for more shaving! Core Contribs Test-fw Kuromoji HDFS *After converting contribs etc to plugins

26. 27 The end user experience Discover plugin Install plugin Stay up to dateDownload Solr «A new plugins tab in the Admin UI lists available ofﬁcial plugins as well as community plugins, installable with a single click.»

27. 28 Plugin Discovery

28. 29 PluginInstall

29. 30 The end user experience Stay up to dateDiscover plugin Install pluginDownload Solr «You will with a single command upgrade to the latest version that is compatible with your Solr version»

30. 31 CLI Usage - upgrade

31. 32 Developer perspective App Store  advantage Simpler  documentation Safe  upgrades Free/  Premium Visibility Distribution bin/solr plugin   install foo Version match SHA checksum PGP signature Paid plugins Try before buy

32. 33 product/project perspective Kitchen sink? Attract developers Modularity Licensing issues Smaller core Faster downloads Sense of  community New ext. points Break out features JIRA patch -> plugin GPL dependencies User decides

33. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Choice of plugin framework • The new plugin bundle • Plugin repositories • Challenges • Recap • Q & A

34. 35 Choice of plugin framework Heavy weight, intrusive Jigsaw solves a different problem Not yet released Lightweight and customisable Provides version, dependencies & repository

35. 36 PF4J (Plugin Framework for Java) «PF4J is an open source, lightweight plugin framework for Java, with minimal dependencies and very extensible» www.pf4j.org

36. 37 The plugin bundle • Deﬁned by PF4J  • Simply «a JAR with a MANIFEST» • Or a ZIP with MANIFEST & dependencies • Lives in SOLR_HOME/plugins Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: decebal Build-Jdk: 1.6.0_17 Plugin-Class: com.example.MyPlugin Plugin-Dependencies: x, y, z Plugin-Id: my-plugin Plugin-Provider: John Doe Plugin-Version: 0.0.1 MANIFEST.mf <dependency org="org.pf4j" name="pf4j" rev="..." />

37. 38 Example contrib --> plugin ZIP analysis-extras-8.0.0.zip

38. 39

39. 40 Bootstrapping

40. Bootstrapping, class loading SolrResourceLoader ClassLoader - lib/ - WEB-INF/lib/ - <lib/> before

41. Bootstrapping, class loading SolrResourceLoader ClassLoader - lib/ - WEB-INF/lib/ - <lib/> PluginBundleClassLoader PluginBundleManager P1 P2 P3 1 2 findClass() (parent ﬁrst) parent (parent last) after

42. 43 Plugin repositories • - • Any static website or folder • Each repo deﬁnes a plugins.json ﬁle <dependency org="org.pf4j" name="pf4j-update" rev="..." /> Solr version requirement

43. 44 Supported repositories dist.apache.org  +mirrors solr-plugins/free  solr-plugins/paid local solr/plugins • Ofﬁcial Apache  • Community moderated  • User added repos  • Local ﬁle shares  • Manually installed some.site org/repo /mnt/solr/repo

44. 45 The Apache dist repo dist.apache.org  +mirrors • Custom PluginUpdateRepository • (1) closest mirror for exact version • (2) apache.org/dist • (3) archive.apache.org/dist • Validates SHA sum and PGP sig

45. 46 Needs more research • Multi version of dependency jars  • Distributed install across cluster  • Utilize runtimeLib / blob store?  • Plugin upgrade requires restart  • How to let plugins add UI menus

46. 47 Future possibilities • Collection declares plugin dependency  <dependency>my-plugin</dependency>  • Slim-JAR +   • Auto load more components using SPI  • Plugin SDK with samples pexels.com/photo/blackboard-business-chalkboard-concept-355988/CC0

47. 48 Next steps • Break up POC into smaller JIRAs • Commit core support to master • Split out a few contribs as plugins • Release 7.x with plugin support • Split out more features as plugins • Currency FieldType • Excel response writer • Kuromoji japanese analyzer • Test-framework? pixabay.com/p-768642/Publicdomain

48. Agenda • Intro • Solr plugins today • The improved plugin system • Technical overview • Recap • Q & A

49. 50 Recap • Already modular • «Kitchen sink» • Hard to locate 3rd party plugins • integration introduces: • a well deﬁned plugin bundle • tooling to work with plugins • an «App Store» approach to distribution • A larger ecosystem!

50. 51 Jan Høydahl Thank You! @cominvent janhoy SOLR-10665

51. Thank You

52. Extra slides

53. 54 How pluggable are we - really?

54. 55 Do you use Solr plugins today?

55. 56 01 Preferred packaging format

56. 57 Top-3 priority features (contd) Not prioritised by anyone: • A "plugin store" to purchase commercial Solr plugins • Plugins must be based on OSGI • Plugins must be Jigsaw modules

57. 58 «All plugins that I used are custom,because [there] doesn’t existanything that fit the needs, or I didn'tfind the plugin at research time» «From a plugin-developer point of view the greatest pain point is keeping up with changes in Lucene, occasionally even interface changes between bug fix releases» «Worst experience is to always reinvent the wheel for plugins after API change in Lucene/Solr»

58. 59 Plugin lifecycle CREATED The manager has loaded the ZIP/JAR DISABLED A plugin that will not be started. May e.g. be do to Solr<—>Plugin version mismatch STARTED Active plugin state, contributes extensions STOPPED A plugin can be explicitly stopped, or implicitly during upgrading (unload old, load new)

59. 60 Implicit loading? • Lucene Analysis plugins loaded with SPI • Enables use of name, e.g synonym instead of class name org.apache.lucene.analysis.synonym.SynonymFilterFactory • Not exposed by Solr :-( • Some UpdateProcessors are pre-initialized (7.0) • Reference by well-defined name • Statically defined, not for 3rd party URPs • All others need to be explicitly configured