1.
Solr’s missing plugin ecosystem
Jan Høydahl
Cominvent AS

2.
Agenda
• Intro
• Solr plugins today
• The improved plugin system
• Technical overview
• Recap
• Q & A
All images used are licensed for commercial re-use

3.
3
• Jan Høydahl
• Norway, 45 years
• Cominvent AS
• Lucene/Solr committer & PMC
Telecom Fast ESP Solr Freelancer
1995 2000 2007 2017
17 years of Search

4.
4

5.
5
«The Monolith»

6.
«Technology ecosystems are product
platforms defined by core components
made by the platform owner and
complemented by applications made by
autonomous companies in the periphery»
-ft.com/lexicon
Ecosystem

7.
7
The Solr Ecosystem
developer
user
isv

8.
Agenda
• Intro
• Solr plugins today
• What is a plugin
• Loading, distribution
• User poll
• The improved plugin system
• Technical overview
• Recap
• Q & A

9.
Java classes pre-defined by
Solr for extension
ZIP/JAR providing a
certain feature
Plugin - two levels

10.
• Examples:
• QueryParser
• UpdateProcessor
• SearchComponent
• ResponseWriter
• TokenFilter
• …
10
What is a Solr plugin?

11.
dataimport
11
Solr contribs
• Optional features bundled as separate JARs
• Released with Solr in the dist and contrib folders
• May contain one or more Java-level plugins
velocity

12.
12
How are plugins loaded?
CLASSPATH SOLR_HOME/lib
WEB-INF/lib
<lib dir="${solr.install.dir}/contrib/velocity/lib" regex=".*.jar" />
<lib dir="${solr.install.dir}/dist/" regex="solr-velocity-d.*.jar" />
<requestHandler name="/browse" class="solr.SearchHandler" … />
Class Loader
ResourceLoader

13.
13

14.
14
Plugin Distribution
http://solr.cool

15.
15

16.
16
Who are you?

17.
17
Pluginusage

18.
18
Rate the following statements
Solr plugins are easy to discover
Solr plugins are easy to install
Solr plugins are easy to configure
Solr plugins are easy to upgrade
It is easy to match plugin to Solr version
I find a plugin for all my needs
It is easy to develop my own plugin
It is easy to distribute my own plugins

19.
19
Top-3 priority features

20.
20
Questions to plugin developers
We need a Solr Plugin SDK
All plugins belong in Apache GIT
Github is a better home for Solr plugins
Commercial plugins in the official repo?
The current plugin types are sufficient
Other languages than Java

21.
Agenda
• Intro
• Solr plugins today
• The improved plugin system
• POC
• The end user perspective
• Deverloper perspective
• Solr product perspective
• Technical overview
• Recap
• Q & A

22.
23
Proof of concept
• Integrated PF4J in Solr
• Design document
• SOLR-10665 with patch
and a download to try
• CLI (bin/solr plugin)
• Plugin repositories

23.
24
Design goals
• User friendly and familiar
• Establish a plugin ecosystem
• Spark further modularisation
• Extensible and customisable

24.
25
The end user perspective
Discover plugin Install plugin Stay up to dateDownload Solr
«Users download a very slim core without any contribs, huge analysers
or esoteric query parsers. A few megs…»

25.
26
Downloading Solr
• 65% size reduction
140 —> 50 Mb*
• From 1min —> 20sec
• Escape the «kitchen sink»
• Room for more shaving!
Core Contribs Test-fw Kuromoji HDFS *After converting contribs etc to plugins

26.
27
The end user experience
Discover plugin Install plugin Stay up to dateDownload Solr
«A new plugins tab in the Admin UI lists available official plugins as
well as community plugins, installable with a single click.»

27.
28
Plugin Discovery

28.
29
PluginInstall

29.
30
The end user experience
Stay up to dateDiscover plugin Install pluginDownload Solr
«You will with a single command upgrade to the latest version that is
compatible with your Solr version»

30.
31
CLI Usage - upgrade

31.
32
Developer perspective
App Store
advantage
Simpler
documentation
Safe
upgrades
Free/
Premium
Visibility
Distribution
bin/solr plugin
install foo
Version match
SHA checksum
PGP signature
Paid plugins
Try before buy

32.
33
product/project perspective
Kitchen sink? Attract developers Modularity Licensing issues
Smaller core
Faster downloads
Sense of
community
New ext. points
Break out features
JIRA patch -> plugin
GPL dependencies
User decides

33.
Agenda
• Intro
• Solr plugins today
• The improved plugin system
• Technical overview
• Choice of plugin framework
• The new plugin bundle
• Plugin repositories
• Challenges
• Recap
• Q & A

34.
35
Choice of plugin framework
Heavy weight, intrusive
Jigsaw solves a different problem
Not yet released
Lightweight and customisable
Provides version, dependencies & repository

35.
36
PF4J (Plugin Framework for Java)
«PF4J is an open
source, lightweight
plugin framework for
Java, with minimal
dependencies and
very extensible»
www.pf4j.org

36.
37
The plugin bundle
• Defined by PF4J
• Simply «a JAR with a MANIFEST»
• Or a ZIP with MANIFEST & dependencies
• Lives in SOLR_HOME/plugins
Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: decebal
Build-Jdk: 1.6.0_17
Plugin-Class: com.example.MyPlugin
Plugin-Dependencies: x, y, z
Plugin-Id: my-plugin
Plugin-Provider: John Doe
Plugin-Version: 0.0.1
MANIFEST.mf
<dependency org="org.pf4j" name="pf4j" rev="..." />

37.
38
Example contrib --> plugin ZIP
analysis-extras-8.0.0.zip

38.
39

39.
40
Bootstrapping

40.
Bootstrapping, class loading
SolrResourceLoader
ClassLoader
- lib/
- WEB-INF/lib/
- <lib/>
before

41.
Bootstrapping, class loading
SolrResourceLoader
ClassLoader
- lib/
- WEB-INF/lib/
- <lib/>
PluginBundleClassLoader
PluginBundleManager
P1 P2 P3
1
2
findClass()
(parent first) parent
(parent last)
after

42.
43
Plugin repositories
• -
• Any static website or folder
• Each repo defines a plugins.json file
<dependency org="org.pf4j" name="pf4j-update" rev="..." />
Solr version requirement

43.
44
Supported repositories
dist.apache.org
+mirrors
solr-plugins/free
solr-plugins/paid
local
solr/plugins
• Official Apache
• Community
moderated
• User added repos
• Local file shares
• Manually installed
some.site
org/repo
/mnt/solr/repo

44.
45
The Apache dist repo
dist.apache.org
+mirrors
• Custom PluginUpdateRepository
• (1) closest mirror for exact version
• (2) apache.org/dist
• (3) archive.apache.org/dist
• Validates SHA sum and PGP sig

45.
46
Needs more research
• Multi version of dependency jars
• Distributed install across cluster
• Utilize runtimeLib / blob store?
• Plugin upgrade requires restart
• How to let plugins add UI menus

46.
47
Future possibilities
• Collection declares plugin dependency
<dependency>my-plugin</dependency>
• Slim-JAR +
• Auto load more components using SPI
• Plugin SDK with samples
pexels.com/photo/blackboard-business-chalkboard-concept-355988/CC0

47.
48
Next steps
• Break up POC into smaller JIRAs
• Commit core support to master
• Split out a few contribs as plugins
• Release 7.x with plugin support
• Split out more features as plugins
• Currency FieldType
• Excel response writer
• Kuromoji japanese analyzer
• Test-framework?
pixabay.com/p-768642/Publicdomain

48.
Agenda
• Intro
• Solr plugins today
• The improved plugin system
• Technical overview
• Recap
• Q & A

49.
50
Recap
• Already modular
• «Kitchen sink»
• Hard to locate 3rd party plugins
• integration introduces:
• a well defined plugin bundle
• tooling to work with plugins
• an «App Store» approach to distribution
• A larger ecosystem!

50.
51
Jan Høydahl
Thank You!
@cominvent
janhoy
SOLR-10665

51.
Thank You

52.
Extra slides

53.
54
How pluggable are we - really?

54.
55
Do you use Solr plugins today?

55.
56
01
Preferred packaging format

56.
57
Top-3 priority features (contd)
Not prioritised by anyone:
• A "plugin store" to purchase
commercial Solr plugins
• Plugins must be based on OSGI
• Plugins must be Jigsaw modules

57.
58
«All plugins that I used are custom,because [there] doesn’t existanything that fit the needs, or I didn'tfind the plugin at research time»
«From a plugin-developer point of
view the greatest pain point is
keeping up with changes in
Lucene, occasionally even interface
changes between bug fix releases»
«Worst experience is to
always reinvent the wheel
for plugins after API
change in Lucene/Solr»

58.
59
Plugin lifecycle
CREATED The manager has loaded the ZIP/JAR
DISABLED A plugin that will not be started.
May e.g. be do to Solr<—>Plugin version mismatch
STARTED Active plugin state, contributes extensions
STOPPED A plugin can be explicitly stopped, or implicitly
during upgrading (unload old, load new)

59.
60
Implicit loading?
• Lucene Analysis plugins loaded with SPI
• Enables use of name, e.g synonym instead of class name
org.apache.lucene.analysis.synonym.SynonymFilterFactory
• Not exposed by Solr :-(
• Some UpdateProcessors are pre-initialized (7.0)
• Reference by well-defined name
• Statically defined, not for 3rd party URPs
• All others need to be explicitly configured