This document provides an overview of integrating Solr with Hadoop for big data search capabilities. It discusses Lucene as the core search library that Solr is built on top of. It then covers ways Solr has been integrated with Hadoop, including putting the Solr index and transaction log directly in HDFS, running Solr on HDFS, and enabling Solr replication on HDFS. Other topics include using MapReduce for scalable index building, integrating Flume and HBase with Solr, and using Morphlines for extraction, transformation, and loading data into Solr.

1. 1
Solr%+%Hadoop%=%Big%Data%Search
Mark%Miller

2. 2
Who$Am$I?
Cloudera$employee,$Lucene/Solr$committer,$Lucene$PMC$member,
Apache$member
First$job$out$of$college$was$in$the$Newspaper$archiving$business.
First$full$time$employee$at$LucidWorks$G$a$startup$around
Lucene/Solr.
Spent$a$couple$years$as$“Core”$engineering$manager,$reporting$to
the$VP$of$engineering.

3. 3
Very%fast%and%feature%rich%‘core’%search%engine%library.%
Compact%and%powerful,%Lucene%is%an%extremely%popular%full>text
search%library.
Provides%low%level%API’s%for%analyzing,%indexing,%and%searching
text,%along%with%a%myriad%of%related%features.
Just%the%core%>%either%you%write%the%‘glue’%or%use%a%higher%level
search%engine%built%with%Lucene.

4. 4
Solr%(pronounced%"solar")%is%an%open%source%enterprise%search
platform%from%the%Apache%Lucene%project.%Its%major%features
include%full;text%search,%hit%highlighting,%faceted%search,%dynamic
clustering,%database%integration,%and%rich%document%(e.g.,%Word,
PDF)%handling.%Providing%distributed%search%and%index
replication,%Solr%is%highly%scalable.%Solr%is%the%most%popular
enterprise%search%engine.
;%Wikipedia

5. 5
Search'on'Hadoop'History
'Katta
'Blur
'SolBase
'HBASE73529
'SOLR71301
'SOLR71045
'Ad7Hoc
•
•
•
•
•
•
•

6. 6
Family'Tree
...

7. 7
Strengthen(the(Family(Bonds
No(need(to(build(something(radically(new(8(we(have(the
pieces(we(need.
Focus(on(integration(points.
Create(high(quality,(first(class(integrations(and(contribute
the(work(to(the(projects(involved.
Focus(on(integration(and(quality(first(8(then(performance
and(scale.
•
•
•
•

8. 8
SolrCloud

9. 9
Solr%Integration
Read%and%Write%directly%to%HDFS
First%Class%Custom%Directory%Support%in%Solr
Support%Solr%Replication%on%HDFS
Other%improvements%around%usability%and%configuration
•
•
•
•

10. 10
Read%and%Write%directly%to%HDFS
Lucene%did%not%historically%support%append%only%file%system
“Flexible%Indexing”%brought%around%support%for%append%only
filesystem%support
Lucene%support%append%only%filesystem%by%default%since%4.2
•
•
•

11. 11
Lucene&Directory&Abstraction
It’s&how&Lucene&interacts&with&index&files.
Solr&uses&the&Lucene&library&and&offers&DirectoryFactory
Class&Directory&{
&&&&&&&&listAll();
&&&&&&&&createOutput(file,&context);
&&&&&&&&openInput(file,&context);
&&&&&&&&deleteFile(file);
&&&&&&&&makeLock(file);
&&&&&&&&clearLock(file);
&&&&&&&&…

12. 12
Putting'the'Index'in'HDFS
Solr'relies'on'the'filesystem'cache'to'operate'at'full'speed.
HDFS'not'known'for'it’s'random'access'speed.
Apache'Blur'has'already'solved'this'with'an'HdfsDirectory'that
works'on'top'of'a'BlockDirectory.
The'“block'cache”'caches'the'hot'blocks'of'the'index'off'heap
(direct'byte'array)'and'takes'the'place'of'the'filesystem'cache.
We'contributed'back'optional'‘write’'caching.
•
•
•
•
•

13. 13
Putting'the'TransactionLog'in'HDFS
HdfsUpdateLog'added'9'extends'UpdateLog
Triggered'by'setting'the'UpdateLog'dataDir'to'something'that
starts'with'hdfs:/'9'no'additional'configuration'necessary.
Same'extensive'testing'as'used'on'UpdateLog
•
•
•

14. 14
Running&Solr&on&HDFS
Set&DirectoryFactory&to&HdfsDirectoryFactory&and&set&the&dataDir&to&a
location&in&hdfs.
Set&LockType&to&‘hdfs’
Use&an&UpdateLog&dataDir&location&that&begins&with&‘hdfs:/’
Or&java&FDsolr.directoryFactory=HdfsDirectoryFactory&
&&&&&&&&&&&&&&&FDsolr.lockType=solr.HdfsLockFactory
&&&&&&&&&&&&&&&FDsolr.updatelog=hdfs://host:port/path&Fjar&start.jar
•
•
•
•

15. 15
Solr%Replication%on%HDFS
While%Solr%has%exposed%a%plug8able%DirectoryFactory%for%a%long
time%now,%it%was%really%quite%limited.
Most%glaring,%only%a%local%file%system%based%Directory%would
work%with%replication.
There%where%also%other%more%minor%areas%that%relied%on%a%local
filesystem%Directory%implementation.
•
•
•

16. 16
Future&Solr&Replication&on&HDFS
Take&advantage&of&“distributed&filesystem”&and&allow&for
something&similar&to&HBase&regions.
If&a&node&goes&down,&the&data&is&still&available&in&HDFS&D&allow
for&that&index&to&be&automatically&served&by&a&node&that&is&still&up
if&it&has&the&capacity.
•
•
Solr&Node Solr&Node Solr&Node
HDFS

17. 17
MR#Index#Building
Scalable#index#creation#via#map8reduce
Many#initial#‘homegrown’#implementations#sent#documents#from#reducer#to
SolrCloud#over#http
To#really#scale,#you#want#the#reducers#to#create#the#indexes#in#HDFS#and
then#load#them#up#with#Solr
The#ideal#impl#will#allow#using#as#many#reducers#as#are#available#in#your
hadoop#cluster,#and#then#merge#the#indexes#down#to#the#correct#number#of
‘shards’
•
•
•
•

18. 18
MR#Index#Building
Mapper:
Parse#input#into
indexable#document
Mapper:
Parse#input#into
indexable#document
Mapper:
Parse#input#into
indexable#document
Index#shard
1
Index#shard
2
Arbitrary#reducing#steps#of#indexing#and#merging
End@Reducer#(shard#1):
Index#document
End@Reducer#(shard#2):
Index#document

19. 19
SolrCloud(Aware
Can(‘inspect’(ZooKeeper(to(learn(about(Solr(cluster.
What(URL’s(to(GoLive(to.
The(Schema(to(use(when(building(indexes.
Match(hash(E>(shard(assignments(of(a(Solr(cluster.
•
•
•
•

20. 20
GoLive
After+building+your+indexes+with+map:reduce,+how+do+you
deploy+them+to+your+Solr+cluster?
We+want+it+to+be+easy+:+so+we+built+the+GoLive+option.
GoLive+allows+you+to+easily+merge+the+indexes+you+have
created+atomically+into+a+live+running+Solr+cluster.
Paired+with+the+ZooKeeper+Aware+ability,+this+allows+you+to
simply+point+your+map:reduce+job+to+your+Solr+cluster+and+it+will
automatically+discover+how+many+shards+to+build+and+what
locations+to+deliver+the+final+indexes+to+in+HDFS.
•
•
•
•

21. 21
Flume&Solr&Sync
Flume&is&a&distributed,&reliable,&and&available&service&for
efficiently&collecting,&aggregating,&and&moving&large&amounts
of&log&data.&It&has&a&simple&and&flexible&architecture&based&on
streaming&data&flows.&It&is&robust&and&fault&tolerant&with
tunable&reliability&mechanisms&and&many&failover&and
recovery&mechanisms.&It&uses&a&simple&extensible&data&model
that&allows&for&online&analytic&application.
=&Apache&Flume&Website

22. Other
Logs
22
Flume.Solr.Sync
HDFS
Flume
Agent
Flume
Agent
Solr

23. 23
SolrCloud(Aware
Can(‘inspect’(ZooKeeper(to(learn(about(Solr(cluster.
What(URL’s(to(send(data(to.
The(Schema(for(the(collection(being(indexed(to.
•
•
•

24. 24
HBase&Integration
Collaboration&between&NGData&&&Cloudera
NGData&are&creators&of&the&Lily&data&management&platform
Lily&HBase&Indexer
Service&which&acts&as&a&HBase&replication&listener
HBase&replication&features,&such&as&filtering,&supported
Replication&updates&trigger&indexing&of&updates&(rows)
Integrates&Morphlines&library&for&ETL&of&rows
AL2&licensed&on&github&https://github.com/ngdata
•
•
•
•
•
•
•
•

25. 25
HBase&Integration
HDFS
HBase
interactive&load
Indexer(s)
Triggers&on&updates
Solr&server
Solr&server
Solr&server
Solr&server
Solr&server

26. 26
Morphlines
A,morphline,is,a,configuration,file,that,allows,you,to,define,ETL
transformation,pipelines
Extract,content,from,input,files,,transform,content,,load,content,(eg
to,Solr)
Uses,Tika,to,extract,content,from,a,large,variety,of,input,documents
Part,of,the,CDK,(Cloudera,Development,Kit)
•
•
•
•

27. 27
Morphlines
syslog
Flume
Agent
Solr3Sink
Command:3readLine
Command:3grok
Command:3loadSolr
Solr
3Open3Source3framework3for3simple3ETL
3Ships3as3part3Cloudera3Developer3Kit3(CDK)
3It’s3a3Java3library
3AL23licensed3on3github
https://github.com/cloudera/cdk
3Similar3to3Unix3pipelines
3Configuration3over3coding
3Supports3common3Hadoop3formats
Avro
Sequence3file
Text
•
•
•
•
•
•
•

28. 28
Morphlines
+Integrate+with+and+load+into+Apache+Solr
+Flexible+log+file+analysis
+Single:line+record,+multi:line+records,+CSV+files+
+Regex+based+pattern+matching+and+extraction+
+Integration+with+Avro+
+Integration+with+Apache+Hadoop+Sequence+Files
+Integration+with+SolrCell+and+all+Apache+Tika+parsers+
+Auto:detection+of+MIME+types+from+binary+data+using+++Apache+Tika
•
•
•
•
•
•
•
•

29. 29
Morphlines
+Scripting+support+for+dynamic+java+code+
+Operations+on+fields+for+assignment+and+comparison
+Operations+on+fields+with+list+and+set+semantics+
+if:then:else+conditionals+
+A+small+rules+engine+(tryRules)
+String+and+timestamp+conversions+
+slf4j+logging
+Yammer+metrics+and+counters+
+Decompression+and+unpacking+of+arbitrarily+nested+container+file
formats
•
•
•
•
•
•
•
•
•

30. 30
Morphlines+Example+Config
morphlines+:+[
+{
+++id+:+morphline1
+++importCommands+:+["com.cloudera.**",+"org.apache.solr.**"]
+++commands+:+[
+++++{+readLine+{}+}++++++++++++++++++++
+++++{+
+++++++grok+{+
+++++++++dictionaryFiles+:+[/tmp/grokFdictionaries]+++++++++++++++++++++++++++++++
+++++++++expressions+:+{+
+++++++++++message+:+"""<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp}+%
{SYSLOGHOST:syslog_hostname}+%{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:+%
{GREEDYDATA:syslog_message}"""
+++++++++}
+++++++}
+++++}
+++++{+loadSolr+{}+}+++++
++++]
+}
]
Example(Input
<164>Feb++4+10:46:14+syslog+sshd[607]:+listening+on+0.0.0.0+port+22
Output(Record
syslog_pri:164
syslog_timestamp:Feb++4+10:46:14
syslog_hostname:syslog
syslog_program:sshd
syslog_pid:607
syslog_message:listening+on+0.0.0.0+port+22.

31. 31
Hue$Integration
Hue
Simple$UI
Navigated,$faceted$drill$down
Customizable$display
Full$text$search,$standard$Solr
API$and$query$language
•
•
•
•

32. 32
Cloudera)Search
https://ccp.cloudera.com/display/SUPPORT/Downloads
Or)Google
“cloudera=search=download”

33. Mark%Miller,%Cloudera @heismark