1. Document Search
Optimization using
Lucene API
Harsha Ummerpillai (harsha07@msu.edu)
Shyam Gedela (gedela@msu.edu)
Michigan State University
11/13/2014
1

2. During KD 2013 we presented MSU’s approach to optimizing
and improving performance of Rice Document search using
Lucene API. We are back this year to share the lessons learned
and results of our implementation.
Background
2

3. Kuali Days 2014
Indianapolis
Introduction
• Background
• Goals for Lucene implementation
• Technical Recap
• Implementation
• Performance Results
• How to
• Demo
3

4. Kuali Days 2014
Indianapolis
Background
• Document Search - Why is it important
• MSU implementation
– Go Live - Jan 1 2011
• Rice 2.1.9
• KFS 5.0.x
• KMM 1.x
• OOI 1.x
– ~4 yrs of operation
– ~4 million documents
– ~50 million search attributes
4

5. Kuali Days 2014
Indianapolis
Goals
• Goals for Lucene implementation
– Fast – Improved and consistent response
times
– Configurable – can be enabled/disabled
using configuration
– Seamless – No change to user screen
– Scalable
– Customizable
5

6. Technical Details Recap
6

7. Kuali Days 2014
Indianapolis
Document search
7
• Client applications define searchable attributes in Data
Dictionary.
• Rice extracts and builds index, saving key value pairs into
DB.
• Attributes saved into 4 different tables based on data types.
• Existing structure – 1 document to n indexed records
• Standard searchable fields
– Status codes
– Initiator
– Approver
– Action dates.
• Custom attributes defined by document types

8. Kuali Days 2014
Indianapolis
Document search
8

9. Kuali Days 2014
Indianapolis
Rice Doc Search
9

10. Kuali Days 2014
Indianapolis
Rice Doc Index w Lucene
10

11. Kuali Days 2014
Indianapolis
Rice Doc Index w Lucene
11

12. Key aspects of the implementation
Implementation
12

13. Kuali Days 2014
Indianapolis
Technical features
• Documents are queued for Lucene indexing with four
separate stages
– WAIT_FOR_REALTIME("0"),
– READY_FOR_REALTIME("1"),
– WAIT_FOR_MASTER("2"),
– READY_FOR_MASTER("3")
• Two indexes; master and real-time
• Master refreshed 3 times a day
• Real-time index refreshed every 5 seconds
• Single master node writes index to shared file storage
13

14. Kuali Days 2014
Indianapolis
Auto warming
14
Index Readers are auto warmed and queued in all nodes

15. Kuali Days 2014
Indianapolis
Index Storage
15
• Directory structure within Lucene Index store
• temp: Storage location before merge into active index
• meta-info: Index stats and message files

16. Results
16

17. Kuali Days 2014
Indianapolis
Performance Test Scenarios
• 7 business scenarios
• Invaluable for daily operations
– E.g. how many payment requests are
department approved but have not been
extracted by PDP (Vendors not paid)
17

18. Kuali Days 2014
Indianapolis
Performance Charts - Comparison
18
0
50000
100000
150000
200000
250000
300000
350000
ACCT Approver PO REQS PCDO PREQ CM
No Lucene
Lucene

19. We have created an open contribution JIRA CONTRIB-95 and
happy to provide latest fixes and patches from our production.
How To
19

20. Kuali Days 2014
Indianapolis
How to guide
• Visit contribution JIRA https://jira.kuali.org/browse/CONTRIB-95
• Download and apply the patch file to rice (base version - 2.1.9)
workspace.
• Add Lucene configuration properties to rice application configuration file.
• Setup shared file store location where index will be saved and shared.
• Add Lucene index queue table using lucene-setup.sql
• Build and start rice application with Lucene configuration enabled
• Visit “Administration > Lucene Administration “ click “Build Master Index”
• Click refresh link to see the status, when index.ready file is listed master
index is ready for use.
• Create a document and see if it is available in search, if real time indexer
is working new document should appear in search results within 5~10
seconds.
• Use administration page to see the latest status and manage the index.
20

21. Technical – Admin page
21

22. Kuali Days 2014
Indianapolis
References
• Lucene http://lucene.apache.org/core/
• KD 2013 Presentation
https://jira.kuali.org/secure/attachment/77886/KD-2013-Optimizing-
Document-Search-using-Lucene.pptx
• CONTRIB-95 https://jira.kuali.org/browse/CONTRIB-95
22