In this slides, I propose a solution for fast prototyping of search engine back end API. It consists of Linux + Django + Solr + Python (LDSP), and all are open source softwares. The solution also provides code repository with automation scripts. Everyone can build a Search Engine back end API in seconds by exploiting LDSP.

1. Search Engine Back End API
Solution for Fast Prototyping
https://github.com/jimmylai/
search_engine_backend_template
Introduction and Tutorial
Jimmy Lai
r97922028 [at] ntu.edu.tw
http://tw.linkedin.com/pub/jimmy-lai/27/4a/536
2013/01/23

2. Requirements:
• A Backend API provides:
1.
2.
3.
4.
5.
Full-text search
Spatial search
RESTful API
Input validation
Output rendering
• Simple and easy for fast prototyping
LDSP
2

3. LDSP:
Linux + Django + Solr + Python
• Django: API serving
– Input validation
– Data mash-up
– Output rendering
• Solr: Indexing and searching
– Schema
– Indexing
– Query
LDSP
3

4. Automation:
Shell command + Fabric
• Installation:
– Solr
– Python packages: django, djangorestframework,
pysolr
• Start services:
– Solr
– Django standalone server
• Data:
– Create Solr core
– Feed data
LDSP
4

5. [Tutorial] Requirement
• Example Dataset: Taiwan movie dataset
http://data.gov.tw/node/7731
• Data source
http://nrchbms.culture.tw/OpenData/API/iCultureAPI.aspx?
type=26&radius=100&format=json
• We’d like to build a geo search API to
find nearby movie items given a
location
LDSP
5

6. [Tutorial] Data preprocessing (1/2)
• Original data
• {"sno":"29277","mainTitle":"「新南光」電影
《孫悟空遊台灣》，描寫孫悟空化為⼤大學⽣生的
外型遊台灣","year":"未
知","format":"508x640pixels","subject":"戲劇
電影歌仔戲","date":"創作⽇日期：⺠民國時期⺠民國
時期","sourcesite":"http://nrch.cca.gov.tw/
ccahome/photo/photo_meta.jsp?
xml_id=0005780574&dofile=cca100009-hptmtp0281_00-0001w.jpg","px":"120.49","py":"23.853"}
LDSP
6

7. [Tutorial] Data preprocessing (2/2)
•
•
•
•
•
•
•
•
•
•
•
•
•
Preprocessed data: normalized the location format
The preprocessed data is stored as data/movie.json
{
"format": "508x640pixels",
"sno": "29277",
"sourcesite": "http://nrch.cca.gov.tw/ccahome/photo/photo_meta.jsp?
xml_id=0005780574&dofile=cca100009-hp-tmtp0281_00-0001-w.jpg",
"mainTitle": "「新南光」電影《孫悟空遊台灣》，描寫孫悟空化為⼤大學⽣生的
外型遊台灣",
"location": "23.853,120.49",
"year": "未知",
"date": "創作⽇日期：⺠民國時期⺠民國時期",
"id": "29277",
"subject": "戲劇電影歌仔戲"
},
LDSP
7

8. [Tutorial] Data indexing
•
•
•
•
•
pip install fabric
fab setup_env
fab create_core:movie
fab run_solr
fab feed_data:movie,data/movie.json
– Core name as the 1st parameter, data file name
as the 2nd parameter
• fab run_django
• We are done!
LDSP
8

9. [Tutorial] Solr Admin
LDSP
9

10. [Tutorial] Djangorestframework UI
LDSP
10

11. [Tutorial] BE API Query, json output
LDSP
11

12. [Tutorial] BE API Query, xml output
LDSP
12

13. The next steps
• Understand the mechanism of:
– Solr:
• Update schema
• Update partial data
• Query language of Solr
– Djangorestframework:
• Add more parameters
• Input validation
• Output rendering
• Deploy to production environment
LDSP
13