[LDSP] Search Engine Back End API Solution for Fast Prototyping

  • 503 views
Uploaded on

In this slides, I propose a solution for fast prototyping of search engine back end API. It consists of Linux + Django + Solr + Python (LDSP), and all are open source softwares. The solution also …

In this slides, I propose a solution for fast prototyping of search engine back end API. It consists of Linux + Django + Solr + Python (LDSP), and all are open source softwares. The solution also provides code repository with automation scripts. Everyone can build a Search Engine back end API in seconds by exploiting LDSP.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
503
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
3
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Search Engine Back End API Solution for Fast Prototyping https://github.com/jimmylai/ search_engine_backend_template Introduction and Tutorial Jimmy Lai r97922028 [at] ntu.edu.tw http://tw.linkedin.com/pub/jimmy-lai/27/4a/536 2013/01/23
  • 2. Requirements: •  A Backend API provides: 1.  2.  3.  4.  5.  Full-text search Spatial search RESTful API Input validation Output rendering •  Simple and easy for fast prototyping LDSP 2
  • 3. LDSP: Linux + Django + Solr + Python •  Django: API serving –  Input validation –  Data mash-up –  Output rendering •  Solr: Indexing and searching –  Schema –  Indexing –  Query LDSP 3
  • 4. Automation: Shell command + Fabric •  Installation: –  Solr –  Python packages: django, djangorestframework, pysolr •  Start services: –  Solr –  Django standalone server •  Data: –  Create Solr core –  Feed data LDSP 4
  • 5. [Tutorial] Requirement •  Example Dataset: Taiwan movie dataset http://data.gov.tw/node/7731 •  Data source http://nrchbms.culture.tw/OpenData/API/iCultureAPI.aspx? type=26&radius=100&format=json •  We’d like to build a geo search API to find nearby movie items given a location LDSP 5
  • 6. [Tutorial] Data preprocessing (1/2) •  Original data •  {"sno":"29277","mainTitle":"「新南光」電影 《孫悟空遊台灣》,描寫孫悟空化為⼤大學⽣生的 外型遊台灣","year":"未 知","format":"508x640pixels","subject":"戲劇 電影歌仔戲","date":"創作⽇日期:⺠民國時期⺠民國 時期","sourcesite":"http://nrch.cca.gov.tw/ ccahome/photo/photo_meta.jsp? xml_id=0005780574&dofile=cca100009-hptmtp0281_00-0001w.jpg","px":"120.49","py":"23.853"} LDSP 6
  • 7. [Tutorial] Data preprocessing (2/2) •  •  •  •  •  •  •  •  •  •  •  •  •  Preprocessed data: normalized the location format The preprocessed data is stored as data/movie.json { "format": "508x640pixels", "sno": "29277", "sourcesite": "http://nrch.cca.gov.tw/ccahome/photo/photo_meta.jsp? xml_id=0005780574&dofile=cca100009-hp-tmtp0281_00-0001-w.jpg", "mainTitle": "「新南光」電影《孫悟空遊台灣》,描寫孫悟空化為⼤大學⽣生的 外型遊台灣", "location": "23.853,120.49", "year": "未知", "date": "創作⽇日期:⺠民國時期⺠民國時期", "id": "29277", "subject": "戲劇電影歌仔戲" }, LDSP 7
  • 8. [Tutorial] Data indexing •  •  •  •  •  pip install fabric fab setup_env fab create_core:movie fab run_solr fab feed_data:movie,data/movie.json –  Core name as the 1st parameter, data file name as the 2nd parameter •  fab run_django •  We are done! LDSP 8
  • 9. [Tutorial] Solr Admin LDSP 9
  • 10. [Tutorial] Djangorestframework UI LDSP 10
  • 11. [Tutorial] BE API Query, json output LDSP 11
  • 12. [Tutorial] BE API Query, xml output LDSP 12
  • 13. The next steps •  Understand the mechanism of: –  Solr: •  Update schema •  Update partial data •  Query language of Solr –  Djangorestframework: •  Add more parameters •  Input validation •  Output rendering •  Deploy to production environment LDSP 13