Download as PDF, PPTX
![Document Upload
http(s)://< document service endpoint >/2013-01-01/documents/
batch!
!
Accept: application/json !
Content-Length: 1176 !
Content-Type: application/json !
Host: doc.imdb-movies-rr2f34ofg56xneuemujamut52i.us-east-1.cloudsearch.amazonaws.com !
!
{ : , : "tt0371746", : { "directors" : [ "Jon Favreau" ],
"release_date" : "2008-04-14T00:00:00Z", "rating" : 7.9, "genres" : [ "Action",
"Adventure", "Sci-Fi" ], "image_url" : "http://ia.media-imdb.com/images/M/
MV5BMTczNTI2ODUwOF5BMl5BanBnXkFtZTcwMTU0NTIzMw@@._V1_SX400_.jpg", "plot" : "When
wealthy industrialist Tony Stark is forced to build an armored suit after a life-
threatening incident, he ultimately decides to use its technology to fight against
evil.", "title" : "Iron Man", "rank" : 171, "running_time_secs" : 7560, "actors" :
[ "Robert Downey Jr.", "Gwyneth Paltrow", "Terrence Howard" ], "year" : 2008 }},!
{ , : "tt0434409"} ]!](https://image.slidesharecdn.com/cloudsearchjpmeetup-140516014742-phpapp01/85/Build-a-Scalable-Search-Engine-With-Amazon-CloudSearch-by-Jon-Handler-32-320.jpg)
![Simple Queries
http(s)/<search endpoint>/2013-01-01/search?q=iron+man!
{"status": {"rid": "oei6zt8oAgq5QOc=",!
"time-ms": 4},!
"hits": {"found": 9, "start": 0,!
"hit": [!
{"id": "tt1228705"},!
{"id": "tt0120744"},!
{"id": "tt0371746"},!
{"id": "tt1866249"},!
{"id": "tt0119558"},!
{"id": "tt0402894"},!
{"id": "tt1258972"},!
{"id": "tt1300854"},!
{"id": "tt0462465"} ] } }!](https://image.slidesharecdn.com/cloudsearchjpmeetup-140516014742-phpapp01/85/Build-a-Scalable-Search-Engine-With-Amazon-CloudSearch-by-Jon-Handler-34-320.jpg)
![IAM Integration
Configuration API Only
{!
"Version":"2012-10-17",!
"Statement": [!
{ "Effect": "Allow",
"Action": ["cloudsearch:*"],
"Resource": "arn:aws:cloudsearch:us-east-1:111122223333:domain/imdb-movies" },!
{ "Effect": "Deny",!
"Action": ["cloudsearch:DeleteDomain"],!
"Resource": "arn:aws:cloudsearch:us-east-1:111122223333:domain/imdb-movies" }!
]!
}!](https://image.slidesharecdn.com/cloudsearchjpmeetup-140516014742-phpapp01/85/Build-a-Scalable-Search-Engine-With-Amazon-CloudSearch-by-Jon-Handler-43-320.jpg)
Uploaded byEiji Shinohara
Build a Scalable Search Engine With Amazon CloudSearch by Jon Handler
This document provides an overview of building a scalable search engine using Amazon CloudSearch. It discusses introducing concepts of search, how Amazon CloudSearch works, and how to build a search service using CloudSearch. Key points include that CloudSearch is a fully managed search service that makes setting up, operating and scaling a search solution easy. It also discusses features like automatic scaling, data uploading, query types, faceting, drilling and ranking.
Build a Scalable Search Engine With Amazon CloudSearch by Jon Handler
- 1. Build a ScalableSearch Engine With Amazon CloudSearch
- 2. Agenda • Introduction toSearch • Amazon CloudSearch • Building with CloudSearch
- 3.
- 4. Search Engines ConnectUs To Data
- 5.
- 6. Representation of aDocument Field Value id tt0371746 title Iron Man description When wealthy industrialist Tony Stark is forced to build an armored suit after a life-threatening incident, he ultimately decides to use its technology to fight against evil. director John Favreau actors Robert Downey Jr., Gwyneth Paltrow, Terrence Howard ... rating 7.9 release_date 2008-05-02T00:00:00Z
- 7.
- 8. Geo • Latlon datatype • Region search • Distance sort • Supports mobile
- 9. Text Processing (Normalization) • Tokenization (parsing) • Downcasing • Stemming • Stopword removal • Synonym Addition When wealthy industrialist Tony Stark is forced to build an armored suit after a life-threatening incident, he ultimately decides to use its technology to fight against evil. when wealth industrial tony stark force build armor suit after life threaten incident ultimate decide use technology fight against evil
- 10. Indexing Term Documents (PostingList) Iron The Man in the Iron Mask Iron Man 2 Iron Man The Iron Giant The Iron Lady ... Man Rain Man The Man in the Moon Iron Man 2 The Lawnmower Man The Third Man Iron Man ...
- 11. Matching The Man inthe Iron Mask Iron Man 2 Iron Man The Iron Giant The Iron Lady Rain Man The Man in the Moon Iron Man 2 The Lawnmower Man The Third Man Iron Man Iron Man 2 Iron Man
- 12. Ranking and Relevance • The meat of the search engine • TF-IDF – uniqueness and presence • Additional Criteria – Measures of document value (e.g. rating) – Observed user behavior – Freshness
- 13. Summary • Search makesdata accessible • Search documents gather information about one search target • Reverse indices provide the basis of text-text matching • Relevance brings the best matches
- 14.
- 15. Building a Searchservice • Build your own – Extend datastores and build custom relevance engine • Open Source – Apache Solr, ElasticSearch • Enterprise Search – FAST, Autonomy, Endeca
- 16. Challenges with buildinga Search service • COMPLEX: Requires extensive search expertise • COSTLY: High upfront expenditure • SLOW: Long time to market. Slows innovation • UNDIFFERENTIATED: Operational overhead that doesn’t add value to core product
- 17. Where CloudSearch fitsin the picture Amazon CloudSearch is a fully managed search service in the cloud that makes it easy to setup, operate, and scale a search solution for your website or application Similar benefits as other AWS Managed Services • Easy to setup and operate (Console, SDK, CLT) • Pay as you go • No need to guess capacity • Experiment fast with low risk • Go Global in minutes
- 18.
- 19. Automatic Scaling SEARCH INSTANCE IndexPartition n Copy 1 SEARCH INSTANCE Index Partition 2 Copy 2 SEARCH INSTANCE Index Partition n Copy 2 SEARCH INSTANCE Index Partition 2 Copy n SEARCH INSTANCE DATA Document Quantity and Size TRAFFIC Search Request Volume and Complexity Index Partition n Copy n SEARCH INSTANCE Index Partition 1 Copy 1 SEARCH INSTANCE Index Partition 2 Copy 1 SEARCH INSTANCE Index Partition 1 Copy 2 SEARCH INSTANCE Index Partition 1 Copy n
- 20.
- 21. Movies > Sci-Fi/Fantasy> 2008 to 2010 > Downey > "Iron"
- 22.
- 23.
- 24. 2014年3月 CloudSearch Launch Arabic, Armenian, Basque, Bulgarian, Catalan, SimplifiedChinese, Traditional Chinese, Czech, Danish, Dutch, English, Finnish, French, Galician, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Korean, Latvian, Norwegian, Persian, Portuguese, Romanian, Russian, Spanish, Swedish, Thai, Turkish • Support for 33 languages
- 25.
- 26. Japanese Text Processing • 形態素解析(Morphological Analysis) – 自然言語で書かれた文を形態素の列に分割し、それぞれの品詞を判別する作業 (http://ja.wikipedia.org/wiki/形態素解析) • 英語のようにスペースで区切られている言語と異なり、 • 日本語は日本語用の構文解析が必要 – 例) 彼はエンジニアだ • 彼(名詞-代名詞)/は(助詞-係助詞)/エンジニア(名詞-一般)/だ(助動詞) • “エンジニア”を抽出してインデックスを作ることにより、 • ”エンジニア”で検索された際に、高速なレスポンスの実現が可能
- 27. Japanese Text Processing • 正規化(Normalize) – ｴﾝｼﾞﾆｱ(半角カナ)で検索された場合も、エンジニア(全角カナ)で検索された場合も、どちら の場合もヒットして欲しい – CloudSearchでサポートされている機能 – 更に突っ込んだ正規化に関しては要件に応じて下記のような実装を自分で行う事が望ま しい場合もある • NFD(Canonical Decomposition): 正規化形式D • NFC(Canonical Composition): 正規化形式C • NFKD(Compatibility Decomposition): 正規化形式KD • NFKC(Compatibility Composition): 正規化形式KC
- 28. Japanese Text Processing • Stemming – 飲んだ → 飲ん(動詞-自立, baseForm:飲む)/だ(助動詞) → 飲む – ステミング辞書への追加 (API/SDKでも追加可能)
- 29. Japanese Text Processing • Stopword Removal – 「の」、「は」、「か」といった意味の無い言葉を除く – ステミング同様Stopword辞書への追加 (API/SDKでも追加可能)
- 30. Japanese Text Processing • Synonym Addition – Synonym = 同義語 • 「ベニス」「ベネチア」「ヴェネチア」 • 「昨年」「去年」 – 同じ意味なので検索された場合にヒットさせる – Stopwords, Stemming同様に追加可能
- 31. Japanese Text Processing • Synonym Addition – シノニム辞書への追加 (API/SDKでも追加可能) • Alias – pupilで検索してstudentのドキュメントがヒット – studentで検索してpupilのドキュメントはヒットしない • Group – 1st, first, oneどれで検索しても – 1st, first, oneの全てのドキュメントがヒット
- 32. Document Upload http(s)://< documentservice endpoint >/2013-01-01/documents/ batch! ! Accept: application/json ! Content-Length: 1176 ! Content-Type: application/json ! Host: doc.imdb-movies-rr2f34ofg56xneuemujamut52i.us-east-1.cloudsearch.amazonaws.com ! ! { : , : "tt0371746", : { "directors" : [ "Jon Favreau" ], "release_date" : "2008-04-14T00:00:00Z", "rating" : 7.9, "genres" : [ "Action", "Adventure", "Sci-Fi" ], "image_url" : "http://ia.media-imdb.com/images/M/ MV5BMTczNTI2ODUwOF5BMl5BanBnXkFtZTcwMTU0NTIzMw@@._V1_SX400_.jpg", "plot" : "When wealthy industrialist Tony Stark is forced to build an armored suit after a life- threatening incident, he ultimately decides to use its technology to fight against evil.", "title" : "Iron Man", "rank" : 171, "running_time_secs" : 7560, "actors" : [ "Robert Downey Jr.", "Gwyneth Paltrow", "Terrence Howard" ], "year" : 2008 }},! { , : "tt0434409"} ]!
- 33. Simple Queries Movies >Sci-Fi/Fantasy > 2008 to 2010 > Downey > "Iron"
- 34. Simple Queries http(s)/<search endpoint>/2013-01-01/search?q=iron+man! {"status":{"rid": "oei6zt8oAgq5QOc=",! "time-ms": 4},! "hits": {"found": 9, "start": 0,! "hit": [! {"id": "tt1228705"},! {"id": "tt0120744"},! {"id": "tt0371746"},! {"id": "tt1866249"},! {"id": "tt0119558"},! {"id": "tt0402894"},! {"id": "tt1258972"},! {"id": "tt1300854"},! {"id": "tt0462465"} ] } }!
- 35. Complex Queries Movies >Sci-Fi/Fantasy > 2008 to 2010 > Downey > "Iron"
- 36. Faceting Movies > Sci-Fi/Fantasy> 2008 to 2010 > Downey > "Iron"
- 37. Drilldown Movies > Sci-Fi/Fantasy> 2008 to 2010 > Downey > "Iron"
- 38. Adjustable Ranking Movies >Sci-Fi/Fantasy > 2008 to 2010 > Downey > "Iron"
- 39. Movies > Sci-Fi/Fantasy> 2008 to 2010 > Downey > "Iron" Highlighting
- 40. Movies > Sci-Fi/Fantasy> 2008 to 2010 > Downey > "Iron"
- 41.
- 42.
- 43. IAM Integration Configuration APIOnly {! "Version":"2012-10-17",! "Statement": [! { "Effect": "Allow", "Action": ["cloudsearch:*"], "Resource": "arn:aws:cloudsearch:us-east-1:111122223333:domain/imdb-movies" },! { "Effect": "Deny",! "Action": ["cloudsearch:DeleteDomain"],! "Resource": "arn:aws:cloudsearch:us-east-1:111122223333:domain/imdb-movies" }! ]! }!
- 44. Closing Thoughts • ContentDiscovery goes hand in hand with Content. Search is everywhere! • Amazon CloudSearch is a fully managed, easy to use, cost effective search service – easy to build, easy to scale • Get the powerful search features found in open source engines (Apache Solr) combined with value add AWS features (easy setup, on demand pricing, auto scaling, Multi-AZ, global availability)
- 45. Questions? Jon Handler (handler@amazon.com) PravinMuthukumar (pravinm@amazon.com)