Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloud powered search

868 views

Published on

Cloud powered search

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Cloud powered search

  1. 1. Search Cloud powered LIVIU MAZILU RADU PINTILIE
  2. 2. April 25, 2015Cloud powered search © EXPERT NETWORK CODECAMP Challenges in distributed applications SQL Azure Federation HDInsight DocumentDB Previous subjects
  3. 3. April 25, 2015Cloud powered search © EXPERT NETWORK Azure Search The need for search Search explained Development Case Scenarios Agenda
  4. 4. April 25, 2015Cloud powered search © EXPERT NETWORK The need for search Why do we search for data? How do we store it to search efficiently? What’s important?
  5. 5. April 25, 2015Cloud powered search © EXPERT NETWORK Is this a search engine? where [field] like “%codecamp%”
  6. 6. April 25, 2015Cloud powered search © EXPERT NETWORK WHAT IS A SEARCH ENGINE? Efficient indexing of data On all fields / combination of fields Analyzing data Text Search Tokenizing Stemming Filtering Understanding locations Relevance scoring
  7. 7. April 25, 2015Cloud powered search © EXPERT NETWORK Lucene Document: collection of fields Field: string based key-value pair Collection: set of documents Inverted index: a term can list the number of documents it contains Score: relevancy for each document matching the query
  8. 8. April 25, 2015Cloud powered search © EXPERT NETWORK How searching works Id Title UserId ViewCount Tags 1 Controller Action ambiguity even with [HttpPost] decoration? (ASP.NET MVC4) 5 352 asp.net asp.net-mvc asp.net-mvc-4 f# 2 Why can't I use a scrollwheel on a webpage? 6 109 c# javascript asp.net asp.net-mvc-4 twitter- bootstrap-3 3 Access session variable of one site in another" 7 78 asp.net .net 4 Check if SIM card exists 5 209 c# windows-phone-8
  9. 9. April 25, 2015Cloud powered search © EXPERT NETWORK Inverted index How searching works Title Access session variable of one site in another" 3 Check if SIM card exists 4 Controller Action ambiguity even with [HttpPost] decoration? (ASP.NET MVC4) 1 Why can't I use a scrollwheel on a webpage? 2 UserID 5 1, 4 6 2 7 3 ViewCount 78 3 109 2 209 4 352 1
  10. 10. April 25, 2015Cloud powered search © EXPERT NETWORK Inverted index How searching works Title Access session variable of one site in another" 3 Check if SIM card exists 4 Controller Action ambiguity even with [HttpPost] decoration? (ASP.NET MVC4) 1 Why can't I use a scrollwheel on a webpage? 2 UserID 5 1, 4 6 2 7 3 ViewCount 78 3 109 2 209 4 352 1 Query: UserID = 5
  11. 11. April 25, 2015Cloud powered search © EXPERT NETWORK Full text search Id Tags 1 asp.net asp.net-mvc asp.net-mvc- 4 f# 2 c# javascript asp.net asp.net-mvc- 4 twitter-bootstrap-3 3 asp.net .net 4 c# windows-phone-8 How searching works Term Doc .net 3 asp.net 1, 2, 3 asp.net-mvc-4 1, 2 c# 2, 4 f# 1 javascript 2 mvc 1 twitter-bootstrap-3 2 windows-phone-8 4
  12. 12. April 25, 2015Cloud powered search © EXPERT NETWORK Full text search Id Tags 1 asp.net asp.net-mvc asp.net-mvc- 4 f# 2 c# javascript asp.net asp.net-mvc- 4 twitter-bootstrap-3 3 asp.net .net 4 c# windows-phone-8 How searching works Term Doc .net 3 asp.net 1, 2, 3 asp.net-mvc-4 1, 2 c# 2, 4 f# 1 javascript 2 mvc 1 twitter-bootstrap-3 2 windows-phone-8 4 Query: “javascript” in Tags
  13. 13. April 25, 2015Cloud powered search © EXPERT NETWORK Full text search Id Tags 1 asp.net asp.net-mvc asp.net-mvc- 4 f# 2 c# javascript asp.net asp.net-mvc- 4 twitter-bootstrap-3 3 asp.net .net 4 c# windows-phone-8 How searching works Term Doc .net 3 asp.net 1, 2, 3 asp.net-mvc-4 1, 2 c# 2, 4 f# 1 javascript 2 mvc 1 twitter-bootstrap-3 2 windows-phone-8 4 Query: “asp.net” in Tags
  14. 14. April 25, 2015Cloud powered search © EXPERT NETWORK Auto-completion Uses
  15. 15. April 25, 2015Cloud powered search © EXPERT NETWORK Auto-correction Phrasing Iframe security – Security in an Iframe Word-level distance grey/gray color/colour Uses
  16. 16. April 25, 2015Cloud powered search © EXPERT NETWORK Elasticsearch Distributed: aggregated results of search performed on multiple shards/indices Schema Less: is document oriented. Supports JSON format RESTful: supports REST interface Faceted Search: support for navigational search functionality Replication: supports index replication Fail over: replication and distributed nature provides inbuilt fail over. Near Real time: supports near real time updates
  17. 17. April 25, 2015Cloud powered search © EXPERT NETWORK Distributed & highly available • Multiple servers (nodes) running in a cluster • Acting as single service • Nodes in cluster that store data or nodes that just help in speeding up search queries. • Sharding • Indeces are sharded (# shards is configurable) • Each shard can have zero or more replicas • Replicas on different servers (server pools) for failover • One in the cluster goes down? No problem. Elasticsearch
  18. 18. April 25, 2015Cloud powered search © EXPERT NETWORK Azure search Elasticsearch as a managed service Platform as a service (PaaS) Admin by Rest API Data exchange with JSON
  19. 19. April 25, 2015Cloud powered search © EXPERT NETWORK Where are we at Service Ease of use Scalability Easy Administration Manual search (SQL) No No Partial Elasticsearch Yes Yes No AzureSearch Yes Yes Yes
  20. 20. April 25, 2015Cloud powered search © EXPERT NETWORK Resource model Service Index (schema type 1) Index (schema type 2) Document Document Field1 Field2 Field3 Field4 Indexers Azure Search
  21. 21. April 25, 2015Cloud powered search © EXPERT NETWORK Management Portal Demo
  22. 22. April 25, 2015Cloud powered search © EXPERT NETWORK Index creation POST https://codecamp-en.search.windows.net/indexes "name": "stackoverflow-posts", "fields": [ { "name": "name_of_field", "type": “data_type", "searchable": true (default where applicable) | false , "filterable": true (default) | false, "sortable": true (default where applicable) | false "facetable": true (default where applicable) | false , "key": true | false (default), "retrievable": true (default) | false } ] … Azure Search
  23. 23. April 25, 2015Cloud powered search © EXPERT NETWORK Index documents Indexers Data sources: Azure SQL Database, DocumentDB Connects data sources with target search indexes An indexer can be used in the following ways: one-time copy of the data to populate an index sync an index with changes from the data source on a schedule invoke on-demand to update an index as needed Azure Search
  24. 24. April 25, 2015Cloud powered search © EXPERT NETWORK CRUD Operations Add, Update, Delete POST https://codecamp-en.search.windows.net/indexes/stackoverflow/docs/index { "@search.action": "upload (default) | merge | mergeOrUpload | delete", "key_field_name": "unique_key_of_document", (key/value pair for key field from index schema) "field_name": field_value (key/value pairs matching index schema) } Azure Search
  25. 25. April 25, 2015Cloud powered search © EXPERT NETWORK Searching through data GET https://codecamp-en.search.windows.net/indexes/stackoverflow/docs? search=[string] + (AND operator “code" and “camp") | (OR operator “code" or “camp" or both) - (NOT operator. “code–camp" “code" term and/or do not have “camp" ) * (Suffix operator. “cod*" - starts with “cod", ignoring case) " (Phrase search operator) ( ) (Precedence operator - code+(camp|workshop) searchMode=any|all searchFields=[string] Azure Search
  26. 26. April 25, 2015Cloud powered search © EXPERT NETWORK Filtering results $filter=[string] - Odata syntax $skip=# $top=# $count=true|false $orderby=[string] $select=[string] Azure Search
  27. 27. April 25, 2015Cloud powered search © EXPERT NETWORK Emphasizing results facet=[string] (field names) count sort values interval highlight=[string] (field names) highlightPreTag=[string] (default is em) highlightPostTag=[string] Azure Search
  28. 28. April 25, 2015Cloud powered search © EXPERT NETWORK Suggestions GET https://codecamp-en.search.windows.net/indexes/stackoverflow/docs/suggest search=[string] suggesterName=[string] fuzzy=[boolean] searchFields=[string] Azure Search
  29. 29. April 25, 2015Cloud powered search © EXPERT NETWORK Stackoverflow Posts 5.215.584 records 212 MB in Title column 118 MB in Tags column 10,5 GB in Body column Sample Data Column Name Data Type Id int CreationDate datetime Score float ViewCount int Body nvarchar OwnerUserId int Title nvarchar Tags nvarchar
  30. 30. April 25, 2015Cloud powered search © EXPERT NETWORK Search API DEMO
  31. 31. April 25, 2015Cloud powered search © EXPERT NETWORK Scaling Capacity measured in Search Units 1 Search Unit 1 Partition 1 Replica Horizontal scaling by increasing the number of partitions and/or replicas Cloud powered search
  32. 32. April 25, 2015Cloud powered search © EXPERT NETWORK Storage Partition limitations: 15 million documents 25 GB data Every Index is split by default in 12 shards Each partition can store 1,2,3,4,6,12 shards Cloud powered search
  33. 33. April 25, 2015Cloud powered search © EXPERT NETWORK SCENARIOS Online retail/ecommerce User generated/social content Not just for the web Hybrid Applications USE CASE
  34. 34. April 25, 2015Cloud powered search © EXPERT NETWORK Conclusions The need for search Search explained Development Case Scenarios
  35. 35. April 25, 2015Cloud powered search © EXPERT NETWORK Questions ?
  36. 36. April 25, 2015Cloud powered search © EXPERT NETWORK THANK YOU

×