0
Search Topology and Optimization
Mike Maadarani
Canada, -UTC 5:00 EST
April 16th /17th, 2014
SharePoint Architect
Navigatus
Ottawa
Canada
Contact
@mikemaadarani
mike@maadarani.com
www.maadarani.com
Session
Time
(GMT)
Time
(EST) Title Level Speaker Twitter Handle
#SP24S090 5:00 1:00
Using the Content Enrichment Web Serv...
Configuring SSA
and PS
Topology
ScenariosAgenda
Relevancy, Query Builder, &
Optimization
SharePoint 2013
Search Overview
A...
Crawl Component
Query Component
SharePoint 2010 Search Service Application
Query
Engine
Property
Store
(SQL)
FAST
Content
SSA
FAST
Query
SSA
FAST back-end components
(managed separately)
Extensibility:
• Sandbox
• Entity
Extraction
SharePoint 2013 Search Service Application
Index
Component
Query
Engine
Content
Pipeline
Content
Processing
Component
Craw...
SharePoint
SP Apps
Devices
Non-SP UX
HTTP
File shares
SharePoint
User profiles
Lotus Notes
Documentum
Exchange folders
Cus...
I just uploaded a document.
Make it searchable, quick!
FAST
EASY
EASY
Search Driven
Applications
noderunner.exe noderunner.exe noderunner.exe noderunner.exe
Windows services
SharePoint Search Host Controller service
Run...
CPU load
Driving factors
QPS
Query transformations
Network load
Driving factors
Number of index partitions
Size of queries...
CPU load
Driving factors
QPS and item count
Guidelines per index component @ 2 GHz CPU
1M items: 5 QPS per CPU core
5M ite...
CPU load
Driving factors
Documents per second
Link discovery
Crawl management
Network load
Driving factors
Downloading ite...
CPU load
Driving factors
Documents per second
Document size and complexity
Feature extraction
Estimate: 5-10 DPS per CPU c...
CPU load
Driving factors
Number of items
Site activity
Disk load
Local disk used for temporary storage
Bulk load, primacy ...
Low CPU and network load
Load increase with more
components in the search
topology
Item
count
DPS QPS
Load impact (relativ...
$SSADB = "SharePoint_Demo_Search"
$SSAName = "Search ServiceApplication SP24"
$SVCAcct = "mcmsp_search"
$SSI = get-spenter...
Host
VM
Index QPC
VM
Admin
Crawl
CPC
APC
Host
VM
Index QPC
VM
Admin
Crawl
CPC
APC
Other
SharePoint
applications
Web front
end
Admin
Crawl
CPC
APC
Index
QPC
Adm
Adm
#2. Extend the Search Topology:
$hostApp1 = Get-SPEnterpriseSearchServiceInstance -Identity
"SP2013"
$hostApp2 = Get-SPEnt...
Challenges: Intent
Where is my talk
Project Plan?
Are Documents held at the
same place?
I wonder if there are
references f...
Conceptual Relevance Flow
For all queries:
Authorities: Level 1: http://employment
Ranking model: {incorporate user rating...
Sites that are important
Sites with low intrinsic relevance
Takes ~24hrs to
propagate
Setting an authority affects all sites connected through hyperlinks
Sites are weighted
by distance to
the authority
Tune Search Results
Created at the SSA, Tenant, Site Collection or Site
SSA
Site Collection
Site
Condition
When Do I apply the rule?
Action
What to do when the rule is matched?
Publishing
When should the rule be active?
 Exact match, beginning or end
 Ad-hoc or term store dictionary
 Match a regex (advanced)
 Is this query more likely a...
Dynamically Ranking Change
Part of the query
Results Ranking
High Availability and Performance
Better Search Quality
Better management
Friendly results and tools
Session Objective and...
Session
Time
(GMT)
Time
(EST) Title Level Speaker Twitter Handle
#SP24S090 5:00 1:00
Using the Content Enrichment Web Serv...
Search Topology and Optimization
Search Topology and Optimization
Search Topology and Optimization
Search Topology and Optimization
Search Topology and Optimization
Search Topology and Optimization
Search Topology and Optimization
Search Topology and Optimization
Upcoming SlideShare
Loading in...5
×

Search Topology and Optimization

331

Published on

Learn about the SharePoint 2013 Search Topology and how it differs from 2010. In this presentation, I explain the details of all search components, how to properly configure your search topology, and your options to extend your search farm in a hybrid “cloud/on-premise” scenario. This session will explain what you need to consider to design your search, in order to handle your organization's needs. I dive into scripting a high availability search topology, keeping it healthy and manage your day-to-day search operations.

Learn about how to optimize your search for best performance and search relevancy, to support reliable search applications.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
331
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Search Topology and Optimization"

  1. 1. Search Topology and Optimization Mike Maadarani Canada, -UTC 5:00 EST April 16th /17th, 2014
  2. 2. SharePoint Architect Navigatus Ottawa Canada Contact @mikemaadarani mike@maadarani.com www.maadarani.com
  3. 3. Session Time (GMT) Time (EST) Title Level Speaker Twitter Handle #SP24S090 5:00 1:00 Using the Content Enrichment Web Service with SharePoint Server 2013 Search Technical - Advanced Sezai Komur - NEC Australia @sezai #SP24S021 6:00 2:00 2013 Search, Display Templates, Query Rules, Result Types Technical - Beginner Omer Zubair - CSC @Omer_Zubair #SP24S047 8:00 4:00 Real-life experience building search-driven applications for product-centric sites Technical - Advanced Marius Constantinescu - blue-infinity SA @c_marius #SP24S032 10:00 6:00 Custom Indexing Connectors - How to integrate external system into your SharePoint Enterprise Search Technical - Advanced Heinrich Ulbricht - Communardo Software GmbH @h_ulbricht #SP24S083 11:00 7:00 Search First Migration with SharePoint 2013 – Benefits / Disadvantages Technical - Intermediate Max Melcher - Alegri International Service GmbH @maxmelcher #SP24S034 14:00 10:00 The Search Immaturity Cycle, and How to Create a Search Strategy Business - Intermediate Jeff Fried - BA Insight @jefffried #SP24S009 16:00 12:00 10 Things to Know about Search in SP2013 and Office 365 Business - Beginner Agnes Molnar - Search Explained @molnaragnes #SP24S019 18:00 14:00 Search Topology and Optimization Technical - Advanced Mike Maadarani - MCM Consulting @mikemaadarani #SP24S050 22:00 18:00 Search Driven Application Development in SharePoint 2013 Technical - Intermediate Matt Youngstrom - Magenic @spguru
  4. 4. Configuring SSA and PS Topology ScenariosAgenda Relevancy, Query Builder, & Optimization SharePoint 2013 Search Overview Architecture and Resource Utilization
  5. 5. Crawl Component Query Component SharePoint 2010 Search Service Application Query Engine Property Store (SQL)
  6. 6. FAST Content SSA FAST Query SSA FAST back-end components (managed separately) Extensibility: • Sandbox • Entity Extraction
  7. 7. SharePoint 2013 Search Service Application Index Component Query Engine Content Pipeline Content Processing Component Crawl Component Query Processing Component Analytics Processing Component Query Pipeline Search Admin Admin Component Entire index on local disk Property Store (SQL) Analysis Engine Crawl Indexing Engine Link/query analysis & recommendations Separate crawl and indexing Extensibility: • Web callout • Entity Extraction
  8. 8. SharePoint SP Apps Devices Non-SP UX HTTP File shares SharePoint User profiles Lotus Notes Documentum Exchange folders Custom - BCS Public API Search topology components
  9. 9. I just uploaded a document. Make it searchable, quick! FAST
  10. 10. EASY
  11. 11. EASY
  12. 12. Search Driven Applications
  13. 13. noderunner.exe noderunner.exe noderunner.exe noderunner.exe Windows services SharePoint Search Host Controller service Runtime/lifecycle control of search components (except crawler)  hostcontrollerservice.exe SharePoint Server Search service Crawl Component mssearch.exe mssdmn.exe Processes Noderunner.exe Runtime environment for search components (except crawler) msseearch.exe mssdmn.exe Crawl Componentnoderunner.exe Search Runtime Environment hostcontrollerservice.exe Host Controller SharePointAppServer Admin entities Search Service Instance: Provisioning of the search service on each box Search Service Application: SharePoint Configuration entity Still there, but only Crawl Component Admin Component Query Processing Component Content Processing Component Index Component Analytics Processing Component
  14. 14. CPU load Driving factors QPS Query transformations Network load Driving factors Number of index partitions Size of queries and results Example: 20 index partitions @ 20 qps => 200/100 Mbit/s in/outbound Item count DPS QPS Load impact (relative) CPU Network Disk
  15. 15. CPU load Driving factors QPS and item count Guidelines per index component @ 2 GHz CPU 1M items: 5 QPS per CPU core 5M items: 2 QPS per CPU core 10M items: 1 QPS per CPU core Disk load Driving factors QPS and item count New content invalidates caches Disk size: 500GB @ 10M items per index component Item count DPS QPS Load impact (relative) CPU Network Disk
  16. 16. CPU load Driving factors Documents per second Link discovery Crawl management Network load Driving factors Downloading items from content sources Passing items on to CPC Disk load All documents are temporarily stored in data folder Item count DPS QPS Load impact (relative) CPU Network Disk
  17. 17. CPU load Driving factors Documents per second Document size and complexity Feature extraction Estimate: 5-10 DPS per CPU core Network load Driving factors Documents per second Document size Item count DPS QPS Load impact (relative) CPU Network Disk
  18. 18. CPU load Driving factors Number of items Site activity Disk load Local disk used for temporary storage Bulk load, primacy concern is load isolation Network load Same as for CPU load PLUS: Network traffic increases when distributing APC across multiple machines Item count DPS QPS Load impact (relative) CPU Network Disk
  19. 19. Low CPU and network load Load increase with more components in the search topology Item count DPS QPS Load impact (relative) CPU Network Disk
  20. 20. $SSADB = "SharePoint_Demo_Search" $SSAName = "Search ServiceApplication SP24" $SVCAcct = "mcmsp_search" $SSI = get-spenterprisesearchserviceinstance -local #1. Start the search services for SSI Start-SPEnterpriseSearchServiceInstance -Identity $SSI #2. Create theApplication Pool $AppPool = new-SPServiceApplicationPool -name $SSAName"- AppPool" -account $SVCAcct #3. Create the search application and set it to a variable $SearchApp = New-SPEnterpriseSearchServiceApplication - Name $SSAName -applicationpool $AppPool -databaseserver SQL2012 -databasename $SSADB #4. Create search service application proxy $SSAProxy = new-SPEnterpriseSearchServiceApplicationProxy - name $SSAName"Application Proxy" -Uri $SearchApp.Uri.AbsoluteURI #5. Provision SearchAdmin Component Set-SPEnterpriseSearchAdministrationComponent - searchapplication $SearchApp -searchserviceinstance $SSI #6. Create the topology $Topology = New-SPEnterpriseSearchTopology - SearchApplication $SearchApp #7.Assign server(s) to the topology $hostApp1 = Get-SPEnterpriseSearchServiceInstance -Identity "SP2013" New-SPEnterpriseSearchAdminComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchCrawlComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchContentProcessingComponent - SearchTopology $Topology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchAnalyticsProcessingComponent - SearchTopology $Topology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchQueryProcessingComponent - SearchTopology $Topology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchIndexComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1 –IndexPartition 0 #8. Create the topology $Topology | Set-SPEnterpriseSearchTopology
  21. 21. Host VM Index QPC VM Admin Crawl CPC APC Host VM Index QPC VM Admin Crawl CPC APC
  22. 22. Other SharePoint applications Web front end Admin Crawl CPC APC Index QPC
  23. 23. Adm Adm
  24. 24. #2. Extend the Search Topology: $hostApp1 = Get-SPEnterpriseSearchServiceInstance -Identity "SP2013" $hostApp2 = Get-SPEnterpriseSearchServiceInstance -Identity "SPSearch" Start-SPEnterpriseSearchServiceInstance -Identity $hostApp1 Start-SPEnterpriseSearchServiceInstance -Identity $hostApp2 #3. Keep running this command until the Status is Online: Get-SPEnterpriseSearchServiceInstance -Identity $hostApp1 Get-SPEnterpriseSearchServiceInstance -Identity $hostApp2 #4. Once the status is online, you can proceed with the following commands: $ssa = Get-SPEnterpriseSearchServiceApplication $active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa –Active $newTopology = New-SPEnterpriseSearchTopology - SearchApplication $ssa New-SPEnterpriseSearchAdminComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchCrawlComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchContentProcessingComponent - SearchTopology $newTopology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchAnalyticsProcessingComponent - SearchTopology $newTopology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchQueryProcessingComponent - SearchTopology $newTopology -SearchServiceInstance $hostApp1 New-SPEnterpriseSearchIndexComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1 –IndexPartition 0 New-SPEnterpriseSearchAdminComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2 New-SPEnterpriseSearchCrawlComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2 New-SPEnterpriseSearchContentProcessingComponent - SearchTopology $newTopology -SearchServiceInstance $hostApp2 New-SPEnterpriseSearchAnalyticsProcessingComponent - SearchTopology $newTopology -SearchServiceInstance $hostApp2 New-SPEnterpriseSearchQueryProcessingComponent - SearchTopology $newTopology -SearchServiceInstance $hostApp2 New-SPEnterpriseSearchIndexComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2 –IndexPartition 1 #5.Activate the topology: Set-SPEnterpriseSearchTopology -Identity $newTopology
  25. 25. Challenges: Intent Where is my talk Project Plan? Are Documents held at the same place? I wonder if there are references from previous projects? Different people have different intents Query Rules help you handle intents There is rarely a single right answer Infrastructure Project
  26. 26. Conceptual Relevance Flow For all queries: Authorities: Level 1: http://employment Ranking model: {incorporate user ratings} Query: HR Employment quarterly report Search Web Part Query Processing Engine Document Collection Thesaurus: HR  Human Resources Best bets: HR Employment /HR/employment (WORDS HR, Human Resources) AND (WORDS employees, employed) AND (WORDS quarterly, quarterlies) AND (WORDS report, reports, reported) Mixed Results for: • HR Employment best bet • HR Employment quarterly report • HR Employment ContentType=reports Dynamic Reordering Rules: Quarterly Report  {prefer docs from http://reports} Query Rule: {Terms} Quarterly Report  {Terms} ContentType=“reports”
  27. 27. Sites that are important Sites with low intrinsic relevance Takes ~24hrs to propagate
  28. 28. Setting an authority affects all sites connected through hyperlinks Sites are weighted by distance to the authority
  29. 29. Tune Search Results Created at the SSA, Tenant, Site Collection or Site SSA Site Collection Site
  30. 30. Condition When Do I apply the rule? Action What to do when the rule is matched? Publishing When should the rule be active?
  31. 31.  Exact match, beginning or end  Ad-hoc or term store dictionary  Match a regex (advanced)  Is this query more likely aimed at the following source…?  Do people mostly click on result of the following type…?  Show a promoted result  Show a block of results  Replace the core results with a different query
  32. 32. Dynamically Ranking Change Part of the query Results Ranking
  33. 33. High Availability and Performance Better Search Quality Better management Friendly results and tools Session Objective and Takeaways
  34. 34. Session Time (GMT) Time (EST) Title Level Speaker Twitter Handle #SP24S090 5:00 1:00 Using the Content Enrichment Web Service with SharePoint Server 2013 Search Technical - Advanced Sezai Komur - NEC Australia @sezai #SP24S021 6:00 2:00 2013 Search, Display Templates, Query Rules, Result Types Technical - Beginner Omer Zubair - CSC @Omer_Zubair #SP24S047 8:00 4:00 Real-life experience building search-driven applications for product-centric sites Technical - Advanced Marius Constantinescu - blue-infinity SA @c_marius #SP24S032 10:00 6:00 Custom Indexing Connectors - How to integrate external system into your SharePoint Enterprise Search Technical - Advanced Heinrich Ulbricht - Communardo Software GmbH @h_ulbricht #SP24S083 11:00 7:00 Search First Migration with SharePoint 2013 – Benefits / Disadvantages Technical - Intermediate Max Melcher - Alegri International Service GmbH @maxmelcher #SP24S034 14:00 10:00 The Search Immaturity Cycle, and How to Create a Search Strategy Business - Intermediate Jeff Fried - BA Insight @jefffried #SP24S009 16:00 12:00 10 Things to Know about Search in SP2013 and Office 365 Business - Beginner Agnes Molnar - Search Explained @molnaragnes #SP24S019 18:00 14:00 Search Topology and Optimization Technical - Advanced Mike Maadarani - MCM Consulting @mikemaadarani #SP24S050 22:00 18:00 Search Driven Application Development in SharePoint 2013 Technical - Intermediate Matt Youngstrom - Magenic @spguru
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×