Delivering a Search-Driven User Experience with SharePoint and FAST


Published on

Do you always know exactly what you are looking for? The traditional Search Engine model requires known and accurate keywords to return meaningful content. This session will cover best use of FAST technology including pipeline extensibility, lemmatisation, fuzzy logic and synonyms which can be combined with SharePoint to break that model and provide a compelling Search-Driven User Experience. An in-depth case study with lessons learned will also be presented including a live demo, in which everything ever recorded by the Government numbering tens of thousands of documents in a variety of unstructured, difficult and lengthy formats is successfully parsed and indexed to deliver a public-facing SharePoint site driven by FAST Search for a great User Experience.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This is the case study track, so I'm going to tell the story of how we built an advanced Government search-driven SharePoint Site underpinned by FAST Search.  However it's not JUST about a particular element of functionality in FAST or in SharePoint or in Search applications generally - Hopefully in about an hour you will realise you NEED a Search Application in your organisation, and if you have one, you will hopefully pick up something that may improve your current one! I'm not claiming that the case study is "the best" however we went through a lot of pain in this exercise, if I can save you some of that, my job will be done. I would really like to hear from anybody afterward about successes as well as any failures. This is not a technical deep dive, although I have a developer background, this session is about the What, When, Why and How to provide better user experiences for your users through search driven applicationsfeel free to contact me after the session
  • IT Pro? Dev? IW? Who uses FAST of any description? Good Conference?
  • What is a Search Application? Anatomy because it can be broken down
  • users, creators, content, engine, and interface.Morville, Peter; Callender, Jeffery (2010-01-14). Search Patterns (Kindle Location 605). OReilly Media - A. Kindle Edition. Platform-agnostic Business Requirements hard to define… especially with upgrades!!
  • Enterprise vs Consumer.. Although a Search Application can be consumer-focused (e.g. e-commerce, travel etc. )“intuitive, meaningful and scalable access to the content”
  • We are interested in the Document Processing pipelineIn FS4SP documents are crawled by the connectorDocument processing stages include We used FAST ESP
  • Query Expansion Spy: Output
  • What is a Search Application? Anatomy because it can be broken down
  • Manual metadata – don’t trust people!
  • Intelligent Linguistic Processing Visual Results“No Keyword”
  • The minutes of meetings of the States started in 1524Beware of migrations…!!!
  • Politicians:Votes & PropositionsPower Users: Very specific information regular usersEmployees: All information about a given topicResidents: Anything – Votes typically, activity
  • Gartner's MarketScope for Enterprise Search examines a group of generalist vendors, many of which our clients frequently ask about, which deliver simply priced, solid enterprise search functionality for common use cases.What You Need to KnowEnterprise search — the simplest and most frequently deployed aspect of information access technology — now dominates the dialogue between organizations and vendors about how to improve people's ability to find information in numerous and disparate repositories. Major vendors have come to dominate the market and, not surprisingly, they dominate the questions that Gartner's clients ask of its analysts. Nevertheless, some smaller vendors remain very effective at delivering the capabilities necessary to create search installations.Simpler projects, such as making an intranet searchable, fall within the scope of this document. Organizations that require specialized search-based applications (knowledge management for a high-tech electronics manufacturer, for example, or collaboration support for pharmaceutical researchers) will want to find a vendor with specialized vocabularies, ontologies and workflow.The best initial step in selecting an enterprise search vendor is to staff the project with professionals who can make decisions about project scope and establish requirements based on that knowledge. Vendors that offer basic solutions and more sophisticated products appear in this MarketScope; organizations that want the most sophisticated platforms or search-based applications, and which are willing to explore vendors that are less well established, should consider those vendors that were excluded because they did not meet the criteria for this report.Gartner puts the compound annual growth rate for the enterprise search market at 11.7% from 2007 to 2013. We believe that the market in 2010 was worth $1.37 billion, and this figure will grow to $1.89 billion in 2013Strengths:Microsoft's broad product line beyond search makes it attractive for projects that have a larger footprint.It is particularly strong at transparently revealing the logical elements that lead to a particular result being returned to users.It has invested significantly in federation as a means of broadening search, while seeking to preserve comparative relevance scoring and results interfaces.It addresses social search effectively, allowing users to collaborate on information gathering.Cautions:Clients express concern that Microsoft will focus on SharePoint to the detriment of non-SharePoint features.Pricing for the Fast search engine is difficult to calculate and deliver for clients not on the SharePoint ECAL.
  • Delivering a Search-Driven User Experience with SharePoint and FAST

    1. 1. Delivering a Search-Driven UX with SharePoint & FAST #CS716 Aonghus Fraser#CS716
    2. 2. Aonghus (Gus) Fraser SharePoint Lead Consultant @ C5 Alliance  ~60 Consultants; ~18 SharePoint & CRM*  Working with SharePoint since WSS 2.0  Developer background (MCPD, MCSD etc.)  Email:  Twitter: @gusfraser  Blog:*probably the highest concentration of SharePoint on the planet (unconfirmed)
    3. 3. Agenda Introductions The Anatomy of a Search Application When/Why Search-Driven UX Case Study: States Assembly Demo Lessons Learned & Top Tips#CS716
    4. 4. Agenda Introductions The Anatomy of a Search Application When/Why Search-Driven UX Case Study: States Assembly Demo Lessons Learned & Top Tips#CS716
    5. 5. The Anatomy of a Search Application Content Roles (Users and Creators) Indexing, Processing & UISource: Search Patterns (Morville/Callender , 2010)#CS716
    6. 6. Search Application vs Internet SearchSearch Application Internet (e.g. Bing, Altavista) Unique result  Multiple results Target Audience  Target Everybody Known users  Anonymous (usually) Complex Formats  Limited Formats Finite Subjects  Multiple Subjects Relevant  Dictionary/History-based Suggest/Autocomplete Suggest/Autocomplete Rich UI  “10 Blue Links”#CS716
    7. 7. FAST Document Processing Engine#CS716
    8. 8. Document Processing Stages EntityExtraction Lemmatisation Synonyms Spy (Debug!) Data Post Pre-processing Manipulation Processing#CS716
    9. 9. Agenda Introductions The Anatomy of a Search Application When/Why Search-Driven UX Case Study: States Assembly Demo Lessons Learned & Top Tips#CS716
    10. 10. When/Why Search-Driven UX? Unknown keywords  Start with refiners Manual metadata  “People” issues QueryingAcross Site Collections Everybody is searching for something  User Context#CS716
    11. 11. Simple Business Case 1,000 Person Company Each Employee loses 1hr a month “searching” = 12,000 hrs/year 25% improvement with a Search Application (Conservative Estimate!) ROI in 1 year if cost < ~£150,000#CS716
    12. 12. Search Driven Examples E.g. Dell, Amazon, Globrix  Known Content & Single Target Audience  Unique Result Desired Legal Sector  Cases/Matters  eDiscovery R&D  Avoid expensive duplication#CS716
    13. 13. Agenda Introductions The Anatomy of a Search Application When/Why Search-Driven UX Case Study: States Assembly Demo Lessons Learned & Top Tips#CS716
    14. 14. States Assembly States of Jersey Government records since 1981 Minutes, Propositions, Statements, Votes, Hansards ~17,000 unstructured .doc, .pdf Migration from a specialised custom ASP.NET solution#CS716
    15. 15. Infrastructure Architecture3 FAST Servers 2 SharePoint Farms  1 Content Authoring (internal)  1 Content Deployment (public)#CS716
    16. 16. Infrastructure Diagram#CS716
    17. 17. Methodology & Objectives Always query FAST (FQL) where possible No SharePoint API or CAML calls Relevant Autocomplete Best Hit & Hit Highlighting should link to specific location in the document#CS716
    18. 18. Hansard Official transcript of everything States Members say during question time, statements and debates in Jersey’s States Assembly Up to 20Mb .doc & .pdf Up to ~130 pages Title vs Name#CS716
    19. 19. Users/Roles Elected Politicians (~50) Power Users (~50) Employees (~7,000) Citizens (~98,000)#CS716
    20. 20. Problems Encountered GrevilleBathe Fund Lack of well-defined test cases  How fuzzy? Comparison with previous system Irrelevant autosuggest Synonyms#CS716
    21. 21. All States of Jersey Documents since 1981STATES ASSEMBLY
    22. 22. How we did itA lot of synonyms  Continue to build from search history Custom regular expressions Custom pipeline stage: entity extraction Rank profile prioritising proximity & body Relevant cached autocomplete Feedback form#CS716
    23. 23. #CS716
    24. 24. Agenda Introductions The Anatomy of a Search Application When/Why Search-Driven UX Case Study: States Assembly Demo Lessons Learned & Top Tips#CS716
    25. 25. Lessons Learned & Top Tips Define all user/role use cases Analyse all content carefully Populate Synonyms from search history Did You Mean? • Spell Tuning > Spell Checking Wireframes (e.g. balsamiq) to define the User Interface Spy Stage to debug Autocomplete with relevant content Use Feedback Form#CS716
    26. 26. Summary Plan for Search up-front Understand & define roles/personas/content Consider FAST for pipeline extensibility, rank tuning & personalisation Beware of upgrade/migration#CS716
    27. 27. Thank you for attending! @gusfraser #CS716#CS716
    28. 28. References & Useful Links Morville/dp/0596802277 SharePoint/dp/0735662223 val-guide-fast-search-server-2010-for-sharepoint-en-us.aspx