The slide deck from my talk at the virtual Haystack 2020 conference via meetup.

1. Search Product Manager: Software PM vs. Enterprise PM
What does that dam PM do?
John T. Kane
JT Kane Consultants
Email: jtkane@gmail.com
Twitter: @EnterpriseSrch
https://www.linkedin.com/in/jtkane/

2. Agenda
I) Introduction – My Search PM Journey
II) Search Use Cases as a Horizontal Technology
III) The Roles of Search and Product Manager within Business
IV) Search & Business as Partners
V) Thoughts on Search & PM Futures…

3. Introduction – My Search PM Journey (1)
I didn’t emerge as a fully-fledged Search PM as every PM has their path.
1998 – Microsoft: SQL Server 7.0 alpha adds Full-text Search
2007 – IntelliSearch (Norway-based) Search Startup (Search Evangelist)
2008 – FAST ESP (Norway-based) acquired by MSFT – Sales Engineer
2010 – Editor/Contributor Book: Pro. Microsoft Search, published.

4. Introduction – My Search PM Journey (2)
I didn’t know that a Search Product Management was my path.
2010 – Joined HP Inc. as BI/Architect & Search Product Manager
2013 – Joined Lucidworks as PM for Fusion (2012: life-changing event).
2015 – Started at Staples as Search PM for migration from Endeca.
2016 – Joined HP Enterprise as Site Search PM using GSS.
2018 – Joined Voyager Search as PM and Solution Architect.
2018 – Joined Zoosk (Online Dating site) with NO Search UI.
2019 – Started at Getty Images as Search PM improve Search Mission.
2020 – Freelance Search PM & Volunteer PM for COVID19 app.

5. Search Use Cases as a Horizontal Technology(1)
Search Technology is Horizontal much like Relational Databases.
As Websites specialize, so too does Search Use Cases get specialized.
Search Use Cases can be broken down into three broad Categories:
Public facing: Site Search (SEO’s Internal Search), eCommerce Search
Dell Laptop Search: https://www.dell.com/koa/search?q=laptop
Getty Images Skiing Search: https://www.gettyimages.com/photos/skiing
HPE Site Search: https://www.hpe.com/us/en/search-results.html?page=1&q=gen%2010&autocomplete=0
Non-Public facing: Intranet Search, Enterprise Search
https://www.microsoft.com/en-us/itshowcase/redefining-the-intranet-site-experience-with-sharepoint-in-office-365 (2019)
https://destwo.net/portfolio/case-study-microsoft-web/ (circa 2014) – Microsoft’s Intranet: MSW
Hybrid or Specialized Search: GeoSpatial Search, Collaborative Search
VoyagerODN Geospatial: https://www.voyagersearch.com/odn-search
Multi-Person or Collaborative Search: SearchTogether, and others.

6. Dynamic Personalized NextGen Search
Dynamic Relevance
Dynamic Users
Dynamic Queries
Dynamic Documents
Dynamic Information Needs
Users change behavior over
time, user history
Topic Trends, Filtering,
document content change
User perceived relevance
changes
Changing query,
reformulation, Mission
Information needs evolve over time
Next Gen.
Search
Engine
All Access points secured
All Device form factors

7. Help! I’m stuck in the head = lost value in the “Long Tail”
Source: Last.fm Dataset
72%

8. The Roles of Search and Product Management
within the Business (1)
As the Internet and Online Business grew, so too did the need for Technologists who understood the business
use cases. The Product Management Role varies by Business & Industry. NO Standards!
My "Five Things" or Search Product Feature Framework
1. Highly-Available (HA / AlwaysOn) = NO one point of failure
2. Fast! (Highly-performant)
3. Relevancy = Topical relevant / domain-specific / content-specific
4. Social / Behavioral / Interactive == Improves Relevancy (external signals, clicks, impressions, metrics)
5. Feedback = Always improving / Real-time feedback
PM Building new Product or "Five Things“ for a Product Manager
1. Understand your target audience, really well.
2. Understand “why?”
3. Drill down.
4. Make a task as simple as possible
5. Forget about feature parity with competitors & Track all pain points.

9. The Roles of Search and Product Management
within the Business (1)
A brief list of Five Traits of Successful Search Product Managers. I’m explicitly avoiding the “good vs. great” PM conversation.
1. Empathy
2. Passion for [Search] products
3. Experimenter’s Mindset / Intellectual Honesty
4. Subject Matter Expert / Domain Knowledge
5. Aspires to build great user experiences

10. Product Management Roles: Software vs. Enterprise
There are overlaps between these Roles and the below is an unordered list of differences (your mileage may vary)
Always have Empathy for your Customers. Know them, Interview them, Survey Them!
Software Search Product Manager
end-product is Software or Service (SaaS)
Software can be Search or other domains.
Software Sales/Subs drive Business Revenue.
Interact with Sales & Sales Engineering.
Interact with Consultants / Pro. Services.
Work with UX Designers, UI Developers.
Work with Backend Search Engineers.
Everyone is a Software Search Expert.
Passion for multi-Search Use Cases.
Empathy for End-Customers, Feel the Pain.
Multi-Use Case experienced KPI’s OKR’s.
Aspires to be great, but always humble.
Able to explain complex topics, simply.
Enterprise Product Manager, Search
Understand the Business drivers & customers.
Understands how Search fits within Business.
SME and Domain (Search) Expert
Interact with internal Stakeholders.
Stakeholders can have conflicting objectives.
Understand your Manager’s goals.
Over communicate with your Manager.
Work with all levels of Engineering Staff.
Understand the Business’ End-Customers
Research Search Uses, Tech & Experiences.
Know Customer Journey & Searcher’s Path.
Ensure Search Query Log & Click-stream exist.
You are your Business’ Search Expert.

11. Sample Search Metrics & KPIs (1)
Search KPI Goals
Increase in visitor-to-site interaction & page depth click-through rate
Increase in visitor clicking through to a linked second page
Increase in visitor highly-engaged to third or fourth linked page depth
Searcher Behavior Metrics
1. Most Popular Searches (must not be viewed in isolation, but in context with % of visitors who use Site Search)
2. Zero Results Searches
3. Top Selected Documents
4. Slow Search Queries
5. Most Popular Facets Value Selected
6. Most Popular Facets Displayed
7. Top X Documents per Top Y Searched Terms of Phrases
Content-based Metrics
8. Crawler Statistics (success/failure) – not possible with GSS
9. Search Business Rules (landing pages or re-directs)
10. Term Frequency
11. Search Query-based Duplicate Content Detection
12. QPS – Queries Per Second – helps determine if we have slow searches
13. Collections – not the right name for GSS, but how can we know in advance that our content is crawled by GSS?
14. Error / Fault Reporting 11

12. Sample Search Metrics & KPIs (2)
Information Find Conversion Rate - Total Number of Visits to the Goal Page / Total Visits
a) This KPI is a variation on the order conversion rate in which we simply loosen the definition of “order” to reflect visitors seeing critical goal pages
b) call this KPI the “image find rate” if the "goal" is for customers who do "research" but do not actually download. (related to searcher intent)
c) consider this KPI in tandem with average page views per visit to monitor for an artificially high conversion rate, essentially visitors viewing multiple overlap
pages
Search to Purchase Conversion Rate - Total Orders Attributed to Searchers / Total Number of Visits to a Search Results Page
a) tracks how searching visitors differentially convert into customers to help understanding the real value investing in new search technology.
b) this too is special case of the order conversion rate, similar to Information Find Conversion Rate above.
Search Results to Site Exits Ratio - Total Site Exits from Search Results Page / Total Number of Visits to a Search Results Page
a) Similar to percent zero yield search results, the ratio of search results to site exits ratio will help you understand whether visitors find Site Search useful.
b) Roughly the opposite of the search to purchase conversion rate, this ratio can help understand how search is driving visitor failure and dissatisfaction.
c) co-present this KPI with average searches per visit, percent zero results searches and percent zero yield searches to provide the proper context.
Percent Visitors Using Search - Total Number of Visitors Who See at Least One “Search Results” Page / All Visitors
a) this KPI is relatively static, any dramatic changes are likely describing some problem caused by recent changes to the site.
Percent Zero Result Searches - Total Number of “Zero Result” Searches / All Search Results
a) Both the numerator and denominator in this calculation should be measured in page views, not visits or visitors (may search multiple times during a visit)
12

13. Sample Search Metrics & KPIs (3)
Percent Zero Yield Searches - Total Number of Search Results Pages from which Visitors Did Not Click a Search Result / All
Search Results
a) If it is easier to track that visitors are clicking on a search result link, you can use that number as well in the numerator (based upon pageview
basis)
b) 1.00 – (Total Number of Search Results Pages from which Visitors Did Click a Search Result / All Search Results) = Percent Zero Yield Searches
(pageview basis)
Average Page Views per Visit - Search technology: If your search functionality is poor, visitors may be forced to click to look for
information.
a) Conversely, if your search functionality is good, visitors may be leveraging search, thusly reducing the number of pages viewed.
b) average page views per visit (diagnose problems) Site Search: percent visitors using search (see above), percent “zero result” searches & average searches
per visit.
Average Searches per Visit - Total Number of Searches (Page Views) / Total Visits
a) present this along with percent visitors using search to provide context. The goal is to monitor the visitor’s relationship with deployed search technology.
Popular Site Search terms Not visible in Site structure (Website Percent High, Medium and Low Time Spent Visits & diagnose
problems)
a) site search logs, looking for commonly searched for terms that may highlight visitor interests not clearly represented in the site’s navigational structure.
Popular Site Search terms Not visible in Site Clicks - (Percent High, Medium and Low Click Depth Visits & diagnose problems)
a) site search logs, looking for commonly searched for terms that may highlight visitor interests not clearly represented in the site’s navigational structure.
13

14. Search & Data Science Prioritization Process
Process Planning Notes:
Quad Matrix: Effort vs. Business vs. Risk (LoE) = Prioritized Features.

15. Search Roadmap – Project Constraints.
Triangle: Effort vs. Time vs. Money (budget) = Limited Resources.

16. What makes a good recommendation? A: RecSys KPIs
• Accuracy
• Coverage
• Long Tail
• Serendipity
• Novelty
• Search and Recommendations are becoming one..

17. Discovery / Exploratory
17

18. Dynamic Searchers: Novice vs. Expert
18

19. Knowledge Graph for Users, Clicks & Image Taxonomy
19

20. Aspects of a Search Architecture
Text-Based Search Ranking (words to match words) – add why.
Define language-specific Noun phrase use case / why matters most. Or biggest impact
Text Search Models (TF/IDF or BM25 to LTR)
Textual Query Preprocessing (via Query Pipeline)
Query Expansion (add synonyms & paraphrases)
Stemming Algorithm (porter vs. lemmatization)
Stopword Removal (some to none)
N-Gram Query Segmentation (Fuzzy or part of words)
Part-of-Speech Tagging (POS for NLP)
Text Sources (add metadata via Indexing Pipeline)
Query Example-Based Search Ranking (Reverse image)
Concept-Based Search Ranking (GKG & extend, content-side)
Query-Concept Mapping (Query-side)

21. Example Search Roadmap Schedule (20% confidence)
Exit Walk: ES-based
Reciprocal
Recommender & Train
Data.
Jan. 2020 Apr. 2020 May 2020
Train Engineers in
on Solr + ML L2R
Develop / Deploy Text Profile, Clicks,
Tagging Features
Monitor Member Usage
Patterns & KPI’s
Determine exit metrics
for Walk phase
Jun. 2020Mar. 2020Feb. 2020
Connect to
Front-end UI
Architect NLP + ML-
based Recommender
Test multiple Search
with same content
Plan Run Phase
Features & Benefits. Run: TBD
Crawl: Standup new
Solr-based Search
Walk 2.0: New ES-
based Features
Jul. 2020
Walk 1.0: ES-based
Recommender +
Business Features

22. References: Books, Blogs &Bits.. (1)
Enterprise Search: Enhancing Business Performance (2nd edition) - He's expanded this to more use cases
https://www.amazon.com/Enterprise-Search-Enhancing-Business-Performance/dp/1491915536/
https://www.cmswire.com/author/martin-white/ - he also writes a column for CMS Wire
Designing the Search Experience: The Information Architecture of Discovery
https://www.amazon.com/Designing-Search-Experience-Information-Architecture/dp/0123969816
Search Patterns
https://www.amazon.com/Search-Patterns-Discovery-Peter-Morville/dp/0596802277
Designing Search: UX Strategies for eCommerce Success
https://www.amazon.com/Designing-Search-Strategies-eCommerce-
Success/dp/0470942231/ref=pd_lpo_14_t_2/144-5441348-2297159
Search Analytics for your Site
https://www.amazon.com/dp/B005EI86HC/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1

23. References: Books, Blogs &Bits.. (2)
Medium.com - is a good source of both business and technical search topics
https://medium.com/search?q=%22ecommerce%20search%22
https://medium.com/search?q=solr
https://medium.com/search?q=elasticsearch
IR or Search Conferences - most papers should be online.
https://sigir.org/sigir2020/
https://ecir2020.org/
https://sigir.org/chiir2020/
Real-Time Visual Navigation in Huge Image Sets using Similarity Graphs 2019
Demo site: https://wikiview.net/ - this is Image Exploratory Search UI site that I spoke about.
Struggling or Exploring? Disambiguating Long Search Sessions
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/HassanWSDM14.pdf
Exploratory search: user behaviour and search engine adaptation by Alan Medlar - Feb 2020 Free access
https://dl.acm.org/doi/abs/10.1145/3352683.3352687
Challenges in Supporting Exploratory Search through Voice Assistants - March 2020
Note: this one is related to your question about Conversational Search, re: Alexi, Echo etc.
https://arxiv.org/pdf/2003.02986.pdf

24. References: Books, Blogs &Bits.. (3)
Serendipity Search
A new online startup serendipitous service to put serendipity back into your life!
http://www.serenflipity.com/#intro
The serendipity society - Serendipitous Resources & papers
https://theserendipitysociety.wordpress.com/resources/
https://twitter.com/ser_soc
Planned Luck: How Incubators Can Facilitate Serendipity for Nascent Entrepreneurs Through Fostering
Network Embeddedness - May 2020 - By Christian Busch
https://journals.sagepub.com/doi/pdf/10.1177/1042258720915798
Observing Serendipity in Digital Information Environments - 2015
https://www.google.co.uk/search?domains=informationr.net&q=serendipity - review of IR papers open
Searching Linked Data with a Twist of Serendipity - 2017
http://www-di.inf.puc-rio.br/~casanova//Publications/Papers/2017-Papers/2017-CAiSE-Eichler.pdf

25. Some Thoughts on Search & PM Futures…
Predicting the Future is difficult. Consider this a list of Dots to Connect for future Search Innovation.
Always be Learning, Growing New Skills, Be open minded, Find your Niche.
Search Technology – Dots to Connect.
Search as a Service (SaaS) is foundational.
Learning to Rank (LTR) is a critical component
Learning to Rank tools will become common.
Online Learning to Rank (OLTR) is Self-Learning.
Deep Learning for Search will have tires kicked.
Carousel UI KPI’s are different than Clicks.
Open Source Solr & ElasticSearch will evolve.
Graphs (Knowledge, Interest, eCommerce)
Niche Search & General Search will co-exist.
Dynamic Search will become more relevant.
Always have built-in Metrics & Analytics.
Aspires to be great, but always humble.
Able to explain complex topics, simply.
Product Manager – Dots to Connect.
Agile Framework will evolve, become practical.
Other Frameworks will emerge, Jobs-to-be-Done.
Always be Learning, Curious, Open-minded.
Stakeholders will always be a Challenge.
Business Location-based work will evolve fast.
Remote Work requires over-communicating
Switch Work hours with Remote-based Time zones.
Resume-based Features are Tempting, Resist!
Your End-Customers Patterns will change more.
Research new Search emerging Technology.
Know Customer Journey & Searcher’s Path.
Search Query Log + Click-stream is your Gold Mine.
You are your Business’ Search Expert.

26. My Search Journey as a Product Manager
Thank You!
John T. Kane
JT Kane Consultants
Email: jtkane@gmail.com
Twitter: @EnterpriseSrch
https://www.linkedin.com/in/jtkane/

27. Background slides below.

28. Search: Consider as the front door
• Today’s web search is built on the
foundation of classic IR
• Treat documents as bags of
keywords that are matched against
user queries through statistical
techniques
• Inverted indexes and TF/IDF are at
the heart of today’s search engines
• Google’s PageRank uses link
structures and dynamic ranking to
improved relevance of results.
• Knowledge Graph & RankBrain
build upon these structures.Inverted Index
w1 → d1 d4 d50
w2 → d3 d10 d32
….
wN → d11 d20 d2003
28

29. Contextual Query Understanding
• User queries can be reformulated and intent inferred based on different types of
context
• User: Short term (interest) and Long term (knowledge) models
• Task: Current session’s objective explicitly (provided by user) or implicitly (derived from page
visitations) captured from the user
• Community: Derived from the links between content and social graph
• Collaboration: Multiple users collaborating to perform a complex task
• Domain/Topic: Query classification leading to better identification of user’s information need
• Time-Space: Temporal and Geographic constraints that relevant information should satisfy
• Application: Constraints on the sources searched and information retrieved
• Web Search uses some of these query contexts to satisfy user’s information needs
• E.g. Geographic constraints in Local Search, User interests in personalized Search
• Limitations of the frameworks that capture, represent and use what is known to the
user
• Each user activity provides some information or knowledge about the topic of interest to the user
and about the user
• Audience intelligence classifies users to a predefined set of categories capturing their interests
• Search history captures information on user’s short-term tasks/interest
• Interpreting such information from different sources provides the context for understanding user query.
29

30. Building NextGen Graph
Discover User-to-Image Graph Interest
Search Users
• Find users/experts with interests in certain topics
• Search for topics in the Entity Graph and project the
results to User Graph
Discover Users
• Find groups that a user would be interested in
• Project user interests from Entity Graph to User Graph to
find group membership.
• Find users by tagging/annotating KB content
• Project tagged content to the User Graph
• User group discovery
• Cluster users based on the semantic clustering of their
interests
Search Knowledge
• Improving Selection and Relevance
• Exploit User interests projected to Entity Graph to select
and rank search results
• Augmenting Search Results
• Recommending related knowledge and resources for a
given query
Discover Knowledge
• Community Interests and Trends
• Cluster and track concepts of interest to a contributor
community
• Search Patterns and Templates
• Repurpose patterns in user navigation as templates for
future search.
30

31. Screenshot for “Expanded Query Terms” usage via Vectors