On-site search design patterns for e-commerce: schema structure, data driven ranking & more

1 / 19
On-site search design patterns for e-commerce: 
schema structure, data driven ranking & more
Dr. Martin Loetzsch, Project A Ventures

Krešimir Slugan, Contorion
https://goo.gl/xJr9Jshttp://project-a.github.io/on-site-search-design-patterns-for-e-commerce/

! ‣
2 / 19https://goo.gl/xJr9Js

! ‣
A naive product centric approach
‣Don’t do this!
{ 
"name": "Fäustel DIN6475 2000g Eschenstiel FORTIS", 
"staple-name": "Fortis Fäustel, mit Eschen-Stiel", 
"description": "Fäustel DIN 6475 Stahlgeschmiedet, Kopf schwarz
lackiert, Bahnen poliert, doppelt geschweifter Eschenstiel mit ozeanblau
lackiertem Handende. SP11968 SP11968", 
"preview_image": "faeustel-din6475-2000g-eschenstiel-fortis-21049292-0-
JlHR5nOi-l.jpg", 
"categories": [ 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
"final_gross_price": 1149, 
"final_net_price": 1003, 
"url": "/handwerkzeug/fortis-faeustel-mit-eschen-stiel-SP11968", 
"manufacturer": "Fortis", 
"hammer_weight": 2000 
}
‣ What are the attributes of your documents?

! ‣
‣Don’t do this!
‣ Huge complexity in schema design and query generation

• Queries need to explicitly list the attributes they want to operate on

• Diﬃcult to use the same attribute in diﬀerent operations

• Whenever new attributes appear, the schema needs to be extended
{ 
JlHR5nOi-l.jpg", 
"categories": [ 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
}

! ‣
‣Don’t do this!



‣ Full potential of data is usually not used
{ 
JlHR5nOi-l.jpg", 
"categories": [ 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
}

! ‣
‣Don’t do this!



‣ Full potential of data is usually not used
‣ Your search shouldn’t need to know that there is such a thing
as the weight of a hammer
{ 
JlHR5nOi-l.jpg", 
"categories": [ 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
}

{ 
"type": "staple", 
"search_result_data": { 
"sku": "SP11968", 
"name": "Fortis Fäustel, mit Eschen-Stiel", 
JlHR5nOi-l.jpg", 
"number_of_products": "4", 
"final_gross_price": "822", 
"final_net_price": "691", 
"base_gross_price": null, 
"base_price_unit": null, 
"url": "/handwerkzeug/fortis-faeustel-mit-eschen-stiel-SP11968" 
}, 
"search_data": [ 
{ 
"full_text": " 21049289 4317784792714 04317784792714 Fäustel DIN
6475 Stahlgeschmiedet, Kopf schwarz lackiert, Bahnen poliert, doppelt
geschweifter Eschenstiel mit ozeanblau lackiertem Handende. SP11968 SP11968", 
"full_text_boosted": " Fortis Fäustel DIN6475 1000g Eschenstiel FORTIS
1000 Fäustel Handwerkzeug Hammer Fäustel Fortis Fäustel, mit Eschen-Stiel
Fortis Fäustel, mit Eschen-Stiel", 
"string_facet": [ 
{ 
"facet-name": "manufacturer", 
"facet-value": "Fortis" 
}, 
{ 
"facet-name": "hammer_weight", 
"facet-value": "1000" 
} 
], 
"number_facet": [ 
{ 
"facet-name": "final_gross_price", 
"facet-value": 822 
} 
] 
}, 
{ 
Usage driven schema & document structure
‣ How do you want to use the attributes of your documents?  ‣
‣Do this!

{ 
"sku": "SP11968", 
JlHR5nOi-l.jpg", 
}, 
"search_data": [ 
{ 
{ 
}, 
{ 
} 
], 
{ 
} 
] 
}, 
{ 
‣Do this!
search result rendering

{ 
"sku": "SP11968", 
JlHR5nOi-l.jpg", 
}, 
"search_data": [ 
{ 
{ 
}, 
{ 
} 
], 
{ 
} 
] 
}, 
{ 
‣Do this!
full-text search

{ 
"sku": "SP11968", 
JlHR5nOi-l.jpg", 
}, 
"search_data": [ 
{ 
{ 
}, 
{ 
} 
], 
{ 
} 
] 
}, 
{ 
{ 
‣Do this!
faceted navigation

{ 
} 
], 
{ 
} 
] 
} 
], 
"completion_terms": [ 
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
"suggestion_terms": [ 
"Fortis Fäustel, mit Eschen-Stiel" 
], 
"number_sort": { 
"final_gross_price": 822 
}, 
"string_sort": { 
"name": "Fortis Fäustel, mit Eschen-Stiel" 
}, 
"scores": { 
"top_seller": 0.91, 
"pdp_impressions": 0.38, 
"sale_impressions_rate": 0.8, 
"data_quality": 0.87, 
"delivery_speed": 0.85, 
"random": 0.75, 
"stock": 1 
}, 
"category": { 
"direct_parents": [ 
"bpka" 
], 
"all_parents": [ 
"bost", 
"boum", 
"boun", 
"bpka" 
], 
‣Do this!
completion

{ 
} 
], 
{ 
} 
] 
} 
], 
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
], 
"number_sort": { 
}, 
"string_sort": { 
}, 
"scores": { 
"random": 0.75, 
"stock": 1 
}, 
"category": { 
"bpka" 
], 
"all_parents": [ 
"bost", 
"boum", 
"boun", 
"bpka" 
], 
‣Do this!
spell-checking

{ 
} 
], 
{ 
} 
] 
} 
], 
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
], 
"number_sort": { 
}, 
"string_sort": { 
}, 
"scores": { 
"random": 0.75, 
"stock": 1 
}, 
"category": { 
"bpka" 
], 
"all_parents": [ 
"bost", 
"boum", 
"boun", 
"bpka" 
], 
‣Do this!
static sorting

{ 
} 
], 
{ 
} 
] 
} 
], 
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
], 
"number_sort": { 
}, 
"string_sort": { 
}, 
"scores": { 
"random": 0.75, 
"stock": 1 
}, 
"category": { 
"bpka" 
], 
"all_parents": [ 
"bost", 
"boum", 
"boun", 
"bpka" 
], 
‣Do this!
dynamic result ranking

{ 
} 
], 
{ 
} 
] 
} 
], 
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer", 
"Fäustel" 
], 
], 
"number_sort": { 
}, 
"string_sort": { 
}, 
"scores": { 
"random": 0.75, 
"stock": 1 
}, 
"category": { 
"bpka" 
], 
"all_parents": [ 
"bost", 
"boum", 
"boun", 
"bpka" 
], 
‣Do this!
‣ Analyse attributes diﬀerently depending on their use

• Best representation for each use case

• Intelligence moves to importers

‣
Faceted navigation

‣
Faceted navigation
Facet counting

‣
Faceted navigation
Filtering

Faceted navigation: the standard way
‣ Put all ﬁlterable attributes in unanalyzed document ﬁeld:  ‣
‣Does not scale to open-ended number of facets
"string_facets" : { 
"hammer_weight": "2000", 
"hammer_color": "Red" 
}

Faceted navigation: the standard way
‣ Put all ﬁlterable attributes in unanalyzed document ﬁeld:  ‣
‣Does not scale to open-ended number of facets
"string_facets" : { 
"hammer_weight": "2000", 
"hammer_color": "Red" 
} 
"aggregations": { 
"facet_manufacturer": { 
"terms": { 
"field": "string_facets.manufacturer" 
} 
}, 
"facet_hammer_weight": { 
"terms": { 
"field": "string_facets.hammer_weight" 
} 
}, 
"facet_hammer_color": { 
"terms": { 
"field": "string_facets.hammer_color" 
} 
} 
}
‣ Explicitly list all facets in the query:

! ‣
Generic faceted navigation
/ 19https://goo.gl/xJr9Js
‣Nicely scales to thousands of facets
‣ Put facet names as values in
documents:
"string_facets": [ 
{ 
}, 
{ 
}, 
{ 
"facet-name": "hammer_color", 
"facet-value": "Red" 
} 
]
6

! ‣
documents:
{ 
}, 
{ 
}, 
{ 
} 
]
‣ Mark string_facets ﬁeld as “nested”
in document:
"string_facets": { 
"type": "nested", 
"properties": { 
"facet-name": { 
"type": "string", 
"index": "not_analyzed" 
}, 
"facet-value": { 
} 
} 
}
6

documents:
{ 
}, 
{ 
}, 
{ 
} 
]
‣ Nested query with nested aggregation:
"agg_string_facet": { 
"nested": { 
"path": "string_facets" 
}, 
"facet_name": { 
"terms": { 
"field": "string_facets.facet-name" 
}, 
"facet_value": { 
"terms": { 
"field": "string_facets.facet-value" 
} 
} 
} 
} 
} 
} 
} 
 
!
in document:
"properties": { 
"facet-name": { 
}, 
"facet-value": { 
} 
} 
}
6

documents:
{ 
}, 
{ 
}, 
{ 
} 
]
"nested": { 
}, 
"facet_name": { 
"terms": { 
}, 
"facet_value": { 
"terms": { 
} 
} 
} 
} 
} 
} 
} 
 
!
in document:
"properties": { 
"facet-name": { 
}, 
"facet-value": { 
} 
} 
}
6
"filter": { 
"nested": { 
"path": "string_facets", 
"filter": { 
"bool": { 
"must": [ 
{ 
"term": { 
"facets.facet-name": "hammer_weight" 
} 
}, 
{ 
"terms": { 
"facets.facet-value": [ 
"2000" 
] 
} 
} 
] 
} 
} 
} 
}
 
!
‣ Nested ﬁlter:

documents:
{ 
}, 
{ 
}, 
{ 
} 
]
"nested": { 
}, 
"facet_name": { 
"terms": { 
}, 
"facet_value": { 
"terms": { 
} 
} 
} 
} 
} 
} 
} 
 
!
in document:
"properties": { 
"facet-name": { 
}, 
"facet-value": { 
} 
} 
}
6
"filter": { 
"nested": { 
"path": "string_facets", 
"filter": { 
"bool": { 
"must": [ 
{ 
"term": { 
"facets.facet-name": "hammer_weight" 
} 
}, 
{ 
"terms": { 
"facets.facet-value": [ 
"2000" 
] 
} 
} 
] 
} 
} 
} 
}
 
!
‣ Nested ﬁlter:
‣ The front-end doesn’t need to know which facets exist

Full-text search
‣

Text analysis
‣ Pre-built Elasticsearch analyzers work, but won’t give the best user experience
‣Invest time in in building proper text analysis chains

Text analysis
Stiel für Schonhammer 800g

Text analysis
Whitespace tokenizer

Text analysis
Stiel für Schonhammer 800g 800 g
Word-delimiter ﬁlter

Text analysis
stiel für schonhammer 800g 800 g
Lowercase ﬁlter

Text analysis
Lowercase ﬁlter
stiel für schonhammer plastikhammer 800g 800 g
Synonym ﬁlter

Text analysis
Lowercase filter
Synonym filter
stiel für schonhammer schon hammer plastikhammer plastik hammer 800g 800 g
Decompounder filter

Text analysis
Lowercase filter
Synonym filter
stiel fur schonhammer schon hammer plastikhammer plastik hammer 800g 800 g
German normalization filter

Text analysis
Lowercase filter
Synonym filter
stiel fur schonhammer schon hammer plastikhammer plastik hammer 800g 800 g
German normalization filter
stiel fur schonhamm scho hamm plastikhamm plastik hamm 800g 800 g
German stemmer filter

Text relevance
‣ Split searchable text by importance in documents:
‣Improve relevance by combining multiple analyzers
"full_text": "21049291 4317784792738 Fäustel DIN
6475 Stahlgeschmiedet, Kopf schwarz lackiert, Bahnen poliert,
doppelt geschweifter Eschenstiel mit ozeanblau lackiertem Handende
SP11968", 
"full_text_boosted": "Fortis Fäustel DIN6475 2000g Eschenstiel FORTIS
2000 Fäustel Handwerkzeug Hammer Fäustel Fortis Fäustel, mit Eschen-
Stiel Fortis Fäustel, mit Eschen-Stiel"

Text relevance
SP11968", 
Stiel Fortis Fäustel, mit Eschen-Stiel" 
"properties": { 
"full_text": { 
"index_analyzer": "full_text_index_analyzer", 
"search_analyzer": "full_text_search_analyzer", 
"fields": { 
"no-decompound": { 
"index_analyzer": "full_text_index_analyzer_no_decompound", 
"search_analyzer": "full_text_search_analyzer_no_decompound" 
}, 
"no-stem": { 
"index_analyzer": "full_text_index_analyzer_no_stem", 
"search_analyzer": "full_text_search_analyzer_no_stem" 
} 
} 
} 
}
‣ Apply multiple analyzers to full-text ﬁelds in mapping:

Text relevance
SP11968", 
Stiel Fortis Fäustel, mit Eschen-Stiel" 
"properties": { 
"full_text": { 
"index_analyzer": "full_text_index_analyzer", 
"search_analyzer": "full_text_search_analyzer", 
"fields": { 
"no-decompound": { 
"index_analyzer": "full_text_index_analyzer_no_decompound", 
"search_analyzer": "full_text_search_analyzer_no_decompound" 
}, 
"no-stem": { 
"index_analyzer": "full_text_index_analyzer_no_stem", 
"search_analyzer": "full_text_search_analyzer_no_stem" 
} 
} 
} 
}
‣ Apply multiple analyzers to full-text ﬁelds in mapping:
"query": { 
"bool": { 
"must": [ 
{ 
"multi_match": { 
"fields": [ 
"search_data.full_text_boosted^7", 
"search_data.full_text^2" 
], 
"type": "cross_fields", 
"analyzer": "full_text_search_analyzer", 
"query": "hammer" 
} 
} 
], 
"should": [ 
{ 
"multi_match": { 
"fields": [ 
"search_data.full_text_boosted.no-stem^7", 
"search_data.full_text.no-stem^2" 
], 
"operator": "OR", 
"analyzer": "full_text_search_analyzer_no_stem", 
} 
}, 
{ 
"multi_match": { 
"fields": [ 
"search_data.full_text_boosted.no-decompound^7", 
"search_data.full_text.no-decompound^2" 
], 
"operator": "OR", 
"analyzer": "full_text_search_analyzer_no_decompound", 
} 
} 
] 
} 
}
‣ Combine
subqueries
to get more
relevant
results

Data driven ranking
! ‣
‣Align formula with business goals

Data driven ranking
! ‣
Result order

Data driven ranking
! ‣‣ Add normalized scores to document that each reﬂect a
heuristic what a “good” result is:
"scores": { 
"random": 0.75, 
"stock": 1 
}
Result order
most scores left out due to their sensitive nature

Data driven ranking
"scores": { 
"random": 0.75, 
"stock": 1 
}
‣ Concrete scores depend on business
Result order

Data driven ranking
"scores": { 
"random": 0.75, 
"stock": 1 
}
‣ Concrete scores depend on business
"query": { 
"filtered": { 
"query": { 
"function_score": { 
"score_mode": "first", 
"boost_mode": "replace", 
"query": {}, 
"functions": [ 
{ 
"script_score": { 
"script": " 
(1 + _score ** 0.5) 
* doc['scores.stock'].value 
* (0.1 * doc['scores.random'].value
+ 0.3 * doc['scores.top_seller'].value
+ 0.1 * doc['scores.pdp_impressions'].value 
+ 0.2 * doc['scores.sale_impressions_rate'].value
+ 0.1 * doc['scores.data_quality'].value
+ 0.3 * doc['scores.delivery_speed'].value)" 
} 
} 
] 
} 
} 
}
}
‣ Combine scores through algebraic formula in query:
(actual weights replaced by random values)

‣ Example I: expected margin
Normalized, evenly-distributed scores I
‣Look at the distribution of a measure across all products
-0.2 0.0 0.2 0.4 0.6 0.8 1.0
05000100001500020000
0.00.51.0
0.5+atan((x−0.1)0.05)π
computation of score expected_margin
#products
x = expected_margin
data from 2014

-0.2 0.0 0.2 0.4 0.6 0.8 1.0
05000100001500020000
0.00.51.0
0.5+atan((x−0.1)0.05)π
#products
x = expected_margin
data from 2014
distribution of the expected margin across all products

-0.2 0.0 0.2 0.4 0.6 0.8 1.0
05000100001500020000
0.00.51.0
0.5+atan((x−0.1)0.05)π
#products
x = expected_margin
data from 2014
score function

-0.2 0.0 0.2 0.4 0.6 0.8 1.0
05000100001500020000
0.00.51.0
0.5+atan((x−0.1)0.05)π
#products
x = expected_margin
data from 2014
mean
stdev
score function

‣ Example II: expected time of delivery
Normalized, evenly-distributed scores II
‣Finding meaningful scores is a requirement for a good ranking
100 200 300 400
020000400006000080000
0.00.51.0
0.5−atan((x−48)12)π
computation of score delivery_speed
#products
x = delivery_time
data from 2014

100 200 300 400
020000400006000080000
0.00.51.0
0.5−atan((x−48)12)π
#products
x = delivery_time
48 hours = neutral → 0.5
data from 2014

100 200 300 400
020000400006000080000
0.00.51.0
0.5−atan((x−48)12)π
#products
x = delivery_time
60 hours = acceptable → 0.25
data from 2014

100 200 300 400
020000400006000080000
0.00.51.0
0.5−atan((x−48)12)π
#products
x = delivery_time
bad
data from 2014

100 200 300 400
020000400006000080000
0.00.51.0
0.5−atan((x−48)12)π
#products
x = delivery_time
good
bad
data from 2014

Multi-term auto-completion
! ‣
‣Allow users to drill down to results by combining multiple attributes

! ‣
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer" 
]
‣ Put all terms to complete on in document

! ‣
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer" 
]
"completion_terms": { 
"analyzer": "completion_analyzer" 
}
‣ Leave terms more or less un-altered in mapping

! ‣
"Fortis", 
"1000", 
"1250", 
"1500", 
"2000", 
"Fäustel", 
"Handwerkzeug", 
"Hammer" 
]
"completion_terms": { 
"analyzer": "completion_analyzer" 
}
‣ Leave terms more or less un-altered in mapping
"aggs": { 
"autocomplete": { 
"terms": { 
"field": "completion_terms", 
"size": 100 
} 
} 
}
‣ Filter by input, aggregate by terms in query

Simple spelling suggestions
‣Very easy to build, and customers expect it
! ‣

! ‣
] 
‣ Put high quality suggestion terms in document:

! ‣
‣ Apply a simple analyzer that that splits text by
whitespace and that lowercases it:
"suggestion_terms" : { 
"type" : "string", 
"analyzer" : "term_suggestion_analyzer", 
}
] 

! ‣
‣ Apply a simple analyzer that that splits text by
whitespace and that lowercases it:
"suggestion_terms" : { 
"type" : "string", 
"analyzer" : "term_suggestion_analyzer", 
}
] 
‣ Look for most similar term by distance in query:
 
"suggest": { 
"spelling-suggestion": { 
"text": "hammmer", 
"term": { 
"field": "suggestion_terms", 
"size": 1 
} 
} 
}

Personalization: dynamic pricing
‣ User speciﬁc discounts per product or category
‣Scripts can be used for many other types of user speciﬁc personalisations
"query": { 
"script_fields": { 
"final_gross_price_discount": { 
"script": “
if (fixed_prices && fixed_prices[doc['sku'].value]) {
return fixed_prices[doc['sku'].value]
}
if(!discounts) {
return
};
 
def discount = 0;
for (String i : doc['discount_categories']) {
if(discounts[i] && discounts[i].value > discount) {
discount = discounts[i].value
}
}
if (discount > 0 && doc['prices.discount_gross_price_level_' + discount].value) {
return doc['prices.discount_gross_price_level_' + discount].value
}", 
"params": { 
"discounts": { 
"47": 5, 
"453": 2, 
"305": 7 
}, 
"fixed_prices": { 
"210417044": 9999, 
"128553": 100 
} 
} 
} 
} 
}

Index pages, not products
‣ Usage driven schemas can be used for any kinds of
searchable documents 
‣
‣Makes brands, categories, editorial content, etc. easily ﬁndable
{ 
"type": "cms-page", 
"id": "7", 
"title": "Versandkosten | contorion.de", 
"name": "Versandinformationen", 
"url": "/versandkosten" 
}, 
"search_data": { 
"full_text_boosted": [ 
"Versandinformationen" 
], 
"full_text": [ 
"Die Versandkosten innerhalb Deutschlands betragen
5,95€ pro Bestellung. Ab einem Warenwert von 50 € liefert
Contorion versandkostenfrei.Contorion.de liefert im
Moment nur nach Deutschland." 
] 
} 
}

Explicit attribute management
‣ Conscious decisions have to be made about how to use each attribute

‣ An explicit representation makes things more maintainable
‣Bonus points for providing an UI
attribute full text
full text
boosted
string
facet
number
facet
completion
terms
suggestion
terms
search
result data
description ✓
manufacturer ✓ ✓ ✓ ✓ ✓
name ✓ ✓ ✓ ✓
sku ✓ ✓
hammer_weight ✓ ✓ ✓
hammer_handle_length ✓ ✓
hammer_handle_material ✓ ✓ ✓
preview_image ✓
url ✓

Product management for search
‣ Best: concrete examples of how search should behave from a user perspective
‣If I search for a hammer, I want to find a hammer
search term issue / expected result dev comment
makita i would expect standard power tools on top (e.g., drilling machine), not a
jacket & a laser
Enhance WHF
akkuschrauber I would expect more search word suggestions, not just Akkuschrauber-Set PM: In specification
schleifscheibe no top sellers on top Add all categories, add popularity
score to category ranking
latt hammer should give back Latthammers first we might ne to recalibrate the
search a little bit
blindnietwerkzeug only gives back products called “Blindnietwerkzeug” but no Blindnietzange
or Blindnietmutter-Handgerät and so on
Please add Tokens to list
bohrmaschine
bosch
top categories should be the ones that actually have “Bohrmaschine” as
it’s name, not Bohrständer and stuff
fixed
tiefenmesschieber customers missing an “s” don’t get any results for TiefenmesSschieber very hard to fix
klopapier synonym for “Toilettenpapier” please create synonyms yourself
duebel doesn’t find products when “ä” is “ae” - that should happen for all umlaute fixed
Handwaschpaste doesn’t find category bug

‣ Structure documents around the usage of attributes

‣ Index pages, not products 
 
Conclusions


 
Conclusions


 
‣Thank you !
Conclusions

On-site search design patterns for e-commerce: schema structure, data driven ranking & more

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

On-site search design patterns for e-commerce: schema structure, data driven ranking & more