Product data is one of the most complex kind of data. Find out how state-of-the-art automated product data preparation is done using a smart mix of probabilistic and statistical methods, paired with latest algorithms of machine learning and artificial intelligence. Onedot's artificial intelligence (AI)-driven software helps businesses reduce manual work in product data management by 20x, speed up time-to-market by 90% and increase revenue up to 10%. Onedot makes this possible by radically improving data quality–automating the data integration, cleaning and categorisation process. Onedot features a user-friendly web interface designed for non-technical users. Unlike traditional rule-based ETL or scripting tools, Onedot is a ‘plug-and-play'​ system that continuously learns from business expert feedback and adapts to constantly changing data formats, structures, and nomenclatures. To learn more about Onedot, visit: https://www.onedot.com.

1. Processes, Methods and Algorithms
4. E-Commerce Meetup @ Liip
Automated
Product Data
Preparation

2. “Big Data is not
the new oil. Small
and messy data,
properly refined, is
the new fuel:
consumable data”
Tobias Widmer
CTO @ Onedot.com
We are hiring!

3. How do you decide
where to buy online?

4. Challenges of Product Data
• Some manufacturers and data suppliers are still in the ERP era
• Missing universal and globally accepted standards and taxonomies
• Complex network of suppliers, distributors and consumers and publishers
• Extreme variety, from food to consumer electronics to construction supplies
• Product data feeds are often inconsistent, incomplete, corrupt, or all of of them
• There is no one-stop shop to get good product data from across categories
• Negotiations with data suppliers regarding data quality are cumbersome
Product data is one of the most complex kind of data to handle.
A new approach to tame product data is required:
1. Stop relying on quasi-standards and embrace the mess out there
2. Take whatever data you can from different data sources
3. Have a reliable and up-to-date product data model on your end
4. Automated data integration and harmonisation to the maximum

5. Introducing Product Data Model
A PDM simplifies governance and enables automation.
A PDM summarizes which products are described with which attributes.
A PDM contains the values needed to provide consistent search filters for end-users.
A PDM indicates which information suppliers should provide.
A PDM helps to set up a comprehensive product data governance with clear processes.
A PDM is required for automated product data preparation.
1
2
3
4
5

6. Automating Data Preparation
General data preparation in data science is an iterative
process with lots of potential to automate.
Identify Sources
Profile Data
Clean Data
Unify Data
Schema Mapping
Entity Resolution
Segmentation
Post-Processing
Potential for Automation
Low Medium High Highest
Data Mining
Probabilistic Methods
Machine Learning
Artificial Intelligence
DataPreparationProcess

7. Going back to University
Automated data preparation pipelines combine probabilistic
methods with advanced machine learning algorithms.

8. The Case for Machine Intelligence
Probabilistic and statistical methods embrace the fuzziness
of product data and allow continuous automated learning.
Manual
• Slow and error-prone
• Limited scalability
• Time-consuming education
Past
Rule-Based
• Costly implementation
• Expensive maintenance
• No automated learning
Present
Artificial Intelligence
• Minimal training using data
• Copes with changing data structure
• Continuous, automated learning
Future

9. Product Data Preparation
6 disciplines for successful product data preparation.
Integration
Automated Schema Mapping
Integrate new products
independent of attribute names
Golden Record Creation
Combine different data records
without the need for a primary key
Transformation
Attribute Normalization
Adapt supplier data to your
conventions, formats and styles
Attribute Extraction
Extract attributes from product
names and descriptions
Categorisation
Product Grouping
Group products according to your
classifications and dynamic groups
Product Variants Identification
Identify product variants across
categories and product groups

10. Schema Mapping
Translate schema mapping into a large-scale segmentation
problem using linear-time blocking algorithms.
Input Schema Target Schema Confidence Level Top 3 Values
address->postcode Zip 98% CF11 8TW, SN5 8YW, BS10 7UH
model Variant Model Variant 96% D V8 S TIPTRONIC S, Diesel Saloon, Coupe GT4
seatsNumber Number of Seats 96% 5, 4, 2
brandColor Exterior Brand Colour 95% White, Rosso Corsa, Jet Black Metallic
m_Unit Mileage Unit 94% miles, km
carOptions Car Options 92% 21-inch 911 Turbo II wheel wit…, 20-inch RS Spyder Deisgn wheel…
Type Model 90% Cayenne, 911, Ghibli
car category Car Type 89% g_km_emission

11. Golden Record Generation
Using entity resolution and data fusion algorithms based on
relevant weighted attributes to unify product information.
Source 1
Source 2
Strategy: Aggregation
Strategy: Authority
Strategy: Min, Max, Avg.,…
Name Colour Size Description ...
T-shirt Red XS This product is
very light
E
Name Colour Size Description ...
T-shirt S Light product E
Name Colour Size Description ...
T-shirt Red S This product is very
light
E
Name Colour Size Description ...
T-shirt Red XS, S This product is very
light, Light product
E
Name Colour Size Description ...
T-shirt Red XS This product is very
light
E

12. Attribute Normalisation
Algorithms based on independent Bayes networks and word
embedding classify arbitrary colours into colour families.
Input: Supplier free-text colours Output: Normalised colour families
Colour
natur beige
aquamarine
olive creative
Swiss Cow style
Manhattan Grau
elegant gold
Colour Colour Family
natur beige Light Brown
aquamarine Blue
olive creative Dark Green
egg yolk Yellow
Manhattan Grau Grey
elegant gold Gold

13. Attribute Extraction
Extract product attributes from unstructured text and put
them into the target data schema using deep learning.
Input: Supplier product catalogues Output: Extracted product attributes
Product Name Model Storage (GB) Colour
Apple iPhone 5S 32 Silver
Apple iPhone 7 64 Gold
Apple iPhone 7 128 Space Grey
Apple iPhone 8 64 Gold
Apple iPhone 8 256 Silver
Apple iPhone 8 256 Gold
Product Description
Apple iPhone 5S 32GB Silver
Apple iPhone 7 64gb gold
Apple iPhone 7 128 Space grey
Apple iPhone 8 64 gigabyte goold
Apple iPhone 8 256 Gb silber
Apple iPhone eight 256 giga-byte Gold

14. Product Categorisation
Using hierarchical, independent Bayes networks to categorise
products in a product category tree.
SKU ID Product Description Predicted Category Confidence Level
21045696 Sanitär-Kreuzschlüssel 200x200mm Chrom Screwdrivers 95%
21045696 Sanitär-Kreuzschlüssel 200x200mm Chrom Screwdrivers 95%
10031686 Bosch Rollenauflage PTA 1000… Filling Stations 93%
10031686 Bosch Rollenauflage PTA 1000… Filling Stations 93%
10031686 Bosch Rollenauflage PTA 1000… Filling Stations 93%
21023947 Mehrzweckhalter 3tlg. verstellb. m. 12Klammer Mounting 86%
94905095 Präz.-Zentrierg. Centro 6-125mm Halmer… Drilling 95%
94905095 Präz.-Zentrierg. Centro 6-125mm Halmer Drilling 95%

15. Product Variant Identification
Generating product variants by segmenting product groups
and aggregating the resulting product group attributes.
Input: Supplier product catalogue Output: Product variants
Category ID Manufacturer Name
Notebook 422102 Lenovo Lenovo Notebook
ThinkPad Yoga 900-13
Notebook 422101 Lenovo Lenovo Notebook
ThinkPad Yoga 900-13
Notebook 376675 Lenovo Lenovo Notebook Yoga
900-13 Silber
Notebook 370921 Lenovo Lenovo Notebook Yoga
900-13 Silber
Category Attribute Examples
Notebook Colour Champagne, Silver, Blue
Notebook Storage Capacity 128 GB, 256 GB, 512 GB
Notebook Display Resolution 1366x768 (WXGA), 1440x900 (WXGA+)
Notebook Usage Business, Consumer, Gaming
Notebook Connectivity 4G, WiFi, Bluetooth
Notebook Processor Family Intel Core i5, Intel Core i7

16. Conclusion
• Using a Product Data Model (PDM) is essential for automated product data preparation
• Adequate data governance helps keeping the PDM in shape
• Most steps in product data preparation can be automated using probabilistic and
statistical approaches
• Advanced machine learning techniques allow a data pipeline to adapt to the changing
data and learn from user feedback
We have shown a few ideas to make automated data
preparation a time- and cost-saving reality.

17. Q&A

18. onedot.com