Food safety risks the elastic stack to the rescue

1. Mihalis Papakonstantinou
Data Engineer/Team Leader @Agroknow
Food Safety
Risks?
The Elastic Stack
to the Rescue

2. SOME
BACKGROUND

3. The food safety intelligence
company that collects,
translates and enriches global
food safety data.
Agroknow

4. EVERYTHING
IS SUPER
CLEAR, NO?

5. The food safety intelligence
company that collects,
translates and enriches global
food safety data.
Agroknow

6. ONE STEP AT
A TIME

7. The food safety intelligence
company that collects,
translates and enriches global
food safety data.
Agroknow

8. FOOD SAFETY
Food is recalled everyday
All over the world food safety
authorities are recalling unsafe
to consume products

9. DATA SOURCES
Food Safety Authorities:
○ Australia
○ Belgium
○ Abu Dhabi
○ Austria
○ Romania
○ China
○ Brazil
○ Germany
○ Czech
○ Canada
○ Hong Kong
○ France
○ Greece
○ USA
○ UK
○ Scotland
○ Ireland
○ South Korea
○ Poland
○ Croatia
○ Israel
○ Italy
○ Denmark
○ Japan
○ New Zealand
○ Netherlands
○ RASFF
○ Luxembourg
○ Singapore
○ Slovenia

10. 22.586
Food product recalls
have taken place so far
within 2020

11. STEP#2

12. The food safety intelligence
company that collects,
translates and enriches global
food safety data.
Agroknow

13. LET’S COLLECT
OUR DATA!

14. AN EXAMPLE

15. WEB
CRAWLING IS
EASY, NO?

16. DATA SOURCES
Food Safety Authorities:
○ Australia
○ Belgium
○ Abu Dhabi
○ Austria
○ Romania
○ China
○ Brazil
○ Germany
○ Czech
○ Canada
○ Hong Kong
○ France
○ Greece
○ USA
○ UK
○ Scotland
○ Ireland
○ South Korea
○ Poland
○ Croatia
○ Israel
○ Italy
○ Denmark
○ Japan
○ New Zealand
○ Netherlands
○ RASFF
○ Luxembourg
○ Singapore
○ Slovenia

17. DATA SOURCES
Food Safety Authorities:
○ Australia
○ Belgium
○ Abu Dhabi
○ Austria
○ Romania
○ China
○ Brazil
○ Germany
○ Czech
○ Canada
○ Hong Kong
○ France
○ Greece
○ USA
○ UK
○ Scotland
○ Ireland
○ South Korea
○ Poland
○ Croatia
○ Israel
○ Italy
○ Denmark
○ Japan
○ New Zealand
○ Netherlands
○ RASFF
○ Luxembourg
○ Singapore
○ Slovenia
RSS
HTML
HTML
HTML
HTML
HTML HTML
HTML
HTML
HTML
RSS
HTML
HTML
HTML
HTML
HTML
HTML
HTML
HTML
HTML
HTML
HTML
HTML
PDF
PDF
PDF
PDF
PDF
XLS
XML

18. (regardless)
LET’S COLLECT
IT!

19. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
AK Transformer
Wrapper API(s)

20. SOME STATS

21. 680.000
Food recalls collected by
our Data Platform so far
going back to 1980

22. WE
COLLECTED
THE DATA,
NOW WHAT?

23. The food safety intelligence
company that collects,
translates and enriches global
food safety data.
Agroknow

24. WHAT IS
IMPORTANT
IN A RECALL?

25. —INGREDIENT
Which was the main
ingredient of the recalled
brand?
FOOD RECALL (IMPORTANT) PROPERTIES#1

26. AN EXAMPLE
By employing product/ingredient
vocabularies & open data

27. —INGREDIENT
Which was the main
ingredient of the recalled
brand?
FOOD RECALL (IMPORTANT) PROPERTIES#2
—HAZARD
Which was the reason
behind the recall?

28. AN EXAMPLE
By employing hazard vocabularies

29. —INGREDIENT
Which was the main
ingredient of the recalled
brand?
FOOD RECALL (IMPORTANT) PROPERTIES#3
—HAZARD
Which was the reason
behind the recall?
—SUPPLIER
Which was the food
company(ies) involved in
the recall?

30. AN EXAMPLE
By employing NER techniques
Relationships can emerge

31. —INGREDIENT
Which was the main
ingredient of the recalled
brand?
FOOD RECALL (IMPORTANT) PROPERTIES#4
—HAZARD
Which was the reason
behind the recall?
—SUPPLIER
Which was the food
company(ies) involved in
the recall?
—BRAND
Which was the product
brand that was recalled?

32. AN EXAMPLE
By integrating open sources of data

33. —INGREDIENT
Which was the main
ingredient of the recalled
brand?
FOOD RECALL (IMPORTANT) PROPERTIES#5
—HAZARD
Which was the reason
behind the recall?
—SUPPLIER
Which was the food
company(ies) involved in
the recall?
—BRAND
Which was the product
brand that was recalled?
—DATE
When did the recall take
place?

34. AN EXAMPLE

35. —INGREDIENT
Which was the main
ingredient of the recalled
brand?
FOOD RECALL (IMPORTANT) PROPERTIES#6
—HAZARD
Which was the reason
behind the recall?
—SUPPLIER
Which was the food
company(ies) involved in
the recall?
—BRAND
Which was the product
brand that was recalled?
—DATE
When did the recall take
place?
—ORIGIN
Where did the recalled
ingredient originate from?

36. LET’S ENRICH
OUR DATA!

37. DATA SOURCES
Food Safety Authorities:
○ Australia
○ Belgium
○ Abu Dhabi
○ Austria
○ Romania
○ China
○ Brazil
○ Germany
○ Czech
○ Canada
○ Hong Kong
○ France
○ Greece
○ USA
○ UK
○ Scotland
○ Ireland
○ South Korea
○ Poland
○ Croatia
○ Israel
○ Italy
○ Denmark
○ Japan
○ New Zealand
○ Netherlands
○ RASFF
○ Luxembourg
○ SIngapore
○ Slovenia

38. BUT...

39. DATA SOURCES
Food Safety Authorities:
○ Australia
○ Belgium
○ Abu Dhabi
○ Austria
○ Romania
○ China
○ Brazil
○ Germany
○ Czech
○ Canada
○ Hong Kong
○ France
○ Greece
○ USA
○ UK
○ Scotland
○ Ireland
○ South Korea
○ Poland
○ Croatia
○ Israel
○ Italy
○ Denmark
○ Japan
○ New Zealand
○ Netherlands
○ RASFF
○ Luxembourg
○ SIngapore
○ Slovenia
en
fr
en
de
ro
cn
pt
de
cz
en
en
fr
el
en
en
en
en
en
en
en
en
fr
kr
pl
hr
il
it
dk
nl
sl

40. LET’S
TRANSLATE
OUR DATA!

41. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
AK Transformer
Wrapper API(s)
Enrichment Data Layer
AK Enricher
Wrapper API(s)

42. (now)
LET’S ENRICH
OUR DATA!

43. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
AK Transformer
Wrapper API(s)
Enrichment Data Layer
AK Enricher
Wrapper API(s)

44. IS THAT ALL?

45. Other data is also important if
we want to holistically cover
the Food Safety Sector

46. OTHER DATA TYPES
○ Laboratory Test Results
○ Country Risk Indicators
○ Country Corruption
Indicators
○ Food Production Data
○ Food Trade Data
○ Companies Data
○ Commodity Prices
○ News
○ Social Media Data
○ Legislation
○ Outbreaks Information
○ Product Brands Recipes
○ Sensor Installations &
Readings
○ Inspections

47. SOME STATS

48. OTHER DATA TYPES
○ Laboratory Test Results (196M)
○ Country Risk Indicators (12K)
○ Country Corruption Indicators (3K)
○ Food Production Data (150K)
○ Food Trade Data (34M)
○ Companies Data (680K)
○ Commodity Prices (390K)
○ News (67K)
○ Social Media Data (7K)
○ Legislation (280)
○ Outbreaks Information (57K)
○ Product Brands Recipes (26K)
○ Sensor Installations & Readings (12M)
○ Inspections (227K)

49. DATA VARIETY

50. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
AK Transformer
Wrapper API(s)
Enrichment Data Layer
AK Enricher
Wrapper API(s)

51. ~600.000.000
Data records have been
collected & processed so
far by our Data Platform

52. DATA QUALITY

53. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
AK Transformer
Wrapper API(s)
Enrichment Data Layer
AK Enricher
Wrapper API(s)
Curation Layer

54. DATA
AVAILABILITY

55. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
AK Transformer
Wrapper API(s)
Enrichment Data Layer
AK Enricher
Wrapper API(s)
Curation Layer Data Persistence Layer
NLP/Analytics Data Layer
Wrapper API(s)
DataAPI

56. SOME (MORE) STATS

57. ~15.000.000 Reqs/Year

58. WHAT ABOUT
PERSONALIZATION
OR
USER PROFILING?

59. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
AK Transformer
Wrapper API(s)
Enrichment Data Layer
AK Enricher
Wrapper API(s)
Curation Layer Data Persistence Layer
NLP/Analytics Data Layer
Wrapper API(s)
DataAPI
Intelligence Layer

60. SOME (MORE) STATS

61. 130 API Endpoints

62. WHAT ABOUT
UPTIME?

63. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
Enrichment Data Layer
AK Transformer AK Enricher
DataAPI
Data Persistence Layer
NLP/Analytics Data Layer
Monitoring Layer
Wrapper API(s)Wrapper API(s)
Intelligence Layer
Wrapper API(s)
Curation Layer

64. AND
ORCHESTRATION?

65. Agroknow Data Platform
CrawlingLayer
Raw Data Persistence Layer
Enrichment Data Layer
AK Transformer AK Enricher
DataAPI
Orchestration Layer
Data Persistence Layer
NLP/Analytics Data Layer
Monitoring Layer
Wrapper API(s)Wrapper API(s)
Intelligence Layer
Wrapper API(s)
Curation Layer

66. WHY ALL THIS?

67. FOODAKAI
Data product helping the Food Industry make
data informed decisions to minimize food
safety risks

68. FOODAKAI

69. FOODAKAI

70. FOODAKAI

71. FOODAKAI

72. FOODAKAI

73. FOODAKAI

74. SOME (FINAL) STATS

75. Elastic Search
Awards Honoree
2019

76. CREDITS: This presentation template was created by Slidesgo,
including icons by Flaticon, and infographics & images by Freepik
THANKS
Do you have any questions?
mihalispapak@agroknow.com
agroknow.com
Got a minute?
https://bit.ly/365oCR2
{agroknow, mihalispapak}
{agroknow, mihalispap}
{@agroknow, @mihalispapak}