Infofarm provides data science and artificial intelligence services including building and maintaining big data architectures using Apache Spark and Hadoop. They help organizations leverage data through training and workshops on data science techniques. Their passion is to extract business value from data by ingesting it from various sources into a datalake, processing it to generate information, and harvesting the value through use cases like personalization. A datalake involves storing raw and processed data in a file system for querying, while use cases may involve predictive analytics using the processed data. Infofarm can help organizations address challenges like data governance for GDPR through architecture best practices.

2. Thomas Bombeke

3. Infofarm
We harvest business value
Data Science Big Data Training & Workshops
Applied Artificial Intelligence services,
Machine Learning & Deep Learning
techniques
Setting up and maintaining on-premise
and Cloud-based Big Data architectures
with Apache Spark and Hadoop
Training and guiding organisations
through the digital era using the power of
Data Science
OUR PASSION
Plow data – Grow information – Harvest value
Powered by

4. Datalake Architecture
Building Blocks
Sinkholes
Use Cases
Concepts

5. Concepts

6. Datalake
“If you think of a datawarehouse as a store of bottled water – cleansed
and packaged and structured for easy consumption – the data lake is a
large body of water in a more natural state. The contents of the data
lake stream in from a source to fill the lake, and various users of the lake
can come to examine, dive in, or take samples” – James Dixon CTO
Pentaho

7. • Based on data source or
category
• Single truth
• Ingests data from different
sources
• Accessible for multiple users
Datalake

8. • Has one schema
• Single type of storage
• Queryable
Puddles
Puddle
Puddle
Puddle

9. • Raw dump
• Not ready for use
• Reprocessing purpose
Raw Puddles

10. • Schema and formats applied
• Timesorted event data
• Ready for any use
Accepted Puddles

11. • Enriched/aggregated data
• Single purpose
• Dashboarding
• Exporting
Master Puddles

12. Streams
Export
Ingestion Puddle
Puddle
Processing

13. Example

14. Ingestion
Click Event Stream
Daily Database Snapshot

15. Processing
Raw
Daily Database Ingest
Click Event Stream
Accepted

16. Processing
Raw Accepted
2019/04/07 2019/04/01/20190407
2019/04/07/20190407

17. Processing
Accepted
Accepted
Master
2019/04/07/20190407
2019/04/07

18. Exporting
Raw
Sends triggerMaster

19. Building Blocks

20. Getting Event Data
• Point in time
• Late arriving data
• Stream or batch

21. Getting Event Data
Click Event Stream
Daily Database Snapshot

22. Getting Entity Data
• No relations
• Avoid if does not bring value
• Reducing load on source
• Snapshot history

23. Getting Entity Data
Click Event Stream
Daily Database Snapshot

24. Storing Data
File System Data Store Streaming Platform
• CSV, Parquet, Avro
• High latency
• Needs work for querying
• Write once, read-many
• Column or row based
• Low latency
• REST, Thrift, JDBC, …
• Updates possible
• Messages
• Log compaction
• Reloading from start

25. Storing Data
Raw
Daily Database Ingest
Click Event Stream
Accepted

26. Processing Data
Batch Streaming
• ”All” the data
• Big data
• Micro-batching
• Rolling window
• Fast data
• More challenges

27. Processing Data
Raw
Accepted
Master
Accepted

28. Exporting Data

29. Scheduling
Cronjob• Frequency
• Dependency
• Execution time

30. Scheduling
1. Ingest click stream hourly
2. Prepare raw clickstream hourly
3. Ingest database everyday at 05:00
4. Create master sessions everyday after step 2
5. Export master sessions after step 4

31. Checkpointing
JOB CHECKPOINT
Ingest tv catalague 2019/04/10 05:00
Ingest click events 2019/04/10 15:00
Create master sessions 2019/04/07 07:00
Export master sessions 2019/04/07 07:20

32. Cloud or On-premise?
• Elastic computing/storage
• No admins needed
• Cheaper
• Hard limits
• “More secure”
• Maintenance
• Better performance

33. Cloud or On-premise?
• Sandbox
• Test environment
• Ad Hoc
• Non-elastic load
• Sensitive data

34. Use Cases

35. Personal Advertisement
Master
Master
Accepted Master
Accepted
?
?

36. Personal Advertisement
Master
Master
Accepted
ConnectorTechType
Datastore
Kudu
Impala
Spark
HBase Hive

37. Personal Advertisement
Master
Master
Accepted
ConnectorTechType
Datastore
Kudu
Impala
Spark
HBase Hive

38. Personal Advertisement
Master
Master
Accepted
ConnectorTechType
Datastore
Kudu
Impala
Spark
HBase Hive

39. Personal Advertisement
Master
Master
Accepted Master
Accepted

40. AWS Predictive autoscaling
adjust scaling

41. AWS Predictive autoscaling
adjust scaling

42. AWS Predictive autoscaling
adjust scaling

43. Sinkholes

44. GDPR
• Retention time
• Right to be informed
• Right to be forgotten

45. GDPR - Retention Time
• What? Puddle schema
• When? Puddle partitioning
• Where? Puddle location

46. GDPR - RTBI
• Puddles are queryable
• Master script that queries all the
puddles

47. GDPR - RTBF
• Datastores with delete
functionality
• What about file systems?

48. GDPR - RTBF
1. Create an alias
2. Replace customer id with alias
during ingestion
3. Replace alias with customer id
during export
4. RTBF? Delete alias

49. Schema Evolution
• Some datastores/file formats can handle schema changes
• Breaking changes? New puddle

50. Small Files/Partitions
• Many small files on filesystem? Compact
• Datastore partitioning? Roll up

51. Data Quality
• Rejected puddle for malformed records
• Stream metrics for data going in and out
• Monitor metrics with conditions or CPD algorithm
• Normalization and formats

52. Wrapping up

53. Get in Touch!
Thomas Bombeke
Big Data Engineer
+32 (0)498 60 66 55
thomas.bombeke@infofarm.be
info@infofarm.be
www.infofarm.be
www.xploregroup.be