It doesn’t come without challenges when you set out to build the architecture necessary to grow Big Data concepts and enable Data Science scenario’s. Especially inside a traditional data warehouse company. About Data lakes and Data warehouses, coping with legacy, politics and getting the most out of scalable cloud services.

1. INTRODUCING THE CONCEPTS OF DATA SCIENCE
AND BIG DATA IN A TRADITIONAL DATA WAREHOUSE
COMPANY
Joris Bos
Manager IT, DDW

2. Dairy Data Warehouse (DDW)
• Start up founded in 2013
• Based in Assen, Netherlands
• DDW offers data services to dairy farm
consultants and corporate stakeholders in
the dairy industry across the globe

3. DDW BIG Dairy Data Platform
1. DATA INPUT
Herd
management
Data capture
devices
Milk recording
data
2. DATA TRANSFORMATION 3. DEEP LEARNING
DDW ETL unique
technology transforms
all data into a standard
common format
Dairy Data Warehouse
Single consistent database
Production
Reproduction
Disease
4. DDW PREDICTIONS

4. ‘Advance to go’
• Daily uploads (batching)
• Slow database XML exports
• Calculations freezing data scientist’ pc
• Valuable data locked away in private networks,
widely spread
• This new ‘thing’!

5. Data warehouses are dead! Or?
• Trendy, all will be superseded
• New paradigm
• On premises or Cloud?
• Clearly differentiate use-cases, strengths
• HDFS != DMS
• Different target audience
• Aim to retain value from previous investments

6. DDW Big data infrastructure
• Set-up from zero, best practices opportunity
• Security, data access and data governance at the core
• New roles and competences within the organization
• Data lake at the center
• Seperation storage / compute
• Support for persisted, historical data now (batch)
• Streaming (real-time) near future
• Audit, audit, audit, audit

7. Data Lake & Data Warehouse

8. Big Data pipeline
Data Engineer Data Scientist
Value

9. Chosen setup (Azure)
• Storage : Azure Data Lake (Active Directory Security / Office365)
• Integration runtimes (Data Factory)
• ELT
• Data warehouse as ‘just’ another data source
• Data Catalog
• Compute : On-demand clusters with Hortonworks distribution platform (HDP)
• ML Deep Learning : Model calculation with GPU powered VMs
• ETL to data warehouse for hot layer going forward

10. Chosen setup

11. Takeaways
• Establish clear language around Big Data and tooling
• Clearly position Big Data in your company (vision, budget, etc.)
• Define new roles, workflows and conventions
• Stick to best practices
• Start small, Determine Minimum Viable Product
• Cloud is great for ad-hoc, scalability demands
• Iterate and experiment through POCs

12. Thank you
Feel free to contact me:
Joris Bos
J.bos@dairydatawarehouse.com