Downloaded 49 times
Uploaded byAvinash Ramineni
Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data Conference 2016
The document is a practical guide to architecting data lakes, covering fundamental concepts such as data lake definitions, features, attributes, governance, and implementation challenges. It discusses the importance of metadata management, data security, and provides various tools for data lake management. The guide also highlights trends in data lakes, including the rise of cloud solutions and IoT data lakes.
More Related Content
Similar to Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data Conference 2016
Practical guide to architecting data lakes - Avinash Ramineni - Phoenix Data Conference 2016
- 1. Practical Guide to ArchitectingData Lakes Presented By Avinash Ramineni
- 2. Agenda • About Clairvoyant •What is Data Lake ? • Features of Data Lake • Tools • Implementation Challenges • Questions
- 3.
- 4.
- 5. 5Page What is aData Lake “ A data lake is an enterprise-wide system for storing and analyzing disparate sources of data in their native formats” “A data lake is a central location in which to store all your data, regardless of its source or format.” “Is Data lake a replacement or complimentary to EDW ? ” “Is Data lake just a storage layer ? ” “ Just having a Hadoop environment is a data lake ? ”
- 6. 6Page Data Lake Attributes •Data Democratization • Data Discovery • Data Lineage • Self-Service capabilities • Metadata Management
- 7.
- 8.
- 9. 9Page Data Governance • DataAcquisition - what, when, where of data • Data Organization – Structure, format • Data Catalog – what data exists in the lake • Capturing Metadata • Data Lineage • Data Quality • Data Profile • Provenance of data at file and record levels • Business names, descriptions • Data Provisioning
- 10.
- 11.
- 12.
- 13. 13Page Guidelines • Expect structured, semi-structure, unstructured data • store a metadata or tag for location of schema, unstructured • Store a copy of raw input • Raw first mile copy of the data so that we can recover our business or almost • Replay the business if we need to • Data Standardization – data clensing as a workflow after ingest • Use a format that supports your data • Automate metadata management
- 14.
- 15.
- 16. 16Page Implementation Challenges • ChangeData Capture • Mysql – binlog readers • Oracle - tungsten • Updating the deltas on to the data lake • Reusable Data movement workflows • One workflow for table ? (Generate Dynamic workflows based on metadata) • Needs to be driven of metadata • Schema changes on the Source end • Streaming Data • Partitioning Strategies on the Data Lake • Configure them into metadata
- 17. 17Page Tools / Products • SmartCatalogs • Waterline Data Inventory • Collibra Catalog • Data Lake Management • Zaloni Bedrock • Informatica Intelligent Data Lake • Data Governance and Metadata Management • Cloudera Navigator • Apache Atlas • Collibra Data Governance • Oracle BigData Catalog
- 18. 18Page Data Lake Trends •Data Lakes on Cloud • IOT Data Lakes • Logical Data Lakes • Unified View of data that exists across data stores • Data Discovery Portals
- 19. 19Page Questions • Principal @Clairvoyant • Email: avinash@clairvoyantsoft.com • LinkedIn: https://www.linkedin.com/in/avinashramineni