Technical Documentation 101 for Data Engineers.pdf
•
0 likes•142 views
This document discusses metadata and data documentation best practices. It begins by defining metadata as data that describes other data, such as author, file size, and date for text files. It recommends documenting the table or database last documented, documenter, business case, tools used, and data quality. Good documentation practices include knowing your audience and purpose, keeping documentation minimal but effective, and building user documentation. Common data documentation templates include CRISP-DM, which outlines phases for documentation like business understanding, data understanding, data preparation, modeling, evaluation, and deployment. Thorough data documentation is important for project understanding, reuse, and governance.
Recommended
Recommended
More Related Content
Similar to Technical Documentation 101 for Data Engineers.pdf
Similar to Technical Documentation 101 for Data Engineers.pdf (20)
More from Shristi Shrestha
More from Shristi Shrestha (20)
Recently uploaded
Recently uploaded (20)
Technical Documentation 101 for Data Engineers.pdf
- 1. TECHNICAL DOCUMENTATION 101 Shristi Shrestha FOR DATA ENGINEERS
- 2. Metadata
- 3. Metadata Metadata is basically information that describes other data. It helps us understand the origin, structure, nature, and context of data.
- 5. author, file size, date created date modified
- 10. Metadata We Care About (Examples) When the table or database was last documented Who it was documented by Business case Tools and resources Queries run against that table Table, column descriptions Data lineage - table level, column level Data quality Metrics
- 11. Metadata Management Software Metadata management is a set of activities, technologies, and policies that target metadata collection, storage, and organizing. Its goal is making data assets understandable and discoverable for users.
- 12. Metadata Management Software: Examples
- 13. Problem 01 02 Additional Software Cost Additions Lots of custom metadata updating code Cost Complication
- 14. Use Anything That Makes Sense
- 15. Good Practice: Where Making it easy to find
- 16. Good Practice: Where Making it easy to find Post a link in the README file in the repository containing the pipeline code
- 17. Source The blue-filled section shows where documentation and lineage are extracted from.
- 19. Document with a purpose Who will consume this documentation? Why do they need this documentation? How would they like to consume documentation? Before you build out your documentation, ask:
- 20. ML and Data Professionals Why do they need this documentation? How would they like to consume documentation?
- 21. Business Stakeholders Why do they need this documentation? How would they like to consume documentation?
- 22. Keep it minimal but effective Remove any un-needed documentation Avoid redundancy across documents Don’t document more than you have to Be flexible and update the documentation as you go through the life cycle of the project
- 23. Know the project's purpose Document the customer’s business objectives Define how your data science project will meet their needs Set a vision for your project or product so that you can steer the team in the right direction Define clear evaluation metrics so that you can objectively determine whether the project was successful Conduct a cost-benefit analysis can help determine project go/no-go and prioritization against other potential projects Document what you are not looking to accomplish
- 25. Document the data What data is being used for the model? Why was this data selected (and other data sets excluded)? How was the data obtained? What are known issues in the data? What does the data look like? (mean, median, mode, skewness, data volume, etc.) How did you alter the data (transformations, imputations, other data cleaning techniques applied, etc.) Where is the data located? How frequently is the data refreshed? Is the data usage compliant with user agreements, data privacy best practices, and relevant regulations? (if not, don’t use it) What security protections do you have data-at-rest and data-in-motion to ensure compliance and data privacy?
- 26. Document the code Can you understand your code after an year? Will another person understand your code if you leave the team?
- 27. Build user documentation Don’t forget your users! Rather, be sure they know how to use your system.
- 29. Depends on Data Lifecycle CRISP-DM (CRoss Industry Standard Process for Data Mining) Microsoft’s TDSP (Team Data Science Process) Usually a custom life-cycle your company follows
- 30. CRISP-DM
- 32. Phases of CRISP-DM Business understanding – What does the business need? Data understanding – What data do we have/need? Is it clean? Data preparation – How do we organize the data for modeling? Modeling – What modeling techniques should we apply? Evaluation – Which model best meets the business objectives? Deployment – How do stakeholders access the results?
- 33. Business Understanding Determine business objectives Assess situation Determine data mining goals Produce project plan
- 34. Data Understanding Collect initial data Describe data Explore data Verify data quality
- 35. Data Preparation Select data Clean data Construct data Integrate data Format data
- 36. Modeling Select modeling techniques Generate test design Build model Assess model
- 37. Evaluation Evaluate results Review process Determine next steps
- 38. Deployment Plan deployment Plan monitoring and maintenance Produce final report Review project
- 40. THANK YOU