Data Project Strategy - A Case Study on Data Platform
Data team usually has to work on three types of projects: data engineering, data analytics, and data science. Data engineering projects focus on how to effectively process and manage data, data analytics projects try to find insights from what happened so as to make better decisions, and data science projects attempt to build models to predict the future. What kind of strategies to use for a project? A strategy is an optimization since we want to optimize our return on investment. However, we have to face many constraints (i.e., money, time, resources). Therefore, a strategy is a trade-off that implies we cannot achieve all the objectives. The process of finding the right strategy involves in asking a lot of questions and figuring out what really matters for a project. The outcome of this process is some guidelines that effectively guide us when working on projects. We usually follow three guidelines: flexible, convenient, and scalable. We built a unified s3-based data platform to achieve one source/format goal. An S3 object stores data in JSON format and some user-defined metadata (i.e., MD5, content size, last modified, UUID) for verification. The key idea is to use a relational database to keep S3 object key, S3 user-defined & system-defined metadata, a logical name (consists of a few fields), documentation, etc. A unified data platform can easily support data analytics and data science tasks well. JSON format is convenient for a human to understand data. For efficient processing, data can be converted into Parquet format via Spark.
Recommended
More Related Content
Viewers also liked
Viewers also liked (14)
Similar to Data Project Strategy - A Case Study on Data Platform
Similar to Data Project Strategy - A Case Study on Data Platform (20)
Recently uploaded
Recently uploaded (20)
Data Project Strategy - A Case Study on Data Platform
- 2. Outline • IntroducFon • Project • Strategy • Data Pla5orm • Conclusion
- 4. IntroducFon – Teams • Product Team – Design – Valuable • Tech Team – Implement – Reliable • Data Team – OpFmize – Smart Product Tech Team Data Team Product Team
- 5. Project – ClassificaFon • Data Engineering – Data – Collect/Crawl/Process/Store/Serve • Data AnalyFcs – Insight – Analysis/Reports/Dashboard • Data Science – Model – Product CategorizaFon/Matching – RecommendaFon/Search/Fraud DetecFon
- 6. Project – RelaFonship • Data Engineering – Process Data – Data Management • Data AnalyFcs – Find Insights – Decision Making • Data Science – Build Models – Predic:on Data Engineering Data AnalyFcs Data Science Data -> Insight -> Model
- 7. Project – Skills • Data Engineering – So=ware Engineering/Data Pipeline/Infrastructure/Pla5orm – Python/Scala/Spark Streaming/AWS/Hadoop/NoSQL/Airflow • Data AnalyFcs – Web Metrics/AARRR/Conversion/Funnel/Cohort – Python/Spark SQL/Pandas/Excel • Data Science – Machine Learning/NLP/Image Processing/Data Mining – Python/Spark MLLib/Scikit-learn/Tensorflow/R
- 8. Strategy – DefiniFon • DefiniFon – Goal: OpFmizaFon – Constraint: Trade-off – Process: QuesFon – Result: Guidelines
- 9. Strategy – Guidelines • Flexible (General/Unified) – Flexible to support different kinds of tasks • Convenient – Convenient for the most frequent tasks • Scalable – Scalable to handle a lot of data
- 10. Data Pla5orm – IntroducFon • Data – MulFple Sources – MulFple Formats • Problems – Process/Monitor – Aggregate/Analyze • Goal – A Unified Data Pla5orm NoSQL Excel Google Analy:cs Mixpanel Report 1 RDBMS Report 2 Report N
- 11. Data Pla5orm – Design Source Sink Transform Stream Stream Load Save Source Source Stream Stream Load Load
- 12. Data Pla5orm – Sources • Data Sources – Internal • RDBMS (MySQL, PostgreSQL) • NoSQL (MongoDB, Cassandra, Redis) • Files (CSV, JSON, Excel, Google Spread Sheet) – External • Google AnalyFcs • Mixpanel • AppsFlyer
- 13. Data Pla5orm – Stream • What we want? – A unified representaFon of data – Independent of data sources/sinks/formats • Stream – A collec:on of bounded/unbounded data items – An item is a JSON or text (serialized format) Item N Item 2 Item 1 . . .
- 14. Data Pla5orm – Sink • Data Sink – Public Cloud (Managed) • Amazon S3 (Separate Storage from Compute) • Amazon EMR – Private Cloud (Dedicated) • Hadoop Cluster – Hybrid Cloud
- 15. Data Pla5orm – Pipeline • Pipeline – Loader: Source -> Stream – Transformer: Stream -> Stream – Saver: Stream -> Sink Source Sink Transform Stream Stream Load Save
- 16. Data Pla5orm – S3 Sink • S3 Sink (Key-Value Store -> File Store) – S3 Object (All objects are under one bucket) • Data: JSON (each line is a JSON string) • User-Defined Metadata – VerificaFon – MD5, last modified, content size, UUID – RDBMS – VerificaFon & Monitoring & DocumentaFon • Logical Name: mulFple fields (i.e., country, project, table, day) • S3 Object Key: path • S3 Object Metadata: MD5, last modified, content size, eTag, UUID • Other: documentaFon, owner, etc. – File System API: read, write, delete, copy, list, …
- 17. Data Pla5orm – API • write(country, project, table, day, data) API RDBMS Stream File Metadata S3 Metadata Metadata Check & Merge File exists? Compute Fetch Upload Write Insert/Update
- 18. Data Pla5orm – API • read(country, project, table, day) API RDBMS Stream S3 Metadata Metadata Exists? Fetch Streaming Fetch Check A File-like object
- 21. Conclusion • Data Projects – Data Engineering (Data) – Data AnalyFcs (Insight) – Data Science (Model) • Data Strategy – Flexible/Convenient/Scalable • Data Pla5orm – Unified: One Source/Format/Schema – Flexible – Stream (JSON) – Convenient – One Loader per Source – Scalable – Store files in S3