Delta from a Data Engineer's Perspective
Your SlideShare is downloading.
×
Check these out next
More Related Content
Related Books
Free with a 30 day trial from Scribd
The Pill for Your Migration Hell
- 1. The Pill for your Migration Hell Roy Levin & Tomer Koren Microsoft
- 2. Session Goals Discuss the challenges in migrating a production pipeline from a legacy Big Data platform to Spark. Present a methodical approach to accomplish this task Measuring quality, optimizing performance and scalability and deciding on the ‘Definition of Done’
- 3. Background On Premises Azure Security Center
- 4. Background
- 5. Background Feature Engineering 1 Features Classification Based Detection Raw Data 2 Alerts Preprocessing Feature Engineering N Current State State Manager Anomaly DetectionTime Series Alerts Processed Data 1 Detection Pipeline 1 Detection Pipeline N Raw Data 1 Raw Data M Previous States Alert Publisher Processed Data K
- 6. Agenda Discussing the Challenges Formalization Defining Metrics & Testing Tuning Performance Summarization
- 7. Expectations Vs Reality Legacy Pipeline Legacy Pipeline
- 8. Maintaining Semantics (Same Input == Same Output) Real Time Constraint ( Over 100 TB per Day) Cost of goods sold (COGS) Challenges
- 9. Maintaining Semantics Code Conversion: • Rewrite UDFs (C# -> Python) • Rewrite Transformations (USQL -> Pyspark) PySparkUSQL
- 10. Maintaining Semantics Code Conversion: • Different Datatypes: • Different ML Libraries
- 11. Maintaining Semantics Schema Changes: Source IP Destination IP Source Hostname Destination Hostname 1.0.0.1 2.0.0.2 Vm1 Vm2 3.0.0.3 4.0.0.4 Vm3 Source IP Destinatio n IP Source Hostname Exist? Hostname Host Id 1.0.0.1 2.0.0.2 True Vm1 111-11 1.0.0.1 2.0.0.2 False Vm2 222-22 3.0.0.3 4.0.0.4 True Vm3 333-33 SparkLegacy
- 12. Real Time Constraint • Multiple ETL Pipeline that runs on an hourly basis • Largest data feed contains ~4TB events per hour • Running time should be less than 60 minutes to avoid accumulated latency.
- 13. Cost of goods sold (COGS)
- 14. Agenda Discussing the Challenges Formalization Defining Metrics & Testing Tuning Performance Summarization
- 15. View Legacy Code in terms of High-Level Components Feature Engineering 1 Features Classification Based Detection Raw Data 2 Alerts Preprocessing Feature Engineering N Current State State Manager Anomaly DetectionTime Series Alerts Processed Data 1 Detection Pipeline 1 Detection Pipeline N Raw Data 1 Raw Data M Previous States Alert Publisher Processed Data K
- 16. View Legacy Code in terms of High-Level Components Feature Engineering 1 Features Classification Based Detection Raw Data 2 Alerts Preprocessing Feature Engineering N Current State State Manager Anomaly DetectionTime Series Alerts Processed Data 1 Detection Pipeline 1 Detection Pipeline N Raw Data 1 Raw Data M Previous States Alert Publisher Processed Data K
- 17. Cosmos Component Spark Component View Legacy Code in terms of High-Level Components
- 18. Cosmos Feature Engineering Spark Feature Engineering CosmosInputData1 CosmosInputData2 CosmosOutputData1 CosmosOutputData2 SparkOutputData2 SparkOutputData1 Schema mismatch! Validating the Migrated Components
- 19. Cosmos Feature Engineering Spark Feature Engineering CosmosInputData1 CosmosInputData2 CosmosOutputData1 CosmosOutputData2 SparkOutputData2 SparkOutputData1 Validating the Migrated Components Translator Translator Comparator Comparator
- 20. Managing it all • This migration process needs to be done for every component of every detection • Some components are reused across detections • What about the connections between the components? • These dependencies need to be represented and validated as well
- 21. Introducing - ¾E⁄ʽ Component Reuse & Connectivity
- 22. MultiTransformer per Component DataItem DataFrame Model DataItem name1 name2 … DataItemnamen MultiDataItem MultiTransformer DataItem DataItem name1 name2 … DataItem namem MultiDataItem
- 23. DAG of Components DataItem DataFrame Model DataItem name1 name2 … DataItemnamen MultiDataItem MT DataItem DataItem name1 name2 … DataItem namem MultiDataItem MT MT MTMT DagMultiTransformer dependencies
- 24. Stateful MultiTransfomers Slice from iterationi-1 StatefulMultiTransformer dataset1 dataset2 Slice of iterationi output-dataset1
- 25. - ¾E⁄ʽ Without CyFlow With CyFlow Deployment Multiple notebooks - one per component Deploy the DAG Unit Testing No framework Standalone Spark with pyunit Shared utility code Import whl files - hard to maintain Use a repository Using notebooks
- 26. - ¾E⁄ʽUsing notebooks Without CyFlow With CyFlow Deployment Multiple notebooks - one per component Deploy the DAG Unit Testing No framework Standalone Spark with pyunit Shared utility code Import whl files - hard to maintain Use a repository Structure Implicit, according to schedules Explicit and visually depicted
- 27. - ¾E⁄ʽ Without CyFlow With CyFlow Deployment Multiple notebooks - one per component Deploy the DAG Unit Testing No framework Standalone Spark with pyunit Shared utility code Import whl files - hard to maintain Use a repository Structure Implicit, according to schedules Explicit and visually depicted Typing No schema checks Schema checks before running Using notebooks
- 28. Agenda Discussing the Challenges Formalization Defining Metrics & Testing Tuning Performance Summarization
- 29. Cosmos Feature Engineering Spark Feature Engineering CosmosInputData1 CosmosInputData2 CosmosOutputData1 CosmosOutputData2 SparkOutputData2 SparkOutputData1 Validating the Migrated Components - revisited Translator Translator Comparator Comparator
- 30. CosmosOutputData2 SparkOutputData2 Validation -- A Closer Look Comparator Recall our challenges • Legacy ML models • Non-reproducable ML models • Indeterministic semantics (e.g.: based row numbers) • Some non-translatable schema changes • Some randomly generated UUIDs (e.g. AlertIds) How Much to invest in achieving full parity?
- 31. Decide Based on Soft Metric of the Final Output Resource Id 1 Res1 2 Res2 3 Res3 4 Res4 5 Res5 6 Res6 7 Res7 Alerts generated by Legacy Component Resource Id 1 Res1 2 Res3 3 Res4 4 Res5 5 Res6 6 Res8 7 Res9 8 Res10 Alerts generated by Spark Component 𝑗𝑠 = 𝑦 ∩ 𝑦′ 𝑦 ∪ 𝑦′ = 5 9 ≅ 0.56 Jaccard Similarity Precision 𝑝𝑟 = 𝑦 ∩ 𝑦′ 𝑦′ = 5 8 ≅ 0.63 Recall 𝑝𝑟 = 𝑦 ∩ 𝑦′ 𝑦 = 5 7 ≅ 0.71 For 𝒚 ∩ 𝒚 we write an alert content validator (for the rest of the column values)
- 32. Agenda Discussing the Challenges Formalization Defining Metrics & Testing Tuning Performance Summarization
- 33. Measure Running Time on Actual Load 14 Hours !! Feature Engineering Classification Based Detection CYFLOW DAG Alert Publisher Processed Data 1 Processed Data 2 Published Alerts
- 34. Finding the culprit Transformation 1 Intermediate Result 1 Feature Engineering Transformation 1 Feature Engineering Transformation 2 Intermediate Result 2 20 minutes13 hours !!
- 35. Tuning Performance • Aggregate hourly state into daily state (end of day) Partition N State 12:00 AMPartition 1 State 12:00 AM Partition N State 01:00 AMPartition 1 State 01:00 AM Partition N State 02:00 AMPartition 1 State 23:00 PM 24HourlySlices Daily Aggregator Partition N State Daily Partition 1 State Daily
- 36. Tuning Performance • Modify Default Partition (number of cores x 3) : • Use of broadcasting when UDF with large signatures are reused. • Cache (be aware for memory failures ) • Unpersist – remove dataframe from cache when no longer needed
- 37. Agenda Discussing the Challenges Formalization Defining Metrics & Testing Tuning Performance Summarization
- 38. Summary • Present the challenges in migrating a large-scale legacy Big Data system to Spark • Preserving Semantics • Realtime constrains • COGS • Introducing - ¾E⁄ʽ -- a framework built over Apache Spark that allows component reuse & connectivity • Discuss different validation strategies • Reducing runtime and COGS: aspects of Spark performance tuning
- 39. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.
