Downloaded 159 times
Uploaded bySpark Summit
Better Visibility into Spark Execution for Faster Application Development-(Shivnath Babubabu and Lance Co Ting Keh, Duke)
The document discusses Spark application development and common problems that can occur. It introduces Unravel Data, a startup that aims to help developers visualize Spark job data, optimize performance through automated analysis and diagnoses, and strategize to prevent problems and meet goals. Key points include discussing common issues like failures, wrong results, poor performance, and resource problems; the difficulty of debugging using logs alone; and demonstrating Unravel's platform to address these challenges.
More Related Content
Similar to Better Visibility into Spark Execution for Faster Application Development-(Shivnath Babubabu and Lance Co Ting Keh, Duke)
Better Visibility into Spark Execution for Faster Application Development-(Shivnath Babubabu and Lance Co Ting Keh, Duke)
- 1. Spark Application Development Made Fastand Easy Shivnath Babu Lance Co Ting Keh
- 2. Lance Co TingKeh Machine Learning @ Box Distributed ML Infrastructure Go Blue Devils! Shivnath Babu Associate Professor @ Duke Chief Scientist at Unravel Data Sys. R&D in Management of Data Systems
- 3.
- 4. What’s so greatabout Spark? From:
- 5.
- 6. Spark Execution part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filterreduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath)
- 7. Spark Execution sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filterreduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 HDFS Stage 0 Stage 1 RDD0 RDD1 RDD2 RDD3 RDD4
- 8. part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filter reduceBykeymap saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS Stage 0 Stage 1 sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Spark Execution
- 9. Spark Execution part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filterreduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 Exec0Exec1Exec2Exec3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Stage 0 Stage 1
- 10. Spark Execution part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filterreduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 Exec0Exec1Exec2Exec3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Stage 0 Stage 1
- 11. Spark Execution part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filterreduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 Exec0Exec1Exec2Exec3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Stage 0 Stage 1
- 12. Anything that cango wrong, will go wrong (at some point)
- 13. What can gowrong? Failures • My query failed after 6 hours! • What does this exception mean?
- 14. What can gowrong? • Failures • My query failed after 6 hours! • What does this exception mean? • Wrong results • Result of my job looks wrong • Bad performance • My app is very slow • Pipeline is not meeting the 4hr SLA • Poor scalability • Oh, but it worked on the dev cluster! • Bad App(le)s • Tom’s query brought the cluster down • Application Problems • Poor choice of transformations • Ineffective caching • Bloated data structures • Data/Storage Problems • Skewed data, load imbalance • Small files, poor data partitioning • Spark Problems • Shuffle • Lazy evaluation causes confusion • Resource Problems • Resource contention • Performance degradation And Why?
- 15. How do applicationdevelopers detect & fix these problems today?
- 17. Too much tolook at!
- 18. Look at Logs? Logsin distributed systems are spread out, incomplete, & usually very difficult to understand
- 19. There has tobe a better way for application developers to detect & fix problems
- 20. Visualize: Show me allrelevant data in one place
- 21. Optimize: Analyze the datafor me and give me diagnoses and fixes
- 22. Strategize: Help me preventthe problems from happening and meet my goals
- 23.
- 24. Visualize: Show me allrelevant data in one place
- 25. Visualize: Show me allrelevant data in one place
- 26. Optimize: Analyze the datafor me and give me diagnoses and fixes
- 27. Strategize: Help me preventthe problems from happening and meet my goals
- 28. Strategize: Help me preventthe problems from happening and meet my goals
- 29. We are hiring:jobs@unraveldata.com Sign up for early access at: bit.ly/unravelspark