LanceShivnathHadoopSummit2015
•Download as PPTX, PDF•
0 likes•285 views
This document discusses Spark application development and common problems that can occur. It notes that failures, wrong results, poor performance, scalability issues, and application, data, storage, and resource problems can all go wrong with Spark applications. It asks how application developers currently detect and fix these issues by looking at logs, but that logs are spread out, incomplete, and difficult to understand. It proposes that a better approach is to visualize all relevant data in one place, analyze the data to provide diagnoses and fixes, and help prevent problems and meet goals. It then lists some existing tools for Hadoop and Spark that provide visualization, optimization, and strategic capabilities.
Recommended
More Related Content
Viewers also liked
Similar to LanceShivnathHadoopSummit2015
Similar to LanceShivnathHadoopSummit2015 (20)
LanceShivnathHadoopSummit2015
- 1. Spark Application Development Made Fast and Easy
- 2. Lance Co Ting Keh Machine Learning @ Box Distributed ML Infrastructure Go Blue Devils! Shivnath Babu Associate Professor @ Duke Chief Scientist at Unravel Data Systems R&D in Management of Data Systems
- 4. Applications Powered by Spark @ Box
- 5. What’s so great about Spark? From http://evinrevello.com/big-data-pocket-knife/
- 6. What’s so great about Spark? From:
- 8. Spark Execution sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filter reduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS
- 9. Spark Execution sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filter reduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 HDFS Stage 0 Stage 1 RDD0 RDD1 RDD2 RDD3 RDD4
- 10. part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filter reduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS Stage 0 Stage 1 sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Spark Execution
- 11. Spark Execution part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filter reduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 Exec0Exec1Exec2 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Stage 0 Stage 1
- 12. Spark Execution part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filter reduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 Exec0Exec1Exec2 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Stage 0 Stage 1
- 13. Spark Execution part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 map filter reduceBykey map saveAsTextFile part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 part-0 part-1 part-2 part-3 Exec0Exec1Exec2 RDD0 RDD1 RDD2 RDD3 RDD4 HDFS sc.textFile(hdfsPath) .map(parseInput) .filter(subThreshold) .reduceByKey(tallyCount) .map(formatOutput) .saveAsTextFile(outPath) Stage 0 Stage 1
- 14. Anything that can go wrong, will go wrong (at some point)
- 15. What can go wrong? • Failures • My query failed after 6 hours! • What does this exception mean? • Wrong results • Result of my job looks wrong • Bad performance • My app is very slow • Pipeline is not meeting the 4hr SLA • Poor scalability • Oh, but it worked on the dev cluster! • Bad App(le)s • Tom’s query brought the cluster down • Application Problems • Poor choice of transformations • Ineffective caching • Bloated data structures • Data/Storage Problems • Skewed data, load imbalance • Small files, poor data partitioning • Spark Problems • Shuffle • Lazy evaluation causes confusion • Resources Problems • Resource contention • Performance degradation And Why?
- 16. How do application developers detect & fix these problems today?
- 19. Look at Logs? Logs in distributed systems are spread out, incomplete, & usually very difficult to understand
- 20. There has to be a better way for application developers to detect & fix problems
- 21. Visualize: Show me all relevant data in one place
- 22. Optimize: Analyze the data for me and give me diagnoses and fixes
- 23. Strategize: Help me prevent the problems from happening and meet my goals
- 24. For Hadoop Visualize Optimize Strategize Ambrose Yes Lipstick Yes ATS /Ambari Yes Inviso Yes Vaidya Yes Dr. Elephant Yes Starfish Yes Yes Yes Unravel Yes Yes Yes
- 25. For Spark Visualize Optimize Strategize Spark UI Yes Spark Debugger Yes Sematext SPM Yes Sparkling Yes Unravel Yes Yes Yes
- 26. Demo
Editor's Notes
- pull stages apart slide for Applications Powered by Spark @Box screenshots of Spark WebUI
- How does it all relate to my app
- Piece together the story and connect it back to app