Why your Spark Job is Failing
•
50 likes•24,828 views
The document discusses Spark job failures and Spark/YARN architecture. It describes a Spark job failure due to a task failing 4 times with a NumberFormatException when parsing a string. It then explains that Spark jobs are divided into stages made up of tasks, and the entire job fails if a stage fails. The document also provides an overview of the Spark and YARN architectures, showing how Spark jobs are submitted to and run via the YARN resource manager.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Why your Spark Job is Failing
Similar to Why your Spark Job is Failing (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
Why your Spark Job is Failing
- 1. 1© Cloudera, Inc. All rights reserved. Why your Spark Job is Failing Kostas Sakellis
- 2. 2© Cloudera, Inc. All rights reserved. Me • Software Engineering at Cloudera • Contributor to Apache Spark • Before that, worked on Cloudera Manager
- 3. 3© Cloudera, Inc. All rights reserved. com.esotericsoftware.kryo.KryoException: Unable to find class: $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC $$iwC$$anonfun$4$$anonfun$apply$3
- 4. 4© Cloudera, Inc. All rights reserved. We go about our day ignoring manholes until… Courtesy of: http://www.independent.co.uk/incoming/article9127706.ece/binary/original/maholev23.jpg
- 5. 5© Cloudera, Inc. All rights reserved. … something goes wrong. Courtesy of: http://greenpointers.com/wp-content/uploads/2015/03/Manhole-Explosion1.jpg
- 6. 6© Cloudera, Inc. All rights reserved. org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, kostas-4.vpc.cloudera.com): java.lang.NumberFormatException: For input string: "3.9166,10.2491,-4.0926,-4.4659,0" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike [...] Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSched uler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche duler.scala:1192) [...]
- 7. 7© Cloudera, Inc. All rights reserved. org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, kostas-4.vpc.cloudera.com): java.lang.NumberFormatException: For input string: "3.9166,10.2491,-4.0926,-4.4659,0" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike [...] Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSched uler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche duler.scala:1192) [...]
- 8. 8© Cloudera, Inc. All rights reserved. Job? What now? Courtesy of:http://calvert.lib.md.us/jobs_pic.jpg
- 9. 9© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum()
- 10. 10© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum()
- 11. 11© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum()
- 12. 12© Cloudera, Inc. All rights reserved. Then what the heck is a stage? Courtesy of: https://writinginadeadworld.files.wordpress.com/2014/03/rock1.jpeg
- 13. 13© Cloudera, Inc. All rights reserved. Partitions sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum() HDFS Partition 1 Partition 2 Partition 3 Partition 4
- 14. 14© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum() …RDD1 HDFS Partition 1 Partition 2 Partition 3 Partition 4
- 15. 15© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum() …RDD1 …RDD2 HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4
- 16. 16© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum() …RDD1 …RDD2 HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4 …RDD3 Partition 1 Partition 2 Partition 3 Partition 4
- 17. 17© Cloudera, Inc. All rights reserved. …RDD1 …RDD2 RDDs HDFS Partition 1 Partition 2 Partition 3 Partition 4 sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum() Partition 1 Partition 2 Partition 3 Partition 4 …RDD3 Partition 1 Partition 2 Partition 3 Partition 4 Sum
- 18. 18© Cloudera, Inc. All rights reserved. …RDD1 …RDD2 RDD Lineage HDFS Partition 1 Partition 2 Partition 3 Partition 4 sc.textFile(“hdfs://…”, 4) .map((x) => x.toInt) .filter(_ > 10) .sum() Partition 1 Partition 2 Partition 3 Partition 4 …RDD3 Partition 1 Partition 2 Partition 3 Partition 4 Sum Lineage
- 19. 19© Cloudera, Inc. All rights reserved. RDD Dependencies …RDD1 …RDD2 HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4 …RDD3 Partition 1 Partition 2 Partition 3 Partition 4 Sum Narrow Dependencies • Narrow and Wide Dependencies
- 20. 20© Cloudera, Inc. All rights reserved. Wide Dependencies • Sometimes records need to be grouped together • Examples • join • groupByKey • Stages created at wide dependency boundaries
- 21. 21© Cloudera, Inc. All rights reserved. A more Interesting Spark Job val rdd1 = sc.textFile(“hdfs://...”) .map(someFunc) .filter(filterFunc) val rdd2 = sc.hadoopFile(“hdfs://...”) .groupByKey() .map(someOtherFunc) val rdd3 = rdd1.join(rdd2) .map(someFunc) rdd3.collect()
- 22. 22© Cloudera, Inc. All rights reserved. A more Interesting Spark Job val rdd1 = sc.textFile(“hdfs://...”) .map(someFunc) .filter(filterFunc) maptextFile filter
- 23. 23© Cloudera, Inc. All rights reserved. A more Interesting Spark Job val rdd2 = sc.hadoopFile(“hdfs://...”) .groupByKey() .map(someOtherFunc) groupByKeyhadoopFile map
- 24. 24© Cloudera, Inc. All rights reserved. A more Interesting Spark Job val rdd3 = rdd1.join(rdd2) .map(someFunc) join map
- 25. 25© Cloudera, Inc. All rights reserved. A more Interesting Spark Job rdd3.collect() maptextFile filter group ByKey hadoop File map join map 1 Wide Dependencies 1 2 3 4
- 26. 26© Cloudera, Inc. All rights reserved. Get to the point before I stop caring!
- 27. 27© Cloudera, Inc. All rights reserved. org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, kostas-4.vpc.cloudera.com): java.lang.NumberFormatException: For input string: "3.9166,10.2491,-4.0926,-4.4659,0" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike [...] Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSched uler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche duler.scala:1192) [...]
- 28. 28© Cloudera, Inc. All rights reserved. What was the failure? org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, kostas-4.vpc.cloudera.com): java.lang.NumberFormatException: For input string: "3.9166,10.2491,-4.0926,-4.4659,0” [...]
- 29. 29© Cloudera, Inc. All rights reserved. What was the failure? Stage Task Task Task Task
- 30. 30© Cloudera, Inc. All rights reserved. What was the failure? Stage Task Task Task Task
- 31. 31© Cloudera, Inc. All rights reserved. What was the failure? Stage Task Task Task Task spark.task.maxFailures=4
- 32. 32© Cloudera, Inc. All rights reserved. org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, kostas-4.vpc.cloudera.com): java.lang.NumberFormatException: For input string: "3.9166,10.2491,-4.0926,-4.4659,0" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike [...] Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSched uler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche duler.scala:1192) [...]
- 33. 33© Cloudera, Inc. All rights reserved. org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, kostas-4.vpc.cloudera.com): java.lang.NumberFormatException: For input string: "3.9166,10.2491,-4.0926,-4.4659,0" at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1250) at java.lang.Double.parseDouble(Double.java:540) at scala.collection.immutable.StringLike [...] Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSched uler$$failJobAndIndependentStages(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGSche duler.scala:1192) [...]
- 34. 34© Cloudera, Inc. All rights reserved. ERROR executor.Executor: Exception in task ID 2866 java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:565) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:648) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:706) at java.io.DataInputStream.read(DataInputStream.java:100) at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:209) at org.apache.hadoop.util.LineReader.readLine(LineReader.java:173) at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206) at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45) at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:164) [...]
- 35. 35© Cloudera, Inc. All rights reserved. Spark Architecture
- 36. 36© Cloudera, Inc. All rights reserved. YARN Architecture Resource Manager Node Manager Container Container Node Manager Container Container Application Master Client Process Process
- 37. 37© Cloudera, Inc. All rights reserved. Spark on YARN Architecture Resource Manager Node Manager Container Container Node Manager Container ContainerClient Process Process
- 38. 38© Cloudera, Inc. All rights reserved. Spark on YARN Architecture Resource Manager Node Manager Container Container Node Manager Container Container Application Master Client Process Process
- 39. 39© Cloudera, Inc. All rights reserved. spark-submit --executor-memory 2g --master yarn-client --num-executors 2 --num-cores 2
- 40. 40© Cloudera, Inc. All rights reserved. Container [pid=63375,containerID=container_1388158490598_0001_01_00 0003] is running beyond physical memory limits. Current usage: 2.2 GB of 2.1 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used. Killing container. [...]
- 41. 41© Cloudera, Inc. All rights reserved. Container [pid=63375,containerID=container_1388158490598_0001_01_00 0003] is running beyond physical memory limits. Current usage: 2.2 GB of 2.1 GB physical memory used; 2.8 GB of 4.2 GB virtual memory used. Killing container. [...]
- 42. 42© Cloudera, Inc. All rights reserved. spark-submit --executor-memory 2g --master yarn-client --num-executors 2 --num-cores 2
- 43. 43© Cloudera, Inc. All rights reserved. yarn.nodemanager.resource.memory-mb Executor Container spark.yarn.executor.memoryOverhead (7%) (10% in 1.4) spark.executor.memory spark.shuffle.memoryFraction (0.4) spark.storage.memoryFraction (0.6) Memory allocation
- 44. 44© Cloudera, Inc. All rights reserved. Sometimes jobs run slow or even… Courtesy of: http://blog.sdrock.com/pastors/files/2013/06/time-clock.jpg
- 45. 45© Cloudera, Inc. All rights reserved. java.lang.OutOfMemoryError: GC overhead limit exceeded at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1986) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) [...]
- 46. 46© Cloudera, Inc. All rights reserved. GC Stalls
- 47. 47© Cloudera, Inc. All rights reserved. Too much spilling! Courtesy of: http://tgnp.me/wp-content/uploads/2014/05/spilled-starbucks.jpg
- 48. 48© Cloudera, Inc. All rights reserved. Shuffle Boundaries maptextFile filter group ByKey hadoop File map join map Shuffle
- 49. 49© Cloudera, Inc. All rights reserved. Most performance issues are in shuffles!
- 50. 50© Cloudera, Inc. All rights reserved. Inside a Task: Fetch & Aggregate ExternalAppendOnlyMapBlock Block deserialize deserialize key1 -> values key2 -> values key3 -> values key4 -> values Sort & Spill key1 -> values key2 -> values key3 -> values
- 51. 51© Cloudera, Inc. All rights reserved. rdd.reduceByKey(reduceFunc, numPartitions=1000) Inside a Task: Specify partitions
- 52. 52© Cloudera, Inc. All rights reserved. Why not set partitions to ∞ ?
- 53. 53© Cloudera, Inc. All rights reserved. Excessive parallelism • Overwhelming scheduler overhead • More fetches -> more disk seeks • Driver needs to track state per-task
- 54. 54© Cloudera, Inc. All rights reserved. So how to choose? • Easy answer: • Keep multiplying by 1.5 and see what works
- 55. 55© Cloudera, Inc. All rights reserved. Is Spark bad? Courtesy of: https://theferkel.files.wordpress.com/2015/04/250474-breaking-bad-quotes.jpg
- 56. 56© Cloudera, Inc. All rights reserved. Thank you
Editor's Notes
- Early on a colleague of ours sent us this exception… this is truncated This talk is going to be about these kinds of errors you sometimes get when running…
- This is probably the most common failure you’re going to see. First of all, in this case, the punchline here is going to be that the problem is your fault. But second of all, what does all this other stuff mean and why is Spark telling me this in this way.
- This is probably the most common failure you’re going to see. First of all, in this case, the punchline here is going to be that the problem is your fault. But second of all, what does all this other stuff mean and why is Spark telling me this in this way.
- Lets start with an example program in Spark.
- The sum() call launches a job
- Lets start with an example program in Spark.
- A chunk of data somewhere Could be on Hadoop File System (HDFS) Could be cached in Spark Defines the degree of parallelism
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- Narrow map filter Wide join groupByKey
- More details: A job is a DAG of stages The scheduler creates a set of tasks per stage Partitions assigned to a task
- This is probably the most common failure you’re going to see. First of all, in this case, the punchline here is going to be that the problem is your fault. But second of all, what does all this other stuff mean and why is Spark telling me this in this way.
- Talk about the relationship between stages and tasks.
- So with all this information in hand, we can come back and interpret this error. We tried to run a job, but it failed because one of its stages failed. Why did that stage fail? Because one of its tasks failed. Tasks will be retried. Mercifully, Spark gives us the exception that caused the most recent failure.
- Mercifully, Spark gives us the exception that caused the most recent failure.
- Mercifully, Spark gives us the exception that caused the most recent failure.
- Lets review the general Spark architecture A driver Where the DAG scheduler lives Drives the sho Single point of failure Executors Communicates with driver Runs the tasks created by the driver Think of this as a ThreadPoolExecutor in java Pluggable cluster managers YARN, Mesos, standalone ----- Meeting Notes (6/10/15 14:57) -----
- Lets review the general Spark architecture
- Lets review the general Spark architecture
- Lets review the general Spark architecture
- This shows up in the YARN NodeManager logs
- mention that this is what happens with a groupByKey or reduceByKey show blocks being deserialized into Java objects and placed into map show spill with fewer tasks, more of these blocks have to go to the same reducer, and more stuff needs to be held in this map
- No Distributed systems are complicated