This document provides recommendations for optimizing Spark jobs. It suggests reducing I/O by running the Spark cluster on the same machines as the data. It recommends avoiding functions that collect data to the driver to reduce memory I/O. It also suggests using caching to avoid read I/O. The document discusses configuring resources like memory and cores and tuning configurations like backpressure to improve performance of Spark streaming jobs. Finally, it recommends using efficient serialization formats like Kryo, Avro and Parquet.

1. Spark Optimization

2. Reduce I/O
 If you are running spark-streaming or spark-sql (hive etc) or your data is residing in
any of distributing platform like in Hadoop as hdfs file or in S3 etc so to avoid
network I/O and for data locality its recommended to setup your spark cluster in
the same machines.
 To avoid the memory I/O or overhead memory try to avoid collectaslist kind a
function because it’ll send data to driver and then further redistribute it.
 To avoid the read I/O should use spark caching option like cache() or persist()
as pert your requirement.

3. Kafka Parallel Read
 Kafka give as option to create partition and bind those into a single consumer
group, can use this in spark to read data in parallel mode, because spark driver
read data from kafka sequentially for each partition but to do parallel consumption
from kafka topic from multiple machines, you have to instantiate multiple
Dstreams.
http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-
example-tutorial/

4. Resource Allocation
 In spark heap size can be control through --executor-memory (from command
line) or spark.executor.memory so as per your heap requirement you can tune it.
 Spark cores property control the number of concurrent task in executor, if we set --
executor-cores 3 each executor can run a maximum 3 tasks at the same time.
 The --num-executors command-line flag or spark.executor.instances
configuration property control the number of executors requested.
 We can set driver memory and cores through --driver-memory and --driver-
cores properties

5. Configuration Changes
 There might be chances in spark streaming where producer generate data faster
than consumer consume due to this memory overhead occur. Spark provide
configuration to handle this called backpressure using
spark.streaming.backpressure.enabled=true, we can avoid this situation.
 Streaming application is a long running process so frequent Garbage Collection
pauses occur and we want to minimize it so using this “--conf
"spark.executor.extraJavaOptions=-XX:+UseConcMarkSweepGC -
Dlog4j.configuration=log4j-eir.properties" “ we can do that.
 Again long running application does generate very large log files using
RollingFileAppender we can limit the size. Also we can turn off console logs using
spark.ui.showConsoleProgress.

6. Serialization and Data Fromat
 In general, Spark uses the deserialized representation for records in memory and
the serialized representation for records stored on disk or being transferred over
the network. The spark.serializer property controls the serializer that’s used to
convert between these two representations. The Kryo serializer,
org.apache.spark.serializer.KryoSerializer, is the preferred option.
 To avoid serialization issue recommend should use Avro, Parquet, Thrift kind a
formats.