Spark is an open-source unified analytics engine that can be used as a replacement for MapReduce. It processes large and diverse datasets using a master-slave principle. Spark has modules for working with structured data using Spark SQL, streaming data analysis with Spark Streaming, machine learning with MLib, and graph analysis with GraphX. The document describes using Spark for spam detection by creating separate RDDs for spam and non-spam emails, applying flatMap processing to transform text to vectors, and using Naive Bayes modeling to classify emails. It concludes that Spark is faster for analyzing a dataset of 10.5 million email records totaling 5GB in size.

1. SPAMDETECTION
By ,Finny, Omkar, Sreenivas

2. DATA
Big data refers to the large, diverse sets of
information that grow at ever-increasing
rates.
Variety
Volume Velocity
BIG

3. • Apache Spark
⚬ open-source unified analytics engine
⚬ large-scale data processing.
• Used as a replacement for MapReduce
• Processes data
⚬ using the Master-Slave principle.

4. Spark SQL is Apache
Spark's module for
working with
structured data.
Spark Streaming
streaming data can be
analyzed in real time
MLib
is a machine learning
library of Apache Spark
GraphX
Graph analysis can be
performed with the
GraphX library

5. The biggest harm of spam emails is that contrary to
popular belief, the
recipient faces more costs than the sender.

6. FLATMAP
collapse the elements of a collection to create a
single collection with elements of the same type
JAVA RDD
Resilient Distributed Datasets - serves as the
building blocks for distributed data processing in
Spark
SPARK CONTEXT
To perform data analysis with Spark, a Spark
Context is required. It serves as a bridge to access
data within the Spark environment.

7. Creating Spark
Context
Java RDD Structure
Separate RDDs for Spam
and Non-Spam Emails
FlatMap Process
Text to Vector
Transformation
Modeling with Naive
Bayes

8. 10.5 M
Records
5 GB
Emails

12. Average Working Time

13. Conclusio
nSpark is faster