Spark is an open-source cluster computing framework that allows processing of large datasets in parallel. It supports multiple languages and provides advanced analytics capabilities. Spark SQL was built to overcome limitations of Apache Hive by running on Spark and providing a unified data access layer, SQL support, and better performance on medium and small datasets. Spark SQL uses DataFrames and a SQLContext to allow SQL queries on different data sources like JSON, Hive tables, and Parquet files. It provides a scalable architecture and integrates with Spark's RDD API.

2. What is Spark?

3. Features of Apache Spark
Speed
Supports multiple languages
Advanced Analytics

4. Spark Built on Hadoop

5. Components of Spark

6. What is RDD?

7. Iterative Operations on Spark RDD

8. Interactive Operations on Spark
RDD

9. Spark Streaming

10. What is SparkContext?

11. Initializing a SparkContext in
Scala
 Import org.apache.spark.SparkConf
 Import org.apache.spark.SparkContext
 Import org.apache.spark.SparkContext._
 Val conf = new
SparkConf().setMaster("local").setAppN
ame("My App") valsc = new
SparkContext(conf)

12. Limitation of Hive
 HIVE uses Map-Reduce which lags in
performance with medium and small
sized data-sets(<200GB).
 No resume capability.
 Hive content drop encrypted Databases.
(SPARK SQL WAS BUILT TO
OVERCOME THE LIMITATIONS OF
APACHE HIVE RUNNING ON TOP OF
SPARK.)

13. What is Spark SQL?

14. Features of Spark SQL
Integrated
Unified Data Access
Hive Compatibility
Standard Connectivity
Scalability

15. Advantages of Spark SQL

16. Spark SQL Architecture

17. Spark SQL-DataFrames

18. Spark SQLContext

19. Spark SQL – Data Sources
A DataFrame interface allows different Data Sources to
work on Spark SQL. It is a temporary table and can be
operated as a normal RDD. Registering a DataFrame as a
table allows you to run SQL queries over its data.
There are different types of data sources available in
Spark SQL, some of which are listed below –
 JSON Datasets
 Hive Tables
 Parquet Files

20. Data Analysis Flow Diagram
Fig: Spark SQL Flow Chart

21. SWITCHING TO SPARK-
SHELL

22.  SWITCHING FROM SPARK-CONTEXT TO
SQL

23. IMPORTING ALL PACKAGES(TO
CONVERT RDD TO DATA-SET)

24. READING THE FILE AND SCHEMA
CHECKING DONE

25. Scala>df.registerTemoTable(“Terror”)
We have to register this table class as a temp table

27. Classify attacks on the basis of gang name.

28. Classify the motive of attack on India.

29. Conclusion

30. References
 Useful Links on Spark SQL
 Spark SQL Wiki - Wikipedia Reference for Spark
SQL.
 https://data-flair.training/blogs/spark-sql-tutorial/
 Useful Books on Spark SQL
 O’Reilly- Learning Spark by Holden Karau, Andy
Konwinski, Patric Wendell &MateiZaharia
 O’Reilly- Advanced Analytics with Spark by Sandy
Ryza, Uri Laserson, Sean Owen & Josh Wills