This document discusses running Sqoop jobs on Apache Spark for faster data ingestion into Hadoop. The authors describe how Sqoop jobs can be executed as Spark jobs by leveraging Spark's faster execution engine compared to MapReduce. They demonstrate running a Sqoop job to ingest data from MySQL to HDFS using Spark and show it is faster than using MapReduce. Some challenges encountered are managing dependencies and job submission, but overall it allows leveraging Sqoop's connectors within Spark's distributed processing framework. Next steps include exploring alternative job submission methods in Spark and adding transformation capabilities to Sqoop connectors.

1. SQOOP on SPARK
for Data Ingestion
Veena Basavaraj & Vinoth Chandar
@Uber

2. Currently @Uber on streaming systems.
@Cloudera on Ingestion for Hadoop.
@Linkedin on front-end service infra.
2
Currently @ Uber focussed on building
a real time pipeline for ingestion to
Hadoop. @linkedin lead on Voldemort.
In the past, worked on log based
replication, HPC and stream
processing.

3. Agenda
• Sqoop for Data Ingestion
• Why Sqoop on Spark?
• Sqoop Jobs on Spark
• Insights & Next Steps
3

4. Sqoop Before
4
SQL HADOOP

5. Data Ingestion
• Data Ingestion needs evolved
– Non SQL like data sources
– Messaging Systems as data sources
– Multi-stage pipeline
5

6. Sqoop Now
• Generic data
Transfer Service
–FROM ANY
source
–TO ANY
target
6
MYSQL KAFKA
HDFS MONGO
FTP HDFS
FROM TO
KAFKA MEMSQL

7. Sqoop How?
• Connectors represent
Pluggable Data Sources
• Connectors are
configurable
•LINK configs
•JOB configs
7

8. Sqoop Connector API
8
Source Targetpartition()
**No Transform (T) stage yet!
extract() load()

9. Agenda
• Sqoop for Data Ingestion
• Why Sqoop on Spark?
• Sqoop Jobs on Spark
• Insights & Next Steps
9

10. It turns out…
• MapReduce is slow!
• We need Connector APIs to support (T)
transformations, not just EL
• Good news! - Execution Engine is also
pluggable
10

11. Why Apache Spark ?
• Why not ? ETL expressed as Spark jobs
• Faster than MapReduce
• Growing Community embracing Apache
Spark
11

12. Why Not Use Spark Data Sources?
12
Sure we can ! but …

13. Why Not Spark DataSources ?
• Recent addition for data sources!
• Run MR Sqoop jobs on Spark with
simple config change
• Leverage incremental EL & job
management within Sqoop

14. Agenda
• Sqoop for Data Ingestion
• Why Sqoop on Spark?
• Sqoop Jobs on Spark
• Insights & Next Steps
14

15. Sqoop on Spark
•Creating a Job
•Job Submission
•Job Execution
15

16. Sqoop Job API
• Create Sqoop Job
–Create FROM and TO job configs
–Create JOB associating FROM and TO configs
• SparkContext holds Sqoop Jobs
• Invoke SqoopSparkJob.execute(conf, context)
16

17. Spark Job Submission
• We explored a few options.!
– Invoke Spark in process within the Sqoop Server to
execute the job
– Use Remote Spark Context used by Hive on Spark to
submit
– Sqoop Job as a driver for the Spark submit command
17

18. Spark Job Submission
• Build a “uber.jar” with the driver and all the sqoop
dependencies
• Programmatically using Spark Yarn Client ( non public) or
directly via command line submit the driver program to
yarn client/
• bin/spark-submit —class org.apache.sqoop.spark.SqoopJDBCHDFSJobDriver
--master yarn /path/to/uber.jar —confDir /path/to/sqoop/server/conf/ —
jdbcString jdbc://myhost:3306/test —u uber —p hadoop —outputDir hdfs://
path/to/output —numE 4 —numL 4
18

19. Spark Job Execution
19
MySQL
partitionRDD extractRDD
.map() .map()
loadRDD.collect()

20. Spark Job Execution
SqoopSparkJob.execute(…)
List<Partition> sp = getPartitions(request,numMappers);
JavaRDD<Partition> partitionRDD = sc.parallelize(sp, sp.size());
20
1
2
3
JavaRDD<List<IntermediateDataFormat<?>>> extractRDD =
partitionRDD.map(new SqoopExtractFunction(request));
extractRDD.map(new SqoopLoadFunction(request)).collect();

21. Spark Job Execution
21
MySQL
Compute Partitions Extract from
MySQL
.map() .mapPartition()
Load
into HDFS
.repartition()
Repartition
to
limit
files on HDFS

22. Agenda
• Sqoop for Data Ingestion
• Why Sqoop on Spark?
• Sqoop Jobs on Spark
• Insights & Next Steps
22

23. Micro Benchmark: MySQL to HDFS
23
Table w/ 300K records, numExtractors = numLoaders

24. Table w/ 2.8M records, numExtractors = numLoaders
good partitioning!!
Micro Benchmark: MySQL to HDFS
24

25. What was Easy?
• NO changes to the Connector API
required.
• Inbuilt support for Standalone and Yarn
Cluster mode for quick end-end testing
and faster iteration
• Scheduling Spark sqoop jobs via Oozie
25

26. What was not Easy?
• No clean public Spark Job Submit API.
Using Yarn UI for Job status and health.
• Bunch of Sqoop core classes such as IDF
had to be made serializable
• Managing Hadoop and spark dependencies
together in Sqoop caused some pain
26

27. Next Steps!
• Explore alternative ways for Spark Sqoop
Job Submission with Spark 1.4 additions
• Connector Filter API (filter, data masking)
• SQOOP-1532
– https://github.com/vybs/sqoop-on-spark
27

28. Sqoop Connector ETL
28
Source Targetpartition()
**With Transform (T) stage!
extract() Transform()
load()

29. Questions!
• Thanks to the Folks @Cloudera
and @Uber !
• You can reach us @vybs,
@byte_array
29