deploy a spark application on EMR AWS

1. Deploy a Spark Application
on a Spark Cluster @
Elastic Map Reduce AWS
Dr. Rim Moussa
University of Carthage

2. Amazon S3 -Amazon Simple Storage Service
 Upload
– S3 bucket for Spark code: .jar
– S3 bucket for Data
 Uploads to S3 might be done via Terminal for big data sets
laumch EC2 instance to upload data into S3 bucket
curl ftp://ftp.ais.dk/ais_data/dk_csv_jun2018.zip | aws s3 cp -
s3://aisdma
 Manipulation of S3 buckets and files can be done via
Terminal
aws s3 ls s3://data.info
aws s3 cp s3://spark.jars/rm-1.0-veracity.jar .
aws s3 rm s3://data.info
2

3. Open Datasets on Amazon
 Amazon has a repository of big datasets
https://registry.opendata.aws/
 Amazon implements a program AWS Public Dataset Program
in order to democratize access to data and encourage the
development of communities that benefit from access to
shared datasets.
https://aws.amazon.com/opendata/public-datasets/
3

4. 4

5. 5

6. Amazon Elastic MapReduce
6

7. Amazon Elastic MapReduce
7

8. Amazon Elastic MapReduce
8

9. Amazon Elastic MapReduce
9

10. Hardware Tab
10

11. Cluster ready
11

12. ssh Master
12

13. → ssh Master
13

14. Click on master
14

15. Click on EC2 instance of master
15

16. Review Master Security Groups
16

17. Check “allow pinging and ssh”
17

18. → ssh master
18

19. Steps towards submitting a workflow
19

20. Create S3 bucket for Workflow output
20

21. Submit job to Spark master
21

22. Job result
22

23. Re-check data.info S3 bucket
23

24. Download results
24

25. Terminate cluster
25

26. Terminate cluster
26