Slides from the 2014 Import.io Data Summit. Includes some slides that have a getting started walk through for setting up Impala on Amazon EMR.

1. ianmas@amazon.com
@IanMmmm
LARGE SCALE DATA
ANALYSIS WITH AWS
Ian Massingham – Technical Evangelist

2. THE MORE DATA YOU COLLECT
THE MORE VALUE YOU CAN
DERIVE FROM IT!

5. THE COST OF DATA
GENERATION IS FALLING!

6. We are constantly producing more data

7. From all types of industries

10. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

11. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Lower cost,
higher throughput

12. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Lower cost,
higher throughput
Highly
constrained

13. + ELASTIC AND HIGHLY SCALABLE
+ NO UPFRONT CAPITAL EXPENSE
+ ONLY PAY FOR WHAT YOU USE
+ AVAILABLE ON-DEMAND
= REMOVE CONSTRAINTS

14. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

15. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
AWS Import / Export
AWS Direct Connect

16. Inbound data transfer is free
Multipart upload to S3
Physical media
AWS Direct Connect

17. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon S3,
Amazon Glacier,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
AWS Storage Gateway,
Data on Amazon EC2

18. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon EC2
Amazon Elastic
MapReduce

20. AMAZON ELASTIC
MAPREDUCE
HADOOP AS A SERVICE!

21. • SPLITS DATA INTO PIECES
• LETS PROCESSING OCCUR
• GATHERS THE RESULTS!

22. HDFS
EMRKinesis
S3 DynamoDB
Data management
Pig
Analytics languages/engines
RDS
Redshift AWS Data Pipeline

23. EMR + IMPALA DEMO

24. STARTING AN EMR CLUSTER
WITH HADOOP ECOSYSTEM
TOOLS PRE-INSTALLED

25. COPY & LOAD OUR DATASET
$
scp
–i
EMRKeyPair.pem
~/aws/hadoop/LHRarrivals*.csv
hadoop@ec2-­‐54-­‐76-­‐242-­‐238.eu-­‐
west-­‐1.compute.amazonaws.com:
$
ssh
–i
EMRKeyPair.pem
hadoop@ec2-­‐54-­‐76-­‐242-­‐238.eu-­‐west-­‐1.compute.amazonaws.com
$
hadoop
fs
-­‐mkdir
/data/
$
hadoop
fs
-­‐put
<uploaded_files>
/data/
$
hadoop
fs
-­‐ls
-­‐h
-­‐R
/data/
or at scale, Distributed Copy using S3DistCp to parallel load from S3
$
.
/home/hadoop/impala/conf/impala.conf
$
hadoop
jar
/home/hadoop/lib/emr-­‐s3distcp-­‐1.0.jar
-­‐Dmapreduce.job.reduces=30
-­‐-­‐
src
s3://s3bucketname/
-­‐-­‐dest
hdfs://$HADOOP_NAMENODE_HOST:$HADOOP_NAMENODE_PORT/
data/
-­‐-­‐outputCodec
'none'
** Run on a cluster master node

26. CREATE EXTERNAL TABLE
$
#check
the
size
of
our
data
set
$
wc
–l
LHRarrivals*.csv
850
LHRarrivals2.csv
1526
LHRarrivals.csv
2376
total
$
impala-­‐shell
Welcome
to
the
Impala
shell.
>
create
EXTERNAL
TABLE
flights
(
input
STRING,
id
BIGINT,
widget
STRING,
source
STRING,
resultnum
BIGINT,
pageurl
STRING,
scheduled
STRING,
flightnumber
STRING,
airport
STRING,
status
STRING,
terminal
STRING
)
ROW
FORMAT
DELIMITED
FIELDS
TERMINATED
BY
','
LOCATION
'/data/';
>
select
count
(*)
from
flights;
Should
return
count(*)
2376
reflecting
the
size
of
the
data
set

27. DEMO OF ODBC ACCESS
Doing this part on Amazon WorkSpaces using the Simba Cloudera
Impala ODBC Driver.!
Set up an SSH tunnel to the master node to allow us to connect to port
25010 from the WorkSpaces desktop to the Impala ODBC port!
A previously configured system DSN allows us to work with the data from
our EMR/Impala cluster directly within Microsoft Excel!

28. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon S3,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
Data on Amazon EC2

29. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

30. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
BATCH
PROCESSING

31. GENERATE ➔ ➔ SHARE!
STREAM
PROCESSING

32. AMAZON KINESIS
REAL-TIME DATA STREAM PROCESSING!

33. Real-time response to content
in semi-structured data streams
Relatively simple computations
on data (aggregates, filters,
sliding window, etc.)

34. Hourly server logs: how your
systems went wrong an hour ago
Weekly / Monthly Bill: What you
spent this past billing cycle
Daily customer report from your
website: tells you what deal or ad
to try next time
Daily fraud reports: tells you if there
was fraud yesterday
Daily business reports: tells me
how customers used AWS services
yesterday
Real-time metrics: what just went
wrong now
Real-time spending alerts/caps:
guaranteeing you can’t overspend
Real-time analysis: what to offer
the current customer now
Real-time detection: blocks
fraudulent use now
Fast ETL into Amazon Redshift:
how are customers using services
now

35. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!

36. GENERATE ➔ STORE ➔ ANALYZE ➔ SHARE!
Amazon S3,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
Data on Amazon EC2
Amazon EC2
Amazon Elastic
MapReduce
Amazon S3,
Amazon Glacier,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
AWS Storage Gateway,
Data on Amazon EC2
AWS Import / Export
AWS Direct Connect

37. GENERATE ➔ ➔ SHARE!
STREAM
PROCESSING

38. GENERATE ➔ ➔ SHARE!
STREAM
PROCESSING
Amazon S3,
Amazon DynamoDB,
Amazon RDS,
Amazon Redshift,
Data on Amazon EC2
Amazon Kinesis
Stream Processing on
Amazon EC2

39. WANT TO KNOW MORE?
aws.amazon.com/solutions/case-studies/big-data/!

40. ianmas@amazon.com
@IanMmmm
LARGE SCALE DATA
ANALYSIS WITH AWS
Ian Massingham – Technical Evangelist