AWS Black Belt Online Seminar 2017 Amazon EMR

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
w
, +1(++( +
r4JF 5YNP 5RYa B YV R FRZV N_s
4ZNg 8@E

•
z
•
w
× w
w w ×
• 7MI w
7 Rk _ CH& 7 Rk _ 9I

v
• – ()/ )) ) n x v v k
–4JF Uaa 2 Nd &NZNg &P Z# vk
• – x zj 4JF
zu j4JF v k
• – v k x z n j
v k
• 7MI U Vd _ e WWVc SZ_UZ_X acZTV bf eVd( 7MI acZTZ_X Zd afS ZT j RgRZ RS V R_U Zd
dfS VTe e TYR_XV Z_ RTT cUR_TV hZeY eYV 7MI 9fde Vc 7XcVV V_e RgRZ RS V Re
Yeea4))Rhd(R Rk _(T )RXcVV V_e)( 7_j acZTZ_X Z_W c ReZ _ Z_T fUVU Z_ eYZd
U Tf V_e Zd ac gZUVU _ j Rd R_ VdeZ ReV W fdRXV TYRcXVd W c 7MI dVcgZTVd SRdVU
_ TVceRZ_ Z_W c ReZ _ eYRe j f YRgV ac gZUVU( C _eY j TYRcXVd hZ SV SRdVU _
j fc RTefR fdV W 7MI dVcgZTVd& R_U Rj gRcj Wc eYV VdeZ ReVd ac gZUVU(

4TR QN
•
• 7 Rk _ CH
• 7 Rk _ CH w , +0)+ $
• 7 Rk _ CH .(i
• w 7 Rk _ CH
•

–~

4ZNg 8@E
❏ , 3 w
• CRaHVUfTV p +
❏ >RU a ×
• FZX& >ZgV& >8RdV& FcVde & >fV&
>RU a ,(i& IaRc ,(i& Z_
❏ 7MI w
• Ia e& 7C& :j_R :8& LF9&
9 fUMReTY& 9B & AZ_VdZd& ACI
→ – @N ERQbPR
– vm
Storage
S3 (EMRFS), HDFS
YARN
Cluster Resource Management
Batch
MapReduce
Interactive
Tez
In Memory
Spark, Flink
Applications
Hive, Pig, Spark SQL/Streaming/ML, Mahout, Sqoop
HBase/Phoenix
Presto
Hue (SQL Interface/Metastore Management)
Zeppelin (Interactive Notebook)
Ganglia (Monitoring)
HiveServer2/Spark Thriftserver (JDBC/ODBC)
Amazon EMR service

n
S3
EMR
EMR Redshift
Machine
Learning
Collect Process Analyze
Store
Data Collection
and Storage
Data
Processing
Event
Processing
Data
Analysis
Data Answers
Athena
Quicksight
Kinesis
DynamoDB
AWS IoT
AWS Lambda
KCL Apps Glue

❏ w
)
• CRdeVc)9 cV)JRd _ UV
• CH I
• 8 edecRa 7TeZ _& IeVad
n
❏
w
• >RU a& O7HD
• >ZgV& FZX& >8RdV& IaRc &
Z_ & FcVde & VeT(
4ZNg 8@E

4ZNg 8@E
n

Task Node
Task Instance Group
4ZNg 8@E n
security group
security group
Master Node
Master Instance Group
Amazon
S3
Amazon
DynamoDB
Amazon
Kinesis
Core Node
Core Instance Group
HDFS HDFS
HDFS HDFS
Task Node
Task Instance Group
w >: I
7MI w
AWS Glue
Data Catalog

4ZNg 8@E @N aR_ aN PR :_ b
❏ CRdeVc D UV +
• RZ gVc
❏ w
• DR VD UV
HVd fcTVCR_RXVc
• 9 cV D UV JRd D UV
Master Node
Master Instance Group
Hadoop1: JobTracker
Hadoop2: ResourceManager
HDFS: NameNode
Hive: HiveServer2, MetaStore
Presto: Coordinator

4ZNg 8@E 6 _R aN PR :_ b
❏ + 9 cV D UV
❏ w
• D UVCR_RXVc
• :ReRD UV w
>: I
• 8Ip
EA
HDFS
Core Node
Core Instance Group
Hadoop1: TaskTracker
Hadoop2: NodeManager
HDFS: DataNode
Presto: Worker

4ZNg 8@E 6 _R aN PR :_ b
❏ 9 cV D UV
• >: I
• 9FK)H7C
❏ >: I
p
• .(+(
HDFS
Core Node
Core Instance Group
HDFS: DataNode
Presto: Worker

4ZNg 8@E GN aN PR :_ b
❏ >: I
9 cV
❏ >: I w 9 cV
D UV
❏ >: I
• >: I w
:VT ZddZ _$
Task Node
Task Instance Group
Presto: Worker

4ZNg 8@E GN aN PR :_ b
❏ =c fa
• Ia e SZU
• _deR_TV JjaV
❏ H
p
Ia e
Task Instance Group 2
Task Instance Group 1
c3.xlarge * 2
bid: $0.1
r3.xlarge * 2
bid: $0.5

F a aN PR
Task Instance GroupCore Instance Group
❏ ❏
❖ B QRZN Q
9 cV D UV
❖ ”
×
E_ UV R_U
❖ F a aN PR JRd
D UV
❖ E_ UV R_U
3
w
4 .(i RcXV 0 + +, p/ $
6 _R A QR B QRZN Q# 4 (-+2o+ Yo0
GN A QR F a# 4 (-+2o/Yo0 % ( .o/Yo0
, +1 ++ + $ w qIa e q
5 +3( 2 +
5 + (1. /

4ZNg 8@E
l

8@E9F2 4ZNg F 79F w
❏ w
❏
• w
❏ w
• w
❏ w
EMR
EMR
Amazon
S3

8@E9F
❏ ld-4))n ”
❏ 7 Rk _ I-
• 4 w 7 Rk _ = RTZVc
❏ 7 Rk _ I- w s
• 7 Rk _ I- w
❏ w

8@E9F 6 V aR a IVRd
❏ 7 Rk _ I-
• FKJ
HVRU RWeVc hcZeV T _dZdeV_Tj$
• FKJ): B J gV_efR
T _dZdeV_Tj$
❏ CH I 9 _dZdeV_e LZVh
• 7 Rk _ :j_R :8 w
• BZde 9 _dZdeV_Tj
• :V VeV 9 _dZdeV_Tj
• HVRU RWeVc hcZeV 9 _dZdeV_Tj
•
Amazon S3 Amazon DynamoDB
CH I
w
Yeead4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV)V c a R_ T _dZdeV_e gZVh(Ye

4ZNg 8@E – v
❏ ”
•
•
❏ w 7 Rk _ I-
• w
• w
t

4ZNg 8@E 2 5 a a_N 4PaV
❏ D UV
• EA
• 8RdY& HfSj& FjeY _& VeT(
• 7 Rk _ I- p
• 7aRTYV 8ZXe a w
w p8 edecRa
Yeea4))Rhd(ejaVaRU(T )dR a), +0) 3)SfZ UZ_X R_U UVa jZ_X Tfde Raa ZTReZ _d hZeY RaRTYV 8ZXe a R_U R Rk _ V c(Ye

4ZNg 8@E 2 FaR
❏ )
• p
• 4 JB >ZgVGB
❏ 7 Rk _ I- KH p p w
• I- Rc
• IecVR Z_X& >ZgV& FZX& IaRc CH w T R_U cf__Vc( Rc $
• dTcZae cf__Vc( Rc SRdY
❏ IeVa
7fe eVc Z_ReV$

4ZNg 8@E
y
❏ IeVa
• 7 Rk _ CH
• w w 9B
• 7MI I:A
• 7C
• )
•
❏ 7MI w
• BR SUR
• :ReR FZaV Z_V
❏ w
Amazon EMR
Amazon EMR
Step API
AWS Data Pipeline
Airflow, Luigi, or other
schedulers on EC2
AWS Lambda

4ZNg 8@E
y
❏ w w
• CRdeVc D UV ddY p p
IIC 7 Rk _ 9, IjdeV d CR_RXVc$ Hf_ T R_U
• >ZgVIVcgVc,& daRc dfS Ze
• PVaaV Z_& >fV& daRc SdVcgVc =K
• BZgj HVde IaRc
• E kZV w w
Web UIs: Hue SQL editor, Zeppelin
notebooks,
R Studio, and more!
Connect with ODBC / JDBC using
HiveServer2/Spark Thriftserver
Use Spark Actions in your Apache
Oozie workflow to create DAGs of jobs.
(start using
start-thriftserver.sh)
Or, use the native APIs and CLIs for
each application

C_VcNaR FbO Ra 8@E z
❏ IfS_Ve FcZgReV
• CH .(,(
❏ FfS ZT
• I-):j_R :8
LF9 V_Ua Z_e
• I-):j_R :8 w
D7J =M
• CH IVcgZTV
IVTfcZej =c fa
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +/)+,) Rf_TY R Rk _ V c T fdeVcd Z_ R Rk _ gaT acZgReV dfS_Ved)

4 NPUR F N_ GRg NQ @N ERQbPR
C_R a n z
Yeea4))U Td(Rhd(R Rk _(T )V c) ReVde)HV VRdV=fZUV)V c UReR V_TcjaeZ _ aeZ _d(Ye
❏ Re cVde pZ_ ecR_dZe p
w
❏ Re cVde
• CH I& >: Ip p w
× w
❏ Z_ ecR_dZe
• JBI
❏ CH ×
• 9 fUW c ReZ _

4ZNg 8@E n (). )( #

4ZNg 8@E n (). )( #
❏ 7aRTYV Z_ w
❏ w
❏ w w w
❏ 7aRTYV >8RdV w I-
❏ 7 Rk _ 9 fUMReTY gV_ed
❏ w w
❏ 7C w w
❏ Z- dVcZVd 7 Rk _ 9, Z_deR_TVd w
❏ 7 Rk _ CH w

4 NPUR 9YV n
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)_ h cf_ cVR eZ V decVR ac TVddZ_X Re dTR V hZeY RaRTYV W Z_
_ R Rk _ V c)
❏ HV VRdV /(+( w
❏ HV VRdV /(3( p7aRTYV Z_ +(-(,
• w p
• IaRc p pIGB
• 7 Rk _ AZ_VdZd IecVR d w
• w
• ViRTe j _TV ×
• wp × w

n
y z
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced WZ_V XcRZ_VU RTTVdd T _ec
fdZ_X T fdeVc eRXd)
❏ CH w p 7C
w
– + +( :Vae5: L w p
×
– + ,( :Vae5: Lp pJVde
w pz
– , +( :Vae5: L p w
– , ,( :VaezLR fV pz
❏ w w 9 fdeVc :
– ( :VdTcZSV9 fdeVc pBZde9 fdeVcd

n n n
❏ HV VRdV .(i /(i w
❏ w
– HV VRdV /(+(
• p
p
w
p w
•
– HV VRdV /(+(
• p w
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced Rfe dTR Z_X R_U T _WZXfcRS V dTR V
U h_ SVYRgZ c)

4 NPUR 5N R n
F z
❏ HV VRdV /(,( w
– w p>8RdV w
I- >: I
– w I- w
❏ HV VRdV /(1(
– >8RdV I-
w
❏ KF
– CH w w I-
– CH I 9 _dZdeV_e LZVh
– M7B Z_ V cj W c McZeV
– /=8 c Ie cV
– Z_ V cj) _ UZd W c HVRU
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)++)R Rk _ V c _ h dfaa ced fdZ_X R Rk _ d- Rd R UReR de cV
W c RaRTYV YSRdV)

4ZNg 6Y bQJNaPU 8cR a
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0)+,)R Rk _ V c _ h dfaa ced VgV_ed R_U _ eZWZTReZ _d hZeY R Rk _
T fUhReTY VgV_ed)
❏ CH
– w
• IJ7HJ D=& HKDD D=& M7 J D=& J HC D7J :& J HC D7J :QM J>Q HHEHI
– w
• Wc H I P D= e HKDD D=z & Wc HKDD D= e H I P D=z
– w w w
• F D: D=& 7JJ79> :& : J79> :& 7 B :
– ×
• F D: D=& 97D9 BQF D: D=& HKDD D=& 9ECFB J :& 97D9 BB :& 7 B :

n n
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c R__ f_TVd Z_deR_TV W VVed W c R Rk _ VT, da e
Z_deR_TVd R_U da e S T d)
❏ p CH w
❏
– w p / r
× r
–
– p+ 0
– C9 × g9FK
• g9FK p 9, g9FK p
CH O7HD D UVCR_RXVc gT cV
– w w
• w × p w w

Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c R__ f_TVd Z_deR_TV W VVed W c R Rk _ VT, da e
Z_deR_TVd R_U da e S T d)
n n
+( ×
,(
-(
.( c ×
/( 8I w
0(
1( g9FK

4ZNg V be 4@
n n
Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1) 1)R Rk _ V c _ h dfaa ced Rf_TYZ_X T fdeVcd hZeY Tfde
R Rk _ Z_fi R Zd)
❏ 7 Rk _ BZ_fi 7 Rk _ CRTYZ_V
RXV 7C $ 7 Rk _ CH w
❏ 7C w p
w 7C
❏ w /(1( 7 Rk _ CH w

V n 4ZNg 86 V aN PR n
BPa OR_ ) ()/#
Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1)+ )R Rk _ V c _ h dfaa ced Z- Z_deR_TVd)
❏ w 7 Rk _ 9, w
Z-
❏ Z- pD _ L ReZ V CV cj iacVdd
DLCV$ II: w p E
E
❏ w /(3( 7 Rk _ CH w

4ZNg 8@E n
Yeead4))Rhd(R Rk _(T ) a)RS fe Rhd)hYRed _Vh), +1)+ )R Rk _ V c _ h dfaa ced aVc dVT _U SZ Z_X)
❏ , +1 + , p 7MI w
❏ w
w
❏ w +
❏ p+
+ $

4ZNg 8@E ,&e

4ZNg 8@E ERYRN R ()- / #
• HV VRdV .( ( m , +/) 1
• HV VRdV .(+( m , +/) 3
• HV VRdV .(,( m , +/)++
• HV VRdV .(-( m , +0) +
• HV VRdV .(.( m , +0) -
• HV VRdV .(/( m , +0) .
• HV VRdV .(0( m , +0) .
• HV VRdV .(1(i m , +0) 0 1
• HV VRdV .(2(i m , +0) 2 , +1) ,
• HV VRdV .(3(+ m , +1) -
• HV VRdV .(3(, m , +1) 1
7aRTYV 8ZXe a w w ×
+, +, w
>RU a ×
Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV)V c cV VRdV T a _V_ed(Ye
• HV VRdV /( ( m , +0) 2
• HV VRdV /( (- m , +0)+
• HV VRdV /(+( m , +0)++
• HV VRdV /(,(i m , +0)+,
• HV VRdV /(-(i m , +1) + ,
• HV VRdV /(.( m , +1) ,
• HV VRdV /(/( m , +1) .
• HV VRdV /(0( m , +1) 0
• HV VRdV /(1( m , +1) 1
• HV VRdV /(2( m , +1) 2
• HV VRdV /(3( m , +1)+

4ZNg 8@E ERYRN R
❏ w
• 7 Rk _ CH p w w
❏ w
• p w
❏ w w p
• GfZT 9cVReV
• pCR_RXV V_e =fZUV HV VRdV =fZUV
7C w w p
http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-release-differences.html

ERYRN R2 n
❏ w w
• w
• 4
• jRc_(cVd fcTV R_RXVc(hVSRaa(RUUcVdd a ce 3 ,0 56 2 22
• )Y V)YRU a)YZgV 56 )fdc) ZS)YZgV& )VeT)YZgV)T _W& )gRc) X)YZgV
❏ w w w
• YRU a w ”
• jRc_ w pYZgV w

ERYRN R2 n
❏ 9 RddZWZTReZ _ % Fc aVceZVd
• 8 edecRa 7TeZ _
• w ”
Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV)V c T _WZXfcV Raad(Ye
e.g. /etc/hadoop/conf/mapred-site.xml

ERYRN R2 n j
❏ w w GfZT 9cVReV
• p w p × - ”
• w = e RUgR_TVU aeZ _d
❏ w
• CR_RXV V_e =fZUV
• 7 Rk _ CH w w
• Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV
• HV VRdV =fZUV
• w w
• Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV

n o-&1&(#
Yeea4))U Td(Rhd(R Rk _(T ))V c) ReVde)HV VRdV=fZUV)V c cV VRdV T a _V_ed(Ye

ERYRN R ,&ey -&e
❏ w
• IaRc ,(i
• FPNYN &))
• >ZgV ,(i
• Z_
• BZgj
❏
• NcN 0
• JVk >ZgV& FZX$
❏
• IeVa
❏ IR_US i
• w
• IR_US i

4ZNg 8@E 4 YVPNaV S _ -&1&(
• >RU a& >ZgV& JVk&
IaRc & FcVde & FZX&
CRY fe& Ib a& Z_
w w
• >8RdV& FY V_Zi
=K
• >fV& PVaaV Z_
w w
• E kZV
• =R_X ZR& >9ReR X&
P VVaVc& YVcf

4 YVPNaV 2
❏ >RU a
• qO7HD w $%>: I w $ p
qCRaHVUfTV + q CH
O7HD p q
❏ >ZgV
• O7HD p IGB q
CRaHVUfTV qI- w q
❏ JVk
• O7HD :7= $ q
>ZgV ” p >ZgV q

4 YVPNaV 2
❏ IaRc
• w w qO7HD pCRaHVUfTV
qIaRc IaRc IGB >ZgV $ IaRc
IecVR Z_X w $& CB ZS $ p
q
❏ Z_
• w w q O7HD p w
p qIaRc p
pIGB q
w ViRTe j _TV × q
❏ FcVde
• IGB qO7HD ” pIGB
q w E D qIGB
7DI IGB q

4 YVPNaV 2
❏ FZX
• CRaHVUfTV :IB q
q
❏ CRY fe
• CRaHVUfTV IaRc q
q
❏ Ib a
• H:8CI p>: I I- w CRaHVUfTV
q

4 YVPNaV 2 n n
❏ >8RdV
• >: I ALIq w
w q
❏ FY V_Zi
• >8RdV w IGB w q

4 YVPNaV 2 :H n n
❏ >fV
• >ZgV ×w p =K q
❏ PVaaV Z_
• =K IaRc w IaRc IGB p
p s w q
❏ E kZV
• w w w qCRaHVUfTV
>ZgV q

4 YVPNaV 2
❏ =R_X ZR
• w 9FK&CV cj&DVeh c $
p q
❏ >9ReR X
• >ZgV ×w FZX w q
❏ P VVaVc
• w w pALI p w w
q >8RdV $ q

4 YVPNaV 2
❏ BZgj
• IaRc CH w H IJ w p
pHF9
w w

n 4ZNg 8@E

4ZNg 8@E x n
❏ w w
•
” w
• HV VRdV BRSV )7C LVcdZ _
w p w
❏ w w
• 7aRTYV 8ZXe a
w w
p8 edecRa
• w
7C
Yeea4))Rhd(ejaVaRU(T )dR a), +0) 3)SfZ UZ_X R_U UVa jZ_X Tfde Raa ZTReZ _d hZeY RaRTYV 8ZXe a R_U R Rk _ V c(Ye

n 2 NQ
❏ O7HD 5 OVe 7_ eYVc HVd fcTV
DVX eZRe c
❏ >RU a, w
• SJcRT Vc
❏ HVd fcTV HVbfVde w w
• 7aa ZTReZ _CRdeVc
p w
HVd fcTVCR_RXVc
MapReduce
Pig Hive Other
Other
HDFS
Other
Map
ReducePig Hive
YARN
HDFS
Hadoop V1
Hadoop V2

K4EA2 ER b_PR@N NTR_
❏ w w
❏ w w
• 9FK& CV cj& VeT(
❏ w
❏
• 7aa ZTReZ _CRdeVc
Yeea4))YRU a(RaRTYV( cX)U Td)TfccV_e)YRU a jRc_)YRU a jRc_ dZeV)O7HD(Ye

K4EA2 A QR@N NTR_
❏ w w
❏ w w
HC
❏ w 9 _eRZ_Vc
• 9 _eRZ_Vc

K4EA2 6 aNV R_
❏ w w
❏ DC
❏
• :VWRf e9 _eRZ_Vc iVTfe c
•
• BZ_fi9 _eRZ_Vc iVTfe c
• TXc fad
• : T Vc9 _eRZ_Vc iVTfe c
• : T Vc

K4EA2 4 YVPNaV @N aR_
❏
9 _eRZ_Vc
• SJcRT Vc
• + 7C
❏
9 _eRZ_Vc

4ZNg 8@E K4EA
❏ CRaHVUfTV& >ZgV& FZX& IaRc _ O7HD
• 7 Rk _ CH pO7HD w
❏ ×
• CH >ZgV IaRc p w

n 2 VcR
❏ IGB Z V p w
w
❏ w w w p
×w
• >: I w w
• 7 Rk _ CH CH I 7 Rk _ I-$
• p7 Rk _ AZ_VdZd& 7 Rk _ :j_R :8
❏
• CRaHVUfTV& JVk& IaRc
Yeead4))YZgV(RaRTYV( cX)

VcR
Metastore
HDFS
Cluster
Amazon
S3
HiveServer2
CREATE
TABLE
SELECT
FROM
+4 ×w
>: I 7 Rk _ I-
w
,4 ×w
w
-4 ×w
.4 ×w
w
/4
04 w
w
Amazon
DynamoDB
Amazon
Kinesis

VcR @RaN a _R
❏ IGB
CVeRde cV
❏ CVeRde cV CjIGB
• 7 Rk _ CH CRdeVc
D UV w
• p7 Rk _ H:I
❏ w 7MI = fV
:ReR 9ReR X CVeRde cV
CVeRde cV
Amazon S3
HDFS

v n BE6 9VYR CN_ bRa
❏ w
• w
•
• EaeZ ZkVU H h 9 f _Rc EH9$ Z V ) FRcbfVe
❏ w
• +
• + p w
• J NJ B & I GK D9 B EH9 w
Yeead4)) cT(RaRTYV( cX)U Td)daVT Z_ec (Ye

n w
❏
• w
• w w
❏ w
❏ w p+ w +

VcR BE6 9VYR CN_ bRa v
❏ ×w
”
❏
CREATE TABLE t (
col1 STRING,
…
) STORED AS [ORC/PARQUET];
INSERT INTO t (…);
SELECT col1 FROM t;

VcR 4ZNg V R V n
❏ 7 Rk _ AZ_VdZd IecVR w p>ZgV ×w
• 7 Rk _ AZ_VdZd p w w
• 4 >ZgV p w
❏ IecVR IYRcU CRaaVc w
• 7 Rk _ AZ_VdZd w
• 7 Rk _ :j_R :8 $ p ”
w
IYRcU
IYRcU +
CRaaVc
CRaaVc
SELECT
FROM
123
id data
1 …
2 …
3 …
Amazon
Kinesis

VcR 4ZNg V R V n
VcR n
7 Rk _ AZ_VdZd
IecVR
D #
>GB ” q
7 Rk _ AZ_VdZd
w

n 2 F N_
❏
w w
• p
• $
❏ w w p
w
• IGB& w & &
http://spark.apache.org/

F N_ FD 7NaN Ra 4C F N_ & #
❏ IaRc IGB
• >ZgVGB p>ZgV CVeRde cV
• :89)E:89
• IGB p
:ReRdVe):ReR cR V
❏ :ReRdVe 7F
• w
• ITR R) RgR
• :ReR cR V
:ReRdVe
Yeead4))UReRScZT d(T )S X), +0) 1)+.)R eR V
W eYcVV RaRTYV daRc RaZd cUUd UReRWcR Vd R_U
UReRdVed(Ye
Yeea4))daRc (RaRTYV( cX)U Td) ReVde)db ac XcR Z_X XfZUV(Ye

❏ :ZdTcVeZkVU IecVR $ p
❏ HVdZ ZV_e :ZdecZSfeVU :ReRdVe $ w
• p
❏ pIaRc ,(, CH HV VRdV /(2( $ pIecfTefcVU
IecVR Z_X 7F =7
DStream
RDD@T1 RDD@T2
Messages
Receiver
F N_ Fa_RNZV T i
Yeea4))daRc (RaRTYV( cX)U Td) ReVde)decVR Z_X ac XcR Z_X XfZUV(Ye
Yeead4))daRc (RaRTYV( cX)U Td) ReVde)decfTefcVU decVR Z_X ac XcR Z_X XfZUV(Ye

4ZNg V R V F N_ Fa_RNZV T
7 Rk _ AZ_VdZd IecVR p7 Rk _ CH IaRc
:IecVR
• IaRc 7 Rk _ AZ_VdZd
• pAZ_VdZd 9 ZV_e BZScRcjzA9B
x p
• w p
http://spark.apache.org/docs/latest/streaming-kinesis-integration.html
IYRcU
IYRcU +
A9B eYcVRU
A9B eYcVRUAmazon
Kinesis
DStream
RDD@T1 RDD@T2

F N_ 7f NZVP ER b_PR 4YY PNaV
❏ iVTfe c H::
• >VRa p
❏ :j_R ZT HVd fcTV 7 TReZ _ 5 iVTfe c
• IecVR Z_X 7aa ZTReZ _ w
• daRc dYV w
• .(.( pdaRc (Uj_R ZT7 TReZ _(V_RS VU
a_bR
• daRc (dYfWW V(dVcgZTV(V_RS VU a_bR
https://spark.apache.org/docs/latest/job-scheduling.html

❏ w
)
n
❏
w
4ZNg 8@E
w p + “ u

p q
• 7 Rk _ CH CR_RXV V_e =fZUV
Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)CR_RXV V_e=fZUV
• 7 Rk _ CH HV VRdV =fZUV
Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)HV VRdV=fZUV
• 7 Rk _ CH 7F HVWVcV_TV
Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde)7F )
• 7 Rk _ CH :VgV aVc =fZUV ,(i R_U -(i $
Yeea4))U Td(Rhd(R Rk _(T ) RdeZTCRaHVUfTV) ReVde):VgV aVc=fZUV)
• 7MI 9B V c
Yeea4))U Td(Rhd(R Rk _(T )T Z) ReVde)cVWVcV_TV)V c)Z_UVi(Ye

D 4

n
7MI w
• Yeea4))Rhd(R Rk _(T ) a)Rhd a Z_ec UfTeZ _)
7MI I feZ _d 7cTYZeVTe
• p w G 7
• Yeea4))Rhd(ejaVaRU(T )dR a)

GdVaaR_ 9NPRO
4JF x
3Nd PY bQMW
p p p
w v u
–
Uaa 2 &SO&ZR )cE0fJZ

4JF jx v
7MI p p
”
Yeead4))Rhd(R Rk _(T ) a)T _eRTe fd)Rhd dR Vd)
7MI ”

4 R QVe

4@ E YR z
, +/ 0 - p 7C H V
7C H V , p
• CH c V
• 7 Rk _ CH p7 Rk _ 9,
• 9, Z_deR_TV ac WZ V
• 7 Rk _ 9, p7 Rk _ I-
Yeea4))Rhd(ejaVaRU(T )dR a), +/) 0)V c R_URe cj ZR faUReV(Ye

ERYRN R ,&(
w w
• >RU a ,(0(
• >ZgV +( (
• FZX (+.(
• CRY fe (+ (
• IaRc +(.(+
7aRTYV 8ZXe a w w ×
• p

ERYRN R ,&)
w
• IaRc +(/(
• >fV -(1(+ $
p w
• FcVde (++3
• PVaaV Z_ (0 I_RadY e$
• E kZV .( (+

ERYRN R ,&
w
• IaRc +(/(,
• =R_X ZR -(0( $
• FcVde (+,/
• PVaaV Z_ (/(/
• E kZV .(,(

ERYRN R ,&
w
• >RU a ,(1(+
• IaRc +(0(
• =R_X ZR -(1(,
• FcVde (+-

ERYRN R ,&,
w
• >9ReR X +( ( $
• CRY fe (++(+
• Ib a +(.(0 $
• FcVde (+-0
• PVaaV Z_ (/(0
IaRc
• :j_R ZT R TReZ _
RgR 2
• 7L7Q>EC ”

ERYRN R ,&-
w
• IaRc +(0(+
• >RU a ,(1(,
• FcVde (+.
CH I
• I- II ACI
• 7MI ACI 9CA

ERYRN R ,&.
w
• >8RdV +(,( $ m D IGB _ >RU a
• P VVaVc -(.(2 $
• FcVde (+.-

ERYRN R ,&/&) t zu ,&/&(–
w
• FY V_Zi .(1( $ m IGB _ >8RdV
• JVk (2(- $ m :7= V_XZ_V _ O7HD
• >8RdV +(,(+
• CRYRfe (+,(
• FcVde (+.1

ERYRN R ,&/&
w
• CRY fe (+,(,
• IaRc +(0(,
• FcVde (+.2

ERYRN R ,&0&(
w
• >8RdV +(,(,
• JVk (2(.
• PVaaV Z_ (0(+
• FcVde (+/+

ERYRN R ,&0&
w
• >RU a ,(1(-
• FcVde (+/,(-

ERYRN R ,&0&
w
• IaRc +(0(-
• FcVde (+/1(+
• P VVaVc -(.(3
• CH :j_R :8
9 __VTe c
– .(,(
– >ZgV& IaRc

ERYRN R ,&1&)
w
•
FjeY _ ,(0 w
• w
>ZgV
• > L 3310
• > L + + 0

ERYRN R -&(&(
w
• >ZgV ,(+(
• FcVde (+/
• IaRc ,( (
• FPNYN &))
• >fV -(+ (
• FZX (+0(
• JVk (2(.
• >8RdV +(,(,
• PVaaV Z_ (0(+
• NcN 0
• JVk >ZgV& FZX$
• IeVa
IR_US i
• w
• IR_US i

85F z
j @, 6, n
CH 7F 8I
• >: I
• p
• 9 fU c ReZ _
8I I9+)IJ+ w
• >: I
C. 9.
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0) ,)R Rk _ V c _ h dfaa ced R Rk _ VSd g f Vd R_U . R_U
T. R Rk _ VT, WR Z ZVd)

4 NPUR VcR 4 NPUR F N_ 8@E
7f NZ 75 n BFF
:j_R :8
:j_R :8 ×w p CH w
E D
=Ze>fS
• Yeead4))XZeYfS(T )Rhd RSd)V c Uj_R US T __VTe c
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +0) 3)eYV R Rk _ V c Uj_R US T __VTe c W c RaRTYV YZgV R_U
RaRTYV daRc Zd _ h aV_ d fcTV)

ERYRN R -&(&
w
• >RU a ,(1(-
• FcVde (+/,(-
• IaRc ,( (+

ERYRN R -&)&(
w
• Z_ +(+(- $
• >8RdV +(,(-
• PVaaV Z_ (0(,

ERYRN R -& &(
w
• IaRc ,( (,
>8RdV _ I-
• >8RdV w
I-

ERYRN R -& &)
w
• FcVde (+/1(+
• P VVaVc -(.(3
• CH :j_R :8
9 __VTe c
– .(,(
– >ZgV& IaRc

ERYRN R -& &(
w
• >ZgV ,(+(+
• >fV -(++(
• IaRc ,(+(
• E kZV .(-(
• Z_ +(+(.

ERYRN R -& &)
w
•
PVaaV Z_
• P FF B D +2/0
• P FF B D +311

ERYRN R -&,&(
w
• FcVde (+00
• FY V_Zi .(3(
• >8RdV +(-(
• Z_ +(,(
• PVaaV Z_ (1(

E, n
Yeead4))Rhd(R Rk _(T )RS fe Rhd)hYRed _Vh), +1) -)R Rk _ V c cV VRdV / . R_U dfaa ce W c c. Z_deR_TVd _ h
RgRZ RS V)
H.
• × w w

ERYRN R -&-&(
w
• >fV -(+,
• FcVde (+1
• B:7F w
• PVaaV Z_ (1(+
• P AVVaVc -(.(+

ERYRN R -&.&(
w
• >8RdV +(-(+
• IaRc ,(+(+
• Z_ +(,(+
• CRY fe (+-(
• IaRc ,(i w
FcVde Z_ ecR_dZe

ERYRN R -&/&(
w
• Z_ +(-(
• FY V_Zi .(++(
• PVaaV Z_ (1(,
w p
7 Rk _ BZ_fi 7C
5N R
• w w
C_R a
• _ UV(ac aVceZVd
K4EA
• T _eRZ_Vc X. (ac aVceZVd
F
• IGEEF ,22 w

ERYRN R -&0&(
w
• 7MI I:A +(++(+0
• Z_ +(-(+
• >ZgV ,(-(
• IaRc ,(,(
• IecfTefcVU IecVR Z_X =7
7MI = fV :ReR 9ReR X
• >ZgV IaRc IGB 7MI
= fV :ReR 9ReR X >ZgV
w
B gVR
• EEP ,1.2 w
bR
• >K /2/3 w
5N R
• CN $ >8RdV w
w w
• w
RgR CR_RXV V_e ieV_dZ _d

ERYRN R -&1&(
w
• 7MI I:A W c RgR +(++(+2-
• Z_ +(-(,
• >fV .( (+
• >fV D eVS W c IaRc w
• FZX (+1(
• FcVde (+2.
• BZgj (.( Z_TfSReZ_X $
Z- dVcZVd 7 Rk _ 9, Z_deR_TVd
w ETe SVc +,& , +1$

ERYRN R -&1&(
F N_
• w p
p7fe ITR Z_X
XcRTVWf
• IF7HA ,+.3. w
LR RYV
• P FF B D ,-11 w
ON R
• >87I +2/--
bR
• >ZgV bfVcj VUZe c
7MI = fV :ReR 9ReR X
w
•

AWS Black Belt Online Seminar 2017 Amazon EMR

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to AWS Black Belt Online Seminar 2017 Amazon EMR

Similar to AWS Black Belt Online Seminar 2017 Amazon EMR (20)

More from Amazon Web Services Japan

More from Amazon Web Services Japan (20)

Recently uploaded

Recently uploaded (20)

AWS Black Belt Online Seminar 2017 Amazon EMR