Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hadoop eco story 이해

766 views

Published on

다양한 Hadoop Eco 기술에 대한 이해

Published in: Software
  • Be the first to comment

Hadoop eco story 이해

  1. 1. BVYdde ?Xd Midgn gZZehl .,-2* ,/ -
  2. 2. -* BVYdde BVYdde : ) Cc -553( >dj =jii c ( FjXZcZ ~ ( c .,,,( deZc hdjgXZ ) Cc .,,-( eVX Z HjiX ~ ( QZW =gVlaZg ) - XdgZ( -A< L G( L C> aZkZa - dc 4 YY idiVa -N< ) CcYZm c gViZh 8 -,, eV Zh + hZX ) a b iZY i Z idiVa cjbWZg d eV Zh id -,, b aa dc ) QZW cYZm c w ) 0 w x ) y Y h ­y CcYZm c * N Z hidgn d BVYdde .
  3. 3. B> M : eVX Z HjiX 0~ hX ZbVaZhh( YjgVWaZ( VcYa c V ajgZ( VjidbVi Vaan gZWVaVcXZY Cc .,,/( Add aZ ejWa h ZY i Z Add aZ aZ MnhiZb eVeZg Cc .,,0( YZkZadeZY HjiX > hig WjiZY aZ MnhiZb ) >jgVWaZ 6 20G WadX ( ) BVcYa c V ajgZ 6 z HdYZ ) jidbVi XVaan gZWVaVcXZ 6 HVbZ HdYZ /
  4. 4. B> M : 0iie6++lll*ha YZh VgZ*cZi+bVYk gjh+ Y h).443-012
  5. 5. B> M : 1
  6. 6. GVeLZYjXZ : H> M :( ) x ( ) xx ( ) * ! Cc .,,0( uGVeLZYjXZ6 M bea ZY >ViV JgdXZhh c dc FVg Z =ajhiZghv 2 GVe HdYZ LZYjXZ GVe
  7. 7. GVeLZYjXZ : 3 -* ldgYXdjci GVeLZYjXZ * .* ( Mea i( ( ( ( … /* /~ Mea i * 0* bVe … 7 (-9 * 1* … Zn * 2* gZYjXZ … kVajZ hjb cVa kVajZ * 3* <ZVg(. =Vg(/ >ZZg(. L kZg(. cVa 7 Zn(kVajZ9 w
  8. 8. 4 RackN NodeManager NodeManager NodeManager Rack2 NodeManager NodeManager NodeManager Rack1 NodeManager NodeManager NodeManager C2.1 C1.4 AM2 C2.2 C2.3 AM1 C1.3 C1.2 C1.1 Hadoop Client 1 Hadoop Client 2 create app2 submit app1 submit app2 create app1 ASM Scheduler queues ASM Containers NM ASM Scheduler Resources .......negotiates....... .......reports to....... .......partitions....... ResourceManager status report GVeLZYjXZ :
  9. 9. BVYdde ! .,,2 BVYdde ! ) B> M GVeLZYjXZ BVYdde =dgZ a W ) .,,2 S BII ­ .,,3 Nl iiZg( VXZWdd ( F c ZYCc .,,4 MjW Jgd_ZXiw ) BWVhZ( Tdd ZZeZg( J SV dd ( B kZ VXZWdd q ) =adjYZgV Add aZ( VXZWdd ( SV dd ~ .,,5 bVodcZ GVeLZYjXZ .,-- Bdgidcldg h SV dd ~ 5
  10. 10. BVYdde** q GVeLZYjXZw !!!! ) B> M ) =ajhiZg ( ) DdW ( egdXZhh c ) =a Zci JC q! <ViX : ) KjZgn( AgVe JgdXZhh c ** q: -,
  11. 11. BVYdde** q w ! w : ) ** ) ** ** B> M GVeLZYjXZ : .,-. GVeLZYjXZ P. S LH --
  12. 12. BVYdde .*, : =ajhiZg … GVeLZYjXZ w BVYdde S LH -.
  13. 13. BVYdde MQw : -/ N Z BVYdde ?XdhnhiZb NVWaZ .,-2 -2, ~ MQ iie6++ VYddeZXdhnhiZbiVWaZ* i jW* d+
  14. 14. BVYdde ?Xd r -0 B> M S LH y HdMKF MKF dc BVYdde x BVYdde)gZaViZY egd_ZXih Vi eVX Z HZlMKF x
  15. 15. BVYdde ?Xd ( MKF dc BVYdde -1 w + !! : B> M S LH GVeLZYjXZ MKF dc BVYdde J ( B kZ( Mi c Zg NZo ( NV_d( CbeVaV( JgZhid( BVl Add aZ >gZbZa .,-- >ViV
  16. 16. BVYdde ?Xd ( MKF dc BVYdde -2 • J .,,4 6 MXgei GVeLZYjXZ • BkZ .,,4 6 MKF GVeLZYjXZ • >g aa .,-- 6 B> M B<VhZ(Gdc d><(M/ GVeL • CbeVaV .,-. 6 y ( ( =adjYZgV • JgZhid .,-/ 6 _dcw VXZWdd • B kZ .*, .,-2 6 B kZ NZo =C> ( ~ • NV_d .,-0 6 >Q dc BVYdde( MKF • MeVg MKF .,-0 6 y • B QK .,-/ 6 GJJ ( sMKF t MKF)Ic)BVYdde : MKF -,w
  17. 17. : y q -3 B> M GVeLZYjXZ J MXg ei B kZ MKF B kZ .*, B kZ iZo >g aa( JgZhid CbeVaV y NZo MeVg kh NZo NV_d BVlf ~ MeVg MKF
  18. 18. q -4 y … y NV_d( B kZ w: MKF >g aa( CbeVaV( JgZhid( B QK( B kZ dc NZo ! MeVg L>> MeVg MKF L>>( B kZ y
  19. 19. q -5 MnhiZb HVbZ Vjai NdaZgVcXZ >ncVb X MX ZYja c Fd N bZ KjZgn Fdl FViZcXn eVX Z NV_d I I I I CbeVaV R R R I Mig c Zg B kZ I I I I JgZhid R R R I MKF)dc)BVYdde :
  20. 20. MKF dc BVYdde ., MKF)Ic)BVYdde y * 78 B> M & WVcYl Yi -, ( 4~( M N 3.,, geb -, & 4~ & -,,G<+hZX 8 4A<+hZX -*1N< B kZ( NV_d( CbeVaV * hZaZXi Xdjci & gdb NUN?MN VXidg " "( " y"( " " * w w w w VXidg * B kZ - NV_d( CbeVaV -*1 p / * -,, ( .,, w * Q Vi YdZh u-,, i bZh VhiZg i Vc B kZv VXijVaan bZVc:
  21. 21. MKF)Ic)BVYdde VXidg .- • BVYdde 6 jaa hXVc • y 6 y gdje Wn( dgYZg Wn • … 6 dgYZg Wn kZ gZYjXZ w * • 6 L= aZ ( IL= aZ( eVgfjZi • 6 IL= * • 6 ( • y 6
  22. 22. BVYdde : ..*** ” IL= JVgfjZi Bdgidcldg h ( DVkV ( L= aZ =adjYZgV ( ( ** ( McVeen ? X Zci >ViV MidgV Z dg cVani Xh l i eVX Z JVgfjZi .*,
  23. 23. K “ ** ./ K JVgfjZi WadX h oZw Y h WadX h oZ w : ?LL 6 JVgfjZi aZ h djaY cdi WZ hea i cid bjai eaZ Y h)WadX h è B kZ eVgfjZi ( beVaV è JVgfjZi Y h WadX h oZ iie6++hiVX dkZg adl*Xdb+fjZhi dch+/,404331+hZi)eVgfjZi)hcVeen)djieji) aZ)h oZ) h) kZ
  24. 24. BVYdde ?Xd ( HdMKF : .0 HdMKF y ~ HdMKF B> M S LH GVeLZYjXZ MKF dc BVYdde BVYdde ><GM q Add aZ < NVWaZ .,-. B> M >< : BWVhZ( =VhVcYgV( Gdc d><
  25. 25. HdMKF w : .1Gdc d><( =VhhVcYgV( B<VhZ y • =VhcVYgV .,,4 6 VXZWdd • B<VhZ .,,2 6 >< • XXjbjad .,-- 6 B<VhZ ( ” • LZYh 6 >< • ?aVhiXMZVgX .,-, 6 FjXZcZ • Gdc d>< 6 -, Zc( w ( Ddc w • >ncVbd>< 6 QM ( ( MM> y( ) =VX Z LZY h >< ) ZaVhi XhZVgX
  26. 26. HIMKF!! = J : .2HdMKF y ! =dch hiZcXn 6 xx w 8 w y * kV aVW a in 6 w 8 w JVgi i dc NdaZgVcXZ 6 8 HQ y HdMKF J w = = L><GM =J w ( B<VhZ J >ncVbd( =VhVcYgV( =djX >< = J = J NZdgZb HdMKF
  27. 27. HIMKF : B> M w ) ­y w ) w HdMKF y ~ .3
  28. 28. HdMKF BWVhZ .4 iie6++lll*ijidg Vahed ci*Xdb+ WVhZ+ WVhZUijidg Va*eY BWVhZ B> M HdMKF ><
  29. 29. HdMKF BWVhZ .5
  30. 30. HdMKF /, HdMKF w è u v q : >< w ( F hi HdMKF y ~ KjZgn
  31. 31. HdMKF : /- … -( .* 6 8 - 6 - /* 6 8 - 6 H 0* y L><GM … ! HdMKF
  32. 32. HdMKF : /. EZn)PVajZ * PVajZ 0w * >< PVajZw Mig c ( F hi( GVe LZY h( >ncVbd><( L V EZn)PVajZ PVajZ ~ =dajbc BWVhZ( =VhVcYgV( >gj Y( XXjbjad =dajbc Vb an PVajZ mba( Dhdc Gdc d><( =djX ><( ?aVhi XhZVgX >dXjbZci
  33. 33. HdMKF // iie6++c.b*Xd* g+c.b.,-,+YdlcadVY+=djX <VhZP-U0*eY • Dd c Dd c w ( HdMKF … >< _d c • HdMKF Dd c w w
  34. 34. HdMKF =dajbc /0
  35. 35. HdMKF >dXjbZci /1 =dajbc Vban >dXjbZci
  36. 36. K =dajbc >< : /2>< w: Xdajbc dg ZciZY YWbh =dgZ w * ) )
  37. 37. HdMKF MKF dc BVYdde** w : /3 HdMKF Hdi Ican MKF … ><GM >< < M? MKF HdMKF L><GM MKF dc BVYdde … = J HdMKF y ! (
  38. 38. HdMKF MKF dc BVYdde** ! /4 B kZ MKF FVnZg : B> M S LH >ViV IM B< M? B kZ MKF =a Zci CchZgi + OeYViZ >ZZe MKF cVanhh B<VhZ VcY B kZ r <ZiiZg Nd Zi Zg ü B< M? >ViV ü ü BkZ ü KjZgn
  39. 39. HdMKF w : HdMKF ee *! è ‘ >< ee w * ee ~ w * ~ P Zl P Zl à w HdMKF : J w =J /5
  40. 40. HIMKF!! = J : < M? : 0, <Vh XVaan kV aVWaZ 6 kV aVWaZ Md i)hiViZ 6 w >ViVw Zme gZ ?kZcijVaan Xdch hiZcXn 6 y >ViV = J < M? HdMKF y ! =dch hiZcXn 6 xx w 8 w y * kV aVW a in 6 w 8 w * JVgi i dc NdaZgVcXZ 6 8 *
  41. 41. HIMKF!! = J : < M? : 0- =C> * ( . … w * < M? ~ * =C>w ~ ( < M? * =C>( < M? = J = J /~ .~ * J ( * ‘( ‘ Jw w = * = : =dch hiZcXn ?kZcijVaan =dch hiZci ~ * HdMKF y ~ r ?kZcijVaan =dch hiZci
  42. 42. HZl MKF: 0.HZlMKF : B> M S LH GVeLZYjXZ MKF dc BVYdde HdMKF HZlMKF =C> Hdc)adX c h VgZY)cdi c HdMKF VcY L><GM … : Pdai><( Hjd><( bZbMKF( =ajhig m
  43. 43. HZl MKF : 0/ HdMKF L><GM MKF =C> < o HZlMKF ><GM ( HZlMKF y HZlMKF LZXdgY FdX c ad c <j Zg edda
  44. 44. HZl MKF : 00 - MKF . igVchVXi dch =C> WZ c( gdaaWVX ( Xdbb i / Hdc)adX c XdcXjggZcXn Xdcigda NgVchVXi dc FdX c Hdc)adX c w * & ( HZlMKF w 0 B eZg)cdYZ eZg dgbVcXZ x ><GM * 1 JVgVaaZa( h VgZY)cdi c VgX iZXijgZ ( w x *
  45. 45. HZlMKF 01 GZbdgn bnMKF ü Pdai>< AHO ( GZbMKF ü MXVaZ>< GnMKF hXVaVW a in ( ü Nd j>< AHO GVg V>< hXVaaVW a in ü =ajhig m> Y hig WjiZY eZZg)id)eZZg MKF YViVWVhZ ü Hjd>< Y hig WjiZY YViVWVhZ bVcV ZbZci s-0 2 HZlMKF =ajhig m( bZbMKF( Hjd><( Mea XZ GVX cZ( NgVch FVii XZ( Pdai>< -2 HdMKF( HZlMKF >ViVWVhZh Nd QViX
  46. 46. HZlMKF( HdMKF( L><GM : 02 ü L><GM dg cZlMKF ü HdMKF dg HZlMKF ü L><GM ü HdMKF ü < o( Cc gV
  47. 47. HZlMKF( HdMKF( L><GM : 03
  48. 48. HZlMKF : 04 FZVYZg’ IgVXaZ( G Xgdhd i( M J( C<G H X Z JaVnZg w … uIcZ h oZ YdZh cdi i Vaav
  49. 49. 05 B> M S LH GVeLZYjXZ MKF dc BVYdde HdMKF ( BVYdde ( q BVYdde : ajbZ( H ( ajZciY( ad hiVh HZlMKF
  50. 50. r w : 1, • MXgWZ .,,4 6 VXZWdd ( • = j lV .,,4 6 SV dd( • ajbZ .,-. 6 ( y ?NF • ajZciY 6 ajbZ eaj c ( ? E w • Fd hiVh 6 * * w • H .,-0 6 ‘ HM • MKIIJ 6 L><GM BVYdde • EV V 6 Fc ZYCc( GZhhV Z KjZjZ >h < o Fd ( ( ) y EV Vw * ? E 6 ?aVhi XhZVgX ajZciY E WVcV Fd gZ Vidg FIA L>< GMA KjZjZ
  51. 51. r : 1- w MdjgXZ )9 = VccZa )9 Mc ) MdjgXZ 6 ) = VccZa 6 ) Mc 6 Jaj c Jaj c ~ w ) 6 ><( JdgdidXda( MHM( B> M ) 6 ><( KjZjZ(
  52. 52. r : 1. : JgdidXda DMIH NZmi RGF < cVgn DMIH ­y w( hoZ w( NZmi JVghc YViV dgbVi N g i( kgd( JgdidWj ­ (
  53. 53. r : 1/ egdidWj w ( i g iw * PLI ) ) ( i g i ) .,,3( VXZWdd ) ( w * egdidWj ) .,,4( Add aZ ) w ( w N g i kh JgdidXda <j Zgh kh kgd ) < VhZY =dbeVg hdc
  54. 54. y + MigZVb c 10 B> M S LH GVeLZYjXZ MKF dc BVYdde HdMKF higZVb c : GZbgdn ?kZci GL … ) C+I HQ( > h ( MeVg ( Midgb( hoboV HZlMKF y
  55. 55. y r w : 11 • M0 .,-- 6 SV dd( higZVbc .,-/ • Midgb .,-/ 6 NliiZg( higZVbc ( • MVboV .,-0 6 Fc ZYCc( higZVbc ( >h • JjahVg .,-1 6 ZWVn( higZVbc =?J( >gjY • MeVg .,-0 6 >ViV<gX h( L>> + ( GF+MKF • a c .,-0 6 YViV gihVch( MeVg ( egdXZhh MeVg a W y MigZVb c < >ViV cZmi hiZe dg eVX Z a c a c
  56. 56. y w: 12 GL NZo MeVg a c >ViV >ViV MVboVMidgb JjahVg B> M( aZ HdMKF … MHM( =Vaa( > A BVYdde > A L>>
  57. 57. 13NZo B> M * GVe LZXYjXZ * GVeLZYjXZ kh NZo 6 ~ :
  58. 58. 14 6 GZbdgn ( > A <ViX MigZVb c w G Xgd <ViX GVeLZYjXZ kh MeVg 6 ~ :
  59. 59. 15 MigZVb c < >ViV6 Midgb( MeVg VcY MVboV 6 GZbdgn ( > A <ViX MigZVb c w G Xgd <ViX eVX Z Midgb
  60. 60. 2,
  61. 61. y y 2-
  62. 62. y 2.
  63. 63. y 2/
  64. 64. Qdg adl( Gdc idg c 20 BVYdde w OC …+ MQ !!!! : bWVg ( =adjYZgV GVcV Zg Ido Z( o VWVc( Fj BVYdde >ViV B> M S LH GVeLZYjXZ MKF dc BVYdde HdMKF HZlMKF y
  65. 65. Qdg adl 21 OC o VWVc F c ZYCc BVYdde o VWVc Ido Z RGF OC Fj Jni dc =dYZ Fj Jni dc ~ Qdg adl ?c cZ =dbeVg hdc
  66. 66. 22 BVYdde =ajhiZg =adjYZgV GVcV Zg =adjYZgV bWVg BdgidcQIg h aVb c d ‘ =adjY cZ < >ViV
  67. 67. 23 BVYdde =ajhiZg bWVg
  68. 68. 24 BVYdde =ajhiZg bWVg

×