Hadoop and EMR Options for Processing Large Data

•Download as KEY, PDF•

36 likes•5,248 views

Tatsuya Sasaki

2010/10/18のJJUG CCC 2010 Fallの講演で使用したスライドです

Technology Spiritual

•
•
• Hadoop (Cloudera)

• Elastic MapReduce
•
•

• (@sasata299)

• 2009 8 JOIN

•
• Hadoop

• Hadoop MySQL

•
- GROUP BY

• 7000 …

• (´Д` )

Hadoop
• Google MapReduce OSS

•
-
-
-
-

Hadoop
master ( )

slave ( )

<key,value>
Map
Shufﬂe & Sort

Hadoop
master ( )

slave ( )

<key,value>
Map Reduce
Shufﬂe & Sort

• Hadoop Streaming (Ruby )

• EC2 Cloudera Hadoop
- Cloudera CDH1
- Hadoop 0.18.3

• S3

MySQL → Hadoop
•
• GROUP BY MapReduce
- ( )
- key

• JOIN MapReduce
•

(1) master
master slave scp

(2) S3

S3 slave scp

MySQL vs Hadoop

7000

MySQL Hadoop
MySQL Hadoop

MySQL vs Hadoop

( Д )
7000
30

MySQL Hadoop
MySQL Hadoop

• Hadoop
-
• Hadoop (HADOOP-6254)
- S3
- SocketTimeoutException

• EMR (Elastic MapReduce)
- Amazon Hadoop

• Cloudera CDH2
-

AMI
(Amazon Machine
UP Image)

EMR

CDH2

EMR
BootStrap Action

Step (Hadoop Job)

Job Flow ( )

•
-
- --alive
• AMI
- AMI

- BootStrap Action

•
- mapred.child.java.opts
- streaming

•
-
- ElasticMapReduce-master 5100

Similar to Hadoop and EMR Options for Processing Large Data

COOKPADでのHadoop利用Tatsuya Sasaki

Hadoop導入事例 in クックパッドTatsuya Sasaki

Amebaサービスのログ解析基盤Toshihiro Suzuki

Hadoop入門とクラウド利用Naoki Yanai

Hadoop and mysql by Chris SchneiderDmitry Makarchuk

Hadoop londonYahoo Developer Network

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)npinto

HadoopThe Hadoop Java Software FrameworkThoughtWorks

Qubole Overview at the Fifth Elephant ConferenceJoydeep Sen Sarma

Hadoop Overview & Architecture EMC

Hadoop Conference Japan 2011 FallRyu Kobayashi

Hadoop Overview kdd2011Milind Bhandarkar

データ解析技術入門(Hadoop編)Takumi Asai

サンプルから見るMap reduceコードShinpei Ohtani

サンプルから見るMapReduceコードShinpei Ohtani

HadoopRajesh Piryani

Brust hadoopecosystemAndrew Brust

Rug hogan-10-03-2012designandanalytics

Hadoop and MapReduceHemanth Kumar Mantri

Azure Data WarehouseNikolay Stanev

Similar to Hadoop and EMR Options for Processing Large Data (20)

COOKPADでのHadoop利用

Hadoop導入事例 in クックパッド

Amebaサービスのログ解析基盤

Hadoop入門とクラウド利用

Hadoop and mysql by Chris Schneider

Hadoop london

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

HadoopThe Hadoop Java Software Framework

Qubole Overview at the Fifth Elephant Conference

Hadoop Overview & Architecture

Hadoop Conference Japan 2011 Fall

Hadoop Overview kdd2011

データ解析技術入門(Hadoop編)

サンプルから見るMap reduceコード

サンプルから見るMapReduceコード

Hadoop

Brust hadoopecosystem

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

Recently uploaded

Pigging Solutions in Pet Food ManufacturingPigging Solutions

APIForce Zurich 5 April Automation LPDGMarianaLemus7

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

DMCC Future of Trade Web3 - Special EditionDubai Multi Commodity Centre

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh

#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada

Understanding the Laravel MVC ArchitecturePixlogix Infotech

Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024BookNet Canada

Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

Recently uploaded (20)

Pigging Solutions in Pet Food Manufacturing

APIForce Zurich 5 April Automation LPDG

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...

Benefits Of Flutter Compared To Other Frameworks

08448380779 Call Girls In Friends Colony Women Seeking Men

DMCC Future of Trade Web3 - Special Edition

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Unleash Your Potential - Namagunga Girls Coding Club

Scanning the Internet for External Cloud Exposures via SSL Certs

Enhancing Worker Digital Experience: A Hands-on Workshop for Partners

FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi

#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024

Understanding the Laravel MVC Architecture

Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024

Maximizing Board Effectiveness 2024 Webinar.pptx

Presentation on how to chat with PDF using ChatGPT code interpreter

Breaking the Kubernetes Kill Chain: Host Path Mount

"Federated learning: out of reach no matter how close",Oleksandr Lapshyn

My Hashitalk Indonesia April 2024 Presentation

Hadoop and EMR Options for Processing Large Data

1. 961

2. • • • Hadoop (Cloudera) • Elastic MapReduce • •

3. • • • Hadoop (Cloudera) • Elastic MapReduce • •

4. • (@sasata299) • 2009 8 JOIN • • Hadoop

5. • • • Hadoop (Cloudera) • Elastic MapReduce • •

7. • 961 • 30 3 1 • - -

8. • • ( , , ...) - - ( , , , …) -

10.

11.

12.

13.

14. • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )

15. • MySQL - • - -

16.

17. • • • Hadoop (Cloudera) • Elastic MapReduce • •

18. Hadoop • Google MapReduce OSS • - - - -

19. Hadoop master ( ) slave ( )

20. Hadoop master ( ) slave ( ) Map

21. Hadoop master ( ) slave ( ) <key,value> Map Shufﬂe & Sort

22. Hadoop master ( ) slave ( ) <key,value> Map Reduce Shufﬂe & Sort

23. • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3

24. MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •

25. (1) master (2) S3

26. (1) master (2) S3

27. (1) master master slave scp (2) S3

28. (1) master master slave scp (2) S3 S3 slave scp

29. MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop

30. MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop

31. Hadoop++ ←Hadoop ↓MySQL

32. • • • Hadoop (Cloudera) • Elastic MapReduce • •

33. • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException

34. • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -

35. AMI (Amazon Machine UP Image) EMR CDH2

36. AMI (Amazon Machine UP Image) EMR CDH2

37. EMR Job Flow ( )

38. EMR BootStrap Action Job Flow ( )

39. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

40. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

41. • - - --alive • AMI - AMI - BootStrap Action

42.

43. Created job ﬂow j-8IXS98OW1WEE ID

44. Hadoop

45.

46. • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100

47. • • • Hadoop (Cloudera) • Elastic MapReduce • •

48. • Map - • Reduce - key Reduce -

49. UU Map Reduce

50. UU Map ID Reduce

51. UU Map Reduce ID

52. UU Map Reduce ID

53. Map Reduce

54. Map ID key Reduce

55. Map Reduce key Reduce

56. Map 100 100 Reduce key Reduce

57. × Map 100 × 100 Reduce key Reduce

58. × Map 100 ×100 Reduce Reduce key sort

59. × Map 100 ×100 Reduce Reduce key sort

60. Hadoop • - Hadoop

61. Hadoop • - Hadoop

62. Hadoop • - Hadoop

63. Hadoop • - Hadoop

64. Hadoop • - Hadoop

65. • • • Hadoop (Cloudera) • Elastic MapReduce • •

66. • Hadoop - - - Reduce

Hadoop and EMR Options for Processing Large Data

Recommended

Recommended

More Related Content

Similar to Hadoop and EMR Options for Processing Large Data

Similar to Hadoop and EMR Options for Processing Large Data (20)

More from Tatsuya Sasaki

More from Tatsuya Sasaki (7)

Recently uploaded

Recently uploaded (20)

Hadoop and EMR Options for Processing Large Data

Editor's Notes