マーケティングのためのHadoop利用

•Download as KEY, PDF•

20 likes•2,133 views

This document discusses Hadoop and its use on Amazon Web Services. It describes how Hadoop can be used to process large amounts of data in parallel across clusters of computers. Specifically, it outlines how to run Hadoop jobs on an Elastic Compute Cloud (EC2) cluster configured with Hadoop and store data in Amazon Simple Storage Service (S3). The document also provides examples of using Hadoop Streaming to run MapReduce jobs written in Ruby on an EC2 Hadoop cluster.

Technology Health & Medicine

• sasata299 ( )
• Hadoop
• NoSQL
•

• http://blog.livedoor.jp/sasata299/

Hadoop

• Google MapReduce OSS
•

•

•

•

GROUP BY !!

MySQL
( 3.5 )

7000
≒292 ……orz

Hadoop
• Hadoop Streaming
• Ruby
• Cloudera CDH1 (0.18.3)
• EC2 Hadoop
• 10-50
• Hadoop S3

hadoop-ec2 push [cluster] mapper.rb
hadoop-ec2 push [cluster] reducer.rb

Hadoop
(EC2)

Hadoop

hadoop-ec2 exec [cluster] [command]

Hadoop S3
(EC2)

target_ids # [21310,12902,15321,..]

ARGF.each do |log|
log.chomp!
id, foo, bar, ... = log.split(/,/)
next if target_ids.include?(id)
end

target_ids 5
…

$# 1000 hash = Hash.new {|h,k| h[k] = []} target_ids.each do |_id| hash[_id.to_s[0,3]] << _id end ARGF.each do |log| log.chomp! id, foo, bar, ... = log.split(/,/) # next if hash[id[0,3]].include?(id) end$

EC2
-> AZ
->

JobTracker
-> 50030
-> hadoop job -list

What's hot

Time to Science, Time to Results. Accelerating Scientific research in the CloudAmazon Web Services

Hadoopを業務で使ってみたTatsuya Sasaki

Upgrading from HDP 2.1 to HDP 2.2SATOSHI TAGOMORI

Elasticsearch x Autoscaling (AWS)Kazuki Matsuda

Elasticwulf Pycon TalkPeter Skomoroch

[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...PROIDEA

BigFoot: Big Data For Every OrganizationMatteo Dell'Amico

Elastic search integration with hadoop leveragebigdataPooja Gupta

Terraform, Ansible, or pure CloudFormation?geekQ

Big Data in the Microsoft PlatformJesus Rodriguez

Terraform, Ansible or pure CloudFormationgeekQ

Enable IPv6 on Route53 AWS ELB, docker and node AppFyllo

M3D - Metadata Driven DevelopmentHenning Kropp

Streaming API, Spark and RubyManohar Amrutkar

Indexing Present1Nat Weerawan

Beyond EC2 and S3Lorenzo Aiello

Deep Dive on Deep Learning with Apache MXNet - AWS Summit Tel Aviv 2017Amazon Web Services

How to reduce hosting costs for Redis based applications on JavaNikita Koksharov

AWS September Webinar Series - Running Microservices with Amazon EC2 Contain...Amazon Web Services

サーバーレスの現実と夢と今Hiroyuki Hara

What's hot (20)

Time to Science, Time to Results. Accelerating Scientific research in the Cloud

Hadoopを業務で使ってみた

Upgrading from HDP 2.1 to HDP 2.2

Elasticsearch x Autoscaling (AWS)

Elasticwulf Pycon Talk

[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...

BigFoot: Big Data For Every Organization

Elastic search integration with hadoop leveragebigdata

Terraform, Ansible, or pure CloudFormation?

Big Data in the Microsoft Platform

Terraform, Ansible or pure CloudFormation

Enable IPv6 on Route53 AWS ELB, docker and node App

M3D - Metadata Driven Development

Streaming API, Spark and Ruby

Indexing Present1

Beyond EC2 and S3

Deep Dive on Deep Learning with Apache MXNet - AWS Summit Tel Aviv 2017

How to reduce hosting costs for Redis based applications on Java

AWS September Webinar Series - Running Microservices with Amazon EC2 Contain...

サーバーレスの現実と夢と今

Similar to マーケティングのためのHadoop利用

Hadoop導入事例 in クックパッドTatsuya Sasaki

Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Bu...HostedbyConfluent

Amazon Aurora로 안전하게 migration 하기Jesang Yoon

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)npinto

Amebaサービスのログ解析基盤Toshihiro Suzuki

From oracle to hadoop with Sqoop and other toolsGuy Harrison

Hadoop PrimerSteve Staso

Hadoop Conference Japan 2011 Fallに行ってきましたmoai kids

Hadoop and mysql by Chris SchneiderDmitry Makarchuk

Hadoop londonYahoo Developer Network

Introduction to Apache HadoopSteve Watt

Brust hadoopecosystemAndrew Brust

(BDT208) A Technical Introduction to Amazon Elastic MapReduceAmazon Web Services

OCF.tw's talk about "Introduction to spark"Giivee The

Hadoop Overview & Architecture EMC

Big data, just an introduction to Hadoop and Scripting LanguagesCorley S.r.l.

EclipseCon Keynote: Apache Hadoop - An IntroductionCloudera, Inc.

Getting started with Hadoop, Hive, and Elastic MapReduceobdit

Internship final report@Treasure Data Inc.Ryuichi ITO

Elephant in the cloudrhatr

Similar to マーケティングのためのHadoop利用 (20)

Hadoop導入事例 in クックパッド

Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Bu...

Amazon Aurora로 안전하게 migration 하기

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

Amebaサービスのログ解析基盤

From oracle to hadoop with Sqoop and other tools

Hadoop Primer

Hadoop Conference Japan 2011 Fallに行ってきました

Hadoop and mysql by Chris Schneider

Hadoop london

Introduction to Apache Hadoop

Brust hadoopecosystem

(BDT208) A Technical Introduction to Amazon Elastic MapReduce

OCF.tw's talk about "Introduction to spark"

Hadoop Overview & Architecture

Big data, just an introduction to Hadoop and Scripting Languages

EclipseCon Keynote: Apache Hadoop - An Introduction

Getting started with Hadoop, Hive, and Elastic MapReduce

Internship final report@Treasure Data Inc.

Elephant in the cloud

Recently uploaded

Artificial intelligence in the post-deep learning eraDeakin University

Unlocking the Potential of the Cloud for IBM Power SystemsPrecisely

My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer

Key Features Of Token Development (1).pptxLBM Solutions

Vulnerability_Management_GRC_by Sohang Sengupta.pptxnull - The Open Security Community

Pigging Solutions in Pet Food ManufacturingPigging Solutions

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

Benefits Of Flutter Compared To Other FrameworksSoftradix Technologies

Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK

My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 3652toLead Limited

Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson

costume and set research powerpoint presentationphoebematthew05

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays

Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community

Build your next Gen AI Breakthrough - April 2024Neo4j

SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren

Recently uploaded (20)

Artificial intelligence in the post-deep learning era

Unlocking the Potential of the Cloud for IBM Power Systems

My INSURER PTE LTD - Insurtech Innovation Award 2024

Key Features Of Token Development (1).pptx

Vulnerability_Management_GRC_by Sohang Sengupta.pptx

Pigging Solutions in Pet Food Manufacturing

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx

Benefits Of Flutter Compared To Other Frameworks

Unblocking The Main Thread Solving ANRs and Frozen Frames

My Hashitalk Indonesia April 2024 Presentation

Human Factors of XR: Using Human Factors to Design XR Systems

Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics

Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365

Are Multi-Cloud and Serverless Good or Bad?

costume and set research powerpoint presentation

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...

Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation

Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx

Build your next Gen AI Breakthrough - April 2024

SQL Database Design For Developers at php[tek] 2024

マーケティングのためのHadoop利用

1. Hadoop

2. • sasata299 ( ) • Hadoop • NoSQL • • http://blog.livedoor.jp/sasata299/

3. Hadoop • Google MapReduce OSS • • • •

4. • Hadoop • Hadoop • • & !?

5. • Hadoop • Hadoop • • & !?

7. 896 30 3 1

8. 896 30 3 1

9. 896 30 3 1 ” ”

10.

11.

12.

13. GROUP BY !! MySQL ( 3.5 )

14. GROUP BY !! MySQL ( 3.5 ) 7000 ≒292 ……orz

15. MySQL

16. … ※

17. !?

18.

19. Hadoop

20. …

21.

22. 7000 30

23.

24. • Hadoop • Hadoop • • & !?

25. Hadoop • Hadoop Streaming • Ruby • Cloudera CDH1 (0.18.3) • EC2 Hadoop • 10-50 • Hadoop S3

26. Hadoop • Hadoop Streaming • Ruby • Cloudera CDH1 (0.18.3) • EC2 Hadoop • 10-50 • Hadoop S3

27.

28.

29.

30.

31.

32. ○○ ×× ○○ ××

33. Hadoop

34.

35.

36. Hadoop AWS

37.

38. Hadoop

39. Hadoop Hadoop (EC2)

40. hadoop-ec2 push [cluster] mapper.rb hadoop-ec2 push [cluster] reducer.rb Hadoop (EC2)

41. hadoop-ec2 push [cluster] mapper.rb hadoop-ec2 push [cluster] reducer.rb Hadoop (EC2)

42. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

43. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

44. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

45. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

46. S3

47. S3

48. Hadoop Hadoop (EC2)

49. Hadoop

50. !!

51. Hadoop

52. 1) 2) Hadoop 3)

53. • Hadoop • Hadoop • • & !?

54. target_ids # [21310,12902,15321,..] ARGF.each do |log| log.chomp! id, foo, bar, ... = log.split(/,/) next if target_ids.include?(id) end target_ids 5 …

55. :-)

56. # 1000 hash = Hash.new {|h,k| h[k] = []} target_ids.each do |_id| hash[_id.to_s[0,3]] << _id end ARGF.each do |log| log.chomp! id, foo, bar, ... = log.split(/,/) # next if hash[id[0,3]].include?(id) end

57.

58. Mapper Reducer - Mapper - Reducer

59. • Hadoop • Hadoop • • & !?

60. EC2 -> AZ -> JobTracker -> 50030 -> hadoop job -list

61. 10h 8h JobTracker

62. !?

63. Amazon Elastic MapReduce

64. Elastic MapReduce