マーケティングのためのHadoop利用

•Download as KEY, PDF•

20 likes•2,134 views

This document discusses Hadoop and its use on Amazon Web Services. It describes how Hadoop can be used to process large amounts of data in parallel across clusters of computers. Specifically, it outlines how to run Hadoop jobs on an Elastic Compute Cloud (EC2) cluster configured with Hadoop and store data in Amazon Simple Storage Service (S3). The document also provides examples of using Hadoop Streaming to run MapReduce jobs written in Ruby on an EC2 Hadoop cluster.

Technology Health & Medicine

• sasata299 ( )
• Hadoop
• NoSQL
•

• http://blog.livedoor.jp/sasata299/

Hadoop

• Google MapReduce OSS
•

•

•

•

GROUP BY !!

MySQL
( 3.5 )

7000
≒292 ……orz

Hadoop
• Hadoop Streaming
• Ruby
• Cloudera CDH1 (0.18.3)
• EC2 Hadoop
• 10-50
• Hadoop S3

hadoop-ec2 push [cluster] mapper.rb
hadoop-ec2 push [cluster] reducer.rb

Hadoop
(EC2)

Hadoop

hadoop-ec2 exec [cluster] [command]

Hadoop S3
(EC2)

target_ids # [21310,12902,15321,..]

ARGF.each do |log|
log.chomp!
id, foo, bar, ... = log.split(/,/)
next if target_ids.include?(id)
end

target_ids 5
…

$# 1000 hash = Hash.new {|h,k| h[k] = []} target_ids.each do |_id| hash[_id.to_s[0,3]] << _id end ARGF.each do |log| log.chomp! id, foo, bar, ... = log.split(/,/) # next if hash[id[0,3]].include?(id) end$

EC2
-> AZ
->

JobTracker
-> 50030
-> hadoop job -list

What's hot

Time to Science, Time to Results. Accelerating Scientific research in the CloudAmazon Web Services

Hadoopを業務で使ってみたTatsuya Sasaki

Upgrading from HDP 2.1 to HDP 2.2SATOSHI TAGOMORI

Elasticsearch x Autoscaling (AWS)Kazuki Matsuda

Elasticwulf Pycon TalkPeter Skomoroch

[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...PROIDEA

BigFoot: Big Data For Every OrganizationMatteo Dell'Amico

Elastic search integration with hadoop leveragebigdataPooja Gupta

Terraform, Ansible, or pure CloudFormation?geekQ

Big Data in the Microsoft PlatformJesus Rodriguez

Terraform, Ansible or pure CloudFormationgeekQ

Enable IPv6 on Route53 AWS ELB, docker and node AppFyllo

M3D - Metadata Driven DevelopmentHenning Kropp

Streaming API, Spark and RubyManohar Amrutkar

Indexing Present1Nat Weerawan

Beyond EC2 and S3Lorenzo Aiello

Deep Dive on Deep Learning with Apache MXNet - AWS Summit Tel Aviv 2017Amazon Web Services

How to reduce hosting costs for Redis based applications on JavaNikita Koksharov

AWS September Webinar Series - Running Microservices with Amazon EC2 Contain...Amazon Web Services

サーバーレスの現実と夢と今Hiroyuki Hara

What's hot (20)

Time to Science, Time to Results. Accelerating Scientific research in the Cloud

Hadoopを業務で使ってみた

Upgrading from HDP 2.1 to HDP 2.2

Elasticsearch x Autoscaling (AWS)

Elasticwulf Pycon Talk

[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...

BigFoot: Big Data For Every Organization

Elastic search integration with hadoop leveragebigdata

Terraform, Ansible, or pure CloudFormation?

Big Data in the Microsoft Platform

Terraform, Ansible or pure CloudFormation

Enable IPv6 on Route53 AWS ELB, docker and node App

M3D - Metadata Driven Development

Streaming API, Spark and Ruby

Indexing Present1

Beyond EC2 and S3

Deep Dive on Deep Learning with Apache MXNet - AWS Summit Tel Aviv 2017

How to reduce hosting costs for Redis based applications on Java

AWS September Webinar Series - Running Microservices with Amazon EC2 Contain...

サーバーレスの現実と夢と今

Similar to マーケティングのためのHadoop利用

Hadoop導入事例 in クックパッドTatsuya Sasaki

Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Bu...HostedbyConfluent

Amazon Aurora로 안전하게 migration 하기Jesang Yoon

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)npinto

Amebaサービスのログ解析基盤Toshihiro Suzuki

From oracle to hadoop with Sqoop and other toolsGuy Harrison

Hadoop PrimerSteve Staso

Hadoop Conference Japan 2011 Fallに行ってきましたmoai kids

Hadoop and mysql by Chris SchneiderDmitry Makarchuk

Hadoop londonYahoo Developer Network

Introduction to Apache HadoopSteve Watt

Brust hadoopecosystemAndrew Brust

(BDT208) A Technical Introduction to Amazon Elastic MapReduceAmazon Web Services

OCF.tw's talk about "Introduction to spark"Giivee The

Hadoop Overview & Architecture EMC

Big data, just an introduction to Hadoop and Scripting LanguagesCorley S.r.l.

EclipseCon Keynote: Apache Hadoop - An IntroductionCloudera, Inc.

Getting started with Hadoop, Hive, and Elastic MapReduceobdit

Internship final report@Treasure Data Inc.Ryuichi ITO

Elephant in the cloudrhatr

Similar to マーケティングのためのHadoop利用 (20)

Hadoop導入事例 in クックパッド

Don’t Forget About Your Past—Optimizing Apache Druid Performance With Neil Bu...

Amazon Aurora로 안전하게 migration 하기

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

Amebaサービスのログ解析基盤

From oracle to hadoop with Sqoop and other tools

Hadoop Primer

Hadoop Conference Japan 2011 Fallに行ってきました

Hadoop and mysql by Chris Schneider

Hadoop london

Introduction to Apache Hadoop

Brust hadoopecosystem

(BDT208) A Technical Introduction to Amazon Elastic MapReduce

OCF.tw's talk about "Introduction to spark"

Hadoop Overview & Architecture

Big data, just an introduction to Hadoop and Scripting Languages

EclipseCon Keynote: Apache Hadoop - An Introduction

Getting started with Hadoop, Hive, and Elastic MapReduce

Internship final report@Treasure Data Inc.

Elephant in the cloud

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 1DianaGray10

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptxDavid Michel

Speed Wins: From Kafka to APIs in Minutesconfluent

PLAI - Acceleration Program for Generative A.I. StartupsStefano

UiPath Test Automation using UiPath Test Suite series, part 2DianaGray10

Exploring UiPath Orchestrator API: updates and limits in 2024 🚀DianaGray10

Strategic AI Integration in Engineering TeamsUXDXConf

Demystifying gRPC in .Net by John StaveleyJohn Staveley

10 Differences between Sales Cloud and CPQ, Blanka DoktorováCzechDreamin

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya HalderCzechDreamin

Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10

Knowledge engineering: from people to machines and backElena Simperl

Agentic RAG What it is its types applications and implementation.pdfChristopherTHyatt

Optimizing NoSQL Performance Through ObservabilityScyllaDB

Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeCzechDreamin

Introduction to Open Source RAG and RAG EvaluationZilliz

To Graph or Not to Graph Knowledge Graph Architectures and LLMsPaul Groth

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra

ECS 2024 Teams Premium - Pretty SecureFemke de Vroome

A Business-Centric Approach to Design System StrategyUXDXConf

Recently uploaded (20)

UiPath Test Automation using UiPath Test Suite series, part 1

Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx

Speed Wins: From Kafka to APIs in Minutes

PLAI - Acceleration Program for Generative A.I. Startups

UiPath Test Automation using UiPath Test Suite series, part 2

Exploring UiPath Orchestrator API: updates and limits in 2024 🚀

Strategic AI Integration in Engineering Teams

Demystifying gRPC in .Net by John Staveley

10 Differences between Sales Cloud and CPQ, Blanka Doktorová

Custom Approval Process: A New Perspective, Pavel Hrbacek & Anindya Halder

Connector Corner: Automate dynamic content and events by pushing a button

Knowledge engineering: from people to machines and back

Agentic RAG What it is its types applications and implementation.pdf

Optimizing NoSQL Performance Through Observability

Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade

Introduction to Open Source RAG and RAG Evaluation

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

ECS 2024 Teams Premium - Pretty Secure

A Business-Centric Approach to Design System Strategy

マーケティングのためのHadoop利用

1. Hadoop

2. • sasata299 ( ) • Hadoop • NoSQL • • http://blog.livedoor.jp/sasata299/

3. Hadoop • Google MapReduce OSS • • • •

4. • Hadoop • Hadoop • • & !?

5. • Hadoop • Hadoop • • & !?

7. 896 30 3 1

8. 896 30 3 1

9. 896 30 3 1 ” ”

10.

11.

12.

13. GROUP BY !! MySQL ( 3.5 )

14. GROUP BY !! MySQL ( 3.5 ) 7000 ≒292 ……orz

15. MySQL

16. … ※

17. !?

18.

19. Hadoop

20. …

21.

22. 7000 30

23.

24. • Hadoop • Hadoop • • & !?

25. Hadoop • Hadoop Streaming • Ruby • Cloudera CDH1 (0.18.3) • EC2 Hadoop • 10-50 • Hadoop S3

26. Hadoop • Hadoop Streaming • Ruby • Cloudera CDH1 (0.18.3) • EC2 Hadoop • 10-50 • Hadoop S3

27.

28.

29.

30.

31.

32. ○○ ×× ○○ ××

33. Hadoop

34.

35.

36. Hadoop AWS

37.

38. Hadoop

39. Hadoop Hadoop (EC2)

40. hadoop-ec2 push [cluster] mapper.rb hadoop-ec2 push [cluster] reducer.rb Hadoop (EC2)

41. hadoop-ec2 push [cluster] mapper.rb hadoop-ec2 push [cluster] reducer.rb Hadoop (EC2)

42. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

43. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

44. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

45. Hadoop hadoop-ec2 exec [cluster] [command] Hadoop S3 (EC2)

46. S3

47. S3

48. Hadoop Hadoop (EC2)

49. Hadoop

50. !!

51. Hadoop

52. 1) 2) Hadoop 3)

53. • Hadoop • Hadoop • • & !?

54. target_ids # [21310,12902,15321,..] ARGF.each do |log| log.chomp! id, foo, bar, ... = log.split(/,/) next if target_ids.include?(id) end target_ids 5 …

55. :-)

56. # 1000 hash = Hash.new {|h,k| h[k] = []} target_ids.each do |_id| hash[_id.to_s[0,3]] << _id end ARGF.each do |log| log.chomp! id, foo, bar, ... = log.split(/,/) # next if hash[id[0,3]].include?(id) end

57.

58. Mapper Reducer - Mapper - Reducer

59. • Hadoop • Hadoop • • & !?

60. EC2 -> AZ -> JobTracker -> 50030 -> hadoop job -list

61. 10h 8h JobTracker

62. !?

63. Amazon Elastic MapReduce

64. Elastic MapReduce