961万人の食卓を支えるデータ解析

•Download as KEY, PDF•

36 likes•5,251 views

This document discusses Hadoop and its ecosystem. It covers Hadoop distributions like Cloudera and Amazon's Elastic MapReduce service. It also discusses running SQL-like queries using MapReduce and moving data between MySQL and Hadoop. Key algorithms like map and reduce functions are explained through examples. Different Hadoop deployment options on EC2 like standalone, Cloudera, and EMR are also summarized.

Technology Spiritual

•
•
• Hadoop (Cloudera)

• Elastic MapReduce
•
•

• (@sasata299)

• 2009 8 JOIN

•
• Hadoop

• Hadoop MySQL

•
- GROUP BY

• 7000 …

• (´Д` )

Hadoop
• Google MapReduce OSS

•
-
-
-
-

Hadoop
master ( )

slave ( )

<key,value>
Map
Shufﬂe & Sort

Hadoop
master ( )

slave ( )

<key,value>
Map Reduce
Shufﬂe & Sort

• Hadoop Streaming (Ruby )

• EC2 Cloudera Hadoop
- Cloudera CDH1
- Hadoop 0.18.3

• S3

MySQL → Hadoop
•
• GROUP BY MapReduce
- ( )
- key

• JOIN MapReduce
•

(1) master
master slave scp

(2) S3

S3 slave scp

MySQL vs Hadoop

7000

MySQL Hadoop
MySQL Hadoop

MySQL vs Hadoop

( Д )
7000
30

MySQL Hadoop
MySQL Hadoop

• Hadoop
-
• Hadoop (HADOOP-6254)
- S3
- SocketTimeoutException

• EMR (Elastic MapReduce)
- Amazon Hadoop

• Cloudera CDH2
-

AMI
(Amazon Machine
UP Image)

EMR

CDH2

EMR
BootStrap Action

Step (Hadoop Job)

Job Flow ( )

•
-
- --alive
• AMI
- AMI

- BootStrap Action

•
- mapred.child.java.opts
- streaming

•
-
- ElasticMapReduce-master 5100

Storage and computation is getting cheaper AND easily accessible on demand in the cloud. We now collect and store some really large data sets Eg: user activity logs, genome sequencing, sensory data etc. Hadoop and the ecosystem of projects built around it present simple and easy to use tools for storing and analyzing such large data collections on commodity hardware. Topics Covered * The Hadoop architecture. * Thinking in MapReduce. * Run some sample MapReduce Jobs (using Hadoop Streaming). * Introduce PigLatin, a easy to use data processing language. Speaker Profile: Mahesh Reddy is an Entrepreneur, chasing dreams. Works on large scale crawl and extraction of structured data from the web. He is a graduate frm IIT Kanpur(2000-05) and previously worked at Yahoo! Labs as Research Engineer/Tech Lead on Search and Advertising products.

Qubole Overview at the Fifth Elephant ConferenceJoydeep Sen Sarma

Hadoop Overview & Architecture

EMC

Hadoop Conference Japan 2011 FallRyu Kobayashi

Hadoop Overview kdd2011

Milind Bhandarkar

データ解析技術入門(Hadoop編)Takumi Asai

サンプルから見るMap reduceコードShinpei Ohtani

サンプルから見るMapReduceコード

Shinpei Ohtani

Hadoop

Rajesh Piryani

Brust hadoopecosystemAndrew Brust

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

からあげエンジニアについて

クックパッドでのemr利用事例

Similar to 961万人の食卓を支えるデータ解析

COOKPADでのHadoop利用Tatsuya Sasaki

Hadoop導入事例 in クックパッド

Tatsuya Sasaki

Amebaサービスのログ解析基盤

Toshihiro Suzuki

Hadoop入門とクラウド利用Naoki Yanai

Hadoop and mysql by Chris SchneiderDmitry Makarchuk

Hadoop london

Yahoo Developer Network

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

npinto

HadoopThe Hadoop Java Software Framework

ThoughtWorks

Qubole Overview at the Fifth Elephant ConferenceJoydeep Sen Sarma

Hadoop Overview & Architecture

EMC

Hadoop Conference Japan 2011 FallRyu Kobayashi

Hadoop Overview kdd2011

Milind Bhandarkar

データ解析技術入門(Hadoop編)Takumi Asai

サンプルから見るMap reduceコードShinpei Ohtani

サンプルから見るMapReduceコード

Shinpei Ohtani

Hadoop

Rajesh Piryani

Brust hadoopecosystemAndrew Brust

Rug hogan-10-03-2012

designandanalytics

Hadoop and MapReduce

Hemanth Kumar Mantri

Azure Data Warehouse

Nikolay Stanev

Similar to 961万人の食卓を支えるデータ解析 (20)

COOKPADでのHadoop利用

Hadoop導入事例 in クックパッド

Amebaサービスのログ解析基盤

Hadoop入門とクラウド利用

Hadoop and mysql by Chris Schneider

Hadoop london

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

HadoopThe Hadoop Java Software Framework

Qubole Overview at the Fifth Elephant Conference

Hadoop Overview & Architecture

Hadoop Conference Japan 2011 Fall

Hadoop Overview kdd2011

データ解析技術入門(Hadoop編)

サンプルから見るMap reduceコード

サンプルから見るMapReduceコード

Hadoop

Brust hadoopecosystem

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

Recently uploaded

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Tobias Schneck

As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other? Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Ramesh Iyer

In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

Bits & Pixels using AI for Good.........

Alison B. Lowndes

DevOps and Testing slides at DASA Connect

Kari Kakkonen

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

Search and Society: Reimagining Information Access for Radical Futures

Bhaskar Mitra

The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.

PHP Frameworks: I want to break free (IPC Berlin 2024)

Ralf Eggert

In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development. This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

FIDO Alliance

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

How world-class product teams are winning in the AI era by CEO and Founder, P...

Product School

Recently uploaded (20)

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024

Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Bits & Pixels using AI for Good.........

DevOps and Testing slides at DASA Connect

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

GraphRAG is All You need? LLM & Knowledge Graph

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Assuring Contact Center Experiences for Your Customers With ThousandEyes

When stars align: studies in data quality, knowledge graphs, and machine lear...

Search and Society: Reimagining Information Access for Radical Futures

PHP Frameworks: I want to break free (IPC Berlin 2024)

FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Key Trends Shaping the Future of Infrastructure.pdf

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Connector Corner: Automate dynamic content and events by pushing a button

How world-class product teams are winning in the AI era by CEO and Founder, P...

961万人の食卓を支えるデータ解析

1. 961

2. • • • Hadoop (Cloudera) • Elastic MapReduce • •

3. • • • Hadoop (Cloudera) • Elastic MapReduce • •

4. • (@sasata299) • 2009 8 JOIN • • Hadoop

5. • • • Hadoop (Cloudera) • Elastic MapReduce • •

7. • 961 • 30 3 1 • - -

8. • • ( , , ...) - - ( , , , …) -

10.

11.

12.

13.

14. • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )

15. • MySQL - • - -

16.

17. • • • Hadoop (Cloudera) • Elastic MapReduce • •

18. Hadoop • Google MapReduce OSS • - - - -

19. Hadoop master ( ) slave ( )

20. Hadoop master ( ) slave ( ) Map

21. Hadoop master ( ) slave ( ) <key,value> Map Shufﬂe & Sort

22. Hadoop master ( ) slave ( ) <key,value> Map Reduce Shufﬂe & Sort

23. • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3

24. MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •

25. (1) master (2) S3

26. (1) master (2) S3

27. (1) master master slave scp (2) S3

28. (1) master master slave scp (2) S3 S3 slave scp

29. MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop

30. MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop

31. Hadoop++ ←Hadoop ↓MySQL

32. • • • Hadoop (Cloudera) • Elastic MapReduce • •

33. • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException

34. • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -

35. AMI (Amazon Machine UP Image) EMR CDH2

36. AMI (Amazon Machine UP Image) EMR CDH2

37. EMR Job Flow ( )

38. EMR BootStrap Action Job Flow ( )

39. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

40. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

41. • - - --alive • AMI - AMI - BootStrap Action

42.

43. Created job ﬂow j-8IXS98OW1WEE ID

44. Hadoop

45.

46. • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100

47. • • • Hadoop (Cloudera) • Elastic MapReduce • •

48. • Map - • Reduce - key Reduce -

49. UU Map Reduce

50. UU Map ID Reduce

51. UU Map Reduce ID

52. UU Map Reduce ID

53. Map Reduce

54. Map ID key Reduce

55. Map Reduce key Reduce

56. Map 100 100 Reduce key Reduce

57. × Map 100 × 100 Reduce key Reduce

58. × Map 100 ×100 Reduce Reduce key sort

59. × Map 100 ×100 Reduce Reduce key sort

60. Hadoop • - Hadoop

61. Hadoop • - Hadoop

62. Hadoop • - Hadoop

63. Hadoop • - Hadoop

64. Hadoop • - Hadoop

65. • • • Hadoop (Cloudera) • Elastic MapReduce • •

66. • Hadoop - - - Reduce

961万人の食卓を支えるデータ解析

Recommended

Recommended

More Related Content

Similar to 961万人の食卓を支えるデータ解析

Similar to 961万人の食卓を支えるデータ解析 (20)

More from Tatsuya Sasaki

More from Tatsuya Sasaki (7)

Recently uploaded

Recently uploaded (20)

961万人の食卓を支えるデータ解析

Editor's Notes