961万人の食卓を支えるデータ解析

•Download as KEY, PDF•

36 likes•5,251 views

This document discusses Hadoop and its ecosystem. It covers Hadoop distributions like Cloudera and Amazon's Elastic MapReduce service. It also discusses running SQL-like queries using MapReduce and moving data between MySQL and Hadoop. Key algorithms like map and reduce functions are explained through examples. Different Hadoop deployment options on EC2 like standalone, Cloudera, and EMR are also summarized.

Technology Spiritual

•
•
• Hadoop (Cloudera)

• Elastic MapReduce
•
•

• (@sasata299)

• 2009 8 JOIN

•
• Hadoop

• Hadoop MySQL

•
- GROUP BY

• 7000 …

• (´Д` )

Hadoop
• Google MapReduce OSS

•
-
-
-
-

Hadoop
master ( )

slave ( )

<key,value>
Map
Shufﬂe & Sort

Hadoop
master ( )

slave ( )

<key,value>
Map Reduce
Shufﬂe & Sort

• Hadoop Streaming (Ruby )

• EC2 Cloudera Hadoop
- Cloudera CDH1
- Hadoop 0.18.3

• S3

MySQL → Hadoop
•
• GROUP BY MapReduce
- ( )
- key

• JOIN MapReduce
•

(1) master
master slave scp

(2) S3

S3 slave scp

MySQL vs Hadoop

7000

MySQL Hadoop
MySQL Hadoop

MySQL vs Hadoop

( Д )
7000
30

MySQL Hadoop
MySQL Hadoop

• Hadoop
-
• Hadoop (HADOOP-6254)
- S3
- SocketTimeoutException

• EMR (Elastic MapReduce)
- Amazon Hadoop

• Cloudera CDH2
-

AMI
(Amazon Machine
UP Image)

EMR

CDH2

EMR
BootStrap Action

Step (Hadoop Job)

Job Flow ( )

•
-
- --alive
• AMI
- AMI

- BootStrap Action

•
- mapred.child.java.opts
- streaming

•
-
- ElasticMapReduce-master 5100

Storage and computation is getting cheaper AND easily accessible on demand in the cloud. We now collect and store some really large data sets Eg: user activity logs, genome sequencing, sensory data etc. Hadoop and the ecosystem of projects built around it present simple and easy to use tools for storing and analyzing such large data collections on commodity hardware. Topics Covered * The Hadoop architecture. * Thinking in MapReduce. * Run some sample MapReduce Jobs (using Hadoop Streaming). * Introduce PigLatin, a easy to use data processing language. Speaker Profile: Mahesh Reddy is an Entrepreneur, chasing dreams. Works on large scale crawl and extraction of structured data from the web. He is a graduate frm IIT Kanpur(2000-05) and previously worked at Yahoo! Labs as Research Engineer/Tech Lead on Search and Advertising products.

Qubole Overview at the Fifth Elephant ConferenceJoydeep Sen Sarma

Hadoop Overview & Architecture

EMC

Hadoop Conference Japan 2011 FallRyu Kobayashi

Hadoop Overview kdd2011

Milind Bhandarkar

データ解析技術入門(Hadoop編)Takumi Asai

サンプルから見るMap reduceコードShinpei Ohtani

サンプルから見るMapReduceコード

Shinpei Ohtani

Hadoop

Rajesh Piryani

Brust hadoopecosystemAndrew Brust

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

からあげエンジニアについて

クックパッドでのemr利用事例

Similar to 961万人の食卓を支えるデータ解析

COOKPADでのHadoop利用Tatsuya Sasaki

Hadoop導入事例 in クックパッド

Tatsuya Sasaki

Amebaサービスのログ解析基盤

Toshihiro Suzuki

Hadoop入門とクラウド利用Naoki Yanai

Hadoop and mysql by Chris SchneiderDmitry Makarchuk

Hadoop london

Yahoo Developer Network

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

npinto

HadoopThe Hadoop Java Software Framework

ThoughtWorks

Qubole Overview at the Fifth Elephant ConferenceJoydeep Sen Sarma

Hadoop Overview & Architecture

EMC

Hadoop Conference Japan 2011 FallRyu Kobayashi

Hadoop Overview kdd2011

Milind Bhandarkar

データ解析技術入門(Hadoop編)Takumi Asai

サンプルから見るMap reduceコードShinpei Ohtani

サンプルから見るMapReduceコード

Shinpei Ohtani

Hadoop

Rajesh Piryani

Brust hadoopecosystemAndrew Brust

Rug hogan-10-03-2012

designandanalytics

Hadoop and MapReduce

Hemanth Kumar Mantri

Azure Data Warehouse

Nikolay Stanev

Similar to 961万人の食卓を支えるデータ解析 (20)

COOKPADでのHadoop利用

Hadoop導入事例 in クックパッド

Amebaサービスのログ解析基盤

Hadoop入門とクラウド利用

Hadoop and mysql by Chris Schneider

Hadoop london

[Harvard CS264] 08b - MapReduce and Hadoop (Zak Stone, Harvard)

HadoopThe Hadoop Java Software Framework

Qubole Overview at the Fifth Elephant Conference

Hadoop Overview & Architecture

Hadoop Conference Japan 2011 Fall

Hadoop Overview kdd2011

データ解析技術入門(Hadoop編)

サンプルから見るMap reduceコード

サンプルから見るMapReduceコード

Hadoop

Brust hadoopecosystem

Rug hogan-10-03-2012

Hadoop and MapReduce

Azure Data Warehouse

Recently uploaded

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Epistemic Interaction - tuning interfaces to provide information for AI support

Alan Dix

Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024 https://alandix.com/academic/papers/synergy2024-epistemic/ As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

Welocme to ViralQR, your best QR code generator.

ViralQR

Welcome to ViralQR, your best QR code generator available on the market! At ViralQR, we design static and dynamic QR codes. Our mission is to make business operations easier and customer engagement more powerful through the use of QR technology. Be it a small-scale business or a huge enterprise, our easy-to-use platform provides multiple choices that can be tailored according to your company's branding and marketing strategies. Our Vision We are here to make the process of creating QR codes easy and smooth, thus enhancing customer interaction and making business more fluid. We very strongly believe in the ability of QR codes to change the world for businesses in their interaction with customers and are set on making that technology accessible and usable far and wide. Our Achievements Ever since its inception, we have successfully served many clients by offering QR codes in their marketing, service delivery, and collection of feedback across various industries. Our platform has been recognized for its ease of use and amazing features, which helped a business to make QR codes. Our Services At ViralQR, here is a comprehensive suite of services that caters to your very needs: Static QR Codes: Create free static QR codes. These QR codes are able to store significant information such as URLs, vCards, plain text, emails and SMS, Wi-Fi credentials, and Bitcoin addresses. Dynamic QR codes: These also have all the advanced features but are subscription-based. They can directly link to PDF files, images, micro-landing pages, social accounts, review forms, business pages, and applications. In addition, they can be branded with CTAs, frames, patterns, colors, and logos to enhance your branding. Pricing and Packages Additionally, there is a 14-day free offer to ViralQR, which is an exceptional opportunity for new users to take a feel of this platform. One can easily subscribe from there and experience the full dynamic of using QR codes. The subscription plans are not only meant for business; they are priced very flexibly so that literally every business could afford to benefit from our service. Why choose us? ViralQR will provide services for marketing, advertising, catering, retail, and the like. The QR codes can be posted on fliers, packaging, merchandise, and banners, as well as to substitute for cash and cards in a restaurant or coffee shop. With QR codes integrated into your business, improve customer engagement and streamline operations. Comprehensive Analytics Subscribers of ViralQR receive detailed analytics and tracking tools in light of having a view of the core values of QR code performance. Our analytics dashboard shows aggregate views and unique views, as well as detailed information about each impression, including time, device, browser, and estimated location by city and country. So, thank you for choosing ViralQR; we have an offer of nothing but the best in terms of QR code services to meet business diversity!

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Assure Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Quantum Computing: Current Landscape and the Future Role of APIs

Vlad Stirbu

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

Free Complete Python - A step towards Data Science

RinaMondal9

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

nkrafacyberclub

Recently uploaded (20)

Assuring Contact Center Experiences for Your Customers With ThousandEyes

A tale of scale & speed: How the US Navy is enabling software delivery from l...

Epistemic Interaction - tuning interfaces to provide information for AI support

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Welocme to ViralQR, your best QR code generator.

UiPath Test Automation using UiPath Test Suite series, part 3

When stars align: studies in data quality, knowledge graphs, and machine lear...

Essentials of Automations: Optimizing FME Workflows with Parameters

Assure Contact Center Experiences for Your Customers With ThousandEyes

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Elevating Tactical DDD Patterns Through Object Calisthenics

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

DevOps and Testing slides at DASA Connect

Quantum Computing: Current Landscape and the Future Role of APIs

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

Free Complete Python - A step towards Data Science

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

961万人の食卓を支えるデータ解析

1. 961

2. • • • Hadoop (Cloudera) • Elastic MapReduce • •

3. • • • Hadoop (Cloudera) • Elastic MapReduce • •

4. • (@sasata299) • 2009 8 JOIN • • Hadoop

5. • • • Hadoop (Cloudera) • Elastic MapReduce • •

7. • 961 • 30 3 1 • - -

8. • • ( , , ...) - - ( , , , …) -

10.

11.

12.

13.

14. • Hadoop MySQL • - GROUP BY • 7000 … • (´Д` )

15. • MySQL - • - -

16.

17. • • • Hadoop (Cloudera) • Elastic MapReduce • •

18. Hadoop • Google MapReduce OSS • - - - -

19. Hadoop master ( ) slave ( )

20. Hadoop master ( ) slave ( ) Map

21. Hadoop master ( ) slave ( ) <key,value> Map Shufﬂe & Sort

22. Hadoop master ( ) slave ( ) <key,value> Map Reduce Shufﬂe & Sort

23. • Hadoop Streaming (Ruby ) • EC2 Cloudera Hadoop - Cloudera CDH1 - Hadoop 0.18.3 • S3

24. MySQL → Hadoop • • GROUP BY MapReduce - ( ) - key • JOIN MapReduce •

25. (1) master (2) S3

26. (1) master (2) S3

27. (1) master master slave scp (2) S3

28. (1) master master slave scp (2) S3 S3 slave scp

29. MySQL vs Hadoop 7000 MySQL Hadoop MySQL Hadoop

30. MySQL vs Hadoop ( Д ) 7000 30 MySQL Hadoop MySQL Hadoop

31. Hadoop++ ←Hadoop ↓MySQL

32. • • • Hadoop (Cloudera) • Elastic MapReduce • •

33. • Hadoop - • Hadoop (HADOOP-6254) - S3 - SocketTimeoutException

34. • EMR (Elastic MapReduce) - Amazon Hadoop • Cloudera CDH2 -

35. AMI (Amazon Machine UP Image) EMR CDH2

36. AMI (Amazon Machine UP Image) EMR CDH2

37. EMR Job Flow ( )

38. EMR BootStrap Action Job Flow ( )

39. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

40. EMR BootStrap Action Step (Hadoop Job) Job Flow ( )

41. • - - --alive • AMI - AMI - BootStrap Action

42.

43. Created job ﬂow j-8IXS98OW1WEE ID

44. Hadoop

45.

46. • - mapred.child.java.opts - streaming • - - ElasticMapReduce-master 5100

47. • • • Hadoop (Cloudera) • Elastic MapReduce • •

48. • Map - • Reduce - key Reduce -

49. UU Map Reduce

50. UU Map ID Reduce

51. UU Map Reduce ID

52. UU Map Reduce ID

53. Map Reduce

54. Map ID key Reduce

55. Map Reduce key Reduce

56. Map 100 100 Reduce key Reduce

57. × Map 100 × 100 Reduce key Reduce

58. × Map 100 ×100 Reduce Reduce key sort

59. × Map 100 ×100 Reduce Reduce key sort

60. Hadoop • - Hadoop

61. Hadoop • - Hadoop

62. Hadoop • - Hadoop

63. Hadoop • - Hadoop

64. Hadoop • - Hadoop

65. • • • Hadoop (Cloudera) • Elastic MapReduce • •

66. • Hadoop - - - Reduce

961万人の食卓を支えるデータ解析

Recommended

Recommended

More Related Content

Similar to 961万人の食卓を支えるデータ解析

Similar to 961万人の食卓を支えるデータ解析 (20)

More from Tatsuya Sasaki

More from Tatsuya Sasaki (7)

Recently uploaded

Recently uploaded (20)

961万人の食卓を支えるデータ解析

Editor's Notes