Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem at Twitter, Dmitriy Ryaboy, Twitter

•Download as PPT, PDF•

17 likes•6,205 views

Hadoop User Group

Technology

Hadoop, Pig, HBase at Twitter ,[object Object],[object Object],[object Object]

Who is this guy, anyway ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

In This Talk ,[object Object],[object Object],[object Object],[object Object],[object Object]

Not In This Talk ,[object Object],[object Object],[object Object],[object Object],[object Object]

Daily workload ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Twitter data pipeline (simplified) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Logs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Tables ,[object Object],[object Object],[object Object],[object Object]

ETL ,[object Object],[object Object],[object Object]

Mutability ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Aren't you guys Cassandra poster boys? poster boys? ,[object Object],[object Object],[object Object],[object Object],[object Object]

HBase schema for MySQL exports, v1. ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HBase schema v1, cont. ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HBase schema, v2. ,[object Object],[object Object],[object Object],[object Object],[object Object]

Why Pig? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HBase Loader enhancements ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

HBase Loader TODOs ,[object Object],[object Object],[object Object],[object Object],[object Object]

Elephant Bird ,[object Object],[object Object],[object Object],[object Object],[object Object]

Bad records kill jobs ,[object Object],[object Object],[object Object],[object Object]

Runaway UDFs kill jobs ,[object Object],[object Object],[object Object],[object Object],[object Object]

Use Counters ,[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],Lazy deserializaton FTW lazy deserialization

Also see ,[object Object],[object Object],[object Object],[object Object],[object Object]

Questions ? Follow me at twitter.com/squarecog TM

Photo Credits ,[object Object],[object Object],[object Object]

What's hot

HUG August 2010: Best practicesHadoop User Group

January 2011 HUG: Howl PresentationYahoo Developer Network

Nov 2010 HUG: Fuzzy Table - B.A.HYahoo Developer Network

January 2011 HUG: Kafka PresentationYahoo Developer Network

Nov HUG 2009: Hadoop Record Reader In PythonYahoo Developer Network

Hadoop at Yahoo! -- University Talksyhadoop

Nextag talkJoydeep Sen Sarma

The Bixo Web Mining ToolkitTom Croucher

Hadoop trainting in hyderabad@kelly technologiesKelly Technologies

Messaging architecture @FB (Fifth Elephant Conference)Joydeep Sen Sarma

Karmasphere Studio for HadoopHadoop User Group

introduction to data processing using Hadoop and PigRicardo Varela

Hadoop Hive Talk At IIT-DelhiJoydeep Sen Sarma

Hadoop Tutorialawesomesos

Hadoop - OverviewJay

ImpalaToGo use caseDavid Groozman

Cloud Optimized Big DataJoydeep Sen Sarma

HUG Nov 2010: HDFS Raid - FacebookYahoo Developer Network

Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)Hari Shankar Sreekumar

Hadoop basicsAntonio Silveira

What's hot (20)

HUG August 2010: Best practices

January 2011 HUG: Howl Presentation

Nov 2010 HUG: Fuzzy Table - B.A.H

January 2011 HUG: Kafka Presentation

Nov HUG 2009: Hadoop Record Reader In Python

Hadoop at Yahoo! -- University Talks

Nextag talk

The Bixo Web Mining Toolkit

Hadoop trainting in hyderabad@kelly technologies

Messaging architecture @FB (Fifth Elephant Conference)

Karmasphere Studio for Hadoop

introduction to data processing using Hadoop and Pig

Hadoop Hive Talk At IIT-Delhi

Hadoop Tutorial

Hadoop - Overview

ImpalaToGo use case

Cloud Optimized Big Data

HUG Nov 2010: HDFS Raid - Facebook

Hadoop architecture (Delhi Hadoop User Group Meetup 10 Sep 2011)

Hadoop basics

Viewers also liked

Yahoo! Mail antispam - Bay area Hadoop user groupHadoop User Group

Common crawlpresentationHadoop User Group

Hdfs high availabilityHadoop User Group

Pig at LinkedinHadoop User Group

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector Yahoo Developer Network

January 2011 HUG: Pig PresentationYahoo Developer Network

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...Yahoo Developer Network

August 2016 HUG: Recent development in Apache OozieYahoo Developer Network

Karmasphere hadoop-productivity-toolsHadoop User Group

Cascalog internal dsl_presoHadoop User Group

Yahoo compares Storm and SparkChicago Hadoop Users Group

Nov 2010 HUG: Business Intelligence for Big DataYahoo Developer Network

Next Generation MapReduceOwen O'Malley

Bay Area HUG Feb 2011 IntroOwen O'Malley

Next Generation Hadoop OperationsOwen O'Malley

Hadoop Summit 2010 Benchmarking And Optimizing HadoopYahoo Developer Network

Rate Limiting at Scale, from SANS AppSec Las Vegas 2012Nick Galbreath

AWS Customer Presentation - eHarmonyAmazon Web Services

Twitter Protobufs And Hadoop Hug 021709Hadoop User Group

Viewers also liked (19)

Yahoo! Mail antispam - Bay area Hadoop user group

Common crawlpresentation

Hdfs high availability

Pig at Linkedin

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

January 2011 HUG: Pig Presentation

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...

August 2016 HUG: Recent development in Apache Oozie

Karmasphere hadoop-productivity-tools

Cascalog internal dsl_preso

Yahoo compares Storm and Spark

Nov 2010 HUG: Business Intelligence for Big Data

Next Generation MapReduce

Bay Area HUG Feb 2011 Intro

Next Generation Hadoop Operations

Hadoop Summit 2010 Benchmarking And Optimizing Hadoop

Rate Limiting at Scale, from SANS AppSec Las Vegas 2012

AWS Customer Presentation - eHarmony

Twitter Protobufs And Hadoop Hug 021709

Similar to Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem at Twitter, Dmitriy Ryaboy, Twitter

Hadoop and Pig at Twitter__HadoopSummit2010Yahoo Developer Network

The other Apache technologies your big data solution needs!gagravarr

Eric Baldeschwieler Keynote from Storage Developers ConferenceHortonworks

Overview of big data & hadoop version 1 - Tony NguyenThanh Nguyen

Overview of Big data, Hadoop and Microsoft BI - version1Thanh Nguyen

Big data Hadoop Analytic and Data warehouse comparison guideDanairat Thanabodithammachari

Big data hadooop analytic and data warehouse comparison guideDanairat Thanabodithammachari

Hadoop Frameworks Panel__HadoopSummit2010Yahoo Developer Network

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...Cloudera, Inc.

Hadoop demo pptPhil Young

Splice Machine OverviewKunal Gupta

Capital onehadoopintroDoug Chang

Sql saturday pig session (wes floyd) v2Wes Floyd

Hadoop ecosystem framework n hadoop in live environmentDelhi/NCR HUG

Treasure Data and OSSN Masahiro

מיכאלsqlserver.co.il

Windows Azure HDInsight ServiceNeil Mackenzie

Hive at bookingDavid Morel

Hadoop summit 2010 frameworks panel elephant birdKevin Weil

Big-Data Hadoop Tutorials - MindScripts Technologies, Pune amrutupre

Similar to Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem at Twitter, Dmitriy Ryaboy, Twitter (20)

Hadoop and Pig at Twitter__HadoopSummit2010

The other Apache technologies your big data solution needs!

Eric Baldeschwieler Keynote from Storage Developers Conference

Overview of big data & hadoop version 1 - Tony Nguyen

Overview of Big data, Hadoop and Microsoft BI - version1

Big data Hadoop Analytic and Data warehouse comparison guide

Big data hadooop analytic and data warehouse comparison guide

Hadoop Frameworks Panel__HadoopSummit2010

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...

Hadoop demo ppt

Splice Machine Overview

Capital onehadoopintro

Sql saturday pig session (wes floyd) v2

Hadoop ecosystem framework n hadoop in live environment

Treasure Data and OSS

מיכאל

Windows Azure HDInsight Service

Hive at booking

Hadoop summit 2010 frameworks panel elephant bird

Big-Data Hadoop Tutorials - MindScripts Technologies, Pune

Recently uploaded

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

A Year of the Servo Reboot: Where Are We Now?Igalia

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal

Artificial Intelligence: Facts and MythsJoaquim Jorge

A Call to Action for Generative AI in 2024Results

Real Time Object Detection Using Open CVKhem

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j

🐬 The future of MySQL is Postgres 🐘RTylerCroy

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

Recently uploaded (20)

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Driving Behavioral Change for Information Management through Data-Driven Gree...

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

A Year of the Servo Reboot: Where Are We Now?

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Axa Assurance Maroc - Insurer Innovation Award 2024

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

The 7 Things I Know About Cyber Security After 25 Years | April 2024

08448380779 Call Girls In Civil Lines Women Seeking Men

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Artificial Intelligence: Facts and Myths

A Call to Action for Generative AI in 2024

Real Time Object Detection Using Open CV

Automating Google Workspace (GWS) & more with Apps Script

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

🐬 The future of MySQL is Postgres 🐘

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Factors to Consider When Choosing Accounts Payable Services Providers.pptx

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...