Transforming Mobile Marketing & Advertising™                        Harnessing s for Big Data                        Analy...
Who am I ?   • Architect @ Flytxt (Big Data Analytics & Automation)   • Passionate about data, distributed computing , mac...
Session Agenda!• Data – Whats the big deal?• What is Hadoop( & What it is not  )• Map-Reduce Model & HDFS• Hadoop Ecosyst...
Five computers & a 640k ;-)                                                             "I think there is a world market  ...
Data Explosion !                                                             Confidential         Copyright © 2010 Flytxt ...
Do I also know what you might do next summer?                                        •     Does your travel company know y...
Don‟t throw away data just because it doesnt „fit‟ •   relational tuples, log files, semi structured textual data (e.g., e...
Lets get to business!!What is Apache Hadoop ?•   Apache Hadoop is an open-source system to    reliably store and process e...
Basics of Hadoop •   Two Core Components – HDFS & Map-Reduce •   Machines are un-reliable •   Separates distributed fault-...
Lead Actors•   Name Node – Book keeping metadata server•   Secondary Name Node – Assistant to Name Node•   Job Tracker – S...
HDFS Write Model                                                            Confidential        Copyright © 2010 Flytxt B....
Map-Reduce Model                                                          Confidential      Copyright © 2010 Flytxt B.V. A...
Map-Reduce Execution Flow                                                          Confidential      Copyright © 2010 Flyt...
Hadoop Ecosystem•   Oozie – Open-source workflow/coordination    service to manage data processing jobs for Apache    Hado...
Hadoop is not a “Holy Grail”•   Not a substitute for a database•   MapReduce is not always the best algorithm•   HDFS is n...
Notable Users of Hadoop(Source: http://en.wikipedia.org/wiki/Hadoop)     • A9.com                               • Meebo   ...
Q&A                                                    www.flytxt.com                                                    C...
THANK YOU      contact us : dev2dev@flytxt.com/ jobin.wilson@flytxt.com                                                   ...
Upcoming SlideShare
Loading in...5
×

Harnessing hadoop for big data analytics v0.1

4,023

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,023
On Slideshare
0
From Embeds
0
Number of Embeds
30
Actions
Shares
0
Downloads
55
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Harnessing hadoop for big data analytics v0.1

  1. 1. Transforming Mobile Marketing & Advertising™ Harnessing s for Big Data Analytics Jobin Wilson jobin.wilson@flytxt.com Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  2. 2. Who am I ? • Architect @ Flytxt (Big Data Analytics & Automation) • Passionate about data, distributed computing , machine learning • Previously •Virtualization & Cloud Lifecycle Management(BMC) • Designed and Implemented Cloud Life Cycle Management Interface for BMC • Large Scale Data Centre Automation(AOL) • Implemented Centralized Data Center Management Framework for AOL •Workflow Systems & Automation (Accenture) • Implemented Service Management Suit for various customers Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  3. 3. Session Agenda!• Data – Whats the big deal?• What is Hadoop( & What it is not  )• Map-Reduce Model & HDFS• Hadoop Ecosystem & Tools• Lets get started!• Q&A 3 Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  4. 4. Five computers & a 640k ;-) "I think there is a world market for about five computers" Moore’s Law Thomas Watson 1943, Chairman of the board of IBM "640k ought to be enough for anybody" Attributed to Bill Gates in 1981. Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  5. 5. Data Explosion ! Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  6. 6. Do I also know what you might do next summer? • Does your travel company know you visited Goa & Cochin twice in the last two years? • Collaborative Filtering • Lots of Data + Statistics = WOW!!! • BTW, don’t worry about the eqn  Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  7. 7. Don‟t throw away data just because it doesnt „fit‟ • relational tuples, log files, semi structured textual data (e.g., e-mail),pictures , videos • User generated data & System generated data • Applications need more than structured data • My application is not “Dumb” any more!! • “I keep saying that the sexy job in the next 10 years will be statisticians, and I’m not kidding.” - Hal Varian (Google’s chief economist) Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  8. 8. Lets get to business!!What is Apache Hadoop ?• Apache Hadoop is an open-source system to reliably store and process extremely large data sets across many commodity computers.• originally developed to support Nutch search engine project.• scales linearly with data size or analysis complexity• Scale-out ,shared nothing architecture• inspired by Googles MapReduce and Google File System (GFS) papers Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  9. 9. Basics of Hadoop • Two Core Components – HDFS & Map-Reduce • Machines are un-reliable • Separates distributed fault-tolerant computing code from application logic. • No need to worry about identity of a machine • lets you interact with a cluster, not a bunch of machines. • Analysis workloads span across multiple machines • runs as a cloud(cluster) & possibly on a cloud (EC2) Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  10. 10. Lead Actors• Name Node – Book keeping metadata server• Secondary Name Node – Assistant to Name Node• Job Tracker – Scheduler• Task Tracker - Task execution• Data Node - Block storage Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  11. 11. HDFS Write Model Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  12. 12. Map-Reduce Model Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  13. 13. Map-Reduce Execution Flow Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  14. 14. Hadoop Ecosystem• Oozie – Open-source workflow/coordination service to manage data processing jobs for Apache Hadoop™ - Developed at Yahoo!• HBase – Column-store database based on Google’s BigTable. Holds extremely large data sets (Petabytes)• Hive – SQL based data warehousing app with features for analyzing very large data sets - Developed at Facebook• Zoo Keeper – Distributed consensus engine providing Leader election, service discovery, distributed locking / mutual exclusion• Pig - platform for analyzing large data sets that consists of a high-level language for expressing data analysis steps• Ganglia - a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  15. 15. Hadoop is not a “Holy Grail”• Not a substitute for a database• MapReduce is not always the best algorithm• HDFS is not a substitute for a High Availability SAN-hosted FS• HDFS is not a Posix file system• Not a place to learn Java programming• Not a place to learn Unix/Linux system administration• Not a place to learn basics of networking Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  16. 16. Notable Users of Hadoop(Source: http://en.wikipedia.org/wiki/Hadoop) • A9.com • Meebo • AOL • Metaweb • EHarmony • The New York Times • eBay • Rackspace • Facebook • StumbleUpon • Fox Interactive Media • Twitter • IBM • Yahoo • Last.fm • Amazon • LinkedIn Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
  17. 17. Q&A www.flytxt.com ConfidentialCopyright © 2010 Flytxt B.V. All rights reserved.
  18. 18. THANK YOU contact us : dev2dev@flytxt.com/ jobin.wilson@flytxt.com www.flytxt.com Confidential 18Copyright © 2010 Flytxt B.V. All rights reserved.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×