• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Harnessing hadoop for big data analytics v0.1
 

Harnessing hadoop for big data analytics v0.1

on

  • 4,133 views

 

Statistics

Views

Total Views
4,133
Views on SlideShare
1,823
Embed Views
2,310

Actions

Likes
1
Downloads
54
Comments
0

39 Embeds 2,310

http://www.flytxt.com 707
http://jobinwilson.blogspot.com 532
http://flytxt.com 418
http://jobinwilson.blogspot.in 304
http://192.168.1.55 143
http://paktolus.com 35
http://192.168.1.100 27
http://192.168.1.185 18
http://jobinwilson.blogspot.kr 18
http://jobinwilson.blogspot.ca 15
http://jobinwilson.blogspot.co.uk 11
http://jobinwilson.blogspot.de 9
http://192.168.1.67 9
http://jobinwilson.blogspot.com.au 9
http://webcache.googleusercontent.com 6
http://jobinwilson.blogspot.com.br 5
http://jobinwilson.blogspot.fr 4
http://jobinwilson.blogspot.it 4
http://jobinwilson.blogspot.com.es 4
http://jobinwilson.blogspot.pt 3
http://jobinwilson.blogspot.jp 3
http://jobinwilson.blogspot.ch 3
http://flytxt.paktolus.com 2
http://jobinwilson.blogspot.sg 2
http://translate.googleusercontent.com 2
http://jobinwilson.blogspot.se 2
http://jobinwilson.blogspot.ro 2
http://jobinwilson.blogspot.co.il 2
http://jobinwilson.blogspot.gr 1
http://jobinwilson.blogspot.hk 1
http://jobinwilson.blogspot.nl 1
http://jobinwilson.blogspot.ae 1
https://nr-029.appspot.com 1
http://jobinwilson.blogspot.ie 1
http://jobinwilson.blogspot.com.ar 1
http://jobinwilson.blogspot.no 1
http://jobinwilson.blogspot.fi 1
http://jobinwilson.blogspot.mx 1
http://jobinwilson.blogspot.tw 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Harnessing hadoop for big data analytics v0.1 Harnessing hadoop for big data analytics v0.1 Presentation Transcript

    • Transforming Mobile Marketing & Advertising™ Harnessing s for Big Data Analytics Jobin Wilson jobin.wilson@flytxt.com Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Who am I ? • Architect @ Flytxt (Big Data Analytics & Automation) • Passionate about data, distributed computing , machine learning • Previously •Virtualization & Cloud Lifecycle Management(BMC) • Designed and Implemented Cloud Life Cycle Management Interface for BMC • Large Scale Data Centre Automation(AOL) • Implemented Centralized Data Center Management Framework for AOL •Workflow Systems & Automation (Accenture) • Implemented Service Management Suit for various customers Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Session Agenda!• Data – Whats the big deal?• What is Hadoop( & What it is not  )• Map-Reduce Model & HDFS• Hadoop Ecosystem & Tools• Lets get started!• Q&A 3 Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Five computers & a 640k ;-) "I think there is a world market for about five computers" Moore’s Law Thomas Watson 1943, Chairman of the board of IBM "640k ought to be enough for anybody" Attributed to Bill Gates in 1981. Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Data Explosion ! Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Do I also know what you might do next summer? • Does your travel company know you visited Goa & Cochin twice in the last two years? • Collaborative Filtering • Lots of Data + Statistics = WOW!!! • BTW, don’t worry about the eqn  Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Don‟t throw away data just because it doesnt „fit‟ • relational tuples, log files, semi structured textual data (e.g., e-mail),pictures , videos • User generated data & System generated data • Applications need more than structured data • My application is not “Dumb” any more!! • “I keep saying that the sexy job in the next 10 years will be statisticians, and I’m not kidding.” - Hal Varian (Google’s chief economist) Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Lets get to business!!What is Apache Hadoop ?• Apache Hadoop is an open-source system to reliably store and process extremely large data sets across many commodity computers.• originally developed to support Nutch search engine project.• scales linearly with data size or analysis complexity• Scale-out ,shared nothing architecture• inspired by Googles MapReduce and Google File System (GFS) papers Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Basics of Hadoop • Two Core Components – HDFS & Map-Reduce • Machines are un-reliable • Separates distributed fault-tolerant computing code from application logic. • No need to worry about identity of a machine • lets you interact with a cluster, not a bunch of machines. • Analysis workloads span across multiple machines • runs as a cloud(cluster) & possibly on a cloud (EC2) Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Lead Actors• Name Node – Book keeping metadata server• Secondary Name Node – Assistant to Name Node• Job Tracker – Scheduler• Task Tracker - Task execution• Data Node - Block storage Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • HDFS Write Model Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Map-Reduce Model Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Map-Reduce Execution Flow Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Hadoop Ecosystem• Oozie – Open-source workflow/coordination service to manage data processing jobs for Apache Hadoop™ - Developed at Yahoo!• HBase – Column-store database based on Google’s BigTable. Holds extremely large data sets (Petabytes)• Hive – SQL based data warehousing app with features for analyzing very large data sets - Developed at Facebook• Zoo Keeper – Distributed consensus engine providing Leader election, service discovery, distributed locking / mutual exclusion• Pig - platform for analyzing large data sets that consists of a high-level language for expressing data analysis steps• Ganglia - a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Hadoop is not a “Holy Grail”• Not a substitute for a database• MapReduce is not always the best algorithm• HDFS is not a substitute for a High Availability SAN-hosted FS• HDFS is not a Posix file system• Not a place to learn Java programming• Not a place to learn Unix/Linux system administration• Not a place to learn basics of networking Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Notable Users of Hadoop(Source: http://en.wikipedia.org/wiki/Hadoop) • A9.com • Meebo • AOL • Metaweb • EHarmony • The New York Times • eBay • Rackspace • Facebook • StumbleUpon • Fox Interactive Media • Twitter • IBM • Yahoo • Last.fm • Amazon • LinkedIn Confidential Copyright © 2010 Flytxt B.V. All rights reserved.
    • Q&A www.flytxt.com ConfidentialCopyright © 2010 Flytxt B.V. All rights reserved.
    • THANK YOU contact us : dev2dev@flytxt.com/ jobin.wilson@flytxt.com www.flytxt.com Confidential 18Copyright © 2010 Flytxt B.V. All rights reserved.