Introduction to Hadoop

1. Hadoop – Taming Big Data Jax ArcSig, June 2012 Ovidiu Dimulescu

2. About @odimulescu •  Working on the Web since 1997 •  Likes stuﬀ well done •  Into engineering cultures and all around automaEon •  Speaker at local user groups •  Organizer for the local Mobile User Group jaxmug.com

3. Agenda •  IntroducEon •  Use cases •  Architecture •  MapReduce Examples •  Q&A

4. What is ? •  Apache Hadoop is an open source Java soSware framework for running data-‐intensive applicaEons on large clusters of commodity hardware •  Created by Doug CuVng (Lucene & Nutch creator) •  Named aSer Doug’s son’s toy elephant

5. What and how is solving? •  Processing diverse large datasets in pracAcal Ame at low cost •  Consolidates data in a distributed ﬁle system •  Moves computaAon to data rather then data to computaEon •  Simpler programming model

6. Why does it maEer? •  Volume, Velocity, Variety and Value •  Datasets do not ﬁt on local HDDs let alone RAM •  Data grows at tremendous pace •  Data is heterogeneous •  Scaling up is expensive (licensing, cpus, disks, interconnects, etc.) •  Scaling up has a ceiling (physical, technical, etc.)

7. Why does it maEer? Data types Complex Data Images, Video 20% Logs Documents Call records Sensor data Mail archives 80% Structured Data User Proﬁles Complex Structured CRM HR Records * Chart Source: IDC White Paper

8. Why does it maEer? •  Need to process a 10TB dataset •  Assume sustained transfer of 75MB/s •  On 1 node -‐ Scanning data ~ 2 days •  On 10 node cluster -‐ Scanning data ~ 5 hrs •  Low $/TB for commodity drives •  Low-‐end servers are mulEcore capable

9. Use Cases •  ETL -‐ Extract Transform Load •  RecommendaEon Engines •  Customer Churn Analysis •  Ad TargeEng •  Data “sandbox”

10. Use Cases -‐ Typical ETL Data Warehouse BI ApplicaAons Live DB ETL 1 ETL 2 ReporAng DB Logs

11. Use Cases -‐ Hadoop ETL Data Warehouse BI ApplicaAons Live DB Data Loading Data Loading ReporAng Hadoop DB Logs

12. Use Cases – Analysis methods •  Pakern recogniEon •  Index building •  Text mining •  CollaboraEve ﬁltering •  PredicEon models •  SenEment analysis •  Graphs creaEon and traversal

13. Who uses it?

14. Who supports it?

15. Why use Hadoop? •  PracEcal to do things that were previously not ü  Shorter execuEon Eme ü  Costs less ü  Simpler programming model •  Open system with greater ﬂexibility •  Large and growing ecosystem

16. Hadoop – Silver bullet? •  Not a database replacement •  Not a data warehousing (complements it) •  Not for interacEve reporEng •  Not a general purpose storage mechanism •  Not for problems that are not parallelizable in a share-‐nothing fashion

17. Architecture – Design Axioms •  System Shall Manage and Heal Itself •  Performance Shall Scale Linearly •  Compute Should Move to Data •  Simple Core, Modular and Extensible

18. Architecture – Core Components HDFS Distributed ﬁlesystem designed for low cost storage and high bandwidth access across the cluster. Map-‐Reduce Programming model for processing and generaEng large data sets.

19. Architecture – Oﬃcial Extensions Management ZooKeeper Chukwa Data Access Pig (Data Flow) Hive (SQL) Avro Data Processing MapReduce Framework Storage HDFS HBase

20. Architecture – CDH DistribuAon 1.  CDH – Cloudera’s DistribuEon of Hadoop 2.  Image credit -‐ Cloudera presentaEon @ Microstrategy World 2011

21. HDFS -‐ Design •  Based on Google’s GFS •  Files are stored as blocks (64MB default size) •  Conﬁgurable data replicaEon (3x, Rack Aware) •  Fault Tolerant, Expects HW failures •  HUGE ﬁles, Expects Streaming not Low Latency •  Mostly WORM

22. HDFS -‐ Architecture Namenode (NN) Client ask NN for ﬁle H NN returns DNs that D host it F Client ask DN for data S Datanode 1 Datanode 2 Datanode N Namenode -‐ Master Datanode -‐ Slaves •  Filesystem metadata •  Reads / Write blocks to/from clients •  Controls read/write to ﬁles •  Replicates blocks at master’s request •  Manages blocks replicaEon •  Applies transacEon log on startup

23. HDFS – Fault tolerance •  DataNode §  Uses CRC to avoid corrupEon §  Data is replicated on other nodes (3x) •  NameNode §  Checkpoint NameNode §  Backup NameNode §  Failover is manual

24. MapReduce -‐ Design •  Based on Google’s MR paper •  Borrows from funcEonal programming •  Simpler programming model §  map (in_key, in_value) -‐> (out_key, intermediate_value) list §  reduce (out_key, intermediate_value list) -‐> out_value list •  No user synchronizaEon and coordinaEon Input -‐> Map -‐> Reduce -‐> Output

25. MapReduce -‐ Architecture Client launches a job J JobsTracker (JT) O -‐ ConﬁguraEon -‐ Mapper B -‐ Reducer S -‐ Input -‐ Output TaskTracker 1 TaskTracker 2 TaskTracker N API JobTracker -‐ Master TaskTracker -‐ Slaves •  Accepts MR jobs submiked by clients •  Run Map and Reduce tasks received •  Assigns Map and Reduce tasks to from Jobtracker TaskTrackers, data locality aware •  Manage storage and transmission of •  Monitors tasks and TaskTracker status, intermediate output re-‐executes tasks upon failure •  SpeculaEve execuEon

26. Hadoop -‐ Core Architecture J JobsTracker O B S API TaskTracker 1 TaskTracker 2 TaskTracker N DataNode 1 DataNode 2 DataNode N H D F S NameNode Mini OS •  File system •  Scheduler

27. MapReduce – Head First Style hkp://www.slideshare.net/esaliya/mapreduce-‐ in-‐simple-‐terms

28. MapReduce – Mapper Types One-‐to-‐One map(k, v) = emit (k, transform(v)) Exploder map(k, v) = foreach p in v: emit (k, p) Filter map(k, v) = if cond(v) then emit (k, v)

29. MapReduce – Reducer Types Sum Reducer reduce(k, vals) = sum = 0 foreach v in vals: sum += v emit (k, sum)

30. MapReduce – High level pipeline K1 K2 K1 K1 K2 K2 K1 K2

31. MapReduce – Detailed pipeline Diagram: hkp://developer.yahoo.com/hadoop/tutorial/module4.html

32. MapReduce – Combiner Phase •  OpEonal •  Runs on mapper nodes aSer map phase •  “ Mini-‐reduce,” only on local map output •  Used to save bandwidth before sending data to full reducer •  The Reducer can be Combiner if 1.  Output key, values are the same as input key, values 2.  CommutaEve and AssociaEve (SUM, MAX ok but AVG not) Diagram: hkp://developer.yahoo.com/hadoop/tutorial/module4.html

33. InstallaAon 1.  Download & conﬁgure single-‐node cluster hadoop.apache.org/common/releases.html 2.  Download a demo VM Cloudera Hortonwork 3.  Use a hosted environment (Amazon’s EMR, Azure)

34. InstallaAon – Pla[orm Notes ProducAon Linux – Oﬃcial Development Linux OSX Windows via Cygwin *Nix

35. MapReduce – Client Languages Java, Any JVM Languages -‐ NaEve hadoop jar jar_path main_class input_path output_path C++ -‐ Pipes framework – Socket IO hadoop pipes -‐input path_in -‐output path_out -‐program exec_program Any – Streaming – Stdin / Stdout hadoop jar hadoop-‐streaming.jar -‐mapper map_prog -‐reducer reduce_prog -‐input path_in -‐output path_out Pig LaEn, Hive HQL, C via JNI

36. MapReduce – Client Anatomy •  Main Program (aka Driver) Conﬁgures the Job IniEates the Job •  Input LocaEon •  Mapper •  Combiner (opEonal) •  Reducer •  Output LocaEon

37. MapReduce – Word Count Example

38. MapReduce – C# Mapper

39. MapReduce – C# Reducer

40. MapReduce – Java Mapper

41. MapReduce – Java Reducer

42. MapReduce – JavaScript Mapper

43. MapReduce – JavaScript Reducer

44. Summary is an economical scalable distributed data processing system which enables data: ü  ConsolidaAon (Structured or Not) ü  Query Flexibility (Any Language) ü  Agility (Evolving Schemas)

45. QuesAons ?

46. References Hadoop at Yahoo!, by Y! Developer Network MapReduce in Simple Terms, by Saliya Ekanayake Hadoop Architecture, by Phillipe Julio 10 Hadoop-‐able Problems, by Cloudera Hadoop, An Industry PerspecEve, by Amr Awadallah Anatomy of a MapReduce Job Run by Tom White MapReduceJobs in Hadoop

Introduction to Hadoop

You are reading a preview.

Check these out next

Introduction to Hadoop

More Related Content

Slideshows for you (20)

Similar to Introduction to Hadoop (20)

More from Ovidiu Dimulescu (9)

Recently uploaded (20)

Introduction to Hadoop