Introduction to hadoop

1. Marc Cluet – Lynx Consultants What’s behind Big Data

8. What is Big Data? ¡  Big Data is the solution! §  Data can be truly dynamic §  Designed to handle Terabytes of data §  Designed for fault tolerance and securing data §  Designed around exploiting hardware to the fullest Lynx Consultants © 2013

9. What is Big Data? ¡  Big Data is the solution! §  Data can be truly dynamic §  Designed to handle Terabytes of data §  Designed for fault tolerance and securing data §  Designed around exploiting hardware to the fullest §  Designed around Map/Reduce Lynx Consultants © 2013

25. What is Hadoop? ¡  Hadoop is one of the big players for Big Data §  Developed as an Open Source implementation to implement Google BigTable §  Mainly developed at Yahoo! §  Current companies behind it: Hortonworks and Cloudera Lynx Consultants © 2013

26. What are the features of Hadoop? ¡  HDFS – Hadoop Distributed File System §  HDFS is a distributed ﬁlesystem across many nodes §  Has many copies of your data (default: 3) §  If one node goes down makes sure all the data is rebalanced Lynx Consultants © 2013

28. What are the features of Hadoop? ¡  HDFS – Hadoop Distributed File System ¡  Hbase – Hadoop NoSQL Database §  Schemaless Key-‐Value storage §  All data exportable in JSON Lynx Consultants © 2013

30. What are the features of Hadoop? ¡  HDFS – Hadoop Distributed File System ¡  Hbase – Hadoop NoSQL Database ¡  Map/Reduce – The key to it all §  This was invented by Google §  Given a dataset we Map all that match a criteria §  Then we Reduce this to a result Lynx Consultants © 2013

32. What are the features of Hadoop? ¡  HDFS – Hadoop Distributed File System ¡  Hbase – Hadoop NoSQL Database ¡  Map/Reduce – The key to it all ¡  Hive – SQL for NoSQL §  Hive provides a SQL language called HiveSQL §  Provides a good entrance for SQL users :) Lynx Consultants © 2013

33. What are the features of Hadoop? ¡  HDFS – Hadoop Distributed File System ¡  Hbase – Hadoop NoSQL Database ¡  Map/Reduce – The key to it all ¡  Hive – SQL for NoSQL ¡  Pig – Map/Reduce made easy §  Creates data results given a reduced language §  Reinvents SQL somehow Lynx Consultants © 2013

36. What are the features of Hadoop? ¡  HDFS – Hadoop Distributed File System ¡  Hbase – Hadoop NoSQL Database ¡  Map/Reduce – The key to it all ¡  Hive – SQL for NoSQL ¡  Pig – Map/Reduce made easy ¡  Flume – Fault Tolerant transport Lynx Consultants © 2013

37. What are the features of Hadoop? ¡  Flume §  Divides in Sources, Channels, Sinks §  Can have multiple of everything, makes it fault tolerant §  Many sources! ▪  Avro, Exec, JMS, Syslog, HTTP, NetCat, Your Own (Java) Lynx Consultants © 2013

38. What are the features of Hadoop? ¡  Flume §  Divides in Sources, Channels, Sinks §  Can have multiple of everything, makes it fault tolerant §  Many sources! §  Many channels! ▪  Memory, File, Your Own (Java) Lynx Consultants © 2013

39. What are the features of Hadoop? ¡  Flume §  Divides in Sources, Channels, Sinks §  Can have multiple of everything, makes it fault tolerant §  Many sources! §  Many channels! §  Many sinks! ▪  Avro, HDFS, Logger, IRC, File, Hbase, ElasticSearch, S3, Community sinks, Your Own (Java) Lynx Consultants © 2013

42. How Hadoop looks like in a DC ¡  Components §  Primary Namenode ▪  Controls all the cluster, knows where the data resides ▪  Runs the job tracker to keep track of Map/Reduce jobs ▪  Biggest point of failure, shadowing it is a potential option §  Secondary Namenode §  Data Node Lynx Consultants © 2013

Introduction to hadoop

You are reading a preview.

Check these out next

Introduction to hadoop

More Related Content

Slideshows for you (20)

Viewers also liked (7)

Similar to Introduction to hadoop (20)

More from Marc Cluet (15)

Recently uploaded (20)