Introduction to hadoop

660 views
566 views

Published on

Lynx Consultants training about Hadoop

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
660
On SlideShare
0
From Embeds
0
Number of Embeds
23
Actions
Shares
0
Downloads
28
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Introduction to hadoop

  1. 1. Marc  Cluet  –  Lynx  Consultants  What’s  behind  Big  Data  
  2. 2. What we’ll cover?¡  Understand  Hadoop  components  ¡  Understand  different  technologies  involved  ¡  Embrace  Big  Data!  Lynx  Consultants  ©  2013  
  3. 3. What is Big Data?Lynx  Consultants  ©  2013  
  4. 4. What is Big Data?¡   SQL  has  a  limited  ability  to  process  changing  data  §  SQL  schemas  are  the  truth,  data  needs  to  fit  that  Lynx  Consultants  ©  2013  
  5. 5. What is Big Data?¡   Big  Data  is  the  solution!  §  Data  can  be  truly  dynamic  Lynx  Consultants  ©  2013  
  6. 6. What is Big Data?¡   Big  Data  is  the  solution!  §  Data  can  be  truly  dynamic  §  Designed  to  handle  Terabytes  of  data  Lynx  Consultants  ©  2013  
  7. 7. What is Big Data?¡   Big  Data  is  the  solution!  §  Data  can  be  truly  dynamic  §  Designed  to  handle  Terabytes  of  data  §  Designed  for  fault  tolerance  and  securing  data  Lynx  Consultants  ©  2013  
  8. 8. What is Big Data?¡   Big  Data  is  the  solution!  §  Data  can  be  truly  dynamic  §  Designed  to  handle  Terabytes  of  data  §  Designed  for  fault  tolerance  and  securing  data  §  Designed  around  exploiting  hardware  to  the  fullest  Lynx  Consultants  ©  2013  
  9. 9. What is Big Data?¡   Big  Data  is  the  solution!  §  Data  can  be  truly  dynamic  §  Designed  to  handle  Terabytes  of  data  §  Designed  for  fault  tolerance  and  securing  data  §  Designed  around  exploiting  hardware  to  the  fullest  §  Designed  around  Map/Reduce  Lynx  Consultants  ©  2013  
  10. 10. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  11. 11. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  12. 12. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  13. 13. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  14. 14. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  15. 15. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  16. 16. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  17. 17. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  18. 18. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  19. 19. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  20. 20. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  21. 21. Who runs Big Data?¡  A  few  small  companies  Lynx  Consultants  ©  2013  
  22. 22. What is Hadoop?Lynx  Consultants  ©  2013  
  23. 23. What is Hadoop?¡   Hadoop  is  one  of  the  big  players  for  Big  Data  §  Developed  as  an  Open  Source  implementation  to  implement  Google  BigTable  Lynx  Consultants  ©  2013  
  24. 24. What is Hadoop?¡   Hadoop  is  one  of  the  big  players  for  Big  Data  §  Developed  as  an  Open  Source  implementation  to  implement  Google  BigTable  §  Mainly  developed  at  Yahoo!  Lynx  Consultants  ©  2013  
  25. 25. What is Hadoop?¡   Hadoop  is  one  of  the  big  players  for  Big  Data  §  Developed  as  an  Open  Source  implementation  to  implement  Google  BigTable  §  Mainly  developed  at  Yahoo!  §  Current  companies  behind  it:  Hortonworks  and  Cloudera  Lynx  Consultants  ©  2013  
  26. 26. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  §  HDFS  is  a  distributed  filesystem  across  many  nodes  §  Has  many  copies  of  your  data  (default:  3)  §  If  one  node  goes  down  makes  sure  all  the  data  is  rebalanced  Lynx  Consultants  ©  2013  
  27. 27. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  Lynx  Consultants  ©  2013  
  28. 28. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  ¡   Hbase  –  Hadoop  NoSQL  Database  §  Schemaless  Key-­‐Value  storage  §  All  data  exportable  in  JSON  Lynx  Consultants  ©  2013  
  29. 29. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  ¡   Hbase  –  Hadoop  NoSQL  Database  Lynx  Consultants  ©  2013  
  30. 30. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  ¡   Hbase  –  Hadoop  NoSQL  Database  ¡   Map/Reduce  –  The  key  to  it  all  §  This  was  invented  by  Google  §  Given  a  dataset  we  Map  all  that  match  a  criteria  §  Then  we  Reduce  this  to  a  result  Lynx  Consultants  ©  2013  
  31. 31. What are the features of Hadoop?¡  Map/Reduce  –  The  key  to  it  all  Lynx  Consultants  ©  2013  
  32. 32. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  ¡   Hbase  –  Hadoop  NoSQL  Database  ¡   Map/Reduce  –  The  key  to  it  all  ¡   Hive  –  SQL  for  NoSQL  §  Hive  provides  a  SQL  language  called  HiveSQL  §  Provides  a  good  entrance  for  SQL  users  :)  Lynx  Consultants  ©  2013  
  33. 33. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  ¡   Hbase  –  Hadoop  NoSQL  Database  ¡   Map/Reduce  –  The  key  to  it  all  ¡   Hive  –  SQL  for  NoSQL  ¡   Pig  –  Map/Reduce  made  easy  §  Creates  data  results  given  a  reduced  language  §  Reinvents  SQL  somehow  Lynx  Consultants  ©  2013  
  34. 34. What are the features of Hadoop?¡   Hive  Lynx  Consultants  ©  2013  
  35. 35. What are the features of Hadoop?¡   Pig  Lynx  Consultants  ©  2013  
  36. 36. What are the features of Hadoop?¡   HDFS  –  Hadoop  Distributed  File  System  ¡   Hbase  –  Hadoop  NoSQL  Database  ¡   Map/Reduce  –  The  key  to  it  all  ¡   Hive  –  SQL  for  NoSQL  ¡   Pig  –  Map/Reduce  made  easy  ¡   Flume  –  Fault  Tolerant  transport  Lynx  Consultants  ©  2013  
  37. 37. What are the features of Hadoop?¡   Flume  §  Divides  in  Sources,  Channels,  Sinks  §  Can  have  multiple  of  everything,  makes  it  fault  tolerant  §  Many  sources!  ▪  Avro,  Exec,  JMS,  Syslog,  HTTP,  NetCat,  Your  Own  (Java)  Lynx  Consultants  ©  2013  
  38. 38. What are the features of Hadoop?¡   Flume  §  Divides  in  Sources,  Channels,  Sinks  §  Can  have  multiple  of  everything,  makes  it  fault  tolerant  §  Many  sources!  §  Many  channels!  ▪  Memory,  File,  Your  Own  (Java)  Lynx  Consultants  ©  2013  
  39. 39. What are the features of Hadoop?¡   Flume  §  Divides  in  Sources,  Channels,  Sinks  §  Can  have  multiple  of  everything,  makes  it  fault  tolerant  §  Many  sources!  §  Many  channels!  §  Many  sinks!  ▪  Avro,  HDFS,  Logger,  IRC,  File,  Hbase,  ElasticSearch,  S3,  Community  sinks,  Your  Own  (Java)  Lynx  Consultants  ©  2013  
  40. 40. What are the features of Hadoop?¡   Flume  Lynx  Consultants  ©  2013  
  41. 41. How Hadoop looks like in a DC¡   Components  §  Primary  Namenode  §  Secondary  Namenode  §  Data  Node  Lynx  Consultants  ©  2013  
  42. 42. How Hadoop looks like in a DC¡   Components  §  Primary  Namenode  ▪  Controls  all  the  cluster,  knows  where  the  data  resides  ▪  Runs  the  job  tracker  to  keep  track  of  Map/Reduce  jobs  ▪  Biggest  point  of  failure,  shadowing  it  is  a  potential  option  §  Secondary  Namenode  §  Data  Node  Lynx  Consultants  ©  2013  
  43. 43. How Hadoop looks like in a DC¡   Components  §  Primary  Namenode  §  Secondary  Namenode  ▪  Performs  secondary  cleanup  options  §  Data  Node  Lynx  Consultants  ©  2013  
  44. 44. How Hadoop looks like in a DC¡   Components  §  Primary  Namenode  §  Secondary  Namenode  §  Data  Node  ▪  Stores  all  the  information  ▪  Runs  Map/Reduce  Lynx  Consultants  ©  2013  
  45. 45. How Hadoop looks like in a DC¡   Components  Lynx  Consultants  ©  2013  
  46. 46. Questions?Lynx  Consultants  ©  2013  

×