Hive sq lfor-hadoop

663 views

Published on

Hive hadoop learning hive for bigdata,Hive hadoop learning hive for bigdata

Published in: Engineering, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
663
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
43
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Hive sq lfor-hadoop

  1. 1. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved Hive: SQL for Hadoop An  Essen8al  Tool  for   Hadoop-­‐based   Data  Warehouses Tuesday, January 10, 12 I’ll  argue  that  Hive  is  indispensable  to  people  crea8ng  “data  warehouses”  with  Hadoop,  because  it  gives  them  a  “similar”  SQL  interface  to  their  data,   making  it  easier  to  migrate  skills  and  even  apps  from  exis8ng  rela8onal  tools  to  Hadoop.
  2. 2. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved Adapted from our Hadoop 3-Day Developer Course info@thinkbiganalytics.com thinkbiganalytics.com Tuesday, January 10, 12 These  notes  are  (heavily)  adapted  from  our  Hadoop  3-­‐day  developer  course.  Of  course,  we  won’t  do  exercises  in  this  talk  ;)  The  second  day  focuses  on   Hive.  We  also  offer  Hive  training  by  itself,  admin  courses,  etc.  
  3. 3. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved Programming Hive Summer, 2012 O’Reilly Tuesday, January 10, 12 I’m  collabora8ng  with  two  other  colleagues  to  write  “Programming  Hive”,  which  will  be  published  by  O’Reilly  this  summer.  Un8l  it’s  ready,  the  best  source   of  documenta8on  is  on  the  Hive  site,  hYp://hive.apache.org.
  4. 4. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved4 programmingscala.compolyglotprogramming.com/fpjava Dean Wampler Functional Programming for Java Developers Dean Wampler 20+ years exp. Think Big Academy Tuesday, January 10, 12
  5. 5. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved5 A True Story... Tuesday, January 10, 12
  6. 6. Once upon an Internet time, in the land of Enterprise, the King’s Warehouse of Data was bursting at the seams. Tuesday, January 10, 12
  7. 7. And lo, the scribes decided they could cut costs and store more data by moving to Hadoop. Tuesday, January 10, 12
  8. 8. But then the scribes complained amongst themselves, “The language of the land of Java hurts our tongues! Tuesday, January 10, 12
  9. 9. “It takes forever to say anything!” Tuesday, January 10, 12
  10. 10. There was a scribe who walked through the Kingdom-wide Labyrinth (kwl), where he found The Book of Faces. Tuesday, January 10, 12
  11. 11. In that book, he found the secret language the Bee Keepers use to query the god Hadoop! Tuesday, January 10, 12
  12. 12. And there was much rejoicing, for the scribes could work with their data again! Tuesday, January 10, 12
  13. 13. The End. Tuesday, January 10, 12
  14. 14. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved14 Enter Hive Tuesday, January 10, 12
  15. 15. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved15 Since your team knows SQL and all your DataWarehouse apps are written in SQL, Hive minimizes the effort of migrating to Hadoop. Tuesday, January 10, 12
  16. 16. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Ideal for data warehousing. • Ad-hoc queries of data. • Familiar SQL dialect. • Analysis of large data sets. • Hadoop MapReduce jobs. 16 Hive Tuesday, January 10, 12 Hive is a killer app, in our opinion, for data warehouse teams migrating to Hadoop, because it gives them a familiar SQL language that hides the complexity of MR programming.
  17. 17. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Invented at Facebook. • Open sourced to Apache in 2008. • http://hive.apache.org 17 Hive Tuesday, January 10, 12
  18. 18. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved18 A Scenario: Mining Daily Click Stream Logs Tuesday, January 10, 12
  19. 19. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved19 Ingest & Transform: • From: file://server1/var/log/clicks.log Jan 9 09:02:17 server1 movies[18]: 1234: search for “vampires in love”. … Tuesday, January 10, 12 As we copy the daily click stream log over to a local staging location, we transform it into the Hive table format we want.
  20. 20. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved20 Ingest & Transform: • From: file://server1/var/log/clicks.log Jan 9 09:02:17 server1 movies[18]: 1234: search for “vampires in love”. … Timestamp Tuesday, January 10, 12
  21. 21. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved21 Ingest & Transform: • From: file://server1/var/log/clicks.log Jan 9 09:02:17 server1 movies[18]: 1234: search for “vampires in love”. … The server Tuesday, January 10, 12
  22. 22. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved22 Ingest & Transform: • From: file://server1/var/log/clicks.log Jan 9 09:02:17 server1 movies[18]: 1234: search for “vampires in love”. … The process (“movies search”) and the process id. Tuesday, January 10, 12
  23. 23. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved23 Ingest & Transform: • From: file://server1/var/log/clicks.log Jan 9 09:02:17 server1 movies[18]: 1234: search for “vampires in love”. … Customer id Tuesday, January 10, 12
  24. 24. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved24 Ingest & Transform: • From: file://server1/var/log/clicks.log Jan 9 09:02:17 server1 movies[18]: 1234: search for “vampires in love”. … The log “message” Tuesday, January 10, 12
  25. 25. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved25 Ingest & Transform: • From: file://server1/var/log/clicks.log 09:02:17^Aserver1^Amovies^A18^A1234^Asea rch for “vampires in love”. … • To: /staging/2012-01-09.log Jan 9 09:02:17 server1 movies[18]: 1234: search for “vampires in love”. … Tuesday, January 10, 12 As we copy the daily click stream log over to a local staging location, we transform it into the Hive table format we want.
  26. 26. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved26 Ingest & Transform: 09:02:17^Aserver1^Amovies^A18^A1234^Asea rch for “vampires in love”. … • To: /staging/2012-01-09.log • Removed month (Jan) and day (09). • Added ^A as field separators (Hive convention). • Separated process id from process name. Tuesday, January 10, 12 The transformations we made. (You could use many different Linux, scripting, code, or Hadoop-related ingestion tools to do this.
  27. 27. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved27 Ingest & Transform: hadoop fs -put /staging/2012-01-09.log /clicks/2012/01/09/log.txt • Put in HDFS: • (The final file name doesn’t matter…) Tuesday, January 10, 12 Here we use the hadoop shell command to put the file where we want it in the file system. Note that the name of the target file doesn’t matter; we’ll just tell Hive to read all files in the directory, so there could be many files there!
  28. 28. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved28 Back to Hive... CREATE EXTERNAL TABLE clicks ( hms STRING, hostname STRING, process STRING, pid INT, uid INT, message STRING) PARTITIONED BY ( year INT, month INT, day INT); • Create an external Hive table: You don’t have to use EXTERNAL and PARTITIONED together…. Tuesday, January 10, 12 Now let’s create an “external” table that will read those files as the “backing store”. Also, we make it partitioned to accelerate queries that limit by year, month or day. (You don’t have to use external and partitioned together…)
  29. 29. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved29 Back to Hive... ALTER TABLE clicks ADD IF NOT EXISTS PARTITION ( year=2012, month=01, day=09) LOCATION '/clicks/2012/01/09'; • Add a partition for 2012-01-09: • A directory in HDFS. Tuesday, January 10, 12 We add a partition for each day. Note the LOCATION path, which is a the directory where we wrote our file.
  30. 30. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved30 Now,Analyze!! SELECT hms, uid, message FROM clicks WHERE message LIKE '%vampire%' AND year = 2012 AND month = 01 AND day = 09; • What’s with the kids and vampires?? • After some MapReduce crunching... … 09:02:29 1234 search for “twilight of the vampires” 09:02:35 1234 add to cart “vampires want their genre back” … Tuesday, January 10, 12 And we can run SQL queries!!
  31. 31. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • SQL analysis with Hive. • Other tools can use the data, too. • Massive scalability with Hadoop. 31 Recap Tuesday, January 10, 12
  32. 32. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved32 Hadoop Master ✲ Job Tracker Name Node HDFS Hive Driver (compiles, optimizes, executes) CLI HWI Thrift Server Metastore JDBC ODBC Karmasphere Others... Tuesday, January 10, 12 Hive queries generate MR jobs. (Some operations don’t invoke Hadoop processes, e.g., some very simple queries and commands that just write updates to the metastore.) CLI = Command Line Interface. HWI = Hive Web Interface.
  33. 33. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • HDFS • MapR • S3 • HBase (new) • Others... 33 Tables Hadoop Master ✲ Job Tracker Name Node HDFS Hive Driver (compiles, optimizes, executes) CLI HWI Thrift Server Metastore JDBC ODBC Karmasphere Others... Tuesday, January 10, 12 There is “early” support for using Hive with HBase. Other databases and distributed file systems will no doubt follow.
  34. 34. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Table metadata stored in a relational DB. 34 Tables Hadoop Master ✲ Job Tracker Name Node HDFS Hive Driver (compiles, optimizes, executes) CLI HWI Thrift Server Metastore JDBC ODBC Karmasphere Others... Tuesday, January 10, 12 For production, you need to set up a MySQL or PostgreSQL database for Hive’s metadata. Out of the box, Hive uses a Derby DB, but it can only be used by a single user and a single process at a time, so it’s fine for personal development only.
  35. 35. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Most queries use MapReduce jobs. 35 Queries Hadoop Master ✲ Job Tracker Name Node HDFS Hive Driver (compiles, optimizes, executes) CLI HWI Thrift Server Metastore JDBC ODBC Karmasphere Others... Tuesday, January 10, 12 Hive generates MapReduce jobs to implement all the but the simplest queries.
  36. 36. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Benefits • Horizontal scalability. • Drawbacks • Latency! 36 MapReduce Queries Tuesday, January 10, 12 The high latency makes Hive unsuitable for “online” database use. (Hive also doesn’t support transactions and has other limitations that are relevant here…) So, these limitations make Hive best for offline (batch mode) use, such as data warehouse apps.
  37. 37. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Benefits • Horizontal scalability. • Data redundancy. • Drawbacks • No insert, update, and delete! 37 HDFS Storage Tuesday, January 10, 12 You can generate new tables or write to local files. Forthcoming versions of HDFS will support appending data.
  38. 38. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Schema on Read • Schema enforcement at query time, not write time. 38 HDFS Storage Tuesday, January 10, 12 Especially for external tables, but even for internal ones since the files are HDFS files, Hive can’t enforce that records written to table files have the specified schema, so it does these checks at query time.
  39. 39. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • No Transactions. • Some SQL features not implemented (yet). 39 Other Limitations Tuesday, January 10, 12
  40. 40. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved40 More on Tables and Schemas Tuesday, January 10, 12
  41. 41. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved41 Data Types • The usual scalar types: •TINYINT, …, BIGNT. •FLOAT, DOUBLE. •BOOLEAN. •STRING. Tuesday, January 10, 12 Like most databases...
  42. 42. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved42 Data Types • The unusual complex types: •STRUCT. •MAP. •ARRAY. Tuesday, January 10, 12 Structs are like “objects” or “c-style structs”. Maps are key-value pairs, and you know what arrays are ;)
  43. 43. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved43 CREATE TABLE employees ( name STRING, salary FLOAT, subordinates ARRAY<STRING>, deductions MAP<STRING,FLOAT>, address STRUCT< street:STRING, city:STRING, state:STRING, zip:INT> ); Tuesday, January 10, 12 subordinates references other records by the employee name. (Hive doesn’t have indexes, in the usual sense, but an indexing feature was recently added.) Deductions is a key-value list of the name of the deduction and a float indicating the amount (e.g., %). Address is like a “class”, “object”, or “c-style struct”, whatever you prefer.
  44. 44. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved44 File & Record Formats CREATE TABLE employees (…) ROW FORMAT DELIMITED FIELDS TERMINATED BY '001' COLLECTION ITEMS TERMINATED BY '002' MAP KEYS TERMINATED BY '003' LINES TERMINATED BY 'n' STORED AS TEXTFILE; All the defaults for text files! Tuesday, January 10, 12 Suppose our employees table has a custom format and field delimiters. We can change them, although here I’m showing all the default values used by Hive!
  45. 45. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved45 Select,Where, Group By, Join,... Tuesday, January 10, 12
  46. 46. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved46 Common SQL... • You get most of the usual suspects for SELECT, WHERE, GROUP BY and JOIN. Tuesday, January 10, 12 We’ll just highlight a few unique features.
  47. 47. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved47 ADD JAR MyUDFs.jar; CREATE TEMPORARY FUNCTION net_salary AS 'com.example.NetCalcUDF'; SELECT name, net_salary(salary, deductions) FROM employees; “User Defined Functions” Tuesday, January 10, 12 Following a Hive defined API, implement your own functions, build, put in a jar, and then use them in your queries. Here we (pretend to) implement a function that takes the employee’s salary and deductions, then computes the net salary.
  48. 48. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved48 SELECT name, salary FROM employees ORDER BY salary ASC; ORDER BY vs. SORT BY • A total ordering - one reducer. SELECT name, salary FROM employees SORT BY salary ASC; • A local ordering - sorts within each reducer. Tuesday, January 10, 12 For a giant data set, piping everything through one reducer might take a very long time. A compromise is to sort “locally”, so each reducer sorts it’s output. However, if you structure your jobs right, you might achieve a total order depending on how data gets to the reducers. (E.g., each reducer handles a year’s worth of data, so joining the files together would be totally sorted.)
  49. 49. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Only equality (x = y). 49 Inner Joins SELECT ... FROM clicks a JOIN clicks b ON ( a.uid = b.uid, a.day = b.day) WHERE a.process = 'movies' AND b.process = 'books' AND a.year > 2012; Tuesday, January 10, 12 Note that the a.year > ‘…’ is in the WHERE clause, not the ON clause for the JOIN. (I’m doing a correlation query; which users searched for movies and books on the same day?) Some outer and semi join constructs supported, as well as some Hadoop-specific optimization constructs.
  50. 50. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved50 A Final Example of Controlling MapReduce... Tuesday, January 10, 12
  51. 51. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Calling out to external programs to perform map and reduce operations. 51 Specify Map & Reduce Processes Tuesday, January 10, 12
  52. 52. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved52 Example FROM ( FROM clicks MAP message USING '/tmp/vampire_extractor' AS item_title, count CLUSTER BY item_title) it INSERT OVERWRITE TABLE vampire_stuff REDUCE it.item_title, it.count USING '/tmp/thing_counter.py' AS item_title, counts; Tuesday, January 10, 12 Note the MAP … USING and REDUCE … USING. We’re also using CLUSTER BY (distributing and sorting on “item_title”).
  53. 53. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved53 Example FROM ( FROM clicks MAP message USING '/tmp/vampire_extractor' AS item_title, count CLUSTER BY item_title) it INSERT OVERWRITE TABLE vampire_stuff REDUCE it.item_title, it.count USING '/tmp/thing_counter.py' AS item_title, counts; Call specific map and reduce processes. Tuesday, January 10, 12 Note the MAP … USING and REDUCE … USING. We’re also using CLUSTER BY (distributing and sorting on “item_title”).
  54. 54. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved54 … And Also: FROM ( FROM clicks MAP message USING '/tmp/vampire_extractor' AS item_title, count CLUSTER BY item_title) it INSERT OVERWRITE TABLE vampire_stuff REDUCE it.item_title, it.count USING '/tmp/thing_counter.py' AS item_title, counts; Like GROUP BY, but directs output to specific reducers. Tuesday, January 10, 12 Note the MAP … USING and REDUCE … USING. We’re also using CLUSTER BY (distributing and sorting on “item_title”).
  55. 55. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved55 … And Also: FROM ( FROM clicks MAP message USING '/tmp/vampire_extractor' AS item_title, count CLUSTER BY item_title) it INSERT OVERWRITE TABLE vampire_stuff REDUCE it.item_title, it.count USING '/tmp/thing_counter.py' AS item_title, counts; How to populate an “internal” table. Tuesday, January 10, 12 Note the MAP … USING and REDUCE … USING. We’re also using CLUSTER BY (distributing and sorting on “item_title”).
  56. 56. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved56 Hive: Conclusions Tuesday, January 10, 12
  57. 57. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Not a real SQL Database. • Transactions, updates, etc. • … but features will grow. • High latency queries. • Documentation poor. 57 Hive Disadvantages Tuesday, January 10, 12
  58. 58. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved • Indispensable for SQL users. • Easier than Java MR API. • Makes porting data warehouse apps to Hadoop much easier. 58 Hive Advantages Tuesday, January 10, 12
  59. 59. Copyright  ©  2011-­‐2012,  Think  Big  Analy8cs,  All  Rights  Reserved59 Questions? dean.wampler@thinkbiganalytics.com thinkbiganalytics.com This presentation: github.com/deanwampler/Presentations Tuesday, January 10, 12

×