Cloudera Impala

  • 4,025 views
Uploaded on

Slides for presentation on Cloudera Impala I gave at the Near Infinity (www.nearinfinity.com) 2013 spring conference.

Slides for presentation on Cloudera Impala I gave at the Near Infinity (www.nearinfinity.com) 2013 spring conference.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • @DavidGroozman Yes I guess technically it's cluster metadata, not database/table metadata. E.g. it contains the metadata about which impalad's are currently in the cluster. Plus, when I made that presentation I was using Impala 0.6, in which the impalad processes read metadata from the statestore at startup, and in 0.6 that was the reason you had to REFRESH whenever you made DDL changes in Hive, e.g. if you created a table in Hive you needed to REFRESH in the impala shell in order to see it. And in 0.6 you could only create tables in Hive but in 0.7+ you can do it from impala directly.
    Are you sure you want to
    Your message goes here
  • @Scottleber. I got this impression from the slide number 31
    Are you sure you want to
    Your message goes here
  • @DavidGroozman Yes you are correct, in that the statestore uses the Hive metastore for database and table information. I think my slides indicate that on slides 36 & 38, don't they? Is there somewhere where I contradict myself, or where it's wrong? Let me know and I will correct the slide, especially as I need to update them for when I give this talk again at the Nothern Virginia Java Users Group in mid July.

    References:

    Here's also a link to Cloudera's documentation on the statestore: http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_concepts.html?scroll=intro_statestore_unique_1

    And, here's a few paragraphs later where the docs detail how Impala works with Hive:

    http://www.cloudera.com/content/cloudera-content/cloudera-docs/Impala/latest/Installing-and-Using-Impala/ciiu_concepts.html#intro_hive_unique_1
    Are you sure you want to
    Your message goes here
  • In best of my understanding StateStore is for the cluster state information. Metadata is left in the Hive metastore. Please correct me if I am wrong.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
4,025
On Slideshare
0
From Embeds
0
Number of Embeds
5

Actions

Shares
Downloads
148
Comments
4
Likes
8

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. ScottLeberknightClouderas
  • 2. Historylesson...
  • 3. Google Map/Reducepaper (2004)Cutting & Cafarellacreate Hadoop (2005)
  • 4. Google Dremel paper (2010)Facebook creates Hive (2007)*
  • 5. Cloudera announces Impala(October 2012)HortonWorks Stinger(February 2013)Apache Drill proposal(August 2012)
  • 6. * Hive => "SQL on Hadoop"Write SQL queriesTranslate into Map/Reduce job(s)Convenient & easyHigh-latency (batch processing)
  • 7. What is Impala?In-memory, distributed SQLquery engine (no Map/Reduce)Native code (C++)Distributed(on HDFS data nodes)
  • 8. Why Impala?Interactive data analysisLow-latency response(roughly, 4-100x Hive)Deploy on existing Hadoop clusters
  • 9. Why Impala? (contd)Data stored in HDFS avoids......duplicate storage...data transformation...moving data
  • 10. Why Impala? (contd)SPEED!
  • 11. Overviewimpalad daemon runs on HDFS nodesQueries run on "relevant" nodesSupports common HDFS file formatsstatestored, uses Hive metastore(for database metadata)
  • 12. Overview (contd)Does not use Map/ReduceNot fault tolerant !(query fails if any query on any node fails)Submit queries via Hue/BeeswaxThrift API, CLI, ODBC, JDBC (future)
  • 13. SQL SupportSELECTProjectionUNIONINSERT OVERWRITEINSERT INTOORDER BY(w/ LIMIT)AggregationSubqueries(uncorrelated)JOIN (equi-join only,subject to memorylimitations)(subset of Hive QL)
  • 14. HBase QueriesMaps HBase tables via Hivemetastore mappingRow key predicates => start/stop rowNon-row key predicates => SingleColumnValueFilterHBase scan translations:
  • 15. (Very) Unscientific Benchmarks
  • 16. 9 queries, run in Impala Demo VMMacbook Pro Retina, mid 201216GB RAM,4GB for VM (VMWare 5),Intel i7 2.6GHz quad-core processorHardwareNo other load on system during queriesPseudo-cluster + Impala daemons
  • 17. Benchmarks (contd)(from simple projection queries tomultiple joins, aggregation, multiplepredicates, and order by)Impala vs. Hive performance"TPC-DS" sample dataset(http://www.tpc.org/tpcds/)
  • 18. Query "A"selectc.c_first_name,c.c_last_namefrom customer climit 50;
  • 19. Query "B"select   c.c_first_name,   c.c_last_name,   ca.ca_city,   ca.ca_county,   ca.ca_statefrom customer c   join customer_address caon c.c_current_addr_sk = ca.ca_address_sklimit 50;
  • 20. Query "C"select   c.c_first_name,   c.c_last_name,   ca.ca_city,   ca.ca_county,   ca.ca_statefrom customer c   join customer_address caon c.c_current_addr_sk = ca.ca_address_skwhere lower(c.c_last_name) like smi%limit 50;
  • 21. Query "D"select distinct cd_credit_ratingfrom customer_demographics;
  • 22. Query "E"select   cd_credit_rating,   count(*)from customer_demographicsgroup by cd_credit_rating;
  • 23. Query "F"select   c.c_first_name,   c.c_last_name,   ca.ca_city,   ca.ca_county,   ca.ca_state,   cd.cd_marital_status,   cd.cd_education_statusfrom customer c   join customer_address ca       on c.c_current_addr_sk = ca.ca_address_sk   join customer_demographics cd       on c.c_current_cdemo_sk = cd.cd_demo_skwhere   lower(c.c_last_name) like smi% and   cd.cd_credit_rating in (Unknown, High Risk)limit 50;
  • 24. Query "G"select   count(c.c_customer_sk)from customer c   join customer_address ca       on c.c_current_addr_sk = ca.ca_address_sk   join customer_demographics cd       on c.c_current_cdemo_sk = cd.cd_demo_skwhere   ca.ca_zip in (20191, 20194) and   cd.cd_credit_rating in (Unknown, High Risk);
  • 25. Query "H"select   c.c_first_name,   c.c_last_name,   ca.ca_city,   ca.ca_county,   ca.ca_state,   cd.cd_marital_status,   cd.cd_education_statusfrom customer c   join customer_address ca       on c.c_current_addr_sk = ca.ca_address_sk   join customer_demographics cd       on c.c_current_cdemo_sk = cd.cd_demo_skwhere   ca.ca_zip in (20191, 20194) and   cd.cd_credit_rating in (Unknown, High Risk)limit 100;
  • 26. select    i_item_id,  s_state,  avg(ss_quantity) agg1,  avg(ss_list_price) agg2,  avg(ss_coupon_amt) agg3,  avg(ss_sales_price) agg4from store_salesjoin date_dim   on (store_sales.ss_sold_date_sk = date_dim.d_date_sk)join item   on (store_sales.ss_item_sk = item.i_item_sk)join customer_demographics   on (store_sales.ss_cdemo_sk = customer_demographics.cd_demo_sk)join store   on (store_sales.ss_store_sk = store.s_store_sk)where  cd_gender = M and  cd_marital_status = S and  cd_education_status = College and  d_year = 2002 and  s_state in (TN,SD, SD, SD, SD, SD)group by  i_item_id,  s_stateorder by  i_item_id,  s_statelimit 100;Query "TPC-DS"
  • 27. Query Hive (sec) # M/R jobs Impala (sec) x Hive perf.A 12.4 1 0.21 59B 30.9 1 0.37 84C 29.6 1 0.33 91D 22.8 1 0.60 38E 22.5 1 0.52 44F 66.4 2 1.56 43G 83.0 3 1.33 62H 66.1 2 1.50 44TPC-DS 248.3 6 3.05 82(remember, unscientific...)
  • 28. Architecture
  • 29. Two daemonsimpaladstatestoredimpalad on each HDFS data nodestatestored - metadataThrift APIs
  • 30. impaladQuery executionQuery coordinationQuery planning
  • 31. impaladQuery CoordinatorQuery PlannerQuery ExecutorHDFS DataNodeHBase RegionServer
  • 32. Queries performed in-memoryIntermediate data never hits disk!Data streamed to clientsC++runtime code generationintrinsics for optimizationExecution engine:
  • 33. statestoredCluster membershipMetadata handling(scheduled for GA release)Not a SPOF(single point of failure)
  • 34. MetadataShares Hive metastoreDaemons cache metadataPush to cluster via statestored(scheduled for GA release)Create tables in Hive(then REFRESH impalad)
  • 35. Next up - how queries work...
  • 36. impaladQuery CoordinatorQuery PlannerQuery ExecutorHDFS DataNodeHBase RegionServerClient Statestore Hive Metastoretablemetadatatablemetadata(cached)SQLqueryimpaladQuery CoordinatorQuery PlannerQuery ExecutorHDFS DataNodeHBase RegionServerimpaladQuery CoordinatorQuery PlannerQuery ExecutorHDFS DataNodeHBase RegionServer
  • 37. Read directly from diskShort-circuit readsBypass HDFS DataNode(avoids overhead of HDFS API)
  • 38. impaladQuery CoordinatorQuery PlannerQuery ExecutorHBaseRegionServerHDFSDataNodeLocal FilesystemReaddirectlyfrom disk
  • 39. Current Limitations(as of beta version 0.6)No join order optimizationNo custom file formats or SerDes or UDFsLimit required when using ORDER BYJoins limited by memory of single node(at GA, aggregate memory of cluster)
  • 40. Current Limitations(as of beta version 0.6)No advanced data structures(arrays, maps, json, etc.)No DDL (do in Hive)Limited file formats (text, sequencew/ snappy/gzip compression)
  • 41. Future - GA & beyond...Structure types (structs,arrays, maps, json, etc.)DDL supportAdditional file formats &compression supportColumnar format(Parquet?)"Performance"Metadata(via statestore)JDBCJoin optimization(e.g. cost-based)UDFs
  • 42. Comparing...
  • 43. Dremel is a scalable, interactive ad-hocquery system for analysis of read-onlynested data. By combining multi-levelexecution trees and columnar data layout, itis capable of running aggregation queriesover trillion-row tables in seconds. Thesystem scales to thousands of CPUs andpetabytes of data, and has thousands ofusers at Google.Comparing Impala to Dremel- http://research.google.com/pubs/pub36632.html
  • 44. Comparing Impala to DremelImpala = Dremel features circa 2010 + joinsupport, assuming columnar data format(but, Google doesnt stand still...)Dremel is production, matureBasis for Googles BigQuery
  • 45. Comparing Impala to HiveHive uses Map/Reduce -> high latencyImpala is in-memory, low-latency query engineSacrifices fault tolerance forperformance
  • 46. Comparing Impala to OthersStingerApache DrillImprove Hive performance (e.g. optimize execution plan)Based on DremelIn very early stages...Support for analytics (e.g. OVER clause, window functions)TEZ framework to optimize executionColumnar file format
  • 47. ReviewIn-memory, distributedSQL query engineIntegrates intoexisting HDFSNot Map/ReduceFocus is onperformance(native code)
  • 48. ReferencesGoogle Dremel - http://research.google.com/pubs/pub36632.htmlApache Drill - http://incubator.apache.org/drill/TPC-DS dataset - http://www.tpc.org/tpcds/Stinger Initiative - http://hortonworks.com/blog/100x-faster-hive/Cloudera Impala resourceshttp://university.cloudera.com/onlineresources/introductionimpala.htmlCloudera Impala: Real-Time Queries in Apache Hadoop, For Realhttp://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/
  • 49. Photo AttributionsImpala - http://www.flickr.com/photos/gerardstolk/5897570970/Measuring tape - http://www.morguefile.com/archive/display/24850Bridge frame - http://www.morguefile.com/archive/display/9699Balance - http://www.morguefile.com/archive/display/93433* All others are iStockPhoto (paid)
  • 50. My Infoscott dot leberknight at nearinfinity dot comtwitter.com/sleberknight www.sleberknight.com/blogwww.nearinfinity.com/blogs/scott_leberknight/all/scott dot leberknight at gmail dot com