Downloaded 25 times
![Lots
of
Commodity
Machines
12
Image:Yahoo! Hadoop cluster [ OSCON ’07 ]](https://image.slidesharecdn.com/impalahadooppdx-140725001959-phpapp02/85/Cloudera-Impala-12-320.jpg)
Uploaded byAlex Moundalexis
Cloudera Impala
The document discusses Cloudera and its offerings related to Hadoop, focusing on Cloudera Impala, a modern open-source SQL query engine for Apache Hadoop. It highlights the capabilities of Impala for fast, interactive SQL queries and contrasts it with traditional MapReduce processing. Additionally, it covers the architecture, design choices, and performance improvements of Impala compared to other querying methods in the Hadoop ecosystem.
Cloudera Impala
- 1. 1 Cloudera Impala Portland Big Data User Group, July 2014 Alex Moundalexis @technmsg
- 2. Thirty Seconds About Alex • SoluGons Architect • aka consultant • government • infrastructure • former coder of Perl • former administrator • fan of Portland 2
- 3. What Does Cloudera Do? • product • distribuGon of Hadoop components, Apache licensed • enterprise tooling • support • training • services (aka consulGng) • community 3
- 4. Disclaimer • Cloudera builds things soPware • most donated to Apache • some closed-‐source • Cloudera “products” I reference are open source • Apache Licensed • source code is on GitHub • hVps://github.com/cloudera 4
- 5. What This Talk Isn’t About • deploying • Puppet, Chef, Ansible, homegrown scripts, intern labor • sizing & tuning • depends heavily on data and workload • coding • unless you count XML or CSV or SQL • algorithms 5
- 6.
- 7. CC BY-‐SA Lilian De Cassai
- 8. cloud·∙e·∙ra im·∙pal·∙a 8 /kloudˈi(ə)rə imˈpalə/ noun a modern, open source, MPP SQL query engine for Apache Hadoop. “Cloudera Impala provides fast, ad hoc SQL query capability for Apache Hadoop, complemenGng tradiGonal MapReduce batch processing.”
- 9. 9 Quick and dirty, for context. The Apache Hadoop Ecosystem
- 10. Why “Ecosystem?” • In the beginning, just Hadoop • HDFS • MapReduce • Today, dozens of interrelated components • I/O • Processing • Specialty ApplicaGons • ConfiguraGon • Workflow 10
- 11. HDFS • Distributed, highly fault-‐tolerant filesystem • OpGmized for large streaming access to data • Based on Google File System • hVp://research.google.com/archive/gfs.html 11
- 12. Lots of Commodity Machines 12 Image:Yahoo! Hadoop cluster [ OSCON ’07 ]
- 13. MapReduce (MR) • Programming paradigm • Batch oriented, not realGme • Works well with distributed compuGng • Lots of Java, but other languages supported • Based on Google’s paper • hVp://research.google.com/archive/mapreduce.html 13
- 14.
- 15. You specify map()and reduce() functions. The framework does the rest. 60
- 16. Apache Hive • AbstracGon of Hadoop’s Java API • HiveQL “compiles” down to MR • a “SQL-‐like” language • Eases analysis using MapReduce 16
- 17. Apache Hive Metastore • Maps HDFS files to DB-‐like resources • Databases • Tables • Column/field names, data types • Roles/users • InputFormat/OutputFormat 17
- 18. WHY DO WE NEED THIS? But wait… 18
- 19.
- 20. 20 I am not a SQL wizard by any means… Super Shady SQL Supplement
- 21. A Simple RelaGonal Database name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011 21
- 22. InteracGng with RelaGonal Data name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011 22 SELECT * FROM people;
- 23. InteracGng with RelaGonal Data name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011 23 SELECT * FROM people;
- 24. RequesGng Specific Fields name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011 24 SELECT name, state FROM people;
- 25. RequesGng Specific Fields name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011 25 SELECT name, state FROM people;
- 26. RequesGng Specific Rows name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011 26 SELECT name, state FROM people WHERE year 2012;
- 27. RequesGng Specific Rows name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011 27 SELECT name, state FROM people WHERE year 2012;
- 28. Two Simple Tables owner species name Alex Cactus Marvin Joey Cat Brain Sean None Paris Unknown 28 name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011
- 29. Joining Two Tables owner species name Alex Cactus Marvin Joey Cat Brain Sean None Paris Unknown 29 SELECT people.name AS owner, people.state AS state, pets.name AS pet FROM people LEFT JOIN pets ON people.name = pets.owner name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011
- 30. Joining Two Tables owner species name Alex Cactus Marvin Joey Cat Brain Sean None Paris Unknown 30 SELECT people.name AS owner, people.state AS state, pets.name AS pet FROM people LEFT JOIN pets ON people.name = pets.owner name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011
- 31. Joining Two Tables owner species name Alex Cactus Marvin Joey Cat Brain Sean None Paris Unknown 31 SELECT people.name AS owner, people.state AS state, pets.name AS pet FROM people LEFT JOIN pets ON people.name = pets.owner name state employer year Alex Maryland Cloudera 2013 Joey Maryland Cloudera 2011 Sean Texas Cloudera 2013 Paris Maryland AOL 2011
- 32. Joining Two Tables 32 SELECT people.name AS owner, people.state AS state, pets.name AS pet FROM people LEFT JOIN pets ON people.name = pets.owner owner state pet Alex Maryland Marvin Joey Maryland Brain Sean Texas Paris Maryland
- 33. Varying ImplementaGon of JOIN 33 SELECT people.name AS owner, people.state AS state, pets.name AS pet FROM people LEFT JOIN pets ON people.name = pets.owner owner state pet Alex Maryland Marvin Joey Maryland Brain Sean Texas ? Paris Maryland ?
- 34. 34 Familiar interface, but more powerful. Cloudera Impala
- 35. Cloudera Impala • InteracGve query on Hadoop • think seconds, not minutes • Nearly ANSI-‐92 standard SQL • compaGble with HiveQL • NaGve MPP query engine • built for low-‐latency queries 35
- 36. Cloudera Impala – Design Choices • NaGve daemons, wriVen in C/C++ • No JVM, no MapReduce • Saturate disks on reads • Uses in-‐memory HDFS caching • Re-‐uses Hive metastore • Not as fault-‐tolerant as MapReduce 36
- 37. Cloudera Impala – Architecture • Impala Daemon • runs on every node • handles client requests • handles query planning execuGon • State Store Daemon • provides name service • metadata distribuGon • used for finding data 37
- 38. Impala Query ExecuGon 38 Query Planner Query Coordinator Query Executor HDFS DN HBase SQL App ODBC Hive Metastore HDFS NN Statestore Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase SQL request 1) Request arrives via ODBC/JDBC/HUE/Shell
- 39. Impala Query ExecuGon 39 Query Planner Query Coordinator Query Executor HDFS DN HBase SQL App ODBC Hive Metastore HDFS NN Statestore Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase 2) Planner turns request into collecPons of plan fragments 3) Coordinator iniPates execuPon on impalad(s) local to data
- 40. Impala Query ExecuGon 40 Query Planner Query Coordinator Query Executor HDFS DN HBase SQL App ODBC Hive Metastore HDFS NN Statestore Query Planner Query Coordinator Query Executor HDFS DN HBase Query Planner Query Coordinator Query Executor HDFS DN HBase 4) Intermediate results are streamed between impalad(s) 5) Query results are streamed back to client Query results
- 41. Cloudera Impala – Results • Allows for fast iteraGon/discovery • How much faster? • 3-‐4x faster on I/O bound workloads • up to 45x faster on mulG-‐MR queries • up to 90x faster on in-‐memory cache 41
- 42. 42 Hold onto something, folks. Demo
- 43. What’s Next? • Download Hadoop! • CDH available at www.cloudera.com • Already done that? Contribute… • Cloudera provides pre-‐loaded VMs • hVp://Gny.cloudera.com/quickstartvm • Clone our repos! • hVps://github.com/cloudera 43
- 44.
- 45. 45 Preferably related to the talk… or not. QuesGons?
- 46. 46 Thank You! Alex Moundalexis @technmsg We’re hiring, kids! Well, not kids.