Cloudera Hue/Beeswax

Hue/Beeswax – A Hive UI 2010.7.6 bc Wong <bcwalrus [at] cloudera.com>

10 A Demo Is Worth 2 Slides

Beeswax Server as the Hive Connector ● A proxy that supports concurrent queries ● Needed to work around HiveServer ● 617 lines of code ● Apache License v2.0

Beeswax Server Talks Thrift QueryHandle query(Query); QueryState get_state(QueryHandle); Results fetch(QueryHandle, bool start_over); ResultsMetadata get_results_metadata(QueryHandle); QueryExplanation explain(Query); string get_log(LogContextId);

A Handle for Each Query QueryHandle query(Query); QueryState get_state(QueryHandle); Results fetch(QueryHandle, bool start_over); { { string query; ResultsMetadata string id; get_results_metadata(QueryHandle); list<string> configuration; LogContextId; string user; } QueryExplanation explain(Query); list<string> groups; } string get_log(LogContextId);

Asynchronous Query Processing QueryHandle query(Query); QueryState get_state(QueryHandle); Results MAIN fetch(QueryHandle, bool start_over); THREAD handle = new QueryHandle(...); ResultsMetadata get_results_metadata(QueryHandle); query = new RunningQueryState(handle, ...); query.compile(); QueryExplanation explain(Query); <thread pool execution> return handle; WORKER THREAD Hive.closeCurrent(); string get_log(LogContextId); Hive.get(conf); driver.execute(); notify();

Hue Polls on the QueryState QueryHandle query(Query); QueryState get_state(QueryHandle); Results fetch(QueryHandle, bool start_over); ResultsMetadata get_results_metadata(QueryHandle); CREATED, INITIALIZED, QueryExplanation COMPILED, explain(Query); RUNNING, FINISHED, string EXCEPTION get_log(LogContextId);

Results Delimiter Is a Big Problem driver.getResults(); QueryHandle query(Query); QueryState get_state(QueryHandle); Results fetch(QueryHandle, bool start_over); ResultsMetadata get_results_metadata(QueryHandle); QueryExplanation explain(Query); { bool ready; list<string> columns; stringlist<string> data; get_log(LogContextId); // list of tab delimited fields int start_row; bool has_more; }

Metadata on Results for Direct Access QueryHandle query(Query); QueryState get_state(QueryHandle); Results fetch(QueryHandle, bool start_over); ResultsMetadata get_results_metadata(QueryHandle); QueryExplanation explain(Query); // Only for “ selects” { string hive_metastore.Schema; get_log(LogContextId); string results_dir; // temporary results dir string table_name; // for select * string delim; }

Logs Collected Per Query QueryHandle Query_a <-- query(Query); LogContextId_a <-- dynamic -- Thread_1 thread to ctx QueryState Query_b <-- get_state(QueryHandle); LogContextId_b <-- map -- Thread_2 Results fetch(QueryHandle, bool start_over); ResultsMetadata get_results_metadata(QueryHandle); LogDivertAppender QueryExplanation explain(Query); string get_log(LogContextId);

Cloudera Hue/Beeswax

by Carl Steinbach

Accessibility

Upload Details

Usage Rights

Statistics