This document provides a comprehensive guide on configuring a Hive high availability connection with Zeppelin, including instructions for setting up a JDBC connection with Kerberos authentication, and integrating with Zookeeper. It outlines the architecture enabling high availability, load balancing, and concurrency while detailing specific configurations required in the Hive server settings. Additionally, it lists necessary JDBC jars and debugging tips to troubleshoot potential connection issues between Zeppelin and Hive.

1. How-to configure a Hive High
Availability connection with
Zeppelin
Hive + Zookeeper + Kerberos + Zeppelin

2. Introduction
With this presentation, you not only should be able to configure a Hive Interpreter on
Zeppelin but also with a High Availability, Load balancing and Concurrency architecture.
It will be created a JDBC connection with kerberos authentication that will communicate
with your Zookeeper on the cluster.

3. Architecture
This architecture enables the following:
● High Availability & Load balancing
● Concurrency on your Data Model
● Access data without locks
● Direct access to Hive Metadata

4. Pre-Assumptions
1. Cluster hostname: cm1.localdomain Zeppelin hostname: cm2.localdomain
2. Cluster & Zeppelin Linux user: zeppelin Kerberos Principal: zeppelin/cm1.localdomain/@DOMAIN.COM
3. Cluster Cloudera Manager: Cloudera Manager 5.12.2
4. Zookeeper: Installed
5. Cluster Authentication Pre-Installed: Kerberos
a. Kerberos Realm DOMAIN.COM
6. Zeppelin version: 0.7.13
7. Zeppelin Machine Authentication: Kerberos

5. Base of Knowledge
JDBC Connections - Hive / Hive + Zookeeper / Impala / Impala (Hive) - with and without Kerberos
JDBC Conn Security URL
Hive None jdbc:hive2://server:10000/db_name
Hive Kerberos
(static)
jdbc:hive2://server:10000/db_name;principal=hive/server@REALM.COM
Hive Kerberos
(user)
jdbc:hive2://server:10000/db_name;principal=hive/server@REALM.COM;auth=kerberos;kerberosAuthType=fromSubject
Hive +
Zookeeper
None jdbc:hive2://zookeeper_quorum:2181/db_name;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Hive +
Zookeeper
Kerberos
(user)
jdbc:hive2://zookeeper_quorum:2181/db_name;principal=hive/server@REALM.COM;serviceDiscoveryMode=zooKeeper;zooKee
perNamespace=hiveserver2
Impala None jdbc:impala://server:21050;authMech=0
Impala
(Hive)
Kerberos jdbc:hive2://server:21050/db_name;principal=impala/server@REALM.COM
Impala Kerberos jdbc:impala://server:21050;AuthMech=1;KrbRealm=REALM.COM;KrbHostFQDN=server.example.com;KrbServiceName=impala

6. Configuring HiveServer2 High Availability
Confirm Hive interaction with Zookeeper is active
In Cloudera Manager on Zookeeper configurations ensure that hive
acknowledge is active.
Confirm Zookeeper authentication with Sentry is active
In Cloudera Manager on Zookeeper configurations ensure that
Zookeeper authentication is made via Sentry.
Note 1: On these flags it will be used the standards (ex: Zookeeper Port 2181).
Note 2: Due to the H.A. in HiveServer2 it's important to enable the lock manager support for concurrency to avoid locks on the metadata.
Enable Zookeeper Dynamic Discovery for HiveServer2 HA
To enable this service it will be necessary to edit Hive Service Advanced
Configuration Snippet (Safety Valve) for hive-site.xml and add the
following Flags:
● Hive.server2.zookeeper.namespace
● Hive.zookeeper.client.port
● Hive.zookeeper.quorum
● Hive.server2.support.dynamic.service.discovery
● Hive.cluster.delegation.token.store.class
● hive.support.concurrency

7. Configuring HiveServer2 High Availability
Hive Service Advanced Configuration Snippet - Part 1
<property>
<name>hive.server2.zookeeper.namespace</name>
<value>hiveserver2</value>
<description>hive.server2.zookeeper.namespace</description>
</property>
<property>
<name>hive.zookeeper.client.port</name>
<value>2181</value>
<description>hive.zookeeper.client.port</description>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>cm1.localdomain:2181</value>
<description>hive.zookeeper.quorum</description>
</property>
Note 1: on the flags zookeeper.namespace and zookeeper.client.port it was given the defaults values.
Note 2: on the flag zookeeper.quorum for this example it was only configured one host, but you can and should add more ex:
<value>cm1.zk_1,cm1.zk_2,cm1.zk_3</value>
Note 3: just need to change the highlighted values for your own values.

8. Configuring HiveServer2 High Availability
Hive Service Advanced Configuration Snippet - Part 2
<property>
<name>hive.server2.support.dynamic.service.discovery</name>
<value>true</value>
<description>hive.server2.support.dynamic.service.discovery</description>
</property>
<property>
<name>hive.cluster.delegation.token.store.class</name>
<value>org.apache.hadoop.hive.thrift.DBTokenStore</value>
<description>hive.cluster.delegation.token.store.class</description>
</property>
<property>
<name>hive.server2.enable.impersonation</name>
<value>true</value>
<description>hive.server2.enable.impersonation</description>
</property>
Note: with dynamic service discovery active will adapted to every new user query in round robin fashion so that all hiveserver2 instances would get
the same load and it is always recommended to use dynamic service discovery in production to avoid HiveServer2 crashes.

9. Configuring HiveServer2 High Availability
Hive Service Advanced Configuration Snippet - Part 3
<property>
<name>hive.support.concurrency</name>
<description>Enable Hive's Table Lock Manager Service</description>
<value>true</value>
</property>
The final JDBC string should be something like this:
jdbc:hive2://<hive.zookeeper.quorum>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Additional point, due to the concurrency on the Cluster and depending on your configuration on the HiveServer2 and
HiveServer1, they can be binding to the same port (10000) , therefore at least one of them has to be changed, example:
<property>
<name>hive.server2.thrift.port</name>
<value>10001</value>
<description>TCP port number to listen on, default 10000</description>
</property>
Note 1: Due to the configuration of H.A. in HiveServer2 it's important to enable the lock manager support for concurrency to avoid locks on the
metadata.
Note 2: just need to change the highlighted values for your own values.

10. Configuring HiveServer2 High Availability
Hive Service Advanced Configuration Snippet - Part 4

11. After all configuration is done, we are going to confirm and test that HiveServer2 High Availability is active.
Connect to Zookeeper
zookeeper-client
List the contents
ls /
Note: If the result is empty please confirm that the configuration for
hive.server2.support.dynamic.service.discovery is active.
Check the configurations
ls /hiveserver2
Note: just need to change the highlighted values for your own values.
Configuring HiveServer2 High Availability
Deregister a particular HiveServer2 instance
delete
/hiveserver2/serverUri=cm1.localdomain:10000;version=1.1.
0-cdh5.12.2;sequence=0000000014
Deregister all HiveServer2 instances
hive --service hiveserver2 --deregister 1.1.0-
cdh5.12.2;sequence=0000000014

12. Before configure the JDBC Interpreter on Zeppelin we must try the connection string on the cluster.
Kinit with Zeppelin Kerberos user
kinit -kt zeppelin.keytab zeppelin/cm1.localdomain@DOMAIN.COM
Test Connection 2º
show databases;
Configuring HiveServer2 High Availability
Test Connection 1º
beeline -u 'jdbc:hive2://cm1.localdomain:2181/db_name;
principal=hive/cm1.localdomain@DOMAIN.COM;serviceDiscoveryMode=zooKe
eper;zooKeeperNamespace=hiveserver2'

13. Hive Memory Configuration
One additional question to be considered is the number of the concurrent connections to Hive, because this will
affect massively the performance of the jobs and obviously the current resources on the Cluster. Therefore the
recommendations from Cloudera are:

14. Preparation to the Interpreter Configuration
Zookeeper JDBC required Jars
Curator-client-2.7.1.jar
Curator-framework-2.7.1.jar
Curator-recipes-2.7.1.jar
Curator-x-discovery-2.7.1.jar
zookeeper-3.4.12.jar
Zeppelin JDBC interpreter brings already some libs, but to configure the Hive + Zookeeper JDBC you will require some
additional jars. These additional jars should be present in every machine (inclusive in Zeppelin machine).
Hive JDBC required Jars
Commons-dbcp2-2.0.1.jar
Commons-logging-1.1.3.jar
Commons-pool2-2.2.jar
Hadoop-auth-2.6.0-cdh5.12.2.jar
Hadoop-common-2.6.0-cdh5.12.2.jar
Hive-jdbc-1.1.0-cdh5.12.2.jar
Hive-jdbc-1.1.0-cdh5.12.2-standalone.jar
Hive-metastore-1.1.0-cdh5.12.2.jar
Hive-service-1.1.0-cdh5.12.2.jar
Hsqldb-1.8.0.10.jar
Libfb303-0.9.3.jar
Libthrift-0.9.3.jar
Log4j-1.2.17.jar
Slf4j-api-1.7.10.jar
Slf4j-log4j12-1.7.10.jar
xercesImpl-2.10.0.jar
Xml-apis-1.4.01.jar

15. Preparation to the Interpreter Configuration
Copy Hadoop Jar's to the Zeppelin JDBC interpreter directory
cd /opt/cloudera/parcels/CDH-5.12.2-1.cdh5.12.2.p0.4/jars/
scp commons*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp curator*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp hadoop-auth*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp hadoop-common*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp hive-jdbc*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp hive-metastore*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp hive-service*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp xercesImpl*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp xml-apis*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc
scp zookeeper*.jar zeppelin@10.222.33.111:/home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc

16. Preparation to the Interpreter Configuration
Confirm the expected Jars in JDBC interpreter folder
ls /home/zeppelin/zeppelin-0.7.3-bin-all/interpreter/jdbc

17. Configure the Hive Interpreter
In the Zeppelin there isn’t any specific Hive JDBC interpreter, therefore we must use the available interpreter group
- JDBC - and reconfigure it.
Create a Hive Interpreter

18. Configure the Hive Interpreter
default.driver
org.apache.hive.jdbc.HiveDriver
default.user (Optional)
zeppelin
zeppelin.interpreter.output.limit
102400
zeppelin.jdbc.auth.type
kerberos
zeppelin.jdbc.concurrent.use
true
zeppelin.jdbc.principal
zeppelin/cm1.localdomain@DOMAIN.COM
default.url (without Kerberos)
jdbc:hive2://cm1.localdomain:2181/db_name;serviceDiscov
eryMode=zooKeeper;zooKeeperNamespace=hiveserver2
default.url (with Kerberos)
jdbc:hive2://cm1.localdomain:2181/db_name;principal=hive
/cm1.localdomain@DOMAIN.COM;serviceDiscoveryMode=z
ooKeeper;zooKeeperNamespace=hiveserver2
zeppelin.jdbc.auth.kerberos.proxy.enable
true
zeppelin.jdbc.concurrent.max_connection
10
zeppelin.jdbc.keytab.location
/home/zeppelin/zeppelin.keytab

19. The expected final result for your Hive JDBC interpreter configurations should be something like this:
Note: just need to change the highlighted values on the previous slide for your own values.
Configure the Hive Interpreter

20. Testing the Hive Interpreter
Just use it like you were using
the HUE from Cloudera Manager.

21. Debug
Due to the intermediate layer that Kerberos is making in all request/authentication process, there can be some issues to
configure the Hive JDBC connection on the Zeppelin, there's even a open bug - Jira Ticket (Zeppelin) due to the transport of
ticket Kerberos on the authentication between Zeppelin JDBC Hive connection with the cluster (HiveServer2).
Start Zookeeper Client on Zookeeper Quorum host with
SERVER_JVMFLAGS="-Djavax.security.auth.useSubjectCredsOnly=false" bin/zkServer start
Additional Java Debug Flags
-Dsun.security.krb5.debug=true
On JDBC Connection String add the following Flags
;LogLevel=6;LogPath=/tmp
Note: just need to change the highlighted values for your own values.The value 6, enables logging with TRACE level - all driver activity.

22. Thanks
Big Data Engineer
Tiago Simões