This document summarizes the process of upgrading an HDP cluster from version 2.1 to 2.4. It describes installing HDP 2.1.5 without Ambari, then upgrading to HDP 2.3.4 using Ambari. Several issues were encountered with Hive on Tez and MR and worked around by changing configurations, increasing memory limits, and switching between Tez and MR. HDP 2.4.0 was then installed which resolved some issues but a block corruption problem remained. The conclusion recommends using HDP 2.4.0 and monitoring upgrades due to ongoing query incompatibility issues.

1. Upgrading
from HDP2.1 to HDP2.4
2016/03/24
@wyukawa
HadoopSCR #hadoopreading

2. About 1 year ago
hFp://www.slideshare.net/tagomoris/upgrading-from-hdp-21-to-hdp-22

3. 2014/12-
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Input data
fluentd-plugin-webhdfs
shib
Execute queries
over hiveserver2 on MR
hive batch on MR

4. HDP2.1.5.0
• Install over Ansible, without Ambari
• Hadoop 2.4.0
• YARN ResourceManager HA
• NameNode HA
• Hive 0.13

5. HDP2.3.4
• Install over Ambari but use Ansible
• Hadoop 2.7.1
– YARN ResourceManager REST API(include kill)
– shib use yarn command to kill on HDP2.1.5.0
• Hive 1.2.1
– backport Hive-12084 Hive queries with ORDER BY
and large LIMIT fails with OutOfMemoryError Java
heap space.
• Tez 0.7.0

6. change Hadoop/Hive sebng by ambari
• dfs.datanode.failed.volumes.tolerated=1
• hive.server2.authendcadon=NOSASL
• hive.server2.enable.doAs=false
• hive.auto.convert.join=false
• change memory size because of presto and fluentd
– increase hadoop/datanode/resourcemanager/
hiveserver2 heap
– decrease yarn.nodemanager.resource.memory-
mb and yarn.scheduler.maximum-allocadon-mb

7. blue green deployment
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Input data
fluentd-plugin-webhdfs
shib
Execute queries
over hiveserver2 on MR
MRv2/Tez/HDFS
Hive
Analysis4(HDP2.3.4)
hive batch on Tez hive batch on MR

8. OOME occured at Hive on Tez
• report as HIVE-13238
– increase hive.tez.container.size
• report as HIVE-13239
Switch to hive batch on MR!
I have the reason to use MR because of shib status feature...

9. Why do I use shib on MR?
• can execute Hive on Tez
– set hive.execudon.engine=tez
• can’t monitor query status
• can’t kill query
• shib use mapreduce.job.name to idendfy
submiFed query

10. OMME occured at Hive on MR
• report as HIVE-13273
– hiveserver2 memory leak

11. need to upgrade HDP2.4
hFps://community.hortonworks.com/quesdons/20116/
logfdscacheflushdmer-thread-increase.html

12. HDP2.4.0
• almost same as HDP2.3.4 except for Kala,
Spark
• backport HIVE-12766 !!!!!!!!!!!!!!!!!!!!!!!!!
• AMBARI-14389

13. upgrading
from HDP2.3.4 to HDP2.4.0
• report as AMBARI-15418
I want hortonworks to fix this issue.
But, anyway, switch to hive batch on Tez!

14. hiveserver2 is stable!!!

15. 2016/3-
MRv2/HDFS
Hive
Analysis3(HDP2.1)
Input data
fluentd-plugin-webhdfs
shib
Execute queries
over hiveserver2 on MR
MRv2/Tez/HDFS
Hive
Analysis4(HDP2.4.0)
hive batch on Tez hive batch on MR

16. somedmes corrupt/missing block...
monitoring with grafana, prometheus, hadoop_exporter

17. Conclusion
• Don’t use HDP2.3.4 if you use hiveserver2
• use HDP2.4.0
• see Apache patch informadon
• I want to see HDP source repository
• somedmes need to change query(HIVE-9390)
• I want Hive commiFer