1) The document discusses rolling upgrades for Hadoop clusters, which allow upgrading services and components while the cluster remains active. 2) It outlines key issues that must be addressed like data safety, service degradation, and maintaining application contexts across versions. 3) It describes enhancements made to components like HDFS, YARN, Hive, and packaging to enable rolling upgrades while minimizing disruption.

1. Page1 © Hortonworks Inc. 2015
Enterprise-Grade Rolling Upgrade for a Live
Hadoop Cluster
Sanjay Radia, Vinod Kumar Vavilapalli
Hortonworks Inc
June 9, 2015

2. Page2 © Hortonworks Inc. 2015
Agenda
•Introduction
•What is Rolling Upgrade?
•Problem – Several key issues to be addressed
–Wire compatibility and side-by-side installs are not sufficient!!
–Must Address: Data safety, Service degradation and disruption
•Enhancements to various components
–Packaging – side-by-side install
–HDFS, YARN, Hive, Oozie, …

3. Page3 © Hortonworks Inc. 2015
Sanjay Radia
•Chief Architect, Founder, Hortonworks
•Part of the Hadoop team at Yahoo! since 2007
–Chief Architect of Hadoop Core at Yahoo!
–Apache Hadoop PMC and Committer
• Prior
–Data center automation, schedulers, virtualization, Java, HA, OSs, File
Systems
– (Startup, Sun Microsystems, Inria …)
–Ph.D., University of Waterloo

4. Page4 © Hortonworks Inc. 2015
Vinod Kumar Vavilapalli
– Long time Hadooper since 2007
– Apache Hadoop Committer / PMC
– Apache Member
– Yahoo! -> Hortonworks
– MapReduce -> YARN from day one

5. Page5 © Hortonworks Inc. 2015
HDP Upgrade: Two Upgrade Modes
Stop the Cluster Upgrade
Shutdown services and cluster and then upgrade.
Traditionally this was the only way
Rolling Upgrade
Upgrade cluster and its services while cluster is
actively running applications
Note: Upgrade time is proportional to # nodes, not data size
Enterprises run critical services and data on a Hadoop cluster.
Need live cluster upgrade that maintains SLAs without degradation

6. Page6 © Hortonworks Inc. 2015
But you can also “Revert to Prior State”
Rollback
Revert bits and state of cluster and its services back to a
checkpoint’d state.
Why? This is an emergency procedure.
Downgrade
Downgrade the service and component to prior version, but
keep any new data and metadata that has been generated
Why? You are not happy with performance, or app compatibility, ….

7. Page7 © Hortonworks Inc. 2015
But aren’t wire compatibility and
side-by-side installs sufficient for
Rolling upgrades?
Unfortunately No!! Not if you want
• Data safety
• Keep running jobs/apps during upgrades; continue to run
correctly
• Maintain SLAs
• Allow downgrade/rollbacks in case of problems

8. Page8 © Hortonworks Inc. 2015
Issues that need to be addressed (1)
• Data safety
• HDFS’s upgrade checkpoint does not work for rolling upgrade
• Service degradation – note every daemon is restarted in rolling fashion
• HDFS write pipeline
• Application Masters on YARN restart
• NodeManagers restart
• Hive server is processing client queries – it cannot restart to new version without loss
• Client must not see failures – many components do not have retry
BUT Hadoop deals with failures, it will fix pipelines, restart tasks –
what is the big deal!!
Service degradation will be high because every daemon is restarted

9. Page9 © Hortonworks Inc. 2015
Issues that need to be addressed (2)
• Maintaining the application submitter’s context (correctness)
• MR tasks get their context from the local node
– In the past the submitters and node’s context were identical
– But with RU, a node’s binaries are being upgraded and hence may be inconsistent with submitter
- Half of the job could execute with old binaries and the other with the new one!!
• Persistent state
• Backward compatibility for upgrade (or convert)
• Forward compatibility for downgrade (or convert)
• Wire compatibility
• With clients (forward and backward)
• Internally (Between Masters and Slaves or Peers)
– Note: the upgrade is in a rolling fashion

10. Page10 © Hortonworks Inc. 2015
Component Enhancements
• Packaging – Side-by-side installs
• HDFS Enhancements
• YARN Enhancements
• Retaining Job/App Context
• Hive Enhancements

11. Page11 © Hortonworks Inc. 2015
Packaging: Side-by-side Installs (1)
• Need side-by-side installs of multiple versions on same node
• Some components are version N, while others are N+1
• For same component, some daemons version N, others N+1 on the same node (e.g. NN and DN)
• HDP’s solution: Use OS-distro standard packaging solution
• Rejected proprietary packing as a solution (no lock-in)
• Want to support RU via Ambari and Manually
• Standard packaging solutions like RPMs have useful tools and mechanisms
– Tools to install, uninstall, query, etc
– Manage dependencies automatically
– Admins do not need to learn new tools and formats
• Side benefits for ‘stop-the-world” upgrade:
• Can install the new binaries before the shutdown

12. Page12 © Hortonworks Inc. 2015
Packaging: Side-by-side installs (2)
• Layout: side-by-side
• /usr/hdp/2.2.0.0/hadoop
• /usr/hdp/2.2.0.0/hive
• /usr/hdp/2.3.0.0/hadoop
• /usr/hdp/2.3.0.0/hive
• Define what is current for each component’s
daemon and clients
• /usr/hdp/current/hdfs-nn->/usr/hdp/2.3.0.0/hadoop
• /usr/hdp/current/hadoop-client->/usr/hdp/2.2.0.0/hadoop
• /usr/hdp/current/hdfs-dn->/usr/hdp/2.2.0.0/hadoop
• Distro-select helps you manage the version switch
• Our solution: the package name contains the version number:
• E.g hadoop_2_2_0_0 is the RPM package name itself
– Hadoop_2_3_0_0 is different peer package
• Bin commands point to current:
/usr/bin/hadoop->/usr/hdp/current/hadoop-client/bin/hadoop

13. Page13 © Hortonworks Inc. 2015
Packaging: Side-by-side installs (3)
• distro-select tool to select current binary
• Per-component, Per-daemon
• Maintain stack consistency – that is what QE tested
• Each component refers to its siblings of same stack version
• Each component knows the “hadoop home” of the same stack
– Wrapper bin-scripts set this up
• Config updates can be optionally synchronized with binary upgrade
• Configs can sit in their old location
• But what if the new binary version requires slightly different config?
• Each binary version has its own config pointer
– /usr/hdp/2.2.0.0/hadoop/conf -> /etc/hadoop/conf

14. Page14 © Hortonworks Inc. 2015
Component Enhancements
• Packaging – Side-by-side installs
• HDFS Enhancements
• YARN Enhancements
• Retaining Job/App Context
• Hive Enhancements

15. Page15 © Hortonworks Inc. 2015
HDFS Enhancements (1)
Data safety
• Since version 2007, HDFS supported an upgrade-checkpoint
• Backups of HDFS not practical – too large
• Protects against HDFS bugs in new version deleting files
– Standard practice to use for ALL upgrade even patch releases
• But this only works for “stop-the-world” full upgrade and does not support downgrade
• Irresponsible to do rolling upgrade without such a mechanism
HDP 2.2 has enhanced upgrade-checkpoint (HDFS-5535)
• Markers for rollback
• “Hardlinks” to protect against deletes due to bugs in the new version of HDFS code
– Old scheme had hardlinks but we now delay the deletes
• Added downgrade capability
• Protobuf based fsImage for compatible extensibility

16. Page16 © Hortonworks Inc. 2015
HDFS Enhancements (2)
Minimize service degradation and retain data safety
• Fast datanode restart (HDFS-5498)
• Write pipeline – every DN will be upgraded and hence many write
pipelines will break and repaired
• Umbrella Jira HDFS-5535
– Repair it to the same DN during RU (avoid replica data copy)
– Retain same number of replicas in pipeline
• Upgrade HA standby and failover (NN HA available for a long time)

17. Page17 © Hortonworks Inc. 2015
Component Enhancements
• Packaging – Side-by-side installs
• HDFS Enhancements
• YARN Enhancements
• Retaining Job/App Context
• Hive Enhancements

18. Page18 © Hortonworks Inc. 2015
YARN Enhancements: Minimize Service Degradation
• YARN RM retains application queue (2013)
• YARN RM fail-over (2014)
– Note this retains the queues but ALL jobs are rekicked
• YARN RM can restart while retaining applications (2015)

19. Page19 © Hortonworks Inc. 2015
YARN Enhancements: Minimize Service Degradation
• A restarted YARN NodeManager retains existing containers (2015)
• Recall: restarting containers will cause serious SLA degradation

20. Page20 © Hortonworks Inc. 2015
YARN Enhancements: Compatibility
• Versioning of state-stores of RM and NMs
• Compatible evolution of tokens over time
• Wire compatibility between mixed versions of RM

21. Page21 © Hortonworks Inc. 2015
Component Enhancements
• Packaging – Side-by-side installs
• HDFS Enhancements
• YARN Enhancements
• Retaining Job/App Context
• Hive Enhancements

22. Page22 © Hortonworks Inc. 2015
Retaining Job/App context
• Previously a Job/Apps used libraries from the local node
• Worked because client-node & compute-nodes had same version
• But during RU, the NodeManager has multiple versions
• Must use the same version as used by the client when submitting a job
• Solution:
• Framework libraries are now installed in HDFS
• Client-context sent as “distro-version” variable in job config
• Has side benefits
– Frameworks now installed in single node and then uploaded to HDFS
• Note Oozie also enhanced to maintain consistent context

23. Page23 © Hortonworks Inc. 2015
YARN Rolling Upgrades: A Cluster Snapshot

24. Page24 © Hortonworks Inc. 2015
Component Enhancements
• Packaging – Side-by-side installs
• HDFS Enhancements
• YARN Enhancements
• Retaining Job/App Context
• Hive Enhancements

25. Page25 © Hortonworks Inc. 2015
Hive Enhancements
• Fast restarts + client-side reconnection
• Hive metastore and Hive client
• Hive-server2: stateful server that submits the client’s query
• Need to keep it running till the old queries complete
• Solution:
• Allow multiple Hive-servers to run, each registered in Zookeeper
• New client requests go to new servers
• Old server completes old queries but does not receive any new ones
– Old-server is removed from Zookeeper
• Side benefits
• HA + Load balancing solution for Hiveserver2

26. Page26 © Hortonworks Inc. 2015
Automated Rolling Upgrade
Via Ambari
Via Your own cluster management scripts

27. Page27 © Hortonworks Inc. 2015
HDP Rolling Upgrades Runbook
Pre-requisites
• HA
• Configs
Prepare
• Install bits
• DB backups
• HDFS
checkpoint
Rolling Upgrade Finalize
Rolling
Downgrade
Rollback
NOT Rolling. Shutdown all
services.
Note: Upgrade time is proportional to # nodes, not data size

28. Page30 © Hortonworks Inc. 2015
Both Manual and Automated Rolling Upgrade
• Ambari supports fully automated upgrades
• Verifies prerequisites
• Performs HDFS upgrade-checkpoint, prompts for DB backups
• Performs rolling upgrade
• All the components, in the right order
• Smoke tests at each critical stages
• Opportunities for Admin verification at critical stages
• Downgrade if you change your mind
• Have published the runbook for those that do not use Ambari
• You can do it manually or automate your own process

29. Page31 © Hortonworks Inc. 2015
Runbook: Rolling Upgrade
Ambari has automated
process for Rolling Upgrades
Services are switched over to
new version in rolling fashion
Any components not installed
on cluster are skipped
Zookeeper
Ranger
Core Masters
Core Slaves
Hive
Oozie
Falcon
Clients
Kafka
Knox
Storm
Slider
Flume
Hue
Finalize
HDFS, YARN, MR,
Tez, HBase, Pig.
Hive, Phoenix,
Mahout
HDFS
YARN
HBase

30. Page32 © Hortonworks Inc. 2015
Runbook: Rolling Downgrade
Zookeeper
Ranger
Core Masters
Core Slaves
Hive
Oozie
Falcon
Clients
Kafka
Knox
Storm
Slider
Flume
Hue
Downgrade
Finalize

31. Page33 © Hortonworks Inc. 2015
Summary
• Enterprises run critical services and data on a Hadoop cluster.
• Need a live cluster upgrade without degradation and maintaining SLAs
• We enhanced Hadoop components for enterprise-grade rolling upgrade
• Non-proprietary packaging solution using OS-standard solution (RPMs, Debs, )
• Data safety
– HDFS checkpoints and write-pipelines
• Maintain SLAs – solve a number of service degradation problems
– HDFS write pipelines, Yarn RM, NM state recovery, Hive, …
• Jobs/apps continue to run correctly with the right context
• Allow downgrade/rollbacks in case of problems
• All enhancements truly open source and pushed back to Apache?
• Yes of course – that is how Hortonworks does business …

32. Page34 © Hortonworks Inc. 2015
Backup slides

33. Page35 © Hortonworks Inc. 2015
Why didn’t you use alternatives
• Alternatives generally keep one version active, not two
• We need to move some services as a pack (clients)
• We need to support managing confs and binaries together and
separately
• Maybe we could have done it, but it was getting complex …..