Hadoop distributions can be combination of 25+ open source projects. Enterprise adoptions have various kinds of workloads, environments with vectors like Operating systems, JDK, Database, Security, Ranger Authorization, Encryption, TDE and so on. Ensuring quality for a complex stack and the combinations can be overwhelming. In this talk we will cover details of technologies involved in automated validation of the stack. Our testing journey begins with ingestion of commits from apache and meets the finish line as we GA the stack distribution. As we speak about this journey, we will walk through how quality is established at various stages like commit, nightly testing, pre prod and readiness. We will go over the challenges we face as we cater to several releases of major, maintenance, hot-fixes all at the same time , how we tackled them with the YARN on YARN infrastructure, using test methodologies to bring efficiencies and how LOG AI comes to the rescue. We will conclude with talk with a case study of end to end workflow test Speaker Sunitha Velpula, Director of Engineering Quality, Hortonworks

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Quality for the Hadoop Zoo
Sunitha Velpula
Director of Engineering, Quality

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Agenda
• The Hadoop Stack
• Test Certification Matrix
• Stages: commit to ship
• What can possibly break?
• Challenges and Solutions

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Hortonworks Data Platform

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
Platform Matrix !

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
• Unit tests
• Smoke Tests
• Nightly Automated Tests
• Pre-Prod
• Long Running Readiness
Typical release
Stages Commit to Ship!

6. 6 © Hortonworks Inc. 2011–2018. All rights reserved
What can possibly
break?

7. 7 © Hortonworks Inc. 2011–2018. All rights reserved
• Functional - Regressions
• Yarn Scheduling
• QueueACLS
• Fault tolerant like agent or Node manager failure
• Config Groups
• Lineage in Atlas
• SafeMode
• Snapshots
• New features
• **This List is a small subset and example only
What can possibly break?

8. 8 © Hortonworks Inc. 2011–2018. All rights reserved
• Reliability
• True HA test example scenario : Test Namenode fail's over works
• # get the active namenode
• # get the standby namenode
• # suspend the active namenode process
• #Validate that failover happened and there is a new active namenode
What can possibly break?

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
• Long Running Workflows
• Zeppelin - Run UI test for creation of notebooks and query with JDBC and LIVY
• Hive tpc-ds, tpc-h with 1TB for weeks finds issues like
• Task container timeouts with long GC pauses
• AM containers killed
• Tez AM goes out of memory
• So on…
What can possibly break?

10. 10 © Hortonworks Inc. 2011–2018. All rights reserved
• Upgrade Types
• Ambari Upgrades
• Stack upgrades
• Express Upgrade
• Rolling Upgrade , Rolling Downgrade
• Patch upgrades
• Challenge : The combination monster !
• Test example:
• Long running application + RU + validate LR , version switch
• Tolerance : Kill the agents on slaves
All done, can you guarantee an upgrade works?

11. 11 © Hortonworks Inc. 2011–2018. All rights reserved
• Challenges:
• Open-source Ingestion
• Compute
• Failure Analysis
• Rescue
• Quality at commit as well as rebase
• YARN on YARN
• Smart Certify
• Log AI
Challenges and Solutions

12. 12 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?

13. 13 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you