The document focuses on the integration of DevOps practices with Hadoop to automate and enhance deployment processes in big data environments. It highlights the importance of continuous integration, testing, and quality management through the use of various tools and configurations, facilitating collaboration between development and operations teams. Additionally, it covers aspects of protecting enterprise assets, easing Hadoop migration, and empowering both administrators and developers with better feedback and monitoring capabilities.

1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hadoop & DevOps :
better together
Maxime Lanciaux
DevOps System Architect
February 2017

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Automate & Speed up deployment

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
Continuous integration workflow
Dev + Ops
Set of practices that emphasize the collaboration
and communication of both software developers and
information technology professionals
while automating the process of software delivery
and infrastructure changes.
Automate deployment with tools to ensure quality

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
 Continuous integration + Build machine
– Jenkins + Maven
 Configuration management tool (Hadoop + non Hadoop tools)
– Ambari + Ansible
 Configuration management (configuration, projects, Jenkins pipeline)
– Git
 Big Data Platform + Database + reporting
– HDP + PostgreSQL + Grafana + Zeppelin
 Others
– Custom log4J config, bash template, custom python scripts
Typical set of tool
You may want to use
virtualization,
containerization,
continuous inspection, code
collaboration tool as well

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
Typical Architecture
DN
DN
DN
MN
MN
Database
Reporting
Edge
Node
Edge
Node
Configuration
Management
(Tag / Branches)
Build
Automation
Continuous
integration
(Parameter, template)
Configuration and Management platform
+
data
Configuration /
workflow / tests

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
Typical Architecture
Database
Reporting
Configuration
Management
Build
Automation
Continuous
integration
Configuration and Management platform
Dev + DR
Prod + Validation
+

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
Typical Organization
 HDFS directory for Dev, Validation, Production, Backup
 Hive databases and Kafka topics with naming convention
 HBase dedicated namespace
 YARN Capacity Scheduler with corresponding queue (Dev, Validation, Production)
 Ranger rules (Project, team, environment with groups)
 All automatically created during init of project (On Dev by user, on Prod by admin)

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automate & Speed up deployment
On the Git side
 Codify everything
 Version everything
 Organize infrastructure as code

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Automate & Speed up deployment
Enforce testing and quality

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
 Use of multi-tenant (1+n) Hadoop clusters
– Which means naming convention and parameter / security
Environments / Context

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
 Use of multi-tenant Hadoop cluster
– Which means naming convention and parameter / security
Environments / Context
Dev Prex Prod Dev Dev

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
 Use of multi-tenant Hadoop cluster
– Which means naming convention and parameter / security
Environments / Context

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
 Use of multi-tenant Hadoop cluster
– Which means naming convention and parameter / security
Environments / Context

15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enforce testing and quality
 Hadoop logs
 Monitoring of full stacks component
 Application logs
 Evolution of performance
 Code review with or without tools
 Test are include in the bundle (for non-reg)
Massive use of Pareto

16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Automate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset

17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Environments
 Configuration management
 Automation
 Impact analysis
 5W
Protect your enterprise asset
Protect your enterprise asset
Your company
TMA #1 TMA #2 TMA #n
DevOps

18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Automate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Install, configure and deploy
 All test to be schedule / run
 Follow performance evolution
 Can be as easy as configuring access to a new edge node + configuration / password /
SSH / certificates / security
 Multi-component environments
Ease Hadoop migration
A lovely hand

20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Automate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration
Empower Administrator and Developer

21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Developer to get a better feedback on their application
 Administrator to check / push back best practices
 Numerous action to be completely automated (ssh)
 Focus more high level task, with direct access to information
 Get directly access to template (no need to reinvent the wheels)
Empower Administrator and Developer
Same tools, same discussion

22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Automate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration
Empower Administrator and Developer
With full monitoring for Human

23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
With full monitoring for human
Per project, per environment, per Hadoop cluster or on all of them

24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
With full monitoring for human
Per project, per environment, per Hadoop cluster or on all of them

25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
With full monitoring for human
Per project, per environment, per Hadoop cluster or on all of them

26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Automate & Speed up deployment
Enforce testing and quality
Protect your enterprise asset
Ease Hadoop migration
Empower Administrator and Developer
With full monitoring for Human
Demo

27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
We are Recruiting !
Email me :
mlanciaux@hortonworks.com

28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You