In shared configuration, the same Hadoop environment supports many applications. Each has specific requirements and criticality (SLA). Yet they all rely on an assembly of shared application bricks. At the same time, the life cycle of a cluster is not static in time. It evolves horizontally, with the arrival of new applications, but also vertically, as the applications grow in load or evolve in terms of functionality. With this in mind, a multi-tenant production cluster presents several challenges including and not limited to: - Maintain a high level of SLA for a set of use cases with heterogeneous needs - Plan and implement the architecture evolution of a cluster in production to ensure the maintenance of SLA throughout the integration of new use cases on it EDF will present how it manages this heterogeneity of SLA inherent of any Big Data cluster. EDF is focusing on how it is renovating its cluster, its organization, its processes and its approach in order to deliver a platform with strong SLA throughout its life cycle. Speaker Edouard Rousseaux, Tech Lead, EDF

1. | 1
MULTI-TENANT HADOOP
THE CHALLENGE OF
MAINTAINING HIGH SLAS
Edouard ROUSSEAUX
EDF-DTEO-DSIT-ITO-DATACENTER
Big Data Tech Lead

2. | 2
SUMMARY
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
1. EDF CONTEXT
PRESENTATION
2. BIG DATA STRATEGY OF EDF-DSIT (IT PRODUCTION DEPARTMENT)
HISTORY
ARCHITECTURE CHOICES
BIG DATA SERVICE OFFER
3. CHALLENGES TO TAKE UP
CURRENT STATE
DIAGNOSTIC
FOCUS
4. ACTION PLAN
TECHNICAL POINTS
ORGANIZATIONAL POINTS

3. | 3
EDF CONTEXT
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
WORLD LEADER IN LOW-CARBON ENERGY, EDF GROUP BRINGS TOGETHER ALL THE BUSINESS OF
PRODUCTION, TRADE AND ELECTRICITY NETWORKS.
EDF-DSIT PROVIDES IT-SERVICES TO SUPPORT
THE GROUP IN ITS DIGITAL TRANSFORMATION

4. | 4
BIG DATA STRATEGY OF EDF-DSIT
A SHARED DATALAKE
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
ARCHITECTURE CHOICES
Shared Platform for all Group businesses:
- Centralization of data
- Economic efficiency
- Sharing / Cross Analyzing data between Business
- Simplify operations
Specific platform per type of use cases in
order to guaranty SLA, performance and flexibility:
- Development / Pre-production
- Production (mainly used as a backend of production
app)
- Backup / Disaster Recovery
- Analytics (soon)
These architectural choices have a very strong
impact on the performance of our infrastructures and
applications
HISTOIRY OF BIG DATA AT EDF
We start in 2012 with a first cluster (Hadoop v1)
- Trade Direction wanted to start cross analyzing data
- 4 recycled hosts
… exponential growth …
Until today :
- 3 physical environments (4th one soon)
- We are using Hortonworks (HDP, HDF)
- 200 hosts
- HDFS ≈ 1,4 PB (usable space)
- YARN ≈ 14,6TB of RAM / 4600 vCores
- HBASE ≈ 8,2TB of RAM
- Biggest HBASE table ≈ 90TB (8k Regions)

5. | 5
BIG DATA STRATEGY OF EDF-DSIT
BIG DATA SERVICE OFFER
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
AN OFFER THAT FOUND ITS PUBLIC
− 5 Business Directions 50 production applications
− +10k Yarn jobs per day (in production)
− 250 HBase Tables / 25k Hbase Regions /
− 150+ Hive DB
− +500 users
− 24/7 applications
− 1 HA applications
− All kind of application type:
• Batch (ELT)
• Streaming / Real-time
• OLAP
• OLTP
• Big volumes / Small volumes
• Critical / Non-critical applications
« BIG DATA » SERVICE OFFER
Business oriented offer :
- A price catalog
- Very simple units of work : TB et vCores
- Global SLA on shared services available on our clusters
(HDFS, HBASE, KAFKA,…)
- Organization, process,…
« self-service » consumption by trades
The design of this service offer also has a strong impact on
the performance of infrastructures and applications
OUR BIG DATA INFRASTRUCTURES HAS BECOME ESSENTIAL TO OUR BUSINESSES

6. | 6
CHALLENGES TO TAKE UP
AN OFFER « VICTIM » OF ITS OWN SUCCESS
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
SITUATION LAST SUMMER
Many critical use cases for businesses
Hard time maintaining the expected level of service :
- Instability / Unavailability
- Difficulty to Communicate
99.5
100 100 100
99.35
99.86
99.1
99
96
97.44
98.85
99.8599.8
99.9
96.3
100
99.2
99.88
100 100
99.6
100 100 100
95
96
97
98
99
100
101
Juin Juillet Août Septembre Octobre Novembre
Availability of services in 2017
Hive Hbase Hdfs Yarn

7. | 7
CHALLENGES TO TAKE UP
DIAGNOSTIC
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
TECHNICAL ISSUES
Improve the way we operate our shared Big Data Infrastructures:
- Insufficient Metrics
- Monitoring didn’t evolve as fast as application types
- Complex diagnostics
- Sizing anticipation were not accurate
Some applications were developed with anti-patterns
Lack of rigor when putting applications in production
- Scale-out and Performance tests
- No code review
Internal billing based on storage and CPU (insufficient) :
- Does not accurately reflect cluster usage
- Does not encourage a virtuous use of clusters
NEED FOR FURTHER TECHNICAL &
PRODUCTION SUPPORT

8. | 8
CHALLENGES TO TAKE UP
DIAGNOSTIC
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
ORGANIZATIONAL ISSUES
« Self-Service » Offer:
- Partial control of what runs on infrastructure
- Lack of a global vision of clusters usage
SLA not accurate enough:
- No concept of degraded service
- Feeling not equal between business and operation teams
Improvement of our production skills
- Shift in skill type required
Shared infrastructure:
- Governance challenge (change management, ...)
- Businesses Use cases impact each other
- Accurate resources allocation is essential (according to needs)
Capacity Planning :
- Not sufficiently detailed
- Not easy to quantify the potential impact of business use cases
(Restricted vision / Sizing)
NEED MORE COORDINATION BETWEEN TEAMS

9. | 9
CHALLENGES TO TAKE UP
IN HINDSIGHT
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
EVOLVING CLUSTER USAGE
Until Summer 2017 : Only « ELT » usage
New type of business use cases :
- Intensive use of HBASE
- Huge volumes
- Critical use cases
Lack of anticipation :
- Impact of those new use cases not qualified
- More than 1000 Regions par Region Servers
- HDFS saturation
- HBASE collapsed
99.5
100 100 100
99.35
99.86
99.1 99
96
97.44
98.85
99.8599.8 99.9
96.3
100
99.2
99.88
100 100
99.6
100 100 100
95
96
97
98
99
100
101
Juin Juillet Août Septembre Octobre Novembre
Availability of services 2017
Hive Hbase Hdfs Yarn
COMPELING EVENT FOR CHANGE

10. | 10
ACTION PLAN
TECHNICAL POINTS
EDITOR SUPPORT
Expertise
- Diagnostic aid
- Improving operating aspect of infrastructure
- Metrics
- Tools
- Sizing / Scaling
- Improvement of application ops
Internal skill improvement
Communication
- Facilitate internal communication between teams
- Best practices
- Shared Diagnostics
- Facilitate dialogue with management
- Investment decisions
- Technical evolution decisions
RESOURCES ISOLATION / SHARING
Fine management of Yarn queues :
- Queues per Business Directions
- Sub-queues per applications or use cases ?
- Overbooking ? Pre-emption ?
Resources Isolation :
- Node labels
- Region Server groups
- Containerization of Elasticsearch/SolR
Protection mechanism of resources :
- HDFS Quotas
- HBASE Quotas
- Kafka Quotas
- Zookeeper Observers
Evaluate multi-tenancy to match business requirements and
enterprise strategy.

11. | 11
ACTION PLAN
CONFIGURATION & ARCHITECTURE
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018

12. | 12
ACTION PLAN
SERVICES AVAILABILITY
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018

13. | 13
ACTION PLAN
MONITORING – HBASE TABLE
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018

14. | 14
ACTION PLAN
MONITORING – HBASE REGION SERVER
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018

15. | 15
ACTION PLAN
MONITORING – KAFKA
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018

16. | 16
ACTION PLAN
MONITORING – HBASE REPLICATION (EVOLUTION OF BAD ROWS)
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018

17. | 17
ACTION PLAN
ORGANIZATION
COMMUNICATION
Build and animate Enterprise Communities
- Templates / GIT,…
- Feedback Sharing
Improve Cross team communication
Share Development guides
Create shared metrics
- Dashboards
- Health Check
- Application Status

18. | 18
ACTION PLAN
ORGANIZATION
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
MANAGEMENT
Reorganization of Application Management :
- Centralized team
- Global vision of jobs
- Centralization of application logs
- Tools
CAPACITY PLANNING
Setup of an « industrial » Capacity Planning
- Definition of the right metrics to predict
- Tools
- Regural meetup to review the planning
SERVICE OFFER
Modification of Billing to reflect actual usage
- Memory and CPU Usage for HBASE
- Number of Region
- Markup for small files
MANAGE EXPECTATIONS & ANTICIPATE
REQUIREMENTS

19. | 19
ACTION PLAN
POSITIVE EFFECTS
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
99.5
100 100 100
99.35
99.86
100 100 100
99.1
99
96
97.44
98.85
99.85
100 100
99.899.8
99.9
96.3
100
99.2
99.88
100 100 100100 100
99.6
100 100 100 100 100 100
95
96
97
98
99
100
101
Juin Juillet Août Septembre Octobre Novembre Décembre Janvier Février
Availability of services
Hive Hbase Hdfs Yarn

20. | 20
THING TO TAKE-AWAY
Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
Service OFFER & Multi-tenancy must be managed globally
Communication & coordination at all level is essential
Get help !!

21. | 21Multi-tenant Hadoop - The challenge of maintaining high SLAS | april 2018
THANK YOU !
QUESTIONS