Hadoop Management Console from eBay at China Hadoop 2015

1. Hadoop Management Console
-User Account
-Share System
-Management Console
2015中国 Hadoop 技术峰会
2015.07

2. About me
Gus, Nie
Hadoop Engineer
Email: fnie@ebay.com
Oracle DBA -
Teradata DBA -
Hadoop Engineer –

3. Agenda
• Problems! Problems! Problems!
• We need Change
• HMC Solution
• Example
• Current & Next
• Q&A

4. Problems
dn2 dn3dn1
dn4 dn5
dn8 rank1dn7
dn2 dn3dn1
dn4 dn5
dn8 rank2dn7
dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn7
dn2 dn3dn1
dn4 dn5
dn8 rank3dn7
dn7
Over 2000 nodes.
Account Increase

5. Problems
• All accounts share the cluster’s resource without limit.
Storage Usage

6. Problems
• All accounts share the cluster’s resource without limit.

7. Problems
• Small files always there. Small Files
>1GB
<1MB
<1GB
<128MB
<10MB
<1MB

8. Problems
• Small files always there.
0
10000000
20000000
30000000
40000000
50000000
60000000
Small Files Accounts
>1GB
<1MB
<1GB
<128MB
<10MB

9. Problems
• Compute resource - Queues Usage.

10. Problems
• Compute resource - Queues Usage.
Account - Queue

11. We need Change
• All accounts share the cluster’s resource without limit.
• Small files always there.
• Queues Usage in a mess.

12. • All accounts share the cluster’s resource without limit.
• Small files always there.
• Queues Access in a mess.
Resource Allocation
Self-Service
Dynamic
We need Change

13. Resource Allocation
Self-Service
Dynamic
Hierarchy Queue
Storage Quota
Apply Request
Auto Approve
Real time release
Pre-define capacity
We need Change

14. HMC Solution
Ruby on Rails
MySQL
KDC
Command Console
Resource Manager
LDAP
HDFS

15. HMC Solution
HMC
Fsimage
Delimiter
File
Upload to
HDFS
Aggregation
to Dir View
Filter Quota
Data
Hadoop Nodes HDFS
Update Real
Time Data
Parse
Scheduler
Dump
Scheduler
Command
Console

16. HMC Solution
HMC
Hadoop Nodes HDFS
Remote
Scripts
Set Name
Quota
Set Space
Quota
Dump to
Scheduler
Scheduler
XML
Refresh
Scheduler
Command
Console

17. HMC Solution
LDAP
Manager • Account
• Kerberos
• Directory

18. HMC Solution
3/6TB
Space quota /
Stand File
Size
Name
Quota
Quota
Admin
Default
more

19. HMC Solution
Default
QueueLDAP

20. HMC Solution
Default
QueueLDAP
SLA Admin

21. Example
Default
Resource

22. Example

23. Current & Next
• All accounts share the cluster’s resource without limit.
Storage Usage

24. Current & Next
• All accounts share the cluster’s resource without limit.
Storage Usage

25. • Small files always there.
Small Files
>1GB
<1MB
<1GB
<128MB
<10MB
<1MB
Current & Next

26. • Small files always there.
Current & Next
Small Files
>1GB
<1MB
<1GB
<128MB
<10MB
<1MB

27. • Small files always there.
Current & Next
0
2000000
4000000
6000000
8000000
10000000
12000000
14000000
16000000
18000000
Small Files Accounts
>1GB
<1MB
<1GB
<128MB
<10MB

28. • Small files always there.
Current & Next
0
0.5
1
1.5
2
2.5
Account - Queue

29. • Time Window Scheduler.
• Dynamic mapping overwrite.
• Account full function self-service.
• ACL for HDFS
Current & Next

30. Q&A