1.
Apache YuniKorn (incubating):
A Big Milestone
Thursday, 18 November 2021
Wilfred Spiegelenburg

2.
© 2020 Cloudera, Inc. All rights reserved. 2
AGENDA
Welcome
Apache YuniKorn (incubating): A Big Milestone
Gang Scheduling
Kubernetes Support
Performance Testing
Future Development

3.
© 2020 Cloudera, Inc. All rights reserved. 3
AGENDA
Welcome
Apache YuniKorn (incubating): A Big Milestone
Gang Scheduling
Kubernetes Support
Performance Testing
Future Development

4.
© 2020 Cloudera, Inc. All rights reserved. 4
Apache YuniKorn (incubating): A Big Milestone
In progress release:
• Apache YuniKorn v0.12
• Renamed from v1.0.0
• Planned release date: December 2021
Next release:
• Apache YuniKorn v1.0.0
• Planned release date: February 2022
• Kubernetes dependency and support

5.
© 2020 Cloudera, Inc. All rights reserved. 5
What to expect in the next release ?
Apache YuniKorn v0.12:
• Kubernetes support added for: v1.20 and v1.21
• Node sorting improvement
• Gang scheduling enhancements
• Supportability improvements
• Scheduler Interface update
• Kubernetes dependency upgrade to v1.20
• Bug fixes

6.
© 2020 Cloudera, Inc. All rights reserved. 6
AGENDA
Welcome
Apache YuniKorn (incubating): A Big Milestone
Gang Scheduling
Kubernetes Support
Performance Testing
Future Development

7.
© 2020 Cloudera, Inc. All rights reserved. 7
Gang
• Predefined set of resource requests
• Defined as part of the application
• Guaranteed resources: all or nothing
Advantages of gang scheduling
• Prevent starvation: FIFO ordering of applications
• Predictable performance
• Improved autoscaling
FEATURE DEEP DIVE: Gang Scheduling
Background

8.
© 2020 Cloudera, Inc. All rights reserved. 8
FEATURE DEEP DIVE: Gang Scheduling
Spark Example: no gang
YuniKorn
Spark Application
Driver
executor
executor
executor

9.
© 2020 Cloudera, Inc. All rights reserved. 9
FEATURE DEEP DIVE: Gang Scheduling
Spark Example: gang
YuniKorn
Spark Application
Driver
gang: 3 executors
executor
executor
executor
gang 1
gang 2
gang 3

10.
© 2020 Cloudera, Inc. All rights reserved. 10
AGENDA
Welcome
Apache YuniKorn (incubating): A Big Milestone
Gang Scheduling
Kubernetes Support
Performance Testing
Future Development

11.
© 2020 Cloudera, Inc. All rights reserved. 11
YuniKorn v0.12.0:
• Support K8s 1.19, 1.20 &
1.21
• Dependency K8s 1.20
YuniKorn v1.0.0:
• Support K8s . . .
• Dependency K8s 1.22
– Deprecated APIs
– Protobuf and gRPC changes
• Scheduler Framework
KUBERNETES SUPPORT
Dependency vs Support

12.
© 2020 Cloudera, Inc. All rights reserved. 12
AGENDA
Welcome
Apache YuniKorn (incubating): A Big Milestone
Gang Scheduling
Kubernetes Support
Performance Testing
Future Development

13.
© 2020 Cloudera, Inc. All rights reserved. 13
PERFORMANCE TESTING
YUNIKORN-842
Last published:
• based on v0.8.0 (YUNIKORN-33)
• Published result (website)
• Run by: Alibaba
New results:
• Based on master (YUNIKORN-842)
• Publishing in progress
• Run by: National Taichung University of Education

14.
© 2020 Cloudera, Inc. All rights reserved. 14
Scheduler Throughput
• 20,000 / 50,000 pods
• 2000 / 4000 nodes
PERFORMANCE TESTING
Preliminary results
YuniKorn compared to Default
• More than 3x faster
• Predicate optimisation possible
for even better performance

15.
© 2020 Cloudera, Inc. All rights reserved. 15
AGENDA
Welcome
Apache YuniKorn (incubating): A Big Milestone
Gang Scheduling
Kubernetes Support
Performance Testing
Future Development

16.
© 2020 Cloudera, Inc. All rights reserved. 16
New functionality and improvements:
• Kubernetes dependency v1.22
– Indirect dependencies
• Admission controller enhancements
• Improved recovery stability
• Scheduler framework (tech preview)
FUTURE DEVELOPMENT
Planned v1.0.0

17.
Thank You