Cloudera DataTalks 2019 Bangalore - YuniKorn A next generation scheduler for YARN, K8s

© Cloudera, Inc. All rights reserved.
YuniKorn : Next Generation Scheduler for YARN, K8s &
the cloud
Vinod Kumar Vavilapalli & Sunil Govindan

© Cloudera, Inc. All rights reserved. 2
SPEAKERS
SUNIL GOVINDAN
Engineering Manager @Cloudera
@sunilgovind
Apache Hadoop PMC & Committer
Apache Hadoop since 2012
VINOD KUMAR VAVILAPALLI
Director of Engineering @Cloudera
@tshooter
Apache Hadoop VP & PMC Chair
Apache Hadoop since 2007

AGENDA
Where are we today?
Introducing YuniKorn
YuniKorn Deep Dive
Future & Open Source Story

BATCH WORKLOADS DEEP LEARNING APPS
CUSTOMER JOURNEY - BIG DATA ECOSYSTEM
BIG DATA ECOSYSTEM - TODAY
PUBLIC CLOUD
STORAGE
100100100
101001000
010010101
STORAGECOMPUTE (on-prem/on-cloud)
HIVE on LLAP
SERVICES

The New Trends
Moving to Cloud
Big Data &
Containerization
Mixed Workloads

WHERE ARE WE TODAY ?
RESOURCE ORCHESTRATOR PERSPECTIVE - STRENGTHS
APACHE YARN
Big Data Ecosystem
KUBERNETES
Cloud Ecosystem
CLOUD NATIVE
Public Cloud
Big Data—Optimized to run Big Data
workloads
Batch Workloads—High Throughput
scheduling for batch workloads
SLA—Better SLA for Big Data
workloads
Multi tenant—Quota Management.
Services—Optimized for
containerized microservices
Networking—Strong Network
management support
Cloud Aware—Better tuned for cloud
use cases
Storage—Persistent volumes
Cost—Budget Centric
∞—Inﬁnite resource for inﬁnite $$

© Cloudera, Inc. All rights reserved. 7© Cloudera, Inc. All rights reserved.
APACHE YARN WORLD
Strengthening Dynamic
Environments
• Improve YARN to work
well for cloud (public &
private)
• Focus on Autoscaling,
Smarter Scheduling etc.
Refer: YARN-9548
Improving capabilities for
persistent volumes
• Added CSI (Container
Storage Interface) support
• Enhancing CSI
implementation to expand
and support storages such
S3, Ozone etc as mounted
volume to YARN
containers
Native Service
enhancements
• Improving native
services support in
YARN
• Micro Service upgrades
What’s happening now in YARN today ?

KUBERNETES WORLD
Demand for Better support
of batch workloads
• Early efforts to support batch
scheduling is in progress by
K8s community.
Efforts on running Spark
on K8s
• Spark and Kubernetes
community is working
towards Spark on K8s
deployments.
• Gaps in running Spark
such dynamic resource
allocation, security etc is
still open.
CDP and Kubernetes
Cloudera
CDP Experiences will be
running on Kubernetes
What’s happening now in KUBERNETES today ?

AGENDA
Where are we today?
YuniKorn Deep Dive

Enter the YuniKorn!
YuniKorn (/ˈyo͞ onəˌkôrn/, Y for YARN, K for K8s, uni- for Uniﬁed)
• A common resource scheduler
• Platform independent
• Enhanced scheduling capabilities

WHAT YuniKorn IS (IS NOT) ?
YuniKorn is
• A better scheduler for the K8s world, for services and batch workloads
• A unified scheduler for the YARN world (FiFo, Fair and Capacity Scheduler)
• Providing unified resource scheduling experience across the YARN and K8s (and beyond)
• Suitable for both finite resource (datacenter) and infinite resources/dollars (cloud) worlds
Is NOT
• A system to port YARN applications to run on K8s w/o modification, or vice versa

SCHEDULING: What do we mean anyways?
On isolation, capacity allocations, scheduling
Faster!
More! Best for my cluster
Throughput
Utilization
Elasticity
Service uptime
Security
ROI
Everything! Right
now!
SLA!

WHAT IS HAPPENING NOW
CHALLENGES - APACHE YARN
APACHE YARN WORLD
• Challenges in managing TWO MAJOR YARN schedulers for different use cases
• Should Deploy & Manage containerized microservices in the same Hadoop Cluster
• Need strong networking and persistent volume support for services
• Need more powerful Auto scaling and budget control in the public cloud

WHAT IS HAPPENING NOW
CHALLENGES - KUBERNETES
KUBERNETES WORLD
• Challenges in running Big Data workloads along with microservices together
• Need much better quota management and better SLA
• No ﬁrst class Application management concept for Big Data workloads

SO HOW DO WE SOLVE THIS ?
Assessment
Native Big Data Apps — Moving
batch workloads (MR, TEZ .. ) from
YARN to Kubernetes looking into
High throughput, Low Latency and
With notion of job etc is costly.
Adaptability — Optimized for
services and running batch
workloads exposes hard-to-bridge
gaps such as run few workloads
with or without docker
Services on YARN — Services and
web farms can run on YARN
however it is not as feature-rich as
Kubernetes.
Multiple Schedulers (YARN) —
Different schedulers are focussed
on speciﬁc use cases and not very
easy to drive continuous feature
enhancements .
One cannot replace another —
Neither Kubernetes can replace
YARN or vice versa in the near future
considering some of the
fundamental architecture
differences.
EXPENSIVE
Higher cost to achieve the goal

WHERE DOES YuniKorn PLAY?
Not Optimized — to balance use-cases
like batch workloads to the needs like
running web farms or services with
respect to scheduling challenges.
Poor Resource Utilization — Not able to
effectively utilize complete resources in
cluster for services and workloads.
FRAGMENTED
Multiple YARN & Kubernetes schedulers are
UNIFIED RESOURCE SCHEDULER AND APPLICATION
MANAGEMENT
What we need is an effort to improve both YARN and
Kubernetes scheduling worlds.
Multiple schedulers power YARN & Kubernetes for different use cases

AGENDA
Where are we today?
YuniKorn Deep Dive

YuniKorn - A UNIFIED RESOURCE SCHEDULER
Capacity Planning
Capacity Planning
Divide cluster resources into resource pools (queues),
deﬁne capacity range based on needs. Enforce
resource quotas and limits.
Resource scheduling
Resource fairness, preemption, high-throughput,
multi-tenant, placement, etc.
Application Management
A central place to monitor application states
Resource Monitoring
A uniﬁed view of cluster resources, a dashboard to
easy track resource usage by queue, user or
organization.
Resource
Scheduling
Application
Management
Resource
Monitoring
...
Explore the feature set

CAPACITY PLANNING
Hierarchy of queues
Queues can be organized in honor of user groups or
organizations, with multiple levels.
Elastic Capacity
Each queue has its min-max capacity, usage is elastic
within this range for multiple users.
Resource Quotas
Resource cap for queues or users. Limited amount of
resources, number of applications etc.
Partition
A set of instances (nodes) that are physically isolated

RESOURCE SCHEDULING
Resource Fairness
Queue/User/App level fairness
ensures each entity gets its own fair
share of resources.
Priorities
Queue priority + App priority
Preemption
Queue demands for more resources
have the chance to preempt resources
from other queues for high priority
apps.
Placement Constraints
Aﬃnity/anti-aﬃnity, node constraints
etc
Services
Low Latency
Long Running
Batch
High-throughput
Short-lived
YuniKorn

RESOURCE MONITORING
Common dashboard to monitor resources
Hierarchy of
queues
Cluster resources
are divided into
hierarchy of
queues, all queue
state is visible
Common View
A common view of
resources, cross
platform.
Resource Centric
Focus on resources,
total/available/used, and
all
Resource
Dashboard

APPLICATION MANAGEMENT
Track applications in a consistent fashion
Application originated
GUI to manage
workloads (instead of
individual pods in
K8s).
Entire application
lifecycle is visible and
trackable.

ARCHITECTURE
YuniKorn Core
Scheduler
Shim
Master
Scheduler Interface
Api-server etcd
kubelets
Resource
Manager
Node Managers
GPRC/API
Resource requests, new application,
node updates
GPRC/API
Container allocation, preemption
Master Node
Slave nodes
MR
Spark
Flink
Tez
MySQL
Spark
Web
Server
Kafka
Client API
Allocate, release container

YuniKorn SCHEDULER vs. OTHERS
Disclosure: this table is summarized based on speakers’ analysis
Scheduler
Capabilities
Resource Sharing
Resource
Fairness Preemption Throughput Gang Scheduling
Hierarchy
queues
Queue
prioritoy
Queue elastic
capacity
Cross queue
fairness
User level
fairness
App level
fairness
Basic
preemption With fairness With priority
Kube-default x x x x x x v x v 100+ allocs/s x
Kube-batch x x x x x v v x v ? v
YARN CS/FS v v v v v v v v v 4k+ allocs/s x
YuniKorn v x (wip) v v v v v v x (wip) ? x (wip)
Key capabilities of a resource scheduler from our perspective

AGENDA
Where are we today?
YuniKorn Deep Dive

YARN cluster
TAKE AWAY
BEFORE AFTER
Scenario 1
Scenario 2
Scenario 3
K8s cluster on cloud/prem
Existing K8s
schedulers
K8s cluster on cloud/prem
YuniKorn
scheduler
K8s cluster
Existing K8s
schedulers
YARN cluster
Capacity
Scheduler
Fair
Scheduler
K8s cluster
YuniKorn Scheduler
YARN cluster
Capacity
Scheduler
Fair
Scheduler
YARN cluster
YuniKorn
scheduler

OPEN SOURCE
Yes, YuniKorn is now open source https://github.com/cloudera/yunikorn-core!
Contributions welcome!
Join us in slack!

ACKNOWLEDGMENTS
A big shout out to the folks who helped to design, develop and make this project
possible.
❏ Wangda Tan
❏ Weiwei Yang
❏ Wilfred Spiegelenburg
❏ Akhil PB
❏ Suma Shivaprasad
❏ and many others...

Cloudera DataTalks 2019 Bangalore - YuniKorn A next generation scheduler for YARN, K8s

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Cloudera DataTalks 2019 Bangalore - YuniKorn A next generation scheduler for YARN, K8s

Similar to Cloudera DataTalks 2019 Bangalore - YuniKorn A next generation scheduler for YARN, K8s (20)

Recently uploaded

Recently uploaded (20)

Cloudera DataTalks 2019 Bangalore - YuniKorn A next generation scheduler for YARN, K8s