SlideShare a Scribd company logo
RESOURCE
AWARE
SCHEDULING IN
APACHE STORM
Presented by Boyang Jerry Peng
2
ABOUT ME
• Apache Storm Committer and PMC member
• Member of the Yahoo’s low latency Team
 Data processing solutions with low latency
• Graduate student @ University of Illinois, Urbana-Champaign
 Research emphasis in distributed systems and stream processing
• Contact:
 jerrypeng@yahoo-inc.com
3
AGENDA
•Overview of Apache Storm
•Problems and Challenges
•Introduction of Resource Aware
Scheduler
•Results
4
OVERVIEW
• Apache Storm is an open source distributed real-time data stream processing
platform
 Real-time analytics
 Online machine learning
 Continuous computation
 Distributed RPC
 ETL
5
STORM TOPOLOGY
• Processing can be represented as a directed graph
• Spouts are sources of information
• Bolts are operators that process data
6
DEFINITIONS OF STORM TERMS
• Stream
 an unbounded sequence of tuples.
• Component
 A processing operator in a Storm
topology that is either a Bolt or Spout
• Executors
 Threads that are spawned in worker
processes that execute the logic of
components
• Worker Process
 A process spawned by Storm that may
run one or more executors.
7
STORM ARCHITECTURE
Master
Node
Cluster
Coordination
Worker
processes
Worker
Nimbus
Zookeeper
Zookeeper
Zookeeper
Supervisor
Supervisor
Supervisor
Supervisor Worker
Worker
Worker
Launches
workers
8
LOGICAL VS PHYSICAL CONNECTION IN STORM
9
OVERVIEW OF SCHEDULING IN STORM
• Default Scheduling Strategy
 Naïve round robin scheduler
 Naïve load limiter (Worker Slots)
• Multitenant Scheduler
 Default Scheduler with multitenant capabilities (supported by
security)
 Can allocate a set of isolated nodes for topology (Soft
Partitioning)
Resource Aware
10
RUNNING STORM AT YAHOO - CHALLENGES
• Increasing heterogeneous clusters
 Isolation Scheduler – handing out dedicated machines
• Low cluster overall resource utilization
 Users not utilizing their isolated allocation very well
• Unbalanced resource usage
 Some machines not used, others over used
• Per topology scheduling strategy
 Different topologies have different scheduling needs (e.g. constraint based
scheduling)
11
RUNNING STORM AT YAHOO – SCALE
600
2300
3500
120
300
680
0
100
200
300
400
500
600
700
800
0
500
1000
1500
2000
2500
3000
3500
4000
2012 2013 2014 2015 2016
Nodes
Year
Total Nodes Running Storm at Yahoo
Total Nodes Largest Cluster Size
12
RESOURCE AWARE SCHEDULING IN STORM
• Scheduling in Storm that takes into account resource availability on
machines and resource requirement of workloads when scheduling
the topology
 Fine grain resource control
 Resource Aware Scheduler (RAS) implements this function
- Includes many nice multi-tenant features
• Built on top of:
 Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar,
and Roy Campbell. "R-storm: Resource-aware scheduling in
storm." In Proceedings of the 16th Annual Middleware Conference,
pp. 149-161. ACM, 2015
13
RAS API
• Fine grain resource control
 Allows users to specify resources requirement for each component (Spout or Bolt) in a Storm Topology:
API to set component memory requirement:
API to set component CPU requirement:
Example of Usage:
public T setMemoryLoad(Number onHeap, Number offHeap)
public T setCPULoad(Number amount)
SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);
s1.setMemoryLoad(1024.0, 512.0);
builder.setBolt("exclaim1", new ExclamationBolt(), 3)
.shuffleGrouping("word").setCPULoad(100.0);
14
CLUSTER CONFIGURATIONS
conf/storm.yaml
.
.
.
supervisor.memory.capacity.mb: 20480.0
supervisor.cpu.capacity: 400.0
.
.
.
15
RAS FEATURES – PLUGGABLE PER TOPOLOGY
SCHEDULING STRATEGIES
• Allows users to specify which scheduling strategy to use
• Default Strategy
- Based on:
• Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm: Resource-
aware scheduling in storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM,
2015.
- Enhancements have been made (e.g. limiting max heap size per worker, better rack selection algorithm, etc)
- Aims to pack topology as tightly as possible on machines to reduce communication latency and increase
utilization
- Collocating components that communication with each other (operator chaining)
• Constraint Based Scheduling Strategy
 CSP problem solver
conf.setTopologyStrategy(DefaultResourceAwareStrategy.class);
16
RAS FEATURES – RESOURCE ISOLATION VIA
CGROUPS (LINUX PLATFORMS ONLY*)
• Replaces resource isolation via isolated nodes
• Resource quotas enforced on a per worker basis
• Each worker should not go over its allocated resource quota
• Guarantee QOS and topology isolation
• Documentation:
https://storm.apache.org/releases/2.0.0-
SNAPSHOT/cgroups_in_storm.html
*RHEL 7 or higher. Potential critical bugs in older RHEL versions.
17
RAS FEATURES – PER USER RESOURCE
GUARANTEES
• Configurable per user resource guarantees
18
RAS FEATURE – TOPOLOGY PRIORITY
• Users can set the priority of a topology to indicate its importance
• The range of topology priorities can range form 0-29. The topologies priorities will
be partitioned into several priority levels that may contain a range of priorities
conf.setTopologyPriority(int priority)
PRODUCTION => 0 – 9
STAGING => 10 – 19
DEV => 20 – 29
19
RAS FEATURES – PLUGGABLE TOPOLOGY
PRIORITY
• Topology Priority Strategy
 Which topology should be scheduled first?
 Cluster wide configuration set in storm.yaml
 Default Topology Priority Strategy
- Takes into account resource guarantees and topology priority
- Schedules topologies from users who is the most under his or her resource
guarantee.
- Topologies of each user is sorted by priority
- More details:
https://storm.apache.org/releases/2.0.0-
SNAPSHOT/Resource_Aware_Scheduler_overview.html
20
RAS FEATURES – PLUGGABLE TOPOLOGY
EVICTION STRATEGIES
• Topology Eviction Strategy
 When there is not enough resource which topology from which user to evict?
 Cluster wide configuration set in storm.yaml
 Default Eviction Strategy
- Based on how much a user’s guarantee has been satisfied
- Priority of the topology
 FIFO Eviction Strategy
- Used on our staging clusters.
- Ad hoc use
 More details:
https://storm.apache.org/releases/2.0.0-
SNAPSHOT/Resource_Aware_Scheduler_overview.html
21
SELECTED RESULTS (THROUGHPUT) FROM PAPER [1] – YAHOO
TOPOLOGIES
47% improvement!
50% improvement!
* Figures used [1]
22
SELECTED RESULTS (THROUGHPUT) FROM PAPER [1] – YAHOO
TOPOLOGIES
23
PRELIMINARY RESULTS IN YAHOO STORM CLUSTERS
24
PRELIMINARY RESULTS IN YAHOO STORM CLUSTERS
25
CONCLUDING REMARKS AND FUTURE WORK
• In Summary
 Built resource aware scheduler
• Migration Process
 In the Progress from migrating from MultitenantScheduler to RAS
 Working through bugs with Cgroups, Java, and Linux kernel
• Future Work
 Improved Scheduling Strategies
 Real-time resource monitoring
 Elasticity
26
QUESTIONS
27
REFERENCES
• [1] Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm:
Resource-aware scheduling in Storm." In Proceedings of the 16th Annual Middleware Conference,
pp. 149-161. ACM, 2015.
 http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf
• [2] Official Resource Aware Scheduler Documentation
 https://storm.apache.org/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.htm
• [3] Umbrella Jira for Resource Aware Scheduling in Storm
 https://issues.apache.org/jira/browse/STORM-893
28
EXTRA SLIDES
29
PROBLEM FORMULATION
• Targeting 3 types of resources
 CPU, Memory, and Network
• Limited resource budget for each node
• Specific resource needs for each task
Goal:
Improve throughput by maximizing
utilization and minimizing network
latency
30
PROBLEM FORMULATION
• Set of all tasks Ƭ = {τ1 , τ2, τ3, …}, each task τi has resource demands
 CPU requirement of cτi
 Network bandwidth requirement of bτi
 Memory requirement of mτi
• Set of all nodes N = {θ1 , θ2, θ3, …}
 Total available CPU budget of W1
 Total available Bandwidth budget of W2
 Total available Memory budget of W3
30
31
PROBLEM FORMULATION
• Qi : Throughput contribution of each node
• Assign tasks to a subset of nodes N’ ∈ N that minimizes the total resource waste:
31
32
PROBLEM FORMULATION
 Quadratic Multiple 3D Knapsack Problem
 We call it QM3DKP!
 NP-Hard!
• Compute optimal solutions or approximate solutions may be hard and time consuming
• Real time systems need fast scheduling
 Re-compute scheduling when failures occur
32
33
SOFT CONSTRAINTS VS HARD CONSTRAINTS
• Soft Constraints
 CPU and Network Resources
 Graceful performance degradation with over subscription
• Hard Constraints
 Memory
 Oversubscribe -> Game over
Your date comes hereYour footer comes here33
34
OBSERVATIONS ON NETWORK LATENCY
1. Inter-rack communication is the slowest
2. Inter-node communication is slow
3. Inter-process communication is faster
4. Intra-process communication is the fastest
Your date comes hereYour footer comes here34
35
HEURISTIC ALGORITHM
35
• Greedy approach
• Designing a 3D resource space
 Each resource maps to an axis
 Can be generalized to nD resource space
 Trivial overhead!
• Based on:
 min (Euclidean distance)
 Satisfy hard constraints
36
HEURISTIC ALGORITHM
Your date comes hereYour footer comes here36
37
HEURISTIC ALGORITHM
Your date comes hereYour footer comes here37
Switch
1 2
3 4 5
6
38
HEURISTIC ALGORITHM
38
• Our proposed heuristic algorithm has the following properties:
1) Tasks of components that communicate will each other will have the highest priority to be scheduled in close network proximity
to each other.
2) No hard resource constraint is violated.
3) Resource waste on nodes are minimized.

More Related Content

What's hot

Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
Jun Rao
 
Keep me in the Loop: INotify in HDFS
Keep me in the Loop: INotify in HDFSKeep me in the Loop: INotify in HDFS
Keep me in the Loop: INotify in HDFS
DataWorks Summit
 
Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...
Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...
Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...
HostedbyConfluent
 
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in KafkaBuilding High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafka
confluent
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
confluent
 
What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?
confluent
 
Reactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka StreamsReactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka Streams
Konrad Malawski
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
Kumar Shivam
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connect
confluent
 
Slope Stability Analysis.pptx
Slope Stability Analysis.pptxSlope Stability Analysis.pptx
Slope Stability Analysis.pptx
Vicky966642
 
Why Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik RamasamyWhy Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik Ramasamy
StreamNative
 
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
涛 吴
 
Introducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes OperatorIntroducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes Operator
Flink Forward
 
Introducing Akka
Introducing AkkaIntroducing Akka
Introducing Akka
Jonas Bonér
 
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
DataWorks Summit/Hadoop Summit
 
Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...
Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...
Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...
HostedbyConfluent
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentUsing the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Flink Forward
 
Aerospike Architecture
Aerospike ArchitectureAerospike Architecture
Aerospike Architecture
Peter Milne
 
URP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to KnowURP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to Know
Todd Palino
 
Deswik Software Suite v5.0: LAS file support
Deswik Software Suite v5.0: LAS file supportDeswik Software Suite v5.0: LAS file support
Deswik Software Suite v5.0: LAS file support
Deswik
 

What's hot (20)

Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
 
Keep me in the Loop: INotify in HDFS
Keep me in the Loop: INotify in HDFSKeep me in the Loop: INotify in HDFS
Keep me in the Loop: INotify in HDFS
 
Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...
Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...
Intelligent Auto-scaling of Kafka Consumers with Workload Prediction | Ming S...
 
Building High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in KafkaBuilding High-Throughput, Low-Latency Pipelines in Kafka
Building High-Throughput, Low-Latency Pipelines in Kafka
 
A Deep Dive into Kafka Controller
A Deep Dive into Kafka ControllerA Deep Dive into Kafka Controller
A Deep Dive into Kafka Controller
 
What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?
 
Reactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka StreamsReactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka Streams
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
From Zero to Hero with Kafka Connect
From Zero to Hero with Kafka ConnectFrom Zero to Hero with Kafka Connect
From Zero to Hero with Kafka Connect
 
Slope Stability Analysis.pptx
Slope Stability Analysis.pptxSlope Stability Analysis.pptx
Slope Stability Analysis.pptx
 
Why Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik RamasamyWhy Splunk Chose Pulsar_Karthik Ramasamy
Why Splunk Chose Pulsar_Karthik Ramasamy
 
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
 
Introducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes OperatorIntroducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes Operator
 
Introducing Akka
Introducing AkkaIntroducing Akka
Introducing Akka
 
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
Building Large-Scale Stream Infrastructures Across Multiple Data Centers with...
 
Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...
Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...
Monitoring Kafka without instrumentation using eBPF with Antón Rodríguez | Ka...
 
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentUsing the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production Deployment
 
Aerospike Architecture
Aerospike ArchitectureAerospike Architecture
Aerospike Architecture
 
URP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to KnowURP? Excuse You! The Three Kafka Metrics You Need to Know
URP? Excuse You! The Three Kafka Metrics You Need to Know
 
Deswik Software Suite v5.0: LAS file support
Deswik Software Suite v5.0: LAS file supportDeswik Software Suite v5.0: LAS file support
Deswik Software Suite v5.0: LAS file support
 

Similar to Resource Aware Scheduling in Apache Storm

Resource Aware Scheduling in Storm (Hadoop Summit 2016)
Resource Aware Scheduling in Storm (Hadoop Summit 2016)Resource Aware Scheduling in Storm (Hadoop Summit 2016)
Resource Aware Scheduling in Storm (Hadoop Summit 2016)
Boyang Jerry Peng
 
참여기관_발표자료-국민대학교 201301 정기회의
참여기관_발표자료-국민대학교 201301 정기회의참여기관_발표자료-국민대학교 201301 정기회의
참여기관_발표자료-국민대학교 201301 정기회의
DzH QWuynh
 
Dynamic Provisioning of Data Intensive Computing Middleware Frameworks
Dynamic Provisioning of Data Intensive Computing Middleware FrameworksDynamic Provisioning of Data Intensive Computing Middleware Frameworks
Dynamic Provisioning of Data Intensive Computing Middleware Frameworks
Linh Ngo
 
A sdn based application aware and network provisioning
A sdn based application aware and network provisioningA sdn based application aware and network provisioning
A sdn based application aware and network provisioning
Stanley Wang
 
Real Time Operating Systems
Real Time Operating SystemsReal Time Operating Systems
Real Time Operating Systems
Pawandeep Kaur
 
Mastering Real-time Linux
Mastering Real-time LinuxMastering Real-time Linux
Mastering Real-time Linux
Jean-François Deverge
 
IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...
IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...
IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...
IRJET Journal
 
FAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDS
FAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDSFAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDS
FAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDS
Maurvi04
 
Apache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and ApplicationsApache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and Applications
Thomas Weise
 
Apache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and Applications Apache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and Applications
Comsysto Reply GmbH
 
High availability and disaster recovery in IBM PureApplication System
High availability and disaster recovery in IBM PureApplication SystemHigh availability and disaster recovery in IBM PureApplication System
High availability and disaster recovery in IBM PureApplication System
Scott Moonen
 
load-balancing-method-for-embedded-rt-system-20120711-0940
load-balancing-method-for-embedded-rt-system-20120711-0940load-balancing-method-for-embedded-rt-system-20120711-0940
load-balancing-method-for-embedded-rt-system-20120711-0940
Samsung Electronics
 
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarUnifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Karthik Ramasamy
 
Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters
Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters
Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters
Sumeet Singh
 
Real time Operating System
Real time Operating SystemReal time Operating System
Real time Operating System
Tech_MX
 
[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...
[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...
[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...
Matteo Ferroni
 
Scheduling Task-parallel Applications in Dynamically Asymmetric Environments
Scheduling Task-parallel Applications in Dynamically Asymmetric EnvironmentsScheduling Task-parallel Applications in Dynamically Asymmetric Environments
Scheduling Task-parallel Applications in Dynamically Asymmetric Environments
LEGATO project
 
HPC Controls Future
HPC Controls FutureHPC Controls Future
HPC Controls Future
rcastain
 
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
 
PMIx Updated Overview
PMIx Updated OverviewPMIx Updated Overview
PMIx Updated Overview
Ralph Castain
 

Similar to Resource Aware Scheduling in Apache Storm (20)

Resource Aware Scheduling in Storm (Hadoop Summit 2016)
Resource Aware Scheduling in Storm (Hadoop Summit 2016)Resource Aware Scheduling in Storm (Hadoop Summit 2016)
Resource Aware Scheduling in Storm (Hadoop Summit 2016)
 
참여기관_발표자료-국민대학교 201301 정기회의
참여기관_발표자료-국민대학교 201301 정기회의참여기관_발표자료-국민대학교 201301 정기회의
참여기관_발표자료-국민대학교 201301 정기회의
 
Dynamic Provisioning of Data Intensive Computing Middleware Frameworks
Dynamic Provisioning of Data Intensive Computing Middleware FrameworksDynamic Provisioning of Data Intensive Computing Middleware Frameworks
Dynamic Provisioning of Data Intensive Computing Middleware Frameworks
 
A sdn based application aware and network provisioning
A sdn based application aware and network provisioningA sdn based application aware and network provisioning
A sdn based application aware and network provisioning
 
Real Time Operating Systems
Real Time Operating SystemsReal Time Operating Systems
Real Time Operating Systems
 
Mastering Real-time Linux
Mastering Real-time LinuxMastering Real-time Linux
Mastering Real-time Linux
 
IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...
IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...
IRJET-Framework for Dynamic Resource Allocation and Efficient Scheduling Stra...
 
FAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDS
FAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDSFAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDS
FAULT TOLERANCE OF RESOURCES IN COMPUTATIONAL GRIDS
 
Apache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and ApplicationsApache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and Applications
 
Apache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and Applications Apache Apex: Stream Processing Architecture and Applications
Apache Apex: Stream Processing Architecture and Applications
 
High availability and disaster recovery in IBM PureApplication System
High availability and disaster recovery in IBM PureApplication SystemHigh availability and disaster recovery in IBM PureApplication System
High availability and disaster recovery in IBM PureApplication System
 
load-balancing-method-for-embedded-rt-system-20120711-0940
load-balancing-method-for-embedded-rt-system-20120711-0940load-balancing-method-for-embedded-rt-system-20120711-0940
load-balancing-method-for-embedded-rt-system-20120711-0940
 
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache PulsarUnifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
Unifying Messaging, Queueing & Light Weight Compute Using Apache Pulsar
 
Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters
Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters
Hadoop Summit San Jose 2015: Towards SLA-based Scheduling on YARN Clusters
 
Real time Operating System
Real time Operating SystemReal time Operating System
Real time Operating System
 
[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...
[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...
[EWiLi2016] Towards a performance-aware power capping orchestrator for the Xe...
 
Scheduling Task-parallel Applications in Dynamically Asymmetric Environments
Scheduling Task-parallel Applications in Dynamically Asymmetric EnvironmentsScheduling Task-parallel Applications in Dynamically Asymmetric Environments
Scheduling Task-parallel Applications in Dynamically Asymmetric Environments
 
HPC Controls Future
HPC Controls FutureHPC Controls Future
HPC Controls Future
 
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
 
PMIx Updated Overview
PMIx Updated OverviewPMIx Updated Overview
PMIx Updated Overview
 

More from DataWorks Summit/Hadoop Summit

Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in ProductionRunning Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in Production
DataWorks Summit/Hadoop Summit
 
State of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache ZeppelinState of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache Zeppelin
DataWorks Summit/Hadoop Summit
 
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache RangerUnleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache Ranger
DataWorks Summit/Hadoop Summit
 
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science PlatformEnabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science Platform
DataWorks Summit/Hadoop Summit
 
Revolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and ZeppelinRevolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and Zeppelin
DataWorks Summit/Hadoop Summit
 
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSenseDouble Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSense
DataWorks Summit/Hadoop Summit
 
Hadoop Crash Course
Hadoop Crash CourseHadoop Crash Course
Hadoop Crash Course
DataWorks Summit/Hadoop Summit
 
Data Science Crash Course
Data Science Crash CourseData Science Crash Course
Data Science Crash Course
DataWorks Summit/Hadoop Summit
 
Apache Spark Crash Course
Apache Spark Crash CourseApache Spark Crash Course
Apache Spark Crash Course
DataWorks Summit/Hadoop Summit
 
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataflow with Apache NiFi
Dataflow with Apache NiFi
DataWorks Summit/Hadoop Summit
 
Schema Registry - Set you Data Free
Schema Registry - Set you Data FreeSchema Registry - Set you Data Free
Schema Registry - Set you Data Free
DataWorks Summit/Hadoop Summit
 
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
DataWorks Summit/Hadoop Summit
 
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
DataWorks Summit/Hadoop Summit
 
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and MLMool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and ML
DataWorks Summit/Hadoop Summit
 
How Hadoop Makes the Natixis Pack More Efficient
How Hadoop Makes the Natixis Pack More Efficient How Hadoop Makes the Natixis Pack More Efficient
How Hadoop Makes the Natixis Pack More Efficient
DataWorks Summit/Hadoop Summit
 
HBase in Practice
HBase in Practice HBase in Practice
HBase in Practice
DataWorks Summit/Hadoop Summit
 
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)
DataWorks Summit/Hadoop Summit
 
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopBreaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
DataWorks Summit/Hadoop Summit
 
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
DataWorks Summit/Hadoop Summit
 
Backup and Disaster Recovery in Hadoop
Backup and Disaster Recovery in Hadoop Backup and Disaster Recovery in Hadoop
Backup and Disaster Recovery in Hadoop
DataWorks Summit/Hadoop Summit
 

More from DataWorks Summit/Hadoop Summit (20)

Running Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in ProductionRunning Apache Spark & Apache Zeppelin in Production
Running Apache Spark & Apache Zeppelin in Production
 
State of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache ZeppelinState of Security: Apache Spark & Apache Zeppelin
State of Security: Apache Spark & Apache Zeppelin
 
Unleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache RangerUnleashing the Power of Apache Atlas with Apache Ranger
Unleashing the Power of Apache Atlas with Apache Ranger
 
Enabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science PlatformEnabling Digital Diagnostics with a Data Science Platform
Enabling Digital Diagnostics with a Data Science Platform
 
Revolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and ZeppelinRevolutionize Text Mining with Spark and Zeppelin
Revolutionize Text Mining with Spark and Zeppelin
 
Double Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSenseDouble Your Hadoop Performance with Hortonworks SmartSense
Double Your Hadoop Performance with Hortonworks SmartSense
 
Hadoop Crash Course
Hadoop Crash CourseHadoop Crash Course
Hadoop Crash Course
 
Data Science Crash Course
Data Science Crash CourseData Science Crash Course
Data Science Crash Course
 
Apache Spark Crash Course
Apache Spark Crash CourseApache Spark Crash Course
Apache Spark Crash Course
 
Dataflow with Apache NiFi
Dataflow with Apache NiFiDataflow with Apache NiFi
Dataflow with Apache NiFi
 
Schema Registry - Set you Data Free
Schema Registry - Set you Data FreeSchema Registry - Set you Data Free
Schema Registry - Set you Data Free
 
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
Building a Large-Scale, Adaptive Recommendation Engine with Apache Flink and ...
 
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on ...
 
Mool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and MLMool - Automated Log Analysis using Data Science and ML
Mool - Automated Log Analysis using Data Science and ML
 
How Hadoop Makes the Natixis Pack More Efficient
How Hadoop Makes the Natixis Pack More Efficient How Hadoop Makes the Natixis Pack More Efficient
How Hadoop Makes the Natixis Pack More Efficient
 
HBase in Practice
HBase in Practice HBase in Practice
HBase in Practice
 
The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)The Challenge of Driving Business Value from the Analytics of Things (AOT)
The Challenge of Driving Business Value from the Analytics of Things (AOT)
 
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS HadoopBreaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
Breaking the 1 Million OPS/SEC Barrier in HOPS Hadoop
 
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
From Regulatory Process Verification to Predictive Maintenance and Beyond wit...
 
Backup and Disaster Recovery in Hadoop
Backup and Disaster Recovery in Hadoop Backup and Disaster Recovery in Hadoop
Backup and Disaster Recovery in Hadoop
 

Recently uploaded

“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
Edge AI and Vision Alliance
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
Safe Software
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
bhumivarma35300
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
Anant Gupta
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Mydbops
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
Jimmy Lai
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
Matthias Neugebauer
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
ChristopherTHyatt
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
rajancomputerfbd
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
Priyanka Aash
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
Shiv Technolabs
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 

Recently uploaded (20)

“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
“Deploying Large Language Models on a Raspberry Pi,” a Presentation from Usef...
 
Data Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining DataData Integration Basics: Merging & Joining Data
Data Integration Basics: Merging & Joining Data
 
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
High Profile Girls call Service Pune 000XX00000 Provide Best And Top Girl Ser...
 
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes..."Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
"Mastering Graphic Design: Essential Tips and Tricks for Beginners and Profes...
 
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - MydbopsScaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
 
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-InTrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
TrustArc Webinar - 2024 Data Privacy Trends: A Mid-Year Check-In
 
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python CodebaseEuroPython 2024 - Streamlining Testing in a Large Python Codebase
EuroPython 2024 - Streamlining Testing in a Large Python Codebase
 
Opencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of MünsterOpencast Summit 2024 — Opencast @ University of Münster
Opencast Summit 2024 — Opencast @ University of Münster
 
How to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdfHow to build a generative AI solution A step-by-step guide (2).pdf
How to build a generative AI solution A step-by-step guide (2).pdf
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
(CISOPlatform Summit & SACON 2024) Keynote _ Power Digital Identities With AI...
 
The Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF GuideThe Role of IoT in Australian Mobile App Development - PDF Guide
The Role of IoT in Australian Mobile App Development - PDF Guide
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
Girls call Kolkata 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 

Resource Aware Scheduling in Apache Storm

  • 2. 2 ABOUT ME • Apache Storm Committer and PMC member • Member of the Yahoo’s low latency Team  Data processing solutions with low latency • Graduate student @ University of Illinois, Urbana-Champaign  Research emphasis in distributed systems and stream processing • Contact:  jerrypeng@yahoo-inc.com
  • 3. 3 AGENDA •Overview of Apache Storm •Problems and Challenges •Introduction of Resource Aware Scheduler •Results
  • 4. 4 OVERVIEW • Apache Storm is an open source distributed real-time data stream processing platform  Real-time analytics  Online machine learning  Continuous computation  Distributed RPC  ETL
  • 5. 5 STORM TOPOLOGY • Processing can be represented as a directed graph • Spouts are sources of information • Bolts are operators that process data
  • 6. 6 DEFINITIONS OF STORM TERMS • Stream  an unbounded sequence of tuples. • Component  A processing operator in a Storm topology that is either a Bolt or Spout • Executors  Threads that are spawned in worker processes that execute the logic of components • Worker Process  A process spawned by Storm that may run one or more executors.
  • 8. 8 LOGICAL VS PHYSICAL CONNECTION IN STORM
  • 9. 9 OVERVIEW OF SCHEDULING IN STORM • Default Scheduling Strategy  Naïve round robin scheduler  Naïve load limiter (Worker Slots) • Multitenant Scheduler  Default Scheduler with multitenant capabilities (supported by security)  Can allocate a set of isolated nodes for topology (Soft Partitioning) Resource Aware
  • 10. 10 RUNNING STORM AT YAHOO - CHALLENGES • Increasing heterogeneous clusters  Isolation Scheduler – handing out dedicated machines • Low cluster overall resource utilization  Users not utilizing their isolated allocation very well • Unbalanced resource usage  Some machines not used, others over used • Per topology scheduling strategy  Different topologies have different scheduling needs (e.g. constraint based scheduling)
  • 11. 11 RUNNING STORM AT YAHOO – SCALE 600 2300 3500 120 300 680 0 100 200 300 400 500 600 700 800 0 500 1000 1500 2000 2500 3000 3500 4000 2012 2013 2014 2015 2016 Nodes Year Total Nodes Running Storm at Yahoo Total Nodes Largest Cluster Size
  • 12. 12 RESOURCE AWARE SCHEDULING IN STORM • Scheduling in Storm that takes into account resource availability on machines and resource requirement of workloads when scheduling the topology  Fine grain resource control  Resource Aware Scheduler (RAS) implements this function - Includes many nice multi-tenant features • Built on top of:  Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm: Resource-aware scheduling in storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM, 2015
  • 13. 13 RAS API • Fine grain resource control  Allows users to specify resources requirement for each component (Spout or Bolt) in a Storm Topology: API to set component memory requirement: API to set component CPU requirement: Example of Usage: public T setMemoryLoad(Number onHeap, Number offHeap) public T setCPULoad(Number amount) SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10); s1.setMemoryLoad(1024.0, 512.0); builder.setBolt("exclaim1", new ExclamationBolt(), 3) .shuffleGrouping("word").setCPULoad(100.0);
  • 15. 15 RAS FEATURES – PLUGGABLE PER TOPOLOGY SCHEDULING STRATEGIES • Allows users to specify which scheduling strategy to use • Default Strategy - Based on: • Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm: Resource- aware scheduling in storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM, 2015. - Enhancements have been made (e.g. limiting max heap size per worker, better rack selection algorithm, etc) - Aims to pack topology as tightly as possible on machines to reduce communication latency and increase utilization - Collocating components that communication with each other (operator chaining) • Constraint Based Scheduling Strategy  CSP problem solver conf.setTopologyStrategy(DefaultResourceAwareStrategy.class);
  • 16. 16 RAS FEATURES – RESOURCE ISOLATION VIA CGROUPS (LINUX PLATFORMS ONLY*) • Replaces resource isolation via isolated nodes • Resource quotas enforced on a per worker basis • Each worker should not go over its allocated resource quota • Guarantee QOS and topology isolation • Documentation: https://storm.apache.org/releases/2.0.0- SNAPSHOT/cgroups_in_storm.html *RHEL 7 or higher. Potential critical bugs in older RHEL versions.
  • 17. 17 RAS FEATURES – PER USER RESOURCE GUARANTEES • Configurable per user resource guarantees
  • 18. 18 RAS FEATURE – TOPOLOGY PRIORITY • Users can set the priority of a topology to indicate its importance • The range of topology priorities can range form 0-29. The topologies priorities will be partitioned into several priority levels that may contain a range of priorities conf.setTopologyPriority(int priority) PRODUCTION => 0 – 9 STAGING => 10 – 19 DEV => 20 – 29
  • 19. 19 RAS FEATURES – PLUGGABLE TOPOLOGY PRIORITY • Topology Priority Strategy  Which topology should be scheduled first?  Cluster wide configuration set in storm.yaml  Default Topology Priority Strategy - Takes into account resource guarantees and topology priority - Schedules topologies from users who is the most under his or her resource guarantee. - Topologies of each user is sorted by priority - More details: https://storm.apache.org/releases/2.0.0- SNAPSHOT/Resource_Aware_Scheduler_overview.html
  • 20. 20 RAS FEATURES – PLUGGABLE TOPOLOGY EVICTION STRATEGIES • Topology Eviction Strategy  When there is not enough resource which topology from which user to evict?  Cluster wide configuration set in storm.yaml  Default Eviction Strategy - Based on how much a user’s guarantee has been satisfied - Priority of the topology  FIFO Eviction Strategy - Used on our staging clusters. - Ad hoc use  More details: https://storm.apache.org/releases/2.0.0- SNAPSHOT/Resource_Aware_Scheduler_overview.html
  • 21. 21 SELECTED RESULTS (THROUGHPUT) FROM PAPER [1] – YAHOO TOPOLOGIES 47% improvement! 50% improvement! * Figures used [1]
  • 22. 22 SELECTED RESULTS (THROUGHPUT) FROM PAPER [1] – YAHOO TOPOLOGIES
  • 23. 23 PRELIMINARY RESULTS IN YAHOO STORM CLUSTERS
  • 24. 24 PRELIMINARY RESULTS IN YAHOO STORM CLUSTERS
  • 25. 25 CONCLUDING REMARKS AND FUTURE WORK • In Summary  Built resource aware scheduler • Migration Process  In the Progress from migrating from MultitenantScheduler to RAS  Working through bugs with Cgroups, Java, and Linux kernel • Future Work  Improved Scheduling Strategies  Real-time resource monitoring  Elasticity
  • 27. 27 REFERENCES • [1] Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm: Resource-aware scheduling in Storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM, 2015.  http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf • [2] Official Resource Aware Scheduler Documentation  https://storm.apache.org/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.htm • [3] Umbrella Jira for Resource Aware Scheduling in Storm  https://issues.apache.org/jira/browse/STORM-893
  • 29. 29 PROBLEM FORMULATION • Targeting 3 types of resources  CPU, Memory, and Network • Limited resource budget for each node • Specific resource needs for each task Goal: Improve throughput by maximizing utilization and minimizing network latency
  • 30. 30 PROBLEM FORMULATION • Set of all tasks Ƭ = {τ1 , τ2, τ3, …}, each task τi has resource demands  CPU requirement of cτi  Network bandwidth requirement of bτi  Memory requirement of mτi • Set of all nodes N = {θ1 , θ2, θ3, …}  Total available CPU budget of W1  Total available Bandwidth budget of W2  Total available Memory budget of W3 30
  • 31. 31 PROBLEM FORMULATION • Qi : Throughput contribution of each node • Assign tasks to a subset of nodes N’ ∈ N that minimizes the total resource waste: 31
  • 32. 32 PROBLEM FORMULATION  Quadratic Multiple 3D Knapsack Problem  We call it QM3DKP!  NP-Hard! • Compute optimal solutions or approximate solutions may be hard and time consuming • Real time systems need fast scheduling  Re-compute scheduling when failures occur 32
  • 33. 33 SOFT CONSTRAINTS VS HARD CONSTRAINTS • Soft Constraints  CPU and Network Resources  Graceful performance degradation with over subscription • Hard Constraints  Memory  Oversubscribe -> Game over Your date comes hereYour footer comes here33
  • 34. 34 OBSERVATIONS ON NETWORK LATENCY 1. Inter-rack communication is the slowest 2. Inter-node communication is slow 3. Inter-process communication is faster 4. Intra-process communication is the fastest Your date comes hereYour footer comes here34
  • 35. 35 HEURISTIC ALGORITHM 35 • Greedy approach • Designing a 3D resource space  Each resource maps to an axis  Can be generalized to nD resource space  Trivial overhead! • Based on:  min (Euclidean distance)  Satisfy hard constraints
  • 36. 36 HEURISTIC ALGORITHM Your date comes hereYour footer comes here36
  • 37. 37 HEURISTIC ALGORITHM Your date comes hereYour footer comes here37 Switch 1 2 3 4 5 6
  • 38. 38 HEURISTIC ALGORITHM 38 • Our proposed heuristic algorithm has the following properties: 1) Tasks of components that communicate will each other will have the highest priority to be scheduled in close network proximity to each other. 2) No hard resource constraint is violated. 3) Resource waste on nodes are minimized.

Editor's Notes

  1. Good afternoon, My name is Boyang Jerry Peng and I am here to present Resource Aware Scheduling in Apache.
  2. A little about me, apache storm committer and pmc member I am currently apart of the low latency team at Yahoo. Our team primarily works on projects that provide data processing solutions with low latency to yahoo and Apache storm is one of the platforms we work on. Prior to me joining Yahoo, I was a graduate student at the University of Iilinois, urbana champaign with a research emphasis in distributed systems.
  3. First, going to provide a brief overview of Apache Storm Then, I will discuss the problems and challenges of running apache storm at yahoo. Next, I will get to the core of this presentation and talk about resource aware scheduling in Storm. Define what it is and how to use it and how it helps us overcome the problems and challenges I have mentioned Lastly, I will present some results.
  4. Apache Storm is a popular open source distributed data stream processing platform used by many companies in industry There are many use cases for Apache Storm such as: Real-time analytics , Online machine learning , Continuous computation , Distributed RPC , and ETL operations
  5. In apache storm, an application or workload is called a Storm topology. A storm topology, like applications in other stream processing systems, can be represented as a directed graph In which each edge represents a flow of data and each vertex a location where processing data occurs. In Storm, there are two types of operators or component. First type is called a spout. Spouts are sources of information and are responsible for injecting data into the storm topology Second type is called a bolt. Bolts consume streams of data, conduct any user defined processing, and potentially emit new streams of data downstream to be processed by other bolts
  6. Briefly go over some definitions in Storm
  7. Two types of nodes in a Storm cluster A master node that runs a daemon called Nimbus. The master node and the Nimbus daemon is responsible (with the help of Apache Zookeeper) for maintaining the active membership of the storm cluster. The nimbus Node is also responsible for computing schedulings of topologies in the Storm cluster. A worker node in Storm is a node that runs a daemon called supervisor that is responsible for retrieving schedulings from nimbus via zookeeper and launching the necessary processes according to the scheduling to realize the computation of the topology
  8. Let me also talk about the difference between logical and physical connections in Storm. The diagram on the left is an example of a storm topology where executors are organized by component. And each line connecting two executors represents a logical connection. In The diagram on your right, executors are organized by the physical machines they are scheduled on and each line represents a physical connection. As you can see logical connections can vary quite a bit from the physical connections that need to be made in a topology This is where the scheduler can play an important part. How the topology is scheduled can have major impacts on performance of the topology.
  9. Let me talk about how scheduling is done in storm Default scheduler schedules executors in a round robin fashion Uses the concept of worker slots to limit the computation load on a single machine. Can only Launch as many worker processes as worker slots. Each worker can run any number of executors that requires any amount of resources to run. Because not resource aware customers want isolated nodes Not very effective Not resource aware. Executors use any arbitrary amount of resources. See some loads overloaded and some nodes empty
  10. Let me talk about some challenges of running storm at yahoo Our clusters have become increasingly heterogeneous. Made up of older nodes and new nodes that have different hardware specs Handing out dedicated nodes heterogeneous cluster, some times nodes on size some time another Not utilizing resources well. Customers used more nodes then they need. Because they don’t think about resource requirements as well. Nothing else can run on those isolated nodes
  11. Fine grain resource control Deprecates the notion of using worker slots to limit load and removes the need to use isolated nodes. Resource isolation via cgroups
  12. Let me go over the some of the core API for scheduling with resource aware scheduler Allows users to specify the resource requirements for each component…
  13. Cluster admins can specify how much of each resource is available for user on each worker machine
  14. Let me talk about some features Resource Aware Scheduler provides One of them is have pluggable per topology scheduling strategies. We have identified that different topologies might have different scheduling needs Constraint based scheduling strategy: An internal user has some scheduling requirements in which Users can can describe these constraints and the strategy will attempt to find a scheduling that satisfies these constraints
  15. Only neat features we developed to support RAS is resource isolation via cgroups Get rid of delagating isolated nodes that was killing out utilization Rhel 7 cgroup and java memory do play well. Bugs in kernel
  16. Taken into account in the scheduling priority and eviction strategies I will mention latter
  17. Taken into account in scheduling priority and eviction strategies
  18. pluggable In what order should the topologies be scheduled
  19. Pluggable Different clusters should have different eviction policies (Production vs Staging) How much over his or her resource guarantee a user is Not enough resources or sudden failure
  20. Still in the process of migration. The average amount of assigned memory has decreased. Which implies that topologies are becoming more resource efficient to run Using less memory to run Run more topologies
  21. Working out the kinks. Cgroup and memory. Complete migration, beta quality
  22. For each task with a certain resource vector that represents its resource requirement we attempt to find the node with the resource vector that represents its resource availability that is closest Based on min (Euclidean distance) while not violating hard constraints
  23. Based on min (Euclidean distance) while not violating hard constraints