Submit Search
Upload
Automatic Self-Tuning Architecture for Batch Scheduler on Large Scale Computing System
•
Download as ODP, PDF
•
2 likes
•
707 views
Sugree Phatanapherom
Follow
Presentation of academic research with 140-char limit
Read less
Read more
Technology
Business
Report
Share
Report
Share
1 of 83
Download now
Recommended
Event Scheduling
Event Scheduling
Ayesha Kanwal
Example Solutions for Scheduling and Work Planning
Example Solutions for Scheduling and Work Planning
SIS Group International
Why Average Response Time is not a right measure of your web application's pe...
Why Average Response Time is not a right measure of your web application's pe...
vodQA
Performance testing basics
Performance testing basics
Charu Anand
Mining model for hotel recommendations (Kaggle Challenge)
Mining model for hotel recommendations (Kaggle Challenge)
Arjun Varma
Time advance mehcanism
Time advance mehcanism
Nikhil Sharma
Predictive control 1 introduction
Predictive control 1 introduction
jamestpp
UC4 - One Automation
UC4 - One Automation
k1k2sdad
Recommended
Event Scheduling
Event Scheduling
Ayesha Kanwal
Example Solutions for Scheduling and Work Planning
Example Solutions for Scheduling and Work Planning
SIS Group International
Why Average Response Time is not a right measure of your web application's pe...
Why Average Response Time is not a right measure of your web application's pe...
vodQA
Performance testing basics
Performance testing basics
Charu Anand
Mining model for hotel recommendations (Kaggle Challenge)
Mining model for hotel recommendations (Kaggle Challenge)
Arjun Varma
Time advance mehcanism
Time advance mehcanism
Nikhil Sharma
Predictive control 1 introduction
Predictive control 1 introduction
jamestpp
UC4 - One Automation
UC4 - One Automation
k1k2sdad
Data Mining and Analytics
Data Mining and Analytics
Nathaniel Palmer
A New Approach for Job Scheduling Using Hybrid GA-ST Optimization-Crimson Pub...
A New Approach for Job Scheduling Using Hybrid GA-ST Optimization-Crimson Pub...
CrimsonPublishersRDMS
Presenting an Algorithm for Tasks Scheduling in Grid Environment along with I...
Presenting an Algorithm for Tasks Scheduling in Grid Environment along with I...
Editor IJCATR
Workforce Management & BPM Integration
Workforce Management & BPM Integration
Nathaniel Palmer
Workforce Management & BPM Integration
Workforce Management & BPM Integration
Nathaniel Palmer
genetic paper
genetic paper
Swathi Rampur
Design and Implementation of a Multi-Agent System for the Job Shop Scheduling...
Design and Implementation of a Multi-Agent System for the Job Shop Scheduling...
CSCJournals
MULTIPROCESSOR AND REAL TIME SCHEDULING.ppt
MULTIPROCESSOR AND REAL TIME SCHEDULING.ppt
CUO VEERANAN VEERANAN
CS 23 Operating System Design Principles_MULTIPROCESSOR AND REAL TIME SCHEDULING
CS 23 Operating System Design Principles_MULTIPROCESSOR AND REAL TIME SCHEDULING
CUO VEERANAN VEERANAN
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
ijceronline
@SIMUL8 Virtual User Group, September: Brian Harrington, Less is More
@SIMUL8 Virtual User Group, September: Brian Harrington, Less is More
SIMUL8 Corporation
Methods of Optimization in Machine Learning
Methods of Optimization in Machine Learning
Knoldus Inc.
Carasik BPM ECM
Carasik BPM ECM
Bob Carasik
Analytics for Process Excellence
Analytics for Process Excellence
Denis Gagné
IRJET- Advance Approach for Load Balancing in Cloud Computing using (HMSO) Hy...
IRJET- Advance Approach for Load Balancing in Cloud Computing using (HMSO) Hy...
IRJET Journal
performance
performance
manogallery
G017314249
G017314249
IOSR Journals
Improved Max-Min Scheduling Algorithm
Improved Max-Min Scheduling Algorithm
iosrjce
STATISTICAL APPROACH TO DETERMINE MOST EFFICIENT VALUE FOR TIME QUANTUM IN RO...
STATISTICAL APPROACH TO DETERMINE MOST EFFICIENT VALUE FOR TIME QUANTUM IN RO...
ijcsit
LEARNING SCHEDULER PARAMETERS FOR ADAPTIVE PREEMPTION
LEARNING SCHEDULER PARAMETERS FOR ADAPTIVE PREEMPTION
cscpconf
Twitter, Facebook and etc: Quick Startup Guide for Marketing
Twitter, Facebook and etc: Quick Startup Guide for Marketing
Sugree Phatanapherom
@sugree and Twitter
@sugree and Twitter
Sugree Phatanapherom
More Related Content
Similar to Automatic Self-Tuning Architecture for Batch Scheduler on Large Scale Computing System
Data Mining and Analytics
Data Mining and Analytics
Nathaniel Palmer
A New Approach for Job Scheduling Using Hybrid GA-ST Optimization-Crimson Pub...
A New Approach for Job Scheduling Using Hybrid GA-ST Optimization-Crimson Pub...
CrimsonPublishersRDMS
Presenting an Algorithm for Tasks Scheduling in Grid Environment along with I...
Presenting an Algorithm for Tasks Scheduling in Grid Environment along with I...
Editor IJCATR
Workforce Management & BPM Integration
Workforce Management & BPM Integration
Nathaniel Palmer
Workforce Management & BPM Integration
Workforce Management & BPM Integration
Nathaniel Palmer
genetic paper
genetic paper
Swathi Rampur
Design and Implementation of a Multi-Agent System for the Job Shop Scheduling...
Design and Implementation of a Multi-Agent System for the Job Shop Scheduling...
CSCJournals
MULTIPROCESSOR AND REAL TIME SCHEDULING.ppt
MULTIPROCESSOR AND REAL TIME SCHEDULING.ppt
CUO VEERANAN VEERANAN
CS 23 Operating System Design Principles_MULTIPROCESSOR AND REAL TIME SCHEDULING
CS 23 Operating System Design Principles_MULTIPROCESSOR AND REAL TIME SCHEDULING
CUO VEERANAN VEERANAN
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
ijceronline
@SIMUL8 Virtual User Group, September: Brian Harrington, Less is More
@SIMUL8 Virtual User Group, September: Brian Harrington, Less is More
SIMUL8 Corporation
Methods of Optimization in Machine Learning
Methods of Optimization in Machine Learning
Knoldus Inc.
Carasik BPM ECM
Carasik BPM ECM
Bob Carasik
Analytics for Process Excellence
Analytics for Process Excellence
Denis Gagné
IRJET- Advance Approach for Load Balancing in Cloud Computing using (HMSO) Hy...
IRJET- Advance Approach for Load Balancing in Cloud Computing using (HMSO) Hy...
IRJET Journal
performance
performance
manogallery
G017314249
G017314249
IOSR Journals
Improved Max-Min Scheduling Algorithm
Improved Max-Min Scheduling Algorithm
iosrjce
STATISTICAL APPROACH TO DETERMINE MOST EFFICIENT VALUE FOR TIME QUANTUM IN RO...
STATISTICAL APPROACH TO DETERMINE MOST EFFICIENT VALUE FOR TIME QUANTUM IN RO...
ijcsit
LEARNING SCHEDULER PARAMETERS FOR ADAPTIVE PREEMPTION
LEARNING SCHEDULER PARAMETERS FOR ADAPTIVE PREEMPTION
cscpconf
Similar to Automatic Self-Tuning Architecture for Batch Scheduler on Large Scale Computing System
(20)
Data Mining and Analytics
Data Mining and Analytics
A New Approach for Job Scheduling Using Hybrid GA-ST Optimization-Crimson Pub...
A New Approach for Job Scheduling Using Hybrid GA-ST Optimization-Crimson Pub...
Presenting an Algorithm for Tasks Scheduling in Grid Environment along with I...
Presenting an Algorithm for Tasks Scheduling in Grid Environment along with I...
Workforce Management & BPM Integration
Workforce Management & BPM Integration
Workforce Management & BPM Integration
Workforce Management & BPM Integration
genetic paper
genetic paper
Design and Implementation of a Multi-Agent System for the Job Shop Scheduling...
Design and Implementation of a Multi-Agent System for the Job Shop Scheduling...
MULTIPROCESSOR AND REAL TIME SCHEDULING.ppt
MULTIPROCESSOR AND REAL TIME SCHEDULING.ppt
CS 23 Operating System Design Principles_MULTIPROCESSOR AND REAL TIME SCHEDULING
CS 23 Operating System Design Principles_MULTIPROCESSOR AND REAL TIME SCHEDULING
International Journal of Computational Engineering Research(IJCER)
International Journal of Computational Engineering Research(IJCER)
@SIMUL8 Virtual User Group, September: Brian Harrington, Less is More
@SIMUL8 Virtual User Group, September: Brian Harrington, Less is More
Methods of Optimization in Machine Learning
Methods of Optimization in Machine Learning
Carasik BPM ECM
Carasik BPM ECM
Analytics for Process Excellence
Analytics for Process Excellence
IRJET- Advance Approach for Load Balancing in Cloud Computing using (HMSO) Hy...
IRJET- Advance Approach for Load Balancing in Cloud Computing using (HMSO) Hy...
performance
performance
G017314249
G017314249
Improved Max-Min Scheduling Algorithm
Improved Max-Min Scheduling Algorithm
STATISTICAL APPROACH TO DETERMINE MOST EFFICIENT VALUE FOR TIME QUANTUM IN RO...
STATISTICAL APPROACH TO DETERMINE MOST EFFICIENT VALUE FOR TIME QUANTUM IN RO...
LEARNING SCHEDULER PARAMETERS FOR ADAPTIVE PREEMPTION
LEARNING SCHEDULER PARAMETERS FOR ADAPTIVE PREEMPTION
More from Sugree Phatanapherom
Twitter, Facebook and etc: Quick Startup Guide for Marketing
Twitter, Facebook and etc: Quick Startup Guide for Marketing
Sugree Phatanapherom
@sugree and Twitter
@sugree and Twitter
Sugree Phatanapherom
Behind the madness
Behind the madness
Sugree Phatanapherom
drupal.in.th
drupal.in.th
Sugree Phatanapherom
Twitter API and Startup Ideas
Twitter API and Startup Ideas
Sugree Phatanapherom
Readme Read Sugree
Readme Read Sugree
Sugree Phatanapherom
SCMSWeb and Condor-G Demonstration
SCMSWeb and Condor-G Demonstration
Sugree Phatanapherom
Hand-on Resources II: Extending SCMSWeb
Hand-on Resources II: Extending SCMSWeb
Sugree Phatanapherom
Drupal: blog and beyond
Drupal: blog and beyond
Sugree Phatanapherom
The Spirit of Open Source
The Spirit of Open Source
Sugree Phatanapherom
mbpurple - the replacement twitter im
mbpurple - the replacement twitter im
Sugree Phatanapherom
jibjib - ultimate twitter client for your phone
jibjib - ultimate twitter client for your phone
Sugree Phatanapherom
Next Web Application - Brainstorm
Next Web Application - Brainstorm
Sugree Phatanapherom
Optimizing Drupal for Mobile Devices
Optimizing Drupal for Mobile Devices
Sugree Phatanapherom
Call for Students: Google Summer of Code 2008
Call for Students: Google Summer of Code 2008
Sugree Phatanapherom
Twitter Rules
Twitter Rules
Sugree Phatanapherom
More from Sugree Phatanapherom
(16)
Twitter, Facebook and etc: Quick Startup Guide for Marketing
Twitter, Facebook and etc: Quick Startup Guide for Marketing
@sugree and Twitter
@sugree and Twitter
Behind the madness
Behind the madness
drupal.in.th
drupal.in.th
Twitter API and Startup Ideas
Twitter API and Startup Ideas
Readme Read Sugree
Readme Read Sugree
SCMSWeb and Condor-G Demonstration
SCMSWeb and Condor-G Demonstration
Hand-on Resources II: Extending SCMSWeb
Hand-on Resources II: Extending SCMSWeb
Drupal: blog and beyond
Drupal: blog and beyond
The Spirit of Open Source
The Spirit of Open Source
mbpurple - the replacement twitter im
mbpurple - the replacement twitter im
jibjib - ultimate twitter client for your phone
jibjib - ultimate twitter client for your phone
Next Web Application - Brainstorm
Next Web Application - Brainstorm
Optimizing Drupal for Mobile Devices
Optimizing Drupal for Mobile Devices
Call for Students: Google Summer of Code 2008
Call for Students: Google Summer of Code 2008
Twitter Rules
Twitter Rules
Recently uploaded
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
CzechDreamin
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
FIDO Alliance
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
中 央社
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
IES VE
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
UXDXConf
Overview of Hyperledger Foundation
Overview of Hyperledger Foundation
Hyperleger Tokyo Meetup
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
Patrick Viafore
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
Samy Fodil
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
panagenda
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
Srushith Repakula
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
Syngulon
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
Jennifer Lim
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
FIDO Alliance
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
Stephen Perrenod
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
CzechDreamin
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
FIDO Alliance
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Exakis Nelite
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
FIDO Alliance
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
ScyllaDB
Recently uploaded
(20)
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
Designing for Hardware Accessibility at Comcast
Designing for Hardware Accessibility at Comcast
Overview of Hyperledger Foundation
Overview of Hyperledger Foundation
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
Easier, Faster, and More Powerful – Notes Document Properties Reimagined
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
Behind the Scenes From the Manager's Chair: Decoding the Secrets of Successfu...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
Automatic Self-Tuning Architecture for Batch Scheduler on Large Scale Computing System
1.
Automatic Self-Tuning Architecture
for Batch Scheduler on Large Scale Computing System
2.
I am Sugree
Phatanapherom from Kasetsart University.
3.
This research is
a co-work with Asst. Prof. Putchong Uthayopas.
4.
Ready, steady, go.
5.
What is batch
scheduler?
6.
Batch scheduler is
responsible to schedule jobs to execute on resources at the right time.
7.
Why do we
need batch scheduler?
8.
To utilize resources
efficiently.
9.
To finish all
jobs as fast as possible.
10.
To minimize power
consumption.
11.
In general, it
is so called "resource scheduling problem".
12.
Jobs, Resources and
Time time resources
13.
In this research,
main criteria is to minimize cost to run the resources.
14.
Back to the
past, most works focused on improving algorithms.
15.
To simplify the
problem, this research limits scope job characteristics to independent sequential jobs.
16.
In short, a
job contains the one and only one task.
17.
In other words,
job = task.
18.
Scheduling Algorithms Scheduling
On-line Batch RR OLB MET MCT MinMin MaxMin Sufferage XSufferage CMinMin CMaxMin CSufferage
19.
There are on-line
and batch scheduling.
20.
The most simple
algorithm is "Round Robin".
21.
"Opportunistic Load Balancing"
assigns job to the next available machine.
22.
"Minimum Execution Time"
assigns job to the fastest machine.
23.
"Minimum Completion Time"
assigns job to the machine with minimum completion time for that job.
24.
Next are batch
scheduling algorithms.
25.
"MinMin" assigns shortest
job to the fastest machine.
26.
"MaxMin" assign longest
job to the fastest machine.
27.
"Sufferage" is reassignable
MaxMin.
28.
"XSufferage" is Sufferage
with data locality.
29.
CMinMin, CMaxMin and
CSufferage are derivative with costing.
30.
How to verify?
How to evaluate?
31.
The answer is
simulation. Why?
32.
Closed. Controllable. Reproducible.
33.
Simulation is assumption
and modeling.
34.
Grid is a
meta-scheduler and underlying cluster schedulers managing hosts.
35.
Grid Grid Scheduler
Cluster Scheduler Host Cluster Scheduler Cluster Scheduler jobs Host
36.
Interconnection between scheduler
and processors are dedicated.
37.
Network Scheduler Processor
Storage Processor Processor Processor
38.
Job consists of
inputs, outputs and executable.
39.
Job Executable Input
Output Machine
40.
Operations are 2
steps; mapping and scheduling.
41.
Mapping "job" to
"machine".
42.
Schedule "job" to
the exact time.
43.
In short, the
result is generic priority index.
44.
45.
Time ready time
execution time deadline period before deadline time
46.
Cost cumulative cost
cost cost
47.
Experimented based on
GAMESS job log in ThaiGrid to assume a small and a big system and named them, KUGrid and ThaiGrid, respectively.
48.
Makespan and cost
are observed.
49.
Makespan is the
period of time from when the first job submitted to the last job finished.
50.
Price-Performance
51.
Cost
52.
Makespan
53.
Looks great! Any
problems? Yes!
54.
Priority index contains
5 factors. What are the right values?
55.
What are the
factors of those factors?
56.
There are so
many dependencies. Job characteristics. Resource characteristics. User characteristics.
57.
This problem is
so called "Multi-variate Optimization".
58.
Plus, a bit
more complex with evaluation in simulator.
59.
How to solve?
60.
Optimization Architecture Optimizer
Simulator Simulator Simulator Simulator Batch Scheduler Monitoring System Accounting System
61.
Optimization Algorithm?
62.
Particle Swarm Optimization
is selected as the first one to try.
63.
The position of
each particle in n-dimension plane represents solution.
64.
PSO is social
influence in various scopes.
65.
Local, neighbor and
global.
66.
Usually, one trust
oneself, friends and the world, respectively. The level of trust.
67.
PSO
68.
How to fully
automate self-tuning process?
69.
Historical data are
the key.
70.
The quality of
solution depends on optimizer.
71.
Running optimizer longer
may return better solution.
72.
Precision of using
historical data depends on data period and amount of data.
73.
How to use
historical data? Log replay or estimation.
74.
How to maximize
solution quality to near optimal?
75.
Just run more
simulations using the whole grid system to optimize itself at night!
76.
Results? Please accept
my apologize. They are not published yet.
77.
Conclusion.
78.
Flexible algorithms introduce
more adjustable factors.
79.
The factors are
vary from time to time.
80.
In other view,
these algorithms are improved by external optimization periodically.
81.
Particle swarm optimization
is selected to solve multi-variate optimization.
82.
Improve scheduler by
scheduler itself.
83.
Any questions?
Download now