2. Table of Contents
• Abstract........................................................................................................................................................1
• 1 Introduction..............................................................................................................................................2
• 2 Background...............................................................................................................................................5
2.1 Grid Computing and Grid Service........................................................................................................5
2.2 Resource Sharing on the Grid...............................................................................................................6
2.2.1 Definition of Resource on the Grid...............................................................................................6
2.2.2 Virtual Organization on the Grid...................................................................................................7
2.3 Grid Information Services....................................................................................................................7
2.3.1 Concept of Discovery and Monitoring..........................................................................................7
2.3.2 MDS Information Services............................................................................................................8
Overview.....................................................................................................................................8
Dynamic Resource Discovery....................................................................................................9
GLUE Schema............................................................................................................................9
ANGExtended GLUE Schema................................................................................................10
2.4 Metascheduling on the Grid..............................................................................................................11
• 3 Motivations.............................................................................................................................................14
• 4 A Survey of Current Metaschedulers...................................................................................................17
4.1 Motivation...........................................................................................................................................17
4.2 Current Condition of ANG.................................................................................................................17
4.3 Survey of Current Opensource MateSchedulers.............................................................................17
4.3.1 Scheduling Functionality.............................................................................................................18
4.3.2 Fault Detection and Recovery Capabilities.................................................................................20
4.3.3 User Interface Functionality........................................................................................................21
4.3.4 Application Support.....................................................................................................................22
4.3.5 Features and Limitations..............................................................................................................23
4.4 Conclusion...........................................................................................................................................23
• 5 Implementation.......................................................................................................................................25
5.1 GridWay..............................................................................................................................................25
5.1.1 GridWay Modular Architecture..................................................................................................25
5.1.2 Working Mechanisms..................................................................................................................27
5.2 Resource Information Representation................................................................................................28
5.2.1 Problem Statement.......................................................................................................................28
5.2.2 A Usage Example........................................................................................................................29
5.2.3 Solution: ANG Resource Information Model.............................................................................31
5.2.4 Identifying Heterogeneous Resources.........................................................................................32
5.3 Software Requirements.......................................................................................................................34
5.3.1 New Variables Supporting Software Requirement.....................................................................34
24. via Gridbus broker API
GridWay
(v5.2)
• Needs to specify the
MDS server
• All the hosts and
queues can be
dynamically
discovered
Yes No direct
support
Expressed by a set of
predefined variables
Supports RSL
Supports JSDL
No support Two builtin scheduling algorithms
available:
• A default builtin scheduling
algorithm using a weighted sum
approach to select the candidate
resource
• A RoundRobin scheduler is
given as an example schedule
User priority and resource priority
are supported. Other scheduling
algorithm should be customized.
NimrodG
(v3.x)
No support for dynamic
resource discovery, the
resources must be
statically specified in
Nimrod database
Yes No direct
support
No direct support,
user has to specify
the resource for the
given job through
command line
No support • Resource needs to be allocated to
the given job
• Two computational economy
scheduling algorithms
• Deadline Scheduling
• Cost Minimization
* Resource Specification Language (RSL): The de facto standard used to describe the requirements of a computational job for
submission to Globus execution system.
* Job Submission Description Language (JSDL): An OGF [29] recommended standard for computational job description and
submission to resources, particularly Grid environments but not restricted to the latter [31].
Table 4.1 Scheduling Functionality
4.3.2 Fault Detection and Recovery Capabilities
• Job Cancellation
A good metascheduler should make the submitted jobs under control, the bottom
line is allowing endusers to cancel the jobs they submitted through the scheduler.
• Fault Tolerance
Because of the dynamics of the grid environment and the network itself, the job
can not be absolutely guaranteed to be finished successfully according to user
requirements for some unpredictable reasons like local or remote system crash or
network disconnection, therefore the metascheduler is supposed to offer basic
mechanism which can friendly handle the job in case of failure, such as retry,
rescheduling or recovery.
Table 4.2 lists comparisons between these metaschedulers’ functionalities regarding to
job cancellation, resource and client fault tolerance, as we can see, all of these meta
schedulers somehow support job cancellation and fault tolerance, like Command Line
20
25. Interface(CLI), Application Programming Interface(API), job retry or resubmission and
job recovery.
Job Cancellation Resource Fault Tolerance Client Fault Tolerance
CondorG(v6.8) Users are allowed to cancel the
job via CLI or API
• Achieved by retrying and restarting
globus JobManager
• Provides enhanced towphase commit
submit protocol
All relevant state for each
submitted job is stored
persistently in the Condor job
queue for job recovery
CSF(v4.0.3) Users are allowed to cancel the
job via Command Line Interface
(CLI) and CSF API
No direct support but can be achieved by CSF job service API
Gridbus Broker
(v3.1)
• No direct command for killing
job via CLI
• Job can be reset by API
Persistence to enable failure management and recovery of an executing
grid application
GridWay(v5.2) Users are allowed to cancel the
job via CLI or API
• Job will be migrated if the job exit
code is not specified
• Job is retried on the same resource in
case of failure
• Job is rescheduled if the number of
retries reaches a specified limit
• The host is banned for a specific time
period if the resource on it fails
The state of GridWay running
on the client host is
periodically saved in order for
job recovery
NimrodG(v3.x) Users are allowed to cancel the
job via CLI
• Achieved by job resubmission Job statuses are recorded in
Nimrod database for
monitoring and recovery
purpose
Table 4.2 Fault Detection and Recovery Capabilities
4.3.3 User Interface Functionality
• Application Scope
As Globus is the de facto standard of Grid Computing, our metascheduler is
supposed to fully work with Globus middleware.
• Command Line Interface (CLI)
Command line tool should provide the endusers with basic functionalities to
submit and monitor the jobs and the status of the resources on the grid
• Application Programming Interface (API)
API support is an advantageous feature when developing advanced grid
application based on the metascheduling system
Table 4.3 lists the comparison of the major metaschedulers' functionalities in terms of
application scope, CLI and API.
21
26. Application Scope Command Line Interface Application Programming
Interface
CondorG(v6.8) Works as a part of Condor
middleware
Rich command line interface No support
CSF(v4.0.3) Works on the top of Globus No special support Supported by a set of Java
APIs
Gridbus
Broker(v3.1)
Works with other
Gridbus middlewares
No special support Supported by a set of Java
APIs
GridWay(v5.2) Works on the top of
Globus
• Similar to that found on
resource management
systems like PBS and
SGE
• Supports job management
and resource monitoring
• Supports multiple users
Full DRMAA* (OGF [29]
standard) support for C
and Java
NimrodG
(v6.8.x)
Works independently No special support
Works independently
No support
* Distributed Resource Management Application API (DRMAA): A highlevel OGF [29] API specification for the submission and
control of jobs to one or more Distributed Resource Management (DRM) systems within a Grid architecture. The scope of the API
covered all the high level functionality required for Grid applications to consign jobs to local Grid DRM systems, and it included
common operations on jobs like termination or suspension.
Table 4.3 User Interface Functionality
4.3.4 Application Support
• Single Job
Supporting single job scheduling is the basic functionality of the metascheduler
• Parameter Sweep Job
Support for parameter sweep job (an array of jobs that use the same execution
program but with different input arguments) is a desirable feature when the jobs
execute same executable with difference input parameters
• Workflow Job
A workflow consists of a serial of jobs restricted by dependencies. Support for basic
workflow is helpful to develop advanced job scheduling system
Table 4.4 lists the comparison of the major metaschedulers' supports in terms of single
job, parameter sweep job and workflow job.
Single Job Parameter Sweep
Job
Workflow Job
CondorG(v6.8) Yes Yes Supported by Condor
DAGMan (Directed Acyclic
Graph Manager) [33]
CSF(v4.0.3) Yes No direct support No direct support
Gridbus
Broker(v3.1)
Yes Yes Supported via working with Gridbus
workflow engine
22
27. GridWay(v5.2) Yes Yes Supports simple workflows by
specifying job dependency
relationship
NimrodG(v6.8.x)Yes Yes No support
Table 4.4 Application Support
4.3.5 Features and Limitations
This section (Table 4.5) generally summarizes the features and limitations of the
investigated metaschedulers, thus giving us an overall perspective of their advantages
and disadvantages.
Features Limitations in Current Release
Condor
G(v6.8)
• Wide middleware support • No checkpoints
• No job exit codes. Job exit codes are not available
• Lacks integration with grid information systems
CSF(v4.0.3) Mainly focuses on providing API for
customization, including advanced job
execution management and resource
reservation services
• No software requirement support
• No VO policy support
Gridbus
Broker(v3.1)
• Can be hosted as a remote WRSF service
• Direct support SRB file protocol
• Direct support parameter sweep job
• Wide middleware support
• Only supports single job submission, job
dependency are only supported via Gridbus
workflow engine
• Needs to work with other Gridbus middlewares
for higher level functionalities
GridWay(v5.2
)
• Flexible and dynamic resource discovery
mechanism
• Rich options for resource selection and rank
expressions
• Modular architecture, easy to modify and
extend
• DRMAA API includes both C and Java
implementations
• Supports multiple user mode
• Accepts jobs from GridGateWay*, a job
submission interface that is similar to
Globus Grid Resource Allocation Manager
(GRAM)*
• Current resource information parser is not
suitable to describe the heterogeneous grid
resource which is now supported by GLUE1.2
Schema
• No software requirement support
• No VO policy support
Nimrod
G(v6.8.x)
Specially designed for parameter sweep
applications
No scheduling functionalities
No software requirement support
No VO policy support
* Grid Resource Allocation Manager (GRAM) [35]: A part of Globus Toolkit, which provides a set of web services to submit, monitor
and cancel jobs on Grid computing resources. It is not a resource scheduler but rather an interface which is used to communicate with
various local resource managements (LRM). GRAM is the de facto protocol between the metascheduling system and the LRM.
* GridGateWay (GGW) [34]: A job submission interface which can be integrated with GRAM but can be used for job submission to
GridWay metascheduling system.
Table 4.5 Features and Limitations
4.4 Conclusion
According to the survey we conducted, we came to the conclusions as following:
23
28. • None of the current major (opensource) schedulers can fully support the latest
WSbased grid information service standards (i.e., Globus MDS4 with GLUE1.2).
• None of these metaschedulers support automatic software searching and
matchmaking according to user requirements, which is a highly desirable for
automating and virtualizing enduser's job execution environment.
• As the resources across the grid are shared by multiple organizations, a usage
control mechanism is very useful to appropriately allocate resources to users from
different virtual organization; however, none of current metaschedulers have such
support. It is noteworthy that GridWay5.2 has similar support (see Table 4.1) but
that is for the GridWay software user rather than the VO user.
• As a further consequence of the lack of support for the latest GLUE standard, none
of current metaschedulers can automatically enforce VOspecific resource usage
polices.
Considering the needs for fully virtualizing resources on the Grid and supporting the
latest grid middlewares and information standards, we chose GridWay as the basic
scheduling framework according to the survey. GridWay (5.2) has the following
advantages when compared with other metaschedulers in the survey:
• Modular software architecture, which makes the scheduling system easy to
customize and extend.
• Specially designed to work on top of Globus, which is used by most of the large
scale grids. A noticeable advantage of GridWay is that it can be integrated with
GridGateWay [34] which has the same interface with GT4 GRAM that accepts job
submission described using RSL (and JSDL [31] in the future release), so by
integrating GridWay and GridGateWay there is no need to make any modification
to existing Grid applications that submit jobs to GT4.
• Already supports dynamic resource discovery, thus making sure that the meta
scheduling system can use the uptodate resource information.
• Provides basic automatic scheduling functionalities such as normal requirements
declaration and failure handling.
• Supports both C and Java programming API, which is useful for more advanced
development in the future.
• The builtin scheduling algorithm makes it easy to integrate with the new resource
usage policies based on VO information.
• A very active project, easy to get technical support.
24
30. Figure 1. GridWay Modular Architecture [36]
Execution Manager (EM) is used to interact with Grid Execution Services, responsible
for lowlevel job execution and management. GridWay distribution includes the following
Execution Middleware Access Drivers (MADs):
• PreWS GRAM (Globus Toolkit 2.4 and above)
• WS GRAM (Globus Toolkit 4.0)
Transfer Manager (TM) is used to interface with Grid Data Management Services,
responsible for file staging; remote working directory setup and remote host clean up.
GridWay distribution includes:
• GridFTP (Grid File Transfer Protocol) [37] server (version 1.1.2 and above)
Information Manager (IM) interfaces with Grid Monitoring Services, being responsible
for host discovery and monitoring. The following Information Drivers are included in
GridWay:
• Static host information data, which can be specified in a configuration file.
• Module for communicating with MDS2 with MDS schema (Globus Toolkit 2.4)
• Module for communicating with MDS2 with GLUE schema (Globus Toolkit 2.4
and LCG [37] middleware)
• Module for communicating with MDS4 (Globus Toolkit 4.0)
Scheduler is used to schedule jobs on the available Grid resources. The following
schedulers are included in GridWay:
• RoundRobin/Flooding: This is a very simple scheduling algorithm. It maximizes
the number of jobs submitted to the Grid. Available resources are ordered by rank
and flooded with user jobs in a roundrobin fashion.
• Builtin Scheduling: The builtin scheduling policy uses a weighted sum approach
to determine the priority of a given resource, the resource which has the highest
sum value will be chosen as the final candidate of the job (if the user requirements
can be satisfied with that resource). Current weighted approach includes the fol
lowing subpolicies:
Fixed Resource Priority Policy: Each resource discovered by the informa
tion manager can have a fixed priority assigned by the system adminis
trator
Rank Policy: Each resource can have a dynamic rank which is calculated
via the RANK expression
Usage Policy: The resources can be prioritiesed by current usage data
and historical usage data recorded in the accounting database
26
39. Requirement
SW_REQ Indicates this is a software
requirement expression
SW_NAME SoftwarePackage.Name The name of the software
SW_VERSION SoftwarePackage.Version Version, following the
syntax adopted by that
software
SW_PATH SoftwarePackage.Path The path on where the
software is installed on the
file system.
SW_MODULE SoftwarePackage .Module The name of the module
that gets loaded (to set the
environment) before the
job runs, here are some
example: java/1.5,
repeatmasker/3.2,
mrbayes/02062007, etc.
SW_SERIAL_AVAIL SoftwarePackage.SerialAvail True if the software can
run in serial mode
SW_PARALLEL_VAIL SoftwarePackage.ParalleAvail True if the software can
run in parallel mode
SW_PARALLE_MAX_CPUS SoftwarePackage.ParallelMaxCpu
s
Number of maximum
parallel CPUs that can be
used if this software is
used in parallel mode
Table 5.1. The new variables supporting software requirements and the corresponding GLUE elements
5.3.2 Matchmaking Algorithm: Problem and Solution
The problem we were faced when adding software requirement support to the meta
scheduler is that the software requirements can not be simply added to GridWay's default
requirement expression because the default matchmaking algorithm can not associate
relevant software requirement components. For example, let’s suppose that we have
specified the job requirements as shown below,
REQUIREMENTS = HOSTNAME = “ng2.sapac.edu.au/*” & SW_NAME = “java” & SW_VERSION =
“1.5*”
Which means the user needs to run software “java” with any “1.5” edition on any
SubCluster (represented by the wildcard *) of grid site “ng2.sapac.edu.au”
Let's also suppose the site “ng2.sapac.edu.au” is running software tools as shown in
Figure 8.
35