Draft - Draft - Grid Engine Enterprise Edition - policies and use cases - Draft - Draft

1. Strict priority scheduling

a) Job based strict priority dispatching

The objective of this setup is to implement a strict hierarchy of job classes. Strict means that higher privileged jobs are started before less privileged jobs if appropriate resources are available. Jobs are submitted with -p <priority>. Scheduler uses jobs priority to derive per job functional share. The higher the jobs priority, the higher the functional share, the higher the number of resulting tickets.

HOWTO:

use SHARE_FUNCTIONAL_SHARES=0 in schedd_params section of global configuration sge_conf(5)
specify functional tickets (e.g. 1000000) in scheduler configuration sched_conf(5)

Remarks:

This setup can be used as substitution for the -p <priority> switch in SGE. If no other scheduling scheme is needed the use of SGE is recommended instead.

b) Project based strict priority dispatching

The objective of this setup is to implement a strict hierarchy of job classes. Strict means that jobs of higher privileged job classes are started before less priviledged jobs if appropriate resources are available. Jobs are submitted into a job class by submitting into a corresponding project

HOWTO:

use SHARE_FUNCTIONAL_SHARES=0 in schedd_params section of sge_conf(5)
specify functional tickets (e.g. 1000000) in scheduler configuration sched_conf(5)
add one project(5) for each scheduling relevant project
the hierarchy between different job classes is expressed by assigning different functional shares to each project
access to higher privileged projects can be controlled via project acl/xacl

2. Functional policy - control resource shares at any time

All the setups in this category have in common that the functional policy is used as main policy. Setups using the functional policy as main policy ensure that a certain share is guaranteed to each user, project or department at any time, i.e. jobs of users, projects or departments that have occupied less resources than supposed (functional share) are prefered when dispatching jobs to idle resources. All the same full resource utilization is guaranteed, because unused share proportions are distributed among those users, projects and departments who need the resources. Past resource consumption is not taken into account.

a) User based functional scheduling

The objective of this setup is a certain share assignment of all the resources combined in the SGEEE cluster to different users. FCFS scheduling is used among jobs of the same user.

HOWTO:

use SHARE_FUNCTIONAL_SHARES=1 in schedd_params section of global configuration sge_conf(5)
specify functional tickets (e.g. 1000000) in scheduler configuration sched_conf(5)
add one user(5) for each scheduling relevant user
assign functional shares to each project

b) Project based functional scheduling

The objective of this setup is a certain share assignment of all the resources combined in the SGEEE cluster to different projects. FCFS scheduling is used among jobs of the same project.

HOWTO:

use SHARE_FUNCTIONAL_SHARES=1 in schedd_params section of global configuration sge_conf(5)
specify functional tickets (e.g. 1000000) in scheduler configuration sched_conf(5)
add one project(5) for each scheduling relevant project
access to higher privileged projects can be controlled via project acl/xacl
assign functional shares to each project

c) Department based functional scheduling

The objective of this setup is a certain share assignment of all the resources combined in the SGEEE cluster to different departments. FCFS scheduling is used among jobs of the same department.

HOWTO:

use SHARE_FUNCTIONAL_SHARES=1 in schedd_params section of global configuration sge_conf(5)
specify functional tickets (e.g. 1000000) in scheduler configuration sched_conf(5)
add departments for each scheduling relevant department
assign functional shares to each department, e.g.

3. Share-tree policy - control resource shares over time

All the setups in this category have in common that the share-tree policy is used as main policy. Setups using the share-tree policy as main policy ensure that a certain share is guaranteed to the instances configured in the share-tree over time, i.e. jobs associated to share-tree branches where less resources were consumed in the past than supposed (share-tree share) are prefered when dispatching jobs to idle resources. All the same full resource utilization is guaranteed, because unused share proportions are still available for pending jobs associated to other share-tree branches.

a) Project based share-tree scheduling with FCFS within each project

The objective of this setup is to guarantee over time a certain share assignment of all the resources combined in the SGEEE cluster to different projects. A FCFS scheduling is used among jobs of the same project.

HOWTO:

specify share tree tickets (e.g. 1000000) in scheduler configuration sched_conf(5)
add one project(5) for each scheduling relevant project
setup a share tree using qmon(1) reflecting the structure of all scheduling relevant projects as leafs

assign share tree shares to the projects

root ------ projectA (75)
 +--------- projectB (25)

b) Project based share-tree scheduling with equal share for each user within each project

The objective of this setup is to guarantee over time a certain share assignment of all the resources combined in the SGEEE cluster to different projects. An equal share is targeted among the jobs competing for resources within the same project.

HOWTO:

specify share tree tickets (e.g. 1000000) in scheduler configuration sched_conf(5)
add one user(5) for each scheduling relevant user
add one project(5) for each scheduling relevant project
setup a share tree using qmon(1) reflecting the structure of all scheduling relevant projects as nodes

assign share tree shares to the projects

root ------ projectA (75) ------- default (10)
 +--------- projectB (25) ------- default (10)

add the user "default" as leaf in the tree below each of these projects

c) Project based share-tree scheduling with per user individual shares within each project

The objective of this setup is to guarantee over time a certain share assignment of all the resources combined in the SGEEE cluster to different projects. Individual share assignments per user is needed.

HOWTO:

specify share tree tickets (e.g. 1000000) in scheduler configuration sched_conf(5)
add one user(5) for each scheduling relevant user
add one project(5) for each scheduling relevant project
setup a share tree using qmon(1) reflecting the structure of all scheduling relevant projects as nodes
assign share tree shares to the projects

add each user as leaf to the projects where they have access and assign individual shares

root ------ projectA (75) ------- user1 (5)
 |                         +----- user2 (90)
 |                         +----- user3 (5)
 +--------- projectB (25) ------- user4 (30)
                           +----- user5 (30)
                           +----- user6 (30)

Remarks:

If individual share assignment is required only for some few users the "default" user can be used in combination with individual users below a project node. I.e. the tree above can be condensed into

root ------ projectA (75) ------- default (5)
 |                         +----- user2 (90)
 +--------- projectB (25) ------- default (30)

4. Manual intervening into automated policies

There are cases in which it becomes necessary for the administrator to intervene manually, to break the automated policies (functional, share-tree) in order to prefer a certain group of jobs for some reason. In such cases the override policy is used by the administrator. One could throw in: "Why not simply changing the settings of the automated policy temporarily, if it's not appropriate?".

This is true at least for the functional policy. Not depending past usage, changes on the functional policy have an immediate effect on scheduling behavior. The share-tree policy however is significantly influenced by past usage. To effectuate an immediate, but temporary preferation of all jobs belonging to certain share-tree node entity (department, project, user) one had to make the scheduler "extrude past usage" temporarily. This appears to be quite circumstantial.

The override policy allows temporarily prefering those jobs by directly assigning override tickets, which is much more intuitive. Another benefit of using override tickets over the way of changing the automated policy is that it leaves the automated policy setup for long-term untouched.

There override policy can be used in two different variations. When the SHARE_OVERRIDE_TICKETS=false setting is used in the schedd_params section of sge_conf(5) the full ticket amount assigned to one of the object instances (user, department, project, job) is assigned to each corresponding single job, this is the default. When the SHARE_OVERRIDE_TICKETS=true setting is used these tickets are shared equally between all corresponding jobs.