...

Job Scheduling

Priority

For the job scheduling, Slurm assigns each job a priority, which is calculated based on several factors (Multifactor Priority Plugin). Jobs with higher priority, run firstJobs on albedo are scheduled based on a priority that is computed by Slurm depending on multiple factors (https://slurm.schedmd.com/priority_multifactor.html).
The higher the priority, the sooner your job begins. (In principle – the backfill scheduling plugin helps making best use of available resources by filling up resources that are reserved (and thus idle) for large higher priority jobs with small (lower priority) jobs.)

On albedo,  the priority is mainly influenced by the

• the fairshare factor (which is based on the user’s recent use of resources) and
• the QOS' priority factor and
• the time your job waits in the queue

Job size (RAM, cores), partitions and/or associations have no influence.

Fairshare

On Albedo all users have the same share of resources, independent of the account used. … TODO…

Sebastian Hinck 

Accounting

TODO...

At AWI, only few of the possible factors are taken into account:

Job_priority =   (PriorityWeightAge) * (age_factor)
                 + (PriorityWeightFairshare) * (fair-share_factor)
                 + (PriorityWeightQOS) * (QOS_factor)
                 - nice_factor

                 
The weights in this formula are set to balance the different factors and might become subject for tuning.
The current values can be assessed by running

$ scontrol show config | grep -i PriorityWeight
PriorityWeightAge       = 3500
PriorityWeightAssoc     = 0
PriorityWeightFairShare = 10000
PriorityWeightJobSize   = 0
PriorityWeightPartition = 0
PriorityWeightQOS       = 5000
PriorityWeightTRES      = (null)

The factors (except of the nice_factor (default is zero), which can be set by the user to downgrade the jobs priority by the setting --nice=...), are numbers in the range from 0 to 1.
They are shortly explained in the following.

FairShare

The fairshare factor is the most important factor here, but also the most difficult factor to understand. This factor is calculated using the "classic" fairshare algorithm of Slurm (https://slurm.schedmd.com/classic_fair_share.html). It computes the fairshare for each user based on the recent usage of the system.
Note, the usage of your associated account is *not* taken into accunt here, as it was the case on ollie!
Usage is basically "CPU seconds", but weighted using the UsageFactor depending on the used QOS (see section QOS). Furthermore, the usage taken into account here decays with time (with a half life time of 7 days).
Fairshare is the calculated by

FS = 2^(- (U_N / S_N) / D),

where the normalized usage U_N is the own usage relative to the total usage of albedo, the normalized share S_N is the share of a user on the entire system (1/(number of albedo users)) and D is a dampening factor. The formula basically assigns users a fairshare > 0,5 who under-use their share and < 0,5 for users who over-use their share. This is shown in the following figure, where the dots are taken from historic data from ollie. D has to be adjusted to account for the many users with an HPC account, who don't use it. This might also need tuning.

Image Added

Fairshare values can be shown with the command

sshare

QOS

To reward usage of the short 30min QOS for jobs, which are easier to schedule, the priority is increased!
See section about QOS.

Age

Job's priority slowly increases with waiting time in the queue.
With the current setting the priority is increased by 500 for each day waiting.
The factor saturates after 7 days.
Note: Jobs waiting for a dependency to finish are not ageing.Sebastian Hinck 

Useful Slurm commands

• sinfo shows existing queues
• scontrol show job <JobID> shows information about specific job
• sstat <JobID> shows resources used by a specific job
• squeue shows information about queues and used nodes
• smap curses-graphic of queues and nodes
• sbatch <script> submits a batch job
• salloc <resources> requests access to compute nodes for interactive use
• scancel <JobID> cancels a batch job
• srun <ressources> <executable> starts a (parallel) code
• sshare and sprio give information on fair share value and job priority

...