Node features

In heterogeneous environments, computing resources are often grouped together into single pools of resources, to make things easier and more accessible. Most applications can run on any type of hardware, so having all resources regrouped in the same partitions maximizes utilization and make job submission much easier, as users don't have dozens of options to choose from.

But for more specific use cases, it may be necessary to specifically select the hardware jobs will run on, either for performance or reproducibility purposes.

To that end, all the compute nodes on Sherlock have feature tags assigned to them. Multiple characteristics are available for each node, such as their class, CPU manufacturer, generation, part number and frequency, as well as Infiniband and GPU characteristics.

Available features#

The table below lists the possible features defined for each node.

Listing the features available in a partition#

All the node features available in a partition can be listed with sh_node_feat command.

For instance, to list all the GPU types in the gpu partition:

$ sh_node_feat -p gpu | grep GPU_SKU
GPU_SKU:P100_PCIE
GPU_SKU:P40
GPU_SKU:RTX_2080Ti
GPU_SKU:V100_PCIE
GPU_SKU:V100S_PCIE
GPU_SKU:V100_SXM2

To list all the CPU generations available in the normal partition:

$ sh_node_feat -p normal | grep CPU_GEN
CPU_GEN:BDW
CPU_GEN:MLN
CPU_GEN:RME
CPU_GEN:SKX

Requesting specific node features#

Those node features can be used in job submission options, as additional constraints for the job, so that the scheduler will only select nodes that match the requested features.

To specify a node feature as a job constraint, the -C/--constraint option can be used.

For instance, to submit a job that should only run on an AMD Rome CPU, you can add the following to your job submission options:

#SBATCH -C CPU_GEN:RME

Or to make sure that your training job will run on a GPU with 80GB of GPU memory:

#SBATCH -G 1
#SBATCH -C GPU_MEM:80GB

Multiple constraints#

For more complex cases, multiple constraints could be composed in different ways, using logical operators.

#SBATCH -C 'CPU_MNF:INTEL&IB:HDR'

error: Job submit/allocate failed: Requested node configuration is not available

Some of the possible logical operations between constraints are listed below:

AND#

Only nodes with all the requested features are eligible to run the job. The ampersand sign (&) is used as the AND operator. For example:

#SBATCH -C 'GPU_MEM:32GB&IB:HDR'

will request a GPU with 32GB of memory on the HDR Infiniband fabric to run the job.

OR#

Only nodes with at least one of specified features will be eligible to run the job. The pipe sign (|) is used as the OR operator.

In multi-node jobs, it means that nodes allocated to the job may end up having different features. For example, the following options:

#SBATCH -N 1
#SBATCH -C "CPU_GEN:RME|CPU_GEN:MLN"

may result in a two-node job where one node as an AMD Rome CPU, and the other node has a AMD Milan CPU.

Matching OR:#

When you need all nodes in a multi-node job to have the same set of features, a matching OR condition can be defined by enclosing the options within square brackets ([,]).

For instance, the following options may be used to request a job to run on nodes with the same frequency, either 2.5 GHz or 2.75GHz:

#SBATCH -C "[CPU_FRQ:2.50GHz|CPU_FRQ:2.75GHz]"

#SBATCH -C "GPU_CC:8.0|GPU_CC:8.6"

For more information, complete details about the --constraints/-C job submission option and its syntax can be found in the official Slurm documentation.