{
"cells": [
{
"cell_type": "markdown",
"id": "e89be12e-f9da-4c8d-9a7f-9e56bdef6787",
"metadata": {},
"source": [
"# [Slurm- Simple Linux Utility for Resource Management](https://slurm.schedmd.com/archive/slurm-22.05.2)"
]
},
{
"cell_type": "markdown",
"id": "b57ae76b-127f-4456-b2a5-1a50abe42911",
"metadata": {},
"source": [
"<img src=\"images/slurm_logo.png\" style=\"float: right; margin: 1rem; width: 200px; display: block\" />\n",
"\n",
"At VSC we use Slurm to manage our user's workload. Slurm uses many concepts that are important in order to understand how job scheduling works.\n",
"\n",
"First of all there is the notion of **nodes** that represent the available physical hardware. The nodes are then grouped into **partitions** which can be thought of as priority ordered job queues.\n",
"\n",
"A **job** then represents resources that are allocated to a user for a specified amount of time. A **job** itself can contain multiple **job steps** that run (possibly parallel) tasks using the jobs resource allocation. Furthermore a job can be a **batch** job or an **interactive** job. A batch job is a job thats usually accompanied by a batch script that will be executed with the requested resources according to the slurm scheduling algorithm. From the batch script (also the **batch job step**) we can then start multiple tasks (other steps). An interactive job is usually one were you need the requested resources rather fast and directly interact with the requested allocation from the terminal.\n",
"\n",
"To influence the priority of scheduled jobs a **quality of service** can be specified. The **qos** also controls which and how many resources are available when using it, runtime limits, the partitions that can be accessed with it (indirectly) and many more parameters (see [Quality of Service](https://slurm.schedmd.com/qos.html)).\n",
"\n",
"At VSC we partition the hardware based on CPU architecture, memory, and available GPUs (e.g. `zen3_0512`, `zen2_0256_a40x2`, ...). There is always a qos with the same name as the partition which gives the users *basic* access to the hardware in the partition. For testing there are also qos with a `_dev` postfix. They have a very limited maximum runtime but a higher priority, so small test scripts will run earlier."
]
},
{
"cell_type": "markdown",
"id": "56ea6b52-fd3d-44df-aa14-b0bbcd344bcf",
"metadata": {},
"source": [
"## Utilities\n",
"\n",
"Slurm comes with a couple of command line utilities to get information about the system, partitions and jobs. Please refer to the [slurm documentation](https://slurm.schedmd.com/man_index.html) for the extensive set of options for each tool.\n",
"\n",
"- `sbatch` - submit a job script for later execution\n",
"- `srun` - blocking submit a job for execution or submit (possibly parallel) job steps if running in a job script\n",
"- `salloc` - allocate resources for a job (request & wait)\n",
"- `scancel` - cancel (or send a signal) to a pending or running job\n",
"- `sinfo` - reports the state of partitions and nodes\n",
"- `sacct` - display tracked information about a job/job step\n",
"- `scontrol` - view (or modify - root required) slurm entities (e.g. job, node, partition, reservations, ...)\n",
"- `squeue` - display jobs in the queue"
]
},
{
"cell_type": "markdown",
"id": "7b608aa0-d0b3-432a-b69b-f17fdcb5f85a",
"metadata": {},
"source": [
"### sqos\n",
"\n",
"`sqos` is a custom utility we provide to users of VSC. It displays information about the *qos* a user can access such as 'qos type' ('cpu' or 'gpu'), total, used and free resources, maximum walltime, priority and an *estimate* of available resources in nodes."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4fc9c6ec-9442-40be-8f56-35229d5c933a",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!sqos"
]
},
{
"cell_type": "markdown",
"id": "e4bda2d0-9046-45e3-a2bd-bb527f752f44",
"metadata": {},
"source": [
"### Display user's jobs with `squeue`"
]
},
{
"cell_type": "markdown",
"id": "45ab1246-a6c0-4c1c-b52a-b14c92d3bdbe",
"metadata": {},
"source": [
"The current user's jobs can be display by using squeue with the `--user` (`-u`) parameter and the shell variable for the current user `$USER`. You may want to adjust the format string for this tool since the default often cuts off the things we are interested in (e.g. `export SQUEUE_FORMAT='%.18i %.25P %.25j %.12u %.2t %.10M %.6D %R'` - more info about the variables are in the the `squeue` man page):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d91d0697-3d96-4d07-b705-ba82984d4d8b",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!squeue -u $USER"
]
},
{
"cell_type": "markdown",
"id": "80a28776-e7d4-4c0d-ba4a-b3734b0f912f",
"metadata": {},
"source": [
"### Display job information with `scontrol`\n",
"\n",
"`scontrol` can show information about many slurm entities. \n",
"\n",
"Here we use it to display the parameters and state of the job the notebook itself is running in by specifying the subcommand `show` with the `job` entity and the notebooks job id."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b719cec5-627f-4e0d-9767-31e469648c2d",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!scontrol show job $SLURM_JOB_ID"
]
},
{
"cell_type": "markdown",
"id": "8f36f372-87c1-4ace-a3f9-d81b2c369c5f",
"metadata": {},
"source": [
"### Cancel a job with `scancel`\n",
"\n",
"A job can be canceled by using the `scancel` command together with the job id\n",
"\n",
"```bash\n",
"$ scancel <JOBID>\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "f30c3327-ec4c-487f-a397-f2b396071b31",
"metadata": {
"tags": []
},
"source": [
"## Submitting jobs with `sbatch`\n",
"\n",
"The command `sbatch [options] job_script` is used to submit jobs to Slurm's job queue, where `options` stands for additional options given to Slurm and `job_script` is the Bash script that details how to execute the job. The output of a job is written to `slurm-<job_ID>.out` in the directory from which the job has been submitted as a default. \n",
"\n",
"Job scripts have to start with a shebang (e.g. `#!/bin/bash`), followed by `#SBATCH` directives and then the actual script. The Slurm options given in the job script can also be directly passed to `sbatch` if required.\n",
"\n",
"The batch script itself runs on the *batch host* and in its own *job step*. If we just want to run a single program with all resources we allocated it is enough to simply run the program (note: all directives also have short versions)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dbb012ba-1bce-4ee7-ab8c-8c6da811911d",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"%%writefile temp/slurm/job-simple.sh\n",
"#!/bin/bash\n",
"\n",
"#SBATCH --job-name=job-simple\n",
"#SBATCH --output=temp/slurm/slurm-%j.out\n",
"\n",
"# set context of resource allocation (e.g. billing, max walltime, priority, ...)\n",
"#SBATCH --account=p70824 # set the account to use (for billing)\n",
"#SBATCH --qos=zen3_0512 # qos\n",
"#SBATCH --reservation=training # during training we have a fixed reservation\n",
"\n",
"# resource allocation\n",
"#SBATCH --nodes=1 # how many nodes to allocate\n",
"#SBATCH --partition=zen3_0512 # hardware to use\n",
"#SBATCH --time=00:05:00 # maximum time of 5 min for testing\n",
"\n",
"./tooling/print_task_info.sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8cc2173a-6a0f-4841-bdc4-1588920790f3",
"metadata": {},
"outputs": [],
"source": [
"!sbatch temp/slurm/job-simple.sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a4e470b0-e091-40a4-88a9-a3e719941e35",
"metadata": {},
"outputs": [],
"source": [
"!squeue -u $USER"
]
},
{
"cell_type": "markdown",
"id": "9eb8f188-2232-4448-869b-b97806e98a07",
"metadata": {},
"source": [
"## `sbatch` with `srun`\n",
"\n",
"If there is the need to run multiple tasks with restricted resources in parallel and possibly on multiple nodes it is necessary to use `srun` (or a similar mechanism) to start the tasks. The batch script that is initially executed, has **only direct access to the batch host**. When using `srun` Slurm will then create separate *job steps* for the tasks and give access to the resources as specified by the task layout of the job (or its arguments).\n",
"\n",
"The *default* is one core per task and memory is not restricted but `sbatch` and `srun` will also inherit the user environment they run in as well as respect implicitly set defaults.\n",
"\n",
"It is important to note the distinction between **resource allocation** (what types, how many and for how long) and **task layout** at this point. Both concepts are often mixed and influence each other which doesn't make it easier to understand.\n",
"\n",
"To illustrate resource allocation consider the following simple example job script"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3dbc7784-477f-4807-a06e-a0e57219a997",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"%%writefile temp/slurm/job-resource-vs-task-hello.sh\n",
"#!/bin/bash\n",
"\n",
"#SBATCH --job-name=job-resource-vs-task-hello\n",
"#SBATCH --output=temp/slurm/slurm-%j.out\n",
"\n",
"# set context of resource allocation (e.g. billing, max walltime, priority, ...)\n",
"#SBATCH --account=p70824 # set the account to use (for billing)\n",
"#SBATCH --qos=zen3_0512 # qos\n",
"#SBATCH --reservation=training # during training we have a fixed reservation\n",
"\n",
"# resource allocation\n",
"#SBATCH --nodes=1 # how many nodes to allocate\n",
"#SBATCH --partition=zen3_0512 # hardware to use\n",
"#SBATCH --time=00:05:00 # maximum time of 5 min for testing\n",
"\n",
"# naive use of srun in sbatch\n",
"srun echo \"hello world!\""
]
},
{
"cell_type": "markdown",
"id": "c504794e-f893-482f-afee-4f5e93cb0e0f",
"metadata": {},
"source": [
"### Some questions beforehand\n",
"1. What do we expect to happen?\n",
"1. What did actually happen?\n",
"1. Why did it happen?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2578d868-3168-4cbb-9095-4f658488d6aa",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!sbatch temp/slurm/job-resource-vs-task-hello.sh"
]
},
{
"cell_type": "markdown",
"id": "c6ce6c09-dd7c-4455-9df6-d4d9e32bfe2d",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"### \"I want answers!\""
]
},
{
"cell_type": "markdown",
"id": "a25149bf-65e9-4369-a926-1a3616e9502d",
"metadata": {},
"source": [
"1. Depending on your point of view you would probably expect either 1 (a task for the whole node), 128 (a task for each physical core) or 256 (a task for each logical thread) tasks to run\n",
"1. However we see a different number of lines in the output file.\n",
"1. When we start the job from within jupyterhub, we are actually in a job resource allocation - Slurm environment variables are set! Does anything match the number of lines we see in the output file?"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4e8986e8-ef29-4de6-aa5c-704a2df1529b",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!env | grep \"SLURM_\" | sort"
]
},
{
"cell_type": "markdown",
"id": "64f175de-a179-4b7d-8415-d6c06a371b09",
"metadata": {
"jp-MarkdownHeadingCollapsed": true
},
"source": [
"### \"Ok. So how do we 'fix' this?\""
]
},
{
"cell_type": "markdown",
"id": "03acc47c-5307-4731-a094-3c865cc4f716",
"metadata": {},
"source": [
"The only way to do it correctly from a jupyter notebook running in a slurm resource allocation is to \n",
"- either set the variables explicitly or\n",
"- unset the specific slurm variables manually.\n",
"\n",
"For this reason we provided a small script that can be sourced in a cell or the jupyter terminal: `source tooling/unset_slurm_env.sh`.\n",
"\n",
"If *no environment variables* are set the (VSC) default is to use the number of physical cores on the node as tasks and assign each task one core.\n",
"\n",
"See the revisited script below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "49e837cf-aa3e-4495-8fd1-9a1fce2d5e3e",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"%%writefile temp/slurm/job-resource-vs-task.sh\n",
"#!/bin/bash\n",
"\n",
"#SBATCH --job-name=job-resource-vs-task\n",
"#SBATCH --output=temp/slurm/slurm-%j.out\n",
"\n",
"# set context of resource allocation (e.g. billing, max walltime, priority, ...)\n",
"#SBATCH --account=p70824 # set the account to use (for billing)\n",
"#SBATCH --qos=zen3_0512 # qos\n",
"#SBATCH --reservation=_training # during training we have a fixed reservation\n",
"\n",
"# resource allocation\n",
"#SBATCH --nodes=1 # how many nodes to allocate\n",
"#SBATCH --partition=zen3_0512 # hardware to use\n",
"#SBATCH --time=00:05:00 # maximum time of 5 min for testing\n",
"\n",
"srun ./tooling/print_task_info.sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dbd4be37-c6b2-4fdc-8753-de3380740f42",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!source tooling/unset_slurm_env.sh && sbatch temp/slurm/job-resource-vs-task.sh"
]
},
{
"cell_type": "markdown",
"id": "ab35f479-7f74-450b-afc4-48c26cdeeb73",
"metadata": {},
"source": [
"Counting the lines that contained \"end on host\" after the job has been finished will tell us how many tasks were started."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3ab5704f-8bcc-4002-a9a8-c42b3e80b7eb",
"metadata": {},
"outputs": [],
"source": [
"!cat temp/slurm/slurm-JOBID.out | grep \"end on host\" | wc -l"
]
},
{
"cell_type": "markdown",
"id": "d5b645ad-8bc2-44ba-a84c-20550a7ee320",
"metadata": {},
"source": [
"**What can we observe now?**"
]
},
{
"cell_type": "markdown",
"id": "e8bd2ef6-dbf1-4db7-bd04-0786cd27f531",
"metadata": {},
"source": [
"What we've looked into so far is considered a **full and exclusive** node allocation in VSC. It means that all cores and memory is available to the allocating user exclusively. This is not necessarily true for Slurm in general and something that is explicitly enforced at VSC.\n",
"\n",
"<div class=\"alert alert-info rounded-pill rounded-5\" style=\"margin: auto auto 10px auto; width:650px; text-align: center;\">\n",
" <strong>\n",
" Specifying a number of nodes always results in a full exclusive node allocation in VSC.\n",
" Whether or not the hardware is actually used does not matter.\n",
" </strong>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"id": "33bd1884-74a1-4c39-96d2-b0b1d1508cb5",
"metadata": {},
"source": [
"## Running multiple tasks on multiple nodes\n",
"\n",
"In order to control the task layout we can specify a couple more directives. First we need to tell slurm how many tasks we want to start in total. This can either be done by using `--ntasks` or by specifying `--ntasks-per-node`. The other argument that is needed is `--cpus-per-task`. Please not that the argument needs to be used with `srun` rather then specified as a diretive - the setting is not inherited by default."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "39fb37df-7615-4be9-b4c2-af72db4fff9f",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"%%writefile temp/slurm/job-multi-node-tasks.sh\n",
"#!/bin/bash\n",
"\n",
"#SBATCH --job-name=job-multi-node-tasks\n",
"#SBATCH --output=temp/slurm/slurm-%j.out\n",
"\n",
"###\n",
"# context info for resource allocation (e.g. billing, max walltime, priority, ...)\n",
"#SBATCH --account=p70824 # set the account to use (for billing)\n",
"#SBATCH --qos=zen3_0512 # qos\n",
"#SBATCH --reservation=training # during training we have a fixed reservation\n",
"\n",
"###\n",
"# actual resource allocation for job\n",
"#SBATCH --nodes=2 # how many nodes to allocate\n",
"#SBATCH --partition=zen3_0512 # hardware to use\n",
"#SBATCH --time=00:05:00 # maximum time of 5 min for testing\n",
"\n",
"###\n",
"# task layout of job\n",
"\n",
"#SBATCH --ntasks=8 # 8 tasks in total\n",
"# OR specify\n",
"##SBATCH --ntasks-per-node=4 # 4 tasks per node\n",
"\n",
"# default distribution of tasks\n",
"# as an example we select *:block here which means that tasks\n",
"# are assigned to nodes using the default algorithm and cpus\n",
"# will be assigned to tasks based on the socket (=numa in our setup)\n",
"# see https://slurm.schedmd.com/sbatch.html#OPT_distribution\n",
"#SBATCH --distribution=*:block\n",
"\n",
"###\n",
"# start tasks\n",
"# see https://slurm.schedmd.com/srun.html#OPT_cpus-per-task\n",
"srun --cpus-per-task=64 --output=temp/slurm/slurm-%j-task-%t.out ./tooling/print_task_info.sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14e91c8b-43c8-407d-b62b-bb86091c94fa",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!source tooling/unset_slurm_env.sh && sbatch temp/slurm/job-multi-node-tasks.sh"
]
},
{
"cell_type": "markdown",
"id": "a82499da-2da1-412a-bc4e-1e916823a633",
"metadata": {},
"source": [
"## A word about task layout and CPU architecture"
]
},
{
"cell_type": "markdown",
"id": "7d4a6728-cd81-4fbe-9989-a3bfb736843a",
"metadata": {},
"source": [
"When thinking about a task layout for our application we also have to think about the cpu architecture. The one we are using right now on VSC-5 is called **Zen3**. The processor model we have in our regular compute nodes is the [AMD EPYC 7713](https://www.amd.com/en/product/10916) and each node has two of them. We can inspect the system with the following commands:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d5c53819-273e-4db1-8af2-62203ee56a1c",
"metadata": {},
"outputs": [],
"source": [
"!lscpu"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "31f2c3d3-4e5d-40ea-af25-9eb6229315c6",
"metadata": {},
"outputs": [],
"source": [
"!numactl --hardware"
]
},
{
"cell_type": "markdown",
"id": "303371e4-9466-4438-906b-490cd96b51b1",
"metadata": {},
"source": [
"<div class=\"alert alert-info rounded-pill rounded-5\" style=\"margin: auto auto 10px auto; width:650px; text-align: center;\">\n",
" <strong>\n",
" Lets have a look at the Topography image generated with lstopo (\"hwloc\" package)<br/>\n",
" in images/vsc5_zen3_topo.svg\n",
" </strong>\n",
"</div>\n",
"\n",
"Lets spell this out:\n",
"- 2 sockets with 2 **processors** (= 128 cores / 256 PU's per system)\n",
"- Each processor has 4 **NUMA nodes** that connects to memory channels (= 64 cores / 128 PU's per processor)\n",
"- Each NUMA node has 2 **core complex dies** (= 16 cores / 32 PU's per NUMA node; sharing the interconnect & memory)\n",
"- Each core complex die in a NUMA node has 8 **cores** and share 32 MB of L3 cache (= 8 cores / 16 PU's per Group; sharing L3 cache)\n",
"- Each core has their own 32 KB L1d, L1i and 512 KB L2 caches, shared by 2 **processing units (PU's)**\n",
"\n",
"If we compare this to the output of a node configuration in Slurm below we will see similarities\n",
"- 256 CPU's (= 256 PU's)\n",
"- 16 CoresPerSocket (= 2x 8 core complexes per NUMA Node)\n",
"- 8 Sockets (= NUMA nodes)\n",
"- 2 ThreadsPerCore (= 2 PU's per core)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8bfaf500-7cb7-43ba-8d0c-189b27556d22",
"metadata": {},
"outputs": [],
"source": [
"!scontrol show node n3503-022 | grep -E \"CoresPerSocket|Sockets|CPU|ThreadsPerCore\""
]
},
{
"cell_type": "markdown",
"id": "242c048f-ebd0-4f07-a7e3-4278f7b5935a",
"metadata": {},
"source": [
"The architecture of the systems we are using always has to be considered when deciding on task layouts and mappings. \n",
"\n",
"For example some loads could profit to run 2 parallel computations in 2 PU's that are next to each other to share e.g. L1 & L2 caches. Some loads need more memory and hence it would be better to distribute the processes onto core complexes, so every process has their own L3 caches. If the load is memory bound it could also make more sense to distribute the processes so that each process has direct access to its own memory channel etc ..."
]
},
{
"cell_type": "markdown",
"id": "0c7e8c27-3467-4052-b335-914b746f7124",
"metadata": {},
"source": [
"## Partial node allocation\n",
"\n",
"For some smaller tasks it makes sense to only allocate a partial node.\n",
"\n",
"To achieve this we **must not** specify the `--nodes` directive on VSC and instead only use `--ntasks`. To request a specific number of cores we simply set `--ntasks` to the needed number of physical cores (where 1 task equals 1 physical core).\n",
"\n",
"Additionally the `--mem` parameter needs to be specified. The default would be to allocate all node memory but this is forbidden by our VSC job plugin.\n",
"\n",
"<div class=\"alert alert-warning rounded-pill rounded-5\" style=\"margin: auto auto 10px auto; width:650px; text-align: center;\">\n",
" <strong>\n",
" Please note that using more memory than allocated will result<br/>\n",
" in automatic job termination by the operating system!\n",
" </strong>\n",
"</div>\n",
"\n",
"Since we don't want to start multiple tasks or have an otherwise complicated task layout in such a scenario, it is **often not necessary** to even use `srun` here - directly running in the allocation and using all available resources works just as well."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc5359dd-5bd0-491e-a7a4-4df9130ba8bc",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"%%writefile temp/slurm/job-partial-node.sh\n",
"#!/bin/bash\n",
"\n",
"#SBATCH --job-name=job-partial-node\n",
"#SBATCH --output=temp/slurm/slurm-%j.out\n",
"\n",
"# set context of resource allocation (e.g. billing, max walltime, priority, ...)\n",
"#SBATCH --account=p70824 # set the account to use (for billing)\n",
"#SBATCH --qos=zen3_0512\n",
"#SBATCH --reservation=training # during training we have a fixed reservation\n",
"\n",
"# resource allocation with tasks\n",
"#SBATCH --ntasks=32 # number of tasks directly translates to number of cpus\n",
"#SBATCH --mem=2GB # restrict memory usage to 2GB\n",
"#SBATCH --partition=zen3_0512 # hardware to use\n",
"#SBATCH --time=00:05:00 # maximum time of 5 min for testing\n",
"\n",
"# directly start the task\n",
"./tooling/print_task_info.sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "313b6551-7999-4652-a6e0-5c54d0dff30a",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!source tooling/unset_slurm_env.sh && sbatch temp/slurm/job-partial-node.sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d63ea332-dbc4-4783-aee5-37ac18e7bbfc",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.6"
}
},
"nbformat": 4,
"nbformat_minor": 5
}