day2 C

C/02_pingpong.ipynb

@@ -95,7 +95,9 @@
   {
    "cell_type": "markdown",
    "id": "55c185ae-cc9d-4fef-a837-fa43ce81db70",
-   "metadata": {},
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
    "source": [
     "## 0. Step by step (according to the pictures above)"
    ]
@@ -201,7 +203,9 @@
   {
    "cell_type": "markdown",
    "id": "d8383b74-f20b-4864-903a-ab7cd45e25c9",
-   "metadata": {},
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
    "source": [
     "## 1. ping"
    ]
@@ -317,6 +321,7 @@
    "cell_type": "markdown",
    "id": "912708e2-cb48-47d4-bcb7-ab9864d1bd92",
    "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
@@ -451,6 +456,7 @@
    "cell_type": "markdown",
    "id": "2e3ccf5a-5a3c-4d46-acf1-cf371adb8c92",
    "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
@@ -557,6 +563,7 @@
    "cell_type": "markdown",
    "id": "c7527f1b-a008-4f95-b14b-7c812172f372",
    "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
@@ -695,6 +702,7 @@
    "cell_type": "markdown",
    "id": "dda10e76-e3ad-49d4-b354-30e04a4b643b",
    "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
@@ -821,7 +829,9 @@
   {
    "cell_type": "markdown",
    "id": "dc77ac5f-5f9d-4898-809f-e9a9dfe943f4",
-   "metadata": {},
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
    "source": [
     "#### Solution (please try to solve the exercise by yourself before looking at the solution)"
    ]
@@ -920,7 +930,9 @@
   {
    "cell_type": "markdown",
    "id": "cfe5e118-ea85-41e7-b147-6c86cad0df96",
-   "metadata": {},
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
    "source": [
     "#### Expected output - What did you measure? Run is a couple of times to see run to run variations!"
    ]
@@ -945,6 +957,7 @@
    "cell_type": "markdown",
    "id": "8430bfc5-c015-4d8b-851d-ac078efed5f4",
    "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
     "tags": []
    },
    "source": [
@@ -1054,7 +1067,9 @@
   {
    "cell_type": "markdown",
    "id": "04914cc2-fdde-4b18-bacb-e1b080368fa1",
-   "metadata": {},
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
    "source": [
     "#### Solution (please try to solve the exercise by yourself before looking at the solution)"
    ]
@@ -1159,9 +1174,11 @@
   {
    "cell_type": "markdown",
    "id": "52981998-f44a-44f8-85a6-88d3b56fcc63",
-   "metadata": {},
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
    "source": [
-    "#### Expected output - What did you measure? Run is a couple of times to see run to run variations!"
+    "#### Expected output - What did you measure? Run it a couple of times to see run to run variations!"
    ]
   },
   {
@@ -1183,7 +1200,9 @@
   {
    "cell_type": "markdown",
    "id": "e2fb16a7-5e88-4709-994b-2917ee452ade",
-   "metadata": {},
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
    "source": [
     "## 5. finish - who wins the race?"
    ]
@@ -1223,6 +1242,7 @@
    "source": [
     "First name:     ________\n",
     "Measurement on: ________\n",
+    "Programming language: C\n",
     "time for 1 ping in micro seconds with     MPI_Send     MPI_Ssend\n",
     "including first ping pong  in  timing     ________     ________\n",
     "excluding first ping pong from timing     ________     ________"
@@ -1252,14 +1272,6 @@
     "  \n",
     "- <b>Contributing and error reporting:</b> Please send an email to: [training@vsc.ac.at](mailto:training@vsc.ac.at)"
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "f2229195-0e73-4940-97d8-2467b4fb3b6a",
-   "metadata": {},
-   "source": [
-    "&nbsp;"
-   ]
   }
  ],
  "metadata": {

C/04_allreduce.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "7b8b3b7c-e15e-466d-b324-a5ce3fd18359",
+   "metadata": {},
+   "source": [
+    "# MPI - MPI_Allreduce (ring)!\n",
+    "\n",
+    "<details>\n",
+    "    <summary markdown=\"span\"><b>Jupyter notebook quick start (click to expand)</b></summary>\n",
+    "\n",
+    "- \\<Shift>+\\<Return> --> runs a cell\n",
+    "- ! --> shell escape [! Linux command line]\n",
+    "- use a (above) and b (below) left of the [ ] to open new cells\n",
+    "- use m (Markdown) or r (Raw) left of the [ ] to change the cell mode\n",
+    "\n",
+    "<details>\n",
+    "    <summary markdown=\"span\"><b>Click to expand</b></summary>\n",
+    "\n",
+    "If you see a little triangle you can click it to expand further explanations."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f77b0a6b-3cd6-4b56-b1d1-cd480029188a",
+   "metadata": {},
+   "source": [
+    "<details>\n",
+    "    <summary markdown=\"span\"><b>Author, acknowledgment, copyright, and license for this notebook</b></summary>\n",
+    "\n",
+    "- <b>Author:</b> Claudia Blaas-Schenner (VSC Research Center, TU Wien), 18 November 2024\n",
+    "\n",
+    "- <b>Based on</b> the [MPI course developed by Rolf Rabenseifner, HLRS](https://www.hlrs.de/training/self-study-materials/mpi-course-material) that is under a quite restrictive copyright by Rolf Rabenseifner and HLRS. The copyrighted material (some images, some exercise descriptions, some code snippets) is used with permission. Some parts taken from the HLRS material are modified and the Jupyter Notebook is extended with own material of the Notebook authors.\n",
+    "\n",
+    "- <b>License:</b> [CC BY-SA 4.0 (Attribution-ShareAlike)](https://creativecommons.org/licenses/by-sa/4.0/)\n",
+    "  \n",
+    "- <b>Contributing and error reporting:</b> Please send an email to: [training@vsc.ac.at](mailto:training@vsc.ac.at)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6b3ee1b9-4262-4c30-83d3-7fd51436e705",
+   "metadata": {},
+   "source": [
+    "#### MPI Standard: [MPI: A Message-Passing Interface Standard Version 4.1 (PDF)](https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5d223967-4a35-407d-a2f7-bdbb8c23f2b9",
+   "metadata": {},
+   "source": [
+    "#### You can edit the exercises directly in the cells.\n",
+    "&nbsp;"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "01ac1316-a516-477c-b5f0-ae32bbc0bf9e",
+   "metadata": {},
+   "source": [
+    "### Contents - explore collective communication:"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8a6701a3-8e8a-4c96-81c5-0da837632879",
+   "metadata": {},
+   "source": [
+    "- Note, in 03_ring we focused on halo communication, but used just one integer (the ranks) instead of a true halo.\n",
+    "  <br> Now, we step away from the halo communication and will do the previous dummy computation in the most easy way.<br>&nbsp;<br>\n",
+    "- 1. ring - MPI_Allreduce<br>\n",
+    "- 2. ring - MPI_Scan<br>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fbaa8ca6-6fdc-432c-950d-e8eac2d687c3",
+   "metadata": {},
+   "source": [
+    "&nbsp;"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d8383b74-f20b-4864-903a-ab7cd45e25c9",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "## 1. ring - MPI_Allreduce"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "03d9d0ce-79ef-4766-9d0a-beb0ff4c9ffc",
+   "metadata": {},
+   "source": [
+    "##### Rrewrite the pass-around-the-ring program:\n",
+    "\n",
+    "- use the MPI global reduction to get the global sum of all ranks of the processes in the ring, print it from all processes\n",
+    "- the pass-around-the-ring communication loop must be substituted by one call to the MPI collective reduction routine\n",
+    "- please look into the MPI standard to see the argument list of [MPI_Allreduce](https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf#page=282)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "adc73ff7-6447-45ab-9344-8ea2967b841d",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "%%writefile tmp/ring_allreduce.c\n",
+    "\n",
+    "#include <stdio.h>\n",
+    "#include <mpi.h>\n",
+    "\n",
+    "int main (int argc, char *argv[])\n",
+    "{\n",
+    "  int rank, size;\n",
+    "  int snd_buf, rcv_buf;\n",
+    "  int right, left;\n",
+    "  int sum, i;\n",
+    "  MPI_Status  status;\n",
+    "  MPI_Request request;\n",
+    "\n",
+    "  MPI_Init(&argc, &argv);\n",
+    "  MPI_Comm_rank(MPI_COMM_WORLD, &rank);\n",
+    "  MPI_Comm_size(MPI_COMM_WORLD, &size);\n",
+    "\n",
+    "  // ----------------------------------------------\n",
+    "  // ------ please substitute whole algorithm -----\n",
+    "  right = (rank+1)      % size;\n",
+    "  left  = (rank-1+size) % size;\n",
+    "\n",
+    "  sum = 0;\n",
+    "  snd_buf = rank;\n",
+    "  for( i = 0; i < size; i++) \n",
+    "  {\n",
+    "    MPI_Issend(&snd_buf, 1, MPI_INT, right, 17, MPI_COMM_WORLD, &request);\n",
+    "    MPI_Recv  (&rcv_buf, 1, MPI_INT, left,  17, MPI_COMM_WORLD, &status);\n",
+    "    MPI_Wait(&request, &status);\n",
+    "    snd_buf = rcv_buf;\n",
+    "    sum += rcv_buf;\n",
+    "  }\n",
+    "  // ------ by one call to a collective routine ---\n",
+    "  // ------  input is rank, output is sum   -------\n",
+    "  // ----------------------------------------------\n",
+    "\n",
+    "  printf (\"PE%i:\\tSum = %i\\n\", rank, sum);\n",
+    "\n",
+    "  MPI_Finalize();\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6e1c939f-db58-4c80-8871-aab667ada929",
+   "metadata": {},
+   "source": [
+    "##### Compile:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5bf06703-8012-4291-ad59-71d83be99107",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpicc ring_allreduce.c -o ring_allreduce"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5112674f-c2d7-4516-b3c7-ed1121fc4ca4",
+   "metadata": {},
+   "source": [
+    "##### Run:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8dd01859-ef72-40d8-a434-b6a499ef2a6c",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpirun -np 4 ./ring_allreduce"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4e3f19cb-cd03-40f3-a7a4-47ad53c18a93",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "#### Play around with different numbers of MPI processes"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "68f76d2b-0516-45ab-bcd8-a5d9f717666e",
+   "metadata": {},
+   "source": [
+    "**Correct result = size * (size-1) / 2**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8ed5866b-a63a-409c-a388-fa7923b76cad",
+   "metadata": {},
+   "source": [
+    "**Run with different numbers of MPI processes:**<br>\n",
+    "E.g., in the VSC Jupyterhub, there are 4 physical cores available for the MPI course.<br>\n",
+    "If you want to run with more MPI processes you have to add the option:\n",
+    "**--oversubscribe**<br>\n",
+    "*Note: For performance runs you want to avoid oversubscribing, but it's okay for our little examples.*<br>\n",
+    "*Note: You can not do oversubscribing and pinning at the same time.*"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3940eeee-e665-4ded-b75a-5863f15ee222",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpirun -np 12 --oversubscribe ./ring_allreduce"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "912708e2-cb48-47d4-bcb7-ab9864d1bd92",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
+    "tags": []
+   },
+   "source": [
+    "#### Solution (please try to solve the exercise by yourself before looking at the solution)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "10a3ae35-acf5-4cf4-9c94-db94e353637a",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "%%writefile tmp/ring_allreduce_solution.c\n",
+    "\n",
+    "#include <stdio.h>\n",
+    "#include <mpi.h>\n",
+    "\n",
+    "int main (int argc, char *argv[])\n",
+    "{\n",
+    "  int rank, size;\n",
+    "  int snd_buf, rcv_buf;\n",
+    "  int right, left;\n",
+    "  int sum, i;\n",
+    "  MPI_Status  status;\n",
+    "  MPI_Request request;\n",
+    "\n",
+    "  MPI_Init(&argc, &argv);\n",
+    "  MPI_Comm_rank(MPI_COMM_WORLD, &rank);\n",
+    "  MPI_Comm_size(MPI_COMM_WORLD, &size);\n",
+    "\n",
+    "  MPI_Allreduce (&rank, &sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);\n",
+    "\n",
+    "  printf (\"PE%i:\\tSum = %i\\n\", rank, sum);\n",
+    "\n",
+    "  MPI_Finalize();\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "34788895-5728-43df-8d90-12a4685d231b",
+   "metadata": {},
+   "source": [
+    "##### Compile the solution:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6f38f54b-4e24-4d21-bba2-37c322dce283",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpicc ring_allreduce_solution.c -o ring_allreduce_solution"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7566ecac-6878-45aa-b545-36fc0d7e97e3",
+   "metadata": {},
+   "source": [
+    "##### Run the solution:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0af3970c-8691-4b43-ae81-3db977614638",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpirun -np 4 ./ring_allreduce_solution"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "613f39de-a970-4121-981e-b8b13e537296",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "#### Expected output (4 MPI processes):"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "28b4a2d7-32e7-4b8b-b579-c80eca0b56b4",
+   "metadata": {},
+   "source": [
+    "PE0:\tSum = 6\n",
+    "PE1:\tSum = 6\n",
+    "PE2:\tSum = 6\n",
+    "PE3:\tSum = 6"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2be9fcee-1be8-47ef-ad53-6ea650bb4365",
+   "metadata": {},
+   "source": [
+    "&nbsp;"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2e3ccf5a-5a3c-4d46-acf1-cf371adb8c92",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
+    "tags": []
+   },
+   "source": [
+    "## 2. ring - MPI_Scan"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "ef02270b-aa74-4387-976d-5124fedf084f",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "##### Rewrite with MPI_Scan (partial sums) instead of MPI_Allreduce (global sum):\n",
+    "\n",
+    "- please look into the MPI standard to see the argument list of [MPI_Scan](https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf#page=289)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c9258630-975c-41ca-aab4-4e15c76d1e42",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "%%writefile tmp/ring_scan.c\n",
+    "\n",
+    "#include <stdio.h>\n",
+    "#include <mpi.h>\n",
+    "\n",
+    "int main (int argc, char *argv[])\n",
+    "{\n",
+    "  int rank, size;\n",
+    "  int snd_buf, rcv_buf;\n",
+    "  int right, left;\n",
+    "  int sum, i;\n",
+    "  MPI_Status  status;\n",
+    "  MPI_Request request;\n",
+    "\n",
+    "  MPI_Init(&argc, &argv);\n",
+    "  MPI_Comm_rank(MPI_COMM_WORLD, &rank);\n",
+    "  MPI_Comm_size(MPI_COMM_WORLD, &size);\n",
+    "\n",
+    "  MPI_Allreduce (&rank, &sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);\n",
+    "\n",
+    "  printf (\"PE%i:\\tSum = %i\\n\", rank, sum);\n",
+    "\n",
+    "  MPI_Finalize();\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "187c78ba-9dc8-4e43-853a-21a6f4b001ab",
+   "metadata": {},
+   "source": [
+    "##### Compile:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "204eeceb-ec83-4230-b7f7-e75874938018",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpicc ring_scan.c -o ring_scan"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87af2510-d138-4a1f-8a6c-0d96830a27c6",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "##### Run (and sort output):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cd8a2825-0677-4bc0-8ff9-09e6fc190360",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpirun -np 4 ./ring_scan | sort"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2d850084-4336-4b54-b0a3-c4b96042b156",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true
+   },
+   "source": [
+    "#### Play around with different numbers of MPI processes"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "bc45973a-e8cb-472f-aa64-e8101f5e3918",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpirun -np 12 --oversubscribe ./ring_scan_solution | sort"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c7527f1b-a008-4f95-b14b-7c812172f372",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
+    "tags": []
+   },
+   "source": [
+    "#### Solution (please try to solve the exercise by yourself before looking at the solution)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9d483267-69ed-44ee-a2fc-57664ecb2524",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "%%writefile tmp/ring_scan_solution.c\n",
+    "\n",
+    "#include <stdio.h>\n",
+    "#include <mpi.h>\n",
+    "\n",
+    "int main (int argc, char *argv[])\n",
+    "{\n",
+    "  int rank, size;\n",
+    "  int snd_buf, rcv_buf;\n",
+    "  int right, left;\n",
+    "  int sum, i;\n",
+    "  MPI_Status  status;\n",
+    "  MPI_Request request;\n",
+    "\n",
+    "  MPI_Init(&argc, &argv);\n",
+    "  MPI_Comm_rank(MPI_COMM_WORLD, &rank);\n",
+    "  MPI_Comm_size(MPI_COMM_WORLD, &size);\n",
+    "\n",
+    "  MPI_Scan (&rank, &sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD );\n",
+    "\n",
+    "  printf (\"PE%i:\\tSum = %i\\n\", rank, sum);\n",
+    "\n",
+    "  MPI_Finalize();\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3b280691-dbba-4814-8ef5-a90dafb02cad",
+   "metadata": {},
+   "source": [
+    "##### Compile the solution:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "31a63e97-3b84-48ab-a301-92eab829c09b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpicc ring_scan_solution.c -o ring_scan_solution"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f4b34423-8112-468a-8462-bd68c47719d9",
+   "metadata": {},
+   "source": [
+    "##### Run the solution (and sort output):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "2dea2fdb-719d-450b-9cf3-17e19127c405",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "!cd tmp; mpirun -np 4 ./ring_scan_solution | sort"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3d60b10b-fc3b-4e10-9132-876426735494",
+   "metadata": {},
+   "source": [
+    "#### Expected output (4 MPI processes):"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "c0efbbff-1e47-40d5-b79d-9c700d6a2071",
+   "metadata": {},
+   "source": [
+    "PE0:\tSum = 0\n",
+    "PE1:\tSum = 1\n",
+    "PE2:\tSum = 3\n",
+    "PE3:\tSum = 6"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "42ba0a93-f2ad-467c-81fd-ff7c3bcb72de",
+   "metadata": {},
+   "source": [
+    "## &nbsp;"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "87674322-36e6-46c4-be63-ffef70c9cc30",
+   "metadata": {},
+   "source": [
+    "<details>\n",
+    "    <summary markdown=\"span\"><b>Author, acknowledgment, copyright, and license for this notebook</b></summary>\n",
+    "\n",
+    "- <b>Author:</b> Claudia Blaas-Schenner (VSC Research Center, TU Wien), 18 November 2024\n",
+    "\n",
+    "- <b>Based on</b> the [MPI course developed by Rolf Rabenseifner, HLRS](https://www.hlrs.de/training/self-study-materials/mpi-course-material) that is under a quite restrictive copyright by Rolf Rabenseifner and HLRS. The copyrighted material (some images, some exercise descriptions, some code snippets) is used with permission. Some parts taken from the HLRS material are modified and the Jupyter Notebook is extended with own material of the Notebook authors.\n",
+    "\n",
+    "- <b>License:</b> [CC BY-SA 4.0 (Attribution-ShareAlike)](https://creativecommons.org/licenses/by-sa/4.0/)\n",
+    "  \n",
+    "- <b>Contributing and error reporting:</b> Please send an email to: [training@vsc.ac.at](mailto:training@vsc.ac.at)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.8"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tasks/C/03_ring/ring.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  // _____
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+  right = (rank+1)      % size;
+  left  = (rank-1+size) % size;
+
+  sum = 0;
+  snd_buf = rank;
+  for( i = 0; i < size; i++) 
+  {
+    MPI_Send(&snd_buf, 1, MPI_INT, right, 17, MPI_COMM_WORLD);
+    MPI_Recv(&snd_buf, 1, MPI_INT, left,  17, MPI_COMM_WORLD, &status);
+    // _____ WRONG program, it will deadlock with a synchronous communication protocol
+    // _____
+    sum += snd_buf;
+  }
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/03_ring/ring_advanced_irecv_issend.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+#define to_right 201
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+  right = (rank+1)      % size;
+  left  = (rank-1+size) % size;
+
+  sum = 0;
+  snd_buf = rank;
+  for( i = 0; i < size; i++) 
+  {
+    MPI_Issend(&snd_buf, 1, MPI_INT, right, to_right, MPI_COMM_WORLD, &request);
+    MPI_Recv  (&rcv_buf, 1, MPI_INT, left,  to_right, MPI_COMM_WORLD, &status);
+    MPI_Wait(&request, &status);
+    snd_buf = rcv_buf;
+    sum += rcv_buf;
+  }
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/03_ring/ring_advanced_irecv_issend_solution.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+#define to_right 201
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  arr_status[2];
+  MPI_Request arr_request[2];
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+  right = (rank+1)      % size;
+  left  = (rank-1+size) % size;
+
+  sum = 0;
+  snd_buf = rank;
+
+  for( i = 0; i < size; i++) 
+  {
+    MPI_Irecv (&rcv_buf, 1, MPI_INT, left,  to_right, MPI_COMM_WORLD, &arr_request[0]);
+    MPI_Issend(&snd_buf, 1, MPI_INT, right, to_right, MPI_COMM_WORLD, &arr_request[1]);
+    MPI_Waitall(2, arr_request, arr_status);
+    snd_buf = rcv_buf;
+    sum += rcv_buf;
+  }
+
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/03_ring/ring_advanced_irecv_ssend.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+#define to_right 201
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+  right = (rank+1)      % size;
+  left  = (rank-1+size) % size;
+
+  sum = 0;
+  snd_buf = rank;
+  for( i = 0; i < size; i++) 
+  {
+    MPI_Issend(&snd_buf, 1, MPI_INT, right, to_right, MPI_COMM_WORLD, &request);
+    MPI_Recv  (&rcv_buf, 1, MPI_INT, left,  to_right, MPI_COMM_WORLD, &status);
+    MPI_Wait(&request, &status);
+    snd_buf = rcv_buf;
+    sum += rcv_buf;
+  }
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/03_ring/ring_advanced_irecv_ssend_solution.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+#define to_right 201
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+  right = (rank+1)      % size;
+  left  = (rank-1+size) % size;
+
+  sum = 0;
+  snd_buf = rank;
+
+  for( i = 0; i < size; i++) 
+  {   
+    MPI_Irecv(&rcv_buf, 1, MPI_INT, left,  to_right, MPI_COMM_WORLD, &request);
+    MPI_Ssend(&snd_buf, 1, MPI_INT, right, to_right, MPI_COMM_WORLD);
+    MPI_Wait(&request, &status);
+    snd_buf = rcv_buf;
+    sum += rcv_buf;
+  }
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/03_ring/ring_solution.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+  right = (rank+1)      % size;
+  left  = (rank-1+size) % size;
+
+  sum = 0;
+  snd_buf = rank;
+  for( i = 0; i < size; i++) 
+  {
+    MPI_Issend(&snd_buf, 1, MPI_INT, right, 17, MPI_COMM_WORLD, &request);
+    MPI_Recv  (&rcv_buf, 1, MPI_INT, left,  17, MPI_COMM_WORLD, &status);
+    MPI_Wait(&request, &status);
+    snd_buf = rcv_buf;
+    sum += rcv_buf;
+  }
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/04_allreduce/ring_allreduce.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+
+  // ----------------------------------------------
+  // ------ please substitute whole algorithm -----
+  right = (rank+1)      % size;
+  left  = (rank-1+size) % size;
+
+  sum = 0;
+  snd_buf = rank;
+  for( i = 0; i < size; i++) 
+  {
+    MPI_Issend(&snd_buf, 1, MPI_INT, right, 17, MPI_COMM_WORLD, &request);
+    MPI_Recv  (&rcv_buf, 1, MPI_INT, left,  17, MPI_COMM_WORLD, &status);
+    MPI_Wait(&request, &status);
+    snd_buf = rcv_buf;
+    sum += rcv_buf;
+  }
+  // ------ by one call to a collective routine ---
+  // ------  input is rank, output is sum   -------
+  // ----------------------------------------------
+
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/04_allreduce/ring_allreduce_solution.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+
+  MPI_Allreduce (&rank, &sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
+
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/04_allreduce/ring_scan.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+
+  MPI_Allreduce (&rank, &sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD);
+
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}

tasks/C/04_allreduce/ring_scan_solution.c

+
+#include <stdio.h>
+#include <mpi.h>
+
+int main (int argc, char *argv[])
+{
+  int rank, size;
+  int snd_buf, rcv_buf;
+  int right, left;
+  int sum, i;
+  MPI_Status  status;
+  MPI_Request request;
+
+  MPI_Init(&argc, &argv);
+  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
+  MPI_Comm_size(MPI_COMM_WORLD, &size);
+
+  MPI_Scan (&rank, &sum, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD );
+
+  printf ("PE%i:\tSum = %i\n", rank, sum);
+
+  MPI_Finalize();
+}