D1_02_Python_overview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b111ab2b-5eaa-4b8b-99eb-900e18061de1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Python Overview\n",
    "\n",
    "<img src=\"images/python-logo.png\" style=\"float: left; margin: 1rem; width: 240px; display: block\" />\n",
    "\n",
    "When we talk about *Python* we mostly refer to the programming language itself and not its actual implementation.\n",
    "\n",
    "Since the language specification ([Python Language Reference](https://docs.python.org/3.13/reference/index.html)) is open and available, many took the opportunity to create their own runtime from it.\n",
    "\n",
    "While they all try to adhere to the standard there are many differences that come with their own advantages (or disadvantages)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ebf9cd3-0244-4728-84f0-000a47ff44eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## [CPython](https://github.com/python/cpython)\n",
    "\n",
    "**CPython** is the reference implementation of python as we know it. It is mostly written in C but also comes with parts that are already written in python. Since it is directly developed by the python core developers it usually represents the most advanced runtime that is available.\n",
    "\n",
    "Although the development started much earlier Python version 1.0 was first released in 1994 by Guido van Rossum. The most recent Version to date is **python 3.13** with python 3.14 currently in development. It runs on over 40 platforms, including many Unix-like systems, as well as Windows, Android and many more.\n",
    "\n",
    "A particular implementation detail is the **Global Interpreter Lock** (or GIL). The GIL ensures that within a single python process only one thread processes python bytecode at any time. This and the slow execution performance in general make python not suitable for CPU-intensive or timing-sensitive operations.\n",
    "\n",
    "Nowadays most of the actually performance critical code is usually not written in python itself: A typical pattern we have all seen is that high level functions such as configuration and organization are written in python and then the control is handed over to a library function that performs the calculations using compiled code (with the optional help of multi-threading). This can either be done via C-API Extensions, where the extension needs to release the GIL for the duration of a long running function, or by using external libraries, that are not restricted by the GIL at all.\n",
    "\n",
    "For a long time the only solution to actually have concurrency in python code was to use e.g. the multiprocessing module to run multiple python interpreter processes. Here each python process has its on GIL but at the same time this introduces communication overhead which is just another form of a scaling program.\n",
    "\n",
    "Although Python 3.13 still uses the GIL there have been major changes in the last versions to prepare for a removal:\n",
    "* Version 3.12 introduced the possibility to have multiple subinterpreters per process where each interpreter has its own GIL\n",
    "* Version 3.13 finally introduced a build flag to disable the GIL ([see PEP703](https://peps.python.org/pep-0703/) and also [Whats new in Python 3.13](https://docs.python.org/3.13/whatsnew/3.13.html#free-threaded-cpython). Many newer linux distributions are now providing a separate **nogil** python package to install.\n",
    "\n",
    "However the **nogil** mode should still be considered somewhat experimental and will not give you immediate performance benefits since the removal of the GIL also comes with a slight single-thread performance degradation and most of the calculation intensive code is running in external libraries anyway.\n",
    "\n",
    "Python 3.13 also saw the addition of a very basic JIT (just in time) compiler. The JIT compiler is still disabled by default but they expect to further improve python performance with it in the next releases.\n",
    "\n",
    "In general CPython is a healthy and active project and especially python performance improvements is a major goal these days."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "451ffff0-d7bc-4ce3-9fbb-cade9bacf095",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## [PyPy](https://www.pypy.org/)\n",
    "\n",
    "**PyPy** is an alternative implementation of python. Compared to CPython it uses a JIT (just in time) compiler to optimize hot code paths and thus *can* be a lot faster than the standard implementation, while still maintaining the use of a [GIL](https://doc.pypy.org/en/latest/faq.html?highlight=gil#does-pypy-have-a-gil-why).\n",
    "\n",
    "PyPy is itself developed in RPython and implements **python 2.7 and 3.10** so it lags behind CPython concerning new language features.\n",
    "\n",
    "Regarding CPU architectures and platforms x86, x86-64, ARM, AArch64, PowerPC and System Z are supported and the runtime should work on Linux, MacOS, Windows, OpenBSD and FreeBSD.\n",
    "\n",
    "The project claims to be much faster than CPython and some results are available on the [pypy speed center website](https://speed.pypy.org/).\n",
    "\n",
    "PyPy works best when used with a long running program, where a signification amount of time is spent in **python code**. It has itself no influence on the speed of library integrations and [compatibility issues](https://www.pypy.org/compat.html) can be encountered with some external libraries. The project mentions compatibility with CFFI, cppyy and other popular libraries such as NumPy, Scikit-learn and more.\n",
    "\n",
    "The project is still in development but seems to focus on the improvement of the JIT. Current packages are available on the project website."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2081eee-914f-450f-bc02-37723b0470fa",
   "metadata": {
    "editable": true,
    "jp-MarkdownHeadingCollapsed": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "## Other Variants\n",
    "\n",
    "There are many other implementations than ones described here but most of them are even lesser known, lack in features or have a specific focus e.g. MicroPython for microcontrollers etc.\n",
    "\n",
    "### [GraalPy](https://www.graalvm.org/python/)\n",
    "\n",
    "Oracle also provides a python variant called **GraalPy** that runs on the GraalVM. This also comes with many tools from the Java ecosystem. The project claims that pure python code runs faster than on CPython and performance with C extensions nearly matches that of CPython. The currently supported python version is **3.11**. Packages are available through conda-forge\n",
    "\n",
    "### [Codon](https://github.com/exaloop/codon)\n",
    "\n",
    "**Codon** is a relatively new implementation that compiles python code to machine code and can thus support native multithreading. However there are some noteable differences to python regarding e.g. datatypes, dynamic typing, etc. The project claims to have speedups to vanilla Python in the order of 10-100x or more.\n",
    "\n",
    "### [IronPython](https://ironpython.net/)\n",
    "\n",
    "**IronPython** is an implementation of python targeting the .NET Framework. It is written entirely in C# and implemented on top of the Dynamic Language Runtime.\n",
    "\n",
    "IronPython supports **python 3.4** since December 2022. The project gained popularity when it was supported by Microsoft but has since been abandoned and is currently maintained by volunteers on GitHub.\n",
    "\n",
    "The performance of IronPython seems to be a mixed bag since it is faster in some and worse in others when compared to CPython. However it may perform better in python programs that make extensive use of threads or multiple cores since it uses a JIT compiler and has no GIL.\n",
    "\n",
    "### [Jython](https://www.jython.org)\n",
    "\n",
    "A very similar player on this field is **Jython** a python implementation that runs on the Java Platform. With Jython it is possible to use any existing Java classes. The last release was in August 2024 and implements **python 2.7**. The plan is to support python 3 with the release of [Jython 3](https://www.jython.org/jython-3-roadmap).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8156578f-d4d1-47ad-a54b-54e17dcf0b2f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Note about python version and runtime\n",
    "\n",
    "Unless noted otherwise the performance details and the implementation notes refer to the latest __Python 3.11__ version of the official __CPython__ runtime."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}