image-stack

Contribution Guidelines

If you wish to contribute your own images to the stack, please adhere to the following contribution guidelines:

• Clone the repo and create a new feature branch (branch from dev)
• When naming your new branch keep the naming convention in mind:
  • features/yournewimage-notebook
  • Start with features/...if you are adding a new image start with fix/... if you are editing an existing notebook
  • Name your image – choose a name that does not exist yet
  • It should reflect as much as possible what the image is about, e.g. you would like to add a datascience notebook call it in a way that others know what it could be about
  • At the end add -notebook
  • Here is an example (that already exists): features/base-notebook
• Work on your new image in the branch you created
  • Create a new folder if you add a new image (called like your branch e.g. mynew-notebook)
  • Push commits to this branch – keep commit messages as short and meaningful as possible
  • Your Dockerfile needs to be multi-staged (at least production stage see detailed how to below)
• Check your work (local testing etc.), once you are happy with it open a new Merge Request
  • Add a meaningful description to your merge request
  • Keep it concise to help the project members to understand your changes better
• Only after creating the merge request the pipeline will trigger
• A project member will take a look at your Merge Request and merge it with the dev branch if everything is ok (pipelines are green, contribution guidelines are kept etc.)
  • You need at least one approval by an eligible project member
• If your Merge Request gets accepted, we will make your image available in the dev environment (https://webportal.dev.austrianopensciencecloud.org)
• Keep in mind that it will not yet be available in the beta environment
• You will need to test your new image in dev manually before it can be merged to beta
  • Does it work how you intended it to in the dev environment?
  • Add test reports to your merge request (from dev to beta)
  • Use your original merge request to dev and create a cherry pick merge request form it (target branch now beta)
• Once merged to beta we will make your image available in the beta environment (https://webportal.beta.austrianopensciencecloud.org/)

Currently you need to be added to the project before you can submitt a merge request. You can request access in the project overview.

Detailed How-To

Merge from fix or feature branch to dev:

1. Clone the git repo https://gitlab.tuwien.ac.at/jupyter/image-stack/-/tree/dev
2. Checkout dev branch and create a new local feature or fix branch (depending on how the notebook already exists)
3. Work in this branch on the (new) image:

   • Please read the contribution guidelines and naming convention above

   • New feature branch:

     • Add locally in your new branch a folder for your new image e.g. mynew-notebook
     • In this folder add your multi stage Dockerfile and some tests if needed
     • Your Dockerfile needs at least a production stage like this (see https://docs.docker.com/develop/develop-images/multistage-build/ for reference)
       Here is an example:

     FROM jupyter/base-notebook as base
     ENV LSB_RELEASE="bionic"
     
     FROM base as production
     RUN apt-get update && apt-get install --assume-yes git

     This is needed so that tests can be added if needed and the pipeline runs for all images, with and without tests.

     The teststage would look like this in your docker image for example:

     FROM python:3-alpine as base
     RUN  apk upgrade --update --no-cache
     
     WORKDIR /usr/src/app
     COPY basic.py basic.py
     
     FROM base as build
     RUN pip install pytest &&\
         pip install pytest-custom_exit_code
     
     FROM build as test
     COPY ./tests .
     RUN pytest test_sample.py --junitxml=/tmp/test/report.xml --suppress-tests-failed-exit-code
     
     FROM base as production
     ENTRYPOINT ["python3" ]
     CMD [ "basic.py" ]

     • Add your image to the .gitlab-ci.yml file like this (mynew-notebook):

     stages:
     - build-test-notebook
     - build-r
     - build-base
     - mynew-notebook

     my-new-image:
      stage: mynew-notebook
     variables:
       workdir: mynew-notebook
       TEST: "False"
     trigger:
      include: .gitlab-ci-image.yml
      strategy: depend
     rules:
      -if: ($CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH || $CI_COMMIT_BRANCH == "dev" || $CI_PIPELINE_SOURCE == 'merge_request_event') # no push to dev directly -> first create Merge Request
      changes:
      - mynew-notebook/**/*

     • Add TEST: "True" if you have test files in your folder and a test stage in your docker file (see above how to do that)
     • For "stage: mynew-notebook" it is important that the stage is called the same as in the above "stages:" list where you added it first

   • Fix Branch:
     If you want to fix something in an already existing notebook (e.g. you added a feature branch for a data science notebook some time ago and now want to change something there)

     • Create a new branch the same way from dev but call it fix/your-notebook (instead of features/your-notebook)
     • Work only in the already existing folder, no need to create a new one
     • No changes in the pipeline are necessary

   • Test and build your local changes to see if it works, add and commit your changes

   4. If you are happy with our local branch you can push it to the gitlab server: git push --set-upstream origin features/mynew-notebook
      Note: This will and should NOT trigger a pipeline yet
   5. Create a Merge Request (from your fix or feature branch to dev) for your new branch here: https://gitlab.tuwien.ac.at/jupyter/image-stack/-/merge_requests
      • Once you created the merge request the pipeline will trigger (without publish and tagging stage)
      • You can then check if your tests were successful for instance, check the security section - if there are a lot of (critical, high) vulnerabilities you might need to upgrade the packages, libraries you used in the image
      • Make modifications if necessary within your branch, the pipeline will now be triggered on each push to your branch, changes are visible within the open merge request
   6. An eligible project member can approve your merge request and then merge it → the pipeline will again trigger on merge now with publish and tagging stage
      • The source branch can be deleted (fix or feature branch that has been merged into dev)

Merge from dev to beta:

1. Once the merge to dev is done the notebook has to be tested manually by the contributor to see if everything works as expected in the dev environment
2. Create test reports
3. Go to the original merge request (from fix or feature branch to dev) and click "cherry pick"
4. Select beta as target branch → this should automatically create a new branch, and create a merge request from it (from cherry pick branch to main)
5. Only your changes should be listed in this merge request
6. Once the merge request is created the pipeline should trigger (again without publish and tagging stage)
7. Check if the pipeline is ok
8. If changes are necessary → checkout the cherry pick branch and make your modifications there, once you push them the pipeline triggers again, the changes are too visible in the merge request
9. Approval and merge by an eligible project member necessary
10. On merge the pipeline triggers only for the changed notebook with publish and tagging stage (to beta registry later)