Automated testing with GitHub Actions

Better Code, Better Science: Chapter 4, Part 7

This is a section from the open-source living textbook Better Code, Better Science, which is being released in sections on Substack. The entire book can be accessed here and the Github repository is here. This material is released under CC-BY-NC.

Automated testing and continuous integration

Once we have a set of tests for a project, it's important to integrate them into our workflow so that we ensure that new changes to the code don't break the existing code (known as a regression).

One useful way to ensure that the tests are run regularly is to run them automatically every time changes are pushed to version control. This is known as "continuous integration" (CI for short), referring to the fact that it allows changes to be continuously integrated into the main code branch once they are confirmed to pass all of the tests. There are a number of different platforms one can use for CI; we will focus on Github Actions since it is the most tightly integrated into the Github version control system. Testing using CI also has a useful side effect: Since the CI system uses a virtual machine to run the tests, the use of CI for testing ensures that the code can run on a separate machine from the one where it was developed. Because setting up the CI system also requires understanding all of the dependencies that are required for the code to run, the CI setup provides a recipe to run the code on any other system.

Using GitHub Actions

GitHub Actions is a system that allows one to automatically execute workflows in response to events related to any GitHub repository. At the time of writing, Github Actions are free and unlimited for public repositories when using the standard GitHub-based workflow runners.

When setting up an automated action using GitHub Actions, there are two primary decisions to specify:

• What is the workflow that I want to run?

• What are the events that I want to trigger the workflow?

As an example, we will implement a workflow running to execute tests for a simple python package that was generated for this book project, called mdnewline. We start by going to the GitHub Actions tab in the repository, and selecting the "Python Package" option, which creates a workflow that builds and tests a Python package. Generating the workflow results in a file that contains a description of the workflow, located at .github/workflows/python-package.yml. Looking more closely at the workflow file, we can see how it works. The first section specifies the name of the workflow, and defines the events that will trigger the workflow:

name: Python package

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

This section specfies that the workflow will be run any time there is a push or a pull request to the main branch of the repo. The next section sets up workflow jobs:

jobs:
  build:
    name: python-build
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.12", "3.13"]

The runs-on argument tells the workflow runner which virtual machine to use to run the workflows; in this case, we will use the default which is the latest release of Ubuntu Linux. We also tell it which versions of Python we want to test the code on, updating it to test on all current versions that are compatible with the package requirements (Python >= 3.12).

We then specify the actual steps in the workflow. The first steps set up the workflow runner so that it can check out the repository, and set up the Python installation. Since this project uses uv to manage packages, we will use the recommended setup code from the uv documentation for multiple Python versions:

    steps:
      - uses: actions/checkout@v4

      - name: Install uv and set the python version
        uses: astral-sh/setup-uv@v6
        with:
          python-version: ${{ matrix.python-version }}

This installs uv with the appropriate python version. We can then install the project using uv sync, pip install the package within the uv environment, and run the tests:

      - name: Install the project
        run: uv sync --locked --all-extras --dev

      - name: pip install the package
        run: uv pip install .

      - name: Run tests
        # For example, using `pytest`
        run: uv run pytest tests

When we commit and push this workflow file, it is automatically run by Github Actions. If we got to the Actions tab in the repository, we will see that the tests failed, and by looking at the logs we can see that the uv installation process failed:

After fixing the `uv` command, we now get the green light!:

It's nice to advertise our testing to the world, which we can do by adding a status badge to our home page, using the markdown generated by Actions for us:

In the next post we will discuss how to speed up long-running tests.