Using organization Python package in Github actions without Python repository

posted on 2022-03-04

Here is the scenario: you have two private Github repositories, one with a Python package, another with a dependent project. And you do not have access to a Python repository you can use to host the package.

One solution is to set up direct access from one repository to the other using a deployment key and have Python Poetry grab the package from the git repository directly. This blog post will go through the steps to set that up.

Set up, or have, two projects

Project super will use project sub as a dependency:

poetry new super
peotry new sub

Push both repositories to github. You can use private repositories here, because we are going to use a git key to get access to the code directly.

Now to use sub in the project super we add the following to the pyproject.toml of the super project:

[tool.poetry.dependencies]
python = "^3.8"
sub = { git = "git@github.com:myorg/sub.git", branch = "main" }

Test if the configuration is correct by running poetry install for super. This should just install everything as if a normal package if you have access to the sub github repository. One issue is that if we try to do this in a Github Action it will fail, because the action from super does not have access to the sub repository.

We are going to solve this by creating an SSH keypair and configure a deployment key on sub to allow the SSH key to read the repository:

ssh-keygen -f sub_deploy_key

Go to the Settings for sub in Github, Deploy keys, Add deploy key. Submit the contents of the sub_deploy_key.pub and allow read-only access.

The private part of the key, the contents of sub_deploy_key we put into a secret in our super repository: Settings, Secrets, Actions, New repository secret. Here create a secret called SUB_DEPLOY_KEY.

The last part of the puzzle is adding a known hosts configuration for github.com to the action secrets of super as well, this will allow us to verify that it's really github we are talking to when we use the key to clone the sub project. You can get this value using ssh-keyscan -H github.com. Create a New repository secret with the name KNOWN_HOSTS and set it to:

|1|jA24BueQ+0e5ZykT9nZWZ/n3/kQ=|9ZuWwIV1/6y1UhNwhDdQIr5KQHs= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==

If all is configured, we can now configure the .github/workflows/main.yml file in the super repository:

name: main

on:
  push:
    branches: [main]
  pull_request:

jobs:
  main:
    runs-on: ubuntu-20.04
    env:
      POETRY_VIRTUALENVS_IN_PROJECT: true
    steps:
      - uses: actions/checkout@v3
      - uses: shimataro/ssh-key-action@v2
        with:
          key: ${{ secrets.SUB_DEPLOY_KEY }}
          known_hosts: ${{ secrets.KNOWN_HOSTS }}
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - run: pip install poetry==1.1.12
      - name: cache venv
        uses: actions/cache@v2
        with:
          path: .venv
          key: venv-${{ runner.os }}-${{ hashFiles('**/poetry.lock') }}
      - run: poetry install

And that is it! You should be up and running with two packages with build dependencies building in a Github Action without an intermediate Python repository.

Now I'm not saying this is scalable or even desirable, but it beats having no dependencies at all or having to use git submodules if you ask me.