Skip to content

PyPI Insiders¤

documentation gitpod gitter

Self-hosted PyPI server with automatic updates for Insiders versions of projects.

Motivation¤

Some open source projects follow the sponsorware release strategy, which means that new features are first exclusively released to sponsors as part of an "Insiders" version of the project. This Insiders version is usually a private fork of the public project.

To use the Insiders projects as dependencies, sponsors have two options:

  1. specify the dependency as a Git URL (HTTPS or SSH), or as a direct HTTPS URL to a build artifact
  2. build and store the artifact in a self-hosted PyPI-like index

The first option is problematic when sponsors' projects are also open source, because most of their contributors will probably not have access to the Insiders version. It means they won't be able to resolve the dependency, even less install it locally. As a result, maintainers must specify the public version of the project as a dependency, and override it with the Insiders version in Continuous Integration / Deployment.

In contrast, the second option allows maintainers to specify the dependency normally, i.e. using the same name/identifier as the public version. Maintainers/contributors with access to the Insiders version will resolve and get the Insiders version, while maintainers/contributors without access to the Insiders version will simply get the public one.

However, self-hosting a PyPI-like index, and building artifacts for each new Insiders version is not a trivial, straight-forward task: companies and organizations might already have such a setup (with an Artifactory server, a Google Cloud registry, etc.), but individual contributors often won't, and automatically updating repositories, building artifacts and uploading them requires custom scripts.

In both cases (company setup or individual contributor) PyPI Insiders comes to the rescue, and manages repository/package updates for you. It comes bundled with a PyPI-like index that you can serve locally, and it watches Insiders repositories, building and uploading distributions to your local index (or any other online index) for each new Insiders version getting published.

See below how to install and use PyPI Insiders!

Installation¤

PyPI Insiders is itself an Insiders project, so this is a bit of a chicken-and-egg 🐔🥚 problem, but here's how to install it within itself:

# clone the repository
git clone git@github.com:pawamoy-insiders/pypi-insiders /tmp/pypi-insiders

# we do the following in a subprocess to keep the current one clean
(
    # install the project in a virtualenv
    cd /tmp/pypi-insiders
    python -m venv .venv
    . .venv/bin/activate
    pip install .

    # add pypi-insiders to your watched repositories
    pypi-insiders repos add pawamoy-insiders/pypi-insiders:pypi-insiders

    # start the server and update all watched repositories / packages
    pypi-insiders server start
    pypi-insiders update
)

# reinstall pypi-insiders with pipx
! command -v pipx &>/dev/null && pip install --user pipx
pipx install pypi-insiders --pip-args="--extra-index-url http://localhost:31411/simple"

The local PyPI server should be running:

pypi-insiders server status

Now, if you wish, you can configure your tools to use your local index by default:

export PIP_INDEX_URL=http://localhost:31411/simple
export PDM_PYPI_URL=http://localhost:31411/simple

Your local index will give precedence to its own packages, and redirect to PyPI.org if it doesn't know the specified packages. It means that Insiders versions will always take precedence over public versions, even if the latter are higher (more recent).

Configuring your tools with environment variables makes it easy to temporarily "deactivate" your local index:

# This will install directly from PyPI.org.
PIP_INDEX_URL= pip install something
PDM_PYPI_URL= pdm install

PyPI Insiders is now part of the watched repositories and will be updated automatically in your local index if you run the watcher or when you manually update packages (see below). Remember to upgrade it through pipx as well, from time to time:

pipx upgrade pypi-insiders

Usage¤

Configure the repositories to watch:

pypi-insiders repos add pawamoy-insiders/pypi-insiders:pypi-insiders

The format is NAMESPACE/PROJECT:DISTRIBUTION_NAME. Only GitHub projects are supported for now.

List watched repositories:

pypi-insiders repos list

Remove watched repositories:

pypi-insiders repos remove pawamoy-insiders/pypi-insiders

Start/stop the local PyPI index, get the server status:

pypi-insiders server start
pypi-insiders server status
pypi-insiders server stop

Manually update all packages from watched repositories:

pypi-insiders update

Start/stop the watcher, get the watcher status:

pypi-insiders watcher start
pypi-insiders watcher status
pypi-insiders watcher stop

Show logs of the server/watcher:

pypi-insiders server logs
pypi-insiders watcher logs