Testing docker image builds when low level libraries change

Background

In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:

• myorg/python-lib1: produces package myorg.lib1
• myorg/python-lib2: produces package myorg.lib2
• myorg/python-app: produces a package myorg.app, depends on myorg.lib1 and myorg.lib2

In our CI for myorg/python-lib1 and myorg/python-lib1 we run unit tests when PRs come in and publish wheels to PyPI on merges to the main branch.

In our CI for myorg/python-app1 we will build test application images when PRs come, deploy them to a test kubernetes cluster, and allow developers to run integration tests against it. It has a Dockerfile like:

# Builds image reistry.myorg.com/python-appFROM reistry.myorg.com/python-base:latestCOPY . /app# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPIRUN pip install /app

Note: We also have some projects using build-packs to achieve a similar goal.

Issue

Because we only run the integration tests for myorg/python-app1, we can't get as much confidence when a change to myorg/python-lib1 and myorg/python-lib2 happens. We have to wait for the libraries to get published to PyPI and then rebuild the application images. When there's an issue in the library, this usually involves us git reverting and fixing the PyPI version. This is super cumbersome when we have a lot of images that pull in the dependency before we detect it needs to be rolled back.

I'm looking to restructure the way we build images in myorg/python-app1 (and other apps that share this pattern) to support pulling in dependencies based off of the PR branches.

I've looked at a lot of tools for containerization but I haven't seen any strategies for building application containers given a change in a lower level dependency. I'm interested in seeing what strategies exist for this. Hopefully the strategy could be applied to other languages as well, where we have a similar strategies for applications written in languages like Java, Golang, and npm.