My Azure Virtual Machine Scaleset instances, used for hosting of an application, are filling up due to excessive disk space usage by Docker.
The deployment of the application is based on an Azure DevOps pipeline that uses a Dockerfile. The intention of the pipeline is to pull the latest source code for the application from Bitbucket, and use it to build the application using Docker so that it can then be deployed using kubernetes (using Azure Kubernetes Service - AKS):
The Dockerfile copies files for the .NET application that are under source control, and uses them to build the application inside the container. I think that what may be happening here is that the application is being copied and built into a container in the Azure container registry, rather than the container image being prepared first and then correctly pushed to the Azure container registry.
Does my assessment of what the pipeline is doing wrong, and what I need to do to remedy it, sound about right? And if so, how should I be doing things in the pipeline in order to correctly build and push an image based on the latest source code to the Azure container registry? I'd appreciate any links to reference information that give me an idea of the correct way to do this.
Guides I've looked at:
I've read this - https://docs.microsoft.com/en-us/azure/devops/pipelines/ecosystems/containers/acr-template?view=azure-devops, but it isn't helping me to understand a good way to build and push a container image that has the required .Net Core dependencies and a built version of the .NET Core app.
The following gives a process of continuous integration and deployment using VSTS and Visual Studio - https://docs.microsoft.com/en-us/azure/devops/pipelines/apps/cd/azure/aspnet-core-to-acr?view=azure-devops, but this isn't what I'm trying to do. I want to be able to fix the deployment by adapting the existing Azure Devops pipeline if it's possible, so that it pulls the latest source code for the application and correctly builds and pushes a docker container image to the Azure container registry.
Details of the server disk space usage that's occurring with the existing build and deployment process
The main use of disk space is in var/lib/docker/overlay2. A number of sub-directories in /var/lib/docker/overlay2 (each named with a large hash) are taking up space. I investigated the largest subdirectory and the main space usage is within its merged sub-directory. The merged subdirectory has a top-level file structure typical of a Linux top-level file structure (e.g. home, lib, tmp, usr etc.). The next largest subdirectory under var/lib/docker/overlay2 is similar i.e. the largest use is in its merged subdirectory.
docker system df on one of the scaleset instances shows 72% of space used by images is reclaimable
Inspecting dangling (i.e. unreferenced) images using docker image ls -f dangling=true only shows 295Mb use by unused images:
docker system prune doesn’t clear much space, but docker system prune -a clears the reclaimable space. The images removes by docker system prune -a all appear to be related to Azure/kubernetes - for example here is the beginning of the output when I check images that have been deleted:
Deleted Images:untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-webhook:prod_20200505.3untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-webhook@sha256:ae17bf0784c2bf340a304e975104ea8dbaf4a92398d25f1cc53606815f8f0464deleted: sha256:2419efef0d87926a6b1c5afd003095a7c915f47cce7980039fe00da6c2d8a62bdeleted: sha256:9d5b880b9f9ae7d70d285dac221ec420516fbee51fabc4046611e0aa4bc67bb3deleted: sha256:2b97796a4d7de2b8c1047f1244b0cbc3a8caf10c75435bb5cddc4800c7c40d45untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-addon-prod:prod_20210325.1untagged: mcr.microsoft.com/azure-policy/policy-kubernetes-addon-prod@sha256:ed990be4cd33c3ead578c2e4f31907ab11f4d5fb582e974cd0529f5965b27c11deleted: sha256:17f17127a71d7bc130274d155617433ffc0a368af61f3a2effb0b4e88ca3938c