Auto-Updating Docker-Compose Images (Using Podman)

Through some simple python coding and systemd services, we have successfully created an automatic update mechanism for Docker-Compose with a Podman backend.

true
2022-07-27

What do we want?

Sykdomspulsen Analytics uses Docker-Compose with a Podman backend.

We use GoCD as the CI/CD tool to build new images, however, deployment of these images is a difficult issue.

Watchtower is a process for automating Docker container base image updates, however, we encountered issues implementing it with a Podman backend.

With our particular setup, each Docker-Compose file is handled via systemd. We need a way to monitor if a new image has been pushed to localhost, which is not consistent with the images currently in use.

Details

/etc/systemd/system/airflow.service is the systemd file that launches the Docker-Compose airflow file. We make sure that the service name is the same as the Docker-Compose folder (e.g. airflow.service and /home/XXXX/images/docker-compose/airflow).

[Unit]
Description=Airflow with Docker Compose
Requires=podman.service
After=podman.service

[Service]
Type=oneshot
RemainAfterExit=true
WorkingDirectory=/home/XXXX/images/docker-compose/airflow
Environment="TMPDIR=/root/tmp"
ExecStartPre=/bin/sh -c "systemctl set-environment HOSTNAME=$(hostname)"
ExecStart=/usr/local/bin/docker-compose up -d
ExecStop=/usr/local/bin/docker-compose down

[Install]
WantedBy=multi-user.target

/etc/systemd/system/composemonitor.service is the systemd file that runs the python script /home/XXXX/images/composemonitor/composemonitor.py.

[Unit]
Description=Monitoring Docker Compose
Requires=podman.service
After=podman.service

[Service]
Type=notify
RemainAfterExit=true
WorkingDirectory=/home/XXXX/images/composemonitor
Environment=PYTHONUNBUFFERED=1
ExecStart=/usr/bin/python -u composemonitor.py
Restart=on-failure

[Install]
WantedBy=multi-user.target

/home/XXXX/images/composemonitor/composemonitor.py which compares the images currently in use with the images in sudo podman images. If these are not consistent, then the service (e.g. airflow.service) is restarted.

It performs this check for the services: airflow.service, gocd.service, and splworkbench.service.

if __name__ == '__main__':
    import subprocess
    import re
    import datetime
    import time
    import systemd.daemon
    
    systemd.daemon.notify('READY=1')
    
    service_names = ["airflow", "gocd", "splworkbench"]
    
    def check_service(service_name):
        print("--------")
        print(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "Checking service: " + service_name)
        
        containers = subprocess.run(["podman", "ps", "-a", "--format", "{{.Names}} {{.Image}} {{.ImageID}}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        containers = containers.stdout.decode('utf-8')
        containers = containers.split()
        
        number_containers = round(len(containers)/3)
        c_container_name = containers[0:len(containers):3]
        c_image_name = containers[1:len(containers):3]
        c_image_id = containers[2:len(containers):3]
        
        index = [i for i, item in enumerate(c_container_name) if re.search("^" + service_name + "_", item)]
        
        c_container_name = [c_container_name[i] for i in index]
        c_image_name = [c_image_name[i] for i in index]
        c_image_id = ['^'+c_image_id[i] for i in index]
        
        needs_update = False
        for i in range(len(c_image_id)):
            image = subprocess.run(["podman", "images", c_image_name[i], "--format", "{{.Id}}"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            image_id = image.stdout.decode('utf-8')
            
            blah = re.search(c_image_id[i],image_id)
            if blah is None:
                print(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), c_container_name[i] + " is out of date")
                needs_update = True
            else:
                print(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), c_container_name[i] + " is up-to-date")
        
        if needs_update:
            print(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "Restarting service " + service_name)
            containers = subprocess.run(["systemctl", "restart", service_name+".service"])
        else:
            print(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "Service " + service_name + " is up-to-date, no action taken")
            
        time.sleep(5)
    
    while True:
        for service_name in service_names:
            check_service(service_name)
        
        print("--------")
        print(datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "Sleeping for 60 seconds")
        
        time.sleep(60)

When running sudo podman images we see that the ID for localhost/airflow:latest is 75b3ca9b3ca2.

We then update the airflow image, with new ID 4942b104ded1.

We check sudo systemctl status composemonitor.service and see that it has detected that the airflow images are out-of-date. The service airflow.service is then rebooted, and the airflow images in use are updated.

We see that sudo podman ps -a shows that the airflow containers have recently been restarted.

Conclusion

Through some simple python coding and systemd services, we have successfully created an automatic update mechanism for Docker-Compose with a Podman backend.

Corrections

If you see mistakes or want to suggest changes, please create an issue on the source repository.

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY 4.0. Source code is available at https://github.com/sykdomspulsen-org/sykdomspulsen-org.github.io, unless otherwise noted. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

Citation

For attribution, please cite this work as

White (2022, July 27). Sykdomspulsen: Auto-Updating Docker-Compose Images (Using Podman). Retrieved from https://docs.sykdomspulsen.no/posts/2022-07-27-auto-updating-docker-compose-images/

BibTeX citation

@misc{white2022auto-updating,
  author = {White, Richard Aubrey},
  title = {Sykdomspulsen: Auto-Updating Docker-Compose Images (Using Podman)},
  url = {https://docs.sykdomspulsen.no/posts/2022-07-27-auto-updating-docker-compose-images/},
  year = {2022}
}