Various container runtimes exist on Linux to run software without installing packages on the host system. The nature of containers implies separation of the host system which sometimes is a gap that needs to be bridged again. For systemd services the "portable service" format allows to run a service with its own dependencies bundled in a filesystem image. However, like a container it still does not make any CLI tools directly available to the host system. Therefore, a common solution is to copy a set of static binaries to the system to use the same deployment mechanism for the service and the CLI tools. The new systemd-sysext format allows to extend the host system through an overlay that integrates the bundled software similar as traditional packages do. The binaries and config files can be updated and managed through a single sysext image file. A version matching logic allows to ensure that a particular host system version is used for depending on certain features or for dynamical linking. We demonstrate how systemd-sysext helps to extend an immutable host system such as Flatcar Container Linux, both for third party user software as well as an internal building block for more modularity.
Linux package managers provide a mature way to install, update, and remove additional software on a system. If using packages is not possible or wanted, containers get used, e.g., through Docker. Containers come with upsides, such as reduced dependency requirements and increased isolation, but also downsides because they don't integrate with the system as well as packages would do. There are workarounds like creating systemd services that start the container and expose expected APIs to the system and using a wrapper script to make the container behave like the CLI tool does if installed through a package. Another approach is to use statically linked binaries for the service and the CLI tools or at least for CLI tools to complement a container. The systemd project introduced support for “portable services” which addresses the integration of a service with the system. It provides a way to set up systemd services from a container-like filesystem image that contains the systemd service definition and the required binaries and dependencies. The recent systemd-sysext format aims to address the extension of the system with additional CLI tools. It works by managing overlay mounts of the sysext images on top of the “/usr” and “/opt” system folders. This has the benefit of bundling a set of binaries inside a single image file that can be added, updated, and removed atomically. There is also a version matching logic that enables safe usage of dynamic linking and depending on certain features of the OS by ensuring that a particular host system version is used. While the primary use case is for deploying additional tools, it can also be used to provide systemd services and their binaries or temporarily overwrite host system files at runtime. Using systemd-sysext for extending the system with additional systemd services requires a small workaround but allows to bundle a service and its CLI tools into a single sysext image. In result, it integrates well with the system and behaves similar to software installed through a regular package.
There are many use cases possible for sysext images, and we will demonstrate some of them for Flatcar Container Linux, which has no package manager but at the same time needs to be open for user customization, selection of container runtimes, and optional cloud vendor tools.