Sealed images: key management for Secure Boot

Unknown — Tue, 05 May 2026 00:00:00 +0000

Sealed images: key management for Secure Boot</h1>
In the previous post</a>, we walked through the security chain that sealed images provide, from firmware through runtime file verification. That chain depends on cryptographic keys at every step. In this post, we'll cover the keys involved, how to generate them, and how they get enrolled into your system's firmware.

UEFI Secure Boot uses a hierarchy of three key types, each serving a distinct role:

Platform Key (PK) -- The root of the Secure Boot trust hierarchy. There is exactly one PK. The PK controls who can modify the Secure Boot key databases. It is used to sign updates to the KEK database. When the PK is enrolled, the firmware transitions from Setup Mode to User Mode, enabling Secure Boot enforcement.

Key Exchange Key (KEK) -- One or more KEKs can be enrolled. A KEK is authorized to sign updates to the signature database (db). The KEK exists to allow delegation: the PK owner can authorize other parties to manage the set of trusted signing keys without giving them full control over the Secure Boot configuration.

Signature Database (db) -- The db contains the keys or hashes that the firmware uses to verify boot binaries. When the firmware loads a bootloader or UKI, it checks the binary's signature against the keys in the db. If the signature matches a db entry, the binary is allowed to execute.

In practice, many organizations will only need to generate a single set of these keys. The db key is the one that does the day-to-day work: it signs your UKIs and bootloader. The PK and KEK exist primarily to control who can modify the db.

If you already have Secure Boot keys provisioned in your infrastructure, you don't need to generate new ones. You only need access to a db key (or the ability to add your signing key to the existing db) in order to sign your UKIs. The rest of this post assumes you're starting from scratch, but the signing steps apply regardless of where your keys came from.

Generating keys</h2>
The following commands generate a complete set of Secure Boot keys. These examples use openssl</code> directly, but you should use whatever key management practices are appropriate for your organization. Hardware security modules (HSMs), vault systems, or other key management infrastructure may be more appropriate for production use.
First, generate a GUID to identify the key owner. This is embedded in the key enrollment data and can be any unique identifier:
$ uuidgen --random > GUID.txt</code></pre> Generate the Platform Key:
$ openssl req -newkey rsa:2048 -nodes -keyout PK.key \ -new -x509 -sha256 -days 3650 \ -subj '/CN=My Platform Key/' -out PK.crt $ openssl x509 -outform DER -in PK.crt -out PK.cer</code></pre> Generate the Key Exchange Key:
$ openssl req -newkey rsa:2048 -nodes -keyout KEK.key \ -new -x509 -sha256 -days 3650 \ -subj '/CN=My Key Exchange Key/' -out KEK.crt $ openssl x509 -outform DER -in KEK.crt -out KEK.cer</code></pre> Generate the Signature Database key:
$ openssl req -newkey rsa:2048 -nodes -keyout db.key \ -new -x509 -sha256 -days 3650 \ -subj '/CN=My Signature Database Key/' -out db.crt $ openssl x509 -outform DER -in db.crt -out db.cer</code></pre> Each command produces three files: a private key (.key</code>), a PEM certificate (.crt</code>), and a DER-encoded certificate (.cer</code>). The private keys are what you need to protect. The certificates are public and are what gets enrolled into the firmware.
Signing boot artifacts</h2> With a db key in hand, you can sign the two boot binaries that need Secure Boot verification: the bootloader (systemd-boot) and the Unified Kernel Image. Signing systemd-boot:
$ sbsign --key db.key --cert db.crt \ --output systemd-bootx64.efi.signed \ /usr/lib/systemd/boot/efi/systemd-bootx64.efi</code></pre> The UKI signing process is more involved because the UKI embeds the composefs digest, which must be computed from the filesystem first. We'll cover building sealed images (including UKI generation and signing) in detail in the next post. For now, the important thing to know is that ukify</code> (the tool that assembles UKIs) accepts the same --secureboot-private-key</code> and --secureboot-certificate</code> options and uses sbsign</code> under the hood to sign the final EFI binary.
Enrolling keys into firmware</h2> The keys need to be enrolled into the UEFI firmware's Secure Boot database before the firmware will use them for verification. There are several ways to do this depending on your environment. Manual enrollment via firmware setup</h3> Most UEFI firmware implementations provide a setup menu (accessed during early boot, typically via a key like F2 or Del) where Secure Boot keys can be managed. The DER-encoded certificates (.cer</code> files) can be enrolled from a USB drive or the EFI System Partition. This is straightforward for individual systems but does not scale well. Programmatic enrollment for virtual machines</h3> For virtual machines using OVMF (the open source UEFI firmware for QEMU/KVM), keys can be enrolled directly into the firmware variable store using the virt-fw-vars</code> tool from the virt-firmware</a> project: $ GUID=$(cat GUID.txt) $ virt-fw-vars \ --input /usr/share/edk2/ovmf/OVMF_VARS.fd \ --secure-boot \ --set-pk $GUID PK.crt \ --add-kek $GUID KEK.crt \ --add-db $GUID db.crt \ -o OVMF_VARS_ENROLLED.fd</code></pre> This produces a firmware variable store with your keys pre-enrolled. When QEMU starts with this variable store, Secure Boot is active and will only accept binaries signed with your db key. This is particularly useful for testing and for VM-based deployments where you control the firmware image. Auto-enrollment via systemd-boot</h3> systemd-boot supports automatic key enrollment when the firmware is in Setup Mode (no Platform Key enrolled). This works by placing signed UEFI authenticated variable files (.auth</code> files) on the EFI System Partition. Creating .auth</code> files requires converting your certificates into UEFI signature lists and signing them with the appropriate parent key: $ GUID=$(cat GUID.txt) $ attr=NON_VOLATILE,RUNTIME_ACCESS,BOOTSERVICE_ACCESS,TIME_BASED_AUTHENTICATED_WRITE_ACCESS # Create EFI signature lists $ sbsiglist --owner $GUID --type x509 --output PK.esl PK.cer $ sbsiglist --owner $GUID --type x509 --output KEK.esl KEK.cer $ sbsiglist --owner $GUID --type x509 --output db.esl db.cer # Sign the variables (PK and KEK signed by PK, db signed by KEK) $ sbvarsign --attr $attr --key PK.key --cert PK.crt --output PK.auth PK PK.esl $ sbvarsign --attr $attr --key PK.key --cert PK.crt --output KEK.auth KEK KEK.esl $ sbvarsign --attr $attr --key KEK.key --cert KEK.crt --output db.auth db db.esl</code></pre> These .auth</code> files can then be placed on the ESP where systemd-boot expects them. bootc can automate this: if you place the .auth</code> files in your container image at /usr/lib/bootc/install/secureboot-keys/<name>/</code>, bootc will copy them to the ESP during installation. On the next boot, systemd-boot can enroll them into the firmware. The enrollment behavior is controlled by the secure-boot-enroll</code> setting in loader.conf</code>. See the systemd-boot documentation</a> for details on the available modes. Key security considerations</h2> A few things worth keeping in mind: Protect your private keys. The .key</code> files are what allow signing boot artifacts. Anyone with access to your db private key can sign binaries that your firmware will trust. Store private keys with appropriate access controls, and consider using an HSM or key management system for production deployments. Key rotation. The examples above use 10-year expiry (-days 3650</code>), which is generous. Plan for key rotation before your certificates expire. The UEFI key hierarchy makes this manageable: you can use your KEK to authorize a new db key without needing to re-enroll the PK. Separation of concerns. In a CI/CD pipeline, the signing step should ideally be isolated from the build step. The build process produces an unsigned UKI, and a separate signing service (with access to the private key) signs it. This limits the blast radius if the build environment is compromised. What's next</h2> With keys generated and an understanding of how they fit into the trust chain, the next post will walk through building a sealed image end-to-end: writing a Containerfile, computing the composefs digest, generating and signing the UKI, and producing a deployable container image.

Containers: pitfalls of incomplete tar archives

Unknown — Mon, 15 Dec 2025 00:00:00 +0000

Containers: pitfalls of incomplete tar archives</h1>
As Colin Walters</a> likes to say, containers are just tarballs wrapped in JSON. We'll mostly ignore the JSON part here, but there's a lot to unpack (pun and foreshadowing intended) with regards to tar and how it interacts with the various pieces that work in conjunction to make containers a reality.

A refresher on

tar</code></h2>
Let's explore a key quirk of tar from which everything else we'll
discuss follows.</p>
First, let's create a basic tar archive, which contains a directory
with a file inside of the directory:</p>

⬢ [jeckersb@toolbx tmp]$ mkdir foo</span></span>
⬢ [jeckersb@toolbx tmp]$ touch foo/bar</span></span>
⬢ [jeckersb@toolbx tmp]$ tar cf foo.tar foo/</span></span></code></pre>
Now let's list the contents of the tar archive:</p>
⬢ [jeckersb@toolbx tmp]$ tar tvf foo.tar</span></span>
drwxr-xr-x jeckersb/jeckersb 0 2025-12-10 11:09 foo/</span></span>
-rw-r--r-- jeckersb/jeckersb 0 2025-12-10 11:09 foo/bar</span></span></code></pre>
Ok, that makes sense.  We see the directory and the file we added to
the archive.</p>
But here's the key bit:  What happens if you add only</em> the file to the archive?</p>
⬢ [jeckersb@toolbx tmp]$ tar cf incomplete.tar foo/bar</span></span>
⬢ [jeckersb@toolbx tmp]$ tar tvf incomplete.tar</span></span>
-rw-r--r-- jeckersb/jeckersb 0 2025-12-10 11:09 foo/bar</span></span></code></pre>
Now the archive has metadata only</em> for the file.  You can deduce from
the path foo/bar</code> that there is a directory foo</code> involved, but you
can't definitively know how foo</code> is supposed to look.  What are the
permissions, owner, group, mtime?  We don't know.  So what happens if
you extract this tar archive?</p>
⬢ [jeckersb@toolbx tmp]$ mkdir incomplete-unpacked && tar xf incomplete.tar -C incomplete-unpacked</span></span>
⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked/</span></span>
total 0</span></span>
drwxr-xr-x. 2 jeckersb jeckersb 60 Dec 10 11:19 foo</span></span>
⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked/foo/</span></span>
total 0</span></span>
-rw-r--r--. 1 jeckersb jeckersb 0 Dec 10 11:09 bar</span></span></code></pre>
It implicitly creates the directory foo</code> in order to put bar</code> at the
correct path.  This makes sense.  But notice foo</code> gets created with
metadata that is dependent on the environment in which the unpacking
of the archive happened.  In the above example, you can see that the
timestamp on the directory is 10 minutes later than the timestamp of
the bar</code> file inside.  This is because it took me 10 minutes of
writing and editing this post before I did the unpack operation.  I
can set my umask to something else and extract it a second time, and
now we'll see a different timestamp and</em> different permissions:</p>
⬢ [jeckersb@toolbx tmp]$ umask 0077</span></span>
⬢ [jeckersb@toolbx tmp]$ mkdir incomplete-unpacked-with-umask && tar xf incomplete.tar -C incomplete-unpacked-with-umask</span></span>
⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked-with-umask/</span></span>
total 0</span></span>
drwx------. 2 jeckersb jeckersb 60 Dec 10 11:25 foo</span></span>
⬢ [jeckersb@toolbx tmp]$ ls -l incomplete-unpacked-with-umask/foo/</span></span>
total 0</span></span>
-rw-------. 1 jeckersb jeckersb 0 Dec 10 11:09 bar</span></span></code></pre>OCI image spec</h2>
OCI container images are serialized as a series of
tarballs</a>
that are applied on top of each other to create the complete
filesystem.  We'll get back to the specifics of that operation a bit
later.</p>
The entire layer specification is worth reading, but we'll focus on
two parts.  First, how to determine changes:</p>
When two directories are compared, the relative root is the top-level directory.</span></span>
The directories are compared, looking for files that have been [added, modified, or removed](#change-types).</span></span>
</span>
For this example, `rootfs-c9d-v1/` and `rootfs-c9d-v1.s1/` are recursively compared, each as relative root path.</span></span>
</span>
The following changeset is found:</span></span>
</span>
Added:      /etc/my-app.d/</span></span>
Added:      /etc/my-app.d/default.cfg</span></span>
Modified:   /bin/my-app-tools</span></span>
Deleted:    /etc/my-app-config</span></span>
</span>
This reflects the removal of `/etc/my-app-config` and creation of a file and directory at `/etc/my-app.d/default.cfg`.</span></span>
`/bin/my-app-tools` has also been replaced with an updated version.</span></span></code></pre>
And then the following section on representing changes:</p>
A [tar archive][tar-archive] is then created which contains _only_ this changeset:</span></span>
</span>
- Added and modified files and directories in their entirety</span></span>
- Deleted files or directories marked with a [whiteout file](#whiteouts)</span></span>
</span>
The resulting tar archive for `rootfs-c9d-v1.s1` has the following entries:</span></span>
</span>
./etc/my-app.d/</span></span>
./etc/my-app.d/default.cfg</span></span>
./bin/my-app-tools</span></span>
./etc/.wh.my-app-config</span></span>
</span>
To signify that the resource `./etc/my-app-config` MUST be removed when the changeset is applied, the basename of the entry is prefixed with `.wh.`.</span></span></code></pre>
But note that the changeset says nothing about the directories higher
in the filesystem.  So in the above example, it would be perfectly
valid (and more correct) to generate changeset tar archives which are
incomplete in the same manner we described above.  Most notably, these
changesets would not</em> contain tar entries for the directories /etc</code>
and /bin</code>.</p>
This oversight in the specification is not new or surprising.  There
is an open issue from
2017</a> which
describes exactly this problem.  There is a linked pull
request</a> which
has been open since late 2022 with intermittent discussion, but no
consensus or resolution has been reached yet.</p>
This has led to some projects implementing their own workarounds to
deal with this behavior.  The containerd</code> project as an example
noted this
issue</a> in 2017
and worked around
it</a> by creating
explicit archive directory entries for parents.</p>
Rechunking in bootc / rpm-ostree</h2>
Base images used via bootc</code> are typically post-processed and
finalized via the rpm-ostree
rechunker</a>.
The general idea (oversimplifying a bit for brevity) is that the tool
can inspect the RPM database and map each file in the container to its
associated RPM.  It then "rechunks" the container by creating a new
layer for each RPM, where the layer contains the files for that RPM
and only that RPM.  This ensures that when a new version of an RPM is
released in the future, when it is integrated into a new base image,
only the layer associated with that RPM will need to be re-downloaded
by the end user.</p>
However, consider a directory like /usr</code>.  This is owned by the
filesystem</code> package which contains all of the shared base directories
in the system.  This directory will be included in the layer
associated with the filesystem</code> package with all of its related
metadata.</p>
Now, almost every other package in the system is going to have some
amount of content that is shipped under the /usr</code> hierarchy.
Originally, the rpm-ostree</code> rechunker would omit the archive entry
for /usr</code> in each package layer.</p>
This however would cause the behavior of indeterminate tar archives
discussed above to be manifest in the final container.  For example,
it was noted in this composefs-rs
issue</a>.</p>
We worked around this issue by creating a new
format-version</a> for
the rpm-ostree</code> rechunker, which creates all of the parent
directories in each tar layer.  This is similar to the fix implemented
as noted above by the containerd</code> project.  This works, but it's not
ideal.  It means that all of the parent directories and their
respective metadata have to be duplicated across every layer in the
image.  This is an unnecessary waste of disk space and bandwidth.</p>
Why isn't this a more widespread problem?</h2>
When I first realized what was going on here, I thought I could
trigger this behavior with a Containerfile</code> like this:</p>
FROM quay.io/fedora/fedora:43</span></span>
</span>
RUN <<EORUN</span></span>
</span>
# Create two levels of directories, because adding a file to a</span></span>
# directory alters the directory mtime.  This means adding a file</span></span>
# under /foo/bar in the next layer will cause the mtime for /foo/bar</span></span>
# to change, but /foo will remain unchanged.</span></span>
mkdir -p /foo/bar</span></span>
</span>
# zero the mtime on foo</span></span>
touch -d @0 /foo</span></span>
EORUN</span></span>
</span>
# Add a file in a derived layer</span></span>
#</span></span>
# In theory, this layer would not contain an entry for /foo, since it</span></span>
# is unchanged.</span></span>
#</span></span>
# In practice... not so much.</span></span>
RUN touch /foo/bar/baz</span></span></code></pre>
But alas:</p>
⬢ [jeckersb@toolbx example]$ podman build -f Containerfile -t localhost/example</span></span>
STEP 1/3: FROM quay.io/fedora/fedora:43</span></span>
STEP 2/3: RUN <<EORUN (# Create two levels of directories, because adding a file to a...)</span></span>
--> d3f72d4d7b90</span></span>
STEP 3/3: RUN touch /foo/bar/baz</span></span>
COMMIT localhost/example</span></span>
--> b730a7409a5a</span></span>
Successfully tagged localhost/example:latest</span></span>
b730a7409a5a5b5450f4f1d142415043aeac48f6ac0d1ac2a632d75071668a60</span></span>
⬢ [jeckersb@toolbx example]$ podman run --rm -it localhost/example stat /foo</span></span>
  File: /foo</span></span>
  Size: 6               Blocks: 0          IO Block: 4096   directory</span></span>
Device: 0,93    Inode: 64851509    Links: 1</span></span>
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)</span></span>
Access: 2025-12-10 21:31:11.973931592 +0000</span></span>
Modify: 1970-01-01 00:00:00.000000000 +0000</span></span>
Change: 2025-12-10 21:31:11.973678196 +0000</span></span>
 Birth: 2025-12-10 21:31:11.973031361 +0000</span></span></code></pre>
But wait... the directory mtime is (correctly) still zeroed from the
first layer.  So... everything up until now is a lie?</p>
No!  If we pull apart the last tar layer, we'll see that it actually
does</em> contain a directory entry for /foo, even though it's not
modified in any way during the last RUN</code> invocation.  I'll skip the
part where I re-exported the image to an oci-dir just to make it
easier to look through the layers.  Just trust me when I say this is
the tarball that gets generated for the last layer:</p>
⬢ [jeckersb@toolbx sha256]$ tar tvf 9402e1d4a1daea10ff057116f7fc6858be340fd82204eb17db566bfec547a911.tar</span></span>
drwxr-xr-x 0/0               0 1969-12-31 19:00 foo/</span></span>
drwxr-xr-x 0/0               0 2025-12-10 16:31 foo/bar/</span></span>
-rw-r--r-- 0/0               0 2025-12-10 16:31 foo/bar/baz</span></span></code></pre>
Ok, so /foo</code> is there.  But... why?  We didn't modify it.  Deeper
down the rabbit hole we go.</p>
Overlayfs</h2>
Spoiler: the answer is
overlayfs</a>.</p>
The OCI spec defines the image format, but notably it does not</em>
specify any particular implementation details on how to store, manage,
and stitch together the final container image from its individual
layers.</p>
In reality, container runtimes such as podman (via
containers-libs</a>,
formerly containers-storage) use the kernel filesystem overlayfs to
stitch together the layers into the final realized view of the world.</p>
A RUN</code> invocation in a Containerfile</code> is in a nutshell this series
of steps:</p>

Mount the previous layer (and recursively, layers prior to that)
as a lowerdir</code> in an overlayfs filesystem.</li>
Perform the RUN</code> invocation in the mounted filesystem.</li>
Collect the contents of the resulting upperdir</code> directory.  This
becomes the tarball for that layer.</li>
</ol>
This process is how we end up with our /foo</code> directory duplicated
into the final layer, even though we did not modify it.  We can mock
this out just using an overlay mount and basic commands.</p>
First, mock our lower layer, and set the mtime:</p>
[root@burd tmp]# mkdir lower upper work merged</span></span>
[root@burd tmp]# mkdir -p lower/foo/bar</span></span>
[root@burd tmp]# touch -d @0 lower/foo</span></span>
[root@burd tmp]# stat lower/foo</span></span>
  File: lower/foo</span></span>
  Size: 60        	Blocks: 0          IO Block: 4096   directory</span></span>
Device: 0,45	Inode: 46858       Links: 3</span></span>
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)</span></span>
Context: unconfined_u:object_r:user_tmp_t:s0</span></span>
Access: 1969-12-31 19:00:00.000000000 -0500</span></span>
Modify: 1969-12-31 19:00:00.000000000 -0500</span></span>
Change: 2025-12-10 16:53:27.174443520 -0500</span></span>
 Birth: 2025-12-10 16:53:20.544455919 -0500</span></span></code></pre>
Next, mount it via overlayfs:</p>
[root@burd tmp]# mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged</span></span>
[root@burd tmp]# cd merged</span></span>
[root@burd merged]# stat foo</span></span>
  File: foo</span></span>
  Size: 60        	Blocks: 0          IO Block: 4096   directory</span></span>
Device: 0,90	Inode: 46858       Links: 3</span></span>
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)</span></span>
Context: unconfined_u:object_r:user_tmp_t:s0</span></span>
Access: 1969-12-31 19:00:00.000000000 -0500</span></span>
Modify: 1969-12-31 19:00:00.000000000 -0500</span></span>
Change: 2025-12-10 16:53:27.174443520 -0500</span></span>
 Birth: 2025-12-10 16:53:20.544455919 -0500</span></span></code></pre>
Now we simulate creating the new layer:</p>
[root@burd merged]# touch foo/bar/baz</span></span></code></pre>
Finally we can unmount the overlayfs mount and observe the contents of
the upper</code> dir:</p>
[root@burd merged]# cd ..</span></span>
[root@burd tmp]# umount merged</span></span>
[root@burd tmp]# find upper/</span></span>
upper/</span></span>
upper/foo</span></span>
upper/foo/bar</span></span>
upper/foo/bar/baz</span></span>
[root@burd tmp]# stat upper/foo</span></span>
  File: upper/foo</span></span>
  Size: 60        	Blocks: 0          IO Block: 4096   directory</span></span>
Device: 0,45	Inode: 46870       Links: 3</span></span>
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)</span></span>
Context: unconfined_u:object_r:user_tmp_t:s0</span></span>
Access: 2025-12-10 16:57:20.828001765 -0500</span></span>
Modify: 1969-12-31 19:00:00.000000000 -0500</span></span>
Change: 2025-12-10 16:56:16.746593313 -0500</span></span>
 Birth: 2025-12-10 16:56:16.745126092 -0500</span></span></code></pre>
So that's where our /foo</code> entry comes from.  This behavior is
explained in the overlayfs
documentation</a>:</p>
Objects that are not directories (files, symlinks, device-special</span></span>
files etc.) are presented either from the upper or lower filesystem as</span></span>
appropriate. When a file in the lower filesystem is accessed in a way</span></span>
that requires write-access, such as opening for write access, changing</span></span>
some metadata etc., the file is first copied from the lower filesystem</span></span>
to the upper filesystem (copy_up). Note that creating a hard-link also</span></span>
requires copy_up, though of course creation of a symlink does not.</span></span>
</span>
The copy_up may turn out to be unnecessary, for example if the file is</span></span>
opened for read-write but the data is not modified.</span></span>
</span>
The copy_up process first makes sure that the containing directory</span></span>
exists in the upper filesystem - creating it and any parents as</span></span>
necessary. It then creates the object with the same metadata (owner,</span></span>
mode, mtime, symlink-target etc.) and then if the object is a file,</span></span>
the data is copied from the lower to the upper filesystem. Finally any</span></span>
extended attributes are copied up.</span></span></code></pre>
Since the content of the layer tarball is taken directly from the
upper</code>, we end up getting a copy of /foo</code> via this process.</p>
Where do we go from here?</h2>
Hopefully I have explained the situation adequately and this all makes
sense.  I have achieved some amount of inner peace that comes with
reaching understanding.</p>
However... I still don't like it.  I think we can do better.</p>
Here's the thing for bootc</code>: in our rechunked images, I don't want to
include the same directories over and over again in every single
layer.  It shouldn't be necessary, and by strict reading of the OCI
spec we shouldn't</em> do that if the directories are unchanged (which
they aren't).</p>
But in the current state of the world, if we don't</em> duplicate those
entries into every layer, we end up with non-deterministic directory
metadata when our OCI image is mounted and viewed through the lens of
overlayfs.  This is fundamentally broken as we move bootc into a
future where
composefs</a> and
fsverity</a> are used
to harden and validate images.  Ideally a container image should be
able to measure its own fsverity hash in a reproducible manner.  Today
that is not possible.</p>
So the following is what I've done as a proof-of-concept (a huge
motivation for writing this post in the first place was to
exhaustively explain my justification for why this is needed).  I've
written and tested a patch (still needs refining before submitting)
for overlayfs which does the following:</p>

Adds a new feature passthrough</code> (naming bikeshedable) which
functions similarly to the metacopy</code> feature.</li>
Uses the trusted.overlay.passthrough</code> to flag directories as
"structural-only".</li>
When encountering a directory with the xattr, searches through the
lowerdirs until it finds a directory which does not have the xattr
set.  The metadata for this layer is used.  (If no unflagged
lowerdir is found, it falls back to the current behavior of using
non-deterministic local metadata)</li>
</ul>
Here's an example.  This is similar to our manual overlayfs example
above, except this uses two lower layers to show how unpacking a tar
layer with incomplete directory information might look.</p>
First, create our base layer again:</p>
[root@fedora ~]# mkdir upper merged work</span></span>
[root@fedora ~]# mkdir -p base_layer/foo/bar</span></span>
[root@fedora ~]# touch -d @0 base_layer/foo</span></span></code></pre>
Then, in our second layer we create the baz</code> file.  This time we set
the passthrough</code> xattr on /foo</code>, which is how I envision this all
working when a container storage engine unpacks a tar</code> archive with
incomplete parent directory information.  Ideally this layer should
include a tar entry for foo/bar/baz</code> and (arguably) foo/bar</code> since
the directory metadata for bar</code> should be updated to reflect the
addition of bar</code>.  But it definitely should not</em> include an entry
for foo/</code>:</p>
[root@fedora ~]# mkdir -p second_layer/foo/bar</span></span>
[root@fedora ~]# touch second_layer/foo/bar/baz</span></span>
[root@fedora ~]# setfattr -n trusted.overlay.passthrough second_layer/foo</span></span></code></pre>
Now let's mount it with the patched overlayfs.  First, we'll specify
passthrough=off</code> to show the existing behavior:</p>
[root@fedora ~]# mount -t overlay overlay -o upperdir=upper,lowerdir=second_layer:base_layer,workdir=work,passthrough=off merged</span></span>
[root@fedora ~]# stat merged/foo</span></span>
  File: merged/foo</span></span>
  Size: 6               Blocks: 0          IO Block: 4096   directory</span></span>
Device: 0,59    Inode: 365132      Links: 1</span></span>
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)</span></span>
Context: unconfined_u:object_r:admin_home_t:s0</span></span>
Access: 2025-12-10 17:33:32.787147889 -0500</span></span>
Modify: 2025-12-10 17:33:32.787147889 -0500</span></span>
Change: 2025-12-10 17:34:01.701054010 -0500</span></span>
 Birth: 2025-12-10 17:33:32.787147889 -0500</span></span></code></pre>
Here the foo</code> directory has whatever metadata we created for it when
we "unpacked" it with the command mkdir -p second_layer/foo/bar</code>
above.  This is what we're trying to avoid.</p>
Now let's unmount and remount with passthrough=on</code>:</p>
[root@fedora ~]# umount merged</span></span>
[root@fedora ~]# mount -t overlay overlay -o upperdir=upper,lowerdir=second_layer:base_layer,workdir=work,passthrough=on merged</span></span>
[root@fedora ~]# stat merged/foo</span></span>
  File: merged/foo</span></span>
  Size: 6               Blocks: 0          IO Block: 4096   directory</span></span>
Device: 0,59    Inode: 365129      Links: 1</span></span>
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)</span></span>
Context: unconfined_u:object_r:admin_home_t:s0</span></span>
Access: 1969-12-31 19:00:00.000000000 -0500</span></span>
Modify: 1969-12-31 19:00:00.000000000 -0500</span></span>
Change: 2025-12-10 17:28:12.268740241 -0500</span></span>
 Birth: 2025-12-10 17:27:39.158847877 -0500</span></span></code></pre>
Now we have the metadata for foo</code> being passed through the second
layer and retrieved from the base layer, which is the behavior we
want.</p>
Conclusion</h2>
Ensuring that OCI image layers are able to only ship content that
they've modified will aid reproducibility and help unlock
optimizations by avoiding having derived layers always need to inherit
metadata from their parent. At the current time, however, tooling
creating OCI images should include their parent information in order
to maximize compatibility. We are looking forward to a future where
that's not necessary though!</p>

Sealed images: deploying and verifying a sealed image</h1>
In the previous post</a> we built a sealed, signed container image. Now it's time to deploy it to a virtual machine and verify that the full security chain is active.</p>

What's next</h2>
At this point we have a sealed, signed container image. The UKI inside is signed with our Secure Boot key and embeds a composefs digest that covers every file in the operating system. In the next post, we'll deploy this image to a system and verify the seal is active.</p>

TODO:</h3>

Laura to explore adding logos to PR.</li>
Laura to add GitHub Actions for publication</li>
Look for info on domain handling for static site</li>
Colin to open a discussion about the AI assisted PRs</li>
Laura to find and share the public Containers Cabal recording</li> </ul>

TODO:</h3>

Put project pavilion application</li>
Keep smaller, more focused meetings</li> </ul>

bootc

bootc ecosystem at DevConf.CZ and Flock to Fedora 2026

Video: sealed bootc with transient /etc and /var

Sealed images: deploying and verifying a sealed image

Sealed images: building a sealed image

Sealed images: key management for Secure Boot

Sealed images: the security chain from firmware to filesystem

Containers: pitfalls of incomplete tar archives

Bootc Community Meeting Notes - 18 July 2025

Bootc Community Meeting Notes - 11 July 2025

bootc

bootc ecosystem at DevConf.CZ and Flock to Fedora 2026

Video: sealed bootc with transient /etc and /var

Sealed images: deploying and verifying a sealed image

Deploying the image</h2> With the sealed image built from the previous post and keys generated from the key management post, deploying is a single command:</p>

Sealed images: building a sealed image

What's next</h2> At this point we have a sealed, signed container image. The UKI inside is signed with our Secure Boot key and embeds a composefs digest that covers every file in the operating system. In the next post, we'll deploy this image to a system and verify the seal is active.</p>

Sealed images: key management for Secure Boot

Sealed images: the security chain from firmware to filesystem

Containers: pitfalls of incomplete tar archives

Bootc Community Meeting Notes - 18 July 2025

Attendees:</h3> Laura Santamaria (she/her; Red Hat)</li> Hristo Marinov</li> Fernando Lozano</li> Colin Walters (he/him; Red Hat)</li> Joseph Marrero Corchado (Red Hat, Inc.)</li> Matteo Piccinini (n/a)</li> Robert Sturla (Tesco Bank)</li> Dusty Mabe</li> Chris Kyrouac</li> Gursewak Mangat</li>

Agenda:</h3> Release 1.5.1</a> Thanks @robert!</li> </ul> </li>

TODO:</h3> Laura to explore adding logos to PR.</li> Laura to add GitHub Actions for publication</li> Look for info on domain handling for static site</li> Colin to open a discussion about the AI assisted PRs</li> Laura to find and share the public Containers Cabal recording</li> </ul>

Bootc Community Meeting Notes - 11 July 2025

TODO:</h3> Put project pavilion application</li> Keep smaller, more focused meetings</li> </ul>

What's next</h2>
At this point we have a sealed, signed container image. The UKI inside is signed with our Secure Boot key and embeds a composefs digest that covers every file in the operating system. In the next post, we'll deploy this image to a system and verify the seal is active.</p>

TODO:</h3>

Laura to explore adding logos to PR.</li>
Laura to add GitHub Actions for publication</li>
Look for info on domain handling for static site</li>
Colin to open a discussion about the AI assisted PRs</li>
Laura to find and share the public Containers Cabal recording</li> </ul>

TODO:</h3>

Put project pavilion application</li>
Keep smaller, more focused meetings</li> </ul>