The vulnerability
OpenStack supports users to upload custom VM images in a number of formats
to be used for creating Virtual Machines. The simplest is a raw copy of a harddisk,
while other formats such as QCOW2 and VMDK are more advanced in being compressed
and allowing for embedding additional information. They also support referencing
other files. The latter feature needs to be controlled by OpenStack to avoid
exposing sensitive files. OpenStack thus disallows such external references
for customer images. The protections were found to be incomplete before which
lead to
CVE-2022-47951
and
CVE-2024-32498.
Unfortunately, the last fixes with CVE-2024-32498 still were incomplete:
Arnaud Morin (OVH) reported a vulnerability in Nova. By supplying a raw format
image which is actually a specially crafted QCOW2 image with a backing file
path or VMDK flat image with a descriptor file path, an authenticated user may
convince systems to return a copy of the referenced file’s contents from the
server resulting in unauthorized access to potentially sensitive data. All Nova
deployments are affected.
The vulnerability has been assigned CVE-2024-40767.
Impact on the SCS reference implementation
The Sovereign Cloud Stack reference implementation
(by OSISM) runs these services in containers and thus
most files from the hypervisor's file system are not directly accessible from
inside the container via this vulnerability. However, this depends on the
service inside the container. Some services have access to data from the
hypervisor or they contain sensitive data themselves (e.g. credentials to
connect to other services), and thus they are accessible by a potential
attacker. They have also access to log files from the services which may
expose sensitive data.
To abuse this vulnerability, the attacker must be an authenticated
user of the platform.
We were not able to exploit this vulnerability in systems that included
the hotfix patches for
CVE-2024-32498
in our testing.
Nova did treat the VMDK and QCOW images declared as raw as raw (as did
Cinder, but Cinder is not suspected of mistreating them).
Boot disks created by Nova from the crafted VMDK and QCOW2 images contain
the raw contents of the images, without their external references being
resolved and thus without exposing any external content.
This is probably caused by SCS not changing the default to
use_cow_images=False. We currently are not aware of working exploits
without that (non-default) setting.
We would argue that mislabeled images should better be rejected early, as
the investigation chain may otherwise be confused which has the potential
to cause exploitable vulnerabilities. So we still highly recommend to patch
the weakness soon.
Operators that are using use_cow_images=False in their nova config are
advised to apply the fixes urgently.
Embargo
The issue has been reported to the OpenStack Vulnerability Management Team in
private. The reporters and upstream developers have worked together to address
the issue with fixes and an embargo date
has been set to Tuesday, 2024-07-23, 15:00 UTC (17:00 CEST). At this point in
time, the patches will get merged and the OpenStack Security Advisory
(OSSA-2024-002) will
be published. The issue is tracked in OpenStack issue
#2071734, which should become
publically accessible after the lift of the embargo and the publication
of this advisory.
Under the used responsible disclosure approach, the information was shared with
a select group of trustable users of OpenStack, so they can prepare updates and
protect their user data in time for the publication.
Mitigation and Fixes
Temporary workarounds that would disable image uploads for QCOW2
and VMDK formats are not desirable and incomplete, as images can also
be created from volumes, which would also have to be blocked.
The complexity of doing all the changes exceeds the complexity of
deploying the fixes.
A better workaround would be to undo the setting of use_cow_images=False
in the nova configuration, as this stops all known exploitable code paths
from being used.
There are patches from the upstream OpenStack developers available. They work
by rejecting images in Nova where contents and the claimed format do not match
and require that a format inspector is available prior to further processing
them. This provides effective protection against confusing Nova's image
handling.
Note that the upstream changes also include improvements to the handling
of ISO images from Jens Harbott (OSISM), which have further tighten the
inspection and which have been merged and backported upstream already without
having received a separate advisory (due to not having fixed known exploitable
situations).
The reference implementation for the IaaS layer of Sovereign Cloud Stack
is provided by the OSISM project — the OSISM team normally builds the
artifacts in a completely public process. To avoid waiting for a few
hours to have all image builds (nova-compute containers)
completed and tested after the disclosure date,
the OSISM project has built private images beforehand which we have tested
to protect against all known scenarios in which these image files are used.
These images were handed to our SCS operators a few hours ahead of time
(without a detailed description) to allow them to protect their infrastructure.
Instructions to install the fixed containers (hotfix on top of
OSISM 7.0.x/OSISM 7.1.0):
{% raw -%}
# Override nova-compute image in your configuration repository:
cat >environments/kolla/images.yml <<EOT
---
my_registry_kolla: osism.harbour.regio.digital
my_namespace2: osism-20240723
nova_compute_tag: build-20240717-stable-2023.2
nova_compute_image: "{{ my_registry_kolla }}/{{ my_namespace2 }}/nova-compute"
EOT
{% endraw -%}
Now, the nova compute image will be downloaded and rolled out on all nova compute nodes:
osism apply -a pull nova -l compute # optional
osism apply -a refresh-containers nova -l compute
If you are not yet on R6 (OSISM 7.0.x), replace stable-2023.2 with the version you are on
(e.g. stable-2023.1 if you are on R5 (OSISM 6.x).
Note that the above instructions do NOT contain cinder and glance images; they should
have been fixed already with the installation of fixes for CVE-2024-32498 or with
the rollout of OSISM 7.1.0.
The deployment will have the following user-visible effects:
- VM (compute service) operations will not be available for some seconds.
Most users won't notice this. Please note that running workloads will not be
affected at all.
After the upstream patches have been merged in the hours after the disclosure
date, OSISM will build, test and publish an official 7.1.1 release
containing the updated containers (nova-compute, glance-api, cinder-volume)
as well as a fix for octavia-api. Please refer to the
OSISM release notes.
You can upgrade from this hotfix toiOSISM 7.1.1 using the regular upgrade mechanism.
If your exposure is low (e.g. private cloud or not using use_cow_images=False)
you may decide to wait for the regular 7.1.1 and go there directly from 7.x.
Warning: If you are still on OSISM 7.0.x (not 7.1.x yet) and create the images.yml as
described here, osism apply -a pull and osism apply -a refresh-containers
will use the base 7.0.x containers with just the single nova_compute
override which would make you vulnerable to the glance and cinder based
attacks from CVE-2024-32498 again.
We recommend to avoid this by using the latest 7.1.0 release.
If you can't, you can either ensure you avoid pulling and refreshing containers
other than the nova_compute one or you construct a combined images.yml from the
description in CVE-2024-32498
and the above section. The latter would ensure you have all needed image overrides for
both CVEs.
Have my systems been compromised?
Providers that are unable to patch this issue on the day of the release
and use use_cow_images=False in their nova configuration
need to consider the possibility of having been attacked. In this case,
passwords/keys for internal communication should be considered to be
leaked and will need to be rotated to reestablish the defense-in-depth
lines. We recommend providers check for suspicious image operations
(QCOW2 and VMDK image registrations, snapshots with QCOW2 signature) in
their logs for the time after the embargo lift.
In particular, every QCOW2 image that has been uploaded should be investigated
for the datafile property; it should be assumed that images that have been
registered and removed again contain the exploit.
VMDK images with external Flatfile references (createType="monolithicFlat")
are suspicious as well. Evil images should also be pruned from the image cache.
Patch status of SCS clouds
Our publically known providers of SCS IaaS platforms have all been
notified and have started to ensure that their use_cow_images setting
does not expose them and have started the rollout of updated images which
we were able to provide them privately a few hours ahead of time (without
sharing all the details on the issue). We expect that none of them has
any exploitable vulnerability left open by the time this gets public.
Please refer to their communication for precise information on the
patch status.
Thanks
The authors would like to thank Arnaud Morin, Jens Harbott, the OpenStack
developers and reviewers, testers and the OpenStack Vulnerability Management
Team for the responsible reporting, careful analysis, fixing, testing and
professional handling of the issue and the OSISM team for providing the fixed
containers.
Sovereign Cloud Stack Security Contact
SCS security contact is security@scs.community, as published on
https://scs.community/.well-known/security.txt.
Version history
- Initial Draft, v0.1, 2024-07-20, 10:00 CEST.
- More precise information on exploitability with
use_cow_images=False, v0.2, 2024-07-20, 11:30 CEST. - Installation instructions, v0.3, 2024-07-20, 14:30 CEST.
- Fix install instructions, v0.4, 2024-07-22, 17:45 CEST.
- Escape jijna templates, v0.5, 2024-07-22, 18:30 CEST.
- Move installation instructions into the fixes section, add combined install insns, v0.6, 2024-07-23, 15:15 CEST.
- Replaced combined install instructions with a wraning, v0.7, 2024-07-23, 16:15 CEST.
- Final review for pubication, v1.0, 2024-07-23, 17:00 CEST.