[bhouse-nexthop]

Description

Resizing a volume on RBD primary storage fails on KVM when the host's libvirt has not cached the image in its view of the pool. A freshly deployed VM's ROOT volume is the usual case:

org.libvirt.LibvirtException: Storage volume not found: no storage vol with matching path '<ceph pool>/<uuid>'

This is a regression from #14188.

Fixes #14622

Why it happens

LibvirtResizeVolumeCommandWrapper gets the disk with pool.getPhysicalDisk(), then looks the volume up through libvirt with conn.storageVolLookupByPath(path) to call StorageVol.resize().

• Before kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188, getPhysicalDisk() went through LibvirtStorageAdaptor.getVolume(), which refreshes the libvirt pool on a miss. The volume was therefore in libvirt's cache by the time the wrapper looked it up.
• kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188 made getPhysicalDisk() stat RBD images through librbd, so nothing refreshes the pool any more.
• RBD volumes are created through librbd, not libvirt. A new volume is not in libvirt's cache on a host until something refreshes the pool there.

Fix

When storageVolLookupByPath() fails with VIR_ERR_NO_STORAGE_VOL on an RBD pool, refresh that pool and look the volume up once more.

• This is what getVolume() did for this path before kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188. The refresh only happens when the volume is not cached.
• Any other libvirt error, and every other pool type, fails the resize as before, with no refresh.
• Nothing else in the resize changes: the flags, StorageVol.resize(), then the resize script, which for RBD is NOTIFYONLY and runs virsh blockresize on a running VM.

I looked at two other fixes:

• Resize RBD images through librbd and skip libvirt for RBD, as the rest of LibvirtStorageAdaptor already does for create, clone and delete. That avoids the refresh entirely, but it replaces the resize itself (shrink handling, the image open, error reporting), so it is a bigger change for a regression fix. It would make a reasonable follow-up.
• Reuse LibvirtStorageAdaptor.getVolume(). It looks the volume up by name on the libvirt pool object and refreshes on any error, so it would change the lookup for volumes that are cached, too.

Other call sites

I checked the KVM plugin for other places that rely on getPhysicalDisk() having refreshed the libvirt pool. There are none:

• LibvirtStorageAdaptor.deletePhysicalDisk() calls getVolume() itself, which still refreshes on a miss.
• LibvirtMigrateCommandWrapper.deleteLocalVolume() uses storageVolLookupByPath(), but only for local files.

Not changed here

execute() already calls pool.refresh() after every resize, before it reads the new size back. For RBD that refresh no longer affects the result, since the size now comes from librbd. On a cache miss, this PR adds a second refresh. Dropping the trailing one for RBD is a separate change.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

Unit tests. A live lab retest will follow.

LibvirtComputingResourceTest passes (273 tests, checkstyle on). The new tests are:

• testResizeVolumeCommandRbdVolumeNotInLibvirtPoolCache: the first lookup throws not found on an RBD pool. The resize refreshes the pool, finds the volume, resizes it, then runs the notify-only script, and answers success with the new size.
• testResizeVolumeCommandRbdVolumeNotFoundAfterRefresh and testResizeVolumeCommandRbdVolumeRetriedAfterFailedRefresh: the volume is still missing after the refresh, whether or not the refresh itself worked. There is exactly one refresh and one retry, then the resize fails.
• testResizeVolumeCommandRbdOtherLookupErrorDoesNotRefresh and testResizeVolumeCommandRbdLookupErrorWithoutCodeDoesNotRefresh: any other lookup error on RBD fails without a refresh.
• testResizeVolumeCommandNonRbdVolumeNotFoundDoesNotRefresh: a missing volume on NFS fails without a refresh, as before.

Without the fix, the first three of these fail. The other three pass with or without it; they check that nothing else changed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 4.08%. Comparing base (d02adcd) to head (aa9683a).

❗ There is a different number of reports uploaded between BASE (d02adcd) and HEAD (aa9683a). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (d02adcd)	HEAD (aa9683a)
unittests	1	0

Additional details and impacted files

@@              Coverage Diff              @@
##               4.22   #14623       +/-   ##
=============================================
- Coverage     18.32%    4.08%   -14.25%     
=============================================
  Files          5939      449     -5490     
  Lines        537768    38282   -499486     
  Branches      65929     7083    -58846     
=============================================
- Hits          98551     1564    -96987     
+ Misses       428008    36503   -391505     
+ Partials      11209      215    -10994     

Flag	Coverage Δ
uitests	4.08% <ø> (ø)
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[weizhouapache]

can you create a method in libvirtUtilitiesHelper ?

[bhouse-nexthop]

Done in aa9683a: the lookup with the RBD pool refresh is now LibvirtUtilitiesHelper.getStorageVolumeByPath(Connect, KVMStoragePool, String), and the resize wrapper calls it. Its tests moved to LibvirtUtilitiesHelperTest; LibvirtComputingResourceTest keeps the end-to-end resize of an RBD volume libvirt has not cached, plus a resize that fails when the lookup fails.

[bhouse-nexthop]

Test report: resizing RBD volumes that libvirt has not cached, with no manual pool refresh

Environment

• A CloudStack 24.0 (main) build with this PR applied, alongside some of my other open PRs, including kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188 (merged) and Allow growing a running KVM VM's ROOT volume through a new disk offering #14602.
• Why the result carries over to 4.22: the resize wrapper and the RBD lookup in getPhysicalDisk() (kvm: look RBD volumes up through librbd instead of refreshing the whole pool #14188) are the same on 4.22 and main.
• Three KVM hosts on Ubuntu 24.04 (libvirt 10.0.0, QEMU 8.2.2), nested virtualisation. Ceph RBD primary storage (squid 19.2.3).
• Ubuntu 24.04 guests on so-r10 (1 vCPU, 1 GB, ROOT on root10, 10 GB), on a fresh deploy each time.
• I did not run virsh pool-refresh at any point. Before each resize, I checked whether the image was in libvirt's view of the pool with virsh vol-list <pool> on the VM's host.

1. Size-only resize of a fresh VM's ROOT, VM running

• Before: the ROOT image was not in virsh vol-list on the host.
• resizeVolume size=12 succeeded in 4 s. The API, virsh domblkinfo, rbd info and the guest's lsblk all showed 12 GiB, with no reboot. The VM stayed Running.
• The agent logged the refresh once:

  INFO  [resource.wrapper.LibvirtResizeVolumeCommandWrapper] Volume cloudstack/<uuid> is not in libvirt's cache of RBD pool <pool>
        (Storage volume not found: no storage vol with matching path 'cloudstack/<uuid>'). Refreshing the pool and looking the volume up again.
  

• After: the image was in virsh vol-list.
• A second resize of the same volume (size=14) succeeded with no refresh line, because the lookup hit the cache. So the refresh happens only on a miss. All four views showed 14 GiB.

On the build without this PR, the same first resize failed with Storage volume not found: no storage vol with matching path 'cloudstack/<uuid>' (see #14622). That is the failure this PR fixes.

2. Data volume, created, attached and resized while attached to the running VM

• A 5 GB volume on the Custom offering, attached to the VM from step 1. The guest saw it as vdb.
• resizeVolume size=8 succeeded in 5 s. API, libvirt, RBD and the guest's lsblk /dev/vdb all showed 8 GiB.
• On this host the image was already in libvirt's cache before the resize, so the resize found it on the first lookup and logged no refresh. The pool refresh from step 1 ran before this volume existed, so the attach must have put it in the cache. This step shows the normal path still works, and that the retry stays out of the way when it isn't needed.

3. resizeVolume diskofferingid=root20 on a fresh running VM (the #14602 path)

• A second fresh VM, on another host. Its ROOT was not in virsh vol-list there.
• resizeVolume diskofferingid=root20 succeeded in 5 s, and the volume is now on root20 at 20 GiB in all four views. The VM stayed Running.
• The agent logged the same single refresh-and-retry line.
• In the guest, growpart and resize2fs grew / to 19G.

Result: pass.

• A resize of an RBD volume that is not in libvirt's pool cache now refreshes the pool once and succeeds, for a size-only ROOT resize, a data volume, and a resize through a new disk offering.
• No manual virsh pool-refresh was needed. When the volume is already cached, the resize doesn't refresh.

Not covered

• A refresh that fails (the warning path), and a volume that is still missing after the refresh. Both are covered by the unit tests.
• Encrypted RBD volumes, and non-RBD pools. Neither goes through the retry.

[weizhouapache]

code lgtm

not tested yet

[weizhouapache]

@blueorangutan package

[blueorangutan]

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.