Repository navigation
kvm: support Host HA on Ceph RBD primary storage - #14320
weizhouapache wants to merge 21 commits into
Conversation
Extend the existing KVM Host-HA heartbeat/VM-activity-check framework (currently limited to NetworkFilesystem and SharedMountPoint pools) to also cover RBD primary storage, based on the approach from the old PR apache#5862, adapted to the current HA architecture and reusing the multi-monitor Ceph support already added in apache#6792. - Add StoragePoolType.RBD to LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT, which is the single switch that makes pool registration, KVMHAMonitor, and the CheckOnHostCommand/CheckVMActivityOnStoragePoolCommand wrappers treat RBD pools as HA-capable. - LibvirtStoragePool: build rbd/rados connection args (--mon-host, pool, and cephx --id/--key when set) from the pool's existing sourceHost/ sourceDir/authUsername/authSecret fields for the heartbeat and VM-activity checks, mirroring the conventions KVMPhysicalDisk already uses to talk to RBD. - Add kvmheartbeat_rbd.sh and kvmvmactivity_rbd.sh: RBD has no shared mount point to write a heartbeat file to, so the heartbeat timestamp is stored as a small RADOS object per host instead, and VM activity is detected via RBD watchers (rbd status) rather than file mtimes. - Minor: fix a stale "NFS storage pool" log message in KVMHAMonitor now that this path also runs for RBD. - Add LibvirtStoragePoolTest#testIsPoolSupportHA covering the new RBD case (no HA/heartbeat tests existed previously for any pool type).
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## main #14320 +/- ##
=============================================
- Coverage 20.75% 3.88% -16.88%
=============================================
Files 6496 493 -6003
Lines 591602 42571 -549031
Branches 72583 8069 -64514
=============================================
- Hits 122814 1653 -121161
+ Misses 455456 40687 -414769
+ Partials 13332 231 -13101
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19426 |
|
@blueorangutan test |
|
@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17079)
|
There was a problem hiding this comment.
🟡 Changes recommended
Critical RBD monitor-port and stale-heartbeat handling issues remain, along with incomplete HA cleanup.
3 open findings
What changed in this PR
Extends KVM Host HA heartbeat and VM-activity checks to Ceph RBD primary storage.
Changes:
- Adds RBD HA support and connection handling.
- Adds RADOS heartbeat and RBD watcher activity scripts.
- Updates logging and adds HA eligibility tests.
| File | Summary |
|---|---|
scripts/vm/hypervisor/kvm/kvmvmactivity_rbd.sh |
Adds RBD watcher-based VM activity detection. |
scripts/vm/hypervisor/kvm/kvmheartbeat_rbd.sh |
Adds RADOS-backed heartbeat handling; stale delays over 255 seconds require bounded status handling. |
plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePoolTest.java |
Tests RBD HA support eligibility. |
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/LibvirtStoragePool.java |
Selects RBD scripts and builds Ceph arguments; monitor ports must be propagated, and missing-script errors should be generic. |
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java |
Generalizes the storage-pool log message. |
engine/components-api/src/main/java/com/cloud/ha/HighAvailabilityManager.java |
Enables RBD HA support; corresponding pool deletion paths must also clean up HA state. |
🧠 Review effort: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- pass the pool's monitor port to the RBD heartbeat/activity scripts, for every monitor, instead of relying on the default port - kvmheartbeat_rbd.sh: keep the heartbeat age out of the function return status, which wraps above 255 and made a long-stale heartbeat look ALIVE - remove pools from the HA monitor on deletion for all storage pool types with HA support, without trying to umount an empty mount path - make the missing-script error message generic - add unit tests for the RBD monitor list (IPv4, IPv6, mixed, with/without port)
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
11 open findings
kvmheartbeat_rbd.shuses-hto derive the per-host RADOS object name… · New-hwas provided; ifHostIPis empty, the heartbeat object… · Newpool.getMountDestPath()directly into a shell command string can break if the path… · NewsourcePort <= 0, this returnssourceHostwithout normalization. IfsourceHostcontains… · NewauthUsername != nullis not sufficient to safely add cephx args:authUsernamecould be… · New-thelp text says 'time on ms', but the caller (LibvirtStoragePool) passes seconds… · New-thelp text says 'time on ms', but the caller (LibvirtStoragePool) passes seconds… · New-thelp text says 'time on ms', but the caller (LibvirtStoragePool) passes seconds… · New
🧠 Review effort: Lite
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19467 |
…of the pool Follow-up to the review of the Host HA on Ceph RBD support: - kvmheartbeat_rbd.sh and kvmvmactivity_rbd.sh: exit with an error when the host IP, which names the RADOS objects, is missing (the self-fencing -c of the heartbeat script does not need it), or when a Ceph user is given without its key - LibvirtStoragePool: trim the Ceph monitors and skip empty entries, with or without a port; pass the Ceph user and key only if both are set and fail with a clear error if only one of them is - KVMHAMonitor: run umount without a shell when removing a pool - kvmvmactivity_rbd.sh: fix the -t help text and drop the unused field from the stored activity state - add unit tests for the monitor list (multiple IPv4 and IPv6 monitors, with and without a port, with spaces) and for the Ceph user and key handling
|
@blueorangutan package |
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
6 open findings
:as an IPv6 literal and will wrap it in brackets, which… · NewgetRbdMonitors()currently branches on the presence of:(which can also appear in… · New--keyexposes it in the process list (both for this script’s args and… · Newintervalis used in a numeric comparison but is never validated when running with-r. If-r… · New${UUIDList//,/ }expansion relies on shell word-splitting and can mis-handle… · Newrados ... statoutput content to detect existence is brittle (output formats can vary, and… · New
11 resolved since last review
-hwas provided; ifHostIPis empty, the heartbeat object…kvmheartbeat_rbd.shuses-hto derive the per-host RADOS object name…-thelp text says 'time on ms', but the caller (LibvirtStoragePool) passes seconds…-thelp text says 'time on ms', but the caller (LibvirtStoragePool) passes seconds…-thelp text says 'time on ms', but the caller (LibvirtStoragePool) passes seconds…authUsername != nullis not sufficient to safely add cephx args:authUsernamecould be…sourcePort <= 0, this returnssourceHostwithout normalization. IfsourceHostcontains…pool.getMountDestPath()directly into a shell command string can break if the path…
🧠 Review effort: Lite
Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.
…st HA checks, and harden the parsing Follow-up to the review of the Host HA on Ceph RBD support: - LibvirtStoragePool: do not add a port to Ceph monitors which have one already, and only wrap bare IPv6 addresses in square brackets; mask the Ceph key in the logged command - kvmheartbeat_rbd.sh, kvmvmactivity_rbd.sh: give the Ceph key to rados and rbd in a temporary keyfile instead of --key - kvmheartbeat_rbd.sh: require -t to be a number when checking a heartbeat - kvmvmactivity_rbd.sh: read the list of volumes without word splitting or globbing, skipping empty entries, and check if the activity object exists by the exit status of rados stat - add unit tests for monitors with a port and for the masked key
|
Thanks @weizhouapache, I went through the GUID change manually and then used Claude Fable to check it again. The naming is consistent on both sides: the checker uses Looks good now. Remove the draft status and it is good to merge. |
great, thanks for the reviews and suggestions, very useful ! @wido @kiranchavala |
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
# Conflicts: # debian/control
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19752 |
…installed in the RBD HA scripts The packages python3-rados and python3-rbd are only suggested by the agent packages, so the scripts check at the start that they can be imported, and say which package is missing: - the heartbeat is not written (exit 1), with the reason - the heartbeat check and the VM activity check are inconclusive (exit 2, no DEAD), so the host is not considered dead The self-fencing (-c) does not use Ceph, so it does not need them.
…os and librbd python3-rados and python3-rbd are in the repository of Ceph, which is not configured on every host, so the agent could not be installed on EL8 without it, even without Ceph storage. The scripts of the Host HA on RBD check that they are installed.
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19773 |
|
[SF] Trillian test result (tid-151)
|
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
…e test PRs Add the commits for apache#14386, apache#14320 and apache#11931 and update the notes.
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19851 |
|
[SF] Trillian test result (tid-168)
|
bhouse-nexthop
left a comment
bhouse-nexthop
left a comment
There was a problem hiding this comment.
Thanks for working on this. RBD-only deployments really need it. I read it closely because we'd like to run it on our RBD-only KVM cluster. I've sorted the findings by severity.
The main concern is that, as it stands, a host that is still running can be declared Down. Ceph's exclusive lock and blocklisting protect the data, but the running VM loses its disk and is restarted elsewhere.
Blocking: a live host can be declared Down
- The watcher check doesn't protect VM HA.
kvmvmactivity_rbd.pyis only reached throughKVMHAProvider.isActive, which is Host HA and needs OOBM.- Without OOBM, RBD in
LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORTturns on the VM-HA path:KVMInvestigator→CheckOnHostCommand→KVMHAChecker→hasHeartBeat. That path uses only the heartbeat age. VM I/O is never consulted before a restart. - Either use the activity check there too, or document that only Host HA has it.
- A missing heartbeat object reads as DEAD (
kvmheartbeat_rbd.pycheck()). An agent that never wrote one is declared dead:- not upgraded yet;
- no bindings;
- a GUID mismatch;
- the first cycle.
- Suggest: missing object → exit 2 (unknown).
- A neighbour with no HA pools answers "dead".
KVMHAChecker.hasHeartBeat()starts atfalse, so an empty pool list (any not-yet-upgraded agent) answers alive=false.- Before this PR, RBD-only clusters never reached it, because
storageSupportsHAwas false. - Now a rolling upgrade (management server first) plus one agent restart marks a running host Down.
- Before this PR, RBD-only clusters never reached it, because
- The heartbeat GUID comes from different sources.
- The writer uses the agent.properties
guid+-+ name. - The checker uses
HostTO.getGuid(), i.e.AgentShell.getGuid(), which prefers the command line and is a random UUID in dev mode. - A missing property gives
KVMHA-hb-null-LibvirtComputingResource, shared by every such host.
- The writer uses the agent.properties
- One bad RBD pool stops the HA monitor thread for every pool.
createHeartBeatCommandcan now throw:- a user/key mismatch;
- an NPE in
getRbdMonitorson a null host; - a missing script.
KVMHAMonitor.run()catches nothing, so the thread dies and NFS/SMP heartbeats stop too. The host then looks dead.
- The verdict is whatever pool comes last. In
KVMHACheckerthe loop overwrites the result per pool, so the verdict comes from the last pool in HashMap order. With mixed NFS and RBD pools, a missing RBD object can outvote a fresh NFS heartbeat.
Blocking: behaviour change outside HA
- RBD pools are now really destroyed on removal.
KVMStoragePoolManager.deleteStoragePool(type, uuid, details)now sends RBD (and SMP) toadaptor.deleteStoragePool(uuid). Before, it went to thedetailsvariant, whichLibvirtStorageAdaptorleaves as the default no-op.- So the libvirt RBD pool is destroyed and its secret undefined, e.g. on
removeHeartbeatForHostsFromPoolduring pool maintenance, while VMs still use it. - Keep the
detailsdelete and only add the HA unregister.
Should fix
- Bindings are only Suggests.
- Without them, the write fails every cycle, and 5 retries × 10 s while holding the
haStoragePoolslock delay NFS heartbeats as well. - With the self-reboot default on, every RBD host reboots.
- Neighbour checks exit 2, so a really dead host never becomes Down.
- Make the bindings Depends/Requires, or fail clearly at registration.
- Without them, the write fails every cycle, and 5 retries × 10 s while holding the
- The freshness thresholds are inconsistent.
kvmvmactivity_rbd.pyhard-codes 61 s for "fresh", ignoringkvm.heartbeat.update.frequency(the heartbeat script gets-t).- The DEAD threshold equals the update interval with no slack, which is tight under Ceph slow ops.
Nits
- RBD special cases are spread across files: monitor, wrapper and pool. Let the pool own the host id and the activity script.
- Duplicate helper:
RbdUtils.composeOptionForMonHostshas no caller and duplicatesKVMPhysicalDisk.composeOptionForMonHosts. - Licence header:
LibvirtCheckVMActivityOnStoragePoolCommandWrapperTestlacks the ASF header.
Looks right
- The key is passed in the environment, not on the command line.
- Volume paths are passed to the activity check.
- A neighbour that can't reach Ceph reports exit 2, not DEAD.
- The large volume lists go through a file.
@bhouse-nexthop |
- Report "cannot tell" instead of dead when the RADOS heartbeat object is missing or malformed, or when the neighbour monitors no HA pool - Decide the verdict over all pools instead of the last one - Keep the HA monitor thread alive when a pool cannot run its heartbeat - Do not monitor an RBD pool whose Ceph Python bindings are missing - Share the heartbeat freshness rule (interval plus slack) between scripts - Keep deleting RBD and SharedMountPoint pools with their details - Move the RBD specific decisions into the storage pool
@bhouse-nexthop
New tests cover I also ran this on a nested environment with KVM hosts and Ceph RBD primary storage. The 9 tests in |
bhouse-nexthop
left a comment
bhouse-nexthop
left a comment
There was a problem hiding this comment.
Thanks for the quick and thorough turnaround, and for the point-by-point reply. I went through fe48966 against each item and ran the unit tests for the changed code. Nearly everything is resolved. One fix from the reply isn't in the code, though, and its new tests fail.
Resolved
- 2. Missing heartbeat object.
kvmheartbeat_rbd.pynow exits 2 for a missing or malformed object. It reports DEAD only when the object exists and is stale. - 3. Neighbour with no HA pools.
KVMHACheckerreturns "alive / can't tell" for an empty pool list, and the old test expectation is updated. - 4. GUID source. I accept the reasoning. Without a GUID the agent refuses to start outside developer mode, and the installer writes the same
guidthathost.guidis built from. With the fix for 2, a mismatched GUID now reads as "unable to tell", not DEAD, so the dangerous half of this item is gone. - 5. A bad pool stops the monitor. Per-pool errors are caught in
executePoolHeartBeatCommand,run()survives unexpected errors, andKVMHAMonitorTestcovers both. - 6. Verdict from the last pool. The any/all rule is correct for both settings of the flag, and
KVMHACheckerTestcovers it. - 9. Thresholds. Both scripts now share
heartbeat_max_age(), and the activity script gets the real interval through-i. I left one small suggestion inline. - 10. RBD special cases.
getHeartBeatHostId,getVmActivityCheckPathandgetHeartBeatPrerequisiteErrornow live on the pool. Neither the monitor nor the wrapper tests for RBD any more. - 12. Licence header. Added.
Reasonably declined or documented
- 1. VM HA uses only the heartbeat. Documented in
KVMHAChecker, and it matches NFS. I'm fine with handling it in a follow-up. Please also say it in the PR description or the cwiki page, since RBD-only users without OOBM will rely on that path. - 8. Bindings stay
Suggests. The EL8 argument holds. The--check-bindingsgate at registration fails clearly, and I checked that it exits 1 with a clear message when the bindings are missing. One consequence is worth documenting. A host without the bindings (or with apython3-rbdtoo old forwatchers_list) monitors no RBD pool, so as a neighbour it answers "alive" for any host it checks.KVMHostActivityChecker.checkHostStatusWithNeighbourHostsreturns on the first "alive" answer, so on an RBD-only cluster that one host stops VM HA from marking any host Down. This fails safe, so it isn't blocking, but the only signal is one line in that host's agent log. I left an optional follow-up idea inline. - 11.
composeOptionForMonHosts. Fine, since #14402 adds the callers.
Remaining
- 7. RBD pools are still destroyed on removal (blocking). The reply says RBD and SharedMountPoint keep the delete that takes the details, but
KVMStoragePoolManager.deleteStoragePool(type, uuid, details)still sends every HA-capable type, now including RBD and SharedMountPoint, toadaptor.deleteStoragePool(uuid). fe48966 doesn't touch that method. The newKVMStoragePoolManagerTestexpects the intended behaviour, and 2 of its 5 tests fail against the current code (details inline). It looks like a hunk got lost in the push.
Minor
- The PR description still describes
kvmheartbeat_rbd.sh/kvmvmactivity_rbd.sh,rbd statusand passing--id/--keyon the command line. It would help reviewers to refresh it for the Python/librados scripts, the key inCEPH_KEY, the GUID-named objects, and theSuggests+--check-bindingsbehaviour.
Unit tests I ran on fe48966: KVMHACheckerTest (7), KVMHAMonitorTest (4), LibvirtCheckVMActivityOnStoragePoolCommandWrapperTest (6), LibvirtStoragePoolTest (29), LibvirtComputingResourceTest (323) and RbdUtilsTest (17) all pass. KVMStoragePoolManagerTest has 2 failures out of 5: testPoolsAreDeletedWithDetailsAsBefore and testPoolsWithHaSupportAreRemovedFromTheMonitorWhenDeleted.
Once 7 and its tests are fixed, I'm happy to approve.
| if (type == StoragePoolType.NetworkFilesystem) { | ||
| if (HighAvailabilityManager.LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT.contains(type)) { | ||
| _haMonitor.unregisterStoragePool(uuid); | ||
| deleteStatus = adaptor.deleteStoragePool(uuid); |
There was a problem hiding this comment.
This is still the delete without details for RBD and SharedMountPoint. LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT now contains both, so ModifyStoragePoolCommand(add=false) (LibvirtModifyStoragePoolCommandWrapper) still destroys the libvirt RBD pool and undefines its secret while VMs may be using it. fe48966 doesn't change this method. Only the HA unregister should depend on the HA type list. The delete itself should stay as it was on main, where only NFS uses the uuid-only delete:
if (HighAvailabilityManager.LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT.contains(type)) {
_haMonitor.unregisterStoragePool(uuid);
}
if (type == StoragePoolType.NetworkFilesystem) {
deleteStatus = adaptor.deleteStoragePool(uuid);
} else {
deleteStatus = adaptor.deleteStoragePool(uuid, details);
}| @Test | ||
| public void testPoolsAreDeletedWithDetailsAsBefore() { | ||
| // the delete which takes the details does not destroy the libvirt pool, which the VMs may still use, e.g. in maintenance of the pool | ||
| for (StoragePoolType type : new StoragePoolType[] {StoragePoolType.NetworkFilesystem, StoragePoolType.RBD, StoragePoolType.SharedMountPoint}) { |
There was a problem hiding this comment.
These tests fail against the current code (2 of 5):
testPoolsAreDeletedWithDetailsAsBeforefails at line 98, because RBD still goes to the unstubbeddeleteStoragePool(uuid), which returns false.- Both delete tests verify
monitor.removeStoragePool(...), but the manager callsunregisterStoragePool(...). That is the right call (no umount), so the tests should verifyunregisterStoragePool(lines 88, 100 and 112). NetworkFilesystemdoesn't belong in this "as before" list. On main, NFS has always useddeleteStoragePool(uuid), which unmounts it. Sending NFS to the details variant (a no-op default inStorageAdaptor) would leave NFS mounts behind. Please split NFS into its own case that expectsdeleteStoragePool(uuid).
| MAX_OBJECT_SIZE = 1024 * 1024 | ||
| # A heartbeat is written every <interval> seconds, but a write which starts on time can end up to TIMEOUT seconds | ||
| # later when Ceph is slow, so the heartbeat of a host which is alive can be older than the interval. | ||
| HEARTBEAT_SLACK = TIMEOUT |
There was a problem hiding this comment.
Suggestion, not blocking. The slack covers one operation timeout, but one heartbeat write can take connect (client_mount_timeout, 30 s) plus the write (rados_osd_op_timeout, 30 s). kvmheartbeat_rbd.py takes the timestamp after the connect, and a reader only sees it once the write lands. The monitor also writes its pools one after another before it sleeps for the interval. So a live host whose Ceph ops are slow but still succeed can show an age of about interval + 60–90 s, beyond the current interval + 30. 2 * TIMEOUT (or the agent's heartbeat.update.timeout, which is what the agent kills a write at) would match the worst case of one write. This is already more forgiving than NFS, so it can be a follow-up.
| // this host monitors no storage pool, e.g. its agent is not upgraded yet and does not monitor the pools of the type | ||
| // which a newer version does: it has nothing to tell about the host, which is not a dead host | ||
| logger.debug("KVMHAChecker has no storage pool to check the heart beat for {}, so it can not tell whether the host is alive.", hostAndPools); | ||
| return true; |
There was a problem hiding this comment.
Optional follow-up, not blocking. true here becomes CheckOnHostAnswer.isAlive(), and KVMHostActivityChecker.checkHostStatusWithNeighbourHosts returns Up on the first such answer. So one neighbour that monitors no pool (for example, the one RBD host without the bindings) stops every host in the cluster from being marked Down through VM HA. That is the safe direction. CheckOnHostAnswer already has a determined flag, though. Answering "undetermined" here, and having the activity checker skip undetermined answers and ask the next neighbour, would let the other hosts still decide.
Description
Extend the existing KVM Host-HA heartbeat/VM-activity-check framework (currently limited to NetworkFilesystem and SharedMountPoint pools) to also cover RBD primary storage, based on the approach from the old PR #5862, adapted to the current HA architecture and reusing the multi-monitor Ceph support already added in #6792.
cwiki: https://cwiki.apache.org/confluence/spaces/CLOUDSTACK/pages/451979203/Host+HA+support+for+Ceph+RBD+primary+storage+KVM
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?