Repository navigation
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14450 +/- ##
============================================
+ Coverage 20.76% 20.84% +0.08%
- Complexity 21903 22054 +151
============================================
Files 6496 6498 +2
Lines 591612 592246 +634
Branches 72585 72665 +80
============================================
+ Hits 122851 123459 +608
+ Misses 455424 455396 -28
- Partials 13337 13391 +54
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
🔴 Test Coverage Grade:
|
| Metric | Value |
|---|---|
| Line coverage | 25.96% |
| Branch coverage | 20.32% |
Grade Scale
| Grade | Line Coverage | Meaning |
|---|---|---|
| 🟢 A | ≥ 80% | Excellent - this code sleeps well at night 😴 |
| 🟡 B | 60-79% | Good - almost there, don't stop now 😉 |
| 🟠 C | 40-59% | Acceptable - your code is wearing a seatbelt, but no airbags 😬 |
| 🔴 D | 20-39% | Marginal - boldly shipping where no test has gone before 🖖 |
| ⛔ F | < 20% | Failing - tests? what tests? 🔥 |
Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run
|
@blueorangutan package |
|
@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19651 |
|
@blueorangutan test |
|
@Pearl1594 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian Build Failed (tid-17194) |
|
@blueorangutan test |
|
@Pearl1594 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17195)
|
|
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch. |
|
Hi @Pearl1594 can you please address the conflicts? |
|
@blueorangutan package |
|
@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✖️ debian ✔️ suse15. SL-JID 19870 |
| * With reportIfHeartBeatFailedForOneStoragePool the host is dead as soon as one pool has no heart beat, | ||
| * otherwise it is dead only when none of the pools has one. | ||
| */ | ||
| private boolean checkPoolsHeartBeat() { |
There was a problem hiding this comment.
| private boolean checkPoolsHeartBeat() { | |
| private boolean doesAnyPoolHasHeartBeat() { |
| * Reads a host's sanlock delta-lease state for a CLVM/CLVM_NG volume group; the lease is renewed | ||
| * by sanlock itself, and a host that stops renewing is reset by its watchdog. | ||
| */ | ||
| public class ClvmSanlockHaChecker { |
There was a problem hiding this comment.
any chance to move this to clvm plugin?
- Check DB lock host first; if LV not active there, read the sanlock lease of the LV from one healthy host (max 2 readers) via QUERY_LEASE_OWNER - Map lease owner to CloudStack host through host.sanlock.id detail - Free lease or dead/restarted owner: activate on preferred host directly - Owner still renewing but agent down: fail fast instead of waiting - Fan-out to all hosts only as fallback (lease unreadable or owner unmapped), now parallel and skipping hosts that did not answer earlier - Lock transfer retries "lock start in progress" refusals - Alert on hosts sharing a sanlock host id at storage connect - Lookup cost grows with disks, not with cluster size
- Treat a template as in use when any volume on any CLVM pool of the same
VG uses it, so a CLVM_NG backing file is never removed under a volume
- Deactivate the template LV on all hosts of the pools sharing the VG
before removing it, releasing stale shared locks ("LV locked by other host")
- Log hosts that cannot deactivate it by name, uuid and state, with the
lvchange command to run there
- Keep the LV when another pool on the same VG still references it
- DestroyCommand reports a failed delete as failure, so the template pool
record is kept and eviction retried instead of leaking the LV
- CLVM cleanup treats only "not found" as already deleted; lvs errors and
exceptions now fail the delete
- Add clvm.lock.query.threads (default 5, restart required) for the parallel
host calls, replacing the hardcoded 16
Pearl1594
changed the title
…ckage, rename heartbeat check
|
|
@blueorangutan package |
|
@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19950 |
Description
This PR fixes: #14178
This PR does 4 things for CLVM and CLVM_NG primary storage (lvmlockd + sanlock).
Host HA support
CLVM and CLVM_NG pools were not in
LIBVIRT_STORAGE_POOL_TYPES_WITH_HA_SUPPORT, so the KVM HA activity check never gave a usable answer for them and a failed host was never resolved.wdmdresets a host that stops renewing it.sanlock client host_status -s lvm_<vg> -D. The host is dead only whenlast_check - last_live >= 8 * io_timeout + 60(140s with the defaultio_timeout). Missing or unparsable state is treated as alive, so a host is never declared dead without evidence.lvmconfig local/host_id) as the host detailhost.sanlock.id.KVMHostActivityCheckerpasses it to the checking host inHostTO.sanlockHostId. Sanlock host ids are host-wide and can repeat across clusters, so the lease is looked up by VG name plus host id.KVMHAMonitorno longer unmounts or checks libvirt for CLVM pools, and deleting a CLVM pool removes it from the monitor.Retry of transient lock refusals
After a holder failed or rebooted, the first exclusive activation (
lvchange -aey) can be refused for a short time: either the previous holder's sanlock lease has not expired yet (LV locked by other host), or the lockspace is still being joined on a rebooted host (lock start in progress). Previously the activation was tried once, the lookup returned no holder, and the VM start went ahead with the LV inactive.The agent then waits for the device node (
KVMStoragePoolManager.getPhysicalDisk, 100 tries every 3 seconds). That wait only checks whether/dev/<vg>/<lv>exists. It never requests the lock, andClvmStorageAdaptor.handleMissingDeviceNodeactivates onlytemplate-volumes. So even after the lease expires nothing activates the LV and the start fails after about 6-7 minutes. HA then retries only afterrestart.retry.interval(600s by default).ClvmPoolManagernow retries the exclusive activation on the chosen host until sanlock grants it or the budget runs out:locked by other hostandlock start in progressare retried. Any other failure or exception is a single attempt, as before.clvm.lock.recovery.retry.timeout(default 180s, counted from the start of the lookup;0disables the retry) andclvm.lock.recovery.retry.interval(default 10s). The lease expires8 x io_timeout + 60seconds after the holder stops renewing, so raise the timeout iflvmlockdruns with a larger--sanlock-timeout.Lock holder lookup from the sanlock lease
Finding the host that holds a volume's exclusive lock used to ask every host in the cluster, one after another, so every dead or rebooting host cost a timeout. During a mass restart this happened for every disk.
The lookup now uses three steps, cheapest first:
clvmLockHostId) is asked first. If the LV is active there, it is the holder: one call.sanlock direct dumpon the<vg>-lvmlockLV, at the offset inlv_lockargs) with the newQUERY_LEASE_OWNERoperation. At most 2 hosts are tried. The lease owner is mapped to a CloudStack host throughhost.sanlock.id, the same detail the HA check uses.Also:
lock start in progresson the destination.clvm.lock.query.threads(default 5, needs a management server restart) sizes the thread pool for the parallel host calls.The cost of the lookup grows with the number of disks, not with the number of hosts. On a 4-host lab with 60 VMs and all hosts reset, all VMs were running again after 380-396s, against 441s with the old lookup (NFS: 366s). The lease owner always mapped to exactly one host, and the ask-every-host fallback never ran.
Template LV removal
Deleting an unused CLVM_NG template left its LV behind. The template LV stays active in shared mode on every host that used it, so
lvremovefailed withLV locked by other host.LibvirtDestroyCommandWrapperignored the result and reported success, so thetemplate_spool_refrecord was dropped and the LV leaked for good.lvchange -an <lv>command to run there.not foundas already deleted.lvserrors and exceptions now fail the delete.Tested on the lab:
Behaviour changes to be aware of
KVMInvestigatorcan now conclude it is down, send the "Host is down" alert, and HA can restart HA-enabled VMs elsewhere. Before this, investigation was skipped for CLVM-only clusters.sanlock,wdmd,lvmlockdandlvmlocksenabled at boot, and a working watchdog device.lvsfailure, exception) now fail and are retried instead of reporting success and leaking the LV.Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Screenshots (if appropriate):
How Has This Been Tested?
HA support
Retry
lock start in progress) and the data volume activated at once. VM Running in 232s. Without the retry the same scenario failed after 6m47s and the VM stayed stopped.locked by other host) over 48s, activated about 136s after the reset, VM Running 142s after the reset.clvm.lock.recovery.retry.timeout=0): one refusal per volume, no retry, the start fails as before. Setting back to 180 restored the retry without a restart.Sanlock lease lookup
lvchange -anon the holder: the lease read showed no owner and the LV was activated on the destination at once.Runningon the other host after ~100s.Template delete
lvremove, which failed withLV locked by other host, but the delete was reported as successful and thetemplate_spool_refrecord was dropped. An 8 GB LV was left with no record.forced=true) while a VM was still running from the template: the template was removed from secondary storage, but the primary LV and its record were kept and the VM kept running.lvchange -ancommand to run, the LV and record were kept, and eviction was logged for retry. The other hosts deactivated it.How did you try to break this feature and the system with this change?