Repository navigation
Conversation
A failed Metal command buffer (e.g. killed by the macOS GPU watchdog,
kIOGPUCommandBufferCallbackErrorImpactingInteractivity / ...Timeout) was
only logged when the stream was drained, and the reconstruction carried on
without its work: on 20 kHz Pb-Pb time frames this gave silently wrong
outputs (271,967-296,957 tracks vs 297,669 on CPU for the same TF).
Worse, a failed buffer never signals the events it was to signal, so
later buffers waiting for them on the GPU, and the host waiting for those,
blocked forever: one such run hung at 0% CPU.
Signal the stream-order event and the recorded event of a buffer from its
completion handler when it fails, so everything queued behind it
completes, and fail in DrainStream through GPUChkErr, like the CUDA and
HIP backends do on a failed synchronisation ("GPU Backend Failure").
Failures seen while the device is torn down are only logged, so the
destructor does not throw.
pbpb30 (M1 Max, abp.sh, 3 rounds, --runs 10): 966.7 ms/ev before, 962.5
after; Metal dumps byte-identical before and after; sim5pp dump still
identical to CPU.
With GPUCA_METAL_TIMELINE=<file>, write one line per command buffer once it completes: stream, status, CPU commit time, GPUStartTime and GPUEndTime, the kernels (with their threadgroup counts), blits, and the events it waits for and signals; and one line per host drain that had to wait. Off by default, the only cost then is a pointer test per encoded command. This is what is needed to see how the streams overlap on the GPU and where they wait for each other, without a profiler attached.
With --debug 1 and deviceTimers (the default), the kernel timers expect the backend to report the kernel's GPU time, and Metal reported none, so all kernel times read 0. Give the timed kernel a command buffer of its own and report its GPUEndTime - GPUStartTime. The times are only set once the completion is processed, so wait for it rather than for the stream's event. pbpb30 on M1 Max: 1012 ms of kernel time per event, vs 1218 ms on the CPU.
StreamWaitForEvents encoded the waits into the stream's open command buffer, after whatever work had accumulated there. With the merger's SectorRefit on all streams, stream 0's buffer (its own 4 sectors of refit) then waited for the 7 other streams: on pbpb30 its GPU work ended 36 ms before the wait was satisfied, and the buffer stayed incomplete meanwhile. Commit the pending work first, so that every GPU-side wait sits at the start of a buffer with no work before it. The host drains the stream right after the waits anyway. pbpb30, M1 Max, 3 rounds of 10 events: 958.8 -> 959.7 ms/ev; 6 rounds of the same change as a runtime switch: 971.2 -> 961.3 ms/ev. Metal dumps byte-identical on pbpb30, sim5pp identical to the CPU.
At 20 kHz occupancy the macOS GPU watchdog killed command buffers, and reset the desktop once, while all 8 streams ran multi-kernel buffers of up to ~1 s at the same time. Host-serialised execution (serializeGPU 3) was the only mode that survived. Add two runtime limits, off by default: - GPUCA_METAL_MAX_KERNELS_PER_BUFFER=N commits a stream's command buffer after N kernels, so that long chains (sector tracking, SectorRefit, TrackFit) become short buffers. Buffers of a stream stay ordered by its order event, the events of the other streams are untouched. - GPUCA_METAL_MAX_STREAMS_IN_FLIGHT=M makes the host wait, before it commits a buffer on an idle stream, until fewer than M streams have committed buffers that are not complete. This cannot deadlock: a buffer only waits for events signalled by buffers committed before it, since RecordMarker commits the signal at once, so the work in flight completes without the host. pbpb30, M1 Max, 6 rounds of 10 events (timeline: longest buffer, streams running at once): default unchanged (931 ms/ev, -3 ms over two run orders); N=1 M=1 1416 ms/ev, 87 ms, 1 stream (serializeGPU 3: 1413 ms/ev); N=1 M=2 1100 ms/ev, 87 ms, 2; N=2 M=4 941 ms/ev, 123 ms, 4. The longest buffers left hold the TrackFit kernel (~85 ms alone). Metal dumps byte-identical on pbpb30 with and without the limits, sim5pp identical to the CPU.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
At 20 kHz occupancy the macOS GPU watchdog killed command buffers, and
reset the desktop once, while all 8 streams ran multi-kernel buffers of
up to ~1 s at the same time. Host-serialised execution (serializeGPU 3)
was the only mode that survived. Add two runtime limits, off by default:
after N kernels, so that long chains (sector tracking, SectorRefit,
TrackFit) become short buffers. Buffers of a stream stay ordered by its
order event, the events of the other streams are untouched.
commits a buffer on an idle stream, until fewer than M streams have
committed buffers that are not complete. This cannot deadlock: a buffer
only waits for events signalled by buffers committed before it, since
RecordMarker commits the signal at once, so the work in flight completes
without the host.
pbpb30, M1 Max, 6 rounds of 10 events (timeline: longest buffer, streams
running at once): default unchanged (931 ms/ev, -3 ms over two run orders);
N=1 M=1 1416 ms/ev, 87 ms, 1 stream (serializeGPU 3: 1413 ms/ev);
N=1 M=2 1100 ms/ev, 87 ms, 2; N=2 M=4 941 ms/ev, 123 ms, 4. The longest
buffers left hold the TrackFit kernel (~85 ms alone). Metal dumps byte-identical on
pbpb30 with and without the limits, sim5pp identical to the CPU.