Engineering Resilient Compute Pipelines: Handling WebGPU Device Loss

The failure mode nobody simulates

Server-side GPU compute runs on dedicated hardware in a controlled environment. The CUDA context persists for the lifetime of the process. Driver updates are scheduled during maintenance windows. If the GPU fails, the orchestrator restarts the pod on a healthy node.

Browser-side GPU compute has none of these guarantees.

The GPU device can be destroyed at any moment. The driver can reset. The OS can reclaim GPU resources. Chrome's GPU watchdog kills compute shaders that exceed 2 seconds. A background tab can have its GPU access revoked. An external GPU can be physically disconnected.

When any of this happens, every GPUBuffer, every GPUComputePipeline, every GPUBindGroup, and every in-flight GPUCommandBuffer becomes invalid. Simultaneously. With no retry mechanism built into the WebGPU API.

If your application does not handle GPUDevice.lost, a single driver hiccup turns your "GPU-accelerated" dashboard into a white screen with an uncaught promise rejection in the console.

How WebGPU device loss works

The WebGPU specification provides exactly one mechanism for detecting device loss: the GPUDevice.lost property, which is a promise that resolves when the device becomes unusable.

const device = await adapter.requestDevice();

device.lost.then((info) => {
  console.log(`Device lost: ${info.reason}`);
  // info.reason is "destroyed" or "unknown"
});

The promise resolves with a GPUDeviceLostInfo object containing a reason field. The reason is either "destroyed" (the application called device.destroy() explicitly) or "unknown" (every other case: driver crash, timeout, resource reclamation, hardware removal).

Two properties of this API make it deceptively simple:

First, it fires once. The promise resolves a single time. There is no event listener, no retry callback, no reconnection hook. After the promise resolves, the device is permanently dead. You must request an entirely new adapter and device.

Second, it is asynchronous. The promise resolves in a microtask, not synchronously at the point of failure. If your code is in the middle of encoding a command buffer when the device is lost, the encoding calls do not throw. They silently produce invalid state. The queue.submit() call may or may not throw, depending on the browser implementation and timing. You cannot rely on synchronous error handling to detect device loss.

The six causes of browser GPU device loss

Understanding what triggers device loss is essential for estimating how often your production users will encounter it.

Cause 1: GPU watchdog timeout

Chrome and Edge enforce a timeout on GPU operations. If a compute shader dispatch or render pass exceeds the browser's threshold (typically 2 seconds on Windows, variable on other platforms), the browser kills the GPU process and triggers device loss for every tab using that GPU.

This affects your application even if your shaders are fast. Another tab running a poorly optimized WebGL game can trigger the watchdog, and your compute pipeline in a different tab loses its device.

Cause 2: Driver crash or update

GPU drivers crash. On Windows, the Display Driver Recovery mechanism (TDR) resets the GPU after a hang and terminates all GPU contexts. On macOS, IOKit can reclaim GPU resources under memory pressure. On Linux, the DRM subsystem resets the GPU on fence timeout.

Driver updates on Windows can occur in the background via Windows Update. When the new driver loads, all existing GPU contexts are invalidated. The user sees no notification. Your application sees GPUDevice.lost with reason "unknown".

Cause 3: External GPU disconnection

Thunderbolt eGPUs are common in enterprise environments. Users dock and undock laptops throughout the day. If your application initialized WebGPU on the external GPU and the user disconnects, the device is lost instantly. The system switches to the integrated GPU, which has different capabilities, different memory bandwidth, and different performance characteristics.

Cause 4: Power management transitions

When a laptop switches from AC to battery, the OS may power down the discrete GPU to conserve energy. Windows Hybrid Graphics and macOS automatic graphics switching handle this transparently for rendering (the compositor migrates to the integrated GPU), but WebGPU compute contexts on the discrete GPU are destroyed.

Cause 5: Background tab throttling

Browsers aggressively throttle background tabs. Chrome may freeze a background tab's GPU access entirely after 5 minutes of inactivity. When the tab returns to the foreground, the GPU device may have been reclaimed. The application must handle re-initialization.

Cause 6: System sleep and resume

After a system sleep/wake cycle, GPU state may or may not survive depending on the OS, driver, and hardware. On many configurations, the GPU context is destroyed during sleep and not automatically restored. The application resumes with a dead device.

What happens to your pipeline state

When the device is lost, every GPU object created from that device becomes invalid:

A compute engine that caches any of these objects (and every performant engine does, because pipeline creation and shader compilation are expensive) must invalidate its entire cache on device loss. If it does not, subsequent operations will attempt to use dead objects and produce GPUValidationError or silent failures.

Our transparent fallback mechanism

Our engine treats device loss as an expected operational event, not an exception. The recovery path has three phases that execute without application-level intervention.

Phase 1: Immediate state invalidation

The GPUDevice.lost callback fires. Within the same microtask, the engine:

Invalidates the pipeline cache. Every cached GPUComputePipeline is removed. The cache is a Map<string, GPUComputePipeline> keyed by shader source hash and pipeline layout. On device loss, the entire map is cleared. No selective invalidation. No stale entry risk.

Drains the buffer pool. The engine maintains a pool of pre-allocated GPUBuffer objects to avoid per-operation allocation overhead. Every buffer in the pool (both idle and in-flight) is marked as dead. The pool's free list and allocation index are reset to empty. No attempt is made to call buffer.destroy() on the dead buffers, as the device is already gone.

Drops cached bind groups. Bind groups reference specific buffers and pipelines. All are invalid. The bind group cache is cleared.

Sets the device state flag. A deviceAvailable boolean flips to false. Every subsequent GPU dispatch call checks this flag before attempting any GPU operation.

device.lost.then((info) => {
  pipelineCache.clear();
  bufferPool.invalidate();
  bindGroupCache.clear();
  deviceAvailable = false;
  currentDevice = null;
  currentAdapter = null;

  if (info.reason !== "destroyed") {
    scheduleReProbe();
  }
});

The entire invalidation completes in under 0.1 ms. It touches only in-memory data structures. No async operations. No GPU calls. No network requests.

Phase 2: In-flight workload re-dispatch

If a compute operation was in progress when device loss occurred, the operation must complete. The caller is waiting for a result. Returning an error is the wrong answer in a production system where the GPU path was an optimization, not a requirement.

The engine maintains a pending operation queue. Each entry contains the operation descriptor, input data (still in JavaScript heap memory as typed arrays), and a result promise. On device loss, the engine iterates the pending queue and re-dispatches each operation to the next available tier.

The tier selection follows the same adaptive dispatch logic, minus the GPU option. If Web Workers are available (navigator.hardwareConcurrency > 1), the operation dispatches to the worker pool. Otherwise, it runs on the main thread.

The caller's promise resolves with the correct result. The caller does not know that the GPU was involved, that it failed, or that the operation was re-dispatched. The latency is higher (CPU path instead of GPU path), but the result is identical.

async function dispatch(operation: Operation, data: TypedArray): Promise<TypedArray> {
  if (deviceAvailable) {
    try {
      return await gpuDispatch(operation, data);
    } catch (e) {
      // Device may have been lost between the flag check and the dispatch.
      // Fall through to CPU path.
    }
  }

  if (navigator.hardwareConcurrency > 1) {
    return workerDispatch(operation, data);
  }

  return cpuDispatch(operation, data);
}

There is a race condition to handle: the device can be lost between the deviceAvailable check and the actual GPU dispatch call. The try/catch around gpuDispatch catches this case. The catch block does not retry on the GPU. It falls through to the CPU path immediately.

Phase 3: Automatic re-probing

After device loss (with reason "unknown", not "destroyed"), the engine schedules a re-probe of GPU capabilities. The re-probe does not happen immediately. It waits for the next compute invocation. There is no point re-probing hardware if the application is idle.

On the next dispatch() call where deviceAvailable is false and the re-probe is pending:

Step 1: Request a new adapter.

const adapter = await navigator.gpu.requestAdapter();

If this returns null, no GPU is available. The engine continues in CPU-only mode. This handles the case where an external GPU was disconnected and no other GPU exists.

If this returns an adapter, the engine reads the adapter info and compares it to the previous adapter. If the vendor, architecture, or device ID has changed (e.g., switched from discrete to integrated GPU after undocking), the engine flags a hardware change.

Step 2: Request a new device.

const device = await adapter.requestDevice({
  requiredLimits: { /* same limits as before */ }
});

A new GPUDevice.lost callback is registered on the new device immediately.

Step 3: Re-run calibration microbenchmarks.

The engine runs the same memory bandwidth and dispatch overhead microbenchmarks used during initial startup. This takes under 200 ms. The calibration ratio is recalculated. If the hardware changed (discrete to integrated, or a driver update altered performance characteristics), the new ratio reflects the current state.

Step 4: Resume GPU dispatch.

The deviceAvailable flag flips to true. The pipeline cache, buffer pool, and bind group cache are empty but functional. Pipelines and buffers will be created on demand as operations arrive. The first few operations after recovery pay a one-time compilation cost (5 to 20 ms for pipeline creation), then subsequent operations hit the warm cache.

The race conditions

GPU device loss introduces three race conditions that a correct implementation must handle.

Race 1: Loss during command encoding

The application calls device.createCommandEncoder(), encodes several compute passes, then calls queue.submit(). If device loss occurs after encoding begins but before submission, the encoder is invalid. The submit() call may throw a GPUValidationError or silently fail.

Our engine catches this by wrapping the encode-submit sequence in a try/catch. If submission fails, the operation enters the re-dispatch path.

Race 2: Loss during buffer readback

After a compute dispatch, the engine calls resultBuffer.mapAsync(GPUMapMode.READ) to read the result back to CPU. If device loss occurs while the map is pending, the mapAsync promise rejects with an OperationError.

Our engine catches this rejection and re-dispatches the operation to the CPU tier. The input data is still in JavaScript heap memory. No data is lost.

Race 3: Loss between adapter probe and device creation

During the re-probe sequence, the adapter may become invalid between requestAdapter() and requestDevice(). This can happen if the GPU is removed during the re-probe itself (unlikely but possible with Thunderbolt eGPUs).

The engine wraps the entire re-probe sequence in a try/catch. If any step fails, the engine remains in CPU-only mode and schedules another re-probe for the next invocation.

What the application developer sees

From the application's perspective, the compute engine exposes a single dispatch() function that accepts an operation and data and returns a promise of results. Device loss is invisible:

// This code works regardless of device loss.
// No error handling for GPU failures needed at this level.
const sorted = await engine.dispatch("radix_sort", inputArray);
const filtered = await engine.dispatch("filter_gt", { data: sorted, threshold: 1000 });
const totals = await engine.dispatch("group_sum", { data: filtered, groupBy: "region" });

If the GPU is available, each call dispatches to the GPU. If the GPU is lost mid-sequence, the failing operation transparently re-dispatches to CPU. Subsequent operations use the CPU until the GPU is re-probed and restored. The application code does not branch on GPU availability. It does not catch GPU-specific errors. It calls dispatch() and gets results.

The only observable difference is latency. The GPU path for a 500,000-element sort takes 3 ms. The CPU fallback takes 12 ms. The user might notice a momentary slowdown on one interaction, then normal speed resumes after re-probing.

Monitoring and observability

Silent recovery is correct behaviour for the end user. For the engineering team, silence is unacceptable. You need to know when devices are lost, why, how often, and how long recovery takes.

Our engine emits structured telemetry events for every state transition:

These events feed into whatever observability pipeline your infrastructure uses. For browser-based dashboards, they can be batched and sent to your logging endpoint. For internal tooling, they can be surfaced in a debug panel.

The telemetry answers the questions your SRE team will ask: "How often are users hitting device loss?" "Which GPU vendors/drivers are most unstable?" "What is the p99 recovery time?" "Are we spending more time on CPU fallback than we should?"

Comparison with server-side GPU failure handling

The browser-side model is simpler in one critical respect: the input data never leaves JavaScript heap memory. The GPU receives a copy via device.queue.writeBuffer(). When the GPU dies, the copy is lost, but the original is intact. There is no checkpointing problem. Re-dispatch is just re-computing on the CPU with data that is already in memory.

Why this matters for production deployments

Enterprise applications run on thousands of machines with heterogeneous hardware, inconsistent driver versions, and unpredictable power management policies. The probability that at least one user experiences GPU device loss in a given day is not low. It is near certain.

A compute engine that crashes on device loss is a liability. A compute engine that degrades gracefully, recovers automatically, and logs the event for your operations team is production infrastructure.

This fault tolerance is not a feature we bolted on after the performance work. It is integral to the adaptive dispatch architecture from the start. The three-tier model (CPU, Workers, GPU) was designed so that every operation has a correct fallback at every tier. The precision analyser ensures the CPU fallback produces numerically identical results. The divergence classifier ensures no workload is GPU-exclusive.

Every path through the system produces correct results. The GPU makes it faster. Losing the GPU makes it slower. It never makes it broken.

That is the standard for enterprise AI automation infrastructure. Not "works when everything is perfect." Works when the GPU crashes, the driver resets, the laptop undocks, and the user never knows.