Reliability — reconnect and retry

BigFred is used on phones over a local WiFi HTTP deployment (club layout or home). Brief radio drops, Android background-tab suspension, and dcc-bus restarts are expected. This section documents the implemented retry and grace mechanisms that keep throttle usable without surprising emergency stops.

Related: §6.3b Throttle overlay, §7e.7 Frontend integration, §7e.4 WebSocket protocol, §4.5 Drive session & DMS.

Overview

The control plane and data plane reconnect independently. Command retries on the data plane wait for the data-plane socket; radio stop retries wait for the control-plane socket.

Command station selection (lazy spawn)

When the driver opens Throttle and picks a command station, the SPA must attach the session to a running dcc-bus daemon.

Automatic attach (ThrottlePage):

1. On connected + session.sessionId + selectedCS > 0, the page calls setCommandStation(selectedCS) on the control plane.
2. The server may lazy-spawn the daemon. While spawning it emits session.commandStationChanged { status: "starting", wsUrl: null }. SocketContext keeps the previous wsUrl during starting so DccBusProvider is not torn down mid-handshake.
3. On success: ack { ok: true } and/or session.commandStationChanged { status: "running", wsUrl }. DccBusProvider opens the data-plane WebSocket at wsUrl.

Manual retry:

• If spawn fails (dcc_bus_unavailable, no_dcc_bus_ports_available, …), the setup panel shows an error Alert with „Spróbuj ponownie” (throttle:retry). Clicking it bumps retryTick, which re-runs the attach useEffect (same as picking the station again).
• Retry is blocked with control_offline when the control-plane socket is down.

Timeouts:

There is no automatic timer retry for failed spawn — only the user-facing button (or changing the picker selection).

WebSocket reconnect — control plane

Implementation: web/src/context/SocketContext.tsx.

Behaviour:

1. On onclose, pending request acks resolve immediately with control_offline (not left hanging until ack timeout).
2. Reconnect is scheduled with backoff: 250 → 500 → 1000 → 2000 ms (reset to 250 ms after a successful onopen).
3. visibilitychange (tab visible) and online (network back) trigger an immediate connect() if the socket is not already OPEN or CONNECTING. This matters on Android, where background tabs freeze timers and kill sockets.
4. reconnecting is true only after the first successful open (avoids flashing reconnect UI on first login).

waitForControlSocket (used by sendAction) polls up to 5 s for an OPEN socket before returning control_offline.

WebSocket reconnect — data plane

Implementation: web/src/context/DccBusContext.tsx.

Uses the same reconnect constants and visibilitychange / online triggers as the control plane.

Additional behaviour:

1. On onclose, all pending command acks resolve with dcc_bus_offline (feeds command retry — see below).
2. Application ping every heartbeatSecs from dcc-bus.opened (daemon default 5 s); server pong resets the dead-man timer.
3. When status returns to "open", ConnectedThrottle re-issues loco.subscribe for the current vehicle / train member addresses.

UI (settings icon in cockpit header):

Reconnect takes priority over command retry in the icon. The setup dialog still shows control/data plane chips; reconnect toasts were removed in favour of the header spinner.

Command retry (throttle driving)

Implementation: web/src/hooks/useRetryingSend.ts, wired through:

• useDebouncedSpeedSend → loco.setSpeed
• useDebouncedTrainSpeedSend → train.setSpeed
• useKeyedRetryingSend → loco.setFunction (per address:fn key)
• RadioStopButton → system.radioStop (dispatchAsync)

Policy

Constants (exported for tests / tuning):

SPEED_RETRY_BACKOFF_MS = 200
SPEED_RETRY_MAX = 2
RETRY_MAX_WAIT_MS = 3_000

Superseding (last-write-wins)

• Speed / train speed: a new slider move calls cancel() on the retry hook and replaces the debounced pending value. A stale speed never lands after a newer target.
• Functions: each ${dccAddress}:${fn} has its own retry chain; toggling F1 does not cancel retry on F2. Re-toggling the same function supersedes only that key.
• Radio stop: a new confirm supersedes the previous dispatchAsync chain via the shared useRetryingSend generation counter.

Debounce before send

sendSpeedNow / direction change / stop bypass debounce but still use retry.

UI feedback

commandRetrying OR radioStopRetrying OR DCC command retry flags drive the sync spinner on the settings icon (ThrottleCockpit). RadioStopButton reports onRetryingChange upward so radio-stop retries share the same indicator.

Radio stop

system.radioStop travels on the control plane (not dcc-bus). RadioStopButton uses dispatchAsync with the same retry policy as driving commands (control_offline + ack_timeout).

The confirmation dialog stays open until res.ok; the run button shows busy for the whole async retry window.

Backend — dead-man's switch and WebSocket grace

Implementation: pkgs/bigfred/dcc-bus/ws/handler.go, pkgs/bigfred/dcc-bus/cmd/session.go.

Daemon flags (defaults from dcc-bus CLI):

Ping silence (watchDeadman)

If the session sends no inbound frames for deadmanSecs, the daemon immediately calls HandleSessionClose with reason deadman and applies emergency stop on that session's subscribed addresses (or all user drive targets when it was the last tab).

WebSocket drop (reconnect grace)

When the browser TCP/WebSocket closes for any reason other than deadman:

1. The session is unregistered from the hub immediately (a new tab can connect).
2. HandleSessionClose / emergency stop is delayed by deadmanSecs (delayedSessionClose).
3. If the user reconnects within that window, isLastSessionForUser sees the new session and skips the layout-wide emergency stop.

This mirrors the ping budget: a short disconnect while the phone switches apps should not stop the train.

Emergency stop direction

applyEmergencyStop preserves the locomotive's cached forward direction from Redis (isLocoPlacedForward) so a stop does not flip consist direction on the layout.

Mobile — screen wake

Implementation: web/src/hooks/useWakeLock.ts + WakeLockKeeper in App.tsx.

BigFred is served over plain HTTP on LAN, so the Screen Wake Lock API is unavailable. The app uses NoSleep.js (muted video fallback) after the first user gesture, re-enabled on visibilitychange when the tab returns. This reduces WiFi power-save and accidental disconnects during driving.

Process-level recovery (supervisord)

When a dcc-bus process exits, supervisord restarts it (autorestart=true, exponential backoff up to ~30 s — see §7d). During restart the data-plane dial fails; the SPA reconnect loop above applies. The control plane stays up so session metadata and spawn state remain available.

Quick reference — source files