A water utility streaming-adjacent job sometimes dies on transient network blips. What resilience setting should they enable?

Water utility jobs that die on transient network blips need automatic restart resilience so brief connectivity losses to storage, metastore, or upstream APIs do not strand the pipeline until a human arrives. Enabling automatic restarts (with sensible bounds) lets Lakeflow recover from ephemeral failures without waiting for a morning desk visit, which is essential for streaming-adjacent or overnight civic telemetry that feeds leak detection, pressure monitoring, or customer outage portals.

Idempotent sinks so the job recovers from transient network failures without waiting for a manual restart address correctness on retry, not revival itself: idempotent sinks prevent duplicated downstream writes when a restarted job replays data, but they do nothing to restart the job automatically when a transient network blip kills it unattended—without an automatic-restart policy, the job still sits dead until someone starts it by hand. Watermark horizons so the job recovers from transient network failures without waiting for a manual restart address event-time completeness, not liveness: watermark horizons control how long to wait for late events before closing windows, but they do not bring a job back online after a transient network failure kills it. Task parallelism so the job recovers from transient network failures without waiting for a manual restart is capacity/performance tuning for sustained volume, not a fix for the job dying outright on a network blip—adding more parallel slots to a job that has already crashed does not restart it.

Exam caveat: automatic restarts help transient faults; they are dangerous for poison-pill failures (bad schema, bad data) unless bounded by a maximum retry/restart limit. Pair restarts with logging and alerts so repeated failures still page humans. Distinguish job/task restart settings from idempotent sinks and watermark horizons—related resilience ideas, different controls.

Operational check: review the job or pipeline restart policy for enabled automatic restarts, confirm a max restart/retry ceiling exists for non-transient poison cases, and simulate a brief network fault in non-prod to verify recovery without manual start. Ensure failure alerts still fire when restarts exhaust.