std::supervisors
Supervisors: long-lived tasks that outlive the code that started them,
restarted by policy when they stop. Supervisor.new makes one, and
only during boot; Supervisor.spawn and Supervisor.spawn_all start
work under it with a Restart disposition and a Backoff schedule,
and GiveUp says what happens when the schedule runs out.
Supervisor.flush waits for its tasks to finish. Where supervisors fit
in a program’s life and how they shut down are language rules
(Concurrency).
Import with import std::supervisors.
Exports
Section titled “Exports”enum Restart
Section titled “enum Restart”enum Restart Temporary Transient Permanent end
Which endings make a supervisor run a task again.
Restart says whether a stopped task runs again. Backoff says when,
and how many times. The three values match the BEAM’s :temporary,
:transient and :permanent.
A restart calls the function the spawn was given again, so whatever state that function sets up is built again.
A cancelled task is never restarted, under any value. Shutdown cancels tasks, and a restart then would keep the program from exiting.
Variants
-
TemporaryNever restarted. This is the default, since most work spawned under a supervisor is meant to finish.
A failure is still reported on standard error.
-
TransientRestarted if it fails, which is a trap in its body. A task that returned is done.
For work with an end, such as sending an email or writing an audit row: it is retried when it breaks and left alone once it finishes.
-
PermanentRestarted whenever it stops, whether it failed or returned.
For work with no end, such as a worker draining a queue or a server reading an inbox. Such a task returning on its own is a bug: its loop took a branch it should not have, or the channel it read was closed. Under
Permanentthat is reported and the task restarted. Under the other two values the worker would stop with no report.
Restart.inspect
Section titled “Restart.inspect”impl Debug for Restart
fn inspect(value: Restart) -> String
impl Debug.inspect
enum Backoff
Section titled “enum Backoff”enum Backoff Exponential { max_restarts: Int = 10, max_elapsed: Duration = Duration.minutes(15) } end
When a supervisor tries a failed task again, and how many times before it gives up.
Variants
-
ExponentialExponential delays with jitter. The first delay is one second, each later one is double the last, and none is longer than a minute. The supervisor gives up after
max_restartsrestarts or aftermax_elapsedspent trying, whichever comes first.Both fields have defaults, so
.Exponential{}is the schedule a supervisor has when none is named. Naming one field keeps the other’s default:.Exponential{max_restarts: 3}still gives up after fifteen minutes.max_restartscounts restarts, not runs, somax_restarts: 3runs the body at most four times.Jitter is always applied. Without it, a supervisor restarting fifty identical tasks would retry them all at the same instant.
A run longer than a minute, the longest delay, counts as recovered: the next failure starts again from the first delay with the whole count and the whole time. So a worker that fails once a month never uses up
max_restarts.
Backoff.inspect
Section titled “Backoff.inspect”impl Debug for Backoff
fn inspect(value: Backoff) -> String
impl Debug.inspect
enum GiveUp
Section titled “enum GiveUp”enum GiveUp Report Exit end
What a supervisor does when it decides a task will not run again:
under Restart.Temporary at its first failure, and under
Backoff.Exponential once max_restarts or max_elapsed runs out.
Neither choice restarts the program. Exit ends the process, and
whatever runs the process decides whether it starts again.
Variants
-
ReportThe default: report the failure on standard error and carry on. The supervisor’s other tasks keep running.
-
ExitEnd the process: every supervisor is drained, then the program exits with a non-zero status.
Nomi does not start the program again. A service manager or an orchestrator does, and the new process runs
bootagain, so it starts with new connections, new state and new supervisors.It is a poor fit for a long-running process with nothing above it to restart it, such as a desktop application or an embedded runtime. There, use
Reportand your own monitoring.
GiveUp.inspect
Section titled “GiveUp.inspect”impl Debug for GiveUp
fn inspect(value: GiveUp) -> String
impl Debug.inspect
enum Wait
Section titled “enum Wait”enum Wait Forever UpTo Duration end
How long Supervisor.flush may block.
Variants
-
ForeverBlock until the supervisor’s work finishes, however long that takes.
-
UpToBlock until the work finishes or this much time passes, whichever comes first. It returns as soon as the work is done. A bound past about 292 years, the longest wait the clock counts, waits that long, as
Foreverdoes.
Wait.inspect
Section titled “Wait.inspect”impl Debug for Wait
fn inspect(value: Wait) -> String
impl Debug.inspect
enum FlushOutcome
Section titled “enum FlushOutcome”enum FlushOutcome Flushed TimedOut end
How a Supervisor.flush ended.
A deadline on the Context in force stops the work that runs past
it. A bound given to flush stops nothing and returns TimedOut, so
the caller decides what a slow flush means.
Variants
-
FlushedEvery task outstanding when
flushwas called has finished. -
TimedOutThe bound ran out with work still outstanding. Nothing was cancelled: the work carries on, and shutdown still bounds it.
FlushOutcome.inspect
Section titled “FlushOutcome.inspect”impl Debug for FlushOutcome
fn inspect(value: FlushOutcome) -> String
impl Debug.inspect
type Supervisor
Section titled “type Supervisor”type Supervisor
An owner for tasks that outlive the call that started them.
A concurrent block does not produce its value until every task in
it has finished, which does not suit a signup handler that responds
now and sends mail later. A function that spawns under a
Supervisor returns while the work keeps running, and the
supervisor decides how long shutdown waits for it. Every task has
one of these two owners.
A supervisor is a set of tasks that share a policy. Its tasks share
one max_running limit and wait on each other for a slot,
Supervisor.flush waits for all of them, and shutdown drains all of
them within one shutdown_timeout. Tasks under different supervisors
do not affect each other. Put work under one supervisor when it
should be treated the same way at shutdown and on failure, not
because it serves the same feature.
Supervisors are fields of the application struct and are created in
boot, one per policy the program needs:
struct App context: Context audit: Supervisor notify: Supervisor end fn boot() -> Result<App, String> Ok(App{ context: Context.root(), audit: Supervisor.new(shutdown_timeout: Duration.seconds(30), max_running: 4), notify: Supervisor.new(shutdown_timeout: Duration.milliseconds(200), max_running: 50), }) end
Supervised work lives in the process and is not durable: a crash after the response is sent and before the email is sent loses the email. That is acceptable for a welcome message and not for a receipt.
Supervisor.new
Section titled “Supervisor.new”fn new(max_running: PositiveInt, shutdown_timeout: Duration = Duration.seconds(5), restart: Restart = .Temporary, backoff: Backoff = .Exponential{}, on_give_up: GiveUp = .Report) -> Supervisor
type function on Supervisor
Creates a supervisor.
max_running is how many of its tasks run at once, and it has no
default. The limit protects whatever the tasks call, such as a
connection pool or a rate-limited service, and only the program
knows that number. Work reaches a supervisor from call sites all
over the program, so no limit would mean no limit for the whole
run. A supervisor that holds one long-lived worker has
max_running: 1.
shutdown_timeout is how long shutdown lets this supervisor’s work
run before cancelling it. Work that finishes within it is never
cancelled. With a zero or negative timeout, shutdown cancels the
work at once. Supervisors drain at the same time, each within its
own timeout, so a 30-second one and a 200-millisecond one do not
wait for each other. When the timeout runs out, the runtime cancels
what is left, waits up to one second for it to unwind and run its
deferred calls, and then abandons it, so a task that never reaches
a safe point cannot hang shutdown. Boot’s deferred calls run after
the drain, so they close what the work used only once the work has
stopped. A timeout past about 292 years, the longest wait the clock
counts, lets the work run until it finishes.
restart, backoff and on_give_up say what happens when a task
stops: see Restart, Backoff and GiveUp.
A supervisor can be created only during boot: in boot itself or
in a constructor boot calls, which is how a server type owns its
supervisor and its restart policy. A call to Supervisor.new that
boot does not reach is a compile error. Every supervisor lives as
long as the program and has its own max_running, so one created
per request would give each request its own limit: max_running: 8
under a thousand requests at once would be eight thousand tasks.
Supervisor.spawn
Section titled “Supervisor.spawn”fn spawn(supervisor: Supervisor, body: () -> Unit) -> Task<Unit>
type function on Supervisor
Starts body as a task under supervisor and returns at once.
The call never blocks. When max_running tasks are already
running, the new task waits for a slot on its own.
It returns a Task, as Task.spawn does, and this one need not be
awaited, because the supervisor drains it at shutdown. Use the
handle to cancel the task or to read its Task.outcome, or discard
it with _ =.
The body must return Unit. Nobody has to await supervised work,
so a returned value would have nowhere to go, and a returned Err
would be dropped without a report. Wrap the call in a function that
handles the failure and reports it.
Supervisor.spawn_all
Section titled “Supervisor.spawn_all”fn spawn_all<T>(supervisor: Supervisor, source: Iter<T>, f: (T) -> Unit) -> List<Task<Unit>>
type function on Supervisor
Starts one task per item of source under supervisor, each
running f on its item.
It takes no max_running: the supervisor’s limit applies to these
tasks and to every other task spawned under it.
f returns Unit, as all supervised work does, so there are no
values to collect. Use the handles to cancel tasks, or pass them to
Task.await_all to wait for this batch and not for everything
under the supervisor. It consumes the whole source, so a source that
is Iter.is_infinite traps before any task is spawned.
Supervisor.flush
Section titled “Supervisor.flush”fn flush(supervisor: Supervisor, bound: Wait = Wait.Forever) -> FlushOutcome
type function on Supervisor
Blocks until every task outstanding under supervisor has finished.
It cancels nothing. Shutdown cancels what is left when
shutdown_timeout runs out; flush waits and reports. Call it
where queued work must have gone out before the program reports
success, or in a test, so the background work a call started
finishes before the test asserts on it.
A task is outstanding if it was spawned before the call: spawn
adds its task before it returns, so work started on an earlier line
is always included, and work a flushed task spawns is waited for
too. Work spawned after the call is not waited for.
With no bound it waits as long as the work takes. Pass
Wait.UpTo(d) to limit the wait; when the time runs out it returns
FlushOutcome.TimedOut and the work carries on:
case Supervisor.flush(App.notify, Wait.UpTo(Duration.seconds(5))) FlushOutcome.Flushed -> report_sent() FlushOutcome.TimedOut -> carry_on() end
Supervisor.inspect
Section titled “Supervisor.inspect”impl Debug for Supervisor
fn inspect(value: Supervisor) -> String
impl Debug.inspect