Worker¶
See Separate workflow and activity processes for a runnable deployment example with distinct registrations, independent slots, remote activity retries and signal-driven shutdown.
Activity execution¶
By default, ordinary synchronous def remote activities run in a dedicated
thread pool bounded by max_concurrent_activity_tasks. Each thread receives
its own activity.context(). A blocked synchronous call does not occupy the
worker event loop, so other activity slots, workflow tasks and worker
heartbeats can progress. Async async def activities run on that event loop
and must use nonblocking I/O. Blocking work inside an async function still
stalls the loop.
From a synchronous activity, call
activity.context().heartbeat_sync(details) to report progress with the
original task, attempt and worker identity. It waits for the owner's event
loop to send the request using the normal client transport timeouts and
raises ActivityCancelled when cancellation is reported. Async activities use
await activity.context().heartbeat(details). Calling the synchronous method
on the owner event loop raises an error instead of deadlocking it.
Threads suit blocking I/O; they do not provide hard isolation or parallel Python CPU execution. Use bounded downstream timeouts and cooperative heartbeat checks. For CPU-heavy or untrusted callbacks, use separately isolated worker processes and their resource limits. The separately negotiated cooperative-cancellation capability retains its supervised process execution; selecting a default thread does not enable that protocol.
Bounded failure reports¶
This behavior is available in Python SDK 2.11.0 and later.
Failure reports retain at most 62 KiB of UTF-8 message text, 128 KiB of stack
text and 64 KiB of serialized details and diagnostics. These budgets include
truncation markers. The complete encoded JSON request body defaults to at most
1 MiB, counting JSON escaping, claim identities, command batches and metadata.
The message budget reserves 2 KiB below the original 64 KiB ceiling for
runtime-derived summaries.
A lower discovered Server limits.max_payload_bytes also applies. Configure
Client(..., failure_report_limit_bytes=...) when a proxy or transport has a
lower request limit. This option can lower the default cap; it cannot raise it.
Query-client clones inherit it. Payload warning thresholds remain advisory.
Oversized diagnostics produce a visible [truncated ...] message marker with
structured omitted-field names and original byte sizes. Activity failures also
carry this information in failure.diagnostics.failure_report; terminal
workflow failures use exception.diagnostics.failure_report. Other existing
report shapes keep the diagnostic JSON in the message marker. Unknown sizes
are null when optional details could not be encoded. Avro details are kept as
an intact valid envelope or omitted whole, before an unnecessary payload upload.
The original failure type, retry/non-retryable choice, authored retry delay and claim identity remain exact. Activity, workflow-task, query and update rejection reports, terminal workflow/update failures and local-activity attempt reports share the complete-request bound. A workflow-task replay failure still leaves the run open for repair; it does not become a terminal workflow failure.
A local activity's failure is bounded before workflow code observes it. First execution and cold replay therefore see the same persisted message. If a later command batch or newly discovered lower limit cannot fit that observed value, reporting fails explicitly rather than changing a value already used by the workflow. Prepared local activities observe the runtime's canonical history.
If essential identity or semantic fields cannot fit even without optional
text, the client raises FailureReportTooLarge before sending the report.
This reporting refusal is separate from the application failure. The worker's
existing reporting-error log identifies it; shortening a task ID or replacing
the application type would change which failure the runtime records.
Poll failures and recovery¶
Worker poll requests advertise optional JSON progress responses and also accept the ordinary JSON response from Servers that do not implement that transport. When a Server sends progress while waiting, each HTTP attempt still has an absolute deadline. Receiving another progress byte does not extend it. The existing transport retry budget and logical poll identity continue to apply. The SDK interprets the terminal status and retry headers before the ordinary error handling, including errors that arrive after HTTP 200 progress began.
If a connection failure, HTTP 502 or HTTP 504 ends a poll before its requested wait, a warning names the observed duration and configured wait. Check the proxy and HTTP timeout budgets. An intentional cancellation does not produce this cutoff warning. These transport observations do not establish that a task was delivered or authorize revoking its lease.
This backoff requires Python SDK 2.10.0 or later. Earlier packages use a fixed one-second poll-error wait.
Workflow, activity and query pollers back off independently after failed polls, including HTTP errors, rejected credentials and unreachable runtimes. The retry ceiling starts at 1 second, doubles on each consecutive failure and stops at 30 seconds. Each delay is chosen randomly between half and all of that ceiling, so workers do not retry in lockstep. Any successful poll, including an empty response, resets the ceiling to 1 second. Shutdown interrupts the wait.
The first failure is logged immediately. An unchanged failure is logged at most once per minute per poller until polling succeeds. HTTP 401 identifies rejected authentication; HTTP 403 identifies denied authorization. Check the worker credentials and their permissions. Once requests succeed again, polling recovers without a worker restart. Poll-error metrics still count every failed poll, including failures whose warning is suppressed.
Starting with Python SDK 2.10.0, warnings name the exception class and static poll route even when an httpx exception has an empty message. Transport exception messages are omitted because they can contain private request details. Server refusals retain their status and bounded reason; authentication failures retain their credential-safe summary. See request outcomes and errors for local cancellation and transport retry behavior.
Typed capacity backpressure follows the Server's advertised retry delay. Storage-admission failures retain their separate identity-preserving retry and worker-stop behavior. Neither uses this generic poll-failure backoff.
Lifecycle and shutdown¶
Worker.run() and Worker.run_until() register through the worker protocol
before polling. After registration succeeds, shutdown stops every poller and
waits for accepted work within the shared shutdown_timeout deadline before
removing that worker-plane registration. Work still running at the deadline is
cancelled. Calling stop() more than once shares the same shutdown operation
and result. No deregistration is sent if registration failed.
Unreleased source behavior, prepared for SDK 2.10.1: the final HTTP call and any recovery use the
original remaining shutdown budget. If Server advertises
worker_deregistration_fencing, Worker retains its registration token and uses
the fenced receipt endpoint. Matching transient responses and transport losses
can retry that same generation. A completed receipt can replay safely even if
a replacement now owns the worker ID. Authentication, malformed or mismatched
receipts, lost authority and expired receipts fail without retrying. Without
advertised fencing, Worker keeps one attempt through the legacy worker-plane
DELETE and does not replay an uncertain result. That legacy route has no
generation fence. Keep worker IDs unique to a process, and do not reuse an ID
until the prior worker has fully stopped.
If drain consumes the deadline, no late deregistration request is sent. A final
HTTP timeout or exhausted recovery raises RuntimeError and leaves removal
unconfirmed. The Server might have committed a request whose response was lost;
an exception is not proof that the registration still exists. Inspect the
runtime before taking an operator recovery action. These changes are not in a
published package yet.
Cancelling an await cannot stop a running synchronous activity thread. Its
physical slot remains occupied until the callback returns, and a late return
does not submit a result after its await was cancelled. If the callback has
not stopped by the shared shutdown deadline, shutdown raises RuntimeError
and retains the registration. Python can keep the process alive while that
thread remains running. Configure callback I/O timeouts and a supervisor
termination policy; a timeout is not proof that an external side effect was
cancelled or that the worker stopped gracefully.
An external stop() also interrupts run_until(); the pending run_until()
call ends with asyncio.CancelledError after its accepted workflow or activity
work has drained or been cancelled. If an async poller, the query-poller thread,
or accepted work cannot be stopped, stop() raises RuntimeError and leaves
the registration active. Treat that as an incomplete shutdown requiring
operator attention rather than continuing as though the worker stopped cleanly.
For a long-running process, coordinate shutdown with the same worker instance
and await both stop() and the run task:
run_task = asyncio.create_task(worker.run())
try:
await shutdown_requested.wait()
finally:
try:
await worker.stop()
finally:
await run_task
Deregistration authentication, protocol, and HTTP failures are raised from
stop() and run(); a process must not treat those failures as a clean
shutdown. When a worker-loop error and deregistration both fail, the loop error
remains the raised exception and the deregistration error is available through
its __cause__.
The worker-plane cleanup operation is separate from
Client.deregister_worker(). That existing control-plane method remains an
operator management action for retiring or recovering worker records and is not
used by normal Worker shutdown.
run_until(timeout=...) includes Server discovery and registration in its
workflow wait budget. A timeout before startup finishes identifies the worker
and workflow in the error. A timeout after startup reports that the workflow
has not reached a terminal state. Neither timeout cancels the durable run.
A replacement worker can continue that same run.
When stop() is called during startup, it waits for registration to finish
before starting the shared drain deadline. Server can record a registration
before its reply reaches the SDK. A worker that never receives that reply has
not begun polling and becomes stale under the Server's heartbeat policy.
Use the existing workflow handle and queue when bringing up a replacement.
Readiness and liveness¶
Worker.status() returns a read-only snapshot that separates readiness
(registered, every configured poller running, latest Server exchange of each
role succeeded) from liveness (event loop progressing, no poller exited, no
request outliving its bound). It performs no network I/O and can be called from
another thread. See Worker readiness and liveness
for states, probe configuration and shutdown guidance.
Long-polling worker that runs workflow and activity tasks.
Worker registers itself with the server for a given task queue, then
spawns poll loops for both workflow tasks and activity tasks. Each received
task is dispatched to the registered workflow class or activity function,
results are serialized, and success/failure commands are sent back to the
server. Workers stop their pollers, drain in-flight tasks on shutdown up to a
configurable shutdown_timeout, and remove successful worker registrations.
Most applications create one Worker per task queue and pass it the
same Client used for control-plane calls, plus
lists of workflow classes and activity callables registered via
durable_workflow.workflow.defn and durable_workflow.activity.defn.
Worker(client, *, task_queue, workflows=(), activities=(), capabilities=(), worker_id=None, build_id=None, poll_timeout=35.0, max_concurrent_workflow_tasks=10, max_concurrent_activity_tasks=10, max_concurrent_worker_sessions=10, sticky_cache_capacity=0, sticky_cache_max_bytes=16 * 1024 * 1024, sticky_cache_ttl_seconds=300, shutdown_timeout=30.0, heartbeat_interval=60.0, metrics=None, interceptors=(), external_storage=None, external_storage_threshold_bytes=None, external_storage_cache=None, require_query_task_authority=False)
Polls workflow and activity tasks and dispatches them to Python callables.
Return replay counters, retained entries and encoded history bytes.
async
¶
Run this worker until a workflow reaches a terminal state.
This is intended for examples, smoke tests, and single-workflow scripts.
Long-running workers should call run and coordinate shutdown from
their process supervisor. timeout includes registration and runtime
admission pauses; shutdown retains its separate drain timeout.
status(*, event_loop_stall_seconds=DEFAULT_EVENT_LOOP_STALL_SECONDS, request_overdue_seconds=None, request_stall_seconds=None)
Return a read-only readiness and liveness snapshot of this worker.
The snapshot performs no network I/O, claims no tasks, renews no leases and does not wait on the worker's event loop, so a health endpoint may call it from any thread, including while the event loop is stalled.
Readiness follows this worker's own registration: it becomes ready
once the Server acknowledges the registration, while every configured
poller is running, the latest Server exchange of every role succeeded
and no outstanding request is older than request_overdue_seconds.
A transient Server or network failure makes the worker unready; the
next successful exchange of that role restores readiness. Liveness
reports local progress: the event loop advanced within
event_loop_stall_seconds, no configured poller has exited, and no
outstanding Server request is older than request_stall_seconds.
Idle long polls, Server outages within the transport retry budget and
long-running activity handlers keep the worker live. Draining workers
stay live but are not ready; stopped and failed workers are neither.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
event_loop_stall_seconds
|
float
|
Maximum age of the event loop's progress tick before the worker is reported not live. |
DEFAULT_EVENT_LOOP_STALL_SECONDS
|
request_overdue_seconds
|
float | None
|
Maximum age of one outstanding poll or heartbeat before the worker is reported not ready. Defaults to one HTTP attempt deadline: the larger of the client request timeout and the poll's absolute HTTP deadline. |
None
|
request_stall_seconds
|
float | None
|
Maximum age of one outstanding poll or heartbeat before the worker is reported not live. Defaults to twice the client's complete transport retry budget for one such request. |
None
|
Returns:
| Type | Description |
|---|---|
WorkerStatus
|
A |
Return a typed lease handle held by this worker until shutdown.
async
¶
Stop polling, drain tasks, and remove the worker registration.
Repeated calls share one shutdown result. Generation-fenced recovery stays within the original shutdown deadline. Unfenced deregistration makes one attempt and does not retry an uncertain outcome.
Read-only worker readiness and liveness status.
durable_workflow.Worker.status returns a WorkerStatus
snapshot that an application health endpoint or orchestrator probe can serve.
Taking a snapshot never performs network I/O, claims tasks, renews leases or
waits on the worker's event loop, so it is safe to call from another thread,
including when the worker's event loop itself is stalled.
Readiness and liveness answer different questions:
- Ready means this worker holds a current registration for its own namespace and task queue, every configured poller is running, the most recent Server exchange of every role succeeded, and no outstanding request is overdue (unanswered beyond one HTTP attempt deadline). A transient Server or network outage makes the worker unready without making it unlive, and the next successful exchange of each affected role restores readiness without a restart.
- Live means the worker's local machinery is progressing: its event loop is advancing, no configured poller has exited, and no outstanding Server request has outlived its stall bound, which exceeds the transport retry budget. An idle long poll, a Server outage within that budget, or a long-running activity handler is not a liveness failure.
Missing observations are reported as unavailable (None), never as
success.
Bases: str, Enum
Lifecycle state of a Worker.
class-attribute
instance-attribute
¶
Constructed or registering; not yet admitted by the Server.
class-attribute
instance-attribute
¶
Registered with the Server; pollers and heartbeats are running.
class-attribute
instance-attribute
¶
stop() was requested; pollers stop and in-flight tasks drain.
class-attribute
instance-attribute
¶
Shutdown completed and the registration was removed.
class-attribute
instance-attribute
¶
Registration was refused, a poller failed, or shutdown could not confirm cleanup.
dataclass
¶
WorkerRoleStatus(role, running, in_flight_seconds, last_success_age_seconds, last_failure_age_seconds, consecutive_failures, last_failure_type, overdue, stalled)
Progress of one configured worker role.
Ages are seconds on the local monotonic clock, measured when the snapshot
was taken. None means the observation has not happened yet.
instance-attribute
¶
Age of the role's outstanding Server request, or None when idle.
instance-attribute
¶
Age of the role's most recent successful Server exchange.
instance-attribute
¶
Age of the role's most recent failed Server exchange.
instance-attribute
¶
Failed exchanges since the role's last success.
instance-attribute
¶
Exception class name of the most recent failure; never its message.
instance-attribute
¶
Whether the outstanding request is unanswered beyond one HTTP attempt deadline.
instance-attribute
¶
Whether the outstanding request has outlived the snapshot's stall bound.
dataclass
¶
WorkerStatus(worker_id, namespace, task_queue, state, ready, ready_reason, live, live_reason, registered_age_seconds, event_loop_progress_age_seconds, roles)
Point-in-time readiness and liveness of a worker.
The snapshot carries identity and progress only: no payloads, task arguments, exception messages or credentials.
instance-attribute
¶
Whether an orchestrator should route work to, or count, this worker.
instance-attribute
¶
Seconds since the Server acknowledged this worker's registration.
instance-attribute
¶
Seconds since the worker's event loop last advanced its progress tick.