External Storage¶

Runtime-mediated transport¶

Client automatically uses the namespace runtime's external-payload transport when cluster discovery advertises it. Payloads above the discovered inline threshold are uploaded with the same runtime URL, namespace, and role credential already used by the client or worker. Workflow requests and worker completions carry a provider-neutral opaque reference; the backing bucket, container, filesystem path, and provider credentials remain inside Server or managed Cloud.

The same mediation applies to payload envelopes nested in workflow and activity polls and completions, signals, queries, updates, schedules, streams, and history exports. Incoming bytes are bounded by the runtime's advertised maximum and checked against both size_bytes and sha256 before Avro decode. A bounded verified-byte cache can reuse content during replay. Cache eviction only removes SDK memory: there is no client delete operation for runtime-owned objects.

Failures are available as typed exceptions:

ExternalPayloadNotFound
ExternalPayloadExpired
ExternalPayloadUnauthorized
ExternalPayloadUnavailable (retryable)
ExternalPayloadOversized
ExternalPayloadUnsupported
ExternalPayloadIntegrityMismatch

An unresolved or malformed runtime reference fails closed and is never returned as decoded workflow user data.

Direct self-hosted adapters¶

LocalFilesystemExternalStorage, S3ExternalStorage, GCSExternalStorage, and AzureBlobExternalStorage are explicit self-hosted integrations. Use them only with a runtime that has separately advertised direct-reference acceptance, and pass the selected driver through Client(external_storage=..., external_storage_threshold_bytes=...) or the serializer helpers. The SDK never chooses one because namespace discovery names the runtime's backing driver.

Provider clients are application-owned optional objects; they are not Durable Workflow SDK dependencies. Direct adapters retain the legacy typed-reference cleanup helper for application-owned objects. Runtime-owned opaque references have no SDK delete path.

external_storage ¶

External payload storage contracts for large Durable Workflow payloads.

ExternalPayloadIntegrityError ¶

ExternalPayloadIntegrityError(message, *, status=None, retryable=None, reference_id=None, body=None)

Bases: ExternalPayloadIntegrityMismatch, ValueError

Raised when fetched external payload bytes do not match their reference.

ExternalStorageDriver ¶

Bases: Protocol

Protocol implemented by pluggable external payload storage drivers.

put ¶

put(data, *, sha256, codec)

Persist data and return a stable URI for later fetches.

get ¶

get(uri)

Fetch previously persisted payload bytes.

delete ¶

delete(uri)

Delete previously persisted payload bytes when retention removes a run.

ExternalPayloadStoragePolicy `dataclass` ¶

ExternalPayloadStoragePolicy(enabled, driver=None, threshold_bytes=None, config=dict(), reference=None, prefix='', mode=None, status=None, integrity_required=True)

Normalized external payload storage policy from server or Cloud APIs.

from_dict `classmethod` ¶

from_dict(data)

Parse a server namespace or Cloud organization storage policy.

ExternalPayloadReference `dataclass` ¶

ExternalPayloadReference(uri, sha256, size_bytes, codec, expires_at=None, schema=EXTERNAL_PAYLOAD_REFERENCE_SCHEMA)

Stable wire envelope for a payload stored outside workflow history.

RuntimeExternalPayloadReference `dataclass` ¶

RuntimeExternalPayloadReference(reference_id, sha256, size_bytes, codec, schema=RUNTIME_EXTERNAL_PAYLOAD_REFERENCE_SCHEMA)

Provider-neutral reference owned by the authenticated namespace runtime.

ExternalPayloadCache ¶

ExternalPayloadCache(*, max_entries=128, max_bytes=16 * 1024 * 1024)

Bounded cache for verified external payload bytes during replay.

Cache entries are keyed by the complete reference identity. Bytes are inserted only by fetch_external_payload after size and sha256 verification has succeeded, so cache hits preserve the same integrity contract as a fresh driver fetch.

discard ¶

discard(reference)

Remove verified bytes for reference after external retention cleanup.

LocalFilesystemExternalStorage ¶

LocalFilesystemExternalStorage(root)

Dependency-free external storage driver for development and tests.

S3ExternalStorage ¶

S3ExternalStorage(client, *, bucket, prefix='')

External storage driver backed by a boto3-compatible S3 client.

The SDK does not depend on boto3. Applications that need S3 pass an already-configured client exposing put_object, get_object, and delete_object.

GCSExternalStorage ¶

GCSExternalStorage(client, *, bucket, prefix='')

External storage driver backed by a google-cloud-storage client.

The SDK does not depend on google-cloud-storage. Applications pass a configured client exposing bucket(name).blob(key).

AzureBlobExternalStorage ¶

AzureBlobExternalStorage(container_client, *, container, prefix='')

External storage driver backed by an azure-storage-blob container client.

The SDK does not depend on azure-storage-blob. Applications pass a configured container client exposing upload_blob, download_blob, and delete_blob.

external_storage_driver_from_policy ¶

external_storage_driver_from_policy(policy, *, s3_client=None, gcs_client=None, azure_container_client=None, local_root=None)

Build an SDK storage driver from a server or Cloud policy payload.

Provider SDK clients remain application-owned. Pass the already-configured S3/GCS/Azure client that matches the policy returned by the control plane.

store_external_payload ¶

store_external_payload(driver, data, *, codec, expires_at=None)

Store encoded payload bytes and return their reference envelope.

fetch_external_payload ¶

fetch_external_payload(driver, reference, *, cache=None)

Fetch payload bytes and verify size/hash before replay or decode.

delete_external_payload ¶

delete_external_payload(driver, reference, *, cache=None)

Delete referenced payload bytes and evict any verified replay cache entry.