Add large-payload blob auto-purge (opt-in singleton job, worker/SDK side) - #758
Add large-payload blob auto-purge (opt-in singleton job, worker/SDK side)#758YunchuWang wants to merge 12 commits into
Conversation
82ae04d to
0ac2dc3
Compare
![github-code-quality[bot]](https://cdn.github.com/u/9919?s=60&v=4){kind=small}
0752610 to
cab0e9a
Compare
YunchuWang
changed the title
cab0e9a to
c05b15a
Compare
Large orchestration payloads are externalized to Azure Blob Storage as `blob:v1:<container>:<blobName>` tokens. The DTS backend stores those tokens but cannot delete the backing blobs (it has no storage credentials) — only this SDK can. This adds an opt-in, whole-scheduler singleton durable entity + orchestration job (mirroring src/ExportHistory) that drains payload rows the backend has soft-deleted and deletes their blobs, then acks so the backend can hard-delete the rows. Design: - PayloadStore.DeleteAsync is virtual (default throws NotSupportedException so it is non-breaking for existing external subclasses); BlobPayloadStore overrides it to decode the token and call DeleteIfExistsAsync (idempotent). - BlobPurgeJob (TaskEntity singleton): Create is a no-op when already Active so racing client processes don't disturb the running job; Run starts a fixed-id orchestrator. - BlobPurgeJobOrchestrator (perpetual): fetch a batch of tombstones, delete the blobs with capped parallelism, ack the successful deletions (failed tokens stay tombstoned to retry), idle on a timer when empty, ContinueAsNew periodically. - ExecuteBlobPurgeJobOperationOrchestrator bridges client -> entity. - Two new unary RPCs on TaskHubSidecarService: GetTombstonedPayloads / AckPurgedPayloads (authoritative proto follow-up: microsoft/durabletask-protobuf#76). - LargePayloadStorageOptions gains AutoPurge (opt-in, default false) and PayloadPurgeBatchSize (default 500). - Client-side BlobPurgeJobStarter (IHostedService) ensures the singleton job when AutoPurge is enabled, without blocking host startup. Worker always registers the entity/orchestrators/activities so a client-enabled job has something to run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
c05b15a to
306d19f
Compare
…er simplification - Drop the `Dto` suffix now that the payload records are first-class public types in `Microsoft.DurableTask.Client` (`TombstonedPayload`, `PayloadPurgeAck`). - Collapse the magic `500` batch-size literal into a single `BlobPurgeConstants.DefaultBatchSize` used everywhere. - Rename `BlobPurgeJobStatus.Stopped` -> `Pending` (still the zero value) and remove the dead `Failed` member (nothing ever set it; the job self-heals). - Make the perpetual orchestrator self-heal: wrap each cycle in try/catch so a transient backend/entity/activity failure logs, backs off, and continues instead of failing the orchestration and killing the eternal loop. - Ack poison tokens: `DeleteExternalBlobActivity` now returns a three-way `BlobDeleteResult` (Deleted/Discarded/Retry). Malformed tokens are discarded and acked so the backend can clear the stuck row instead of re-streaming it forever; transient failures stay tombstoned to retry. - Replace the single-value `BlobPurgeJobCreationOptions` record with a plain `int` on `BlobPurgeJob.Create`. - Guard the client fetch RPC: `GetTombstonedPayloadsAsync` throws `ArgumentOutOfRangeException` unless `0 < limit < 1000`. - Simplify `BlobPurgeJobStarter` to a fixed-instance-id fire-once: drop the entity-active pre-check and schedule the Create bridge once with a fixed instance id, retrying only until the backend is reachable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The store's consumer is the interceptor wired up in UseExternalizedPayloadsCore, so Core is the single fallback registration site. Move the TryAddSingleton<PayloadStore> out of each configure overload and into Core, symmetrically for the client and worker extensions. TryAdd keeps reusing a shared store (AddExternalizedPayloadStore or the sibling builder) and never creates a redundant one. The worker overload switches from AddSingleton to the shared TryAddSingleton in Core as part of the move. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…store Restore the original design: the client never registers a PayloadStore, it only consumes a shared/external one (via AddExternalizedPayloadStore or an in-process worker). Only the worker self-registers a fallback store. Remove the client Core's TryAddSingleton<PayloadStore> block (and its now-unused Microsoft.Extensions.DependencyInjection.Extensions using); the client Core PostConfigure still resolves PayloadStore from the shared/worker registration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adds an opt-in, worker/SDK-driven auto-purge mechanism for Azure Blob externalized payloads, enabling the SDK (which has storage credentials) to delete backing blobs after the backend soft-deletes payload rows and exposes tombstones via new gRPC RPCs.
Changes:
- Adds a singleton durable-entity + perpetual orchestrator “blob purge job” with activities to fetch tombstones, delete blobs, and ACK successful deletions.
- Extends the public client surface (
DurableTaskClient+GrpcDurableTaskClient) with purge RPC methods and adds the corresponding RPCs/messages to the vendored sidecar proto. - Introduces
LargePayloadStorageOptions.AutoPurge+PayloadPurgeBatchSize, a client-side hosted starter, and a new test project covering deletion/token behavior, DI, job create semantics, and options defaults.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/AzureBlobPayloads.Tests/PayloadStore/BlobPayloadStoreTests.cs | Adds unit coverage for token decoding and idempotent blob deletion behavior. |
| test/AzureBlobPayloads.Tests/Options/LargePayloadStorageOptionsTests.cs | Verifies AutoPurge and batch-size default values. |
| test/AzureBlobPayloads.Tests/DependencyInjection/UseExternalizedPayloadsTests.cs | Validates conditional DI registration of the hosted purge starter. |
| test/AzureBlobPayloads.Tests/AzureBlobPayloads.Tests.csproj | Introduces a dedicated test project for the AzureBlobPayloads extension. |
| test/AzureBlobPayloads.Tests/AutoPurge/DeleteExternalBlobActivityTests.cs | Covers poison vs transient failure classification for delete activity. |
| test/AzureBlobPayloads.Tests/AutoPurge/BlobPurgeJobTests.cs | Tests singleton entity Create semantics and batch-size persistence/defaulting. |
| src/Grpc/orchestrator_service.proto | Adds new unary RPCs and messages for tombstone fetch + purge ACK. |
| src/Extensions/AzureBlobPayloads/PayloadStore/PayloadStore.cs | Adds virtual DeleteAsync to support deletable external stores without breaking subclasses. |
| src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs | Implements token-based blob deletion via DeleteIfExistsAsync(IncludeSnapshots). |
| src/Extensions/AzureBlobPayloads/Options/LargePayloadStorageOptions.cs | Adds AutoPurge enablement + batch-size option. |
| src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskWorkerBuilderExtensions.AzureBlobPayloads.cs | Ensures purge entity/orchestrations/activities are registered and avoids redundant PayloadStore registration. |
| src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskClientBuilderExtensions.AzureBlobPayloads.cs | Conditionally registers the hosted starter when AutoPurge is enabled via configuration callback. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Orchestrations/ExecuteBlobPurgeJobOperationOrchestrator.cs | Adds client→entity bridge orchestration for driving entity ops through orchestration surface. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Orchestrations/BlobPurgeJobOrchestrator.cs | Implements the perpetual purge loop with capped parallelism, idle delay, and periodic ContinueAsNew. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Models/BlobPurgeJobStatus.cs | Adds job status model for the singleton entity. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Models/BlobPurgeJobState.cs | Adds persisted entity state (timestamps, counts, batch size, last error). |
| src/Extensions/AzureBlobPayloads/AutoPurge/Models/BlobDeleteResult.cs | Adds delete outcome enum to drive ACK vs retry decisions. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Logs.cs | Adds structured log messages for purge job lifecycle and failures. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Entity/BlobPurgeJob.cs | Adds singleton entity to manage activation and orchestrator startup. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Constants/BlobPurgeConstants.cs | Centralizes job IDs and defaults for purge job components. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs | Adds hosted service to ensure the singleton purge job without blocking startup. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Activities/GetTombstonedPayloadsActivity.cs | Fetches tombstoned payloads via injected DurableTaskClient. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Activities/DeleteExternalBlobActivity.cs | Deletes a blob token and classifies outcomes to avoid batch failure. |
| src/Extensions/AzureBlobPayloads/AutoPurge/Activities/AckPurgedPayloadsActivity.cs | ACKs successful deletions to backend via DurableTaskClient. |
| src/Client/Grpc/GrpcDurableTaskClient.cs | Implements the new purge RPC client methods. |
| src/Client/Core/TombstonedPayload.cs | Adds serializable model for purge tombstones across activity boundary. |
| src/Client/Core/PayloadPurgeAck.cs | Adds serializable model for purge ACKs across activity boundary. |
| src/Client/Core/DurableTaskClient.cs | Adds new virtual purge methods to the public core client API. |
| Microsoft.DurableTask.sln | Adds the new AzureBlobPayloads test project to the solution. |
…t-of-range); drop redundant downstream coercions Validate the batch size at its single point of specification - LargePayloadStorageOptions.PayloadPurgeBatchSize - and fail fast when out of range, mirroring the existing ThresholdBytes setter. Valid range is 1..999 inclusive, matching the gRPC GetTombstonedPayloadsAsync contract (which rejects limits >= 1000); add BlobPurgeConstants.MaxBatchSize = 999 to express the upper bound. Now that the value is guaranteed valid at specification, remove the redundant '> 0 ? x : DefaultBatchSize' coercions from every downstream consumer (starter, entity, orchestrator, activity); they simply use the value directly. Tests: add setter validation tests (throws for 0, -1, 1000, 1001; accepts 1, 500, 999) and repurpose the entity's non-positive test to assert the entity now stores the batch size verbatim (no coercion). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…edPayloadsAsync/AckPurgedPayloadsAsync Both new methods previously let RpcException(StatusCode.Cancelled) escape, unlike every other method in this client. Wrap each method's gRPC call in a try/catch that translates Cancelled to OperationCanceledException, mirroring the existing pattern used by GetOrchestrationHistoryAsync and the other client methods. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| builder.Services.Configure(builder.Name, configure); | ||
|
|
||
| UseExternalizedPayloadsCore(builder); | ||
|
|
| await blob.DeleteIfExistsAsync( | ||
| DeleteSnapshotsOption.IncludeSnapshots, | ||
| conditions: null, | ||
| cancellationToken); |
| processedCycles++; | ||
| if (processedCycles > ContinueAsNewFrequency) | ||
| { | ||
| context.ContinueAsNew(new BlobPurgeJobRunRequest(input.JobEntityId, batchSize, ProcessedCycles: 0)); | ||
| return null!; | ||
| } |
| BlobDeleteResult result = await context.CallActivityAsync<BlobDeleteResult>( | ||
| nameof(DeleteExternalBlobActivity), | ||
| tombstone.Token); |
…stonedPayloads limit check to allow 1000 1000 is a valid, accepted batch size end-to-end: the proto documents no numeric range and the backend only clamps via min(limit, configuredCap) without rejecting >= 1000, so the whole range is contained in this SDK. Raise BlobPurgeConstants.MaxBatchSize from 999 to 1000 (the options setter already validates against it) and relax the gRPC GetTombstonedPayloadsAsync guard from 'limit >= 1000' to 'limit > 1000'. Update the two PayloadPurgeBatchSize theories so 1000 is asserted accepted and 1001 still throws. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| // Conditional DI: register the auto-purge starter only when the caller opted into auto-purge. Peek the | ||
| // flag now by running the configure delegate against a probe (options configurators are pure setters). | ||
| LargePayloadStorageOptions probe = new(); | ||
| configure(probe); | ||
| if (probe.AutoPurge) |
| BlobDeleteResult result = await context.CallActivityAsync<BlobDeleteResult>( | ||
| nameof(DeleteExternalBlobActivity), | ||
| tombstone.Token); |
| internal BlobPayloadStore(BlobContainerClient containerClient, LargePayloadStorageOptions options) | ||
| { | ||
| this.containerClient = containerClient ?? throw new ArgumentNullException(nameof(containerClient)); | ||
| this.options = options ?? throw new ArgumentNullException(nameof(options)); | ||
| } |
| } | ||
|
|
||
| return EncodeToken(this.containerClient.Name, blobName); | ||
| return EncodeToken(blob.Uri); |
| public void Create(TaskEntityContext context, int purgeBatchSize) | ||
| { | ||
| if (this.State.Status == BlobPurgeJobStatus.Active) | ||
| { | ||
| logger.BlobPurgeJobAlreadyRunning(context.Id.Key); | ||
| return; | ||
| } | ||
|
|
||
| this.State.Status = BlobPurgeJobStatus.Active; | ||
| this.State.PurgeBatchSize = purgeBatchSize; | ||
| this.State.CreatedAt ??= DateTimeOffset.UtcNow; | ||
| this.State.LastModifiedAt = DateTimeOffset.UtcNow; | ||
| this.State.LastError = null; | ||
|
|
da8a190 to
e74f633
Compare
| await blob.DeleteIfExistsAsync( | ||
| DeleteSnapshotsOption.IncludeSnapshots, | ||
| conditions: null, | ||
| cancellationToken); |
| // Conditional DI: register the auto-purge starter only when the caller opted into auto-purge. Peek the | ||
| // flag now by running the configure delegate against a probe (options configurators are pure setters). | ||
| LargePayloadStorageOptions probe = new(); | ||
| configure(probe); | ||
| if (probe.AutoPurge) |
| public void Create(TaskEntityContext context, int purgeBatchSize) | ||
| { | ||
| if (this.State.Status == BlobPurgeJobStatus.Active) | ||
| { | ||
| logger.BlobPurgeJobAlreadyRunning(context.Id.Key); |
berndverst
left a comment
berndverst
left a comment
There was a problem hiding this comment.
Performance and reliability review
Requesting changes before merge.
The Azure Identity and Blob client lifetime itself is sound: BlobPayloadStore is a singleton on the working DI paths, it creates one container client/pipeline, per-blob clients reuse that pipeline and its HTTP connection pool, and Azure.Core caches and renews the access token rather than acquiring one for every delete. The gRPC client/channel is likewise long-lived when DI is configured correctly. I do not see an Entra token-acquisition or socket-lifetime leak in the normal path.
The main risks are in the purge control plane: one durable activity per blob creates excessive orchestration history/replay; all-retry batches immediately refetch the same rows; the Blob SDK retry budget can hold each 32-item wave for a long time; several supported client/worker/named-client DI topologies fail; a container mismatch is acknowledged even though the blob was not removed; and the singleton job is not self-healing once its perpetual orchestrator terminates. Mixed backend versions and Blob versioning/soft-delete semantics also need explicit handling.
Before merge, I recommend batching deletes within activities (or using Blob Batch), adding adaptive job-level backoff plus backend lease/attempt metadata, fixing the DI and named-client resolution paths, separating malformed-token errors from configuration mismatches, reconciling the actual perpetual orchestration, and adding provider-resolution and full-cycle tests.
References: credential reuse, Azure SDK client lifetime, Blob retry guidance, delete semantics, and versioning.
|
|
||
| async Task<DeleteOutcome> DeleteOneAsync(TaskOrchestrationContext context, TombstonedPayload tombstone) | ||
| { | ||
| BlobDeleteResult result = await context.CallActivityAsync<BlobDeleteResult>( |
There was a problem hiding this comment.
[High / performance] This creates one durable activity per blob. At the 1,000-item limit, a cycle emits 1,000 activity scheduled/completed pairs; five full cycles before ContinueAsNew produce at least ~10,000 history events and roughly 160 replay waves. The durable control-plane cost can dominate the DELETE calls. Please move deletion into chunked activities (for example 25-100 tokens each), or use Blob Batch where appropriate, and parallelize within the activity.
|
|
||
| List<PayloadPurgeAck> acks = await this.DeleteBatchAsync(context, tombstones); | ||
|
|
||
| if (acks.Count > 0) |
There was a problem hiding this comment.
[High / reliability] If every delete returns Retry, acks.Count is zero, no exception is raised, and this loop immediately fetches the same non-empty batch again. A persistent 403, lease/configuration failure, unsupported store, or storage outage can therefore create a hot loop and keep the oldest rows blocking later work. Add exponential backoff with jitter for zero/high-failure batches, and consider cursor/lease/attempt metadata in the backend contract so failed rows do not cause head-of-line blocking.
|
|
||
| static async Task DrainAsync(List<Task<DeleteOutcome>> tasks, List<PayloadPurgeAck> acks) | ||
| { | ||
| DeleteOutcome[] outcomes = await Task.WhenAll(tasks); |
There was a problem hiding this comment.
[High / performance] Bounded concurrency is good, but each of these 32 activities invokes the existing Blob pipeline configured with eight retries and a two-minute network timeout. A timeout path can occupy a wave for roughly 18 minutes before the orchestrator advances; processing 1,000 rows can consequently take many hours, followed by an immediate refetch if none succeeded. For bulk purge, use a smaller per-operation SDK retry budget and let the job-level policy provide adaptive backoff/circuit breaking. Fast persistent 403s generally bypass SDK retries and hit the hot-loop path even sooner.
|
|
||
| builder.Services.Configure(builder.Name, configure); | ||
|
|
||
| UseExternalizedPayloadsCore(builder); |
There was a problem hiding this comment.
[High / DI] This configured client overload never registers PayloadStore, while UseExternalizedPayloadsCore adds a PostConfigure<PayloadStore, ...> dependency. A client-only host using this overload fails when the client/options are resolved unless an undocumented worker/shared registration happens to provide the store. Please register the store here or make AddExternalizedPayloadStore an explicit, validated prerequisite.
| r.AddEntity<BlobPurgeJob>(); | ||
| r.AddOrchestrator<ExecuteBlobPurgeJobOperationOrchestrator>(); | ||
| r.AddOrchestrator<BlobPurgeJobOrchestrator>(); | ||
| r.AddActivity<GetTombstonedPayloadsActivity>(); |
There was a problem hiding this comment.
[High / DI] GetTombstonedPayloadsActivity and AckPurgedPayloadsActivity both require DurableTaskClient, but AddDurableTaskWorker does not register one. In a supported split client/worker deployment, activity activation fails at dispatch. Please either perform fetch/ack through the worker's existing backend channel or explicitly register/require a client for this topology.
| // id and no dedupe policy the backend would purge and replace the terminal instance on every | ||
| // host restart.) Only (re)schedule when the bridge is absent, or ended in a Failed/Terminated | ||
| // state that may never have applied Create - which lets a failed setup self-heal. | ||
| OrchestrationMetadata? existing = await this.client.GetInstanceAsync( |
There was a problem hiding this comment.
[High / lifecycle] This checks only the short-lived bridge orchestration. Once that bridge is Completed, the starter returns even if the actual fixed-ID BlobPurgeJobOrchestrator later fails or is terminated; the entity remains Active, so Create also no-ops and the job never recovers. Please reconcile the perpetual instance itself and define restart/stop/update behavior so disabling auto-purge or changing batch size is not permanently ignored.
| // Conditional DI: register the auto-purge starter only when the caller opted into auto-purge. Peek the | ||
| // flag now by running the configure delegate against a probe (options configurators are pure setters). | ||
| LargePayloadStorageOptions probe = new(); | ||
| configure(probe); |
There was a problem hiding this comment.
[Medium / configuration] Invoking the caller's configure callback a second time assumes it is a pure setter; it may construct a credential or have other side effects. More importantly, this probe is the only place the starter is registered, so shared configuration via AddExternalizedPayloadStore(... AutoPurge = true) plus the no-argument overload silently ignores auto-purge. Register the hosted service once and have it inspect the resolved named options in StartAsync.
| new P.GetTombstonedPayloadsRequest { Limit = limit }, | ||
| cancellationToken: cancellation); | ||
| } | ||
| catch (RpcException e) when (e.StatusCode == StatusCode.Cancelled) |
There was a problem hiding this comment.
[Medium / compatibility] During a mixed rollout, an older backend returns gRPC Unimplemented for this new RPC. Only cancellation is translated here, so the activity/orchestrator retries the unsupported operation indefinitely without a clear terminal diagnostic. Please add capability negotiation or map Unimplemented to an explicit unsupported-backend state that stops/disables the purge job.
|
|
||
| // Idempotent by design: DeleteIfExistsAsync returns false (rather than throwing) when the blob is | ||
| // already gone, so re-delivered tombstones and concurrent purges from multiple worker replicas are safe. | ||
| await blob.DeleteIfExistsAsync( |
There was a problem hiding this comment.
[Medium / storage semantics] IncludeSnapshots does not delete blob versions. With versioning enabled, deleting the base blob turns the current version into a retained previous version, so acknowledging this as purged may reclaim no storage; soft delete likewise retains bytes until expiration. Please either require/document an appropriate lifecycle policy, explicitly enumerate/delete versions, or make the user-visible status clear that this only deletes the current base blob.
| builder.Object.UseExternalizedPayloads(options => options.AutoPurge = true); | ||
|
|
||
| // Assert - the purge-job starter is the only IHostedService this path registers. | ||
| services.Should().ContainSingle(d => d.ServiceType == typeof(IHostedService)); |
There was a problem hiding this comment.
[Medium / tests] Descriptor presence does not exercise the runtime dependency graph, which is why the missing client-side PayloadStore, worker-only DurableTaskClient, and named-client resolution failures are not detected. Please build the provider and resolve the actual named/default client, worker task factories, options, and hosted service for client-only, worker-only, combined, and named configurations. The purge orchestrator/starter also need full-cycle tests for zero acknowledgements, replay/history growth, restart behavior, and backend Unimplemented.
…worker; fixes client-only DI failure) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44c27836-c49c-45fd-ae2b-3309f9c3f0f0
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
src/Extensions/AzureBlobPayloads/DependencyInjection/DurableTaskClientBuilderExtensions.AzureBlobPayloads.cs:44
- This invokes the caller-provided
configuredelegate twice (once viaServices.Configure(...)and again against a probe instance). If the delegate has side-effects or is non-deterministic, this can lead to incorrect AutoPurge detection or unexpected behavior. Consider registering the hosted service unconditionally and having it no-op whenAutoPurgeis false (or otherwise avoid double-invoking the delegate).
// Conditional DI: register the auto-purge starter only when the caller opted into auto-purge. Peek the
// flag now by running the configure delegate against a probe (options configurators are pure setters).
LargePayloadStorageOptions probe = new();
configure(probe);
if (probe.AutoPurge)
src/Extensions/AzureBlobPayloads/PayloadStore/BlobPayloadStore.cs:176
- This call mixes a named argument (
conditions:) with a later positional argument (cancellationToken), which is not valid C# and will fail to compile. The cancellation token should be passed as a named argument as well.
await blob.DeleteIfExistsAsync(
DeleteSnapshotsOption.IncludeSnapshots,
conditions: null,
cancellationToken);
src/Extensions/AzureBlobPayloads/AutoPurge/Client/BlobPurgeJobStarter.cs:68
Task.WhenAny(...)returns without observing exceptions fromensureTaskwhen it faults. This can lead to unobserved task exceptions (and can surface viaTaskScheduler.UnobservedTaskException). If the task completes, it should be awaited (inside a try/catch) to observe any fault/cancellation.
if (pending is not null)
{
// The ensure loop observes cancellation and returns promptly; swallow any faulted/cancelled result.
await Task.WhenAny(pending, Task.Delay(Timeout.Infinite, cancellationToken)).ConfigureAwait(false);
}
src/Extensions/AzureBlobPayloads/PayloadStore/PayloadStore.cs:35
- Adding a new virtual member is generally binary compatible, but it can still be a behavioral/source-breaking change for existing external subclasses that already defined a same-signature
DeleteAsyncmethod (it may become hidden, and calls via aPayloadStorereference will bind to the new base implementation). The remarks currently state this is non-breaking; it would be safer to document this edge case explicitly.
/// The default implementation throws <see cref="NotSupportedException"/>. Stores that externalize
/// payloads to deletable storage (for example Azure Blob Storage) should override it. It is declared
/// virtual rather than abstract so that adding it does not break existing external subclasses.
/// </remarks>
Summary
Large orchestration payloads are externalized to Azure Blob Storage by the
AzureBlobPayloadsextension asblob:v1:<container>:<blobName>tokens. The DTS backend stores those tokens but cannot delete the backing blobs — it has no storage credentials; only this SDK can. This PR implements the worker/SDK side of large-payload blob auto-purge.Design (opt-in singleton durable entity + orchestration job)
Instead of an always-on background stream, this mirrors the existing
src/ExportHistoryfeature: an opt-in, whole-scheduler singleton durable entity + orchestration job that drains soft-deleted payload rows the backend exposes and deletes their blobs, then acks so the backend can hard-delete the rows.PayloadStore.DeleteAsync— added as avirtualmethod (default throwsNotSupportedException, so it is non-breaking for existing external subclasses).BlobPayloadStoreoverrides it to decode the token and callDeleteIfExistsAsync(idempotent — deleting a missing blob is a no-op).BlobPurgeJob(TaskEntitysingleton) —Createis a no-op when already Active so racing client processes don't disturb the running job (intentionally softer thanExportJob.Create, which throws).Runstarts a fixed-instance-id orchestrator.BlobPurgeJobOrchestrator(perpetual) — each cycle: fetch a batch of tombstones, delete the blobs with capped parallelism (32), ack only the successful deletions (failed tokens stay tombstoned to retry), idle on a 1-minute timer when there's nothing to purge, andContinueAsNewevery 5 cycles to keep history small. Activities use a small retry policy.ExecuteBlobPurgeJobOperationOrchestrator— client → entity bridge (mirrors export).GetTombstonedPayloadsActivity,DeleteExternalBlobActivity(returnsfalse+ logs on failure so one bad token can't fail the batch),AckPurgedPayloadsActivity.DurableTaskClient(GetTombstonedPayloadsAsync/AckPurgedPayloadsAsync), overridden inGrpcDurableTaskClient— mirroring how ExportHistory addedListInstanceIdsAsync/GetOrchestrationHistoryAsync. The purge activities injectDurableTaskClientdirectly; no dedicated gRPC client is needed. AzureManaged reusesGrpcDurableTaskClient, so it inherits these methods with no extra client.LargePayloadStorageOptions.AutoPurge(opt-in, defaultfalse) andPayloadPurgeBatchSize(default500).BlobPurgeJobStarter(IHostedService) ensures the singleton job whenAutoPurgeis enabled, on a background task that does not block host startup and retries until the backend is reachable. The worker always registers the entity/orchestrators/activities (not gated onAutoPurge) so a client-enabled job always has something to execute.gRPC contract
Two new unary RPCs added to
TaskHubSidecarServiceinsrc/Grpc/orchestrator_service.proto(worker is the client; wire paths/TaskHubSidecarService/GetTombstonedPayloadsand/AckPurgedPayloads):C# stubs are generated at build time by
Grpc.Tools(not committed). The authoritative proto change is a follow-up in microsoft/durabletask-protobuf#76; once it merges,src/Grpc/orchestrator_service.protoshould be re-synced from upstream (content identical to what's here). The vendored change lets this PR build/test standalone.Testing
dotnet build Microsoft.DurableTask.sln— succeeds, 0 errors.dotnet test test/AzureBlobPayloads.Tests— 11 passed (BlobPayloadStore delete/idempotency +BlobPurgeJob.Createno-op-when-Active + options defaults).dotnet test test/ExportHistory.Tests— 147 passed (shared patterns unaffected).dotnet test test/Client/Core.Tests+test/Client/Grpc.Tests— 43 + 47 passed (coreDurableTaskClient/GrpcDurableTaskClientedits).Notes / deviations
BlobPurgeJobStatus.Pendingis the0/default value (mirroringExportJobStatus.Pending=0) so a freshly initialized entity never appearsActive.RecordPurgedentity op to track a cumulativePurgedCount.Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com