Retry retryable ack failures instead of abandoning the work item

[pinodeca]

Fixes #41.

Problem

The worker dispatcher abandoned an activity work item on any ack_work_item error, ignoring the error's retry classification. Abandoning redelivers the item and re-runs the handler, so a few milliseconds of database contention cost a full redundant activity execution.

The dispatcher poll loop in the same file already gets this right for fetch_work_item — it branches on e.is_retryable() and backs off. The ack path did not.

Changes

src/runtime/dispatchers/worker.rs — add ack_work_item_with_retry: up to 5 attempts with exponential backoff (100ms → 1s, ~1.5s total budget) for retryable errors, then fall back to the existing abandon path. Used by the success, application-error, and cancellation completion paths.

Permanent errors still abandon immediately, so poison handling and worker_abandon_on_ack_failure_enables_retry are unaffected.

tests/session_e2e_tests.rs — test_multi_worker_heterogeneous_config asserted a + b == 3, i.e. exactly-once activity execution. Activities are documented as at-least-once (docs/durable-futures-internals.md), so any redelivery makes equality wrong. Relaxed to a + b >= 3; the existing b >= 1 assertion still carries the test's actual intent (overflow sessions reach worker B).

tests/common/fault_injection.rs — FailingProvider could only inject permanent ack failures. Added fail_next_ack_work_item_retryable(n) plus remaining_retryable_ack_failures() so tests can distinguish "abandon" from "retry in place".

tests/worker_reliability_test.rs — new worker_retries_retryable_ack_failure_without_reexecuting_activity. Injects 3 retryable ack failures, then asserts the orchestration completes, all 3 injections were consumed, and the activity ran exactly once. worker_lock_timeout is 30s so a redelivery could not come from lock expiry.

Verified non-vacuous — reverting only the worker.rs change fails it:

FAIL worker_retries_retryable_ack_failure_without_reexecuting_activity
assertion `left == right` failed: Activity must not be re-executed when the ack failure was retryable

Why this showed up as a flake

SqliteProvider::new_in_memory uses sqlite::memory:?cache=shared. Per the SQLite docs, shared-cache mode uses table-level locks and returns SQLITE_LOCKED on a read→write upgrade collision — without invoking the busy handler ("If a required table lock cannot be obtained, the query fails and SQLITE_LOCKED is returned to the caller"). So the provider's PRAGMA busy_timeout = 60000 gives zero protection against this specific error; recovering properly needs sqlite3_unlock_notify, which sqlx doesn't wire up.

test_multi_worker_heterogeneous_config runs 2 runtimes × (2 worker + 2 orchestration) dispatchers = 8 concurrent pollers against one shared cache with a 5-connection pool, so it hit this readily: 2/8 failures in-suite and 1/5 in isolation before this change.

Note the shared-cache path is only used by new_in_memory — file-backed deployments use WAL and get SQLITE_BUSY, which is covered by busy_timeout. The SQLITE_LOCKED trigger is therefore test-infrastructure-only, but the abandon-on-retryable-error behavior this PR fixes applies to any provider.

I did not change the shared-cache configuration here. SQLite documents shared-cache as obsolete and discourages it, so that's worth revisiting, but it's a larger change to the provider's concurrency model and doesn't belong in this fix.

Testing

• ./run-tests.sh — both passes green (1109 with --all-features, 1106 without), exit 0
• cargo test --doc --all-features — 46 passed
• cargo clippy --all-targets --all-features — no new warnings

Measuring the runtime fix specifically

Simply re-running the suite after this change proves little, because relaxing the assertion to >= 3 makes that test unable to fail from a redelivery. To isolate the worker.rs fix, I kept the strict assert_eq!(a + b, 3) (exactly-once) assertion, applied only the runtime change, and ran 20 iterations with --success-output immediate:

	Failures	Ack retries fired	SQLITE_LOCKED events
Before (strict assertion)	3 / 14	n/a	—
After (strict assertion)	0 / 20	19	741

741 SQLITE_LOCKED events across the 20 runs confirms the contention was still occurring, and the new retry path fired 19 times — each one a work item that would previously have been abandoned and re-executed. Against the pre-fix rate of 3/14 (21%), 0/20 gives p ≈ 0.008.

The relaxed >= 3 assertion is still the correct one to commit, since at-least-once means the runtime cannot guarantee equality under lock expiry. The strict version was used only as a measurement probe.