[codex] Fix large fan-in replay joins

[mwotton]

Summary

• Add a 1024-way completed sub-orchestration fan-in replay regression test that verifies each joined result.
• Store schedule-to-token bindings so replay completion delivery is O(1) instead of scanning every token binding.
• Replace the replay no-op waker with a private wake state, and register durable-future token wakers so replayed completions can drive wake-based progress.

Background

This was distilled from a spin loop bug in the wild. Parent bulk orchestrations had very large fan-in, all child SubOrchestrationCompleted events were already persisted, and no queue work remained, but replay still failed to produce a terminal parent event.

Root Cause

There were two related issues.

First, replay polled orchestration futures with a no-op waker. For large inputs, futures::future::join_all uses wake-driven readiness internally, so simply polling the parent future again is not enough: completed durable children need to wake the task. Without those wakes, a fully replayed large fan-in can return Continue instead of reaching Completed.

Second, replay completion delivery only had token-to-schedule bindings, so each completed schedule searched all bound tokens. With large fan-in, that made history replay O(n^2) just to deliver completions.

Validation

• ./run-tests.sh
• cargo fmt --check
• git diff --check upstream/main...HEAD

[mwotton]

@microsoft-github-policy-service agree

[mwotton]

the real meat here is just the first commit which is the test that shows the bug - second commit is my attempt at fixing it, but it's entirely possible it's non-idiomatic or has problems I haven't seen.

[qbradley]

It seems that the purpose of must_poll is to avoid unnecessary extra polling, based on the assumption that only application of history events can wake up pending futures.

If that assumption is incorrect, then one naive fix would be to remove must_poll field entirely, and just always poll after applying every history event, and after applying all history. Redundant polling is always safe, as long as a completed future is not polled.

The risk is that CPU usage of repeatedly polling non-trivial futures could be wasteful.

[qbradley]

After digging into the implementation more, I observe that the root issue is OrchestrationContext::join is using futures::future::join_all which uses a non-deterministic scheduling strategy for >30 futures.

I provide an alternative fix here, which is to only use deterministic scheduling for OrchestrationContext combinators:

#19

[affandar]

@mwotton thank you for catching, reporting and proposing a fix!

@qbradley had a simpler approach to solving the root cause which I took, but your test case was merged as part of it.

If you want you can send the replay optimization as a separate PR and I'll be happy to review that.

[mwotton]

fantastic! thanks for the merge - i'll see how it works on my fork without the optimisation and see if performance is still a problem. (I think I only saw it because I'm using a turso provider ported from the sqlite provider, but it's a macro-driven monstrosity right now, need a bit more thought before I PR that.)