Skip to content

Fix async context loss when field injection is unavailable - #12030

Draft
CharlyRien wants to merge 2 commits into
DataDog:masterfrom
CharlyRien:fix-weak-context-store-drop-10479
Draft

Fix async context loss when field injection is unavailable#12030
CharlyRien wants to merge 2 commits into
DataDog:masterfrom
CharlyRien:fix-weak-context-store-drop-10479

Conversation

@CharlyRien

@CharlyRien CharlyRien commented Jul 22, 2026

Copy link
Copy Markdown

What does this PR do?

Removes the silent-drop size cap from WeakMapContextStore, the fall-back used when context-store field injection is unavailable, and rebuilds it on ConcurrentHashMap + identity-based weak keys with ReferenceQueue cleanup.

Motivation

Fixes #10479. Under JDK 25 AOT class linking (-XX:AOTCache, on by default) carrier classes come pre-linked from the cache, field injection fails, and every ContextStore access falls back to this weak map. Its put() silently dropped entries past a 50k cap measured with WeakConcurrentMap.approximateSize(), which also counts collected-but-unexpunged entries (expunge runs ~1/s) — so past ~50k context writes/sec live async context was discarded and spans were orphaned. A reproducer loses 98.5% of async context with class linking on, 0% without; with this fix, 0% everywhere (verified up to ~1.4M writes/sec).

There is deliberately no cap anymore: growth is bounded by live carriers, exactly like the injected-field path. Dead entries are drained on writes and by a 1s background task (same AgentTaskScheduler mechanism the old WeakMaps-backed map used), so idle/read-only stores don't retain collected contexts. Reads are allocation-free (reused thread-local lookup key), and context factories run outside the map's own locks so they may re-enter the store.

JMH benchmark included (old vs new, 1/10/100 threads): ~30% faster single-threaded, parity under contention, 0 B/op on the hot paths.

This aligns with the direction of the mcculls/global-weak-context-store branch (uncapped weak storage + ReferenceQueue cleanup), scoped to the fall-back store only.

Memory characteristics

Measured retained heap (after full GC) of old vs new store:

Scenario old (capped) new
200k live carriers 4.1 MB / 50,000 entries kept 15.3 MB / 200,000 entries kept
2M dead-carrier churn burst 4.0 MB (97.5% of writes dropped) ~110 MB retained, 0 entries
  • Per-entry cost is the same (~80 B/entry). The old store only looks smaller because it dropped everything past the cap — which is the bug.
  • The churn number is a worst-case synthetic burst with no GC pressure: nothing gets enqueued mid-burst, the map peaks at ~2M dead entries, and the backing ConcurrentHashMap table grows to match and never shrinks. This is a one-time high-water mark, not a leak — repeated bursts on the same store do not accumulate (subsequent rounds retained 8–9 MB when GC ran mid-burst, which is the realistic case since real churn comes with allocation pressure). The uncapped WeakConcurrentMap direction has the same property; it is inherent to keeping the data instead of dropping it.

Additional Notes

Regression tests drive real GC (dead-carrier churn past the former cap) and lock in the factory-reentrancy contract.

🤖 Generated with Claude Code

When context-store fields cannot be injected (e.g. JDK 25 AOT class
linking pre-links carrier classes before the agent can transform them),
every ContextStore access falls back to WeakMapContextStore. Its put()
silently dropped new entries past a 50k cap measured with
WeakConcurrentMap.approximateSize(), which also counts collected but
not-yet-expunged entries (expunged ~once per second) - so past ~50k
context writes/sec live async context was discarded and spans were
orphaned (DataDog#10479).

Rebuild the fall-back store on ConcurrentHashMap with identity-based
weak keys and no cap - growth is bounded by live carriers, exactly like
the injected-field path:

- collected keys are drained from a ReferenceQueue on every write and
  by a periodic background task, so dead entries don't accumulate on
  idle or read-only stores
- reads are allocation-free (reused thread-local lookup key)
- context factories run outside the map's own locks, so they may
  re-enter the store (CHM.computeIfAbsent forbids this)

JMH benchmark (old vs new, 1/10/100 threads) shows ~30% faster
single-threaded and parity under contention.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@CharlyRien
CharlyRien force-pushed the fix-weak-context-store-drop-10479 branch from 785f022 to 78f023d Compare July 22, 2026 12:16
@mcculls

mcculls commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

FYI, we plan to incorporate the new approach under a feature flag - as that will allow switching between the approaches, rather than completely remove the limit (which would work for AOT but could also have an unforeseen impact on non-AOT users)

For example: a misbehaving instrumentation could leak store entries - and with no limit or forcible reclamation that would eat up the heap and impact the application.

@CharlyRien

Copy link
Copy Markdown
Author

FYI, we plan to incorporate the new approach under a feature flag - as that will allow switching between the approaches, rather than completely remove the limit (which would work for AOT but could also have an unforeseen impact on non-AOT users)

Yes it would be great. Thanks for letting me know.
Do you have an ETA already? No rush, it is just for information.

@CharlyRien

Copy link
Copy Markdown
Author

FYI, we plan to incorporate the new approach under a feature flag - as that will allow switching between the approaches, rather than completely remove the limit (which would work for AOT but could also have an unforeseen impact on non-AOT users)

For example: a misbehaving instrumentation could leak store entries - and with no limit or forcible reclamation that would eat up the heap and impact the application.

For the misbehaving instrumentation, I had exactly this one in mind also. 😓
Do you want me to add the feature flag in this PR? dd.trace.weak-context-store.unbounded for example?

@mcculls

mcculls commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Do you want me to add the feature flag in this PR? dd.trace.weak-context-store.unbounded for example?

I'd prefer to leave the current WeakMapContextStore untouched for the moment - rather than mix changes to the old approach and the incoming work (which should land in the next week or so)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Traces do not appear in Datadog Web UI when AOT Cache is enabled,

3 participants