[fault-injection] Profiler::recordSample() stack walk is unprotected - #687
[fault-injection] Profiler::recordSample() stack walk is unprotected#687zhengyu123 wants to merge 61 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors and extends crash-protection for native stack walking by moving fault recovery from HotspotSupport::checkFault() into Profiler::checkFault(), adding an outer sigsetjmp/siglongjmp protection region around Profiler::recordSample()’s unwind path, and updating fault-injection tests/counters accordingly.
Changes:
- Introduces
Profiler::checkFault(ProfiledThread*, siginfo_t*, void*)and wires it intoProfiler::crashHandlerInternal(), removing the old HotSpot-specificcheckFault. - Adds an outer crash-protection region in
Profiler::recordSample()to recover from faults in “unprotected” metadata reads around stack walking. - Renames the longjmp recovery counter and expands fault-injection tests to validate chaining/restoration behavior.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| ddprof-lib/src/test/cpp/hotspot_crash_protection_ut.cpp | Switches the guard-clause test to call Profiler::checkFault() instead of the removed HotSpot helper. |
| ddprof-lib/src/test/cpp/faultInjection_ut.cpp | Updates fault injection signal wrapper to use Profiler::checkFault() and adds a new regression test for recordSample’s outer setjmp/restore protocol. |
| ddprof-lib/src/main/cpp/vmEntry.cpp | Adds an assertion ensuring the JVM library lookup succeeded before publishing _libjvm. |
| ddprof-lib/src/main/cpp/threadLocalData.h | Changes the protected-jump context type stored in TLS from jmp_buf* to sigjmp_buf*. |
| ddprof-lib/src/main/cpp/profiler.h | Declares Profiler::checkFault(ProfiledThread*, siginfo_t*, void*). |
| ddprof-lib/src/main/cpp/profiler.cpp | Adds outer unwind crash protection in recordSample(), computes profiler address range at signal-handler setup, and implements Profiler::checkFault(). |
| ddprof-lib/src/main/cpp/hotspot/hotspotSupport.h | Removes the HotspotSupport::checkFault() declaration. |
| ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp | Switches walkVM crash protection to sigsetjmp/sigjmp_buf and removes the old HotSpot-specific checkFault() implementation. |
| ddprof-lib/src/main/cpp/counters.h | Renames the recovery counter from WALKVM_LONGJMP_RECOVERED to STACKWALK_LONGJMP_RECOVERED. |
Comments suppressed due to low confidence (2)
ddprof-lib/src/test/cpp/faultInjection_ut.cpp:64
- Profiler::checkFault increments Counters from the signal handler; without pre-initializing Counters in SetUp(), the first injected fault can trigger Counters' lazy aligned_alloc/memset from inside the signal path, which is not async-signal-safe.
Profiler::checkFault(ProfiledThread::current()); // longjmp if protected
ddprof-lib/src/test/cpp/faultInjection_ut.cpp:236
- This test uses setjmp/jmp_buf but the recovery path uses siglongjmp (via Profiler::checkFault). Mixing setjmp with siglongjmp is undefined; use sigsetjmp/sigjmp_buf consistently.
jmp_buf unwind_ctx;
jmp_buf* prev_jmp_buf = t->getJmpCtx();
ASSERT_EQ(&grandparent_ctx, prev_jmp_buf);
t->setFiRng(0xC0FFEEC0FFEEC0FFULL);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Scan-Build Report
Bug Summary
Reports
|
||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (4)
ddprof-lib/src/test/cpp/faultInjection_ut.cpp:214
ProfiledThread::setJmpCtxnow usessigjmp_buf*, so the test's outer context needs to besigjmp_bufas well (otherwise this won't compile).
jmp_buf grandparent_ctx;
t->setJmpCtx(&grandparent_ctx);
ddprof-lib/src/test/cpp/faultInjection_ut.cpp:238
- This test still uses
jmp_buf/setjmp, but the recovery path usessiglongjmpviaProfiler::checkFault. Usesigjmp_buf+sigsetjmp(..., 1)so the types and semantics match.
jmp_buf unwind_ctx;
jmp_buf* prev_jmp_buf = t->getJmpCtx();
ASSERT_EQ(&grandparent_ctx, prev_jmp_buf);
t->setFiRng(0xC0FFEEC0FFEEC0FFULL);
int jmp_rc = setjmp(unwind_ctx);
ddprof-lib/src/main/cpp/threadLocalData.h:66
- The comment above
_jmp_bufsays it's "Used by hotspot only", but the newProfiler::recordSample()crash-protection also installs a jump context. Updating the comment helps keep the contract accurate for future maintainers.
std::atomic<sigjmp_buf*> _jmp_buf;
ddprof-lib/src/test/cpp/faultInjection_ut.cpp:172
- This test uses
Profiler::checkFault()(via the installed signal handler), which now recovers viasiglongjmp, but the unwind context installed withProfiledThread::setJmpCtxis still ajmp_bufcreated withsetjmp. This mismatch will not compile after switchingProfiledThreadtosigjmp_buf*and should be updated tosigjmp_buf+sigsetjmp(..., 1).
for (int i = 0; i < 5000 && faults == 0; i++) {
// Raw deref of the (possibly poisoned) base — mirrors walkVM's raw reads.
uintptr_t v = *(uintptr_t*)INJECT_FAULT_ADDRESS_LIKELY(base);
// Optimization barrier: tell the compiler `v` is read/write and clobber memory to prevent
// reordering/optimizing away the load.
CI Test ResultsRun: #30701708406 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-08-01 13:44:26 UTC |
Benchmark Results (commit bbed591)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/126986264 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
ddprof-lib/src/main/cpp/profiler.cpp:688
- The
sigsetjmpresult is assigned twice (int jmp_rc = jmp_rc = ...), which is almost certainly a typo and can trigger warnings or confusion. It should be a single assignment.
sigjmp_buf unwind_ctx;
sigjmp_buf *prev_jmp_buf = walk_thread->getJmpCtx();
int jmp_rc = jmp_rc = sigsetjmp(unwind_ctx, 1);
if (jmp_rc != 0) {
ddprof-lib/src/test/cpp/faultInjection_ut.cpp:215
- This test sets a
jmp_bufas the thread’s jump context, butProfiledThread::setJmpCtxnow expects asigjmp_buf*. Usingjmp_bufhere will fail to compile (and is inconsistent withProfiler::checkFaultusingsiglongjmp).
// Simulate a pre-existing outer protection context, as recordSample must
// support when nested inside another sampler's own protected region.
jmp_buf grandparent_ctx;
t->setJmpCtx(&grandparent_ctx);
ddprof-lib/src/test/cpp/faultInjection_ut.cpp:239
- The unwind context in this test uses
jmp_buf/setjmp, but the signal handler recovery path usessiglongjmp.siglongjmpmust return to asigsetjmpsite, so this should usesigjmp_buf+sigsetjmpto avoid undefined behavior.
jmp_buf unwind_ctx;
jmp_buf* prev_jmp_buf = t->getJmpCtx();
ASSERT_EQ(&grandparent_ctx, prev_jmp_buf);
t->setFiRng(0xC0FFEEC0FFEEC0FFULL);
int jmp_rc = setjmp(unwind_ctx);
if (jmp_rc != 0) {
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c6a3d7735
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (1)
ddprof-lib/src/main/cpp/profiler.cpp:1087
- The
__FAULT_INJECTION__preprocessor block is indented, unlike other directives in this file (e.g.,profiler.cpp:148uses#ifdefat column 1). Also, the comment says the guard-region init runs “once handlers are installed”, but it currently runs beforereplaceSigsegvHandler/replaceSigbusHandler.
Consider unindenting the directives and updating the comment to reflect the actual order (or move the init below handler installation).
#ifdef __FAULT_INJECTION__
// Reserve the PROT_NONE guard region used to poison memory-access sites.
// Done here (off the signal path) once handlers are installed.
faultinj::init();
#endif
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 42432ff581
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.
Suppressed comments (3)
ddprof-lib/src/main/cpp/profiler.h:517
force_stackwalk_crash_envtriggers a null-pointer write to simulate a SIGSEGV. Dereferencingnullptris undefined behavior and can be optimized away or turned into a different kind of trap, which may make the intended crash/recovery tests flaky and may not reliably yield a SIGSEGV with a PC inside the profiler library.
if (force_stackwalk_crash_env) {
TEST_LOG("FORCE_SIGSEGV");
int* p = nullptr;
*p = 1;
ddprof-lib/src/main/cpp/profiler.cpp:1081
Libraries::findLibraryByAddress()can return NULL; in non-debug buildsassert(prof_lib != nullptr)is compiled out, soprof_lib->minAddress()/maxAddress()can become a null dereference during signal handler setup. Consider handling the NULL case explicitly and leaving the address range unset (with checkFault refusing to recover when the range is missing in production).
Libraries* libs = Libraries::instance();
CodeCache* prof_lib = libs->findLibraryByAddress((const void*)&Profiler::setupSignalHandlers);
assert(prof_lib != nullptr);
profiler_min_address = reinterpret_cast<uintptr_t>(prof_lib->minAddress());
profiler_max_address = reinterpret_cast<uintptr_t>(prof_lib->maxAddress());
ddprof-lib/src/main/cpp/profiler.cpp:2039
- In non-UNIT_TEST builds, if
profiler_min_address/profiler_max_addressare unexpectedly left as 0, the current logic falls back to recovering unconditionally (the safety-tightening range gate is effectively disabled in release builds whereassert()is compiled out). It’s safer to refuse recovery when the range is missing in production builds.
// If the profiler address range is not initialized (e.g. unit tests), fall back
// to recovering unconditionally when a protection context is installed.
#if !defined(UNIT_TEST)
assert(min != 0 && max != 0);
#endif
Benchmark Results (commit 42432ff)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128337560 Commit: ✅ Within expected boundariesNo significant runtime deltas (all within run-to-run noise) and no internal-counter outliers. Runtime details (per benchmark × JDK)
Internal counter details (ddprof)ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
|
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.
Suppressed comments (5)
ddprof-lib/src/main/cpp/profiler.h:517
- The debug-only
force_stackwalk_crash_envpath triggers a null-pointer write via a non-volatile pointer. Because this is undefined behavior, compilers are allowed to optimize the store away, which could make the forced-crash path unreliable (especially under LTO/optimizations).
if (force_stackwalk_crash_env) {
TEST_LOG("FORCE_SIGSEGV");
int* p = nullptr;
*p = 1;
ddprof-lib/src/main/cpp/profiler.cpp:1081
setupSignalHandlers()dereferencesprof_libafter anassert(prof_lib != nullptr). In release builds assertions may be compiled out, so a failure to resolve the profiler library would become a null dereference during signal-handler setup.
Libraries* libs = Libraries::instance();
CodeCache* prof_lib = libs->findLibraryByAddress((const void*)&Profiler::setupSignalHandlers);
assert(prof_lib != nullptr);
profiler_min_address = reinterpret_cast<uintptr_t>(prof_lib->minAddress());
profiler_max_address = reinterpret_cast<uintptr_t>(prof_lib->maxAddress());
ddprof-lib/src/main/cpp/threadLocalData.h:63
- The comment above
_jmp_bufstill says it is "Used by hotspot only", but this PR extends crash-recovery to StackWalker::walkFP()/walkDwarf() for all JVMs. This comment is now misleading and should be updated to match current behavior.
// siglongjmp buffer. Used by hotspot only at this moment.
// Published in HotspotSupport::walkVM()/walkJavaStack() and StackWalker::walkFP()/walkDwarf() (all VMs),
// consumed in Profiler::checkFault() from an asynchronous SEGV-handler context on the same thread
ddprof-lib/src/main/cpp/profiler.cpp:2039
checkFault()relies on anassert(min != 0 && max != 0)to ensure the profiler address range is initialized, but in non-UNIT_TEST release builds this assert is compiled out. If initialization is ever missed, the current logic silently falls back to unconditional recovery, undermining the intended "only recover for profiler PC range" tightening.
#if !defined(UNIT_TEST)
assert(min != 0 && max != 0);
#endif
if ((min != 0 && max != 0) && (pc < min || pc >= max)) {
return;
ddprof-lib/src/test/cpp/stackWalker_ut.cpp:224
- This test uses a hard-coded 4096 byte mapping and unmapping size. On Linux builds with a non-4K page size (e.g., 64K),
munmap(..., 4096)can fail with EINVAL. Prefer using the project-wideOS::page_size(already available) and assertmunmapsucceeds.
_bad_page = mmap(nullptr, 4096, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
ASSERT_NE(MAP_FAILED, _bad_page);
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47b385dc13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.
Suppressed comments (3)
ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp:1277
- Same as above:
async_trace_activeis never set around thisAsyncSampleMutexregion, so a siglongjmp recovery can bypass the destructor and leave_unwinding_Javastuck true, effectively disabling async sampling for this thread.
AsyncSampleMutex mutex(ProfiledThread::current());
if (mutex.acquired()) {
java_frames = getJavaTraceAsync(ucontext, frames, max_depth, java_ctx, truncated);
if (java_frames > 0 && java_ctx->pc != NULL && VMStructs::hasMethodStructs()) {
VMNMethod* nmethod = CodeHeap::findNMethod(java_ctx->pc);
ddprof-lib/src/main/cpp/profiler.cpp:1081
setupSignalHandlers()derives the profiler library address range viaLibraries::findLibraryByAddress(), butLibraries::_native_libsis only populated after a symbol scan (e.g.Libraries::updateSymbols/refresh). If this runs before any scan,prof_libcan be null, causing an assert in debug builds and a null-deref in release builds.
// Get address range of java profiler library
Libraries* libs = Libraries::instance();
CodeCache* prof_lib = libs->findLibraryByAddress((const void*)&Profiler::setupSignalHandlers);
assert(prof_lib != nullptr);
profiler_min_address = reinterpret_cast<uintptr_t>(prof_lib->minAddress());
profiler_max_address = reinterpret_cast<uintptr_t>(prof_lib->maxAddress());
ddprof-lib/src/main/cpp/hotspot/hotspotSupport.cpp:1242
async_trace_activeis intended to let the siglongjmp recovery path releaseAsyncSampleMutex’s per-thread_unwinding_Javaguard, but in the hook-prefixed async path the flag is never set when the mutex is acquired. If a fault is recovered insidegetJavaTraceAsync(), the mutex destructor is bypassed and_unwinding_Javacan remain stuck true.
This issue also appears on line 1273 of the same file.
if (prof_thread != nullptr) {
prof_thread->setJmpCtx(&crash_protection_ctx);
}
Benchmark Results (commit be7e84d)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128359370 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10343 ms (21 iters) | ✅ 10415 ms (21 iters) | ≈ +0.7% (±11.4%) | — / — |
| akka-uct | 25 | ✅ 8762 ms (24 iters) | ✅ 8988 ms (24 iters) | ≈ +2.6% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 6022 ms (33 iters) | ✅ 5998 ms (33 iters) | ≈ -0.4% (±24.9%) | |
| finagle-chirper | 25 | ✅ 5440 ms (36 iters) | ✅ 5411 ms (36 iters) | ≈ -0.5% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2750 ms (68 iters) | ✅ 2652 ms (70 iters) | 🟢 -3.6% | — / — |
| future-genetic | 25 | ✅ 2017 ms (92 iters) | ✅ 2068 ms (89 iters) | ≈ +2.5% (±2.7%) | — / — |
| naive-bayes | 21 | ✅ 1280 ms (134 iters) | ✅ 1265 ms (135 iters) | ≈ -1.2% (±32.6%) | — / — |
| naive-bayes | 25 | ✅ 1017 ms (168 iters) | ✅ 1012 ms (169 iters) | ≈ -0.5% (±31.8%) | — / — |
| reactors | 21 | ✅ 15457 ms (15 iters) | ✅ 16576 ms (15 iters) | 🔴 +7.2% | — / — |
| reactors | 25 | ✅ 18689 ms (15 iters) | ✅ 18545 ms (15 iters) | ≈ -0.8% (±4.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 1996 / 1934 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 4 | 2291 / 2359 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 4 | 8383 / 8239 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 8601 / 8133 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1246 / 1257 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 1270 / 1277 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2986 / 2898 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 2931 / 2862 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 4 | 3519 / 3491 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 3463 / 3445 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1532 / 1692 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1870 / 1821 | ✅ / ✅ | ✅ / ✅ |
Benchmark Results (commit 47b385d)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128364274 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10357 ms (21 iters) | ✅ 10282 ms (21 iters) | ≈ -0.7% (±11.2%) | — / — |
| akka-uct | 25 | ✅ 8922 ms (24 iters) | ✅ 8910 ms (24 iters) | ≈ -0.1% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 6002 ms (33 iters) | ✅ 6106 ms (33 iters) | ≈ +1.7% (±25.7%) | |
| finagle-chirper | 25 | ✅ 5460 ms (36 iters) | ✅ 5477 ms (36 iters) | ≈ +0.3% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2676 ms (69 iters) | ✅ 2672 ms (69 iters) | ≈ -0.1% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2811 ms (66 iters) | ✅ 2824 ms (66 iters) | ≈ +0.5% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2060 ms (90 iters) | ✅ 2070 ms (90 iters) | ≈ +0.5% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2117 ms (88 iters) | ✅ 2038 ms (91 iters) | 🟢 -3.7% | — / — |
| naive-bayes | 21 | ✅ 1267 ms (135 iters) | ✅ 1290 ms (133 iters) | ≈ +1.8% (±33.2%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1030 ms (166 iters) | ≈ +1.8% (±32%) | — / — |
| reactors | 21 | ✅ 16799 ms (15 iters) | ✅ 16575 ms (15 iters) | ≈ -1.3% (±9.5%) | — / — |
| reactors | 25 | ✅ 18800 ms (15 iters) | ✅ 18156 ms (15 iters) | ≈ -3.4% (±3.8%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1931 / 1922 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2197 / 2226 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 8759 / 9117 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 8644 / 8380 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 2 | 1263 / 1244 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / ✅ | 1265 / 1267 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2985 / 3007 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2842 / 2834 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 4 | 3519 / 3521 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 4 / 5 | 3472 / 3522 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 1565 / 1835 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1950 / 1830 | ✅ / ✅ | ✅ / ✅ |
…nto zgu/recordSample
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Suppressed comments (5)
ddprof-lib/src/main/cpp/profiler.cpp:2038
Profiler::checkFaultreadsprofiler_min_address/profiler_max_addresswithmemory_order_relaxed, but those atomics are written during initialization and then read from arbitrary threads in an async SIGSEGV/SIGBUS handler. On weakly-ordered architectures (arm64), relaxed loads can observe stale 0/old values, re-enabling the "uninitialized" fallback (unconditional recovery) or otherwise weakening the intended PC-range gate.
const uintptr_t pc = (uintptr_t)StackFrame(ucontext).pc();
const uintptr_t min = profiler_min_address.load(std::memory_order_relaxed);
const uintptr_t max = profiler_max_address.load(std::memory_order_relaxed);
// If the profiler address range is not initialized (e.g. unit tests), fall back
// to recovering unconditionally when a protection context is installed.
#if !defined(UNIT_TEST)
assert(min != 0 && max != 0);
#endif
if ((min != 0 && max != 0) && (pc < min || pc >= max)) {
return;
ddprof-lib/src/main/cpp/profiler.cpp:1067
- The newly added block inside
setupSignalHandlers()is indented inconsistently (extra spaces) compared to the rest of the file, which makes the control-flow harder to read and risks style/tooling churn (e.g.,spotlessApply).
// Initialize infrastructure before enabling signal handler
// Eagerly initialize the Counters singleton off the signal path, before any
// handler that increments counters is installed. The crash handler
// (crashHandlerInternal -> SafeAccess::handle_safefetch) bumps
ddprof-lib/src/main/cpp/profiler.cpp:1087
- The
__FAULT_INJECTION__preprocessor block is indented (and its comment says "once handlers are installed"), but at this point insetupSignalHandlers()the SIGSEGV/SIGBUS handlers have not been installed yet. Keeping#ifdef/#endifat column 1 (as elsewhere) and correcting the comment would avoid confusion.
#ifdef __FAULT_INJECTION__
// Reserve the PROT_NONE guard region used to poison memory-access sites.
// Done here (off the signal path) once handlers are installed.
faultinj::init();
#endif
ddprof-lib/src/main/cpp/profiler.cpp:1089
setupSignalHandlers()still installs the SIGSEGV/SIGBUS handlers only for HotSpot and OpenJ9. The PR description states that non-HotSpot JVMs (including Zing) now get crash recovery; if Zing is intended to be covered, it likely also needs to be included here (or the description adjusted if Zing cannot tolerate interposed handlers).
if (VM::isHotspot() || VM::isOpenJ9()) {
ddprof-lib/src/test/cpp/stackWalker_ut.cpp:211
- In
StackWalkerCrashRecoveryTest::SetUp, signal handlers are replaced before the mmap() + ASSERT. If the ASSERT fails, gtest will skipTearDown(), leaving the process with modified signal handlers for subsequent tests. Reordering to perform the mmap/assert first reduces the risk of leaking global handler state on SetUp failure.
_orig_segv = OS::replaceSigsegvHandler(Profiler::segvHandler);
_orig_bus = OS::replaceSigbusHandler(Profiler::busHandler);
_bad_page = mmap(nullptr, 4096, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
ASSERT_NE(MAP_FAILED, _bad_page);
Benchmark Results (commit 5b97812)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128369356 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10285 ms (21 iters) | ✅ 10263 ms (21 iters) | ≈ -0.2% (±11%) | — / — |
| akka-uct | 25 | ✅ 8884 ms (24 iters) | ✅ 8901 ms (24 iters) | ≈ +0.2% (±10.2%) | — / — |
| finagle-chirper | 21 | ✅ 5973 ms (33 iters) | ✅ 6060 ms (33 iters) | ≈ +1.5% (±25.5%) | |
| finagle-chirper | 25 | ✅ 5392 ms (36 iters) | ✅ 5538 ms (36 iters) | ≈ +2.7% (±24.3%) | |
| fj-kmeans | 21 | ✅ 2707 ms (69 iters) | ✅ 2637 ms (72 iters) | ≈ -2.6% (±2.6%) | — / — |
| fj-kmeans | 25 | ✅ 2818 ms (66 iters) | ✅ 2829 ms (66 iters) | ≈ +0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2111 ms (88 iters) | ✅ 2171 ms (86 iters) | 🔴 +2.8% | — / — |
| future-genetic | 25 | ✅ 2061 ms (91 iters) | ✅ 2122 ms (87 iters) | 🔴 +3% | — / — |
| naive-bayes | 21 | ✅ 1306 ms (132 iters) | ✅ 1256 ms (136 iters) | ≈ -3.8% (±31.9%) | — / — |
| naive-bayes | 25 | ✅ 1023 ms (167 iters) | ✅ 1010 ms (170 iters) | ≈ -1.3% (±31.7%) | — / — |
| reactors | 21 | ✅ 16596 ms (15 iters) | ✅ 16226 ms (15 iters) | ≈ -2.2% (±7.6%) | — / — |
| reactors | 25 | ✅ 18608 ms (15 iters) | ✅ 18740 ms (15 iters) | ≈ +0.7% (±3.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1946 / 1933 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2292 / 2287 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 3 | 8678 / 8754 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8610 / 8478 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 4 | 1234 / 1282 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1282 / 1274 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2947 / 3018 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2920 / 2943 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 6 / 4 | 3539 / 3506 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 3486 / 3470 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 1679 / 1465 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1907 / 1895 | ✅ / ✅ | ✅ / ✅ |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.
Suppressed comments (2)
ddprof-lib/src/main/cpp/profiler.cpp:2039
- In non-UNIT_TEST builds, the "range not initialized" path currently falls back to unconditional recovery (because the
assert(min != 0 && max != 0)is compiled out underNDEBUG). That undermines the new safety invariant of only recovering faults whose PC is inside the profiler .so. Consider refusing to recover when the range is unset in production builds, and keep unconditional recovery only for UNIT_TEST builds.
#if !defined(UNIT_TEST)
assert(min != 0 && max != 0);
#endif
if ((min != 0 && max != 0) && (pc < min || pc >= max)) {
return;
ddprof-lib/src/main/cpp/profiler.cpp:1087
- The comment above
faultinj::init()says it runs "once handlers are installed", but it currently executes beforeprotectSignalHandlers()/replaceSig*Handler()installs the profiler handlers. Updating the comment would avoid misleading future readers about the ordering constraints here.
#ifdef __FAULT_INJECTION__
// Reserve the PROT_NONE guard region used to poison memory-access sites.
// Done here (off the signal path) once handlers are installed.
faultinj::init();
Benchmark Results (commit 2492825)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128377456 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10219 ms (21 iters) | ✅ 10326 ms (21 iters) | ≈ +1% (±11%) | — / — |
| akka-uct | 25 | ✅ 8902 ms (24 iters) | ✅ 8908 ms (24 iters) | ≈ +0.1% (±10.7%) | — / — |
| finagle-chirper | 21 | ✅ 5984 ms (33 iters) | ✅ 5961 ms (33 iters) | ≈ -0.4% (±25%) | |
| finagle-chirper | 25 | ✅ 5442 ms (36 iters) | ✅ 5433 ms (36 iters) | ≈ -0.2% (±24.5%) | |
| fj-kmeans | 21 | ✅ 2781 ms (67 iters) | ✅ 2656 ms (70 iters) | 🟢 -4.5% | — / — |
| fj-kmeans | 25 | ✅ 2843 ms (66 iters) | ✅ 2854 ms (66 iters) | ≈ +0.4% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2081 ms (89 iters) | ✅ 2071 ms (90 iters) | ≈ -0.5% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 2061 ms (90 iters) | ✅ 2022 ms (91 iters) | ≈ -1.9% (±2.6%) | — / — |
| naive-bayes | 21 | ✅ 1257 ms (136 iters) | ✅ 1217 ms (141 iters) | ≈ -3.2% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1023 ms (167 iters) | ✅ 1010 ms (169 iters) | ≈ -1.3% (±31.7%) | — / — |
| reactors | 21 | ✅ 15287 ms (15 iters) | ✅ 15475 ms (16 iters) | ≈ +1.2% (±7.4%) | — / — |
| reactors | 25 | ✅ 18639 ms (15 iters) | ✅ 18137 ms (15 iters) | ≈ -2.7% (±4.6%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 4 | 2076 / 1973 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2300 / 2212 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8538 / 8460 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 8049 / 8595 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 3 | 1212 / 1287 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 3 | 1303 / 1303 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 3011 / 2991 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2884 / 2910 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 3497 / 3531 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3501 / 3513 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1487 / 1636 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1837 / 1876 | ✅ / ✅ | ✅ / ✅ |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated no new comments.
Suppressed comments (4)
ddprof-lib/src/main/cpp/profiler.cpp:2039
- These preprocessor directives are indented, but the rest of this file keeps preprocessor directives at column 1. Unindenting them keeps formatting consistent and avoids accidental whitespace-only diffs.
#if !defined(UNIT_TEST)
assert(min != 0 && max != 0);
#endif
ddprof-lib/src/main/cpp/profiler.cpp:1083
- Typo in the comment: "stores pass" should be "stores past".
// Prevents the compiler from moving profiler_min_address/profiler_max_address stores pass
// signal handler setup.
ddprof-lib/src/main/cpp/profiler.cpp:1090
- Preprocessor directives are indented here, but elsewhere in this file directives are consistently placed at column 1 (e.g.,
#ifdef UNIT_TESTaround line 1048). Keeping them unindented improves readability and avoids accidental whitespace-diff churn.
This issue also appears on line 2037 of the same file.
#ifdef __FAULT_INJECTION__
// Reserve the PROT_NONE guard region used to poison memory-access sites.
// Done here (off the signal path) once handlers are installed.
faultinj::init();
#endif
ddprof-lib/src/main/cpp/profiler.h:517
- This intentionally crashes via a null-pointer write, but
int* p = nullptr; *p = 1;is undefined behavior and can be optimized into something that doesn’t reliably raise SIGSEGV (or changes the signal type) under some optimization/sanitizer combinations. Using a volatile store to address 0 makes the intended SIGSEGV much more deterministic.
int* p = nullptr;
*p = 1;
Benchmark Results (commit 5c39ccb)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128412890 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10268 ms (21 iters) | ✅ 10333 ms (21 iters) | ≈ +0.6% (±11.5%) | — / — |
| akka-uct | 25 | ✅ 8855 ms (24 iters) | ✅ 8872 ms (24 iters) | ≈ +0.2% (±10%) | — / — |
| finagle-chirper | 21 | ✅ 5960 ms (33 iters) | ✅ 5955 ms (33 iters) | ≈ -0.1% (±25.1%) | |
| finagle-chirper | 25 | ✅ 5480 ms (36 iters) | ✅ 5404 ms (36 iters) | ≈ -1.4% (±23.8%) | |
| fj-kmeans | 21 | ✅ 2749 ms (68 iters) | ✅ 2629 ms (72 iters) | 🟢 -4.4% | — / — |
| fj-kmeans | 25 | ✅ 2761 ms (68 iters) | ✅ 2840 ms (66 iters) | 🔴 +2.9% | — / — |
| future-genetic | 21 | ✅ 2151 ms (86 iters) | ✅ 2187 ms (85 iters) | ≈ +1.7% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1993 ms (94 iters) | ✅ 2055 ms (90 iters) | 🔴 +3.1% | — / — |
| naive-bayes | 21 | ✅ 1224 ms (140 iters) | ✅ 1231 ms (139 iters) | ≈ +0.6% (±32.2%) | — / — |
| naive-bayes | 25 | ✅ 1022 ms (167 iters) | ✅ 970 ms (176 iters) | ≈ -5.1% (±31.1%) | — / — |
| reactors | 21 | ✅ 15954 ms (15 iters) | ✅ 15729 ms (16 iters) | ≈ -1.4% (±7.4%) | — / — |
| reactors | 25 | ✅ 18364 ms (15 iters) | ✅ 18624 ms (15 iters) | ≈ +1.4% (±5.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2021 / 1984 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 4 / ✅ | 2276 / 2308 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 5 | 8499 / 8786 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 8360 / 8313 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 1284 / 1303 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 6 | 1286 / 1288 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 2997 / 2923 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2855 / 2932 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 6 | 3514 / 3496 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 3451 / 3445 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1579 / 1508 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 1829 / 1927 | ✅ / ✅ | ✅ / ✅ |
What does this PR do?:
This PR generalizes the sigsetjmp/siglongjmp crash-recovery mechanism that previously protected only
HotspotSupport::walkVM():Moves
checkFault()fromHotspotSupporttoProfiler, and — critically — moves the call site incrashHandlerInternal()outside theif (VM::isHotspot())block. Previously, non-HotSpot JVMs (J9, Zing) got no crash recovery at all; this fixes that.Adds the same
sigsetjmp/siglongjmpprotection toHotspotSupport::walkJavaStack(),StackWalker::walkFP(), andStackWalker::walkDwarf().walkFP/walkDwarfback all native-frame sampling (Profiler:: native-trace collection, called regardless of JVM type), so this closes a real, previously-unprotected crash surface for every deployment, not just HotSpot.Tightens recovery semantics:
checkFault()now only recovers if the faulting PC falls inside the profiler's own .so address range (profiler_min_address/profiler_max_address, computed once insetupSignalHandlers()), rather than unconditionally recovering whenever ajmp ctxhappens to be installed. This is a meaningful safety tightening — it stops the recovery path from swallowing unrelated faults that merely race with an installedjmp ctx.Introduces
NO_INJECTION_ASSERTso debug asserts don't fire on invariants that fault injection deliberately violates, and fixesforce_stackwalk_crash_envto fault via a real null-pointer write instead ofraise(SIGSEGV)— necessary because raise()'s PC lands in libc, not profiler code, which would now fail the new PC-range check.Adds substantial new unit-test coverage (
WalkJavaStackAsyncMutexRecoveryTest,StackWalkerCrashRecoveryTest) exercising both the recovery and rejection sides of the new range check via UNIT_TEST-only test hooks (setAddressRangeForTest/resetAddressRangeForTest).Motivation:
Fix
Profiler::recordSample()crash discovered by fault-injection.Additional Notes:
How to test the change?:
-PenableFaultInjection), some of tests crashed without this fix, but no crash with this fix.For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!