Carve slow/e2e tests into separate testSlow tasks - #700
Conversation
…re-release Tag long-running tests with @tag("slow"), exclude them from the default test tasks, and add testSlow${Config} Gradle tasks (wired to gtest deps) run only by the nightly and release-validated CI workflows. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI Test ResultsRun: #30641088505 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-07-31 15:33:21 UTC |
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Benchmark Results (commit 5502908)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128197042 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10277 ms (21 iters) | ✅ 10216 ms (21 iters) | ≈ -0.6% (±11.6%) | — / — |
| akka-uct | 25 | ✅ 8828 ms (24 iters) | ✅ 8927 ms (24 iters) | ≈ +1.1% (±9.8%) | — / — |
| finagle-chirper | 21 | ✅ 5946 ms (33 iters) | ✅ 6015 ms (33 iters) | ≈ +1.2% (±25.2%) | |
| finagle-chirper | 25 | ✅ 5437 ms (36 iters) | ✅ 5470 ms (36 iters) | ≈ +0.6% (±23.9%) | |
| fj-kmeans | 21 | ✅ 2782 ms (67 iters) | ✅ 2657 ms (70 iters) | 🟢 -4.5% | — / — |
| fj-kmeans | 25 | ✅ 2778 ms (67 iters) | ✅ 2837 ms (66 iters) | ≈ +2.1% (±2.7%) | — / — |
| future-genetic | 21 | ✅ 2080 ms (89 iters) | ✅ 2075 ms (89 iters) | ≈ -0.2% (±2.6%) | — / — |
| future-genetic | 25 | ✅ 1971 ms (93 iters) | ✅ 2056 ms (90 iters) | 🔴 +4.3% | — / — |
| naive-bayes | 21 | ✅ 1241 ms (137 iters) | ✅ 1311 ms (131 iters) | ≈ +5.6% (±33.5%) | — / — |
| naive-bayes | 25 | ✅ 1012 ms (169 iters) | ✅ 1001 ms (171 iters) | ≈ -1.1% (±31.5%) | — / — |
| reactors | 21 | ✅ 17177 ms (15 iters) | ✅ 17395 ms (15 iters) | ≈ +1.3% (±7.5%) | — / — |
| reactors | 25 | ✅ 18447 ms (15 iters) | ✅ 18630 ms (15 iters) | ≈ +1% (±4.3%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1995 / 2020 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / 2 | 2171 / 2126 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 8768 / 8817 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 8597 / 8311 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 3 | 1268 / 1300 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 3 / 2 | 1250 / 1293 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 2 / 4 | 2937 / 2949 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 2872 / 2881 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 4 | 3506 / 3486 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 3502 / 3469 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 1769 / 1817 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1944 / 1918 | ✅ / ✅ | ✅ / ✅ |
![chatgpt-codex-connector[bot]](https://cdn.github.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 734072b35d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| uses: ./.github/workflows/test_workflow.yml | ||
| with: | ||
| configuration: '["debug"]' | ||
| slow_tests: true |
There was a problem hiding this comment.
For every normal release validation, this call and pre-release-tests both instantiate the full debug matrix, but test_workflow.yml names uploaded artifacts only by platform, JDK, and configuration (for example, line 170), without distinguishing regular from slow tests. Since actions/upload-artifact artifacts are immutable and names must be unique within the workflow run, the second debug job to upload each name fails, causing one of these required reusable-workflow jobs to fail and blocking create-release; include the suite in every artifact name.
Useful? React with 👍 / 👎.
| if: ${{ inputs.dry_run != true && inputs.skip_tests != true && inputs.release_type != 'retag' }} | ||
| uses: ./.github/workflows/test_workflow.yml | ||
| with: | ||
| configuration: '["debug"]' |
There was a problem hiding this comment.
Configuring the slow suite only for debug means the newly tagged tests are excluded from both nightly and pre-release ASan tasks, while no testSlowAsan invocation replaces that coverage. This is particularly harmful for WriteStackTracesAfterClassUnloadTest, whose own documentation says non-sanitized runs can stay green on the use-after-free and that ASan/UBSan is the authoritative signal; include asan in a slow-suite workflow so this regression remains detectable.
AGENTS.md reference: AGENTS.md:L324-L328
Useful? React with 👍 / 👎.
| import org.junit.platform.engine.support.descriptor.MethodSource; | ||
| import org.junit.platform.launcher.Launcher; | ||
| import org.junit.platform.launcher.LauncherDiscoveryRequest; | ||
| import org.junit.platform.launcher.TagFilter; |
There was a problem hiding this comment.
This commit modifies ProfilerTestRunner.java, but the file still begins directly with its package declaration and has no Datadog copyright header. Add the current-year header required for any modified file that lacks one.
AGENTS.md reference: AGENTS.md:L372-L374
Useful? React with 👍 / 👎.
Benchmark Results (commit 734072b)Pipeline: https://gitlab.ddbuild.io/DataDog/apm-reliability/benchmarking-platform/-/pipelines/128272132 Commit:
|
| Benchmark | JDK | Latest | Dev | Δ (dev vs latest) | Issues L/D |
|---|---|---|---|---|---|
| akka-uct | 21 | ✅ 10402 ms (21 iters) | ✅ 10250 ms (21 iters) | ≈ -1.5% (±10.6%) | — / — |
| akka-uct | 25 | ✅ 8930 ms (24 iters) | ✅ 8826 ms (24 iters) | ≈ -1.2% (±10.4%) | — / — |
| finagle-chirper | 21 | ✅ 5953 ms (33 iters) | ✅ 5935 ms (33 iters) | ≈ -0.3% (±24.6%) | |
| finagle-chirper | 25 | ✅ 5494 ms (36 iters) | ✅ 5473 ms (36 iters) | ≈ -0.4% (±24.8%) | |
| fj-kmeans | 21 | ✅ 2727 ms (68 iters) | ✅ 2776 ms (67 iters) | ≈ +1.8% (±2.7%) | — / — |
| fj-kmeans | 25 | ✅ 2837 ms (66 iters) | ✅ 2828 ms (66 iters) | ≈ -0.3% (±2.6%) | — / — |
| future-genetic | 21 | ✅ 2133 ms (87 iters) | ✅ 2065 ms (90 iters) | 🟢 -3.2% | — / — |
| future-genetic | 25 | ✅ 2009 ms (92 iters) | ✅ 2077 ms (89 iters) | 🔴 +3.4% | — / — |
| naive-bayes | 21 | ✅ 1314 ms (130 iters) | ✅ 1234 ms (138 iters) | ≈ -6.1% (±32%) | — / — |
| naive-bayes | 25 | ✅ 1022 ms (167 iters) | ✅ 1012 ms (168 iters) | ≈ -1% (±31.8%) | — / — |
| reactors | 21 | ✅ 16450 ms (15 iters) | ✅ 15792 ms (15 iters) | ≈ -4% (±7.6%) | — / — |
| reactors | 25 | ✅ 18869 ms (15 iters) | ✅ 18773 ms (15 iters) | ≈ -0.5% (±5.1%) | — / — |
Internal counter details (ddprof)
ddprof internal counters, latest / dev (✅ = 0, · = unavailable):
| Benchmark | JDK | Dropped rec | Dropped jvmti | Dropped trace | Skipped WC | AGCT fail | Unwind fail |
|---|---|---|---|---|---|---|---|
| akka-uct | 21 | ✅ / ✅ | ✅ / ✅ | ✅ / 1 | 2009 / 1922 | ✅ / ✅ | ✅ / ✅ |
| akka-uct | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 2201 / 2301 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 21 | ✅ / ✅ | ✅ / ✅ | 5 / 4 | 8503 / 8481 | ✅ / ✅ | ✅ / ✅ |
| finagle-chirper | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 2 | 8717 / 8472 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 3 | 1261 / 1255 | ✅ / ✅ | ✅ / ✅ |
| fj-kmeans | 25 | ✅ / ✅ | ✅ / ✅ | 2 / 1 | 1280 / 1266 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 21 | ✅ / ✅ | ✅ / ✅ | 3 / 1 | 2964 / 2991 | ✅ / ✅ | ✅ / ✅ |
| future-genetic | 25 | ✅ / ✅ | ✅ / ✅ | ✅ / ✅ | 2834 / 2958 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 21 | ✅ / ✅ | ✅ / ✅ | 4 / 1 | 3483 / 3477 | ✅ / ✅ | ✅ / ✅ |
| naive-bayes | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 3 | 3463 / 3437 | ✅ / ✅ | ✅ / ✅ |
| reactors | 21 | ✅ / ✅ | ✅ / ✅ | 1 / ✅ | 1797 / 1623 | ✅ / ✅ | ✅ / ✅ |
| reactors | 25 | ✅ / ✅ | ✅ / ✅ | 1 / 1 | 1933 / 1917 | ✅ / ✅ | ✅ / ✅ |
What does this PR do?:
Tags long-running (45-70s+) tests with
@Tag("slow"), excludes them from the defaulttest${Config}Gradle tasks, and adds newtestSlow${Config}tasks (with the same gtest dependency wiring) that run only in the nightly and pre-release-validated CI workflows.Motivation:
Several tests (reference-chain tracking, class-unload/leak stress tests, dump-while-churning-threads) individually take 45-70+ seconds, inflating the regular per-PR test run. They belong in a slow/e2e suite that runs nightly and before releases, not on every PR.
Additional Notes:
ProfilerTestRunner(the musl Exec-task path) gained-Dtest.tags.include/-Dtest.tags.excludesystem-property-driven tag filtering, mirroring the native GradleTesttask'sincludeTags/excludeTags.testSlow${Config}tasks depend on the samegtest${Config}task as theirtest${Config}sibling, since each is an independent CI job.How to test the change?:
./gradlew :ddprof-test:tasksliststestDebug,testRelease,testSlowDebug,testSlowReleasewith correct descriptions../gradlew -q :ddprof-test:testSlowDebug --dry-runconfirms thegtestDebugtask graph is a dependency.nightly.ymlandrelease-validated.ymlvalidated as valid YAML.For Datadog employees:
credentials of any kind, I've requested a security review (run the
dd:platform-security-reviewskill, or file a request via the PSEC review form).
bewairealso runs automatically on every PR.Unsure? Have a question? Request a review!