Add smoke test JUnit framework - #12111
Conversation
PerfectSlayer
added
type: feature
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
4a50f04 to
11530c9
Compare
Bits has a CI fix ready🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready
View in Datadog | Reviewed commit 4abdba0 · Any feedback? Reach out in #deveng-pr-agent |
11530c9 to
4abdba0
Compare
|
@codex review |
![chatgpt-codex-connector[bot]](https://cdn.github.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4abdba0567
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| new PollingConditions(timeoutSeconds) | ||
| .eventually(() -> SmokeTraceAssertions.assertTraces(getTraces(), options, matchers)); |
There was a problem hiding this comment.
For the default count-exact assertion, PollingConditions returns immediately when one snapshot matches. Because both backends receive traces asynchronously, the expected traces can match first and an additional trace from the same test action can arrive just afterward, causing the test to pass even though the final collection violates the exact-count contract; this also makes IGNORE_ADDITIONAL_TRACES ineffective as the distinction between strict and permissive behavior in that scenario. The strict path needs a stabilization/final-count check rather than accepting the first transient match.
Useful? React with 👍 / 👎.
b169c19 to
9c11764
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c1176468d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @Override | ||
| public void beforeEach(ExtensionContext context) { | ||
| if (clearsBetweenTests()) { | ||
| clear(); | ||
| } |
There was a problem hiding this comment.
When a SmokeCliApp uses a backend that is also registered as a shared extension, the CLI process can finish and emit all of its traces during beforeAll, after which this beforeEach callback clears them before the test body runs. The CLI app's deliberate lack of an owned-backend reset cannot prevent this; TestAgentBackend only works if callers discover and enable retainAcrossTests(), while MockAgentBackend offers no retention mode, so shared CLI trace assertions are otherwise unusable.
Useful? React with 👍 / 👎.
9c11764 to
b6b0809
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b6b0809cde
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
b6b0809 to
20e23f3
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 20e23f3f65
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| this.process.destroyForcibly(); | ||
| } |
There was a problem hiding this comment.
When teardown is interrupted while waiting for the child, this branch restores the interrupt flag and returns immediately after destroyForcibly(), which is asynchronous and does not guarantee the process has exited. During Gradle cancellation or another teardown interrupt, the child can therefore retain its port and files after the extension completes; the interrupted path should still confirm termination while preserving the interrupt status.
Useful? React with 👍 / 👎.
| private void collect(HandlerApi api, TraceFormat format) { | ||
| DecodedMessage message = format.decode(api.getRequest().getBody()); | ||
| this.traces.addAll(message.getTraces()); | ||
| api.getResponse().status(200).sendWithType(JSON, "{}"); |
There was a problem hiding this comment.
When a tracer regression sends an absent or incorrect X-Datadog-Trace-Count, the mock backend decodes and records the payload and returns 200 without checking the header, so smoke tests using this documented backend can pass despite a malformed intake request. The existing AbstractSmokeTest.handleTrace checks the decoded count against this header, and the test-agent backend enables trace_count_header by default, so the mock path should preserve the same validation rather than silently accepting the regression.
Useful? React with 👍 / 👎.
This comment has been minimized.
This comment has been minimized.
20e23f3 to
76991be
Compare
![datadog-datadog-us1-prod[bot]](https://cdn.github.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The new JUnit smoke framework cannot compile as configured: the PR removes the module's direct JUnit Jupiter compile dependency while its new main sources import JUnit extension APIs. Restore that compile-only dependency before shipping. No additional tests recommended: this is a build-configuration defect, and the existing tests cannot compile until the dependency is restored.
🤖 Datadog Autotest · Commit 76991be · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
|
|
||
| dependencies { | ||
| api libs.okhttp | ||
| api libs.testcontainers |
There was a problem hiding this comment.
The dd-smoke-tests module fails at compilation, so the new JUnit smoke framework cannot be built or consumed by smoke-test modules.
Assertion details
- Input: Build dd-smoke-tests with the new AbstractSmokeApp, SmokeCliApp, SmokeServerApp, and backend sources present.
- Expected:
The smoke-test main source set compiles with org.junit.jupiter.api.extension available. - Actual:
The PR removes compileOnly(libs.junit.jupiter) from dd-smoke-tests/build.gradle, while new main sources import org.junit.jupiter.api.extension.AfterAllCallback, BeforeAllCallback, BeforeEachCallback, and ExtensionContext. Gradle therefore cannot compile the new framework unless another direct/transitive dependency supplies those APIs; the module's existing project dependencies do not declare JUnit Jupiter as an API dependency.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| api libs.testcontainers | |
| api libs.testcontainers | |
| compileOnly(libs.junit.jupiter) |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76991be5ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
76991be to
0f9f704
Compare
1 participant
What Does This Do
This PR adds a JUnit 5 / Java smoke test framework that allows to write smoke tests using Java/JUnit 5 instead of Groovy/Spock.
The framework has four parts:
App runners:
AbstractSmokeAppwithSmokeServerApp(long-running/HTTP) andSmokeCliApp(batch) concrete types, registered as a@RegisterExtension.Owns the launched app's lifecycle and wires in the agent jar + test agent session token.
Backends: a common
TraceBackendfacade with two interchangeable implementations:MockAgentBackend(in-processJavaTestHttpServer, no Docker) andTestAgentBackend(a Testcontainers-managed, or external/CI, dd-apm-test-agent, scoped per test via
X-Datadog-Test-Session-Token).A test written against the facade runs unchanged on either backend but the goal is moving toward
TestAgentBackendby default.Capture surfaces:
Traces(msgpack/JSON decoded through the shared decoder),Logs(std out/error logs with clear, waitFor capabilities),Telemetry(with message-batch flattening), andRemoteConfig(push a config thetracer polls for, and read back its
/v0.7/configpolls, with products/capabilitiesdecoders).
Traces, telemetry, and remote config are all isolated per test for owned backends.
Motivation
The goal is to provide framework for testing tracing instrumentations and products.
Unlike the existing solution, it verifies the signals against an agent, not our our decoders, and it no longer relies on Groovy / Spock but Java and JUnit.
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: APMLP-1247