raft: add streaming multi-group soak evidence

[bootjp]

Summary

• add an executable three-group, three-node-per-group gRPC transport soak with reconnect, flow-control backpressure, snapshot, and recovery sentinels
• integrate a five-node, three-group DynamoDB Jepsen run and fail-closed cumulative transport-metrics verification
• expose observational transport counters and record schema-v2 evidence
• mark the streaming transport design as implemented with its acceptance evidence and operational instructions

Risk

The runtime change is limited to observational transport counters. Raft message routing, acknowledgements, quorum decisions, and fallback behavior are unchanged. The soak and verifier fail closed when required topology, delivery, reconnect, snapshot, or backpressure evidence is missing.

Test evidence

• make lint
• go test -race ./internal/raftengine/etcd/transportsoak ./internal/raftengine/etcd ./monitoring ./cmd/elastickv-raft-stream-soak
• go run ./cmd/elastickv-raft-stream-soak -output docs/evidence/raft_streaming_multigroup_soak.json
• go run ./cmd/elastickv-raft-stream-soak -verify docs/evidence/raft_streaming_multigroup_soak.json
• bash -n scripts/run-jepsen-raft-streaming-multigroup-soak.sh scripts/verify-raft-streaming-multigroup-metrics.sh
• shellcheck scripts/run-jepsen-raft-streaming-multigroup-soak.sh scripts/verify-raft-streaming-multigroup-metrics.sh
• cd jepsen && lein test elastickv.cli-test elastickv.dynamodb-multi-table-workload-test (23 tests, 42 assertions)

[coderabbitai]

Warning

Review limit reached

@bootjp, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6dcdf34f-b8db-4b5e-a3d3-79db043fadc6

📥 Commits

Reviewing files that changed from the base of the PR and between 37277ea and b6d1b8c.

📒 Files selected for processing (15)

• cmd/elastickv-raft-stream-soak/main.go
• docs/design/2026_04_18_implemented_raft_grpc_streaming_transport.md
• docs/evidence/raft_streaming_multigroup_soak.json
• internal/raftengine/etcd/engine.go
• internal/raftengine/etcd/grpc_transport.go
• internal/raftengine/etcd/transportsoak/soak.go
• internal/raftengine/etcd/transportsoak/soak_test.go
• jepsen/src/elastickv/cli.clj
• jepsen/src/elastickv/db.clj
• jepsen/src/elastickv/dynamodb_multi_table_workload.clj
• jepsen/test/elastickv/dynamodb_multi_table_workload_test.clj
• monitoring/hotpath.go
• monitoring/hotpath_test.go
• scripts/run-jepsen-raft-streaming-multigroup-soak.sh
• scripts/verify-raft-streaming-multigroup-metrics.sh

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[bootjp]

Latest-head evidence for da2f2c180ebaead0fcd824e00f357518fe690002:

• signed commit verified for bootjp <contact@bootjp.me>
• full repository lint hook: 0 issues
• focused race tests passed for transport soak, Raft transport/engine, monitoring, and the soak command
• deterministic 3-group x 3-node TCP gRPC soak passed; schema-v2 evidence regenerated and independently verified
• targeted Jepsen tests passed: 23 tests, 42 assertions
• shell syntax, ShellCheck, and diff checks passed
• design status is Implemented; no proposed/partial path references remain

The local full go test ./... run passed every reported package except adapter, whose existing integration suite hit the global 10-minute test timeout under heavy concurrent local builds. The changed Raft, transport-soak, monitoring, store, and command packages passed, and CI is running the clean hosted suite.

@codex review

[gemini-code-assist]

Code Review

This pull request introduces a deterministic transport soak tool, updates Jepsen workloads to support multi-group streaming soak tests, and adds Prometheus metrics to track outbound stream opens, reconnects, messages, and snapshot activity. A critical issue was identified in the Jepsen database setup where server environment variables are silently ignored because start-daemon! does not support the :env key; wrapping the daemon startup with c/with-env was suggested to resolve this.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[bootjp]

Latest-head review update for da2f2c180ebaead0fcd824e00f357518fe690002:

The high Jepsen environment finding was checked against the pinned jepsen 0.3.11 artifact. Its start-daemon! implementation explicitly consumes :env and executes start-stop-daemon with that environment. I replied with the dependency-source evidence and resolved the thread; no code change was needed. CI remains in progress.

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da2f2c180e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Latest-head evidence for b6d1b8c0aef78a4720d86c2b1d4e6aebd71676ce:

• fixed all three current-head review findings and resolved their threads
• built and passed elastickv-list-routes plus distributed n1:50051 to the routing setup hook
• bounded metrics collection with 2s connect and 5s total curl deadlines
• wired every node/group Raft address to its Dynamo endpoint and audited both start-node! callers (initial setup and restart)
• targeted Jepsen: 24 tests, 43 assertions, 0 failures/errors
• transport soak test, helper builds, repository lint hook, shell syntax, ShellCheck, signature, and diff checks passed

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: b6d1b8c0ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bootjp]

Review-loop stop recorded for b6d1b8c0aef78a4720d86c2b1d4e6aebd71676ce: all CI checks are successful, the latest-head connector review reports no major issues, and the inline surface has no new finding whose original_commit_id matches this head. No merge performed.