Skip to content

Add a dormant v4 RDS Postgres dual-write to the CI benchmark emitters - #8513

Merged
connortsui20 merged 2 commits into
developfrom
ct/bench-v4-emitter-dual-write
Jun 19, 2026
Merged

Add a dormant v4 RDS Postgres dual-write to the CI benchmark emitters#8513
connortsui20 merged 2 commits into
developfrom
ct/bench-v4-emitter-dual-write

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jun 19, 2026

Copy link
Copy Markdown
Member

The CI benchmark emitters currently need a manual vortex-bench-migrate run to populate the new v4 RDS Postgres database. This adds a best-effort, env-gated dual-write so the emitters populate it directly: a new --postgres mode in scripts/post-ingest.py (IAM auth, verify-full TLS, a five-table-plus-commit upsert keyed on measurement_id, NaN/Inf guard, best-effort site revalidate), plus scripts/_measurement_id.py, a byte-for-byte port of the Rust measurement_id hash so re-ingests upsert the existing row instead of duplicating. The existing v2 and v3 write paths are untouched, and the v3 --server path stays stdlib-only.

It ships dormant: the v4 step added to bench.yml, sql-benchmarks.yml, and v3-commit-metadata.yml is gated on vars.GH_BENCH_INGEST_ROLE_ARN and carries continue-on-error: true, so with the variable unset the new code is dead-but-safe and a v4 failure can never fail a workflow. Provisioning the IAM role and setting that variable is a separate ops step after this merges.

The measurement_id port is verified against the cross-language golden vectors (65 pass); those vectors and the parity test live in vortex-data/benchmarks-website with the Rust source of truth rather than in this repo, and the heavier testcontainer and revalidate checks are run out-of-band.

🤖 Generated with Claude Code

Note from human (connor): This is the only thing that needs to be merged on this repo. Everything else after that needs to be done outside of this repo. That's really just configuring AWS (Vortex account), vercel project, migration of the old data, and repo secrets / variables.

Teach scripts/post-ingest.py a --postgres mode that connects to the v4 RDS
Postgres (IAM auth, verify-full TLS) and upserts benchmark results directly via
INSERT ... ON CONFLICT (measurement_id) DO UPDATE, in one transaction with a
NaN/Inf guard and a best-effort site revalidate. The pre-existing v3 --server
path is unchanged and stays stdlib-only; the v4 path lazily imports its
third-party deps (psycopg[binary], boto3, xxhash) supplied at the call site via
uv run --no-project --with, so importing the module or running --server never
needs them.

scripts/_measurement_id.py is a byte-for-byte port of the Rust measurement_id
xxhash64 in vortex-data/benchmarks-website's server/src/db.rs. The writer computes
the row key client-side so re-ingesting an existing (commit, dimension) upserts
the existing row instead of inserting a duplicate. The port is verified against
the golden vectors generated by that repo's measurement_id_golden_vectors test
(65 vectors pass as of this commit); the vectors and the parity test live with
the Rust source of truth rather than in this repo.

Signed-off-by: Connor Tsui <connor@spiraldb.com>
…kflows

Add the v4 ingest step block to bench.yml, sql-benchmarks.yml, and
v3-commit-metadata.yml (the last also gains id-token: write). Each block assumes
the ingest role via OIDC and runs post-ingest.py --postgres after the existing v3
--server step. Every v4 step is gated on vars.GH_BENCH_INGEST_ROLE_ARN != '' and
carries continue-on-error: true, so with the variable unset the code is dormant
and a v4 failure can never fail a workflow or disturb the v2/v3 paths. Setting the
variable is the later live cutover.

Signed-off-by: Connor Tsui <connor@spiraldb.com>
@codspeed-hq

codspeed-hq Bot commented Jun 19, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 14.56%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 7 regressed benchmarks
✅ 1573 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation decompress_rd[f64, (10000, 0.01)] 108.9 µs 139.3 µs -21.82%
Simulation decompress_rd[f64, (10000, 0.1)] 109.2 µs 139.6 µs -21.79%
Simulation decompress_rd[f64, (10000, 0.0)] 108.9 µs 139.2 µs -21.77%
Simulation decompress_rd[f32, (100000, 0.0)] 496.1 µs 583.9 µs -15.04%
Simulation decompress_rd[f32, (10000, 0.1)] 78.2 µs 91.4 µs -14.4%
Simulation decompress_rd[f32, (10000, 0.01)] 78.2 µs 91.1 µs -14.16%
Simulation decompress_rd[f32, (10000, 0.0)] 78.7 µs 91.3 µs -13.88%
Simulation eq_i64_constant 317.9 µs 287.9 µs +10.42%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/bench-v4-emitter-dual-write (aa5a15a) with develop (9814173)

Open in CodSpeed

connortsui20 added a commit that referenced this pull request Jun 19, 2026
Record phase A (ops) done: GitHubBenchmarkIngestRole created and verified (trust
repo:vortex-data/vortex on develop + ct/bench-v4; rds-db:connect for bench_ingest).
User ran the prod IAM commands after the auto-mode classifier blocked the agent.
Set Current Position to the phase-A boundary gate; phases C and B stay hard-gated
on demo-over + go-ahead (B also needs #8513 merged).

Signed-off-by: Connor Tsui <connor@spiraldb.com>
@connortsui20
connortsui20 marked this pull request as ready for review June 19, 2026 18:51
@connortsui20
connortsui20 requested a review from a team June 19, 2026 18:51
connortsui20 added a commit that referenced this pull request Jun 19, 2026
Demo confirmed over; user runs the prod ops themselves (harness blocks the agent
from prod mutations). Record the C/B cutover scripts (/tmp/cutover-phase-c.sh,
/tmp/cutover-phase-b.sh), the generated revalidate token, the exact var/secret
changes, and the C -> merge #8513 -> B order, in the ledger for resume.

Signed-off-by: Connor Tsui <connor@spiraldb.com>
connortsui20 added a commit that referenced this pull request Jun 19, 2026
Record phase C (ops) done: BENCH_REVALIDATE_TOKEN aligned on the v4 Vercel project
+ monorepo secret, v4 redeployed via an empty commit to benchmarks-website develop
(dashboard Redeploy was locked; vercel --prod hit the file-upload cap), verified
200 {revalidated:true}. Advance Current Position to phase B (flip the var),
pending #8513 admin-merge + the cutover-phase-b.sh run.

Signed-off-by: Connor Tsui <connor@spiraldb.com>
connortsui20 added a commit that referenced this pull request Jun 19, 2026
Record the one-time historical backfill as Phase 5: re-run vortex-bench-migrate to
load pre-cutover data into v4 RDS, in a quiet no-merge window. Safe after cutover
(keyed measurement_id upserts; no dupes/clobber). v4 not primary yet so users are
unaffected; DNS cutover stays out of scope and must follow the backfill. Immediate
finish (merge #8513 -> phase B) is unchanged.

Signed-off-by: Connor Tsui <connor@spiraldb.com>
@connortsui20
connortsui20 merged commit 97850e9 into develop Jun 19, 2026
76 of 78 checks passed
@connortsui20
connortsui20 deleted the ct/bench-v4-emitter-dual-write branch June 19, 2026 19:26
connortsui20 added a commit that referenced this pull request Jun 19, 2026
Post-merge bench-CI scare investigated via ci-failure-analysis: the bench job
failures were `cancelled` benchmark steps (GitHub infra blip during 3 rapid
develop pushes; bench.yml has no concurrency block), the PutObject
PreconditionFailed is pre-existing cat-s3.sh optimistic-concurrency retry (#7296,
untouched by #8513), and the 503 is the continue-on-error v4 revalidate / S3
SlowDown. #8513's workflow diff is pure addition (no permissions/matrix/existing-
step change); post-ingest.py does no S3 PutObject. Logged the open soak item
(confirm a clean bench run writes inserted/updated > 0 to v4).

Signed-off-by: Connor Tsui <connor@spiraldb.com>
connortsui20 added a commit that referenced this pull request Jun 20, 2026
The bench-v4 CI emitter dual-write project shipped to develop via separate
code-only PRs (#8513 emitter + workflows, #8516 reorder CI fix + commit-metadata
rename, #8517 binary-only uv install). This orchestration branch is retired down
to just its `.big-plans/` record -- the spine (goal, decisions, phase map, verdict
ledger), the design spec, and the per-sub-phase task plans -- as a durable archive
of the planning + execution history. The commit history is squashed; the shipped
code lives on develop.

Signed-off-by: Connor Tsui <connor@spiraldb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants