Add a dormant v4 RDS Postgres dual-write to the CI benchmark emitters - #8513
Merged
Conversation
Teach scripts/post-ingest.py a --postgres mode that connects to the v4 RDS Postgres (IAM auth, verify-full TLS) and upserts benchmark results directly via INSERT ... ON CONFLICT (measurement_id) DO UPDATE, in one transaction with a NaN/Inf guard and a best-effort site revalidate. The pre-existing v3 --server path is unchanged and stays stdlib-only; the v4 path lazily imports its third-party deps (psycopg[binary], boto3, xxhash) supplied at the call site via uv run --no-project --with, so importing the module or running --server never needs them. scripts/_measurement_id.py is a byte-for-byte port of the Rust measurement_id xxhash64 in vortex-data/benchmarks-website's server/src/db.rs. The writer computes the row key client-side so re-ingesting an existing (commit, dimension) upserts the existing row instead of inserting a duplicate. The port is verified against the golden vectors generated by that repo's measurement_id_golden_vectors test (65 vectors pass as of this commit); the vectors and the parity test live with the Rust source of truth rather than in this repo. Signed-off-by: Connor Tsui <connor@spiraldb.com>
…kflows Add the v4 ingest step block to bench.yml, sql-benchmarks.yml, and v3-commit-metadata.yml (the last also gains id-token: write). Each block assumes the ingest role via OIDC and runs post-ingest.py --postgres after the existing v3 --server step. Every v4 step is gated on vars.GH_BENCH_INGEST_ROLE_ARN != '' and carries continue-on-error: true, so with the variable unset the code is dormant and a v4 failure can never fail a workflow or disturb the v2/v3 paths. Setting the variable is the later live cutover. Signed-off-by: Connor Tsui <connor@spiraldb.com>
Merging this PR will degrade performance by 14.56%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | decompress_rd[f64, (10000, 0.01)] |
108.9 µs | 139.3 µs | -21.82% |
| ❌ | Simulation | decompress_rd[f64, (10000, 0.1)] |
109.2 µs | 139.6 µs | -21.79% |
| ❌ | Simulation | decompress_rd[f64, (10000, 0.0)] |
108.9 µs | 139.2 µs | -21.77% |
| ❌ | Simulation | decompress_rd[f32, (100000, 0.0)] |
496.1 µs | 583.9 µs | -15.04% |
| ❌ | Simulation | decompress_rd[f32, (10000, 0.1)] |
78.2 µs | 91.4 µs | -14.4% |
| ❌ | Simulation | decompress_rd[f32, (10000, 0.01)] |
78.2 µs | 91.1 µs | -14.16% |
| ❌ | Simulation | decompress_rd[f32, (10000, 0.0)] |
78.7 µs | 91.3 µs | -13.88% |
| ⚡ | Simulation | eq_i64_constant |
317.9 µs | 287.9 µs | +10.42% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ct/bench-v4-emitter-dual-write (aa5a15a) with develop (9814173)
connortsui20
added a commit
that referenced
this pull request
Jun 19, 2026
Record phase A (ops) done: GitHubBenchmarkIngestRole created and verified (trust repo:vortex-data/vortex on develop + ct/bench-v4; rds-db:connect for bench_ingest). User ran the prod IAM commands after the auto-mode classifier blocked the agent. Set Current Position to the phase-A boundary gate; phases C and B stay hard-gated on demo-over + go-ahead (B also needs #8513 merged). Signed-off-by: Connor Tsui <connor@spiraldb.com>
connortsui20
marked this pull request as ready for review
June 19, 2026 18:51
connortsui20
added a commit
that referenced
this pull request
Jun 19, 2026
Demo confirmed over; user runs the prod ops themselves (harness blocks the agent from prod mutations). Record the C/B cutover scripts (/tmp/cutover-phase-c.sh, /tmp/cutover-phase-b.sh), the generated revalidate token, the exact var/secret changes, and the C -> merge #8513 -> B order, in the ledger for resume. Signed-off-by: Connor Tsui <connor@spiraldb.com>
connortsui20
added a commit
that referenced
this pull request
Jun 19, 2026
Record phase C (ops) done: BENCH_REVALIDATE_TOKEN aligned on the v4 Vercel project
+ monorepo secret, v4 redeployed via an empty commit to benchmarks-website develop
(dashboard Redeploy was locked; vercel --prod hit the file-upload cap), verified
200 {revalidated:true}. Advance Current Position to phase B (flip the var),
pending #8513 admin-merge + the cutover-phase-b.sh run.
Signed-off-by: Connor Tsui <connor@spiraldb.com>
connortsui20
added a commit
that referenced
this pull request
Jun 19, 2026
Record the one-time historical backfill as Phase 5: re-run vortex-bench-migrate to load pre-cutover data into v4 RDS, in a quiet no-merge window. Safe after cutover (keyed measurement_id upserts; no dupes/clobber). v4 not primary yet so users are unaffected; DNS cutover stays out of scope and must follow the backfill. Immediate finish (merge #8513 -> phase B) is unchanged. Signed-off-by: Connor Tsui <connor@spiraldb.com>
lwwmanning
approved these changes
Jun 19, 2026
connortsui20
added a commit
that referenced
this pull request
Jun 19, 2026
Post-merge bench-CI scare investigated via ci-failure-analysis: the bench job failures were `cancelled` benchmark steps (GitHub infra blip during 3 rapid develop pushes; bench.yml has no concurrency block), the PutObject PreconditionFailed is pre-existing cat-s3.sh optimistic-concurrency retry (#7296, untouched by #8513), and the 503 is the continue-on-error v4 revalidate / S3 SlowDown. #8513's workflow diff is pure addition (no permissions/matrix/existing- step change); post-ingest.py does no S3 PutObject. Logged the open soak item (confirm a clean bench run writes inserted/updated > 0 to v4). Signed-off-by: Connor Tsui <connor@spiraldb.com>
connortsui20
added a commit
that referenced
this pull request
Jun 20, 2026
The bench-v4 CI emitter dual-write project shipped to develop via separate code-only PRs (#8513 emitter + workflows, #8516 reorder CI fix + commit-metadata rename, #8517 binary-only uv install). This orchestration branch is retired down to just its `.big-plans/` record -- the spine (goal, decisions, phase map, verdict ledger), the design spec, and the per-sub-phase task plans -- as a durable archive of the planning + execution history. The commit history is squashed; the shipped code lives on develop. Signed-off-by: Connor Tsui <connor@spiraldb.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CI benchmark emitters currently need a manual
vortex-bench-migraterun to populate the new v4 RDS Postgres database. This adds a best-effort, env-gated dual-write so the emitters populate it directly: a new--postgresmode inscripts/post-ingest.py(IAM auth, verify-full TLS, a five-table-plus-commit upsert keyed onmeasurement_id, NaN/Inf guard, best-effort site revalidate), plusscripts/_measurement_id.py, a byte-for-byte port of the Rustmeasurement_idhash so re-ingests upsert the existing row instead of duplicating. The existing v2 and v3 write paths are untouched, and the v3--serverpath stays stdlib-only.It ships dormant: the v4 step added to
bench.yml,sql-benchmarks.yml, andv3-commit-metadata.ymlis gated onvars.GH_BENCH_INGEST_ROLE_ARNand carriescontinue-on-error: true, so with the variable unset the new code is dead-but-safe and a v4 failure can never fail a workflow. Provisioning the IAM role and setting that variable is a separate ops step after this merges.The
measurement_idport is verified against the cross-language golden vectors (65 pass); those vectors and the parity test live invortex-data/benchmarks-websitewith the Rust source of truth rather than in this repo, and the heavier testcontainer and revalidate checks are run out-of-band.🤖 Generated with Claude Code
Note from human (connor): This is the only thing that needs to be merged on this repo. Everything else after that needs to be done outside of this repo. That's really just configuring AWS (Vortex account), vercel project, migration of the old data, and repo secrets / variables.