Build and test mssql-odbc across many Linux distros via prebuilt CI artifacts

[saurabh500]

Summary

Extends the main-branch CI pipeline (not PR builds) to build the mssql-odbc driver + C++ gtest e2e binaries once per (arch, libc) and publish them as pipeline artifacts, then download and rerun those prebuilt binaries against a live SQL Server inside many distro containers — mirroring the existing mssql-tds nextest-archive → artifact → multi-distro pattern.

This is a draft for iteration; the full e2e runs can only be validated by running the pipeline (no live SQL Server locally). All YAML and shell scripts pass yaml.safe_load and bash -n.

Depends on #128 — the container-image sync changes (import redhat/ubi8, build + mirror the manylinux_2_28_{x86_64,aarch64}_rust images to GHCR) needed by the glibc-2.28 / RHEL 8 track were split into that PR to keep this one scoped to the ODBC build/test pipeline. #128 must merge and the sync pipeline must run first so the _rust images land in GHCR before these tracks can pull them.

Approach: build once, test everywhere

The e2e flow was split into two halves:

• build_e2e.sh [--release] [--out=DIR] — builds the Rust driver + ctest binaries and stages build/ (with libmsodbcsql18.so copied inside) into a drop dir that gets published as an artifact.
• run_e2e.sh --skip-build — skips all compilation, restores the prebuilt build/ tree, auto-resolves the driver from build/libmsodbcsql18.so, and reruns the binaries via CTest.

CTestTestfile.cmake bakes absolute paths, so build and test jobs both mount the repo at /workspace to keep paths consistent.

Build tracks (libc / OpenSSL specific)

Track	Build base	Artifact	Reused on
glibc modern x64	Ubuntu 22.04 (glibc 2.35, OpenSSL 3)	Odbc_Linux_x64	Debian bookworm, Ubuntu 22.04/24.04, Azure Linux 3
glibc modern arm64	Ubuntu 22.04 arm64	Odbc_Linux_arm64	same (cross-pool x64 SQL host)
musl x64 / arm64	Alpine 3.18 (musl, OpenSSL 3)	Odbc_Musl_x64 / Odbc_Musl_arm64	Alpine 3.18–3.21
glibc 2.28 x64	manylinux_2_28_x86_64_rust (AlmaLinux 8, OpenSSL 1.1)	Odbc_glibc228_x64	RHEL 8 / UBI 8 x64
glibc 2.28 arm64	manylinux_2_28_aarch64_rust (AlmaLinux 8, OpenSSL 1.1)	Odbc_glibc228_arm64	RHEL 8 / UBI 8 arm64 (cross-pool x64 SQL host)

A glibc-2.35 / OpenSSL-3 binary won't load on RHEL 8 (glibc 2.28, libssl.so.1.1), so the glibc-2.28 track is built separately, on both x64 and arm64. RHEL 9 / Oracle 9 (glibc 2.34) are intentionally excluded from the modern track.

The glibc-2.28 tracks build in the pre-baked manylinux_2_28_{x86_64,aarch64}_rust images (from #128), which ship rust + cmake + unixODBC-devel + OpenSSL 1.1 dev, so nothing is installed at build time.

New CI stages

The ODBC e2e reuses the existing Test_amd64 / Test_arm64 stages (no new stages / agent allocation):

• glibc modern — Debian bookworm, Ubuntu 22.04/24.04, Azure Linux 3 (x64 + arm64)
• musl — Alpine 3.18–3.21 (x64 + arm64)
• glibc 2.28 — RHEL 8 / UBI 8 (x64 + arm64)

arm64 tests use the existing cross-pool x64 SQL host orchestration (sql-host-template.yml: derive password, poll-for-endpoint, teardown sentinels), passing ODBC_TEST_SERVER=host,port. The arm64 RHEL 8 job publishes an odbc_RHEL8 teardown sentinel and uses a nameSuffix on the ODBC matrix job to avoid a duplicate job name alongside the modern-glibc arm64 ODBC matrix.

Files

New scripts

• mssql-odbc/tests/e2e/build_e2e.sh — build half
• .pipeline/scripts/containerized-odbc-build.sh (Ubuntu), containerized-odbc-build-glibc228.sh (manylinux_2_28 _rust)
• scripts/dockerentry/alpine-odbc-build.sh (musl build)
• scripts/dockerentry/odbc-e2e-{deb,rhel,azlinux3,alpine}.sh (test entrypoints)

New templates

• build-odbc-template.yml, build-odbc-alpine-template.yml, build-odbc-glibc228-template.yml
• test-matrix-odbc-template.yml, test-matrix-odbc-template-arm64.yml

Modified

• mssql-odbc/tests/e2e/run_e2e.sh — --skip-build / --driver flags
• .pipeline/templates/validation-stages.yml — build calls (incl. glibc-2.28 arm64) + ODBC test matrices (incl. arm64 RHEL 8, shared + per-job)
• .pipeline/templates/build-odbc-glibc228-template.yml — architecture param; default buildImage → GHCR manylinux_2_28_x86_64_rust
• .pipeline/scripts/containerized-odbc-build-glibc228.sh — dropped runtime dnf install + rustup (now baked into the _rust image)
• .pipeline/templates/test-matrix-odbc-template-arm64.yml — nameSuffix param for the shared-mode job name
• mssql-odbc/tests/e2e/README.md — document the flow

Container-image sync (.pipeline/sync-container-images.yml) and the manylinux_2_28_*_rust Dockerfiles are handled in #128, not here.

Open items / assumptions to verify in-pipeline

• musl .so runtime deps on Alpine (libssl3, unixodbc, libstdc++).
• Driver accepts Server=host,port; for the arm64 cross-pool endpoint (assumed from BuildConnectionString).
• No green live CI run yet for any ODBC track; the arm64 shared-host sentinel accounting is the highest-risk part.

Work item

AB#46492

Verification done

• All new/edited pipeline YAML validated with yaml.safe_load.
• All new/edited shell scripts pass bash -n.
• Fixed a real bug: dropped --entrypoint bash (Alpine base images have no bash) in favor of executing entry scripts via their shebang, matching the nextest templates.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 100%

🎯 Overall Coverage 90.9%

📦 Project: mssql-tds + mssql-py-core ℹ️ Note: diff coverage is reported, not enforced.

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

---

🔗 Quick Links

View Azure DevOps Build · Coverage Report

[Copilot]

Pull request overview

Extends the main-branch CI pipeline to build mssql-odbc + its CTest/gtest e2e binaries once per (arch, libc/OpenSSL track), publish them as pipeline artifacts, and then re-run those prebuilt binaries across a multi-distro container matrix (including arm64 cross-pool SQL host orchestration), mirroring the existing nextest archive pattern used elsewhere in the repo.

Changes:

• Add split e2e scripts (build_e2e.sh + run_e2e.sh --skip-build) to enable artifact reuse across distro containers.
• Introduce new Azure Pipelines templates to build/publish ODBC drops for glibc (modern), musl (Alpine), and glibc 2.28 (RHEL8) tracks, plus corresponding test matrices (amd64 + arm64).
• Add distro-specific container entry scripts and update image sync to ensure required base images are available (e.g., UBI8, manylinux_2_28 mirror).

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/dockerentry/odbc-e2e-rhel.sh	RHEL-family test container entrypoint (dnf) to install runtime deps and run prebuilt ODBC e2e.
scripts/dockerentry/odbc-e2e-deb.sh	Debian/Ubuntu test container entrypoint (apt) to install runtime deps and run prebuilt ODBC e2e.
scripts/dockerentry/odbc-e2e-azlinux3.sh	Azure Linux 3 test container entrypoint (tdnf) to install runtime deps and run prebuilt ODBC e2e.
scripts/dockerentry/odbc-e2e-alpine.sh	Alpine (musl) test container entrypoint (apk) to install runtime deps and run prebuilt ODBC e2e.
scripts/dockerentry/alpine-odbc-build.sh	Alpine (musl) build container entrypoint to produce and stage a reusable ODBC e2e drop.
mssql-odbc/tests/e2e/run_e2e.sh	Add --skip-build / --driver support and adjust help output for prebuilt artifact execution.
mssql-odbc/tests/e2e/README.md	Document the “build once, test everywhere” artifact flow and libc/OpenSSL track constraints.
mssql-odbc/tests/e2e/build_e2e.sh	New build-only script to produce the driver + CMake build tree and stage them into an artifact drop dir.
.pipeline/templates/validation-stages.yml	Wire new ODBC build steps and ODBC test matrices into the main-branch validation stages (non-PR).
.pipeline/templates/test-matrix-template-arm64.yml	Add manageSqlHost to allow sharing a single cross-pool SQL host with another matrix in the same stage.
.pipeline/templates/test-matrix-template-alpine_arm64.yml	Add manageSqlHost to support sharing the musl cross-pool SQL host with ODBC musl arm64 matrix.
.pipeline/templates/test-matrix-odbc-template.yml	New amd64 ODBC e2e test matrix template that downloads an ODBC drop and reruns via containers.
.pipeline/templates/test-matrix-odbc-template-arm64.yml	New arm64 ODBC e2e matrix template that uses cross-pool SQL host sentinels and reruns prebuilt binaries.
.pipeline/templates/build-odbc-template.yml	New template to build/publish a modern-glibc ODBC drop from Ubuntu 22.04 build container.
.pipeline/templates/build-odbc-glibc228-template.yml	New template to build/publish a glibc 2.28 / OpenSSL 1.1 ODBC drop for RHEL8 compatibility.
.pipeline/templates/build-odbc-alpine-template.yml	New template to build/publish a musl ODBC drop from Alpine build container.
.pipeline/sync-container-images.yml	Import/mirror additional base images required by the new ODBC distro matrices (e.g., UBI8, manylinux_2_28).
.pipeline/scripts/containerized-odbc-build.sh	New build-container helper script to install deps and run build_e2e.sh for Ubuntu-based track.
.pipeline/scripts/containerized-odbc-build-glibc228.sh	New build-container helper script to install deps + rustup and run build_e2e.sh for glibc 2.28 track.

[David-Engel]

Summary

Extends the main-branch CI to build the mssql-odbc driver + C++ gtest e2e binaries once per (arch, libc) track and rerun the prebuilt binaries across many distro containers — mirroring the existing mssql-tds nextest-archive pattern. The design is sound, the split build/run flow is well-reasoned, and the cross-job SQL-host sentinel accounting checks out. Findings below are non-blocking; the real validation is a green live pipeline run (which requires #128 first).

Verified correct (the parts flagged as risky)

• arm64 shared-host sentinel accounting balances. The Test_arm64 shared host declares 12 expectedSentinels, and the three consumer matrices publish exactly those 12: nextest (7) + ODBC modern odbc_{DebianBookworm,Ubuntu22,Ubuntu24,AzLinux3} (4) + ODBC glibc228 odbc_RHEL8 (1). Test_alpine_arm64 similarly balances (4 nextest + 4 odbc_ = 8). No orphaned/missing sentinels that would hang or prematurely release the host.
• The format/replace sentinel-prefix expression in test-matrix-odbc-template-arm64.yml correctly produces odbc_A,odbc_B (prefix applied to every element, not just after commas) and degrades to a plain join when sentinelPrefix is ''.
• chmod +x restore is complete. The find … -name '*_test' in run_e2e.sh matches every compiled binary — CMakeLists names them all *_test via add_odbc_test(...). Good catch on the artifact-strips-exec-bit problem.
• Absolute-path / /workspace mount reasoning holds — both build and test jobs mount the repo at /workspace, so the paths baked into CTestTestfile.cmake resolve on rerun. The dedicated CARGO_TARGET_DIR=/workspace/target-glibc228 doesn't affect this since the driver is copied into build/.
• amd64 SQL image hardcoding matches convention — test-matrix-template.yml also hardcodes mssql/server:2025-latest, so the new ODBC amd64 template isn't a deviation.

Suggestions

• Heavy duplication in validation-stages.yml. The Alpine and deb/rhel target lists are now repeated across shared-mode and per-job-mode for both nextest and ODBC (6+ copies of near-identical targets: blocks). Adding/removing a distro requires editing many places and invites drift. Consider hoisting each target list into a reusable variable / YAML anchor or a parameter default. Mirrors the existing pattern, so not merge-blocking.
• ~130-line block duplicated between shared-mode and per-job-mode in test-matrix-odbc-template-arm64.yml; the two steps: sequences differ only in how container/entry/sentinel names are sourced. Hard to factor further in ADO templates, but worth a note so future edits stay in sync.
• Dead artifact output: build_e2e.sh writes "$OUT_DIR/DRIVER_FILE", but run_e2e.sh --skip-build re-derives the lib name from uname and never reads it. Either consume it in run_e2e.sh (single source of truth) or drop the write.

Nits

• Inconsistent architecture casing: the glibc228 arm64 call passes architecture: arm64 while sibling calls use ARM64/x64. Display-only (step name), so harmless, but worth normalizing.
• The RHEL/UBI8 dnf install unixODBC cmake in odbc-e2e-rhel.sh should resolve from ubi-8-appstream, but confirm in the first live run (along with the already-listed musl runtime-dep and Server=host,port assumptions).

Merge ordering

The glibc-2.28 / RHEL8 tracks pull manylinux_2_28_{x86_64,aarch64}_rust and import/redhat/ubi8 from GHCR, which #128 provides. As the description states, #128 must merge and its sync pipeline must run before these tracks can pull the images — otherwise those matrix legs fail on image pull. Worth reconfirming ordering at merge time.

Overall: clean, well-documented, correctly wired. Main pre-merge action is a green live pipeline run; the duplication items are quality-of-life, not blockers.

[saurabh500]

Thanks for the review, David. Addressed in 4a1c7af:

• Removed the dead OUT_DIR/DRIVER_FILE write in build_e2e.sh — run_e2e.sh --skip-build re-derives the driver name from uname and never read it.
• Normalized the glibc-2.28 arm64 build call to architecture: ARM64 to match its sibling ODBC/nextest calls (display-only string).

On the "verify in a live run" items — these all passed in the last full green run (build 161627): the RHEL8/ubi8 dnf install of msodbcsql18, the musl runtime-dep resolution on Alpine, and the Server=host,port e2e connections across all glibc/musl/glibc-2.28 x64+arm64 distros.

I filed the duplication cleanups (target lists in validation-stages.yml + the arm64 template block) as #138 rather than reworking this green PR — happy to prioritize if you'd prefer it here.