Add SqlClient performance test pipeline with baseline comparison, noise reduction, and ADX ingestion

[cheenamalhotra]

Summary

Adds a performance-test pipeline for Microsoft.Data.SqlClient under eng/pipelines/perf/. It runs the BenchmarkDotNet perf suite on a dedicated performance test lab (Azure Dedicated Hosts), compares the branch under test against a released NuGet baseline, applies noise-reduction controls, and optionally ingests results into an Azure Data Explorer (Kusto) database.

What's included

• Pipeline (sqlclient-perf-pipeline.yml) — extends the reusable Perf.Test.Job template; provisions a VM, runs benchmarks over SSH, collects results, and runs post-test translation/ingestion on the agent.
• On-VM harness (scripts/run-perf-tests.sh / .ps1) — installs the SDK, creates the perf DB, injects the VM connection string, pins the client to a reserved CPU set, and runs the benchmarks.
• Two-pass build model — current build via ProjectReference (branch under test) vs. baseline via PackageReference (released MDS from NuGet.org), pinned with VersionOverride under Central Package Management.
• Noise reduction — interleaved per-unit baseline↔candidate runs + best-of-N confirmation (interleave_perf.py), client CPU pinning, warm-up, allocator/network tuning on Linux, fail-loud guards, and diagnostics capture.
• Comparison (compare_perf.py) — matches benchmarks by (Type, Method, Parameters), emits comparison.md/comparison.json, and can gate the run on confirmed regressions.
• Kusto ingestion (perf_to_kusto.py, ingest_kusto.py) — translates BenchmarkDotNet JSON into PerfRun + PerfBenchmarkResult NDJSON and performs a queued ingestion. ADX cluster/database/service-connection are sourced from a pipeline library variable group (ADX Cluster Variables), not hard-coded.
• Runner harness changes — PerformanceTests/Program.cs gains PERF_LIST_BENCHMARKS / PERF_BENCHMARK modes to support single-unit interleaving; BenchmarkConfig.cs enables the JSON "full" exporter.

Notes

• The pipeline runs on internal perf-lab infrastructure and consumes an internal extends template; it is manual/scheduled only (pr: none, trigger: none) and does not run on public PRs or commits.
• Ingestion is optional and gated at runtime — the pipeline still runs and compares when the ADX variables are empty, publishing the translated NDJSON as an artifact for manual/backfill ingestion.

Checklist

• Tests added or updated (perf harness/runner)
• Public API changes documented (none — no public API changes)
• Verified against customer repro (N/A)
• Ensure no breaking changes introduced

[Copilot]

Pull request overview

Adds an internal Azure DevOps performance-testing pipeline for Microsoft.Data.SqlClient, including on-VM harness scripts to run BenchmarkDotNet in a noise-reduced baseline-vs-candidate model and (optionally) translate/ingest results into ADX (Kusto). This fits the repo’s existing eng/pipelines/ infrastructure and extends the existing perf test project to support single-unit execution needed for interleaving.

Changes:

• Adds a new perf pipeline (eng/pipelines/perf/sqlclient-perf-pipeline.yml) that provisions a perf-lab VM, runs benchmarks, publishes results, and optionally ingests translated NDJSON into Kusto.
• Adds Linux/Windows VM harness scripts plus Python utilities for interleaving (best-of-N confirmation), baseline comparison, and Kusto translation/ingestion.
• Updates the perf test runner to support enumerating/running individual benchmark “units”, and ensures BenchmarkDotNet “full” JSON reports are emitted.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Program.cs	Adds PERF_LIST_BENCHMARKS / PERF_BENCHMARK modes to enable interleaved per-unit runs.
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Microsoft.Data.SqlClient.PerformanceTests.csproj	Enables baseline pinning in Package mode via MdsPackageVersion + VersionOverride under CPM.
src/Microsoft.Data.SqlClient/tests/PerformanceTests/Config/BenchmarkConfig.cs	Enables BenchmarkDotNet JSON “full” exporter needed for comparison/translation.
eng/pipelines/perf/sqlclient-perf-pipeline.yml	New ADO perf pipeline extending internal perf-lab template; publishes artifacts and optionally ingests to Kusto.
eng/pipelines/perf/scripts/run-perf-tests.sh	Linux on-VM harness: install SDK, create DB, run interleaved/sequential passes, compare results.
eng/pipelines/perf/scripts/run-perf-tests.ps1	Windows on-VM harness counterpart (affinity pinning via ProcessorAffinity).
eng/pipelines/perf/scripts/interleave_perf.py	Interleaved + best-of-N orchestrator invoking single-unit runner mode and producing comparison outputs.
eng/pipelines/perf/scripts/compare_perf.py	Baseline vs current comparison over *-report-full.json with markdown/JSON outputs.
eng/pipelines/perf/scripts/perf_to_kusto.py	Translates BenchmarkDotNet “full” JSON into Kusto PerfRun/PerfBenchmarkResult NDJSON.
eng/pipelines/perf/scripts/ingest_kusto.py	Queued ingestion into ADX using az-cli auth + post-ingest verification polling.
eng/pipelines/perf/README.md	Documents pipeline architecture, parameters, noise-reduction controls, and ingestion setup.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

eng/pipelines/perf/scripts/run-perf-tests.sh:167

• The “reuse pre-installed SDK” check doesn’t actually verify the pinned SDK version from global.json. It only checks for any 10.0.* SDK, which could pick an incompatible feature band/patch and cause builds/restores to fail (global.json pins 10.0.300). Also, DOTNET_ROOT is derived from the dotnet shim path; if dotnet is a symlink in /usr/bin this can point to the wrong root.

# Reuse a pre-installed SDK only if it already satisfies global.json; otherwise install locally.
if command -v dotnet >/dev/null 2>&1 && dotnet --list-sdks 2>/dev/null | grep -q '10\.0\.'; then
    echo "Using pre-installed dotnet: $(command -v dotnet)"
    export DOTNET_ROOT="$(dirname -- "$(command -v dotnet)")"
else

eng/pipelines/perf/scripts/run-perf-tests.ps1:127

• The pre-installed dotnet detection checks only for any 10.0.* SDK, but global.json pins a specific SDK version (10.0.300). If the VM image has a different 10.0 SDK, builds can fail despite this check passing. Prefer verifying the exact pinned version (or at least the pinned feature band) before skipping Install-DotNet.

$hasNet10Sdk = $false
if (Get-Command dotnet -ErrorAction SilentlyContinue) {
    if ((dotnet --list-sdks) -match '^10\.0\.') { $hasNet10Sdk = $true }
}
if ($hasNet10Sdk) {
    Write-Host "Using pre-installed dotnet: $((Get-Command dotnet).Source)"
} else {
    Install-DotNet
}

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

eng/pipelines/perf/scripts/run-perf-tests.sh:167

• The preinstalled-SDK check is too permissive and can skip installing the SDK required by global.json (e.g., a host with 10.0.1xx will satisfy grep '10\.0\.' but dotnet build will later fail because global.json requires 10.0.3xx). Also, setting DOTNET_ROOT to the directory containing the dotnet shim (often /usr/bin) is incorrect and can break host resolution.

Consider checking for a compatible feature band derived from global.json and avoiding DOTNET_ROOT overrides when using the system installation.

# Reuse a pre-installed SDK only if it already satisfies global.json; otherwise install locally.
if command -v dotnet >/dev/null 2>&1 && dotnet --list-sdks 2>/dev/null | grep -q '10\.0\.'; then
    echo "Using pre-installed dotnet: $(command -v dotnet)"
    export DOTNET_ROOT="$(dirname -- "$(command -v dotnet)")"
else

eng/pipelines/perf/scripts/run-perf-tests.ps1:127

• The script decides whether to install the SDK based on a generic ^10\.0\. check, which can accept a preinstalled SDK that doesn't satisfy global.json (e.g., 10.0.1xx vs required 10.0.3xx). This can cause the subsequent build to fail.

Consider deriving the required SDK feature band from global.json (rollForward=patch ⇒ same feature band) and checking dotnet --list-sdks against that instead.

$hasNet10Sdk = $false
if (Get-Command dotnet -ErrorAction SilentlyContinue) {
    if ((dotnet --list-sdks) -match '^10\.0\.') { $hasNet10Sdk = $true }
}
if ($hasNet10Sdk) {
    Write-Host "Using pre-installed dotnet: $((Get-Command dotnet).Source)"
} else {
    Install-DotNet
}

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

eng/pipelines/perf/sqlclient-perf-pipeline.yml:308

• The Kusto ingestion step is gated only on KustoClusterUri and KustoServiceConnection. If those are set but KustoDatabase is missing/empty in the variable group, the ingest_kusto.py --database argument becomes empty and the step will fail after running pip installs. Consider gating on KustoDatabase being non-empty as well.

      # provisioned.
      - task: AzureCLI@2
        displayName: 'Ingest results into Kusto'
        condition: and(succeededOrFailed(), ne(variables['KustoClusterUri'], ''), ne(variables['KustoDatabase'], ''), ne(variables['KustoServiceConnection'], ''))

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

eng/pipelines/perf/scripts/interleave_perf.py:372

• After removing the unused fail_on_regression parameter from orchestrate(), update this call site to stop passing args.fail_on_regression. The regression gate is already applied after orchestrate() returns.

    entries, confirmed, unconfirmed = orchestrate(
        runner, units, results_dir, args.threshold, args.reps, args.fail_on_regression)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.56%. Comparing base (d040700) to head (6406866).

❗ There is a different number of reports uploaded between BASE (d040700) and HEAD (6406866). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (d040700)	HEAD (6406866)
CI-SqlClient	1	0

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4473      +/-   ##
==========================================
- Coverage   70.88%   63.56%   -7.32%     
==========================================
  Files         288      283       -5     
  Lines       43935    66821   +22886     
==========================================
+ Hits        31142    42477   +11335     
- Misses      12793    24344   +11551     

Flag	Coverage Δ
CI-SqlClient	?
PR-SqlClient-Project	63.56% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.