Migrate to xunit.v3

[Youssef1313]

No description provided.

[rainersigwald]

fyi I'm working on this in #13215, but won't be mad if you beat me to it. Chasing what looks like some Windows hangs at the moment, and as a low-pri background thing.

[Youssef1313]

Oh, I didn't know you are already working on it.

[Youssef1313]

@rainersigwald I tried to pull in some changes you did in the other PR here to see if it gets me here in a better state. We can sync next Tuesday to align the efforts.

Generally speaking, a big difference between the two PRs is that in this PR I'm running with MTP, while the other PR runs with VSTest. But I would think most test failures will be related to the xunit 2 to xunit 3 move and not VSTest/MTP differences. In addition, in this PR I'm keeping the tests wrapped in TestEnvironment.Create() and TestEnvironment.Dispose() to assert that the tests don't change env variables.

One thing to note is that xunit.v3 might be running the tests in a different order compared to xunit 2, which might (or might not) be the cause for some of the failures (i.e, some tests might have been modifying global state without resetting it - that was somehow unnoticed with xunit 2 but might become apparent now that the ordering changed)

[rainersigwald]

In addition, in this PR I'm keeping the tests wrapped in TestEnvironment.Create() and TestEnvironment.Dispose() to assert that the tests don't change env variables.

I believe this is not currently working--my interpretation of a bunch of failures in my attempt was "something is making this work now and failing a bunch of tests that should have been failing before but weren't". I'd love to have all those tests fixed of course . . .

[Youssef1313]

In addition, in this PR I'm keeping the tests wrapped in TestEnvironment.Create() and TestEnvironment.Dispose() to assert that the tests don't change env variables.

I believe this is not currently working--my interpretation of a bunch of failures in my attempt was "something is making this work now and failing a bunch of tests that should have been failing before but weren't". I'd love to have all those tests fixed of course . . .

In the other PR, I commented on why the approach there won't work. But I think it might have been working on main already.

[Copilot]

Pull request overview

This PR migrates the repo’s unit test infrastructure from xUnit v2 patterns to xUnit v3, aligning execution with Microsoft.Testing.Platform and updating shared test bootstrapping so test environment initialization remains consistent across the suite.

Changes:

• Switched test dependencies to xUnit v3 (and Verify.XunitV3), removed xUnit v2 console runner usage, and updated test runner/coverage configuration.
• Replaced the custom xUnit v2 “AssemblyFixtureSupport” framework with xUnit v3 pipeline startup + a custom test framework wrapper to create TestEnvironment per test case.
• Updated many test projects/files to remove Xunit.Abstractions usages and stabilize a few test-global/static state interactions.

Reviewed changes

Copilot reviewed 160 out of 160 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/Xunit.NetCore.Extensions/Xunit.NetCore.Extensions.csproj	Switch extension package reference to xUnit v3 extensibility core and adjust TFMs/settings.
src/Xunit.NetCore.Extensions/UseInvariantCultureAttribute.cs	Update BeforeAfterTestAttribute overrides to xUnit v3 signatures.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestMethodRunnerWithAssemblyFixture.cs	Removed legacy xUnit v2 assembly fixture runner implementation.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestFrameworkWithAssemblyFixture.cs	Removed legacy xUnit v2 custom test framework.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestFrameworkExecutorWithAssemblyFixture.cs	Removed legacy xUnit v2 executor wrapper.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestCollectionRunnerWithAssemblyFixture.cs	Removed legacy xUnit v2 collection runner wrapper.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestClassRunnerWithAssemblyFixture.cs	Removed legacy xUnit v2 class runner wrapper.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/XunitTestAssemblyRunnerWithAssemblyFixture.cs	Removed legacy xUnit v2 assembly runner wrapper.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/README.md	Removed docs for the deleted xUnit v2 assembly fixture support.
src/Xunit.NetCore.Extensions/AssemblyFixtureSupport/AssemblyFixtureAttribute.cs	Removed legacy assembly-level fixture attribute.
src/Utilities.UnitTests/ToolTask_Tests.cs	Remove xUnit v2 abstractions import.
src/Utilities.UnitTests/ToolLocationHelper_Tests.cs	Remove xUnit v2 abstractions import.
src/Utilities.UnitTests/ProcessExtensions_Tests.cs	Remove xUnit v2 abstractions import.
src/UnitTests.Shared/TestEnvironment.cs	Update default output helper implementation for xUnit v3 interface surface.
src/UnitTests.Shared/RunnerUtilities.cs	Update xUnit output helper import namespace.
src/UnitTests.Shared/ObjectModelHelpers.cs	Remove xUnit v2 abstractions import.
src/UnitTests.Shared/MockLogger.cs	Remove xUnit v2 abstractions import.
src/UnitTests.Shared/MockEngine.cs	Update xUnit output helper import namespace.
src/UnitTests.Shared/Microsoft.Build.UnitTests.Shared.csproj	Replace xUnit v2 packages with xUnit v3 assert/extensibility packages.
src/Tasks.UnitTests/XamlDataDrivenToolTask_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/WriteLinesToFile_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/VerifyFileHash_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/Unzip_Tests.cs	Replace conditional attribute usage with runtime skip under xUnit v3.
src/Tasks.UnitTests/TestResources/TestBinary.cs	Update xUnit-related imports.
src/Tasks.UnitTests/RoslynCodeTaskFactory_Tests.cs	Remove Verify-specific attribute usage.
src/Tasks.UnitTests/ResourceHandling/ResGenDependencies_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/ResourceHandling/MSBuildResXReader_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/ResourceHandling/GenerateResourceOutOfProc_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/ResourceHandling/GenerateResource_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/ResolveNonMSBuildProjectOutput_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/ResolveAssemblyReference_CustomCultureTests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/RemoveDir_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/RegressionTests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/PortableTasks_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/OutputPathTests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/MSBuildInternalMessage_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/MSBuild_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/Microsoft.Build.Tasks.UnitTests.csproj	Switch Verify package to xUnit v3 variant.
src/Tasks.UnitTests/HintPathResolver_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/GetSDKReference_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/GetFileHash_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/GetCompatiblePlatform_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/GetAssembliesMetadata_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/GenerateBindingRedirects_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/FormatUrl_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/Exec_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/CreateVisualBasicManifestResourceName_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/CreateItem_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/CreateCSharpManifestResourceName_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/Copy_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssignProjectConfiguration_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/WinMDTests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/VerifyTargetFrameworkHigherThanRedist.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/VerifyTargetFrameworkAttribute.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/VerifyIgnoreVersionForFrameworkReference.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/SuggestedRedirects.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependencyAutoUnify.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependencyAppConfig.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/StronglyNamedDependency.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/SpecificVersionPrimary.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/ResolveAssemblyReferenceTestFixture.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/ReferenceTests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/Perf.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/NonSpecificVersionStrictPrimary.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/Node/OutOfProcRarNode_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/Miscellaneous.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/InstalledSDKResolverFixture.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/GlobalAssemblyCacheTests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/FilePrimary.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AssemblyDependency/AssemblyFoldersFromConfig_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/Al_Tests.cs	Remove xUnit v2 abstractions import.
src/Tasks.UnitTests/AddToWin32Manifest_Tests.cs	Remove xUnit v2 abstractions import.
src/StringTools.UnitTests/StringTools.UnitTests.net35.csproj	Force OutputType Exe for net35 test project under new runner approach.
src/StringTools.UnitTests/InterningTestData.cs	Update xUnit-related imports.
src/Shared/UnitTests/xunit.runner.json	Remove xUnit v2 runner settings (shadow copy/app domain).
src/Shared/UnitTests/TypeLoader_Tests.cs	Remove xUnit v2 abstractions import.
src/Shared/UnitTests/TestProgram.cs	Removed custom xUnit v2 console entrypoint program.
src/Shared/UnitTests/TestAssemblyInfo.cs	Introduce xUnit v3 pipeline startup + custom test framework wrapper for TestEnvironment.
src/MSBuild.UnitTests/XMake_Tests.cs	Remove unused test platform + xUnit v2 abstractions imports.
src/MSBuild.UnitTests/XMake_BinlogSwitch_Tests.cs	Remove xUnit v2 abstractions import.
src/MSBuild.UnitTests/PerfLog_Tests.cs	Remove xUnit v2 abstractions import.
src/MSBuild.UnitTests/MSBuildServer_Tests.cs	Remove xUnit v2 abstractions import.
src/MSBuild.UnitTests/MSBuildMultithreaded_Tests.cs	Remove xUnit v2 abstractions import.
src/MSBuild.UnitTests/Microsoft.Build.CommandLine.UnitTests.csproj	Switch Verify package to xUnit v3 variant.
src/MSBuild.UnitTests/CommandLineSwitches_Tests.cs	Stabilize change-wave state setup for a test.
src/Framework.UnitTests/FileUtilities_Tests.cs	Replace conditional fact usage with runtime skip under xUnit v3.
src/Framework.UnitTests/FileMatcher_Tests.cs	Remove xUnit v2 abstractions import.
src/Directory.Build.targets	Update runner args/trait filtering, add xUnit v3 extensions + MTP coverage integration, remove xUnit console runner wiring.
src/BuildCheck.UnitTests/TaskInvocationAnalysisDataTests.cs	Remove xUnit v2 abstractions import.
src/BuildCheck.UnitTests/EndToEndTests.cs	Remove xUnit v2 abstractions import.
src/BuildCheck.UnitTests/BuildCheckManagerProviderTests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/WarningsAsMessagesAndErrors_Tests.cs	Adjust assembly location resolution logic for task loading.
src/Build.UnitTests/Utilities_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/TerminalLoggerConfiguration_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/TerminalLogger_Tests.cs	Remove Verify-specific attribute usage and xUnit v2 abstractions import.
src/Build.UnitTests/Telemetry/Telemetry_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/TaskHostFactoryLifecycle_E2E_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Resources_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/ProjectCache/ProjectCacheTests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/NodeStatus_Transition_Tests.cs	Remove Verify-specific attribute usage.
src/Build.UnitTests/NodeStatus_SizeChange_Tests.cs	Remove Verify-specific attribute usage.
src/Build.UnitTests/NetTaskHost_E2E_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/MSBuildTaskHostTests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Microsoft.Build.Engine.UnitTests.csproj	Switch Verify package to xUnit v3 variant.
src/Build.UnitTests/InvalidProjectFileException_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Instance/ProjectInstance_Internal_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Graph/ResultCacheBasedBuilds_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Graph/ProjectGraph_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Graph/IsolateProjects_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Graph/GraphLoadedFromSolution_tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/ExpressionTreeExpression_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/EvaluationProfiler_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Evaluation/SdkResultEvaluation_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Evaluation/ExpanderFunction_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Evaluation/Expander_Tests.cs	Remove xUnit v2 abstractions import; reset change-wave state for stability.
src/Build.UnitTests/EscapingInProjects_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/EndToEndCondition_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Construction/SolutionProjectGenerator_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Construction/SolutionFilter_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Construction/SolutionFile_OldParser_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Construction/SolutionFile_NewParser_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/Construction/ProjectRootElement_Tests.cs	Clear cached read-only flags to prevent cross-test pollution in same process.
src/Build.UnitTests/ConsoleLogger_Tests.cs	Ensure Traits.LogAllEnvironmentVariables is restored after tests.
src/Build.UnitTests/CLR2TaskHost_E2E_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/ChangeWaves_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BinaryLogger_Tests.cs	Stabilize test behavior around Traits.Instance state and environment variables.
src/Build.UnitTests/BackEnd/TaskRouter_IntegrationTests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/TaskRegistry_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/TaskHostFactory_Tests.cs	Adjust assembly location resolution logic for <UsingTask AssemblyFile=...>.
src/Build.UnitTests/BackEnd/TaskHostCallback_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/TaskHost_MultiThreadableTask_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/TaskBuilder_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/TargetUpToDateChecker_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/TargetEntry_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/SdkResultOutOfProc_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/SdkResolverLoader_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/MSBuild_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/LoggingContext_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/DebugUtils_tests.cs	Avoid Directory.GetFiles when debug dump dir doesn’t exist.
src/Build.UnitTests/BackEnd/BuildRequestConfiguration_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/BuildManager_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/BuildManager_Logging_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/AssemblyTaskFactory_E2E_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.UnitTests/BackEnd/AppHostSupport_Tests.cs	Ensure DOTNET_ROOT_X86 is tracked for restoration in tests.
src/Build.OM.UnitTests/NugetRestoreTests.cs	Remove xUnit v2 abstractions import.
src/Build.OM.UnitTests/Instance/ProjectInstance_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.OM.UnitTests/Definition/ProjectCollection_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.OM.UnitTests/Definition/Project_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.OM.UnitTests/Construction/ProjectMetadataElement_Tests.cs	Remove xUnit v2 abstractions import.
src/Build.OM.UnitTests/Construction/ProjectFormatting_Tests.cs	Remove xUnit v2 abstractions import.
global.json	Configure test runner as Microsoft.Testing.Platform.
eng/Version.Details.xml	Rename Arcade dependency to Microsoft.DotNet.XUnitV3Extensions.
eng/Version.Details.props	Update Arcade xUnit v3 extensions version properties.
eng/dependabot/Directory.Packages.props	Add MTP coverage package; switch Verify dependency to Verify.XunitV3.
Directory.Packages.props	Switch xUnit extensions package and remove xunit.console version pin.
Directory.Build.props	Add xUnit v3/MTP-related properties (TestRunnerName, versions, OutputType for test projects, analyzer suppression).
CoverageWindowsFull.runsettings	Removed legacy VS CodeCoverage runsettings.
CoverageWindowsFull.config	Added new coverage configuration format for MTP coverage integration.
Coverage.runsettings	Removed legacy non-Windows runsettings.
Coverage.config	Added new non-Windows coverage configuration format.

[JanProvaznik]

/review

[github-actions]

✅ Expert Code Review (command) completed successfully!

[github-actions]

Additional findings from deeper static analysis of the xUnit v3 migration (TestAssemblyInfo.cs):

🔴 TestEnvironment.Dispose() exceptions lost — not captured by the aggregator

In the new MSBuildTestCase.Run:

using var _ = TestEnvironment.Create();
return await XunitRunnerHelper.RunXunitTestCase(this, ...);

TestEnvironment.Dispose() calls AssertInvariant which can throw (via Assert.True invariant checks). In xUnit v3's ISelfExecutingXunitTestCase pattern, exceptions from cleanup must flow through the ExceptionAggregator so they are reported as test failures rather than crashing the test runner. The old MSBuildTestEnvironmentFixture went through xUnit's standard fixture teardown which correctly routes disposal exceptions through the aggregator.

Fix: replace the using with an explicit try/finally that calls aggregator.Run(env.Dispose):

var env = TestEnvironment.Create();
try
{
    return await XunitRunnerHelper.RunXunitTestCase(
        this, messageBus, cancellationTokenSource,
        aggregator.Clone(), explicitOption, constructorArguments);
}
finally
{
    aggregator.Run(env.Dispose);
}

🔴 MSBuildTestPipelineStartup.StopAsync() — NRE if constructor throws mid-way

If the constructor of MSBuildTestPipelineStartup throws before _testEnvironment is assigned, StopAsync() will NullReferenceException on _testEnvironment.Dispose(). Change to:

_testEnvironment?.Dispose();

🟡 Reflection calls with no null guards

testInfoType.GetField("s_runningTests", ...) returns null if the field is moved or renamed. The subsequent SetValue call will throw a cryptic NRE rather than a diagnostic message. Adding a null check with a clear exception message would make future failures diagnosable.

Generated by Expert Code Review (command) for issue #13482 · ● 25.9M · ◷