Inside Out | How dotnet test works?

This post walks through the internals of dotnet test command. We will look at bits and pieces of code on the way.

A command line toolset

The dotnet command line tools (dotnet-cli) are a collection of command line utilities, each named after the invocation verb. E.g. new verb is handled by a tool named dotnet-new. The test run is handled by an utility called dotnet-test↗.

With the move↗ to csproj based projects, an operation that requires knowledge of the project properties depends on msbuild to find them. The test entrypoint does invoke msbuild to ensure project related information like the output assembly etc..

A word on the scenarios…

In the test operations, there are two key user motivations:

During development, user may want to run the tests from the project. This requires an understanding of target frameworks, bitness, dependencies etc..
In a CDP scenario, user may want to directly run the output binary since the source code may not be available on the deployment agent box. Here user provides the test binary directly as input to the runner. All dependencies are available in the published output directory.

The development inner loop scenario is triggered from an IDE/Editor and command line. dotnet test supports the inner loop, where as dotnet vstest is the entrypoint for advanced scenarios (like IDE/Editor integration).

dotnet-test flow

The dotnet-test takes a project file as input (testapp.csproj) to run the tests. It does some prework to compute the outputs of a project, and then invokes the test runner. Let’s look at the flow.

dotnet test flow

1. `dotnet test` evaluates input options and invokes `msbuild /t:VSTest`.

Source: dotnet-cli/src/dotnet/commands/dotnet-test↗

Let’s find the msbuild invocation from dotnet test.

:::bash
$ dotnet test --list-tests --no-build -v:diag | head -n 1

# Invokes msbuild with args translated to Properties for VSTest target
$ ~/dotnet/sdk/1.0.0-preview5-004262/MSBuild.dll.exe /nologo
/Logger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,~/dotnet/sdk/1.0.0-preview5-004262/dotnet.dll
/m /p:VSTestListTests=true /p:VSTestNoBuild=true /t:VSTest /v:m
/verbosity:diag
~/tmp/samples/testapp/testapp.csproj

The interesting properties in this case are:

--list-tests translated to /p:VSTestListTests=true
--no-build translated to /p:VSTestNoBuild=true

2. The VSTest target

Source: vstest/src/Microsoft.TestPlatform.Build/Microsoft.TestPlatform.targets↗

VSTest target tries to build the test project if required, calls VSTest task with evaluated parameters. It adds a few project context specific parameters like the path to test assembly.

Below is the code excerpt.

:::xml
<!--                                                                                                                                       
   ============================================================
   Test target
   Main entry point for running tests through vstest.console.exe
   ============================================================
-->
<Target Name="VSTest" >
  <CallTarget Condition="'$(VSTestNoBuild)' != 'true'" Targets="BuildProject" />
  
  <CallTarget Targets="ShowCallOfVSTestTaskWithParameter" />

  <Microsoft.TestPlatform.Build.Tasks.VSTestTask
    TestFileFullPath="$(TargetPath)"
    VSTestSetting="$(VSTestSetting)"
    VSTestTestAdapterPath="$(VSTestTestAdapterPath)"
    VSTestFramework="$(TargetFrameworkMoniker)"
    VSTestPlatform="$(PlatformTarget)"
    VSTestTestCaseFilter="$(VSTestTestCaseFilter)"
    VSTestLogger="$(VSTestLogger)"
    VSTestListTests="$(VSTestListTests)"
    VSTestDiag="$(VSTestDiag)"
  />
</Target>

The VSTest task is defined in Microsoft.TestPlatform.Build nuget package and is packages as ~/dotnet/sdk/1.0.0-preview5-004262/Microsoft.TestPlatform.Build.dll in dotnet-cli.

3. The VSTest task

Source: vstest/src/Microsoft.TestPlatform.Build/Tasks/VSTestTask.cs↗

VSTest task parses the parameters, runs vstest.console runner with test assembly. It listens to standard output/error of the console runner and relays it back to msbuild.

Let’s use diagnostic trace to find what’s going on!

:::bash
$ export VSTEST_BUILD_TRACE=1    # enable tracing for VSTest task
$ dotnet test --list-tests --no-build
Test run for /home/codito/tmp/samples/mstesttest/bin/Debug/netcoreapp1.0/mstesttest.dll(.NETCoreApp,Version=v1.0)
VSTest: Starting vstest.console...
VSTest: Arguments: dotnet exec\
"~/dotnet/sdk/1.0.0-preview5-004262/vstest.console.dll"\
--framework:".NETCoreApp,Version=v1.0"\
--listTests\
"~/tmp/samples/testapp/bin/Debug/netcoreapp1.0/testapp.dll"
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

The following Tests are available:
    PassingTest
    FailingTest

VSTest: Exit code: 0

The output explains it all :)

We’ve one last piece of the build puzzle left: multitargeted test projects.

Source: dotnet-cli/src/resources/MSBuildImports/15.0/Microsoft.Common.CrossTargeting.targets/ImportAfter/Microsoft.TestPlatform.CrossTargeting.targets↗

In case of multitargeted test project, msbuild evaluates the TargetFrameworks array and calls VSTest target once for each TargetFramework. VSTest target invokes the console runner (vstest.console) for each.

This file is packaged as ~/dotnet/sdk/1.0.0-preview5-004262/15.0/Microsoft.Common.CrossTargeting.targets/ImportAfter/Microsoft.TestPlatform.CrossTargeting.targets

4. Inside Microsoft.NET.Test.Sdk

There are three components that come together to run a test with test platform:

A test framework. E.g. mstest, xunit, nunit etc.
An adapter which can identify and execute tests written with the framework
A test host that finds all adapters, and orchestrates the run

In the earlier versions of test platform, test framework and adapter could be provided by nuget packages. The test host was an integral part of the test platform. Currently the test host comes with Microsoft.NET.Test.Sdk nuget package. A test project carries everything required to run a test with itself.

Source: vstest/src/Microsoft.NET.Test.Sdk.targets↗

In addition to bringing in test host reference in dependency closure, this package also provides build time support for test project to hide some complexities. E.g. a target that auto generates the program entrypoint

::: xml
<Target Name="GenerateProgramFile"
          BeforeTargets="CoreCompile"
          DependsOnTargets="PrepareForBuild;CoreGenerateProgramFile"
          Condition="'$(GenerateProgramFile)' == 'true'" />

The GenerateProgramFile target runs before the CoreCompile target. The generated program file lives in the intermediate output directory and is passed to the compiler. That saves every test project from having a program entrypoint.

That’s all for now :) This platform is now open source. If you have any feedback, please create an issue at vstest↗ repository!

How dotnet test works?