How dotnet test works?
January 15, 2017
This post walks through the internals of
dotnet test command. We will look at
bits and pieces of code on the way.
A command line toolset
The dotnet command line tools (dotnet-cli) are a collection of command line
utilities, each named after the invocation verb. E.g.
new verb is handled by a
dotnet-new. The test run is handled by an utility called
With the move to csproj based projects, an operation that requires knowledge of the project properties depends on msbuild to find them. The test entrypoint does invoke msbuild to ensure project related information like the output assembly etc..
A word on the scenarios…
In the test operations, there are two key user motivations:
- During development, user may want to run the tests from the project. This requires an understanding of target frameworks, bitness, dependencies etc..
- In a CDP scenario, user may want to directly run the output binary since the source code may not be available on the deployment agent box. Here user provides the test binary directly as input to the runner. All dependencies are available in the published output directory.
The development inner loop scenario is triggered from an IDE/Editor and command
dotnet test supports the inner loop, where as
dotnet vstest is the
entrypoint for advanced scenarios (like IDE/Editor integration).
dotnet-test takes a project file as input (
testapp.csproj) to run the
tests. It does some prework to compute the outputs of a project, and then
invokes the test runner. Let’s look at the flow.
dotnet test evaluates input options and invokes
Let’s find the msbuild invocation from
:::bash $ dotnet test --list-tests --no-build -v:diag | head -n 1 # Invokes msbuild with args translated to Properties for VSTest target $ ~/dotnet/sdk/1.0.0-preview5-004262/MSBuild.dll.exe /nologo /Logger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,~/dotnet/sdk/1.0.0-preview5-004262/dotnet.dll /m /p:VSTestListTests=true /p:VSTestNoBuild=true /t:VSTest /v:m /verbosity:diag ~/tmp/samples/testapp/testapp.csproj
The interesting properties in this case are:
2. The VSTest target
VSTest target tries to build the test project if required, calls
VSTest task with evaluated parameters. It adds a few project context
specific parameters like the path to test assembly.
Below is the code excerpt.
:::xml <!-- ============================================================ Test target Main entry point for running tests through vstest.console.exe ============================================================ --> <Target Name="VSTest" > <CallTarget Condition="'$(VSTestNoBuild)' != 'true'" Targets="BuildProject" /> <CallTarget Targets="ShowCallOfVSTestTaskWithParameter" /> <Microsoft.TestPlatform.Build.Tasks.VSTestTask TestFileFullPath="$(TargetPath)" VSTestSetting="$(VSTestSetting)" VSTestTestAdapterPath="$(VSTestTestAdapterPath)" VSTestFramework="$(TargetFrameworkMoniker)" VSTestPlatform="$(PlatformTarget)" VSTestTestCaseFilter="$(VSTestTestCaseFilter)" VSTestLogger="$(VSTestLogger)" VSTestListTests="$(VSTestListTests)" VSTestDiag="$(VSTestDiag)" /> </Target>
VSTest task is defined in
Microsoft.TestPlatform.Build nuget package and
is packages as
3. The VSTest task
VSTest task parses the parameters, runs
vstest.console runner with test
assembly. It listens to standard output/error of the console runner and relays
it back to msbuild.
Let’s use diagnostic trace to find what’s going on!
:::bash $ export VSTEST_BUILD_TRACE=1 # enable tracing for VSTest task $ dotnet test --list-tests --no-build Test run for /home/codito/tmp/samples/mstesttest/bin/Debug/netcoreapp1.0/mstesttest.dll(.NETCoreApp,Version=v1.0) VSTest: Starting vstest.console... VSTest: Arguments: dotnet exec\ "~/dotnet/sdk/1.0.0-preview5-004262/vstest.console.dll"\ --framework:".NETCoreApp,Version=v1.0"\ --listTests\ "~/tmp/samples/testapp/bin/Debug/netcoreapp1.0/testapp.dll" Microsoft (R) Test Execution Command Line Tool Version 22.214.171.124 Copyright (c) Microsoft Corporation. All rights reserved. The following Tests are available: PassingTest FailingTest VSTest: Exit code: 0
The output explains it all :)
We’ve one last piece of the build puzzle left: multitargeted test projects.
In case of multitargeted test project, msbuild evaluates the
array and calls
VSTest target once for each
invokes the console runner (
vstest.console) for each.
This file is packaged as
4. Inside Microsoft.NET.Test.Sdk
There are three components that come together to run a test with test platform:
- A test framework. E.g. mstest, xunit, nunit etc.
- An adapter which can identify and execute tests written with the framework
- A test host that finds all adapters, and orchestrates the run
In the earlier versions of test platform, test framework and adapter could be
provided by nuget packages. The test host was an integral part of the test
platform. Currently the test host comes with
package. A test project carries everything required to run a test with itself.
In addition to bringing in test host reference in dependency closure, this package also provides build time support for test project to hide some complexities. E.g. a target that auto generates the program entrypoint
::: xml <Target Name="GenerateProgramFile" BeforeTargets="CoreCompile" DependsOnTargets="PrepareForBuild;CoreGenerateProgramFile" Condition="'$(GenerateProgramFile)' == 'true'" />
GenerateProgramFile target runs before the
CoreCompile target. The
generated program file lives in the intermediate output directory and is passed
to the compiler. That saves every test project from having a program entrypoint.
That’s all for now :) This platform is now open source. If you have any feedback, please create an issue at vstest repository!