Skip to main content

16 - Testing

Why test

  • Until you execute a line of code, you don't know if that line can work at all.
  • Until you execute your code with a representative set of basic use cases, you don't know if the code can work end-to-end.
  • If you and possibly other people are going to modify your code, it's very easy to break it in unexpected ways. A suite of automated unit and integration tests gives you confidence you've not broken anything significant.

Dif ways to test

  • Functional
    • Unit testing
    • Integration testing
    • System testing
    • Acceptance testing
  • Non-functional
    • Load/Stress testing
    • Security testing
    • Usability testing
    • Compatibility testing

  • Manual/Explorative testing, Black/White box, UI tests, ….

Unit testing

  • A unit test is a test that exercises individual software components or methods, also known as "unit of work".
  • Unit tests should only test code within the developer's control.
  • They do not test infrastructure concerns.
  • Infrastructure concerns include interacting with databases, file systems, and network resources.

Integration testing

  • An integration test differs from a unit test in that it exercises two or more software components' ability to function together, also known as their "integration”.
  • These tests operate on a broader spectrum of the system under test, whereas unit tests focus on individual components.
  • Often, integration tests do include infrastructure concerns.

Load/stress testing

  • A load test aims to determine whether or not a system can handle a specified load, for example, the number of concurrent users using an application and the app's ability to handle interactions responsively.
  • Load tests: Test whether the app can handle a specified load of users for a certain scenario while still satisfying the response goal. The app is run under normal conditions.
  • Stress tests: Test app stability when running under extreme conditions, often for a long period of time. The tests place high user load, either spikes or gradually increasing load, on the app, or they limit the app's computing resources.
  • Stress tests determine if an app under stress can recover from failure and gracefully return to expected behaviour. Under stress, the app isn't run under normal conditions.

Test frameworks

  • xUnit, NUnit, MSTest, ...
  • xUnit is a free, open source, community-focused unit testing tool for .NET. Written by the original inventor of NUnit v2, xUnit.net is the latest technology for unit testing .NET apps.
  • xUnit.net works with ReSharper, CodeRush, TestDriven.NET, and Xamarin

Unit testing

  • A typical unit test contains 3 phases
    • Initializes a small piece of an application it wants to test (also known as the system under test, or SUT).
    • Applies some stimulus to the system under test (usually by calling a method on it)
    • Observes the resulting behavior. If the observed behavior is consistent with the expectations, the unit test passes, otherwise, it fails, indicating that there is a problem somewhere in the system under test.
  • These three unit test phases are also known as Arrange, Act and Assert, or simply AAA.

xUnit vs NUnit

  • We will have a test class and lots of test methods in them

  • NUnit creates instance of the test class and runs all the test methods
  • xUnit creates new instance of the test class for every test method

xUnit

  • Add new project to solution
  • Category: Unit Test Project
  • Type: xUnit

Test

Testing controller with DbContext

  • Simple controller method

  • Dependecies

    • ILogger
    • AppDbContext
public class HomeController : Controller
{
private readonly ILogger<HomeController> _logger;
private readonly AppDbContext _context;


public HomeController(ILogger<HomeController> logger, AppDbContext context)
{
_logger = logger;
_context = context;
}

public async Task<IActionResult> Test()
{
var vm = new TestViewModel
{
ContactTypes = await _context.ContactTypes.ToListAsync()
};
return View(vm);
}

Creating dependecies

  • Unit testing should test code in isolation – independent of database, logging, etc.
  • So we need to create mocked (fake) dependecies, with fixed/controllable behaviour

mocking DbContext

  • DbContext has special engine for testing – InMemory
  • Since it is based on lists, it is not real SQL. LinqToObjects not LinqToSql.
  • Nuget: Microsoft.EntityFrameworkCore.InMemory
// set up mock database - inmemory
var optionsBuilder = new DbContextOptionsBuilder<AppDbContext>();

// use random guid as db instance id
optionsBuilder.UseInMemoryDatabase(Guid.NewGuid().ToString());
_ctx = new AppDbContext(optionsBuilder.Options);

// reset db
_ctx.Database.EnsureDeleted();
_ctx.Database.EnsureCreated();

mocking ILogger

  • We are not testing/asserting logging functionality of our controllers, so this is not really mocked
using var logFactory = LoggerFactory.Create(builder => builder.AddConsole());
var logger = logFactory.CreateLogger<HomeController>();

test class output

xUnit has special console like functionality for output – ITestOutputHelper

_testOutputHelper.WriteLine(result?.ToString());

Test class, constructor

  • Test class setup is done in constructor
  • Arrange phase
public class UnitTestHomeController
{
private readonly ITestOutputHelper _testOutputHelper;
private readonly HomeController _homeController;
private readonly AppDbContext _ctx;

public UnitTestHomeController(ITestOutputHelper testOutputHelper)
{
_testOutputHelper = testOutputHelper;

// set up mock database - inmemory
var optionsBuilder = new DbContextOptionsBuilder<AppDbContext>();
// use random guid as db instance id
optionsBuilder.UseInMemoryDatabase(Guid.NewGuid().ToString());
_ctx = new AppDbContext(optionsBuilder.Options);
// reset db
_ctx.Database.EnsureDeleted();
_ctx.Database.EnsureCreated();

// set up logger - it is not mocked, we are not testing logging functionality
using var logFactory = LoggerFactory.Create(builder => builder.AddConsole());
var logger = logFactory.CreateLogger<HomeController>();

//set up SUT
_homeController = new HomeController(logger, _ctx);
}

xUnit test methods

  • Test methods are declared using attribute [Fact]
  • Test methods will do Act and Assert steps
[Fact]
public async Task TestAction_ReturnsVm()
{
// ACT
var result = await _homeController.Test() as ViewResult;

// ASSERT
Assert.NotNull(result);
Assert.NotNull(result!.Model);
Assert.IsType<TestViewModel>(result.Model);
}

test methods 1

  • More complex behaviour testing – with data
  • Seed test data
private async Task SeedDataAsync()
{
_ctx.ContactTypes.Add(new ContactType()
{
ContactTypeDescription = new AppLangString("Skype"),
ContactTypeValue = new AppLangString("Skype is still used?")
});
await _ctx.SaveChangesAsync();
}

test methods 2

  • Arrange
  • Act
  • Assert
[Fact]
public async Task TestAction_ReturnsVm_WithData()
{
await SeedDataAsync();

// ACT
var result = await _homeController.Test() as ViewResult;

// ASSERT
Assert.NotNull(result);
Assert.NotNull(result!.Model);
Assert.IsType<TestViewModel>(result.Model);
var vm = result.Model as TestViewModel;
Assert.NotNull(vm!.ContactTypes);
Assert.Equal(1, vm!.ContactTypes!.Count);
}

assertion methods

  • Equal, NotEqual
  • Null, NotNull, IsType<SomeType>, IsNotType<>, Same, NotSame
  • StartsWith, EndsWith
  • Matches, DoesNotMatch – regex
  • True, False
  • Contains, DoesNotContain
Assert.Contains("text", "This is a text.");
var names = new List<string> { "Picard", "Kirk" };
Assert.Contains(names, n => n == "Kirk");
  • Empty, NotEmpty
  • InRange, NotInRange

asserting exceptions

No asserts for "no exception"

// exact exception
Assert.Throws<NullReferenceException>(() => new NullReferenceException());
// exact or derived exception
Assert.ThrowsAny<Exception>(() => new ApplicationException());

third party tools

  • FluentAssertions – makes assertions read like human language

  • Nuget: FluentAssertions

  • Shouldly, others

  • using FluentAssertions;

[Fact]
public async Task TestAction_ReturnsVm_WithData_Fluent()
{
await SeedDataAsync();

// ACT
var result = await _homeController.Test() as ViewResult;

result.Should().NotBeNull();
result!.Model.Should().NotBeNull();
result.Model.Should().BeOfType<TestViewModel>();
(result.Model as TestViewModel)!
.ContactTypes.Should().NotBeNull().And.HaveCount(1);
}

parameterised test

  • [Fact] – parameterless unit test, tests invariants of code

  • [Theory] – parameterised unit test

  • Supply parameters with [InlineData(param1, param2, …)]

[Theory]
[InlineData(1)]
[InlineData(10)]
public async Task TestAction_ReturnsVm_WithDataCounts_Fluent(int count)
{
await SeedDataAsync(count);

// ACT
var result = await _homeController.Test() as ViewResult;

result.Should().NotBeNull();
result!.Model.Should().NotBeNull();
result.Model.Should().BeOfType<TestViewModel>();
(result.Model as TestViewModel)!.ContactTypes.Should().NotBeNull().And.HaveCount(count);
}

Generate test data

public class TestDataGenerator : IEnumerable<object[]>
{
private readonly List<object[]> _data = new()
{
new object[] {5},
new object[] {7},
new object[] {9},
};
public IEnumerator<object[]> GetEnumerator() => _data.GetEnumerator();
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

Use generated data

Supply parameters via generator

[Theory]
[ClassData(typeof(TestDataGenerator))]
public async Task TestAction_ReturnsVm_WithDataCounts_Fluent(int count)
{
await SeedDataAsync(count);

// ACT
var result = await _homeController.Test() as ViewResult;

result.Should().NotBeNull();
result!.Model.Should().NotBeNull();
result.Model.Should().BeOfType<TestViewModel>();
(result.Model as TestViewModel)!.ContactTypes.Should().NotBeNull().And.HaveCount(count);
}

Test Driven Development

  • Software development approach in which test cases are developed to specify and validate what the code will do.
    Test cases for each functionality are created and tested first and if the test fails then the new code is written in order to pass the test and making code simple and bug-free.
  • Philosophical problems – tdd is almost impossible while you are learning new language/framework.

TDD

TDD

  • Early bug notification.
    Developers test their code but in the database world, this often consists of manual tests or one-off scripts. Using TDD you build up, over time, a suite of automated tests that you and any other developer can rerun at will.
  • Better Designed, cleaner and more extensible code.
    It helps to understand how the code will be used and how it interacts with other modules. It results in better design decision and more maintainable code. TDD allows writing smaller code having single responsibility rather than monolithic procedures with multiple responsibilities. This makes the code simpler to understand. TDD also forces to write only production code to pass tests based on user requirements.
  • Confidence to Refactor
    If you refactor code, there can be possibilities of breaking the code. So having a set of automated tests you can fix those problems before release. Proper warning will be given if errors are found when automated tests are used. Using TDD should result in faster, more extensible code with fewer bugs that can be updated with minimal risks.
  • Good for teamwork
    In the absence of any team member, other team members can easily pick up and work on the code. It also aids knowledge sharing, thereby making the team more effective overall.
  • Good for Developers
    Though developers have to spend more time in writing TDD test cases, it takes a lot less time for debugging and developing new features. You will write cleaner, less complicated code.
  • Tests as specification
    TDD tests will form the specification of what the code must do.

Mocking and stubbing

  • Mocking is primarily used in unit testing. An object under test may have dependencies on other (complex) objects. To isolate the behavior of the object you want to replace the other objects by mocks that simulate the behavior of the real objects. This is useful if the real objects are impractical to incorporate into the unit test.
  • In short, mocking is creating objects that simulate the behavior of real objects.
  • Stub is a "minimal" simulated object. The stub implements just enough behavior to allow the object under test to execute the test.
  • A mock is like a stub but the test will also verify that the object under test calls the mock as expected. Part of the test is verifying that the mock was used correctly.

  • Moq, NSubstitute, ...
  • Moq seems to be more popular
  • Nuget: Moq

Mocking

Object to mock

public class ApiAdapter
{
public virtual void SaveStuff(string a)
{
// Expensive operation, paid api call
}

public virtual string GetString(int a)
{
// Expensive operation, paid api call
return a.ToString();
}
}

SUT

public IActionResult TestApiCall()
{
string a = _adapter.GetString(1);
_adapter.SaveStuff(a);
string b = _adapter.GetString(2);
return Ok(b);
}

Mocking framework allows partially replacement of mocked object

private readonly Mock<ApiAdapter> _adapterMock;

public UnitTestHomeController(ITestOutputHelper testOutputHelper)
{
_adapterMock = new Mock<ApiAdapter>();
_adapterMock.Setup(x => x.GetString(It.Is<int>(a => a == 1))).Returns("3");
_adapterMock.Setup(x => x.GetString(It.Is<int>(a => a == 2))).Returns("2");
_adapterMock.Setup(x => x.SaveStuff(It.IsAny<string>())).Verifiable();

Test

[Fact]
public void TestMockableDependency()
{
var result = _homeController.TestApiCall();

_adapterMock.Verify(x => x.SaveStuff(It.Is<string>(a => a == "3")), Times.Once);
_adapterMock.Verify(x => x.SaveStuff(It.Is<string>(a => a == "2")), Times.Never);

result.Should().BeOfType<OkObjectResult>();
Assert.Equal("2", (result as OkObjectResult)!.Value as string);
}

Integration tests

  • Broader tests are used to test the app's infrastructure and whole framework, often including the following components:

    • Database
    • File system
    • Network appliances
    • Request-response pipeline
  • Unit tests use fabricated components, known as fakes or mock objects, in place of infrastructure components.

  • In contrast to unit tests, integration tests:

    • Use the actual components that the app uses in production.
    • Require more code and data processing.
    • Take longer to run.

ASP.NET integration tests

Integration tests follow a sequence of events that include the usual Arrange, Act, and Assert test steps:

  • The SUT's web host is configured.
  • A test server client is created to submit requests to the app.
  • The Arrange test step is executed: The test app prepares a request.
  • The Act test step is executed: The client submits the request and receives the response.
  • The Assert test step is executed: The actual response is validated as a pass or fail based on an expected response.

Web Host

  • Nuget: Microsoft.AspNetCore.Mvc.Testing
  • Create custom factory to modify app startup – replace services in DI as needed

  • Find DbContext, replace
  • Seed data as needed

Add at the end of program.cs - needed for integration testing to get correct access levels

public partial class Program
{

}

Factory to create correct webhost, with modified di setup

using System;
using System.Linq;
using DAL.App.EF;
using DAL.App.EF.Seeding;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using WebApp.Tests.Helpers;

namespace WebApp.Tests
{
// ReSharper disable once ClassNeverInstantiated.Global
public class CustomWebApplicationFactory<TStartup>
: WebApplicationFactory<TStartup> where TStartup: class
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureServices(services =>
{
// find DbContext
var descriptor = services.SingleOrDefault(
d => d.ServiceType ==
typeof(DbContextOptions<AppDbContext>));

// if found - remove
if (descriptor != null)
{
services.Remove(descriptor);
}

// and new DbContext
services.AddDbContext<AppDbContext>(options =>
{
options.UseInMemoryDatabase("InMemoryDbForTesting");
});

// create db and seed data
var sp = services.BuildServiceProvider();
using var scope = sp.CreateScope();
var scopedServices = scope.ServiceProvider;
var db = scopedServices.GetRequiredService<AppDbContext>();
var logger = scopedServices
.GetRequiredService<ILogger<CustomWebApplicationFactory<TStartup>>>();

db.Database.EnsureCreated();

try
{
DataSeeder.SeedData(db);
}
catch (Exception ex)
{
logger.LogError(ex, "An error occurred seeding the " +
"database with test messages. Error: {Message}", ex.Message);
}
});
}
}
}

Integration test class

  • Test the response from httpClient
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using AngleSharp.Html.Dom;
using Microsoft.AspNetCore.Mvc.Testing;
using WebApp.Tests.Helpers;
using Xunit;

namespace WebApp.Tests.Controllers
{
public class IntegrationTestHomeController : IClassFixture<CustomWebApplicationFactory<Program>>
{
private readonly HttpClient _client;
private readonly CustomWebApplicationFactory<Program> _factory;


public IntegrationTestHomeController(CustomWebApplicationFactory<Program> factory)
{
_factory = factory;
_client = factory.CreateClient(new WebApplicationFactoryClientOptions
{
AllowAutoRedirect = false
});
}


[Fact]
public async Task Get_Index()
{
// Arrange

// Act
var response = await _client.GetAsync("/");

// Assert
response.EnsureSuccessStatusCode();
}

}
}

Testing http responses

  • DOM parser for analyzing response html
  • Browserlike context for producing next request (cookies, etc)
  • Nuget: AngleSharp.Io

Create response context
Create browser like dom tree and context in c#

using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading;
using System.Threading.Tasks;
using AngleSharp;
using AngleSharp.Html.Dom;
using AngleSharp.Io;

namespace WebApp.Tests.Helpers
{
public static class HtmlHelpers
{
public static async Task<IHtmlDocument> GetDocumentAsync(HttpResponseMessage response)
{
var content = await response.Content.ReadAsStringAsync();
var document = await BrowsingContext.New()
.OpenAsync(ResponseFactory, CancellationToken.None);
return (IHtmlDocument)document;

void ResponseFactory(VirtualResponse htmlResponse)
{
htmlResponse
.Address(response.RequestMessage?.RequestUri)
.Status(response.StatusCode);

MapHeaders(response.Headers);
MapHeaders(response.Content.Headers);

htmlResponse.Content(content);

void MapHeaders(HttpHeaders headers)
{
foreach (var header in headers)
{
foreach (var value in header.Value)
{
htmlResponse.Header(header.Key, value);
}
}
}
}
}
}
}

Integration test

Parse actual response, assert expected content

[Fact]
public async Task TestAction_HasCorrectData()
{
// ARRANGE
const string uri = "/Home/Test";
// ACT
var getTestResponse = await _client.GetAsync(uri);
getTestResponse.EnsureSuccessStatusCode();

// get the actual content from response
var getTestContent = await HtmlHelpers.GetDocumentAsync(getTestResponse);
var testTable = (IHtmlTableElement) getTestContent.QuerySelector("#table");

// ASSERT
Assert.NotNull(testTable);
var testTableSection = testTable.Bodies.FirstOrDefault();
Assert.NotNull(testTableSection);
Assert.Equal(1, testTableSection!.Rows.Length);
}

Testing get->submit->redirect->get cycle

private async Task RegisterUserAsync()
{
// ARRANGE
const string registerUri = "/Identity/Account/Register";
// ACT
var getRegisterResponse = await _client.GetAsync(registerUri);
// ASSERT
getRegisterResponse.EnsureSuccessStatusCode();
// ARRANGE get the actual content from response
var getRegisterContent = await HtmlHelpers.GetDocumentAsync(getRegisterResponse);
// get the form element from page content
var formRegister = (IHtmlFormElement) getRegisterContent.QuerySelector("#form-register");
// set up the form values - username, pwd, etc
var formRegisterValues = new Dictionary<string, string>
{
["Input_Email"] = "test@user.ee",
["Input_Password"] = "Foo.bar1",
["Input_ConfirmPassword"] = "Foo.bar1",
};
// ACT send form with data to server, method (POST) is detected from form element
var postRegisterResponse = await _client.SendAsync(formRegister, formRegisterValues);
// ASSERT found - 302 - ie user was created and we should redirect
Assert.Equal(HttpStatusCode.Found, postRegisterResponse.StatusCode);
}

submitting form

private static IHtmlFormElement SetFormValues(IHtmlFormElement form, IEnumerable<KeyValuePair<string, string>> formValues)
{
foreach (var (key, value) in formValues)
{
switch (form[key])
{
case IHtmlInputElement:
{
(form[key] as IHtmlInputElement)!.Value = value;
if ((form[key] as IHtmlInputElement)!.Type == "checkbox" && bool.Parse(value))
{
(form[key] as IHtmlInputElement)!.IsChecked = true;
}
break;
}
case IHtmlSelectElement:
{
(form[key] as IHtmlSelectElement)!.Value = value;
break;
}
}
}
return form;
}

submitting form

Post

public static Task<HttpResponseMessage> SendAsync(
this HttpClient client, IHtmlFormElement form,
IHtmlElement submitButton, IEnumerable<KeyValuePair<string, string>> formValues)
{
form = SetFormValues(form, formValues);

var submit = form.GetSubmission(submitButton);
var target = (Uri) submit.Target;
if (submitButton.HasAttribute("formaction"))
{
var formaction = submitButton.GetAttribute("formaction");
target = new Uri(formaction, UriKind.Relative);
}

var submission = new HttpRequestMessage(new HttpMethod(submit.Method.ToString()), target)
{
Content = new StreamContent(submit.Body)
};

foreach (var (key, value) in submit.Headers)
{
submission.Headers.TryAddWithoutValidation(key, value);
submission.Content.Headers.TryAddWithoutValidation(key, value);
}
return client.SendAsync(submission);
}

Http Client Extensions

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AngleSharp.Html.Dom;
using Xunit;

namespace WebApp.Tests.Helpers
{
public static class HttpClientExtensions
{
public static Task<HttpResponseMessage> SendAsync(
this HttpClient client,
IHtmlFormElement form,
IHtmlElement submitButton)
{
return client.SendAsync(form, submitButton, new Dictionary<string, string>());
}

public static Task<HttpResponseMessage> SendAsync(
this HttpClient client,
IHtmlFormElement form,
IEnumerable<KeyValuePair<string, string>> formValues)
{
var submitElement = Assert.Single(form.QuerySelectorAll("[type=submit]"));
var submitButton = Assert.IsAssignableFrom<IHtmlElement>(submitElement);

return client.SendAsync(form, submitButton, formValues);
}

public static Task<HttpResponseMessage> SendAsync(
this HttpClient client,
IHtmlFormElement form,
IHtmlElement submitButton,
IEnumerable<KeyValuePair<string, string>> formValues)
{
foreach (var (key, value) in formValues)
{
switch (form[key])
{
case IHtmlInputElement:
{
(form[key] as IHtmlInputElement)!.Value = value;
if ((form[key] as IHtmlInputElement)!.Type == "checkbox" && bool.Parse(value))
{
(form[key] as IHtmlInputElement)!.IsChecked = true;
}

break;
}
case IHtmlSelectElement:
{
(form[key] as IHtmlSelectElement)!.Value = value;
break;
}
}
}

var submit = form.GetSubmission(submitButton);
var target = (Uri) submit!.Target;
if (submitButton.HasAttribute("formaction"))
{
var formaction = submitButton.GetAttribute("formaction");
if (!string.IsNullOrEmpty(formaction))
target = new Uri(formaction, UriKind.Relative);
}

var submission = new HttpRequestMessage(new HttpMethod(submit.Method.ToString()), target)
{
Content = new StreamContent(submit.Body)
};

foreach (var (key, value) in submit.Headers)
{
submission.Headers.TryAddWithoutValidation(key, value);
submission.Content.Headers.TryAddWithoutValidation(key, value);
}

return client.SendAsync(submission);
}
}
}

xUnit.runner.json

{
"shadowCopy": false
}

test csproj

<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<IsPackable>false</IsPackable>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="AngleSharp.Io" Version="0.16.0" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="6.0.4" />
<PackageReference Include="Microsoft.EntityFrameworkCore.InMemory" Version="6.0.4" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.1.0" />
<PackageReference Include="Moq" Version="4.17.2" />
<PackageReference Include="xunit" Version="2.4.1" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.3">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="coverlet.collector" Version="3.1.2">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\WebApp\WebApp.csproj" />
</ItemGroup>

</Project>

WebApp csproj additions

    <ItemGroup>
<InternalsVisibleTo Include="WebApp.Tests" />
</ItemGroup>