Fallacies of unit testing

Speaker: Alexey Golub @Tyrrrz
Fallacies of Unit Testing
Road to reality-driven testing

WARNING
Controversial topic

Clarification
Unit testing is not inherently bad, but the practices & beliefs surrounding it
often lead to cargo cult mentality
Some subjects in the talk may be opinionated and come from my own
experience, which might not represent yours
I will rely on “classic” definitions, where…
• Unit tests target smallest separable parts of code
• End-to-end tests target the application boundaries
• Integration tests target everything in-between

/whois ${speaker}
• Open-source developer ✨
• Conference speaker & blogger 🌐️
• C#, F#, JavaScript 💻
• Cloud & web ☁️
• Automation & DevOps ⚙️

Unit tests…
Unit
Small isolated part
No formal definition
Method of a class
Function of a module

Unit
Unit
Unit
Unit
Unit tests must be isolated
Dependency inversion

Unit
Unit
Unit
Unit
Abstraction
Unit tests must be isolated

Unit tests must be pure
Unit
DB
File
system

• Unit tests achieve isolation by stubbing dependencies
• Unit tests don’t cover integrations between units
• Unit tests mostly avoid exercising effectful code
• Unit tests are primarily useful to verify business logic

public class LocationProvider : IDisposable
{
private readonly HttpClient _httpClient = new HttpClient();
// Gets location by query
public async Task<Location> GetLocationAsync(string locationQuery) { /* ... */ }
// Gets current location by IP
public async Task<Location> GetLocationAsync() { /* ... */ }
public void Dispose() => _httpClient.Dispose();
}
Design is fine, but
not unit-testable
public class SolarCalculator : IDiposable
{
private readonly LocationProvider _locationProvider = new LocationProvider();
// Gets solar times for current location and specified date
public async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) { /* //. // }
public void Dispose() /> _locationProvider.Dispose();
}

public interface ILocationProvider
{
Task<Location> GetLocationAsync(string locationQuery);
Task<Location> GetLocationAsync();
}
public class LocationProvider : ILocationProvider
{
private readonly HttpClient _httpClient;
public LocationProvider(HttpClient httpClient) => _httpClient = httpClient;
public async Task<Location> GetLocationAsync(string locationQuery) { /* ... */ }
public async Task<Location> GetLocationAsync() { /* ... */ }
}
public interface ISolarCalculator
{
Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date);
}
public class SolarCalculator : ISolarCalculator
{
private readonly ILocationProvider _locationProvider;
public SolarCalculator(ILocationProvider locationProvider) => _locationProvider = locationProvider;
public async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) { /* ... */ }
}
Isolation through
dependency
inversion

Design influence
• Isolation comes at the expense of fragmentation
• Objects no longer encapsulate their dependencies
• Abstractions for the sake of testing, not polymorphism
• Design driven by testability, rather than domain

public class SolarCalculatorTests
{
[Fact]
public async Task GetSolarTimesAsync_ForKyiv_ReturnsCorrectSolarTimes()
{
// Arrange
var location = new Location(50.45, 30.52);
var date = new DateTimeOffset(2019, 11, 04, 00, 00, 00, TimeSpan.FromHours(+2));
var expectedSolarTimes = new SolarTimes(
new TimeSpan(06, 55, 00),
new TimeSpan(16, 29, 00)
);
var locationProvider = Mock.Of<ILocationProvider>(lp =>
lp.GetLocationAsync() == Task.FromResult(location)
);
var solarCalculator = new SolarCalculator(locationProvider);
// Act
var solarTimes = await solarCalculator.GetSolarTimesAsync(date);
// Assert
solarTimes.Should().BeEquivalentTo(expectedSolarTimes);
}
}
Implementation-
aware testing
Tight coupling to a
specific class/method

Consider the drawbacks
• Unit tests rely on implementation details
Unit
Dep A
Dep B
Dep C
Which method does it use?
How is it used?
When does it call it?
How many times does it call it?

• Unit tests often lead to more complicated design
Unit Unit
Unit
Unit Unit
Unit

• Unit tests can get quite expensive
UnitDep
Mock
A
Mock
B
Mock
C

• Unit tests are limited in purpose
Payroll
System
CRUD
application
Business logic (%)

• Unit tests don’t exercise user behavior
User
API
UI
CLI
Unit
Unit
Unit
Unit

Pyramid-driven testing

Development tests need to be
• Fast
• Cheap
• Predictable
• Stable

High-level tests are
• Not Fast
• Not Cheap
• Not Predictable
• Not Stable
(allegedly)

Handled by QAs
Often
avoided
entirely
Development
testing
Someone
else’s
problem

Integration tests are not slow
• Just because they can be, doesn’t mean they should be
• Technologies make things easier
• Architecture dictates testability

Return on investment is not linear
• Isolation isn’t free
• Higher-level tests are often cheaper
• Cheap & fast tests aren’t automatically valuable

Business logic is not a constant
• Value of unit testing is contingent on the amount of business logic
• Modern software contains a lot of “glue” code
• Depending on the project, the pyramid may be misleading

Development
testing
Unit
testing≠

Reality-driven testing

Component
System
Confidence
(value)
User
Systemboundary
Unit level End-to-end levelIntegration scope

User needs to be able get solar times for their location
User needs to be able get solar times during periods of midnight sun
User needs to be able get solar times even if their location cannot be determined
Functional requirements

public class SolarTimesSpecs
{
[Fact]
public async Task User_can_get_solar_times_automatically_for_their_location() { /* //. // }
[Fact]
public async Task User_can_get_solar_times_during_periods_of_midnight_sun() { /* //. // }
[Fact]
public async Task User_can_get_solar_times_if_their_location_cannot_be_determined() { /* //. // }
}
Functional tests

Functional tests Powered by xUnit’s
test display options
✓ Live documentation
✓ Human-readable test
names
✓ No dependency on code
structure

Functional requirements can be…
• Formal
Software specification documents, user stories, etc.
• Informal
JIRA tickets, readme, API documentation, usage instructions, assumed, etc.
Every software has them, in one form or the other (yes, even yours)

Guidelines
• Write tests that are as highly integrated as possible ⚙️
• Keep their speed, complexity, and stability acceptable 🚀
• Partition tests based on threads of behavior 🖱️
• Test suite should not mirror code structure 🔗

Example
Functional test suite for an ASP.NET Core application

public class LocationProvider
{
private readonly HttpClient _httpClient;
public LocationProvider(HttpClient httpClient) />
_httpClient = httpClient;
public async Task<Location> GetLocationAsync(IPAddress ip)
{
// If IP is local, just don't pass anything (useful when running on localhost)
var ipFormatted = !ip.IsLocal() ? ip.MapToIPv4().ToString() : "";
var json = await _httpClient.GetJsonAsync($"http://ip-api.com/json/{ipFormatted}");
var latitude = json.GetProperty("lat").GetDouble();
var longitude = json.GetProperty("lon").GetDouble();
return new Location
{
Latitude = latitude,
Longitude = longitude
};
}
}

public class SolarCalculator
{
private readonly LocationProvider _locationProvider;
public SolarCalculator(LocationProvider locationProvider) />
_locationProvider = locationProvider;
private static TimeSpan CalculateSolarTimeOffset(Location location, DateTimeOffset instant,
double zenith, bool isSunrise)
{
/* //. //
}
public async Task<SolarTimes> GetSolarTimesAsync(IPAddress ip, DateTimeOffset date)
{
var location = await _locationProvider.GetLocationAsync(ip);
var sunriseOffset = CalculateSolarTimeOffset(location, date, 90.83, true);
var sunsetOffset = CalculateSolarTimeOffset(location, date, 90.83, false);
var sunrise = date.ResetTimeOfDay().Add(sunriseOffset);
var sunset = date.ResetTimeOfDay().Add(sunsetOffset);
return new SolarTimes
{
Sunrise = sunrise,
Sunset = sunset
};
}
}

[ApiController, Route("solartimes")]
public class SolarTimeController : ControllerBase
{
private readonly SolarCalculator _solarCalculator;
private readonly CachingLayer _cachingLayer;
public SolarTimeController(SolarCalculator solarCalculator, CachingLayer cachingLayer)
{
_solarCalculator = solarCalculator;
_cachingLayer = cachingLayer;
}
[HttpGet("by_ip")]
public async Task<IActionResult> GetByIp(DateTimeOffset? date)
{
var ip = HttpContext.Connection.RemoteIpAddress;
var cacheKey = $"{ip},{date}";
var cachedSolarTimes = await _cachingLayer.TryGetAsync<SolarTimes>(cacheKey);
if (cachedSolarTimes != null)
return Ok(cachedSolarTimes);
var solarTimes = await _solarCalculator.GetSolarTimesAsync(ip, date ?? DateTimeOffset.Now);
await _cachingLayer.SetAsync(cacheKey, solarTimes);
return Ok(solarTimes);
}
}

public class CachingLayer
{
private readonly IConnectionMultiplexer _redis;
public CachingLayer(IConnectionMultiplexer connectionMultiplexer) =>
_redis = connectionMultiplexer;
public async Task<T> TryGetAsync<T>(string key) where T : class
{
var result = await _redis.GetDatabase().StringGetAsync(key);
if (result.HasValue)
return JsonSerializer.Deserialize<T>(result.ToString());
return null;
}
public async Task SetAsync<T>(string key, T obj) where T : class =>
await _redis.GetDatabase().StringSetAsync(key, JsonSerializer.Serialize(obj));
}

public class Startup
{
// //.
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc(o /> o.EnableEndpointRouting = false);
services.AddSingleton<IConnectionMultiplexer>(
ConnectionMultiplexer.Connect(GetRedisConnectionString()));
services.AddSingleton<CachingLayer>();
services.AddHttpClient<LocationProvider>();
services.AddTransient<SolarCalculator>();
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
if (env.IsDevelopment())
app.UseDeveloperExceptionPage();
app.UseMvcWithDefaultRoute();
}
}

public class FakeApp : IDisposable
{
private readonly WebApplicationFactory<Startup> _appFactory;
public HttpClient Client { get; }
public FakeApp()
{
_appFactory = new WebApplicationFactory<Startup>();
Client = _appFactory.CreateClient();
}
public void Dispose()
{
Client.Dispose();
_appFactory.Dispose();
}
}
Runs the entire HTTP
pipeline in-memory
Get a client to talk to
our app

public class RedisFixture : IAsyncLifetime
{
private string _containerId;
public async Task InitializeAsync()
{
var result = await Cli.Wrap("docker")
.WithArguments("run -d -p 6379:6379 redis")
.ExecuteBufferedAsync();
_containerId = result.StandardOutput.Trim();
}
public async Task ResetAsync() />
await Cli.Wrap("docker")
.WithArguments($"exec {_containerId} redis-cli FLUSHALL")
.ExecuteAsync();
public async Task DisposeAsync() />
await Cli.Wrap("docker")
.WithArguments($"container kill {_containerId}")
.ExecuteAsync();
}
Ideally should bind
to a random port
Can also use:
• Docker.NET
• .NET TestContainers

public class SolarTimeSpecs : IClassFixture<RedisFixture>, IAsyncLifetime
{
private readonly RedisFixture _redisFixture;
public SolarTimeSpecs(RedisFixture redisFixture) =>
_redisFixture = redisFixture;
public async Task InitializeAsync() => await _redisFixture.ResetAsync();
[Fact]
public async Task User_can_get_solar_times_for_their_location_by_ip()
{
// Arrange
using var app = new FakeApp();
// Act
var response = await app.Client.GetStringAsync("/solartimes/by_ip");
var solarTimes = JsonSerializer.Deserialize<SolarTimes>(response);
// Assert
solarTimes.Sunset.Should().BeWithin(TimeSpan.FromDays(1)).After(solarTimes.Sunrise);
solarTimes.Sunrise.Should().BeCloseTo(DateTimeOffset.Now, TimeSpan.FromDays(1));
solarTimes.Sunset.Should().BeCloseTo(DateTimeOffset.Now, TimeSpan.FromDays(1));
}
}
Redis is reset
between tests
Requirement-driven test
Property-based assertions

public class FakeIpStartupFilter : IStartupFilter
{
public IPAddress Ip { get; set; } = IPAddress.Parse("::1");
public Action<IApplicationBuilder> Configure(Action<IApplicationBuilder> nextFilter)
{
return app =>
{
app.Use(async (ctx, next) =>
{
ctx.Connection.RemoteIpAddress = Ip;
await next();
});
nextFilter(app);
};
}
}

public class FakeApp : IDisposable
{
private readonly WebApplicationFactory<Startup> _appFactory;
private readonly FakeIpStartupFilter _fakeIpStartupFilter = new FakeIpStartupFilter();
public HttpClient Client { get; }
public IPAddress ClientIp
{
get => _fakeIpStartupFilter.Ip;
set => _fakeIpStartupFilter.Ip = value;
}
public FakeApp()
{
_appFactory = new WebApplicationFactory<Startup>().WithWebHostBuilder(o =>
{
o.ConfigureServices(s =>
{
s.AddSingleton<IStartupFilter>(_fakeIpStartupFilter);
});
});
Client = _appFactory.CreateClient();
}
// ...
}
Inject fake IP provider
into pipeline

[Fact]
public async Task User_can_get_solar_times_for_their_location_by_ip()
{
// Arrange
using var app = new FakeApp
{
ClientIp = IPAddress.Parse("20.112.101.1")
};
var date = new DateTimeOffset(2020, 07, 03, 0, 0, 0, TimeSpan.FromHours(-5));
var expectedSunrise = new DateTimeOffset(2020, 07, 03, 05, 20, 37, TimeSpan.FromHours(-5));
var expectedSunset = new DateTimeOffset(2020, 07, 03, 20, 28, 54, TimeSpan.FromHours(-5));
// Act
var query = new QueryBuilder
{
{"date", date.ToString("O", CultureInfo.InvariantCulture)}
};
var response = await app.Client.GetStringAsync($"/solartimes/by_ip{query}");
var solarTimes = JsonSerializer.Deserialize<SolarTimes>(response);
// Assert
solarTimes.Sunrise.Should().BeCloseTo(expectedSunrise, TimeSpan.FromSeconds(1));
solarTimes.Sunset.Should().BeCloseTo(expectedSunset, TimeSpan.FromSeconds(1));
}
Strict assertions

✓ No mocks
✓ Real infrastructural dependencies
✓ Full HTTP pipeline
✓ High confidence
✓ Reasonable cost & speed

The goal is not to cover everything end-to-end,
but to gain the highest return on investment
If you can test at the highest level, do it.
If not, reduce scope until the results are acceptable.
Confidence
Usability

Summary
1. Think critically and challenge best practices
2. Don’t rely on the test pyramid too much
3. Separate tests by functionality, rather than by classes or scope
4. Aim for highest level of integration with reasonable cost, speed, reliability
5. Avoid sacrificing software design for testability
6. Consider mocking only as a last resort

Resources
• Unit Testing is Overrated (Alexey Golub)
https://tyrrrz.me/blog/unit-testing-is-overrated
• Write tests. Not too many. Mostly integration (Kent C. Dodds)
https://kentcdodds.com/blog/write-tests
• Mocking is a Code Smell (Eric Elliott)
https://medium.com/javascript-scene/mocking-is-a-code-smell-944a70c90a6a
• Test-induced design damage (David Heinemeier Hansson)
https://dhh.dk/2014/test-induced-design-damage.html

Thank you!

Fallacies of unit testing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Fallacies of unit testing

Similar to Fallacies of unit testing (20)

More from Oleksii Holub

More from Oleksii Holub (7)

Recently uploaded

Recently uploaded (20)

Fallacies of unit testing