The document provides a comprehensive overview of benchmarking software and infrastructure performance targeting .NET developers, emphasizing its importance in understanding product behavior, reliability, and performance under stress. It introduces BenchmarkDotNet, a popular open-source library for .NET benchmarks, discussing installation, core concepts, and advanced techniques for effective performance evaluation. Additionally, it covers integrating benchmarks into CI/CD pipelines and modern topics such as cloud-native performance testing and sustainable software engineering.

1. Benchmarking
Software & Infrastructure
Performance
A Practical Overview for .NET Developers
Part 1
tech-afternoon Aug 2024, Amin Mesbahi

2. Introduction
Benchmarking Software & Infrastructure Performance
2015 2020 2024
https://t.me/techafternoon
tech-afternoon
Amin Mesbahi
Solutions Architect, former Engineering Manager, and Lifetime Developer
amin.mesbahi@gmail.com

3. Agenda
Benchmarking Software & Infrastructure Performance
Introduction
Why Benchmarking?
BenchmarkDotNet
1. Introduction to BenchmarkDotNet
1.1. Overview of BenchmarkDotNet
1.2. Predecessors and Alternatives
2. Getting Started
2.1. Installation and Setup
2.2. Core Concepts and Terminology
2.3. Writing Your First Benchmark
3. Intermediate Concepts and Examples
3.1. Configuring Benchmarks
3.2. Diagnosing Performance
3.3. Comparing Multiple Implementations
4. Advanced Concepts and Topics
4.1. Advanced Benchmarking Techniques
4.2. Avoiding Common Pitfalls
4.3. Custom Exporters and Analyzers
5. BenchmarkDotNet in Production
5.1. Integrating Benchmarks into CI/CD
5.2. Enterprise-Level Performance Testing
5.3. Tracking and Reporting
6. Complementary Tools and Best Practices
6.1. Additional Tools
6.2. Best Practices across teams
7. Q&A and Live Coding Session
7.1. Live Examples
7.2. Final Q&A
What’s next?

4. Why Benchmarking?
Benchmarking Software & Infrastructure Performance
We need to get to know our
product behavior, reliability, load
capacity, speed, stability, and
performance
under stress.

5. What is Benchmarking?
Benchmarking Software & Infrastructure Performance
• Benchmarking is the process of running a series of tests to
measure the performance of software, hardware, and
infrastructure under specific conditions.
• It provides a quantitative basis to evaluate speed, reliability,
load capacity, and stability, particularly under stress.
• Benchmarking is not limited to code; it encompasses system,
application, hardware, network, and storage performance.

6. Why Benchmarking?
Benchmarking Software & Infrastructure Performance
But benchmarking is something beyond code!
System benchmarking
performance of an entire system, including its hardware, software, and network components.
Application benchmarking
performance of specific applications, such as databases, web applications, and office productivity applications.
Hardware benchmarking
performance of specific hardware components, such as processors, graphics cards, and memory.
Network benchmarking
performance of network systems, including local area networks (LANs) and wide area networks (WANs).
Storage benchmarking
performance of storage systems, including hard drives, solid-state drives (SSDs), and storage area networks (SANs).

7. Why Benchmarking?
Benchmarking Software & Infrastructure Performance
A benchmark is a non-functional test that you run to
measure how fast your code is.
Purpose of Benchmarking:
• Understanding product behavior in real-world scenarios.
• Identifying performance bottlenecks.
• Ensuring the application meets performance requirements before deployment.
• Providing a baseline for future improvements and regression analysis.

8. BenchmarkDotNet, overview:
Benchmarking Software & Infrastructure Performance
• A popular, open-source .NET library for benchmarking .NET code.
• License: MIT | Downloads (Sep 2024): Total 32.5M, Daily 8.1K
• Project Owner: Andrey Akinshin *(Mathematician, software developer, Ph.D.), Currently works @ JetBrains
• Provides accurate performance measurements and detailed reports.
• Supports a wide range of runtime environments, from .NET Core and .NET Framework to Mono.
Predecessors and Alternatives:
• Earlier tools included custom scripts, microbenchmarks, and libraries like xUnit's Performance or
NBench.
• Alternatives to BenchmarkDotNet include tools like dotTrace, PerfView, or profilers integrated into IDEs.

9. Getting Started with BenchmarkDotNet
Benchmarking Software & Infrastructure Performance
Installation and Setup:
1. Add the BenchmarkDotNet NuGet package to your project.
2. Configure the benchmark in Program.cs using BenchmarkRunner.Run<YourBenchmarkClass>();.
3. Run the application in Release mode

10. Core Concepts and Terminology:
Benchmarking Software & Infrastructure Performance
• Jobs: Configurations defining the environment and execution
strategy for benchmarks (e.g., SimpleJob, LongRunJob).
• Diagnosers: Tools to capture additional information
(e.g., MemoryDiagnoser for memory allocations).
• Attributes: Annotations like [Benchmark], [Config], and
[MemoryDiagnoser] to define benchmark methods and
configurations.

11. Let’s make something with code!
Benchmarking Software & Infrastructure Performance

12. BenchmarkDotNet in Production
Benchmarking Software & Infrastructure Performance
• Integrating Benchmarks into CI/CD:
• Automate benchmarks in Continuous Integration/Continuous
Deployment pipelines.
• Use tools like Azure DevOps, GitHub Actions, or Jenkins to
automate performance testing.
• Enterprise-Level Performance Testing:
• Scale benchmarks to test enterprise-grade applications, including
distributed systems and microservices.
• Use cloud-based tools and environments (e.g., Azure or AWS) to
simulate real-world conditions.
• Tracking and Reporting:
• Track performance over time to detect regressions.
• Generate detailed reports and dashboards using tools like Grafana,
Datadog, or custom reporting solutions.

13. Modern Topics!
Benchmarking Software & Infrastructure Performance
• Cloud-Native Performance Testing:
• containers, serverless functions.
• k6.io | modern Open source and SaaS load testing tool
• Observability and Performance Monitoring:
• Prometheus, OpenTelemetry for real-time insights
• Shift-Left Testing:
• The importance of incorporating performance testing early in
development to catch issues sooner
• Microservices and Distributed Systems:
• Benchmarking in a distributed environment, focusing on network
latency, service discovery, and inter-service communication.
• Sustainable Software Engineering:
• Benchmarking in reducing energy consumption and making
software more sustainable.

14. Thank you for your TIME!
Benchmarking Software & Infrastructure Performance