Десь рік тому ми почали працювати над новою версією наших продуктів. Саме тоді ми почали випробовувати різні технології, архітектури, підходи, а головне це — міряти performance, бо без цього в highload проектах взагалі не вижити. При проектуванні “любої” системи нам потрібно знати її ліміти: скільки паралельних запитів може обробити мікросервіс за допустиму latency? як багато запитів може витримати база даних, яку ми використовуємо? як довго потрібно чекати на Push повідомлення? як довго триває розподілена транзакція та між якими сервісами відбувається найбільша затримка? І таких питань у нас було безліч. В процесі тестування ми використовували різний tooling: JMeter, ab, Gatling, але всі вони надавали дуже лімітовані можливості. Нам не вдавалося нормально покрити push flow (WebSockets/SSE), різні бази даних, було складно імітувати різний workloads (update/read). На цій зустрічі я розповім про наш досвід застосування load testing: що використовуємо для тестування баз даних, мікросервісів; як готуємо Pull/Push тести та як адаптуємо тести під різні протоколи (HTTP/WebSockets/SSE); які виникають проблеми з замірами latency. Моя доповідь дуже практична, тому після неї ви зможете з легкістю почати застосовувати load testing у себе на проекті.

1. which you
always wanted
LOAD TESTING
by @antyadev

2. @antyadev

5. • Document size ~15Kb
• 6K sport events available for betting
• 73K markets (categories of bets)
• 500K selections (positions to win an event)
• Up to 1M incoming events per minute consumed by one
process
• 8500 outgoing events per second
• 10TB of streaming data per day

6. Sports Data
API

7. Agenda
- The real need of load testing
- Load testing basics
- Current state of load testing on 2019
- NBomber – your drug to load testing

8. Do you know
how many
concurrent requests
your service
can handle?

9. The real need of Load Testing
- Choose technology stack
- Prove architecture design (POC)
- System regression (CI/CD pipeline)
- Explore system limits

10. Load Testing basics
- in many cases, you don’t need a cluster
- make sure that targeting node is fully loaded (100% CPU) to
get max RPS
- make sure that you have long running tests
- make sure that you have controllability (your test
input/output should be consistent)

11. Current state
of
Load Testing

12. Load Testing
is not easy

13. Current state of Load Testing
- Easy to cover HTTP only systems
- Hard to cover any other Pull/Push systems
- Complicated and not flexible API
- Limited cluster support (different
workloads)

14. Current state of Load Testing
- Easy to cover HTTP only systems
- Hard to cover any other Pull/Push systems
- Complicated and not flexible API
- Limited cluster support

15. Batch Update Process 1
Batch Update Process 3
MongoDB
Batch Update Process 2
insert
read
read

16. Producer
KAFKA
ETL
KAFKA
No SQL
1 2
34

17. WebSockets
API

19. What about
1 Million
connections???

20. WebSockets
Users
API
WebSockets
Payments
API
WebSockets
Orders
API
Combine to one connection

21. let start = getCurrentTimer()
execFunc()
let end = getCurrentTimer()
let latency = end - start

22. let start = getCurrentTimer()
sendHttpReqeust()
let end = getCurrentTimer()
let latency = end - start

23. let start = getCurrentTimer()
queryMongoDb()
let end = getCurrentTimer()
let latency = end - start

24. let start = getCurrentTimer()
publishToKafka()
let end = getCurrentTimer()
let latency = end - start

25. let start = getCurrentTimer()
let! response = websockets.ReceivePush()
let end = getCurrentTimer()
let latency = end - start

26. Load test any system
https://nbomber.com
PM> Install-Package NBomber

27. Why another {x} framework for load testing?
The main reasons are:
- To be technology agnostic as much as possible (no dependency on any protocol:
HTTP, WebSockets, SSE etc).
- To be able to test .NET implementation of specific driver. During testing, it was
identified many times that the performance could be slightly different because of
the virtual machine(.NET, Java, PHP, Js, Erlang, different settings for GC) or just
quality of drivers. For example there were cases that drivers written in C++ and
invoked from NodeJs app worked faster than drivers written in C#/.NET. Therefore,
it does make sense to load test your app using your concrete driver and runtime.

28. - Technology agnostic
(no dependency on HTTP, WebSockets, SSE)
- Very simple API
(to test Pull and Push scenarios)
- CI/CD integration

29. Pull and Push
Scenarios
xUnit/NUnit
integration
Sequential flow
ReportingDistributed clusterPlugins
Features

30. type Scenario = {
ScenarioName: string
TestInit: (unit -> Task) option
TestClean: (unit -> Task) option
Steps: Step[] // these steps will be executed sequentially
Assertions: Assertion[]
ConcurrentCopies: int
WarmupDuration: TimeSpan
Duration: TimeSpan
}
type Step = {
StepName: string
Execute: Request -> Task<Response>
}
Very simple API

31. Very simple API
Step.Create("step", () => {...})
ScenarioBuilder
.CreateScenario("scenario", step1, step2)
.WithConcurrentCopies(10)
.WithDuration(TimeSpan.FromSeconds(60))
NBomberRunner.RegisterScenarios(scenario)
.RunInConsole();
Step.create("step", fun () -> {...})
Scenario.create("scenario", [step1; step2])
|> Scenario.withConcurrentCopies(10)
|> Scenario.withDuration(TimeSpan.FromSeconds(60))
NBomberRunner.registerScenario(scenario)
|> NBomberRunner.runInConsole

32. Step 1
Scenario
while true
{
}

33. Scenario
Step 2Step 1
}
while true
{

34. Scenario
Step 3Step 1 Step 2
}
while true
{

35. var scenario = ScenarioBuilder
.CreateScenario("Hello World from NBomber!", step)
.WithConcurrentCopies(10)
.WithDuration(TimeSpan.FromSeconds(10));
var step = Step.Create("simple step", pool, async context => {
// you can do any logic here: go to http, websocket etc
await Task.Delay(TimeSpan.FromSeconds(0.1));
return Response.Ok();
});
NBomberRunner.RegisterScenarios(scenario)
.RunInConsole();

36. let step = Step.create("simple step", pool, fun context -> task {
// you can do any logic here: go to http, websocket etc
do! Task.Delay(TimeSpan.FromSeconds(0.1))
return Response.Ok()
})
Scenario.create("Hello World from NBomber!", [step])
|> Scenario.withConcurrentCopies(10)
|> Scenario.withDuration(TimeSpan.FromSeconds(10.0))
|> NBomberRunner.registerScenario
|> NBomberRunner.runInConsole

37. Scenario withConcurrentCopies 2

38. Scenario withConcurrentCopies 2
Task<T>
Task<T>
Step 2Step 1
Step 2Step 1

39. var client = new HttpClient();
var step1 = Step.Create("GET html", ConnectionPool.None, async context =>
{
var request = new HttpRequestMessage(HttpMethod.Get, "https://nbomber.com");
var response = await client.SendAsync(request);
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var scenario = ScenarioBuilder.CreateScenario("test_nbomber", step1);
NBomberRunner.RegisterScenarios(scenario)
.RunInConsole();
Http example

40. Assertions[Test]
public void Test()
{
var assertions = new[] {
Assertion.ForStep("simple step", stats => stats.OkCount > 2),
Assertion.ForStep("simple step", stats => stats.RPS > 8),
Assertion.ForStep("simple step", stats => stats.Min > 8),
Assertion.ForStep("simple step", stats => stats.Max > 8),
Assertion.ForStep("simple step", stats => stats.Percent95 >= 102),
Assertion.ForStep("simple step", stats => stats.DataMinKb == 1.0),
Assertion.ForStep("simple step", stats => stats.AllDataMB >= 0.01)
};
var scenario = BuildScenario()
.WithConcurrentCopies(1)
.WithWarmUpDuration(TimeSpan.FromSeconds(0))
.WithDuration(TimeSpan.FromSeconds(2))
.WithAssertions(assertions);
NBomberRunner.RegisterScenarios(scenario)
.RunTest();
}

41. .Http
https://nbomber.com
PM> Install-Package NBomber.Http

42. var client = new HttpClient();
var step1 = Step.Create("GET html", ConnectionPool.None, async context =>
{
var request = new HttpRequestMessage(HttpMethod.Get, "https://nbomber.com");
var response = await client.SendAsync(request);
return response.IsSuccessStatusCode
? Response.Ok()
: Response.Fail();
});
var scenario = ScenarioBuilder.CreateScenario("test_nbomber", step1);
NBomberRunner.RegisterScenarios(scenario)
.RunInConsole();
Http example

43. var step1 = HttpStep.CreateRequest("GET", "https://nbomber.com")
.WithHeader("accept", "application/json")
.WithBody(new StringContent("{ some json }"))
.BuildStep("GET html");
var scenario = ScenarioBuilder.CreateScenario("test_nbomber", step1);
NBomberRunner.RegisterScenarios(scenario)
.RunInConsole();
Nbomber.Http

44. let step1 = HttpStep.createRequest("GET", "https://nbomber.com")
|> HttpStep.withHeader("accept", "application/json")
|> HttpStep.withBody(StringContent("{ some json }"))
|> HttpStep.build("GET html")
Scenario.create("test_nbomber", [step1])
|> NBomberRunner.registerScenario
|> NBomberRunner.runInConsole
Nbomber.Http

46. Advanced
Example

47. Batch Update Process 1
Batch Update Process 3
MongoDB
Batch Update Process 2
write
read
read

48. var insertStep = Step.Create("insert", ConnectionPool.None, async context =>
{
await mongoCollection.InsertManyAsync { ... };
});
var pause_200 = Step.Create("pause 200", ConnectionPool.None, async context =>
{
await Task.Delay(TimeSpan.FromMilliseconds(200));
});
var readStep = HttpStep.CreateRequest("GET", "https://somehost")
.BuildStep("GET TOP 100 events");
// insert scenario: batch insert per 200 ms
var insertScenario = ScenarioBuilder.CreateScenario("mongo", insertStep, pause_200);
// read scenario: in parallel, we want to read data
var readScenario = ScenarioBuilder.CreateScenario("http", readStep);
NBomberRunner.RegisterScenarios(insertScenario, readScenario)
.RunInConsole();

49. The Kids
want cluster

50. Agent
Agent
Coordinator
MongoDB
read
write
read

51. Test
Scenarios.fs
Compile
Test
Scenarios.EXE
Upload
Test
Scenarios.EXE
config
Test
Scenarios.EXE
config

52. Test
Scenarios.EXE
config
Test
Scenarios.EXE
config

53. Test
Scenarios.EXE
coordinator_config
Test
Scenarios.EXE
agent_config

54. Test
Scenarios.EXE
coordinator_config
Test
Scenarios.EXE
agent_config
Test
Scenarios.EXE
agent_config

55. {
"ClusterSettings": {
"Agent": {
"ClusterId": "test_cluster",
"Port": 4000
}
}
}
Agent
Agent
Agent Config

56. Coordinator
Config
{
"GlobalSettings": {
"ScenariosSettings": [
{
"ScenarioName": "read",
"Duration": "00:00:10",
"ConcurrentCopies": 200
}
]
},
"ClusterSettings": {
"Coordinator": {
"ClusterId": "test_cluster",
"TargetScenarios": [ "insert" ]
"Agents": [
{
"Host": "35.187.178.41",
"Port": 4000
"TargetScenarios": [ "read" ]
}
]
},
}
} Coordinator

57. The Kids
want PUSH

58. var url = "ws://localhost:53231";
var webSocketsPool = ConnectionPool.Create("webSocketsPool", openConnection: () =>
{
var ws = new ClientWebSocket();
ws.ConnectAsync(new Uri(url), CancellationToken.None).Wait();
return ws;
});
var pingStep = Step.Create("ping", webSocketsPool, async context =>
{
var msg = new WebSocketRequest
{
CorrelationId = context.CorrelationId,
RequestType = RequestType.Ping
};
await context.Connection.SendAsync(msg, WebSocketMessageType.Text);
return Response.Ok();
});

59. var pingStep = Step.Create("ping", webSocketsPool, async context =>
{
var msg = new WebSocketRequest
{
CorrelationId = context.CorrelationId,
RequestType = RequestType.Ping
};
await context.Connection.SendAsync(msg, WebSocketMessageType.Text);
return Response.Ok();
});
var pongStep = Step.Create("pong", webSocketsPool, async context =>
{
while (true)
{
var message = await context.Connection.ReceiveMessage();
if (msg.CorrelationId == context.CorrelationId)
{
return Response.Ok(msg);
}
}
});

60. Internals

61. NBomber
Runner

62. let run (dep: Dependency, context: NBomberContext) =
asyncResult {
let! ctx = Validation.validateContext(context)
let! nodeStats =
NBomberContext.tryGetClusterSettings(ctx)
|> Option.map(function
| Coordinator c -> runClusterCoordinator(dep, ctx, c)
| Agent a -> runClusterAgent(dep, ctx, a))
|> Option.orElseWith(fun _ -> runSingleNode(dep, ctx))
|> Option.get
return nodeStats
|> calcStatistics(dep, ctx)
|> saveStatistics(ctx)
|> applyAsserts(ctx)
|> buildReport(dep)
|> saveReport(dep, ctx)
}
|> AsyncResult.mapError(handleError(dep))
|> Async.RunSynchronously
|> ignore

63. Validation

64. type Result<TOk, TError> =
<!>
<*>
>>=
>=>

65. module ScenarioValidation =
let validate (context: NBomberContext) =
context.Scenarios
|> checkEmptyName
>>= checkDuplicateName
>>= checkEmptyStepName
>>= checkDuplicateStepName
>>= checkDuration
>>= checkConcurrentCopies
>>= fun _ -> Ok context

66. module GlobalSettingsValidation =
let validate (context: NBomberContext) =
context.NBomberConfig
|> Option.bind(fun x -> x.GlobalSettings)
|> Option.map(fun glSettings ->
glSettings
|> checkEmptyTarget
>>= checkAvailableTarget
>>= checkDuration
>>= checkConcurrentCopies
>>= checkEmptyReportName
>>= fun _ -> Ok context
)
|> Option.defaultValue(Ok context)

67. module ScenarioValidation =
let validate (context) = ...
module GlobalSettingsValidation =
let validate (context) = ...
module Validation =
let validate =
ScenarioValidation.validate >=> GlobalSettingsValidation.validate

68. Type Safe
Cluster
(>>=)

69. type IClusterCoordinator =
abstract StartNewSession: ScenarioSetting[] -> Task<Result<unit,Error>>
abstract WaitOnAllAgentsReady: unit -> Task<Result<unit,Error>>
abstract StartWarmUp: unit -> Task<Result<unit,Error>>
abstract StartBombing: unit -> Task<Result<unit,Error>>
abstract GetStatistics: unit -> Task<Result<NodeStats[],Error>>

70. let runCoordinator (cluster: IClusterCoordinator, localHost: IScenariosHost,
settings: ScenarioSetting[], targetScns: string[]) =
asyncResult {
do! cluster.SendStartNewSession(settings)
do! localHost.InitScenarios(settings, targetScns)
do! cluster.WaitOnAllAgentsReady()
do! cluster.SendStartWarmUp()
do! localHost.WarmUpScenarios()
do! cluster.WaitOnAllAgentsReady()
do! cluster.SendStartBombing() // Task<Result<unit,Error>>
do! localHost.StartBombing()
do! cluster.WaitOnAllAgentsReady()
let localStats = localHost.GetStatistics()
let! agentsStats = cluster.GetStatistics()
let allStats = Array.append [|localStats|] agentsStats
localHost.StopScenarios()
return allStats
}

71. - (2663) LOC
- (6) OOP objects
- (~160) functions
Summary

78. COORDINATED
OMISSION

80. GC pause
????
API

81. THANKS
@antyadev
antyadev@gmail.com
http://nbomber.com