Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

QA Fest 2019. Антон Молдован. Load testing which you always wanted

28 views

Published on

Десь рік тому ми почали працювати над новою версією наших продуктів. Саме тоді ми почали випробовувати різні технології, архітектури, підходи, а головне це — міряти performance, бо без цього в highload проектах взагалі не вижити.
При проектуванні “любої” системи нам потрібно знати її ліміти:
- скільки паралельних запитів може обробити мікросервіс за допустиму latency?
- як багато запитів може витримати база даних, яку ми використовуємо?
- як довго потрібно чекати на Push повідомлення?
- як довго триває розподілена транзакція та між якими сервісами відбувається найбільша затримка?
І таких питань у нас було безліч. В процесі тестування ми використовували різний tooling: JMeter, ab, Gatling, але всі вони надавали дуже лімітовані можливості. Нам не вдавалося нормально покрити push flow (WebSockets/SSE), різні бази даних, було складно імітувати різний workloads (update/read).
На цій зустрічі я розповім про наш досвід застосування load testing:
- що використовуємо для тестування баз даних, мікросервісів;
- як готуємо Pull/Push тести та як адаптуємо тести під різні протоколи (HTTP/WebSockets/SSE);
- які виникають проблеми з замірами latency.
Моя доповідь дуже практична, тому після неї ви зможете з легкістю почати застосовувати load testing у себе на проекті.

Published in: Education
  • Be the first to comment

  • Be the first to like this

QA Fest 2019. Антон Молдован. Load testing which you always wanted

  1. 1. which you always wanted LOAD TESTING by @antyadev
  2. 2. Data Feed Terminals
  3. 3. select * from Games where isLive = true order by totalBets desc limit 10 PUSH changes
  4. 4. Sports Data API
  5. 5. • Document size ~15Kb • 5K sport games available for betting • 73K markets (categories of bets) • 500K selections (positions to win an event) • Up to 1M incoming events per minute consumed by one process • 10K outgoing events per second • 10TB of streaming data per day
  6. 6. { update } { push } { pull } Data Consistency Problem
  7. 7. { update } { push } { pull } Data Propagation Problem DB
  8. 8. Agenda - The real need of load testing - Load testing basics - Current state of load testing on 2019 - NBomber – your drug to load testing
  9. 9. Do you know how many concurrent requests your service can handle?
  10. 10. The real need of Load Testing - Choose technology stack - Prove architecture design (POC) - System regression (CI/CD pipeline) - Explore system limits
  11. 11. Load Testing basics - in many cases, you don’t need a cluster - make sure that targeting node is fully loaded (100% CPU) to get max RPS - make sure that you have long running tests - make sure that you have controllability (your test input/output should be consistent)
  12. 12. Current state of Load Testing
  13. 13. Load Testing is not easy
  14. 14. Current state of Load Testing Gatling (open-source, Scala, Akka): - free + commercial version - test as code - very good for HTTP (has API for WebSockets, SSE) - very nice reports - has integration with Grafana, Jenkins - has UI recorder to playback actions - no back pressure - not easy to customize - no cluster in free version
  15. 15. class BasicSimulation extends Simulation { val httpConf = http .baseURL("http://computer-database.gatling.io") .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" .doNotTrackHeader("1") .acceptLanguageHeader("en-US,en;q=0.5") .acceptEncodingHeader("gzip, deflate") .userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/ val scn = scenario("BasicSimulation") .exec( http("request_1").get("/")) .pause(5) setUp( scn.inject(atOnceUsers(1)) ).protocols(httpConf) }
  16. 16. Current state of Load Testing - Easy to cover HTTP only systems - Hard to cover any other Pull/Push systems - Complicated and not flexible API - Limited cluster support (different workloads)
  17. 17. Current state of Load Testing - Easy to cover HTTP only systems - Hard to cover any other Pull/Push systems - Complicated and not flexible API - Limited cluster support
  18. 18. Batch Update Process 1 Batch Update Process 3 MongoDB Batch Update Process 2 insert read read
  19. 19. Producer KAFKA ETL KAFKA No SQL 1 2 34
  20. 20. WebSockets API
  21. 21. What about 1 Million connections???
  22. 22. WebSockets Users API WebSockets Payments API WebSockets Orders API Combine to one connection
  23. 23. let start = getCurrentTimer() execFunc() let end = getCurrentTimer() let latency = end - start
  24. 24. let start = getCurrentTimer() sendHttpReqeust() let end = getCurrentTimer() let latency = end - start
  25. 25. let start = getCurrentTimer() queryMongoDb() let end = getCurrentTimer() let latency = end - start
  26. 26. let start = getCurrentTimer() publishToKafka() let end = getCurrentTimer() let latency = end - start
  27. 27. let start = getCurrentTimer() let! response = websockets.ReceivePush() let end = getCurrentTimer() let latency = end - start
  28. 28. Load test any system https://nbomber.com PM> Install-Package NBomber
  29. 29. Why another {x} framework for load testing? The main reasons are: - To be technology agnostic as much as possible (no dependency on any protocol: HTTP, WebSockets, SSE etc). - To be able to test .NET implementation of specific driver. During testing, it was identified many times that the performance could be slightly different because of the virtual machine(.NET, Java, PHP, Js, Erlang, different settings for GC) or just quality of drivers. For example there were cases that drivers written in C++ and invoked from NodeJs app worked faster than drivers written in C#/.NET. Therefore, it does make sense to load test your app using your concrete driver and runtime.
  30. 30. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  31. 31. Pull and Push Scenarios xUnit/NUnit integration Sequential flow ReportingDistributed clusterPlugins Features
  32. 32. type Scenario = { ScenarioName: string TestInit: (ScenarioContext -> Task) option TestClean: (ScenarioContext -> Task) option Steps: Step[] // these steps will be executed sequentially Assertions: Assertion[] ConcurrentCopies: int WarmupDuration: TimeSpan Duration: TimeSpan } type Step = { StepName: string Execute: StepContext -> Task<Response> } Very simple API
  33. 33. Very simple API Step.Create("step", () => {...}) ScenarioBuilder .CreateScenario("scenario", step1, step2) .WithConcurrentCopies(10) .WithDuration(TimeSpan.FromSeconds(60)) NBomberRunner.RegisterScenarios(scenario) .RunInConsole(); Step.create("step", fun () -> {...}) Scenario.create("scenario", [step1; step2]) |> Scenario.withConcurrentCopies(10) |> Scenario.withDuration(TimeSpan.FromSeconds(60)) NBomberRunner.registerScenario(scenario) |> NBomberRunner.runInConsole
  34. 34. Step 1 Scenario while true { }
  35. 35. Scenario Step 2Step 1 } while true {
  36. 36. Scenario Step 3Step 1 Step 2 } while true {
  37. 37. var scenario = ScenarioBuilder .CreateScenario("Hello World from NBomber!", step) .WithConcurrentCopies(10) .WithDuration(TimeSpan.FromSeconds(10)); var step = Step.Create("simple step", async context => { // you can do any logic here: go to http, websocket etc await Task.Delay(TimeSpan.FromSeconds(0.1)); return Response.Ok(); }); NBomberRunner.RegisterScenarios(scenario) .RunInConsole();
  38. 38. let step = Step.create("simple step", fun context -> task { // you can do any logic here: go to http, websocket etc do! Task.Delay(TimeSpan.FromSeconds(0.1)) return Response.Ok() }) Scenario.create("Hello World from NBomber!", [step]) |> Scenario.withConcurrentCopies(10) |> Scenario.withDuration(TimeSpan.FromSeconds(10.0)) |> NBomberRunner.registerScenario |> NBomberRunner.runInConsole
  39. 39. Scenario withConcurrentCopies 2
  40. 40. Scenario withConcurrentCopies 2 Task<T> Task<T> Step 2Step 1 Step 2Step 1
  41. 41. var client = new HttpClient(); var step1 = Step.Create("GET html", async context => { var request = new HttpRequestMessage(HttpMethod.Get, "https://nbomber.com"); var response = await client.SendAsync(request); return response.IsSuccessStatusCode ? Response.Ok() : Response.Fail(); }); var scenario = ScenarioBuilder.CreateScenario("test_nbomber", step1); NBomberRunner.RegisterScenarios(scenario) .RunInConsole(); Http example
  42. 42. Assertions[Test] public void Test() { var assertions = new[] { Assertion.ForStep("simple step", stats => stats.OkCount > 2), Assertion.ForStep("simple step", stats => stats.RPS > 8), Assertion.ForStep("simple step", stats => stats.Min > 8), Assertion.ForStep("simple step", stats => stats.Max > 8), Assertion.ForStep("simple step", stats => stats.Percent95 >= 102), Assertion.ForStep("simple step", stats => stats.DataMinKb == 1.0), Assertion.ForStep("simple step", stats => stats.AllDataMB >= 0.01) }; var scenario = BuildScenario() .WithConcurrentCopies(1) .WithWarmUpDuration(TimeSpan.FromSeconds(0)) .WithDuration(TimeSpan.FromSeconds(2)) .WithAssertions(assertions); NBomberRunner.RegisterScenarios(scenario) .RunTest(); }
  43. 43. .Http https://nbomber.com PM> Install-Package NBomber.Http
  44. 44. var client = new HttpClient(); var step1 = Step.Create("GET html", async context => { var request = new HttpRequestMessage(HttpMethod.Get, "https://nbomber.com"); var response = await client.SendAsync(request); return response.IsSuccessStatusCode ? Response.Ok() : Response.Fail(); }); var scenario = ScenarioBuilder.CreateScenario("test_nbomber", step1); NBomberRunner.RegisterScenarios(scenario) .RunInConsole(); Http example
  45. 45. var step1 = HttpStep.Create("simple step", async (context) => Http.CreateRequest("GET", "https://nbomber.com") .WithHeader("accept", "application/json")); var scenario = ScenarioBuilder.CreateScenario("test_nbomber", step1); NBomberRunner.RegisterScenarios(scenario) .RunInConsole(); Nbomber.Http
  46. 46. Advanced Example
  47. 47. Batch Update Process 1 Batch Update Process 3 MongoDB Batch Update Process 2 write read read
  48. 48. var insertStep = Step.Create("insert", async context => { await mongoCollection.InsertManyAsync { ... }; }); var pause_200 = Step.Create("pause 200", async context => { await Task.Delay(TimeSpan.FromMilliseconds(200)); }); var readStep = HttpStep.Create("TOP 100 games", async (context) => Http.CreateRequest("GET", "https://sbtech/games?top=100") .WithHeader("accept", "application/json")); // insert scenario: batch insert per 200 ms var insertScenario = ScenarioBuilder.CreateScenario("mongo", insertStep, pause_200); // read scenario: in parallel, we want to read data var readScenario = ScenarioBuilder.CreateScenario("http", readStep); NBomberRunner.RegisterScenarios(insertScenario, readScenario) .RunInConsole();
  49. 49. The Kids want cluster
  50. 50. Agent Agent Coordinator MongoDB read write read
  51. 51. Test Scenarios.fs Compile Test Scenarios.EXE Upload Test Scenarios.EXE config Test Scenarios.EXE config
  52. 52. Test Scenarios.EXE config Test Scenarios.EXE config
  53. 53. Test Scenarios.EXE coordinator_config Test Scenarios.EXE agent_config
  54. 54. Test Scenarios.EXE coordinator_config Test Scenarios.EXE agent_config Test Scenarios.EXE agent_config
  55. 55. { "ClusterSettings": { "Agent": { "ClusterId": "test_cluster", "TargetGroup": "group_1", "MqttServer": "104.155.3.33" } } } Agent Agent Agent Config
  56. 56. Coordinator Config { "ClusterSettings": { "Coordinator": { "ClusterId": "test_cluster", "TargetScenarios": [ "mongo_insert" ] "Agents": [ { "TargetGroup": "group_1", "TargetScenarios": [ "mongo_read" ] } ] }, } } Coordinator
  57. 57. The Kids want PUSH
  58. 58. var url = "ws://localhost:53231"; var webSocketsPool = ConnectionPool.Create("webSocketsPool", openConnection: () => { var ws = new ClientWebSocket(); ws.ConnectAsync(new Uri(url), CancellationToken.None).Wait(); return ws; }); var pingStep = Step.Create("ping", webSocketsPool, async context => { var msg = new WebSocketRequest { CorrelationId = context.CorrelationId, RequestType = RequestType.Ping }; await context.Connection.SendAsync(msg, WebSocketMessageType.Text); return Response.Ok(); });
  59. 59. var pingStep = Step.Create("ping", webSocketsPool, async context => { var msg = new WebSocketRequest { CorrelationId = context.CorrelationId, RequestType = RequestType.Ping }; await context.Connection.SendAsync(msg, WebSocketMessageType.Text); return Response.Ok(); }); var pongStep = Step.Create("pong", webSocketsPool, async context => { while (true) { var message = await context.Connection.ReceiveMessage(); if (msg.CorrelationId == context.CorrelationId) { return Response.Ok(msg); } } });
  60. 60. Internals
  61. 61. NBomber Runner
  62. 62. let run (dep: Dependency, context: NBomberContext) = asyncResult { let! ctx = Validation.validateContext(context) let! nodeStats = NBomberContext.tryGetClusterSettings(ctx) |> Option.map(function | Coordinator c -> runClusterCoordinator(dep, ctx, c) | Agent a -> runClusterAgent(dep, ctx, a)) |> Option.orElseWith(fun _ -> runSingleNode(dep, ctx)) |> Option.get return nodeStats |> calcStatistics(dep, ctx) |> saveStatistics(ctx) |> applyAsserts(ctx) |> buildReport(dep) |> saveReport(dep, ctx) } |> AsyncResult.mapError(handleError(dep)) |> Async.RunSynchronously |> ignore
  63. 63. Validation
  64. 64. type Result<TOk, TError> = <!> <*> >>= >=>
  65. 65. module ScenarioValidation = let validate (context: NBomberContext) = context.Scenarios |> checkEmptyName >>= checkDuplicateName >>= checkEmptyStepName >>= checkDuplicateStepName >>= checkDuration >>= checkConcurrentCopies >>= fun _ -> Ok context
  66. 66. module GlobalSettingsValidation = let validate (context: NBomberContext) = context.NBomberConfig |> Option.bind(fun x -> x.GlobalSettings) |> Option.map(fun glSettings -> glSettings |> checkEmptyTarget >>= checkAvailableTarget >>= checkDuration >>= checkConcurrentCopies >>= checkEmptyReportName >>= fun _ -> Ok context ) |> Option.defaultValue(Ok context)
  67. 67. module ScenarioValidation = let validate (context) = ... module GlobalSettingsValidation = let validate (context) = ... module Validation = let validate = ScenarioValidation.validate >=> GlobalSettingsValidation.validate
  68. 68. Type Safe Cluster (>>=)
  69. 69. type IClusterCoordinator = abstract StartNewSession: ScenarioSetting[] -> Task<Result<unit,Error>> abstract WaitOnAllAgentsReady: unit -> Task<Result<unit,Error>> abstract StartWarmUp: unit -> Task<Result<unit,Error>> abstract StartBombing: unit -> Task<Result<unit,Error>> abstract GetStatistics: unit -> Task<Result<NodeStats[],Error>>
  70. 70. let runCoordinator (cluster: IClusterCoordinator, localHost: IScenariosHost, settings: ScenarioSetting[], targetScns: string[]) = asyncResult { do! cluster.SendStartNewSession(settings) do! localHost.InitScenarios(settings, targetScns) do! cluster.WaitOnAllAgentsReady() do! cluster.SendStartWarmUp() do! localHost.WarmUpScenarios() do! cluster.WaitOnAllAgentsReady() do! cluster.SendStartBombing() // Task<Result<unit,Error>> do! localHost.StartBombing() do! cluster.WaitOnAllAgentsReady() let localStats = localHost.GetStatistics() let! agentsStats = cluster.GetStatistics() let allStats = Array.append [|localStats|] agentsStats localHost.StopScenarios() return allStats }
  71. 71. - (2663) LOC - (6) OOP objects - (~160) functions Summary
  72. 72. THANKS @antyadev antyadev@gmail.com http://nbomber.com

×