Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

.NET Fest 2019. Андрей Винда. Создание REST API с поддержкой высокой нагрузки

59 views

Published on

Я хочу рассказать о нашем опыте создания высоконагруженного REST API, который имеет следующие параметры:
* Интуитивно понятный
* Масштабируемый
* Отказоустойчивой
* Защита от атаки
* Высокая пропускная способность
Также я расскажу о том:
* как мы делали нагрузочное тестирование
* как мы пережили множество DDoS-атак
* как мы сделали Blue-Green deploy
Использовался следующий список технологий: .NET Core 2, ASP.NET Core 2, Consul, Fabio, Orleans.NET, Kafka, RabbitMQ, Serilog, xMetrics, Grafana, ELK

Published in: Education
  • Be the first to comment

  • Be the first to like this

.NET Fest 2019. Андрей Винда. Создание REST API с поддержкой высокой нагрузки

  1. 1. Тема доклада Тема доклада Тема доклада KYIV 2019 Andrey Vinda How to build & support high load REST API .NET CONFERENCE #1 IN UKRAINE
  2. 2. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Betting revenue
  3. 3. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Load statistics Active users: 30 000 Passive users: 5000 Logins: 5000 (min)
  4. 4. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Highload
  5. 5. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Highload
  6. 6. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 15+ years of experience Many different products Like new technologies Launched several APIs About myself
  7. 7. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Business
  8. 8. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Business Business
  9. 9. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Users
  10. 10. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Opps, something went wrong
  11. 11. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Business needs APIs
  12. 12. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Login API Authenticate Fast response High throughput Resistance to DDoS and Brute Force attacks High Availability (99.99 %)
  13. 13. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Standard building blocks DATABASE CACHE SEARCH INDEXES QUEUE
  14. 14. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Application with stateless services
  15. 15. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Data shipping paradigm
  16. 16. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Stateless architecture: Pros Services are easy to scale No state in services When service server is dead – a new one could be up without any crucial affect
  17. 17. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Stateless architecture: Cons Add latency Non efficient access to data Limits scalability • Master / slave replication • Sharding Concurrency
  18. 18. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Stateless architecture with Cache: Pros Add latency Non efficient access to data Limits scalability •Master / slave replication •Sharding Concurrency Cache invalidation
  19. 19. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 What we want DATA LOCALITY EASY SCALING DEFENSE
  20. 20. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Defence Fast analyze of input requests In-Memory statistics of success/failed requests Calculation on the fly Fast and serializable access to the cache
  21. 21. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Data locality Moving compute to data is typically faster than moving data to compute For low latency flow Data intensive service
  22. 22. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 .NET alternatives
  23. 23. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Actors as the universal primitives of concurrent computation. Actor can: • Receive messages • Make local decisions • Create more actors • Send more messages • Determine how to respond to the next message received Actor model
  24. 24. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • .NET Framework used to create Scalable, Distributed, .NET Applications • Focused on low response latency and high concurrency • Usable in any .NET application (but frequently used with WebAPI Applications) • Based on a system of VirtualActors Orleans
  25. 25. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Virtual actors Actor instances always exist, virtually • Application neither creates nor deletes them. They never fail. • Code can always call methods on an actor Activations are created on-demand • If there is no existing activation, a message sent to it triggers instantiation • Transparent recovery from server failures • Lifecycle is managed by the runtime • Runtime can create multiple activations of stateless actors (for performance) Location transparency • Actors can pass around references to one another or persist them • These are logical (virtual) references, always valid, not tied to a specific activation
  26. 26. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 • Grains are C# classes • Derived from GrainBase • Implement an Interface (e.g. IPlayer, ISession) • Messages passing = Calling Interface methods • Ex: AddLoginSession (int playerId, Guid sessionId) • Can be Stateless or Stateful Orleans: Grain = Virtual actor
  27. 27. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Orleans: Grains Grains: Individually isolated objects that are messaging
  28. 28. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Orleans: Concurrency model for Grains Orleans uses a cooperative multithreading scheduler Scheduler schedules only one message at a time for a grain A message is processed completely before another message is scheduled A message is processed as a sequence of one or more turns (continuations)
  29. 29. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Transparent scalability by default New Silos can be added at any time Location transparency Multiplexed communication Efficient scheduling Orleans cluster Grains Silo
  30. 30. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Orleans cluster
  31. 31. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Smart cache Most popular pattern Use cases Read-only, write-though or write behind cache State usually is backed by persistence storage Orleans solution Actor per data item Time-based and/or explicit validation
  32. 32. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 An actor which keeps track of a set of other actors • Does not intervene in iteration with the individual actors • Its own state is just a list of references to other actors Registry
  33. 33. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Orleans grains Session Grain Player Grain Online Grain Operator Grain IP Grain Login Grain Cache layer Statistics layerOnline layer Grain Cache Service Operator Cache Grain Player Cache Grain
  34. 34. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Cache pre-population on startup Update cache on a regular basis (Kafka loader) In-memory statistics of all login attempts • Regulation specific data is being stored in DB All logged in players are in memory Time regulation features are based on Timer Orleans usage
  35. 35. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Demo
  36. 36. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Technologies
  37. 37. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Protection & Security HOW TO IDENTIFY POSSIBLE FRAUD? HOW TO PROTECT? WHAT TO ANALYZE? WHERE TO STORE DATA FOR ANALYZING?
  38. 38. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Protection INCAPSULA CAPTCHA BLOCKS
  39. 39. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Cookie challenge: If the client supports cookies, we respond to an HTTP request with a cookie. Web browsers typically will store and resend this cookie. Most bots do not support cookies and therefore will not respond. JS cookie challenge: After receiving an HTTP request, we respond with a JS cookie, instructing the browser to perform an action. Web browsers typically will execute the JavaScript instructions, on the other hand most bots do not support a JS engine and therefore will not respond Incapsula
  40. 40. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Captcha Send a CAPTCHA challenge, expecting a human response to the challenge LoginAPI generates captcha value for current session and stores it inside Orleans Use a non-white background with interspersed or roughened
  41. 41. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Fraud detection Analyze IP addresses User-Agent Amount of failed login attempts per minute Amount of success login attempts per minute
  42. 42. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Block players from the login base on the analysis of failed and success attempts per period Blocks
  43. 43. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Communication schema Client Login API JWT Secret Credentials Anonymous JWT Credentials Anonymous JWT OKAuth JWT
  44. 44. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Blue/Green deployment
  45. 45. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Blue/Green deployment Login API-A Login API-B Client Credentials * Cluster (default=A) * Cluster A Cluster B
  46. 46. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Observability Observability Logs Metrics Traces
  47. 47. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Orleans dashboard
  48. 48. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Metrics
  49. 49. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Metrics
  50. 50. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Metrics
  51. 51. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Fails • Kafka consumer in the main thread • Wrong configuration: connect App to Orleans • Load testing (NBomber, JMeter, Gatling) • Improper Captcha counters calculation
  52. 52. Тема доклада Тема доклада Тема доклада .NET LEVEL UP .NET CONFERENCE #1 IN UKRAINE KYIV 2019 Questions FB: https://www.facebook.com/andrey.vinda Email: vindaav@gmail.com Skype: vinda.andrew

×