Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech

78 views

Published on

В 2017 году мы начали активно использовать F# для построения high-load push-based queryable API, а также обработки больших потоков данных (stateful stream processing). На тот момент времени никто в наших командах не имел предыдущего опыта по внедрению и применению F# но мы решили попробовать. На этом докладе я расскажу о нашем опыте внедрения F#, его проблемах и недостатках, о том как мы его научились готовить, где имеет смысл его применять и как подружить C#/OOP с F#/FP в одном проекте.
Данный доклад нацелен на аудиторию не имеющую предыдущего опыта с FP/F#.
Agenda:
- Why did we choose F# over C#?
- A high-level overview of the architecture of our push-based queryable API.
- Adopting F# for C#/OOP developers (inconveniences, C# interoperability, code style, DDD, TDD)

Published in: Education
  • Be the first to comment

  • Be the first to like this

.NET Fest 2018. Антон Молдован. One year of using F# in production at SBTech

  1. 1. About me: @AntyaDev like types*
  2. 2. - Biggest provider in sport offering - Supports a lot of regulated markets - About 1000 microservices (200 distinct types) - 5 datacenters maintained fully by SBTech - About 500 concurrent live events at pick time - On average we handle about 100K+ RPS
  3. 3. highload and near real time
  4. 4. Why F#?
  5. 5. Why F#? If we have sexy C#
  6. 6. If we have sexy C# class Person { public string FirstName; // Not null public string? MiddleName; // May be null public string LastName; // Not null }
  7. 7. If we have sexy C# switch (shape) { case Circle c: WriteLine($"circle with radius {c.Radius}"); break; case Square s when (s.Length == s.Height): WriteLine($"{s.Length} x {s.Height} square"); break; case Rectangle r: WriteLine($"{r.Length} x {r.Height} rectangle"); break; }
  8. 8. If we have sexy C# public (string, string) LookupName(long id) { // some code.. return (first, middle); } var names = LookupName(id); WriteLine($"found {names.first} {names.last}.");
  9. 9. Why F#? If we have sexy C#
  10. 10. 1 day
  11. 11. Story of a new C# project Get("/api/bets", async request => { var result = Validator.Validate(request); // unit testable if (result.IsOk) { var response = await _httpClient.Call(request); return BetsMapper.MapResponse(response); // unit testable } else return Errors.MapError(result); // unit testable });
  12. 12. 1 day
  13. 13. In 1 week
  14. 14. public class BetsProviderService : IBetsProviderService { readonly IOpenBetsRepository _openBetsRepository; readonly ISettledBetsRepository _settledBetsRepository; public BetsProviderService(IOpenBetsRepository openBetsRepository, ISettledBetsRepository settledBetsRepository) { _openBetsRepository = openBetsRepository; _settledBetsRepository = settledBetsRepository; } } Story of a new C# project
  15. 15. Different Culture OOP FP abstraction extensibility purity composability correctness
  16. 16. Triggers: Much better tooling support Community growth Recursive modules
  17. 17. Tooling support Visual Studio RiderVisual Studio Code
  18. 18. Recursive modules type Plane() = member x.Start() = PlaneLogic.start(x) // types and modules can't currently be mutually // referential at all module private PlaneLogic = let start (x: Plane) = ...
  19. 19. Recursive modules module M type First = | T of Second and Second = | T of First
  20. 20. Recursive modules module M type First = | T of Second type Second = | T of First
  21. 21. Recursive modules module rec M type First = | T of Second type Second = | T of First
  22. 22. Recursive modules module M let private helper () = ... let publicApiFunc () = helper()
  23. 23. Recursive modules module rec M let publicApiFunc () = helper() let private helper () = ...
  24. 24. Community growth • To begin, F# has grown to be bigger than ever, at least as far as we can measure, through product telemetry, twitter activity, GitHub activity, and F# Software Foundation activity. • Active unique users of F# we can measure are in the tens of thousands. • Measured unique users of Visual Studio Code with Ionide increased by over 50% this year, to become far larger than ever. • Measured unique users of Visual Studio who use F# increased by over 20% since last year to be larger than ever, despite quality issues earlier in the year that we believe have inhibited growth. • Much of the measured growth coincides with the release of .NET Core 2.0, which has shown significant interest in the F# community.
  25. 25. namespace SportData.ETL.Core type ItemRawUpdate<'Item> = { Id: string Data: 'Item option } module TransformationFlow = let getUpdates (feed: ChangeFeed, lastOffset: uint32) = Code Style
  26. 26. Code sample let createGame (globalCache: GlobalItemsCache, masterEvent: MasterEventItem, currentTime: DateTime) = trial { let! league = Cache.get(globalCache.Leagues, masterEvent.LeagueID) let! country = Cache.get(globalCache.Countries, league.CountryID) let! sport = Cache.get(globalCache.Branches, masterEvent.BranchID) ... return { Id = masterEvent.ID.ToString() Type = getEventType(masterEvent) } }
  27. 27. I need website Website SBTech story
  28. 28. I need an API to build unique UI The Problem Oops?!
  29. 29. The Problem Oops?! 1. well defined contracts (Event, Market, Selection) 2. flexible way to query data and subscribe on changes 3. push updates notifications about changes 4. near real time change delivery (1 sec delay) We need to provide:
  30. 30. PUSH based Queryable API (Change Feed)
  31. 31. select * from Games where isLive = true order by totalBets desc limit 10 PUSH changes
  32. 32. Change Feed (Actor) Change Stream Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool
  33. 33. Subscribers Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool Change Log Query Change Feed (Actor)
  34. 34. Subscribers type NodeName = string type LastHeartBeat = DateTime type NodeSubscribers = Map<NodeName, LastHeartBeat> type FeedStatus = | ReadyForDeactivation | HasSubscribers | NoSubscribers type ChangeFeed = { Id: FeedId View: FeedView ChangeLog: ChangeLog Subscribers: NodeSubscribers Query: Query Status: FeedStatus } Query Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool Change Feed Change Log
  35. 35. module ChangeFeed let getSnapshot (feed: ChangeFeed, fastPreloadAmount: uint32) = ... let getUpdates (feed: ChangeFeed, lastOffset: uint32) = ... let getLatestUpdate (feed: ChangeFeed) = ... let subscribe (feed: ChangeFeed, name: NodeName, currentTime: DateTime) = ... let subscribeByOffset (feed: ChangeFeed, name: NodeName, lastOffset: uint32) = ... let unsubscribe (feed: ChangeFeed, name: NodeName) = ... let getInactiveNodes (feed: ChangeFeed, currentTime: DateTime, inactivePeriod: TimeSpan) = ... let reloadView (feed: ChangeFeed, queryResults: QueryResult seq) = ...
  36. 36. Feed View 2:2 1:0 Chelsea - Milan Milan - Liverpool type PartialView = { EntityType: EntityType Entities: IEntity seq } type FeedView = { OrderType: EntityType OrderIds: string seq OrderFilter: FilterFn option Views: Map<EntityType, PartialView> } let tryCreate (queryResults: QueryResult seq, orderType: EntityType, orderFilter: FilterFn option) = match queryResults with | NotMatchFor orderType -> fail <| Errors.OrderTypeIsNotMatch | Empty -> ok <| { OrderType = orderType; OrderIds = Seq.empty OrderFilter = orderFilter; Views = views } | _ -> ok <| ...
  37. 37. [<Property>] let ``empty ChangeLog should start with offset 0 and requested maxSize`` (maxSize: uint32) = let changeLog = ChangeLog.create(maxSize) Assert.Equal(0u, changeLog.Offset) Assert.Equal(maxSize, changeLog.MaxSize) [<Property(Arbitrary=[|typeof<Generators.Generator>|])>] let ``all changes in changeLog.stream should have UpdateType.Update`` (payloads: Payload array) = let mutable changeLog = ChangeLog.create(5u) for p in payloads do changeLog <- ChangeLog.append(changeLog, p) let result = changeLog.Stream.All(fun change -> change.Type = UpdateType.Update) Assert.True(result) TDD without Mocks
  38. 38. We have our own frontend but we need a Data The Next Challenge Oops?!
  39. 39. SBTech All changes Operator
  40. 40. RDBMS All changes from 40 tables ETL Operator Well defined entities compose entities react on changes recalculate odds (data locality)
  41. 41. C# F#Asp NET Core Orleans DB Drivers [ DDD; Tests] Infrastructure business logic
  42. 42. public Task<FullFeedUpdate> GetSnapshot(GetSnapshot msg) { // invoke F# code from C# var updates = ChangeFeedModule.getSnapshot(FeedState, msg.FastPreloadAmount); return updates; }
  43. 43. Applied at the heart of the system Zero Null Reference exceptions Domain errors instead of exceptions DDD with types (strong determinism) Dependency Rejection TDD without mocks (property based testing)
  44. 44. http://nbomber.com
  45. 45. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  46. 46. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  47. 47. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  48. 48. - Technology agnostic (no dependency on HTTP, WebSockets, SSE) - Very simple API (to test Pull and Push scenarios) - CI/CD integration
  49. 49. type Step = | Request of RequestStep // to model Request-response pattern | Listener of ListenerStep // to model Pub/Sub pattern | Pause of TimeSpan // to model pause in your test flow type TestFlow = { FlowName: string Steps: Step[] ConcurrentCopies: int } type Scenario = { ScenarioName: string TestInit: Step option TestFlows: TestFlow[] Duration: TimeSpan }
  50. 50. let execStep (step: Step, req: Request, timer: Stopwatch) = task { timer.Restart() match step with | Request r -> let! resp = r.Execute(req) timer.Stop() let latency = Convert.ToInt64(timer.Elapsed.TotalMilliseconds) return (resp, latency) | Listener l -> let listener = l.Listeners.Get(req.CorrelationId) let! resp = listener.GetResponse() timer.Stop() let latency = Convert.ToInt64(timer.Elapsed.TotalMilliseconds) return (resp, latency) | Pause time -> do! Task.Delay(time) return (Response.Ok(req), 0L) }
  51. 51. type TestFlowRunner(flow: TestFlow) = let createActors (flow: TestFlow) = flow.CorrelationIds |> Set.toArray |> Array.map(fun id -> TestFlowActor(id, flow)) let actors = createActors(flow) member x.Run() = actors |> Array.iter(fun x -> x.Run()) member x.Stop() = actors |> Array.iter(fun x -> x.Stop()) member x.GetResult() = actors |> Array.collect(fun actor -> actor.GetResults()) |> TestFlowStats.create(flow)
  52. 52. // C# example var flow = new TestFlow(flowName: "READ Users", steps: new[] {step1, step2, step3}, concurrentCopies: 10); new ScenarioBuilder(scenarioName: "Test MongoDb") .AddTestFlow(flow) .Build(duration: TimeSpan.FromSeconds(10)) .RunInConsole();
  53. 53. // F# example let flow = { FlowName = "GET flow" Steps = [step1; step2; step3] ConcurrentCopies = 100 } Scenario.create("Test HTTP https://github.com") |> Scenario.addTestFlow(flow) |> Scenario.withDuration(TimeSpan.FromSeconds(10.0)) |> Scenario.runInConsole
  54. 54. var httpClient = new HttpClient(); var httpStep = Step.CreateRequest("GET request", execute: async _ => { var response = await httpClient.SendAsync(request); return response.IsSuccessStatusCode ? Response.Ok() : Response.Fail(); }); var pause_1s = Step.CreatePause(duration: TimeSpan.FromSeconds(1)); var scenario = new ScenarioBuilder("Test HTTP") .AddTestFlow("PULL", new[] { httpStep, pause_1s }, concurrentCopies: 10) .Build(duration: TimeSpan.FromSeconds(10)); scenario.RunInConsole();
  55. 55. RDBMS Storage Kafka ETL Kafka NoSQL Storage Push API
  56. 56. - 1470 LOC - (4) OOP objects - (~100) functions - try/catch used 2 times
  57. 57. async { let! x = myFunction() match x with | Some v -> ... | None -> ... } F# vNext match!
  58. 58. async { match! myFunction() with | Some v -> ... | None -> ... } F# vNext match!
  59. 59. F# vNext Struct Records type Vector3 = { X: float; Y: float; Z: float } [<Struct>] type Vector3 = { X: float; Y: float; Z: float }
  60. 60. F# vNext Struct Records [<Struct>] type Result<'T,'TError> = | Ok of 'T | Error of 'Terror [<Struct>] type ValueOption<'T> = | Some of ‘T | None
  61. 61. F# vNext Anonymous Records type Data = { X: int Y: string } let data = { X = 1; Y = "abc" }
  62. 62. F# vNext Anonymous Records let data = {| X = 1; Y = "abc" |}
  63. 63. F# vNext Structural Type System type User() = member this.GetName() = "test user" type Car() = member this.GetName() = "test car" let inline printName (x: ^T when ^T : (member GetName: unit -> string)) = x.GetName() |> Console.WriteLine
  64. 64. F# vNext Applicative functors validate { let! name = validateName(name) let! age = validateAge(age) let! email = validateEmail(email) return User(name, age, email) }
  65. 65. F# vNext Applicative functors validate { let! name = validateName(name) and! age = validateAge(age) and! email = validateEmail(email) return User(name, age, email) }
  66. 66. F# vNext Applicative functors parallel { let! x = slowRequestX() and! y = slowRequestY() and! z = slowRequestZ() return f(x, y, z) }
  67. 67. THANKS @antyadev antyadev@gmail.com @nbombertest http://nbomber.com

×