1. Yan Cui (@theburningmonk)
BOLOGNA 28 march 2015 LambdaCon
F# in the Real World

2. agenda

3. Hi, my name is Yan Cui.

5. 1MILLION USERS
ACTIVE
DAILY

6. 250MILLION DAY
PER
REQUEST

7. 2MONTH
TB
P E R
sec
ops
25,000

8. why F#?

9. time to market

10. correctness

11. efficient

12. tame complexity

13. 17th April!
London

15. Collectables
Wager Size
Special Symbol
Avg Wager Size
Web Server call

16. • Line Win!
– X number of matching symbols on adjacent columns!
– Positions have to be a ‘line’!
– Wild symbols substitute for other symbols!
!
• Scatter Win!
– X number of matching symbols anywhere!
– Triggers bonus game

17. What symbols should land?!
Any special symbol wins?!
Did the player win anything?

18. What the player’s new
average wager?

19. Should the player receive
collectables?

20. type Symbol = Standard! of string!
! ! ! | Wild

21. type Symbol = Standard! of string!
! ! ! | Wild
e.g.! Standard “hat”!
! Standard “shoe”!
! Standard “bonus”!
! …

22. type Symbol = Standard! of string!
! ! ! | Wild
i.e.! Wild

23. !
!
!
type Win = LineWin of int * Symbol * int!
! | ScatterWin of Symbol * int

24. !
!
!
type Win = LineWin of int * Symbol * int!
! | ScatterWin of Symbol * int
e.g.! LineWin (5, Standard “shoe”, 4)

25. !
!
!
type Win = LineWin of int * Symbol * int!
! | ScatterWin of Symbol * int
e.g.! ScatterWin (Standard “bonus”, 3)

26. type LineNum = int!
type Count! = int!
!
type Win = LineWin of LineNum * Symbol * Count!
! | ScatterWin of Symbol * Count

27. type LineNum = int!
type Count! = int!
!
type Win = LineWin of LineNum * Symbol * Count!
! | ScatterWin of Symbol * Count
e.g.! LineWin (5, Standard “shoe”, 4)

28. type LineNum = int!
type Count! = int!
!
type Win = LineWin of LineNum * Symbol * Count!
! | ScatterWin of Symbol * Count
e.g.! ScatterWin (Standard “bonus”, 3)

29. make invalid states
unrepresentable

30. NASA orbiter crashed
because one engineer
accidentally used miles
instead of kilometres

31. you’re never too smart
to make mistakes

32. [<Measure>]!
type Pence
e.g.! 42<Pence>!
! 153<Pence>!
! …

33. 10<Meter> / 2<Second> ! = 5<Meter/Second>!
10<Meter> * 2<Second> ! = 20<Meter Second> !
10<Meter> + 10<Meter> ! = 20<Meter>!
10<Meter> * 10! ! ! = 100<Meter>!
10<Meter> * 10<Meter> ! = 100<Meter2>!
10<Meter> + 2<Second> ! // error!
10<Meter> + 2 ! ! ! // error

34. 10<Meter> / 2<Second> ! = 5<Meter/Second>!
10<Meter> * 2<Second> ! = 20<Meter Second> !
10<Meter> + 10<Meter> ! = 20<Meter>!
10<Meter> * 10! ! ! = 100<Meter>!
10<Meter> * 10<Meter> ! = 100<Meter2>!
10<Meter> + 2<Second> ! // error!
10<Meter> + 2 ! ! ! // error

35. type Wager = int64<Pence>!
type Multiplier = int!
!
type Payout = Coins! of Wager!
! ! | MultipliedCoins of Multiplier * Wager!
! ! | Multi! of Payout list!
! ! | BonusGame

36. type Wager = int64<Pence>!
type Multiplier = int!
!
type Payout = Coins! of Wager!
! ! | MultipliedCoins of Multiplier * Wager!
! ! | Multi! of Payout list!
! ! | BonusGame

37. type Wager = int64<Pence>!
type Multiplier = int!
!
type Payout = Coins! of Wager!
! ! | MultipliedCoins of Multiplier * Wager!
! ! | Multi! of Payout list!
! ! | BonusGame

38. type State = !
{!
AvgWager! : Wager!
SpecialSymbol : Symbol!
Collectables : Map<Collectable, Count>!
}

40. New avg wager Got a Collectable!
A pay line win!

42. Betting small reduces avg wager!
Bonus Game!

45. type MonopolyBoardSpace = !
| RailRoad! ! of bool!
| Property! ! of int!
| ElectricCompany! of bool!
| WaterCompany! of bool!
| Chance! | CommunityChest!
| GotoJail! | Jail!
| FreeParking! | Go

46. type MonopolyBoardSpace = !
| RailRoad! ! of bool!
| Property! ! of int!
| ElectricCompany! of bool!
| WaterCompany! of bool!
| Chance! | CommunityChest!
| GotoJail! | Jail!
| FreeParking! | Go

47. type MonopolyBoardSpace = !
| RailRoad! ! of bool!
| Property! ! of int!
| ElectricCompany! of bool!
| WaterCompany! of bool!
| Chance! | CommunityChest!
| GotoJail! | Jail!
| FreeParking! | Go

48. type MonopolyBoardSpace = !
| RailRoad! ! of bool!
| Property! ! of int!
| ElectricCompany! of bool!
| WaterCompany! of bool!
| Chance! | CommunityChest!
| GotoJail! | Jail!
| FreeParking! | Go

49. [<Measure>]!
type Position!
!
type MonopolyBoard =!
{!
Spaces : MonopolyBoardSpace [ ]!
}!
member this.Item with !
get (pos : int<Position>) =!
this.Spaces.[int pos]

50. [<Measure>]!
type Position!
!
type MonopolyBoard =!
{!
Spaces : MonopolyBoardSpace [ ]!
}!
member this.Item with !
get (pos : int<Position>) = !
this.Spaces.[int pos]

52. And a pay line win!
Coin size brought over
from main game
Houses = multiplier on wins
GAME OVER
Collected in the bonus game.
Gives player extra ‘lives’.

53. type BonusGameState =!
{!
Board ! : MonopolyBoard!
Position ! : int<Position>!
Lives ! : int<Life>!
Doubles! : int<Double>!
CoinSize! : Wager!
TotalWin! : Wager!
}

54. Coin size brought over
from main game

55. type BonusGameState =!
{!
Board ! : MonopolyBoard!
Position ! : int<Position>!
Lives ! : int<Life>!
Doubles! : int<Double>!
CoinSize! : Wager!
TotalWin! : Wager!
}

56. And a pay line win!
Houses = multiplier on wins

57. type BonusGameState =!
{!
Board ! : MonopolyBoard!
Position ! : int<Position>!
Lives ! : int<Life>!
Doubles! : int<Double>!
CoinSize! : Wager!
TotalWin! : Wager!
}

60. Collected in the bonus game.
Gives player extra ‘lives’.

61. type BonusGameState =!
{!
Board ! : MonopolyBoard!
Position ! : int<Position>!
Lives ! : int<Life>!
Doubles! : int<Double>!
CoinSize! : Wager!
TotalWin! : Wager!
}

62. 0<Position>

63. let jail = 6<Position>

64. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

65. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

66. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

67. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

68. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

69. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

70. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

71. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

72. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
…

73. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| RailRoad true ! -> awardRailRoadMultiplier state!
| Property n when n > 0 -> awardPropertyMultiplier n state!
| ElectricityCompany true!
| WaterCompany true -> awardUtilityMultiplier state!
…

74. let rec move newPos state =!
match state.Board.[newPos] with!
…!
| CommunityChest! -> playCommunityChestCard state!
| Chance ! ! -> playChanceCard state!
…

75. let rec move newPos state =!
match state.Board.[newPos] with!
| RailRoad true ! -> awardRailRoadMultiplier state!
| Property n when n > 0 -> awardPropertyMultiplier n state!
| ElectricityCompany true!
| WaterCompany true -> awardUtilityMultiplier state!
| CommunityChest! -> playCommunityChestCard state!
| Chance ! ! -> playChanceCard state!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
| _ ! ! ! -> state

76. let rec move newPos state =!
match state.Board.[newPos] with!
| RailRoad true ! -> awardRailRoadMultiplier state!
| Property n when n > 0 -> awardPropertyMultiplier n state!
| ElectricityCompany true!
| WaterCompany true -> awardUtilityMultiplier state!
| CommunityChest! -> playCommunityChestCard state!
| Chance ! ! -> playChanceCard state!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
| _ ! ! ! -> state

77. let rec move newPos state =!
match state.Board.[newPos] with!
| RailRoad true ! -> awardRailRoadMultiplier state!
| Property n when n > 0 -> awardPropertyMultiplier n state!
| ElectricityCompany true!
| WaterCompany true -> awardUtilityMultiplier state!
| CommunityChest! -> playCommunityChestCard state!
| Chance ! ! -> playChanceCard state!
| GotoJail when state.Lives = 0<Life> -> !
! move jail state |> gameOver!
| GotoJail ! ! -> move jail state!
| _ ! ! ! -> state

79. Recap

80. lightweight syntax to
create types and
hierarchies

81. great for domain
modelling

82. make invalid states
unrepresentable

83. clear, concise way to
handle branching

84. better correctness

85. order of magnitude
increase in productivity

88. player states are big

89. Stateless Server DatabaseClient

90. 1:1 read-write ratio

91. stateless = !
Inefficient & Expensive

92. ServerClient
Session 1
Session 2

93. ServerClient Database

94. Elastic Load Balancer
S3
Auto scaling Group
Server A Server B
...
EC2
CloudFront
Stateful Server

95. The Actor Model
An actor is the fundamental unit of computation
which embodies the 3 things!
• Processing!
• Storage!
• Communication!
that are essential to computation.!
!
-Carl Hewitt*
* http://bit.ly/HoNHbG

96. The Actor Model
• Everything is an actor!
• An actor has a mailbox!
• When an actor receives a message it can:!
– Create new actors!
– Send messages to actors!
– Designate how to handle the next message

97. Stateful Server
• Gatekeeper!
– Manages the local list of active workers!
– Spawns new workers!
!
• Worker!
– Manages the states for a player!
– Optimistic locking!
– Persist state after period of inactivity

98. Stateful Server
Game Server
Player A
Player B
S3
Worker C
Worker B
Gatekeeper
RequestHandlers
Asynchronous

99. Stateful Server
Game Server
Worker C
Worker B
Gatekeeper
Worker A
ok
RequestHandlers
Player A
Player B
Asynchronous
S3

100. Stateful Server
Game Server
S3
Worker C
Worker B
Gatekeeper
Worker A
RequestHandlers
Player A
Player B
Asynchronous

101. Stateful Server
Game Server
S3
Worker C
Gatekeeper
Worker A
RequestHandlers
Player A
Player B
Asynchronous

102. Stateful Server
Game Server
Worker C
Worker A
Gatekeeper
error
RequestHandlers
Player A
Player B
S3
Asynchronous

103. type Agent<‘T> = MailboxProcessor<‘T>!
!
!
!

104. type Agent<‘T> = MailboxProcessor<‘T>!
!
type Message = !
| Get of AsyncReplyChannel<…>!
| Put of State * Version * AsyncReplyChannel<…>

105. type Agent<‘T> = MailboxProcessor<‘T>!
!
type Message = !
| Get of AsyncReplyChannel<…>!
| Put of State * Version * AsyncReplyChannel<…>

106. type Result<‘T> =!
| Success! of ’T!
| Failure! of Exception!
!
!
!

107. type Result<‘T> =!
| Success! of ’T!
| Failure! of Exception!
!
type GetResult = Result<State * Version>!
type PutResult = Result<unit>!

108. type Agent<‘T> = MailboxProcessor<‘T>!
!
type Message = !
| Get of AsyncReplyChannel<GetResult>!
| Put of State * Version * AsyncReplyChannel<PutResult>

109. type Agent<‘T> = MailboxProcessor<‘T>!
!
type Message = !
| Get of AsyncReplyChannel<GetResult>!
| Put of State * Version * AsyncReplyChannel<PutResult>

110. type Worker (playerId) =!
let agent = Agent<Message>.Start(fun inbox ->!
let state = getCurrentState playerId!
!
let rec workingState (state, version) = !
async { … }!
and closedState () =!
async { … }!
!
workingState (state, 1))

111. type Worker (playerId) =!
let agent = Agent<Message>.Start(fun inbox ->!
let state = getCurrentState playerId!
!
let rec workingState (state, version) = !
async { … }!
and closedState () =!
async { … }!
!
workingState (state, 1))

112. type Worker (playerId) =!
let agent = Agent<Message>.Start(fun inbox ->!
let state = getCurrentState playerId!
!
let rec workingState (state, version) = !
async { … }!
and closedState () =!
async { … }!
!
workingState (state, 1))

113. type Worker (playerId) =!
let agent = Agent<Message>.Start(fun inbox ->!
let state = getCurrentState playerId!
!
let rec workingState (state, version) = !
async { … }!
and closedState () =!
async { … }!
!
workingState (state, 1))

114. type Worker (playerId) =!
let agent = Agent<Message>.Start(fun inbox ->!
let state = getCurrentState playerId!
!
let rec workingState (state, version) = !
async { … }!
and closedState () =!
async { … }!
!
workingState (state, 1))

115. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
| None -> !
do! persist state!
return! closedState()!
…!
}!

116. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
| None -> !
do! persist state!
return! closedState()!
…!
}!

117. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
| None -> !
do! persist state!
return! closedState()!
…!
}!

118. non-blocking I/O

119. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
| None -> !
do! persist state!
return! closedState()!
…!
}!

120. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
| None -> !
do! persist state!
return! closedState()!
…!
}!

121. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
…!
| Some(Get(reply)) ->!
reply.Reply <| Success(state, version)!
return! workingState(state, version)!
…!
}!

122. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
…!
| Some(Put(newState, v, reply)) when version = v ->!
reply.Reply <| Success()!
return! workingState(newState, version+1)!
…!
}

123. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
…!
| Some(Put(_, v, reply)) ->!
reply.Reply <| Failure(VersionMismatch(version, v))!
return! workingState(state, version)!
}

124. let rec workingState (state, version) = !
async { !
let! msg = inbox.TryReceive(60000)!
match msg with!
| None ! ! ! ! ! ! -> …!
| Some(Get(reply)) ! ! ! ! -> …!
| Some(Put(newState, v, reply)) when version = v -> …!
| Some(Put(_, v, reply)) ! ! ! -> …!
}

125. and closedState () = !
async { !
let! msg = inbox.Receive()!
match msg with!
| Get(reply) ->!
reply.Reply <| Failure(WorkerStopped)!
return! closedState()!
| Put(_, _, reply) ->!
reply.Reply <| Failure(WorkerStopped)!
return! closedState()!
}

126. and closedState () = !
async { !
let! msg = inbox.Receive()!
match msg with!
| Get(reply) ->!
reply.Reply <| Failure(WorkerStopped)!
return! closedState()!
| Put(_, _, reply) ->!
reply.Reply <| Failure(WorkerStopped)!
return! closedState()!
}

128. 5x efficient
improvement

129. 60% latency drop

130. no databases

131. 90+% cost saving

132. need to ensure
server affinity

133. need to balance load

134. need to avoid
hotspots

135. • Persist player state after short inactivity!
• Move player to another server after persistence

136. need to gracefully
scale down

137. Recap

138. Agents!
• no locks!
• async message passing!
!

139. Agents!
• no locks!
• async message passing!
• self-contained!
• easy to reason with

140. Agents!
• low level!
• exceptions kills agents!
• primitive error monitoring!
• no supervision!
• no distribution

141. MS Orleans
Cricket
akka.Net

143. Async Workflow!
• non-blocking I/O!
• no callbacks

144. Pattern Matching!
• clear & concise

147. Caught a Gnome

148. EXP Item Gold
Quest Progress
Caught a Gnome

149. Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete

150. Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
New Quest
Quest Complete

151. Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New Quest
Level Up

152. Quest Progress
New Quest
Achievement
Progress
EXP Item Gold
Quest Complete
Level Up
Caught a Gnome

153. Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New Quest
Achievement
Progress
Achievement
Complete

154. Level Up
Quest Progress
EXP Item Gold
Caught a Gnome
Quest Complete
New Quest
Achievement
Progress
Achievement
Complete

155. 100+ actions

156. triggered by different
abstraction layers

157. non-functional
requirements

158. message-broker
pattern

159. Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner
Reporting

160. Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner
Reporting
Ignore
Process
Process
Process
Process
Ignore

161. Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner
Reporting
EXP
Item
Gold

162. Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner
Reporting
EXP
Item
Gold

163. Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner
Reporting
EXP
Item
Gold
Process
Ignore
Ignore
Ignore
Process
Ignore

164. Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner
Reporting
EXP
Item
Gold
Level Up

165. Caught Gnome Trapping
Queue
Levelling
Quests
Achievements
Analytics
Partner
Reporting
EXP
Item
Gold
Level Up

166. need lots of facts

167. type Fact =!
| GotExp! ! of Exp!
| GotGold! ! of Gold!
| GotItem! ! of Item * Count!
| CaughtMonster! of Monster * Bait * Location!
| LevelUp! ! of OldLevel * NewLevel!
…

168. type Reward =!
| GotExp! ! of Exp!
| GotGold! ! of Gold!
| GotItem! ! of Item * Count!
!
type StateChange =!
| LevelUp! ! of OldLevel * NewLevel!
…

169. type Fact =!
| StateChange! of StateChange!
| Reward! ! of Reward!
| Trapping! ! of Trapping!
| Fishing! ! of Fishing!
…

170. let process fact =!
match fact with!
| StateChange(stateChange)! -> …!
| Reward(reward)! ! ! -> …!
| Trapping(trapping)! ! -> …!
…

171. C# interop

172. …!
var fact = Fact.NewStateChange(!
! StateChange.NewLevelUp(oldLvl, newLvl));!
…

173. type IFact = interface end!
!
type Reward =!
| GotExp! ! of Exp!
| GotGold! ! of Gold!
| GotItem! ! of Item * Count!
interface IFact

174. type IFact = interface end!
!
type Reward =!
| GotExp! ! of Exp!
| GotGold! ! of Gold!
| GotItem! ! of Item * Count!
interface IFact

175. let process (fact : IFact) =!
match fact with!
| :? StateChange ! as stateChange!-> …!
| :? Reward ! ! as reward! ! -> …!
| :? Trapping! ! as trapping! -> …!
…!
| _ -> raise <| NotSupportedFact fact

176. let process (fact : IFact) =!
match fact with!
| :? StateChange ! as stateChange!-> …!
| :? Reward ! ! as reward! ! -> …!
| :? Trapping! ! as trapping! -> …!
…!
| _ -> raise <| NotSupportedFact fact

177. simple

178. flexible

179. saver

181. @theburningmonk
github.com/theburningmonk
theburningmonk.com

183. http://hire.jobvite.com/m?3OHq4hwL
F#
C#
AWS
Erlang
Docker
micro-services
elasticsearch
redis
Google AppEngine
Google
BigQuery
PostSharpNeo4j
Python DevOps
PagerDuty
NewRelic

184. BOLOGNA 28 march 2015 LambdaCon
Leave your feedback on Joind.in!
https://joind.in/13652