Yan Cui, profile picture
Uploaded byYan Cui
21,078 views

Tame cloud complexity with F#-powered DSLs

The document discusses various aspects of using Amazon DynamoDB, AWS Simple Workflow, and CloudWatch along with F# Domain-Specific Languages (DSLs) for managing complex workflows and data queries efficiently. It highlights the importance of abstraction to simplify interactions with these services while avoiding complexity leaks. Additionally, it provides examples of queries and configuration for settings in DynamoDB and workflows in AWS, emphasizing the need for performance and scalability.

Embed presentation

Downloaded 51 times
1 / 204
2 / 204
3 / 204
4 / 204
5 / 204
6 / 204
7 / 204
8 / 204
9 / 204
10 / 204
11 / 204
12 / 204
13 / 204
14 / 204
15 / 204
16 / 204
17 / 204
18 / 204
19 / 204
20 / 204
21 / 204
22 / 204
23 / 204
24 / 204
25 / 204
26 / 204
27 / 204
28 / 204
29 / 204
30 / 204
31 / 204
32 / 204
33 / 204
34 / 204
35 / 204
36 / 204
37 / 204
38 / 204
39 / 204
40 / 204
41 / 204
42 / 204
43 / 204
44 / 204
45 / 204
46 / 204
47 / 204
48 / 204
49 / 204
50 / 204
51 / 204
52 / 204
53 / 204
54 / 204
55 / 204
56 / 204
57 / 204
58 / 204
59 / 204
60 / 204
61 / 204
62 / 204
63 / 204
64 / 204
65 / 204
66 / 204
67 / 204
68 / 204
69 / 204
70 / 204
71 / 204
72 / 204
73 / 204
74 / 204
75 / 204
76 / 204
77 / 204
78 / 204
79 / 204
80 / 204
81 / 204
82 / 204
83 / 204
84 / 204
85 / 204
86 / 204
87 / 204
88 / 204
89 / 204
90 / 204
91 / 204
92 / 204
93 / 204
94 / 204
95 / 204
96 / 204
97 / 204
98 / 204
99 / 204
100 / 204
101 / 204
102 / 204
103 / 204
104 / 204
105 / 204
106 / 204
107 / 204
108 / 204
109 / 204
110 / 204
111 / 204
112 / 204
113 / 204
114 / 204
115 / 204
116 / 204
117 / 204
118 / 204
119 / 204
120 / 204
121 / 204
122 / 204
123 / 204
124 / 204
125 / 204
126 / 204
127 / 204
128 / 204
129 / 204
130 / 204
131 / 204
132 / 204
133 / 204
134 / 204
135 / 204
136 / 204
137 / 204
138 / 204
139 / 204
140 / 204
141 / 204
142 / 204
143 / 204
144 / 204
145 / 204
146 / 204
147 / 204
148 / 204
149 / 204
150 / 204
151 / 204
152 / 204
153 / 204
154 / 204
155 / 204
156 / 204
157 / 204
158 / 204
159 / 204
160 / 204
161 / 204
162 / 204
163 / 204
164 / 204
165 / 204
166 / 204
167 / 204
168 / 204
169 / 204
170 / 204
171 / 204
172 / 204
173 / 204
174 / 204
175 / 204
176 / 204
177 / 204
178 / 204
179 / 204
180 / 204
181 / 204
182 / 204
183 / 204
184 / 204
185 / 204
186 / 204
187 / 204
188 / 204
189 / 204
190 / 204
191 / 204
192 / 204
193 / 204
194 / 204
195 / 204
196 / 204
197 / 204
198 / 204
199 / 204
200 / 204
201 / 204
202 / 204
203 / 204
204 / 204
@theburningmonk tame |> (cloud <| complexity) |> with |> (fsharp >> powered >> DSLs)
Principal Engineer
image by nerovivo license : https://creativecommons.org/licenses/by-sa/2.0/
@theburningmonk Under-Abstraction
@theburningmonk Oversimplification
@theburningmonk Impedance Mismatch
@theburningmonk Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
@theburningmonk F# DSLs. Awesome!
@theburningmonk Amazon DynamoDB Amazon SimpleW Amazon CloudWatch CASE STUDY
@theburningmonk
@theburningmonk managed key-value store
@theburningmonk redundancy 9-9s guarantee
@theburningmonk great performance
@theburningmonk name your throughput
@theburningmonk
@theburningmonk can be changed on-the-fly
@theburningmonk
@theburningmonk infinitely scalable (but you still have to pay for it)
@theburningmonk
Hash Key Range Key
@theburningmonk Query: given a hash key filter on range key, or local secondary index
@theburningmonk
Hash Key Range Key Local Secondary Index Global Secondary Index
@theburningmonk Scan: FULL TABLE search (performance + cost concern)
@theburningmonk
Hash Key Range Key Local Secondary Index Global Secondary Index
Hash Key Range Key Local Secondary Index Who are the TOP 3 players in “Starship X” with a score of at least 1000? Global Secondary Index
@theburningmonk
@theburningmonk select GameTitle, UserId, TopScore from GameScores where GameTitle = “Starship X” and TopScore >= 1000 order desc limit 3 with (NoConsistentRead, Index(GameTitleIndex, true))
DynamoDB.SQL github.com/fsprojects/DynamoDb.SQL
@theburningmonk GOAL Disguise complexity
@theburningmonk GOAL Prevent abstraction leak
@theburningmonk GOAL SELECT UserId, TopScore FROM GameScore WHERE GameTitle CONTAINS “Zelda” ORDER DESC LIMIT 3 WITH (NoConsistentRead)
@theburningmonk Query AST Execution
F# & FParsec* *www.quanttec.com/fparsec External DSL via
@theburningmonk
@theburningmonk SELECT * FROM GameScore Abstract Syntax Tree (AST) FParsec
@theburningmonk SELECT * FROM GameScore keyword keyword * | attribute, attribute, … table name
@theburningmonk SELECT * FROM GameScore type Attributes = | Asterisk | Attributes of string[]
@theburningmonk SELECT * FROM GameScore type Query = { Attributes : Attributes Table : string }
@theburningmonk SELECT * FROM GameScore Parser for “SELECT” keyword pSelect
@theburningmonk SELECT * FROM GameScore pSelect let pSelect = skipStringCI "select"
@theburningmonk SELECT * FROM GameScore pSelect let pSelect = skipStringCI "select" matches the string “select” (Case Insensitive) and ignores it
@theburningmonk SELECT * FROM GameScore pFrom let pFrom = skipStringCI "from"
@theburningmonk SELECT * FROM GameScore Parser for a string that represents the table name pTableName
@theburningmonk SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
@theburningmonk SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
@theburningmonk SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
@theburningmonk SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
@theburningmonk SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName parses a sequence of one or more chars that satisfies the predicate function
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, …
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let pAsterisk = stringCIReturn "*" Asterisk
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let pAsterisk = stringCIReturn "*" Asterisk matches the specified string and return the given value
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let isAttributeName = isLetter <||> isDigit let pAttributeName = many1Satisfy isAttributeName
@theburningmonk SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pComma = skipStringCI ","
@theburningmonk SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pAttributeNames = sepBy1 pAttributeName pComma
@theburningmonk SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pAttributeNames = sepBy1 pAttributeName pComma parses one or more occurrences of pAttributeName separated by pComma
@theburningmonk SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pComma sepBy1 pAttributeNames pAsterisk *
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, …
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … let pAttribute = pAsterisk <|> pAttributeNames
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … let pAttribute = pAsterisk <|> pAttributeNames
@theburningmonk SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … choice pAttribute
@theburningmonk SELECT * FROM GameScore pAttribute
@theburningmonk SELECT * FROM GameScore pAttribute pTableNamepFrompSelect
@theburningmonk SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
@theburningmonk SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
@theburningmonk SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
@theburningmonk SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table }) Query
@theburningmonk SELECT * FROM GameScore pAttribute pTableNamepFrompSelect tuple4 pQuery
@theburningmonk
@theburningmonk < 50 lines of code
@theburningmonk Amazing F# + FParsec =
@theburningmonk Recap
@theburningmonk
@theburningmonk select GameTitle, UserId, TopScore from GameScores where GameTitle = “Starship X” and TopScore >= 1000 order desc limit 3 with (NoConsistentRead, Index(GameTitleIndex, true))
@theburningmonk Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
@theburningmonk
@theburningmonk Decision Worker
@theburningmonk Decision Worker Poll
@theburningmonk Decision Worker Decision Task
@theburningmonk Decision WorkerDecide
@theburningmonk Activity Worker Decision Worker
@theburningmonk Activity Worker Decision Worker Poll
@theburningmonk Activity Worker Decision Worker Activity Task
@theburningmonk Activity Worker Decision Worker Complete
@theburningmonk Workers can run from anywhere
@theburningmonk
@theburningmonk input = “Yan” result = “Hello Yan!” Start Finish Activity
@theburningmonk
image by Ryan Hageman license : https://creativecommons.org/licenses/by-sa/2.0/
SWF-based Application
API
Heartbeats Error Handling Polling API
Activity Worker Decision Worker Heartbeats Error Handling Polling API
Activity Worker Decision Worker Heartbeats Error Handling Polling API Boilerplate
– Kris Jordan “Good simplicity is less with leverage, not less with less. Good simplicity is complexity disguised, not complexity denied.”
http://bit.ly/1pOLeKl
Start Finish Activity ?
@theburningmonk the workflow is implied by decision worker logic…
@theburningmonk instead..
@theburningmonk the workflow should drive decision worker logic
@theburningmonk the workflow should driveautomate decision worker logic
Amazon .SimpleWorkflow .Extensions github.com/fsprojects/Amazon.SimpleWorkflow.Extensions
@theburningmonk GOAL Remove boilerplates
@theburningmonk GOAL Code that matches the way you think
@theburningmonk Start Finish Activity
Start Finish Activity
@theburningmonk Workflows can be nested
input result
@theburningmonk Recap
Activity Worker Decision Worker Heartbeats Error Handling Polling API
@theburningmonk Amazon DynamoDB Amazon SimpleW Amazon CloudWatch CASE STUDY
@theburningmonk
@theburningmonk
@theburningmonk
@theburningmonk wanna find correlations?
@theburningmonk wanna find correlations? you can DIY it! ;-)
@theburningmonk “what latencies spiked at the same time as payment service?”
Amazon .CloudWatch .Selector github.com/fsprojects/Amazon.CloudWatch.Selector
@theburningmonk Find metrics whose 5 min average exceeded 1 second during last 12 hours
@theburningmonk cloudWatch.Select( unitIs “milliseconds” + average (>) 1000.0 @ last 12 hours |> intervalOf 5 minutes)
@theburningmonk cloudWatch.Select(“ unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes”)
@theburningmonk “did any cache nodes’ CPU spike yesterday?”
@theburningmonk cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes)
@theburningmonk cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes) Regex
@theburningmonk
@theburningmonk
@theburningmonk
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes Filters
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes TimeFrame
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes Period
@theburningmonk type Query = { Filter : Filter TimeFrame : TimeFrame Period : Period option }
@theburningmonk Query Internal DSL External DSL
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
@theburningmonk type MetricTerm = Namespace | Name type Filter = | MetricFilter of MetricTerm * (string -> bool)
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
@theburningmonk type MetricTerm = Namespace | Name type Unit = | Unit type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool)
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
@theburningmonk type MetricTerm = Namespace | Name type Unit = | Unit type StatsTerm = | Average | Min | Max | Sum | SampleCount type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool) | StatsFilter of StatsTerm * (float -> bool)
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
@theburningmonk type MetricTerm = Namespace | Name type Unit = | Unit type StatsTerm = | Average | Min | Max | Sum | SampleCount type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool) | StatsFilter of StatsTerm * (float -> bool) | CompositeFilter of Filter * Filter
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
@theburningmonk type TimeFrame = | Last of TimeSpan | Since of DateTime | Between of DateTime * DateTime
@theburningmonk namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
@theburningmonk type Period = | Period of TimeSpan
@theburningmonk Active Patterns a primer on
@theburningmonk allow patterns to be abstracted away into named functions
@theburningmonk Single-Case Patterns
@theburningmonk let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input
@theburningmonk let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input
@theburningmonk let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input Float : string -> float
@theburningmonk match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
@theburningmonk let (|Float|) input = match Double.TryParse input with | true, n -> n match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
@theburningmonk let (|Float|) input = match Double.TryParse input with | true, n -> n match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
@theburningmonk match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x let (|Float|) input = match Double.TryParse input with | true, n -> n
@theburningmonk match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x let (|Float|) input = match Double.TryParse input with | true, n -> n
@theburningmonk match “42” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
@theburningmonk match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x Error!!!
@theburningmonk Partial Patterns
@theburningmonk let (|Float|_|) input = match Double.TryParse input with | true, n -> Some n | _ -> None
@theburningmonk let (|Float|_|) input = match Double.TryParse input with | true, n -> Some n | _ -> None Float : string -> float option
@theburningmonk match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x | _ -> “not a float”
@theburningmonk match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x | _ -> “not a float”
@theburningmonk Multi-Case Patterns
@theburningmonk let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
@theburningmonk let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
@theburningmonk let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
@theburningmonk let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
@theburningmonk let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
@theburningmonk match someString with | Prime n -> … | NotPrime n -> … | NaN -> …
@theburningmonk match someString with | Prime n & Float 11.0 -> … | Prime n -> … | Float 42.0 | Float 8.0 -> … | NotPrime n -> … | NaN -> …
@theburningmonk match someString with | Prime n & Float 11.0 -> … | Prime n -> … | Float 42.0 | Float 8.0 -> … | NotPrime n -> … | NaN -> …
@theburningmonk match someString with | Prime (IsPalindrome n) -> … | Prime (IsEven n) -> … | _ -> …
@theburningmonk Tokenise (string -> string list)
@theburningmonk [ “namespaceIs”; “‘JustEat’”; “and”; “nameLike”; “‘cpu’”; “and”; … ]
@theburningmonk Visual Studio time…
@theburningmonk Parse Filter (string list -> Filter * string list)
@theburningmonk let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
@theburningmonk let (|NamespaceIs|_|) = function | StringCI "NamespaceIs"::QuotedString ns::tl -> (eqFilter MetricFilter Namespace ns, tl) |> Some | _ -> None let (|NamespaceLike|_|) = function | StringCI "NamespaceLike"::QuotedString pattern::tl -> (regexFilter MetricFilter Namespace pattern, tl) |> Some | _ -> None
@theburningmonk Parse (string list -> Query)
@theburningmonk let parse (input : string) = input |> tokenize |> parseFilter |> parseTimeFrame |> parsePeriod
@theburningmonk Amazing F# =
@theburningmonk usable from anywhere you can run F# code e.g. F# REPL, executable, .. Internal DSL
@theburningmonk useful for building tools e.g. CLI, … External DSL
@theburningmonk
@theburningmonk
@theburningmonk
@theburningmonk Recap
@theburningmonk
@theburningmonk
@theburningmonk Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
@theburningmonk theburningmonk.com github.com/theburningmonk
is hiring :-) http://tech.just-eat.com/jobs

More Related Content

PDF
F# in the real world (NDC)
byYan Cui
 
PDF
The Magic Of Elixir
byGabriele Lana
 
PDF
F# in the real world (CodeMotion Dubai)
byYan Cui
 
PDF
Slaying the Dragon: Implementing a Programming Language in Ruby
byJason Yeo Jie Shun
 
KEY
Refactor like a boss
bygsterndale
 
PDF
F# at GameSys
byYan Cui
 
PDF
Investigating Python Wats
byAmy Hanlon
 
ODP
Very basic functional design patterns
byTomasz Kowal
 
F# in the real world (NDC)
byYan Cui
 
The Magic Of Elixir
byGabriele Lana
 
F# in the real world (CodeMotion Dubai)
byYan Cui
 
Slaying the Dragon: Implementing a Programming Language in Ruby
byJason Yeo Jie Shun
 
Refactor like a boss
bygsterndale
 
F# at GameSys
byYan Cui
 
Investigating Python Wats
byAmy Hanlon
 
Very basic functional design patterns
byTomasz Kowal
 

What's hot

PDF
A swift introduction to Swift
byGiordano Scalzo
 
PDF
Pooya Khaloo Presentation on IWMC 2015
byIran Entrepreneurship Association
 
PDF
Introducción rápida a SQL
byCarlos Hernando
 
PDF
optim function
bySupri Amir
 
PDF
Slides
byMohamed Mustaq Ahmed
 
PDF
Swift 함수 커링 사용하기
by진성 오
 
PPT
A little exercise with clojure macro
byZehua Liu
 
PDF
Nik Graf - Get started with Reason and ReasonReact
byOdessaJS Conf
 
PDF
The Ring programming language version 1.8 book - Part 96 of 202
byMahmoud Samir Fayed
 
PDF
RxSwift 시작하기
bySuyeol Jeon
 
PDF
To be Continued - multithreading with Project Loom and Kotlin's Coroutines
byArtur Skowroński
 
KEY
1 the ruby way
byLuis Doubrava
 
PDF
2 BytesC++ course_2014_c2_ flow of control
bykinan keshkeh
 
PDF
How fast ist it really? Benchmarking in practice
byTobias Pfeiffer
 
PDF
Elixir & Phoenix – fast, concurrent and explicit
byTobias Pfeiffer
 
PDF
Turn Hours into Seconds - Concurrent event processing in Elixir using Flow
byEmil Soman
 
PDF
Elixir & Phoenix – fast, concurrent and explicit
byTobias Pfeiffer
 
PDF
Good Code
byKevlin Henney
 
A swift introduction to Swift
byGiordano Scalzo
 
Pooya Khaloo Presentation on IWMC 2015
byIran Entrepreneurship Association
 
Introducción rápida a SQL
byCarlos Hernando
 
optim function
bySupri Amir
 
Slides
byMohamed Mustaq Ahmed
 
Swift 함수 커링 사용하기
by진성 오
 
A little exercise with clojure macro
byZehua Liu
 
Nik Graf - Get started with Reason and ReasonReact
byOdessaJS Conf
 
The Ring programming language version 1.8 book - Part 96 of 202
byMahmoud Samir Fayed
 
RxSwift 시작하기
bySuyeol Jeon
 
To be Continued - multithreading with Project Loom and Kotlin's Coroutines
byArtur Skowroński
 
1 the ruby way
byLuis Doubrava
 
2 BytesC++ course_2014_c2_ flow of control
bykinan keshkeh
 
How fast ist it really? Benchmarking in practice
byTobias Pfeiffer
 
Elixir & Phoenix – fast, concurrent and explicit
byTobias Pfeiffer
 
Turn Hours into Seconds - Concurrent event processing in Elixir using Flow
byEmil Soman
 
Elixir & Phoenix – fast, concurrent and explicit
byTobias Pfeiffer
 
Good Code
byKevlin Henney
 

Similar to Tame cloud complexity with F#-powered DSLs

PDF
MongoDB World 2019: Tips and Tricks++ for Querying and Indexing MongoDB
byMongoDB
 
PPTX
Introduction to SQL Antipatterns
byKrishnakumar S
 
PDF
Advanced query optimization
byMYXPLAIN
 
PPT
Drill / SQL / Optiq
byJulian Hyde
 
PPTX
Qure Tech Presentation
bysqlserver.co.il
 
PPT
Cypher
byMax De Marzi
 
PDF
Tame Cloud Complex with F# powered DSLs
byYan Cui
 
PDF
Tame cloud complexity with F# powered DSLs (build stuff)
byYan Cui
 
PDF
MongoDB .local Houston 2019:Tips and Tricks++ for Querying and Indexing MongoDB
byMongoDB
 
PDF
Rob Sullivan at Heroku's Waza 2013: Your Database -- A Story of Indifference
byHeroku
 
PPTX
Data
byRithikRaj25
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
byMongoDB
 
PDF
MongoDB .local Bengaluru 2019: Tips and Tricks++ for Querying and Indexing Mo...
byMongoDB
 
PDF
JDD 2016 - Tomasz Borek - DB for next project? Why, Postgres, of course
byPROIDEA
 
PDF
Dynamodb
byJean-Tiare LE BIGOT
 
PDF
The ultimate-guide-to-sql
byMcNamaraChiwaye
 
PPTX
Database Management System Review
byKaya Ota
 
PPTX
SQL Server Select Topics
byJay Coskey
 
ZIP
Postgres is easier
bygisborne
 
PDF
Dbms narrative question answers
byshakhawat02
 
MongoDB World 2019: Tips and Tricks++ for Querying and Indexing MongoDB
byMongoDB
 
Introduction to SQL Antipatterns
byKrishnakumar S
 
Advanced query optimization
byMYXPLAIN
 
Drill / SQL / Optiq
byJulian Hyde
 
Qure Tech Presentation
bysqlserver.co.il
 
Cypher
byMax De Marzi
 
Tame Cloud Complex with F# powered DSLs
byYan Cui
 
Tame cloud complexity with F# powered DSLs (build stuff)
byYan Cui
 
MongoDB .local Houston 2019:Tips and Tricks++ for Querying and Indexing MongoDB
byMongoDB
 
Rob Sullivan at Heroku's Waza 2013: Your Database -- A Story of Indifference
byHeroku
 
Data
byRithikRaj25
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
byMongoDB
 
MongoDB .local Bengaluru 2019: Tips and Tricks++ for Querying and Indexing Mo...
byMongoDB
 
JDD 2016 - Tomasz Borek - DB for next project? Why, Postgres, of course
byPROIDEA
 
Dynamodb
byJean-Tiare LE BIGOT
 
The ultimate-guide-to-sql
byMcNamaraChiwaye
 
Database Management System Review
byKaya Ota
 
SQL Server Select Topics
byJay Coskey
 
Postgres is easier
bygisborne
 
Dbms narrative question answers
byshakhawat02
 

More from Yan Cui

PDF
Patterns and practices for building resilient serverless applications.pdf
byYan Cui
 
PDF
Migrating existing monolith to serverless in 8 steps
byYan Cui
 
PDF
Patterns and practices for building resilient serverless applications
byYan Cui
 
PDF
Why your next serverless project should use AWS AppSync
byYan Cui
 
PDF
How to win the game of trade-offs
byYan Cui
 
PDF
Lambda and DynamoDB best practices
byYan Cui
 
PDF
How to choose the right messaging service
byYan Cui
 
PDF
How to choose the right messaging service for your workload
byYan Cui
 
PDF
How to bring chaos engineering to serverless
byYan Cui
 
PDF
A chaos experiment a day, keeping the outage away
byYan Cui
 
PDF
Build social network in 4 weeks
byYan Cui
 
PDF
Serverless observability - a hero's perspective
byYan Cui
 
PDF
Building a social network in under 4 weeks with Serverless and GraphQL
byYan Cui
 
PDF
Lessons from running AppSync in prod
byYan Cui
 
PDF
How to debug slow lambda response times
byYan Cui
 
PDF
How to ship customer value faster with step functions
byYan Cui
 
PDF
How to improve lambda cold starts
byYan Cui
 
PDF
FinDev as a business advantage in the post covid19 economy
byYan Cui
 
PDF
What can you do with lambda in 2020
byYan Cui
 
PDF
How serverless changes the cost paradigm
byYan Cui
 
Patterns and practices for building resilient serverless applications.pdf
byYan Cui
 
Migrating existing monolith to serverless in 8 steps
byYan Cui
 
Patterns and practices for building resilient serverless applications
byYan Cui
 
Why your next serverless project should use AWS AppSync
byYan Cui
 
How to win the game of trade-offs
byYan Cui
 
Lambda and DynamoDB best practices
byYan Cui
 
How to choose the right messaging service
byYan Cui
 
How to choose the right messaging service for your workload
byYan Cui
 
How to bring chaos engineering to serverless
byYan Cui
 
A chaos experiment a day, keeping the outage away
byYan Cui
 
Build social network in 4 weeks
byYan Cui
 
Serverless observability - a hero's perspective
byYan Cui
 
Building a social network in under 4 weeks with Serverless and GraphQL
byYan Cui
 
Lessons from running AppSync in prod
byYan Cui
 
How to debug slow lambda response times
byYan Cui
 
How to ship customer value faster with step functions
byYan Cui
 
How to improve lambda cold starts
byYan Cui
 
FinDev as a business advantage in the post covid19 economy
byYan Cui
 
What can you do with lambda in 2020
byYan Cui
 
How serverless changes the cost paradigm
byYan Cui
 

Recently uploaded

PDF
How to Evaluate a High Performance Database
byScyllaDB
 
PPTX
Building Agents in Microsoft Agent Framework.pptx
byUdaiappa Ramachandran
 
PPTX
communication-skills-with-technology tools
byJaleto Sunkemo
 
PPTX
Gemini vs ChatGPT : Comparing Top AI Models
byDigicarrom
 
PPTX
Top _HR Technology Trends _for 2026_.pptx
byAutomationEdge Technologies
 
PDF
Exam Prep Plan Overview: Amazon Web Services (AWS) Certified
byVICTOR MAESTRE RAMIREZ
 
PDF
CompTIA Cybersecurity Analyst (CySA+) CS0-003: Unit 5
byVICTOR MAESTRE RAMIREZ
 
PDF
Data Virtualization in Action: Scaling APIs and Apps with FME
bySafe Software
 
PDF
Day 4 - Access, Deployments, and Monitoring - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PPTX
Conversational Agents – Building Intelligent Assistants [Virtual Hands-on Wor...
byUiPathCommunity
 
PDF
Fundamentals and Frontiers of Post Quantum Cryptography
byGokul Alex
 
PPTX
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
PDF
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
PDF
Internet_of_Things_IoT_for_Next_Generation_Smart_Systems_Utilizing.pdf
byRoshanSyed12
 
PPTX
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
PDF
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PDF
AI Powered Document Processing and Data Extraction
byRobert McDermott
 
PDF
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
PDF
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
How to Evaluate a High Performance Database
byScyllaDB
 
Building Agents in Microsoft Agent Framework.pptx
byUdaiappa Ramachandran
 
communication-skills-with-technology tools
byJaleto Sunkemo
 
Gemini vs ChatGPT : Comparing Top AI Models
byDigicarrom
 
Top _HR Technology Trends _for 2026_.pptx
byAutomationEdge Technologies
 
Exam Prep Plan Overview: Amazon Web Services (AWS) Certified
byVICTOR MAESTRE RAMIREZ
 
CompTIA Cybersecurity Analyst (CySA+) CS0-003: Unit 5
byVICTOR MAESTRE RAMIREZ
 
Data Virtualization in Action: Scaling APIs and Apps with FME
bySafe Software
 
Day 4 - Access, Deployments, and Monitoring - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
Conversational Agents – Building Intelligent Assistants [Virtual Hands-on Wor...
byUiPathCommunity
 
Fundamentals and Frontiers of Post Quantum Cryptography
byGokul Alex
 
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
Internet_of_Things_IoT_for_Next_Generation_Smart_Systems_Utilizing.pdf
byRoshanSyed12
 
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
AI Powered Document Processing and Data Extraction
byRobert McDermott
 
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 

Tame cloud complexity with F#-powered DSLs

Editor's Notes

  • #2 good afternoon, and welcome to this talk on taming complex APIs with DSLs and how FSharp can help you build these DSLs easily.
  • #3 My name is Yan Cui and I often go by the online alias of ‘theburningmonk’
  • #4 I work for a company called Gamesys, we're based in central London and are one of the market leaders in the real-money gaming business. Me and my team focus on freemium games for a more social audience and as a backend developer, I have built the backend for a number of our social games on Facebook and mobile.
  • #5 Across our social games, we have around 1 million DAU and, 250 millions requests per day.
  • #6 Pretty much every user action in our games are recorded and analyzed, we capture around 2TB of data a month for analytics purpose alone, which doesn’t take into account the significant amount of data we generate and store to facilitate the actual gameplay.
  • #7 All our core game services are deployed to Amazon Web Services, and we make extensive use of its many services..
  • #8 which has given us first-hand experience of the different kind of…
  • #9 complexities that can arise from some of these APIs.
  • #10 Some complexities are visible, they tend to be the result of the inherent complexities with the operations that the API allows you to perform
  • #11 Whilst other APIs might appear to be simple at what it does but pushes the complexities to you instead. These complexities tend to surface only when you start working with the API.
  • #12 And sometimes it’s merely a case of an impedance mismatch between what the API designer think you’ll do and what you actually need from the API
  • #13 We’ll use three AWS APIs that we use regularly as case studies to illustrate the different types of complexities and how…
  • #14 ..F# can be an effective weapon in tackling these complexities by simplifying the task of creating both internal and external DSLs.
  • #15 We’ll start with DynamoDB…
  • #16 which is to this day still the fastest growing service that Amazon has and the DB of choice for our small team…
  • #17 It is a managed key-value store…
  • #18 with built-in redundancy and 9 9s guarantee…
  • #19 it has great performance thanks to the fact that it runs on SSD drives…
  • #20 It differs from other Amazon services in that it doesn’t operate with the usual pay-as-you-go model where you pay for actual amount of usage you have for any particular service…
  • #21 Instead, when you create a table, you specify the throughput you require, Amazon would reserve enough capacity to meet your requirements and you pay for the amount of resources Amazon has to reserve even if you don’t end up using all that throughput…
  • #22 Once created, you can still change the throughput of a table on the fly without impacting performance or needing downtime…
  • #23 and the amount you pay will be adjusted when you change the throughput settings.
  • #24 You can create a DB that can handle a million transactions/s with a few clicks but it will cost you dearly if you need that level of throughput 24/7. It does enable an interesting usage pattern though, I know of an advertising firm which sees little traffic all year round but gets 4 million hits/s during the Superbowl, they were able to bump their throughput requirements all the way up during the Superbowl and then change them back afterwards. They managed to get through the Superbowl for a few thousand bucks and didn’t need to create and maintain a super-scalable, but expensive infrastructure that goes unused 99% of the time.
  • #25 It is semi-schema’d which means the only schema info you need to provide is the shape of the data you are going to use as the key in your table…
  • #26 For every hash key, you can specify an additional range key which you can filter on when making a query against your data. Whilst you’ll most likely be doing simple CRUD operations against specific keys - like you do in a key-value store - DynamoDb supports a limited set of queriability.
  • #27 A query in DynamoDB must be provided with a hash key, accompanied with a set of filters against the range key. If you created a local secondary index on your table then you can also filter against the local secondary index instead. Speaking of indices…
  • #28 It supports two types of indices, local secondary index which gives you an alternative range key to query with. Whereas global secondary index effectively allows you to specify an alternative set of hash and range key for the whole table.
  • #29 In our example here, most of the time we’ll be accessing data for individual players, so we use the UserId as Hashkey, and allow us to query the player’s scores by the game Title. The local index allows us to also query, for a give user, his top score. The global index, allows us to use Game Title has the hash key, and the TopScore as the range key. Which makes it easy for us to do queries such as - give me the top 10 players for Donkey Kong by score.
  • #30 DynamoDB also supports full table scans, but these tend to eat up your provisioned throughput, and take a long time to complete. Whilst it’s running you’re also more likely to have your normal queries throttled by DynamoDB if the Scan takes up too much throughput.
  • #31 If you consult the documentation for the Query API, you quickly get the sense that it’s anything but straight forward
  • #32 As an example, take the table we saw earlier, if we were to ask for…
  • #33 The top 3 player in Starship X by their Top Score, we might write…
  • #34 this… which is a) a lot of code, and b) the code is not easily comprehensible because there are a lot of noise. wouldn’t it be nice if you can write something like…
  • #35 this instead? Which is far less code and more expressive of our intent.
  • #36 Since data access is such a common task, it makes sense for us to create a DAL layer that provides a nicer abstraction for our game developers to work with.
  • #37 It’s also a great place to implement Timeout, Circuit Breaker, Caching, and other useful patterns for better performance, resilience and fault tolerance
  • #38 However, it’s hard to abstract over a complex API like Query, so the game developers end up having to work with the request objects from the AWSSDK directly whenever they want to Query/Scan the table.
  • #39 Which means the low-level abstractions provided by the AWSSDK itself is now leaking into the abstraction above…
  • #40 which is why we created…
  • #41 DynamoDB.SQL The goals of the project include…
  • #42 Providing a DSL that hides the complexity of the query/scan API and to…
  • #43 prevent AWSSDK from leaking through to downstream code
  • #44 We wanted to create an external DSL that is SQL like, so that it’s familiar to most developers, esp someone new joining the team. To do so…
  • #45 we need to parse the query into an AST and then translate that AST into a request and execute it against DynamoDB
  • #46 To create this solution, we used FSharp with FParsec, which is a …
  • #47 Parser combinator library. It basically means that you create basic, simple parsers and then combine them using functions to create more complex parsers. The functions that combine parsers together are called combinators.
  • #48 Take this simple query for instance…
  • #49 We have SELECT and FROM as keywords, GameScore is the name of the table, and for attributes we can either use a wildcard to fetch all the attributes or provide the name of the attributes we want in a comma separated list.
  • #50 In the AST, we can represent the attributes as a FSharp…
  • #51 DU, which you can think of as an Enum where each name can be associated with an arbitrary data type.
  • #52 Here, we say that attributes can be either an Asterisk, or an array of strings
  • #53 As for the whole query, we need to know the attributes, the table name, and we can store them inside a FSharp…
  • #54 Record type, which is a lightweight data container where the fields are…
  • #55 immutable by default.
  • #56 To build a parser for this simple query, we first need a parser for the select keyword.
  • #57 in FParsec, you can use the skipStringCI function which…
  • #58 matches against the string “select” in a case insensitive way, and ignores it
  • #59 we can do the same with the FROM keyword
  • #60 for the table name..
  • #61 DynamoDB requires a table name to consist of only letters and digits, so…
  • #62 we create a predicate function by combining the isLetter and isDigit predicates built into FParsec, using…
  • #63 a custom combinator that combines two predicates using OR. Since isLetter and isDigit predicates work against individual characters in a string, so…
  • #64 to match a string that is NOT EMPTY, and consists of only letters and digits we use the FParsec combinator many1Satisfy and provide it with our isTableName predicate function
  • #65 This creates a parser for our table name
  • #66 For the attributes, we first need a parser for …
  • #67 asterisk, and the stringCIReturn function here differs from the skipStringCI function we saw earlier in that..
  • #68 when matched, it will return the specified value, in our case, when we match the * we will return the union case Asterisk
  • #69 to parse a comma separated list of attribute names, we need a parser for individual attribute names, similar to the table name parser we saw previously
  • #70 we will also need a parser for commas, which is another use of the skipStringCI function
  • #71 we can then combine them together using the sepBy1 combinator function to create a parser that…
  • #72 parses one or more attribute names separated by commas
  • #74 now that we have both parsers, time to combine them…
  • #75 with the choice combinator, which can be denoted as…
  • #76 this special operator, that says…
  • #77 an attribute can be either an asterisk or a comma separated list of attribute names
  • #78 we now have all…
  • #79 the parsers that we need, to put everything together…
  • #80 we use the tuple4 combinator which takes in..
  • #81 four parsers that need to match in the specified order, this creates a parser which outputs a tuple of 4 items. The output can then be forwarded using this…
  • #82 combinator operator to a function that maps the tuple into…
  • #83 the query type we defined earlier, hence creating a parser that returns an instance of the Query type
  • #84 You can then incrementally add on the WHERE, ORDER, LIMIT, and WITH clauses to the parser, all and all…
  • #85 the query syntax took less than…
  • #86 50 LOC, which just goes to show how…
  • #87 much productivity you get from using F# and FParsec.
  • #88 So to recap, we looked at…
  • #89 DynamoDB’s query operation as an example of a complex API that you can simplify with an external DSL…
  • #90 and how such a DSL can be easily implemented with F# and FParsec
  • #91 Next, let’s look at Amazon SimpleWorkflow, which presented a whole different kind of complexity.
  • #92 Simple Workflow is an orchestration service that manages the state of your workflow and provides the reporting, heartbeat and retry capabilities.
  • #93 To build an application on top of SWF, you need to implement a decision worker, which is responsible for…
  • #94 pulling tasks from the service…
  • #95 the decision task it gets back contains information about the current state of the workflow and the events that have occurred so far. Based on this, the worker needs to…
  • #96 decide what happens next in the workflow, whether the it should be completed, cancelled, or some more work needs to be scheduled. And when you need to do some work, you need an…
  • #97 activity worker, which also…
  • #98 pull for tasks…
  • #99 and the activity tasks it receives contains payloads for the activity that it needs to perform, be it to process some credit card payment or to encode some video, and when it’s done it…
  • #100 needs to report back to the service that the activity was completes. This triggers another decision task to be made available to the decision worker to process in order to decide what to do next.
  • #101 Your decision and activity workers doesn’t have to run from EC2, instead it can run from anywhere where you can access the SWF service.
  • #102 Everything that happens in a workflow execution is recorded and can be viewed from the management console, the input and result of every action, when it started and when it ended. You can retry failed executions, and workflows can be nested.
  • #103 As a hello world example, imagine we have a workflow with only one activity - which takes an input string and returns a string as result, here is what the implementation look like…
  • #104 which is pretty insane how much work was required, and we haven’t even implemented common concerns such as error handling and heartbeats. And the only thing in this craziness that is even relevant to what I wanted to get out of the service is…
  • #105 this line at the top…
  • #106 there’s gotta be a better way. Writing a truck load of code should never be your first-solution to a problem, every line of code you write has a cost - from the time it takes to write it, and then subsequently reading it, plus the time it takes to comprehend it, and then maintaining it. And considering that by making it easier for you to write more code, you’ve also made it easier for you to create cost to your company, which has a non-zero probability of outweighing the value that line of code brings.
  • #107 Before we look at our interpretation of a better way is, let’s look at the problem more closely. For any application that is built on top of Amazon simple workflow…
  • #108 first, there’s the service API…
  • #109 then there’re the recurring concerns such as polling, heartbeats, health checks and error handling…
  • #110 Then finally you have the things that are specific to your workflow, the activity and decision workers. That’s a lot of things you need to implement just to get going, but really, the only thing that is unique to your workflow are the activities it needs to perform, and everything else is…
  • #111 just glorified boilerplate. So whilst Simple Workflow’s APIs are pretty simple, as a consumer of the API, there are a lot of complexities that have been pushed down to you. Which reminds me of a post that Kris Jordan wrote a while back called ‘the complexity of simplicity’…
  • #112 where he said that…(read)…he went on to expand on this point by talking about how Google hides all the complexities around internet search and gives its users a simple UI with which they can do more with less. The guys at Stripe has done a very similar thing with taking credit card payments.
  • #113 It’s a really good read, here’s a bit.ly link to the post for you to have a look later on.
  • #114 in the decision worker implementation earlier, you know, the thing that orchestrates the workflow, plumbing aside, the highlighted block of code is responsible for scheduling the activity, and completing or failing the workflow after the activity either completes or fails. Looking at this block of code, I can’t easily reconstruct the mental model I had…
  • #115 of the workflow I set out to implement? The problem here is that…
  • #116 the workflow itself is never made explicit, but rather, implied by the logic that’s coded up in the decision worker, which is…
  • #117 far from ideal. Instead, the mental model you have of a workflow should be…
  • #118 driving the decision worker logic, or better, it should…
  • #119 automate the decision worker logic…
  • #120 and that’s why we created…
  • #121 a simple workflow extensions library which aims to…
  • #122 remove the boilerplate code you need to write, and…
  • #123 allow you to write code that actually matches your mental model of workflows
  • #124 and with that, let’s have a look how we’d implement our hello world example using our DSL
  • #125 The DSL allows you to configure the Workflow and attach activities to it using a simple…
  • #126 arrow-like operator, and when you read this code, you just follow the arrow to see…
  • #127 what your workflow does
  • #128 which each activity you add, you need to provide some configuration values and a delegate function which will be called when an activity task is received. It also automatically registers the workflow and activities with the service, something that you’d have to do manually otherwise
  • #129 Simple workflow allows you to nest workflows, so you can start and wait for another workflow to complete before continuing on in the parent workflow, to do this in the DSL…
  • #130 you might write something along the lines of…
  • #131 notice how we created a child workflow here, with its own activties, and…
  • #132 simply added it to a parent workflow as if it’s another activity
  • #133 the DSL takes care of the propagation of input and results so that the result of the previous activity or child workflow is passed as the input for the next activity or workflow
  • #134 In case of exceptions, it also takes care of capturing exception details and reporting back to the service so that you can see what went wrong in the management console
  • #135 you can optionally specify how many times an activity or child workflow should be retried
  • #136 Now, to recap, we looked at…
  • #137 Simple Workflow and how its simple API ends up pushing a lot of the complexities to you, the consumer, and how we addressed these complexities with…
  • #138 a super-simple internal DSL in FSharp that consists of an operator, and two types, that…
  • #139 …removes the need for boilerplate, and lets you create workflows with code that visually informs of what the workflow is going to do. It takes care of the heavy lifting involved so that you can…
  • #140 focus on building things that actually add value to your customers
  • #141 Finally, let’s have a look at Amazon CloudWatch..
  • #142 CloudWatch is a monitoring service that lets you collect and monitor metrics, and set alarms on those metrics. You can monitor anything from CPU usage, to database latencies, as well as any custom metrics you track from your application.
  • #143 It also comes with a nice web UI for browsing your metrics and charting them. It’s an immensely important and useful service, but, there are shortcomings…
  • #144 Presumably for performance reasons, the web UI limits you to 200 metrics when browsing, beyond that you have to know what you’re looking for to find it, so discovery is out of the question, but more importantly…
  • #145 to find correlations between events such as latency spikes, you have to manually search for, and inspect every latency metric yourself…
  • #146 it’s DevOps where you do all the monkey work, and you can’t even easily answer simple questions such as…
  • #147 …(read)… driven by pain and anger we decided to automate with…
  • #148 Amazon.CloudWatch.Selector, which gives us the ability to express queries using both internal and external DSLs
  • #149 For instance, if we want to find latency metrics that, at some point in the last 12 hours, exceeded a 1s average…
  • #150 we can express our question using an internal DSL that is very humanly readable, or as an…
  • #151 external DSL that is very similar in syntax and identical in capability
  • #152 or, suppose you want to find out if any of your cache nodes had a CPU spike in the last 24 hours…
  • #153 you might try something that targets specific metrics, in our case, CPU metrics under the Amazon ElastiCache namespace, and…
  • #154 whenever you see function names that end with ‘like’ it means they support regex
  • #155 overall the syntax for the DSL is quite small…
  • #156 as you can see, the internal DSL is implemented in a handful of lines…
  • #157 the external DSL is slightly more involved, and this time around I chose to implement it using active patterns, which is a…
  • #158 language feature in FSharp that allows you to abstract away, and give names to patterns so that they can be easily reused in pattern matching…
  • #159 there’s the single-case patterns which is similar to a normal function, this pattern here…
  • #160 matches against a string parameter called input, and either returns a float, or errors…
  • #161 and we’ll give this pattern a name, enclosed in what we call the banana clip operators…
  • #162 this pattern has a type signature of taking in a string, and returning a float, to use this pattern you could apply it…
  • #163 in a match…with clause where you can stack them up, or even compose different patterns together…
  • #164 and notice that the return value of the pattern can be bound and used in the pattern matching too…
  • #165 but, if someone pass in a string that can’t be parsed to a float…
  • #166 then this code will error. But sometimes you need a pattern that doesn’t always match, which is when you can use a partial active pattern instead…
  • #167 So here we can rewrite the earlier pattern to return an option…
  • #168 so that if we’re able to parse the string as a float, we’ll…
  • #169 the value of the float as Some, which is equivalent to Haskell’s Just.
  • #170 And when we can’t parse it as a float, then we return None, which is again, equivalent to Haskell’s Nothing.
  • #171 So now we can rewrite our pattern matching code earlier with an additional clause that catches cases where the input string…
  • #172  cannot be parsed as a float.
  • #173 and finally, if the input value cannot be classified into just something or nothing, then you can use the multi-case active pattern…
  • #174 here we’ll try to parse the input string as a float and if the resulting float is a prime number, then return it as a Prime…
  • #175 alternatively, return it with as a NotPrime…
  • #176 and if we couldn’t even parse the input then return NaN instead.
  • #177 and from here you can use these patterns in pattern matching as you would with any other. You can even compose them together to make your pattern matching code even more expressive…
  • #178 and from here you can use these patterns in pattern matching as you would with any other. You can use active patterns to really easily create parsers, and compose them to make more interesting parsers like we did with FParsec…
  • #179 one of the most pleasing thing for me was that I was able to rely on the type system to guide me along the way and after I wrote both DSLs in a 2 hour sprint everything compiled and worked as expected at the first time of asking! Which gave me a beautiful feeling, and just what I needed at 4am!
  • #180 so even on its own using only vanilla functions and active patterns, F# is still awesome at building DSLs with
  • #181 for the internal DSL, it can be used from anywhere you can run F# code, including the REPL, or executables…
  • #182 the external DSL is mainly useful for building tools with, such as a CLI…
  • #183 which one is available via Chocolatey
  • #184 with the CLI tool, you can write your queries using the external DSL syntax and if we find any matching metrics, you can plot them on graphs for visual inspection
  • #186 To recap, we looked at CloudWatch and how its impedance mismatch renders routine post-mortem investigations into an exercise…
  • #187 of finding a needle in the haystack, and how simple internal and external DSLs make that needle…
  • #188 that much easier to find
  • #189 and that concludes our case studies, but F#’s awesomeness doesn’t end here
  • #190 in fact, we have used F# to build a number of tools and frameworks for working with specific Amazon services, including…
  • #191 a type provider for S3…
  • #193 Which allows you to interactively navigate, browse objects in your S3 account, without even leaving Visual Studio, and you get full intellisense support on the buckets and objects you find
  • #194 you can also search for items in S3, which is useful for buckets with large number of objects, our buckets for player states have millions of objects, each with up to thousands of versions.
  • #195 we also have Reacto-Kinesix, a framework…
  • #196 for building real-time stream processing applications with Amazon Kinesis service
  • #197 and Metricano, a library for…
  • #198 for collecting and publishing custom metrics to services such as Amazon CloudWatch, whereby you can…
  • #199 either track metrics manually from your code or to use PostSharp aspects to auto instrument your methods to record execution count and time metrics. These metrics can then be hooked up to publishers such as the builtin CloudWatch publisher or…
  • #200 one of your own.
  • #201 You can use the PostSharp aspects to target individual methods, or multi-cast to all methods in a class or assembly to apply tracking to all your service entry points, or database calls…
  • #202 and to publish the metrics to CloudWatch, you just need to call Publish.With with an instance of CloudWatchPublisher
  • #203 and then you’ll be able to see your metrics on the CloudWatch management console, simple!
  • #204 and with that, thank you very much for listening
  • #205 We have the first ever FSharp Exchange in London next April, so hope to see you there.
  • #206 If you have any questions for me after today, feel free to get in touch