Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Tame cloud complexity with F# powered DSLs (build stuff)

437 views

Published on

The emergence of Cloud platforms has fundamentally changed the IT landscape. However, attempting to ride on this ever-expanding platform ecosystem wave has created a new set of challenges.

Join Yan Cui in this talk as he draws on his extensive experience with AWS over the last 7 years to illustrate, with real-world use examples, how you can use F# to build internal and external DSLs to tame the complexity from these cloud services.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Tame cloud complexity with F# powered DSLs (build stuff)

  1. 1. tame |> (cloud <| complexity) |> with |> (fsharp >> powered >> DSLs)
  2. 2. hi,I’mYanCui
  3. 3. AWS user since 2009
  4. 4. image by nerovivo license : https://creativecommons.org/licenses/by-sa/2.0/
  5. 5. Under-Abstraction
  6. 6. Oversimplification
  7. 7. Impedance Mismatch
  8. 8. Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
  9. 9. F# DSLs. Awesome!
  10. 10. Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
  11. 11. managed key-value store
  12. 12. redundancy 9-9s guarantee
  13. 13. great performance
  14. 14. name your throughput
  15. 15. can be changed on-the-fly
  16. 16. infinitely scalable (but you still have to pay for it)
  17. 17. Hash Key Range Key
  18. 18. Query: given a hash key filter on range key, or local secondary index
  19. 19. Hash Key Range Key Local Secondary Index Global Secondary Index
  20. 20. Scan: FULL TABLE search (performance + cost concern)
  21. 21. Hash Key Range Key Local Secondary Index Global Secondary Index
  22. 22. Hash Key Range Key Local Secondary Index Who are the TOP 3 players in “Starship X” with a score of at least 1000? Global Secondary Index
  23. 23. select GameTitle, UserId, TopScore from GameScores where GameTitle = “Starship X” and TopScore >= 1000 order desc limit 3 with (NoConsistentRead, Index(GameTitleIndex, true))
  24. 24. DynamoDB.SQL github.com/fsprojects/DynamoDb.SQL
  25. 25. GOAL Disguise complexity
  26. 26. GOAL Prevent abstraction leak
  27. 27. GOAL SELECT UserId, TopScore FROM GameScore WHERE GameTitle CONTAINS “Zelda” ORDER DESC LIMIT 3 WITH (NoConsistentRead)
  28. 28. Query AST Execution
  29. 29. F# & FParsec* *www.quanttec.com/fparsec External DSL via
  30. 30. SELECT * FROM GameScore Abstract Syntax Tree (AST) FParsec
  31. 31. SELECT * FROM GameScore keyword keyword * | attribute, attribute, … table name
  32. 32. SELECT * FROM GameScore type Attributes = | Asterisk | Attributes of string[]
  33. 33. SELECT * FROM GameScore type Query = { Attributes : Attributes Table : string }
  34. 34. SELECT * FROM GameScore Parser for “SELECT” keyword pSelect
  35. 35. SELECT * FROM GameScore pSelect let pSelect = skipStringCI "select"
  36. 36. SELECT * FROM GameScore pSelect let pSelect = skipStringCI "select" matches the string “select” (Case Insensitive) and ignores it
  37. 37. SELECT * FROM GameScore pFrom let pFrom = skipStringCI "from"
  38. 38. SELECT * FROM GameScore Parser for a string that represents the table name pTableName
  39. 39. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
  40. 40. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
  41. 41. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
  42. 42. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName
  43. 43. SELECT * FROM GameScore pTableName let isTableName = isLetter <||> isDigit let pTableName = many1Satisfy isTableName parses a sequence of one or more chars that satisfies the predicate function
  44. 44. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, …
  45. 45. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let pAsterisk = stringCIReturn "*" Asterisk
  46. 46. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let pAsterisk = stringCIReturn "*" Asterisk matches the specified string and return the given value
  47. 47. SELECT * FROM GameScore pAsterisk * pAttributeName UserId, GameTitle, TopScore, … let isAttributeName = isLetter <||> isDigit let pAttributeName = many1Satisfy isAttributeName
  48. 48. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pComma = skipStringCI ","
  49. 49. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pAttributeNames = sepBy1 pAttributeName pComma
  50. 50. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pCommapAsterisk * let pAttributeNames = sepBy1 pAttributeName pComma parses one or more occurrences of pAttributeName separated by pComma
  51. 51. SELECT * FROM GameScore UserId, GameTitle, TopScore, … pAttributeName pComma sepBy1 pAttributeNames pAsterisk *
  52. 52. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, …
  53. 53. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … let pAttribute = pAsterisk <|> pAttributeNames
  54. 54. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … let pAttribute = pAsterisk <|> pAttributeNames
  55. 55. SELECT * FROM GameScore pAsterisk * pAttributeNames UserId, GameTitle, TopScore, … choice pAttribute
  56. 56. SELECT * FROM GameScore pAttribute
  57. 57. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect
  58. 58. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
  59. 59. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
  60. 60. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table })
  61. 61. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect let pQuery = tuple4 pSelect pAttribute pFrom pTableName |>> (fun (_, attributes, _, table) -> { Attributes = attributes Table = table }) Query
  62. 62. SELECT * FROM GameScore pAttribute pTableNamepFrompSelect tuple4 pQuery
  63. 63. < 50 lines of code
  64. 64. Amazing F# + FParsec =
  65. 65. Recap
  66. 66. select GameTitle, UserId, TopScore from GameScores where GameTitle = “Starship X” and TopScore >= 1000 order desc limit 3 with (NoConsistentRead, Index(GameTitleIndex, true))
  67. 67. Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
  68. 68. Decision Worker
  69. 69. Decision Worker Poll
  70. 70. Decision Worker Decision Task
  71. 71. Decision WorkerDecide
  72. 72. Activity Worker Decision Worker
  73. 73. Activity Worker Decision Worker Poll
  74. 74. Activity Worker Decision Worker Activity Task
  75. 75. Activity Worker Decision Worker Complete
  76. 76. Workers can run from anywhere
  77. 77. input = “Yan” result = “Hello Yan!” Start Finish Activity
  78. 78. image by Ryan Hageman license : https://creativecommons.org/licenses/by-sa/2.0/
  79. 79. SWF-based Application
  80. 80. API
  81. 81. Heartbeats Error Handling Polling API
  82. 82. Activity Worker Decision Worker Heartbeats Error Handling Polling API
  83. 83. Activity Worker Decision Worker Heartbeats Error Handling Polling API Boilerplate
  84. 84. – Kris Jordan “Good simplicity is less with leverage, not less with less. Good simplicity is complexity disguised, not complexity denied.”
  85. 85. http://bit.ly/1pOLeKl
  86. 86. Start Finish Activity ?
  87. 87. the workflow is implied by decision worker logic…
  88. 88. instead..
  89. 89. the workflow should drive decision worker logic
  90. 90. the workflow should driveautomate decision worker logic
  91. 91. Amazon .SimpleWorkflow .Extensions github.com/fsprojects/Amazon.SimpleWorkflow.Extensions
  92. 92. GOAL Remove boilerplates
  93. 93. GOAL Code that matches the way you think
  94. 94. Start Finish Activity
  95. 95. Start Finish Activity
  96. 96. Workflows can be nested
  97. 97. input result
  98. 98. Recap
  99. 99. Activity Worker Decision Worker Heartbeats Error Handling Polling API
  100. 100. Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
  101. 101. wanna find correlations?
  102. 102. wanna find correlations? you can DIY it! ;-)
  103. 103. “what latencies spiked at the same time as payment service?”
  104. 104. Amazon .CloudWatch .Selector github.com/fsprojects/Amazon.CloudWatch.Selector
  105. 105. Find metrics whose 5 min average exceeded 1 second during last 12 hours
  106. 106. cloudWatch.Select( unitIs “milliseconds” + average (>) 1000.0 @ last 12 hours |> intervalOf 5 minutes)
  107. 107. cloudWatch.Select(“ unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes”)
  108. 108. “did any cache nodes’ CPU spike yesterday?”
  109. 109. cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes)
  110. 110. cloudWatch.Select( namespaceLike “elasticache” + nameLike “cpu” + max (>) 80.0 @ last 24 hours |> intervalOf 15 minutes) Regex
  111. 111. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
  112. 112. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes Filters
  113. 113. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes TimeFrame
  114. 114. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes Period
  115. 115. type Query = { Filter : Filter TimeFrame : TimeFrame Period : Period option }
  116. 116. Query Internal DSL External DSL
  117. 117. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
  118. 118. type MetricTerm = Namespace | Name type Filter = | MetricFilter of MetricTerm * (string -> bool)
  119. 119. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
  120. 120. type MetricTerm = Namespace | Name type Unit = | Unit type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool)
  121. 121. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
  122. 122. type MetricTerm = Namespace | Name type Unit = | Unit type StatsTerm = | Average | Min | Max | Sum | SampleCount type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool) | StatsFilter of StatsTerm * (float -> bool)
  123. 123. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
  124. 124. type MetricTerm = Namespace | Name type Unit = | Unit type StatsTerm = | Average | Min | Max | Sum | SampleCount type Filter = | MetricFilter of MetricTerm * (string -> bool) | UnitFilter of Unit * (string -> bool) | StatsFilter of StatsTerm * (float -> bool) | CompositeFilter of Filter * Filter
  125. 125. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
  126. 126. type TimeFrame = | Last of TimeSpan | Since of DateTime | Between of DateTime * DateTime
  127. 127. namespaceIs ‘JustEat’ and nameLike ‘cpu’ and unitIs ‘milliseconds’ and average > 1000.0 duringLast 12 hours at intervalOf 5 minutes
  128. 128. type Period = | Period of TimeSpan
  129. 129. type Query = { Filter : Filter TimeFrame : TimeFrame Period : Period option }
  130. 130. Active Patterns a primer on
  131. 131. allow patterns to be abstracted away into named functions
  132. 132. Single-Case Patterns
  133. 133. let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input
  134. 134. let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input
  135. 135. let (|Float|) input = match Double.TryParse input with | true, n -> n | _ -> failwithf “not a float [%s]” input Float : string -> float
  136. 136. match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
  137. 137. let (|Float|) input = match Double.TryParse input with | true, n -> n match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
  138. 138. let (|Float|) input = match Double.TryParse input with | true, n -> n match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
  139. 139. match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x let (|Float|) input = match Double.TryParse input with | true, n -> n
  140. 140. match someString with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x let (|Float|) input = match Double.TryParse input with | true, n -> n
  141. 141. match “42” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x
  142. 142. match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x Error!!!
  143. 143. Partial Patterns
  144. 144. let (|Float|_|) input = match Double.TryParse input with | true, n -> Some n | _ -> None
  145. 145. let (|Float|_|) input = match Double.TryParse input with | true, n -> Some n | _ -> None Float : string -> float option
  146. 146. match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x | _ -> “not a float”
  147. 147. match “boo” with | Float 42.0 -> “ftw” | Float 11.0 -> “palprime” | Float x -> sprintf “just %f” x | _ -> “not a float”
  148. 148. Multi-Case Patterns
  149. 149. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
  150. 150. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
  151. 151. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
  152. 152. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
  153. 153. let (|Prime|NotPrime|NaN|) input = match Double.TryParse input with | true, n when isPrime n -> Prime n | true, n -> NotPrime n | _ -> NaN
  154. 154. match someString with | Prime n -> … | NotPrime n -> … | NaN -> …
  155. 155. match someString with | Prime n & Float 11.0 -> … | Prime n -> … | Float 42.0 | Float 8.0 -> … | NotPrime n -> … | NaN -> …
  156. 156. match someString with | Prime n & Float 11.0 -> … | Prime n -> … | Float 42.0 | Float 8.0 -> … | NotPrime n -> … | NaN -> …
  157. 157. match someString with | Prime (IsPalindrome n) -> … | Prime (IsEven n) -> … | _ -> …
  158. 158. Tokenise (string -> string list)
  159. 159. [ “namespaceIs”; “‘JustEat’”; “and”; “nameLike”; “‘cpu’”; “and”; … ]
  160. 160. “namespaceIs” “and” … F# List = LinkedList “‘JustEat’”
  161. 161. “namespaceIs” “and” … F# List = LinkedList “namespaceIs” “‘JustEat’” “and”:::: :: tail cons (::) operator to prepend “‘JustEat’”
  162. 162. “namespaceIs” “and” … F# List = LinkedList “namespaceIs” “‘JustEat’” “and”:::: :: tail match aList with | “‘JustEat’” works as a pattern too!
  163. 163. let (|StringCI|_|) (str : string) (input : string) = if str.ToLower() = input.ToLower() then Some () else None let (|Float|_|) input = match System.Double.TryParse input with | true, n -> Some n | _ -> None
  164. 164. let (|EmptyString|_|) input = if String.IsNullOrWhiteSpace input then Some () else None let (|StartsWith|_|) char (input : string) = if input.[0] = char then Some () else None let (|EndsWith|_|) char (input : string) = if input.[input.Length-1] = char then Some () else None
  165. 165. let (|QuotedString|_|) (input : string) = match input with | EmptyString -> None | str when str.Length < 2 -> None | StartsWith ''' & EndsWith ''' -> Some <| input.Substring(1, input.Length - 2) | _ -> None
  166. 166. let (|QuotedString|_|) (input : string) = match input with | EmptyString -> None | str when str.Length < 2 -> None | StartsWith ''' & EndsWith ''' -> Some <| input.Substring(1, input.Length - 2) | _ -> None
  167. 167. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None
  168. 168. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None namespaceIs ‘JustEat’ and nameLike ‘cpu’ and …
  169. 169. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None “namespaceIs” “‘JustEat’” :::: …
  170. 170. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None type MetricTerm = Namespace | Name type Filter = | MetricFilter of MetricTerm * (string -> bool)
  171. 171. let (|NamespaceIs|_|) = function | StringCI “NamespaceIs" :: QuotedString ns :: tl -> Some (eqFilter MetricFilter Namespace ns, tl) | _ -> None “and” :: “nameLike” :: “‘cpu’” :: “and” :: …
  172. 172. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
  173. 173. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
  174. 174. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
  175. 175. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
  176. 176. let rec loop acc = function | NamespaceIs (filter, tl) | NamespaceLike (filter, tl) | NameIs (filter, tl) | NameLike (filter, tl) | UnitIs (filter, tl) | Average (filter, tl) | Sum (filter, tl) | Min (filter, tl) | Max (filter, tl) | SampleCount (filter, tl) -> match tl with | And tl -> loop (filter::acc) tl | _ -> flatten (filter::acc), tl | _ -> failwith “No filters?!?!?”
  177. 177. let parse (input : string) = input |> tokenize |> parseFilter |> parseTimeFrame |> parsePeriod
  178. 178. Amazing F# =
  179. 179. usable from anywhere you can run F# code e.g. F# REPL, executable, .. Internal DSL
  180. 180. useful for building tools e.g. CLI, … External DSL
  181. 181. Recap
  182. 182. Amazon DynamoDB Amazon SimpleWorkflow Amazon CloudWatch CASE STUDY
  183. 183. @theburningmonk theburningmonk.com github.com/theburningmonk

×