Jet.com is an e-commerce startup competing with Amazon. We're heavy users of F#, and have based our architecture around Azure-based event-driven functional microservices. Over the last several months, we've schooled ourselves on what works and what doesn't for F# and microservices. This session will walk you through the lessons we have learned on our way to developing our platform.
3. We plan to be the new Amazon.com
Launched July 22, 2015
• Both Apple & Android named our app
as one of their top 5 for 2015
• Over 20k orders per day
• Over 10.5 million SKUs
• #4 marketplace worldwide
• 700 microservices
We’re hiring!
http://jet.com/about-us/working-at-jet
4. Azure Web sites
Cloud
services VMs Service bus
queues
Services
bus topics
Blob storage
Table
storage Queues Hadoop DNS Active
directory
SQL Azure R
F# Paket FSharp.Data Chessie Unquote SQLProvider Python
Deedle
FAK
E
FSharp.Async React Node Angular SAS
Storm Elastic
Search
Xamarin Microservices Consul Kafka PDW
Splunk Redis SQL Puppet Jenkins
Apache
Hive
EventStore
6. Microservices
An application of the single responsibility principle at the service level.
Has an input, produces an output.
Easy scalability
Independent releasability
More even distribution of complexity
Benefits
“A class should have one, and only one, reason to change.”
9. Events
Any significant change in
state that has happened
in your domain
• Past tense
• Immutable
• Contains only relevant
data to transaction
All events should be represented as
verbs in the past tense such as
CustomerRelocated,
CargoShipped, or
InventoryLossageRecorded.
For those who speak French, it should be
passé composé, they are things that have
completed in the past.
Greg Young
13. Compare to relational
model which captures
only the latest state
change. These sets
are then related to
each other.
Event-sourced
• Event-sourced is about how you model the domain.
• An append-only sequence of events as data store.
• Keep track of all state changes.
• Can REPLAY these event streams.
19. The F# solution offers us an order of magnitude
increase in productivity and allows one developer to
perform the work [of] a team of dedicated
developers…
Yan Cui
Lead Server Engineer, Gamesys
“
“ “
23. Concise & powerful code
public abstract class Transport{ }
public abstract class Car : Transport {
public string Make { get; private set; }
public string Model { get; private set; }
public Car (string make, string model) {
this.Make = make;
this.Model = model;
}
}
public abstract class Bus : Transport {
public int Route { get; private set; }
public Bus (int route) {
this.Route = route;
}
}
public class Bicycle: Transport {
public Bicycle() {
}
}
type Transport =
| Car of Make:string * Model:string
| Bus of Route:int
| Bicycle
C# F#
Trivial to pattern match on!
25. Concise & powerful code
public abstract class Transport{ }
public abstract class Car : Transport {
public string Make { get; private set; }
public string Model { get; private set; }
public Car (string make, string model) {
this.Make = make;
this.Model = model;
}
}
public abstract class Bus : Transport {
public int Route { get; private set; }
public Bus (int route) {
this.Route = route;
}
}
public class Bicycle: Transport {
public Bicycle() {
}
}
type Transport =
| Car of Make:string * Model:string
| Bus of Route:int
| Bicycle
| Train of Line:int
let getThereVia (transport:Transport) =
match transport with
| Car (make,model) -> ...
| Bus route -> ...
| Bicycle -> ...
Warning FS0025: Incomplete pattern
matches on this expression. For example,
the value ’Train' may indicate a case not
covered by the pattern(s)
C# F#
28. type Booking =
| Basic of Plane
| Combo of Combo
| FullPack of Plane * Hotel * Car
and Plane = {Outbound: DateTime; Return: DateTime; Destination: Country}
and Combo =
| ``With Hotel`` of Plane * Hotel
| ``With Car`` of Plane * Car
and Hotel = {Arrival: DateTime; Departure: DateTime; Location: Country}
and Car = {From: DateTime; To: DateTime; Location: Country}
and Country = {Name: String; ``ISO 3166-1``: char*char}
32. F# way 33
type Year = int
type [<Measure>] percent
type Customer = Simple | Valuable | MostValuable
type AccountStatus =
| Registered of Customer * since:Year
| Unregistered
let customerDiscount = function
| Simple -> 1<percent>
| Valuable -> 3<percent>
| MostValuable -> 5<percent>
let yearsDiscount = function
| years when years > 5 -> 5<percent>
| years -> 1<percent> * years
let accountDiscount = function
| Registered (customer, years) ->
customerDiscount customer, yearsDiscount years
| Unregistered -> 0<percent>, 0<percent>
let asPercent p = decimal p / 100m
let reducePriceBy discount price =
price - price * (asPercent discount)
let calculateDiscountedPrice price account =
let custDiscount, yrsDiscount =
accountDiscount account
price
|> reducePriceBy custDiscount
|> reducePriceBy yrsDiscount
34. Be functional!
Prefer immutability
Avoid state changes,
side effects, and
mutable data
Use data in data out
transformations
Think about mapping
inputs to outputs.
Look at problems
recursively
Consider successively
smaller chunks of the
same problem
Treat functions as
unit of work
Higher-order functions
35. Don’t abstract
This one magical service could write to ALL of the following:
…badly.
Event
store
Nservice
Bus
MSMQ 0MQ
SQL
Server
36. Isolate side effects
Submit order
microservice
Insert order to
SQL microservice
Send thank you
email microservice
Updates SQL
Sends “Thank you for
ordering” Email
Updates SQL
Sends “Thank you for
ordering” Email
37. Use a backup service to replay events
Service 1 runs in production as normal
Backup service 1 replays events until up-to-date.
Switch over. Instantly live with changes!
Also stage a copy of any aggregate/data store
until stream has completed replaying!
38. type Input =
| Product of Product
type Output =
| ProductPriceNile of Product * decimal
| ProductPriceCheckFailed of PriceCheckFailed
let handle (input:Input) =
async {
return Some(ProductPriceNile({Sku="343434"; ProductId = 17; ProductDescription = "My
amazing product"; CostPer=1.96M}, 3.96M))
}
let interpret id output =
match output with
| Some (Output.ProductPriceNile (e, price)) -> async {()} // write to event store
| Some (Output.ProductPriceCheckFailed e) -> async {()} // log failure
| None -> async {{ }} // log failure
let consume = EventStoreQueue.consume (decodeT Input.Product) handle interpret
What do our services look like?
Define inputs
& outputs
Define how input
transforms to output
Define what to do
with output
Read events,
handle, & interpret
39. Microservices should not control own lifecycle
Execution
runtime
Deployment Configuration Restarting
Versioning Scaling Availability
Grouping by
subsystem
Scheduling
Input-output
static analysis
Dashboard
Think
IoC!
40. Torch YAML files
torchVer: 2.0.0
subSystem: PriceCheck
name: PriceCheck
description: checks prices on nile
ver: 0.0.1
autoStart: always
compile: true
ha: aa ##active-active. Could be ap for active-passive
scriptPath: PriceCheckNilePriceCheckNile.fsx
libPath: binrelease
args: --jsonConfig=PriceCheckNile.json
It used to mean “Yet Another Markup Language” but was backronymed to clarify its focus as data-oriented.
YAML = “YAML Ain’t Markup Language”
42. For more information 43
F#
fsharp.org
fsharpforfunandprofit.com
Event sourcing
http://www.infoq.com/news/2014/09/greg-young-event-sourcing
https://www.youtube.com/watch?v=JHGkaShoyNs
Microservices
martinfowler.com
microservices.io
Right tool for the job company. Chaos testing is the right tool.
Azure:
Storage Q for At-least-once
Svc bus for At-most-once, transactions, persistence, order guarantee
and Kafka for event distribution
5-7 mins
More even distribution of complexity: easier to create & maintain services (very small amount of logic), harder to manage all services.
Shift complexity from biz logic to infrastructure. Better than one giant monolithic solution with 25 projects!
Before moving on, let’s define events.
-“Event” refers to both the event itself, as well as the notification message that’s passed to the rest of the system.
Considering events as notification messages.
This is push-based notification..
Reactive manifesto:
Responsive
Message-driven
Resilient
Elastic
Microservice input should treated as an Observable and event-sourced.
How many folks do event-sourcing now?
Account history or ledger.
Can replay and adjust if needed. – if need a ref number for all transactions now. CAN CREATE THIS
Found a bug.
Ran out of sockets.
Updated SQL schema.
Re-indexing Elasticsearch.
Why microservices?
F#: data in -> data out; inputs -> outputs. A function.
A microservice = just a function that naturally maps to an f# script, with input/out via the network.
Also, naming: Best practices = naming services after the function they perform.
Bad: “sku service” , good: “import skus"
No explicit meeting where we decided to do microservices, one day we just realized we had a bunch of microservices
Pattern #1.
Fewer bugs come from less code.
Rewrote into prod in 6 weeks!!
More features that I’ll talk about tomorrow.
NullReferenceException can be obviated because of option types.
Similar to nullable<int> but different because:
Can be used on any type (strings, functions)
When pattern matched on, forces you to consider None case
Can use map, iter, etc. functions.
Nestable: Option<Option<int>> is valid.
Option type is a simple discriminated union.
Both are idiomatic.
C# version still lacks structural equality.
Would need to override equality, comparison, AND GetHashCode.
Also not proper immutability because private set is still mutable.
-- there should be no setter, and the backing field should be readonly.
Pattern matching is like a switch statement, but WAY more powerful.
Option type is a simple discriminated union.
Both are idiomatic.
C# version still lacks structural equality (based on contents): that means overriding the equality implementation, the comparison implementation, AND GetHashCode. (generates a hash code from your object instance).
Also not proper immutability because private set is still mutable. To get real immutability, there should be no setter, and the backing field should be readonly.
Pattern matching is like a switch statement, but WAY more powerful.
Paul's team. BA asked for "notes from meeting", was actual code.
Look at the code a little closer
An article came out on CodeProject very recently.
https://twitter.com/Functional_S/status/709457140021399552
25 lines of code to
118 lines of code.
Doesn’t even show tests.
WTF WHY.
31 lines of code.
Once inputs and outputs are defined, the service is just a series of transformations. Can compose or pipe the operations. Trivial to make concurrent, to scale.
Will help Composing handlers and avoiding handler’s hell.
Subscription and handlers should be defined as a series of functions, outside the Micro Service. Functional! Using Async.Seq
OO Patterns: command pattern, visitor pattern are both just higher order functions.
HO fns often can more easily express a complex work flow.
(no slide, but) IDEMPOTENCY.
Where you can’t avoid side effects, isolate them!
Can’t replay this service easily if SQL schema changes or if you find a bug -- will resend all emails!
Aggregate = row in SQL table with latest aggregated info. Eg. Order shipped.
Keep staged version of aggregated info until stream has **completed** replaying.
Otherwise, you’re writing events to table. Might turn on, then turn off an account.
(again, isolate side effects)
If we change the SQL schema. Just replay events into new schema.
Keep a second service around. Spin that one up to replay.
Then switch over to it and live instant new schema.
Decode >> Handle >> interpret
Allows one fun to be called to implement.
Being functional allows composition here.
Use Docker, Consul, etc.
We rolled our own because:
We can deploy a new build in 30s (instead of 15-20 minutes for a cloud service.)
Scaling should include adding a new VM with assorted relevant services (automatically).
Active/active — Traffic intended for the failed node is either passed onto an existing node or load balanced across the remaining nodes. This is usually only possible when the nodes use a homogeneous software configuration.
Active/passive — Provides a fully redundant instance of each node, which is only brought online when its associated primary node fails.[1] This configuration typically requires the most extra hardware.