Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Microservices and FP
Michael Neale
@michaelneale
developer.cloudbees.com
Wednesday, 9 October 13
Microservices and FP
Given that:
We have lots of services
- Everything has an API
Functional programming:
- quality improv...
Definitions
Functional programming:
- avoid/manage side effects
- immutability over mutation
- higher order functions
- first...
Definitions
Micro-service:
- remotely accessible service (typically http)
- does “one thing” (and does it well?)
- executes...
Definitions
Micro-service:
- “small”
- independently deployed
- loosely coupled
- not necessarily reusable (but may be)
Wed...
Definitions
Like a unix command line app that does ONE THING
AND ONE THING ONLY!
service1 | service2 | service3 ....
Wednes...
Example scenario
Wednesday, 9 October 13
Jenkins CI
PaaS repos autoscale
management console web apps
how to compose???
provisioning
licencingsubscriptions
Wednesda...
Identify some micro
services
repos: artifact storage
- store blobs in users account (via s3)
autoscale:
- look at a stream...
All in common:
All small!
All do one thing!
Wednesday, 9 October 13
Aside: FP (more later)
Some services lend themselves to “purity”
eg: Automatic scaling
F(last minute stats) = recommendati...
Why not monolithic?
Can easily do
@Produces(MediaType.JSON)
@Path("/apps")
@POST
...
Thanks to jax-rs in one app? why not?...
Why not monolithic?
Apps grow (teams grow too!)
Different parts change at different rates
Different execution container needs...
Why not monolithic?
Fear.
Risk.
Fear of deploying a change.
Risk of trying a new technique, new framework.
Wednesday, 9 Oc...
Why not monolithic?
Deploying a change to a microservice does not
increase blood pressure as much as deploying the
ONE MAS...
Wednesday, 9 October 13
Monolithic debugging
(Often in production)
???
Wednesday, 9 October 13
micro-service debugging
Wednesday, 9 October 13
Shift to dev-ops
More deployment expected
More *people* doing the deployment
Wednesday, 9 October 13
Smooth deployment
Microservices need:
- easy rollback
- no “over the wall” deployment
- easy as-needed deployment
- access...
Transport
Typically (but not always) HTTP
Typically (but not always) JSON
(autoscale service was message bus + thrift name...
Containers
Execution environment:
- anything really
- consumers of APIs don’t care
- but developers do, deployers do
- JVM...
Convinced?
By this point I hope I have convinced you
microservices are good.
Right?
Wednesday, 9 October 13
Wednesday, 9 October 13
Functional Programming
with microservices
First, some background...
Wednesday, 9 October 13
Context
New Company
New Team
New Product
You might not be this “lucky”
Wednesday, 9 October 13
History
2010 Started: JVM stack parts - Scala not
controversial (I had experience) - working mostly
“lone wolf”
2011 - ano...
Erlang!
Me
other team members
Wednesday, 9 October 13
More FP
Given “success” with Scala, more FP (services,
languages) was not viewed as a great risk.
Wednesday, 9 October 13
Observation
FP means one person can do more &&
People like me like to work alone
∴ risk of staying with one-person-per mic...
Objections and resistance
Ask why:
- resistance to the “new” (unnecessary risk)
- resistance to FP ideas (hype?)
- polyglo...
Maintainability objection
It goes:
- how will anyone be able to maintain this after you?
My Experience:
- projects featuri...
Where we may differ
Small teams - many systems.
∴ Little overlap in jobs.
We get to use a great PaaS !
(our own!) for all ...
What did we do with FP
Manage horrendous public cloud APIs
Automatic scaling (microservice)
Github crawling (microservice)...
Surprisingly practical things
No real calculations, no explicit maths.
Just boring every day error prone stuff.
Wednesday, ...
For example (provisioning)
Wednesday, 9 October 13
Providore Evil cloud
api
Build masters
build workers Workspace
storage
Wednesday, 9 October 13
New build required. Check the pool, talk to the Evil
api, ask for a new server, it fails, ask again. Wait for
server to be...
How can we solve this?
TDD? problems only manifest under load/in-
production. APIs are buggy, change over time.
Industry b...
Yes
Types (providore is written in scala)
- Specifically: Option, Either
- Closed Data Types (servers only in so many state...
Cloud API
launch_server: Server
at best hopes to be:
launch_server: Option[Server]
launch_server: Either[Server, OhGodWhyW...
Cloud Monad
Cloud APIs are like IO
Slow, horrible, misbehaving IO
...and then the APIs other people write
All want to be m...
val validation: String / Subscription = (for {
account <- extractAccount
_ <- validateSubscription(account)
callback <- ex...
Types
So hard to catch things without them
Monadic IO + types mean you catch things before
you try
Trying/experimenting ca...
Types helped with
Ignored messages
Bad pattern matching
Misconfiguration/timing of server creation
Avoiding “stringly typed...
Types didn’t help with...
Wednesday, 9 October 13
def ec2
Fog::Compute.new(:provider => 'AWS',
:aws_secret_access_key => ENV['EC2_SECRET_KEY'],
:aws_access_key_id => ENV['E...
True story
Wednesday, 9 October 13
Currying
Server lifecycle: Reserved->Launching->Update
(user data)->Volume Create->Volume Attach-
>initialise/start
Accumu...
Small things
But every little bit helps.
The pure FP is my ideal, rarely reached (so far)
Wednesday, 9 October 13
Another example
(autoscale)
Wednesday, 9 October 13
Message Bus Autoscale
Controller
apps/statistics
Wednesday, 9 October 13
Auto scaling
F(last minute stats, previous data
window) -> “Suggestion” to scale up, or down
(or in or out)
Fundamentally ...
Side effects
Push out side effects to other side of the message
bus
Let a nasty app handle the side effects
Messages are sig...
Advocating to Developers
Don’t “sell” to your developers by:
- saying monad too often
- saying “it’s easy”
- showing that ...
Find the functions
Find the functions in what they do
Find the calculations (eg core of autoscaling)
Advanced:
Separate th...
‘Unlearning’ OO
Clojure: excellent at teaching people to forget about
OO.
Scala: challenge. Temptation always there.
∴ Use...
Some Frameworks for FP &
Microservices
Quick tour:
Wednesday, 9 October 13
Quick development for web apps and APIs
RESTful by design
“batteries included” - has everything.
Easy reuse of all java li...
Wednesday, 9 October 13
Easy deployment (jar/zip file)
Can support DBs/evolution
Excellent JSON/http/web support
Security, Async, Background jobs, ...
Compojure: just enough web framework
Light - small apps
Deploy as war, or embedded jetty
Great libraries, reuse java libs
...
Wednesday, 9 October 13
(:use cheshire.core)
(let [clojure-data (parse-string json-data)])
(generate-string clojure-data)
;;magic!
https://github....
To the Cloud!
Want to try things at your company:
Cloud Platforms (hint! vendor shilling!) make it easy
to try things out ...
Mixed language projects
Good idea??
Relevant to JVM
Ease-into-it
Jury is out...
Wednesday, 9 October 13
Bad questions to hear
(when advocating)
If you hear these asked, probably will have a bad
time:
How will we hire people?
D...
Final Observation
Developers who have fondness for emacs/vim (over
IDE) find things easier
FP invites “change this small bi...
Conclusion
FP and Microservices *naturally* go together
Can be about Risk Reduction in new rapid
deployment world
Cloud
A ...
Thank you
Michael Neale
https://twitter.com/michaelneale
http://www.cloudbees.com
Wednesday, 9 October 13
Upcoming SlideShare
Loading in …5
×

Microservices and functional programming

21,795 views

Published on

A talk I did recently on microservices and functional programming. Microservices are small, single purpose apps that are run as a service, which are usually composed together to provide the real app.

Published in: Technology

Microservices and functional programming

  1. 1. Microservices and FP Michael Neale @michaelneale developer.cloudbees.com Wednesday, 9 October 13
  2. 2. Microservices and FP Given that: We have lots of services - Everything has an API Functional programming: - quality improvements, popular - applied easily to “new” services/projects Combine them! Wednesday, 9 October 13
  3. 3. Definitions Functional programming: - avoid/manage side effects - immutability over mutation - higher order functions - first class functions - composition using functions over OO Wednesday, 9 October 13
  4. 4. Definitions Micro-service: - remotely accessible service (typically http) - does “one thing” (and does it well?) - executes stand alone in a container (JVM - in this case) - may depend on other services - can be composed into a larger service Wednesday, 9 October 13
  5. 5. Definitions Micro-service: - “small” - independently deployed - loosely coupled - not necessarily reusable (but may be) Wednesday, 9 October 13
  6. 6. Definitions Like a unix command line app that does ONE THING AND ONE THING ONLY! service1 | service2 | service3 .... Wednesday, 9 October 13
  7. 7. Example scenario Wednesday, 9 October 13
  8. 8. Jenkins CI PaaS repos autoscale management console web apps how to compose??? provisioning licencingsubscriptions Wednesday, 9 October 13
  9. 9. Identify some micro services repos: artifact storage - store blobs in users account (via s3) autoscale: - look at a stream of stats - decide to scale app (note: won’t scale app - just recommends) subscriptions: - check entitlements, record usage Wednesday, 9 October 13
  10. 10. All in common: All small! All do one thing! Wednesday, 9 October 13
  11. 11. Aside: FP (more later) Some services lend themselves to “purity” eg: Automatic scaling F(last minute stats) = recommendation to scale up, or down (or no change) No state, no persistence. Perfect candidate for FP. Wednesday, 9 October 13
  12. 12. Why not monolithic? Can easily do @Produces(MediaType.JSON) @Path("/apps") @POST ... Thanks to jax-rs in one app? why not? Different endpoints via @Path, different methods, different classes? Wednesday, 9 October 13
  13. 13. Why not monolithic? Apps grow (teams grow too!) Different parts change at different rates Different execution container needs (resources) Different teams? Company grows? But most of all: Wednesday, 9 October 13
  14. 14. Why not monolithic? Fear. Risk. Fear of deploying a change. Risk of trying a new technique, new framework. Wednesday, 9 October 13
  15. 15. Why not monolithic? Deploying a change to a microservice does not increase blood pressure as much as deploying the ONE MASSIVE APP THAT RUNS EVERYTHING Wednesday, 9 October 13
  16. 16. Wednesday, 9 October 13
  17. 17. Monolithic debugging (Often in production) ??? Wednesday, 9 October 13
  18. 18. micro-service debugging Wednesday, 9 October 13
  19. 19. Shift to dev-ops More deployment expected More *people* doing the deployment Wednesday, 9 October 13
  20. 20. Smooth deployment Microservices need: - easy rollback - no “over the wall” deployment - easy as-needed deployment - accessible deploy environment A PaaS can help (private or public) - they were born for this. Wednesday, 9 October 13
  21. 21. Transport Typically (but not always) HTTP Typically (but not always) JSON (autoscale service was message bus + thrift name/ value pairs) But *are* always loosely coupled (ie never RMI) Wednesday, 9 October 13
  22. 22. Containers Execution environment: - anything really - consumers of APIs don’t care - but developers do, deployers do - JVM ideal: enough controls/constraints - many languages - many frameworks If you had to pick one: JVM wins. Wednesday, 9 October 13
  23. 23. Convinced? By this point I hope I have convinced you microservices are good. Right? Wednesday, 9 October 13
  24. 24. Wednesday, 9 October 13
  25. 25. Functional Programming with microservices First, some background... Wednesday, 9 October 13
  26. 26. Context New Company New Team New Product You might not be this “lucky” Wednesday, 9 October 13
  27. 27. History 2010 Started: JVM stack parts - Scala not controversial (I had experience) - working mostly “lone wolf” 2011 - another team added - brought Erlang Wednesday, 9 October 13
  28. 28. Erlang! Me other team members Wednesday, 9 October 13
  29. 29. More FP Given “success” with Scala, more FP (services, languages) was not viewed as a great risk. Wednesday, 9 October 13
  30. 30. Observation FP means one person can do more && People like me like to work alone ∴ risk of staying with one-person-per micro service (is this a bad thing?) Wednesday, 9 October 13
  31. 31. Objections and resistance Ask why: - resistance to the “new” (unnecessary risk) - resistance to FP ideas (hype?) - polyglot fear (realistically multiple languages will be used) Wednesday, 9 October 13
  32. 32. Maintainability objection It goes: - how will anyone be able to maintain this after you? My Experience: - projects featuring FP handed over successfully - new developers able to pick up FP easily - seasoned developers too Wednesday, 9 October 13
  33. 33. Where we may differ Small teams - many systems. ∴ Little overlap in jobs. We get to use a great PaaS ! (our own!) for all these micro services Wednesday, 9 October 13
  34. 34. What did we do with FP Manage horrendous public cloud APIs Automatic scaling (microservice) Github crawling (microservice) Subscription/signup/entitlements (microservice) ... and microservices Wednesday, 9 October 13
  35. 35. Surprisingly practical things No real calculations, no explicit maths. Just boring every day error prone stuff. Wednesday, 9 October 13
  36. 36. For example (provisioning) Wednesday, 9 October 13
  37. 37. Providore Evil cloud api Build masters build workers Workspace storage Wednesday, 9 October 13
  38. 38. New build required. Check the pool, talk to the Evil api, ask for a new server, it fails, ask again. Wait for server to be up, no, I mean really up. Ask for a new disk, wait for a new disk, it fails, try again, attach the new disk, is the disk attached? fails, try again, damn it the snapshot isn’t available. The problem: Wednesday, 9 October 13
  39. 39. How can we solve this? TDD? problems only manifest under load/in- production. APIs are buggy, change over time. Industry best practice: hack something together as a script and some of the time it works. Can FP help us? Wednesday, 9 October 13
  40. 40. Yes Types (providore is written in scala) - Specifically: Option, Either - Closed Data Types (servers only in so many states) The M word: Monads Currying Wednesday, 9 October 13
  41. 41. Cloud API launch_server: Server at best hopes to be: launch_server: Option[Server] launch_server: Either[Server, OhGodWhyWhyWhy] (not an actual pure function of course) Wednesday, 9 October 13
  42. 42. Cloud Monad Cloud APIs are like IO Slow, horrible, misbehaving IO ...and then the APIs other people write All want to be monadic Wednesday, 9 October 13
  43. 43. val validation: String / Subscription = (for { account <- extractAccount _ <- validateSubscription(account) callback <- extractCallBack plan <- validatePlan billing_day <- extractBillingDay subscription <- createSub(account, plan, callback, ... } yield subscription).run(req.body) (scala) ReaderT to help you compose http://debasishg.blogspot.com.au/2011/07/ monad-transformers-in-scala.html Need to “organise code in monadic way” Wednesday, 9 October 13
  44. 44. Types So hard to catch things without them Monadic IO + types mean you catch things before you try Trying/experimenting can be $$ expensive... (still learning this, all new to me) Wednesday, 9 October 13
  45. 45. Types helped with Ignored messages Bad pattern matching Misconfiguration/timing of server creation Avoiding “stringly typed” messages All “real world” things types have help us catch with a friendly compile error** ** may not actually be friendly Wednesday, 9 October 13
  46. 46. Types didn’t help with... Wednesday, 9 October 13
  47. 47. def ec2 Fog::Compute.new(:provider => 'AWS', :aws_secret_access_key => ENV['EC2_SECRET_KEY'], :aws_access_key_id => ENV['EC2_ACCESS_KEY']) end   def tenured? (instance) instance.created_at && (instance.created_at < Chronic.parse('50 minutes ago')) end   def alive? (instance) instance.state == 'running' or instance.state == 'stopped' end   zombies = ec2.servers.select { |i| i.tags.empty? && tenured?(i) && alive?(i) } Parallel.each(zombies, :in_threads => 15) do |zombie| begin puts "Terminating zombie node #{zombie.id}" ec2.servers.get(zombie.id).destroy end end Wednesday, 9 October 13
  48. 48. True story Wednesday, 9 October 13
  49. 49. Currying Server lifecycle: Reserved->Launching->Update (user data)->Volume Create->Volume Attach- >initialise/start Accumulate setup data via partial application Instead of an object that has mutating state, a function you partially apply to accumulate data. Good “beginner” FP concept (powerful, simple) Wednesday, 9 October 13
  50. 50. Small things But every little bit helps. The pure FP is my ideal, rarely reached (so far) Wednesday, 9 October 13
  51. 51. Another example (autoscale) Wednesday, 9 October 13
  52. 52. Message Bus Autoscale Controller apps/statistics Wednesday, 9 October 13
  53. 53. Auto scaling F(last minute stats, previous data window) -> “Suggestion” to scale up, or down (or in or out) Fundamentally calculation, fundamentally functional. Wednesday, 9 October 13
  54. 54. Side effects Push out side effects to other side of the message bus Let a nasty app handle the side effects Messages are signals, suggestions, idempotent Wednesday, 9 October 13
  55. 55. Advocating to Developers Don’t “sell” to your developers by: - saying monad too often - saying “it’s easy” - showing that it can be just as easy/familiar as what they have Instead... Wednesday, 9 October 13
  56. 56. Find the functions Find the functions in what they do Find the calculations (eg core of autoscaling) Advanced: Separate the program definition from the execution (monads) Intermediate: Currying, Higher order functions, Types (good ones) Wednesday, 9 October 13
  57. 57. ‘Unlearning’ OO Clojure: excellent at teaching people to forget about OO. Scala: challenge. Temptation always there. ∴ Use object/package as namespace (avoid classes) Aside: lack of implicit state allows “Erlang Magic” Wednesday, 9 October 13
  58. 58. Some Frameworks for FP & Microservices Quick tour: Wednesday, 9 October 13
  59. 59. Quick development for web apps and APIs RESTful by design “batteries included” - has everything. Easy reuse of all java libraries. www.playframework.com Wednesday, 9 October 13
  60. 60. Wednesday, 9 October 13
  61. 61. Easy deployment (jar/zip file) Can support DBs/evolution Excellent JSON/http/web support Security, Async, Background jobs, and more Scala or Java. Wednesday, 9 October 13
  62. 62. Compojure: just enough web framework Light - small apps Deploy as war, or embedded jetty Great libraries, reuse java libs Effortless JSON -> clojure https://github.com/weavejester/compojure Wednesday, 9 October 13
  63. 63. Wednesday, 9 October 13
  64. 64. (:use cheshire.core) (let [clojure-data (parse-string json-data)]) (generate-string clojure-data) ;;magic! https://github.com/dakrone/cheshire Wednesday, 9 October 13
  65. 65. To the Cloud! Want to try things at your company: Cloud Platforms (hint! vendor shilling!) make it easy to try things out that may be used seriously. Clojure anywhere, Scala anywhere Say “cloud cloud” a lot and people will listen. Wednesday, 9 October 13
  66. 66. Mixed language projects Good idea?? Relevant to JVM Ease-into-it Jury is out... Wednesday, 9 October 13
  67. 67. Bad questions to hear (when advocating) If you hear these asked, probably will have a bad time: How will we hire people? Does it work for large teams? Wednesday, 9 October 13
  68. 68. Final Observation Developers who have fondness for emacs/vim (over IDE) find things easier FP invites “change this small bit, see what happens” exploration No real tool barriers. If I can do this, anyone can. Wednesday, 9 October 13
  69. 69. Conclusion FP and Microservices *naturally* go together Can be about Risk Reduction in new rapid deployment world Cloud A good place to try FP Wednesday, 9 October 13
  70. 70. Thank you Michael Neale https://twitter.com/michaelneale http://www.cloudbees.com Wednesday, 9 October 13

×