Real-World Functional Programming @ Incubaid

6,326 views

Published on

Presentation on Functional Programming in a for-profit company using OCaml, presented on December 14th 2011 at Ghent University (UGent).

Due to the Haskell background of the attendees, OCaml was introduced with this in mind.

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,326
On SlideShare
0
From Embeds
0
Number of Embeds
3,170
Actions
Shares
0
Downloads
36
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Real-World Functional Programming @ Incubaid

  1. 1. About Incubaid Functional Programming @ Incubaid OCaml ConclusionReal-World Functional Programming @ Incubaid Romain Slootmaekers Nicolas Trangez Incubaid BVBA {romain,nicolas}@incubaid.com Twitter: @incubaid Team Blog: blog.incubaid.com Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  2. 2. About Incubaid Functional Programming @ Incubaid OCaml Conclusion1 About Incubaid2 Functional Programming @ Incubaid Why FP? Current Projects3 OCaml OCaml vs. Haskell Code Teaser4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  3. 3. About Incubaid Functional Programming @ Incubaid OCaml ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  4. 4. About Incubaid Functional Programming @ Incubaid OCaml ConclusionCompany Overview Technology incubator Creating & providing services to startups Not a VC, collaboration with external parties Main activities: Datacenter & Cloud Computing Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  5. 5. About Incubaid Functional Programming @ Incubaid OCaml ConclusionHistory Dedigate (2000) acquired by Terremark (2005) Hostbasket (2000) acquired by Telenet (2008) DataCenter Technologies (2002) acquired by Veritas/Symantec (2005) Q-Layer (2007) acquired by Sun Microsystems (2009) Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  6. 6. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  7. 7. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  8. 8. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionDevelopment Benefits Code quality Correctness Easy refactoring Performance Developer performance (Getting Things Done) Developer satisfaction Developer quality (hiring!) Not the average “Enterprise Developer” ;-) Mathematical background or interests Good at logical, conceptual reasoning Smaller prototype-to-production cycle Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  9. 9. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionIdeal fit for our Application Domains Handling large opaque data blobs (DSS) System & application automation (SSO) Highly scalable distributed systems Large scale implies constant random failure Transactions? Where, why, how? Concurrency Immutability of data, purity of operations, idempotent actions,. . . allow to reason about systems and their behaviour Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  10. 10. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionFP: more than languages Note Functional Programming is not only about languages. Insight in FP concepts and techniques can bring radical improvements to the architecture and design of software, especially in large-scale, highly-unreliable environments. Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  11. 11. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  12. 12. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionAmplidata DSS Next-generation large object storage Redundant storage of petabytes of big files Dispersed storage using rateless codes Initial version: C++ Current version: OCaml (90%), C (9%), Asm (1%) More info @ amplidata.com Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  13. 13. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionAmplidata DSS (Cont.) Math 2 a ... a0,n 3 b0 2 3 0,0 6 a1,0 ... a1,n 7 2 3 6 b1 7 7 x0 6. . . . . . . . . . . . . . . . . . .7 6 6 ... 7 6 6 7 x 7 6 a 6 n,0 ... an,n 7 6 1 7 = 6 bn 7 7 . . .5 74 6 7 6 an+1,0 ... an+1,n 7 6 bn+1 7 6 6 7 5 xn 4 ... 5 ................... 4 an+m,0 ... an+m,n bn+m Dataflow Storage0 Client codec ........ StorageN Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  14. 14. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionArakoon Distributed, consistent, persistent key-value store Multi-Paxos implementation TokyoCabinet backend OCaml Open Source, see http://www.arakoon.org/ A nursery of two 3-node clusters Cluster A A0 client0 client2 Cluster B B0 client1 A1 B1 A2 B2 Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  15. 15. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionBaardskeerder Local, in-process key-value database Append-only B-tree-ish, persistent Replace TokyoCabinet which can’t cope with large values Benefits of and doesn’t kill SSD drives OCaml (future Arakoon backend) Not publicly available yet Follow @incubaid or blog.incubaid.com for updates Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  16. 16. About Incubaid Functional Programming @ Incubaid Why FP? OCaml Current Projects ConclusionBaardskeerder (Cont.) Tree Construction 0: Value "A" 1: Leaf [ " a " , Outer 0 ] 2: Commit ( Outer 1 ) a b c 3: Value "B" 4: Leaf [ " a " , Outer 0 ; " b " , Outer 3 ] 5: Commit ( Outer 4 ) 6: Value "C" A B C 7: Leaf [ " a " , Outer 0 ; " b " , Outer 3 ; " c " , Outer 6 ] 8: Commit ( Outer 7 ) Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  17. 17. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  18. 18. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionIntroduction Objective Caml Objective Categorical Abstract Machine/Meta Language Dialect of ML Descendents include Microsoft F#, JoCaml (, Scala) Static typing using type inference Eager evaluation (by default) Emphasis on performance, both in language features as well as standard library implementation Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  19. 19. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionTimeline 1970 1980 1990 2000 2010 Caml Ocaml F# ML Miranda Haskell Standard ML Prolog Erlang C C++ Scala Java C# 1970 1980 1990 2000 2010 Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  20. 20. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionCompiler and Runtime Main implementation from INRIA (.fr) Bytecode and native targets Very fast compilation Straight-forward compiler backend & code generation Easy to reason about efficiency, predict compiler output If necessary (unlike GHC) native assembly can be read/interpreted by humans Thanks to this, we were able to debug an obscure non-reproducible bug in Arakoon once, which turned out to be a flaw in one of the libraries we use. Details on request! Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  21. 21. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionConcurrency & Parallellism Using native/system threads (1-to-1 mapping) No parallel runtime (only one OCaml execution thread at all times), a reentrant runtime is WIP Lightweight “Monadic Threads” for concurrency using Lwt, Async,. . . See W.L.Harrison, “Cheap (But Functional) Threads” Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  22. 22. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionApplications Ocsigen: Web application framework, parent project of Lwt: lightweight threading Js_of_ocaml: OCaml to JavaScript compiler Mirage: Xen-based exokernel Coq: Proof assistant MLDonkey: EDonkey P2P client Unison: File synchronization haXe compiler: High-level multiplatform language, compiled to JS/SWF/AS3/PHP5/C++/. . . Coccinelle: Semantic patches Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  23. 23. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  24. 24. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionKey Differences (1/3) OCaml evaluation is eager by default I once defined (>>) as (>>) a b = a >>= fun _ −> b. Lost quite some time debugging the unexpected runtime behaviour. No overloading (cfr. Haskell type-classes) + : int −> int −> int +. : float −> float −> float (notice the dot) + : num −> num −> num Both structural and referential equality operators: = and == Optional & named arguments Higher-order modules & functors (not the Haskell ones) Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  25. 25. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionKey Differences (2/3) OO support Clean semantics Quirky syntax Type operators / coercions: :#, :> Polymorphic variants (union types) # [ ‘ Number 1 ; ‘ F l o a t 3 . 1 4 ; ‘ U n i t ; ] ; ; − : [ > ‘ F l o a t o f f l o a t | ‘ Number o f i n t | ‘ Unit ] l i s t = [ ‘ Number 1 ; ‘ F l o a t 3 . 1 4 ; ‘ U n i t ] No purity restrictions (cfr. IO monad) Side-effects (incl. IO) possible everywhere Record fields can be mutable In-place mutable variables (refs) implemented on top of this Strings are in-place mutable Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  26. 26. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionKey Differences (3/3) Sugared syntax: "abc ".[1] , [| 1; 2; 3; |].(1) vs. String .get, Array.get Exceptions are common, also in pure code (e.g. Not_found) Smaller “standard library” compared to Haskell Platform Syntax extensions Macros: (IFDEF ... THEN ... ELSE ... END) Comprehensions: [x | x <− [1;2;3]; x<2] Succinctness for libraries Normal lwt.syntax lwt x = calc_x() in c a l c _ x ( ) >>= fun x −> lwt y = calc_y() in c a l c _ y ( ) >>= fun y −> ... ... (* concurrent evaluation! *) lwt a = f () and b = g () in ... Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  27. 27. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionGotcha: Value Restriction (1/4) Definition The “value restriction” is a way to maintain correct typing in the presence of side-effects: the type of an expression can only be generalized if the expression is a “syntactic value” (or “non-expansive expression”): a literal or identifier: 3, ’ a’ , . . . an abstraction: fun x −> 2 ∗ x, . . . a constructor applied to a syntactic value: Some 1, . . . Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  28. 28. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionGotcha: Value Restriction (2/4) Example # l e t i = fun x −> x ; ; v a l i : ’ a −> ’ a = <fun > # let r = ref i ; ; The type of i is ∀α.(α → α). Without value restriction, the type of r would be ∀α.(α → α) ref. Since it’d be polymorphic, it could be used as type (bool −> bool) ref or ( int −> int) ref , so r : = fun ( t r u e | f a l s e ) −> f a l s e ; (! r ) 0 would both type-check but the last expression would result in a runtime error. Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  29. 29. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionGotcha: Value Restriction (3/4) Demo Here’s what happens for real: # let r = ref i ; ; v a l r : ( ’ _a −> ’ _a ) r e f = { c o n t e n t s = <fun >} # r : = fun ( t r u e | f a l s e ) −> t r u e ; ; − : unit = () # r ;; − : ( b o o l −> b o o l ) r e f = { c o n t e n t s = <fun >} # let s = ref i ; ; v a l s : ( ’ _a −> ’ _a ) r e f = { c o n t e n t s = <fun >} # (! s) 0;; − : int = 0 # s;; − : ( i n t −> i n t ) r e f = { c o n t e n t s = <fun >} # l e t t = l e t l = r e f [ ] i n fun e −> l : = ( e : : (! l )); (! l );; v a l t : ’ _a −> ’ _a l i s t = <fun > # t 1;; − : int l i s t = [1] # t ;; − : i n t −> i n t l i s t = <fun > # t 2;; − : i n t l i s t = [ 2 ; 1] Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  30. 30. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionGotcha: Value Restriction (4/4) Work-around using eta-expansion # let id x = x ; ; v a l i d : ’ a −> ’ a = <fun > # ( i d 1 , i d "abc" ) ; ; − : i n t ∗ s t r i n g = ( 1 , "abc" ) # l e t id ’ = i d i d ; ; v a l i d ’ : ’ _a −> ’ _a = <fun > # id ’ 1 ; ; − : int = 1 # id ’ ; ; − : i n t −> i n t = <fun > # i d ’ "abc" ; ; E r r o r : T h i s e x p r e s s i o n has type s t r i n g b u t an e x p r e s s i o n was expected of type i n t # l e t i d ’ ’ = fun x −> ( i d i d ) x ; ; v a l i d ’ ’ : ’ a −> ’ a = <fun > # ( i d ’ ’ 1 , i d ’ ’ "abc" ) ; ; − : i n t ∗ s t r i n g = ( 1 , "abc" ) # l e t f 1 = L i s t . map i d ; ; v a l f 1 : ’ _a l i s t −> ’ _a l i s t = <fun > # l e t f 2 = fun l −> L i s t . map i d l ; ; v a l f 2 : ’ a l i s t −> ’ a l i s t = <fun > Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  31. 31. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  32. 32. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser ConclusionRsync algo Implementation of the classic “rsync” algorithm Full source @ https://github.com/Incubaid/rsync-demo Programming “in the large” Expose use of OO Modules, functors Imperative programming, mutability ... but not necessarily the best, most advisable approach Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  33. 33. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser Conclusionmodule type WEAK = sig type t v a l make : u n i t −> t v a l from : s t r i n g −> i n t −> i n t −> t v a l r e s e t : t −> u n i t v a l d i g e s t : t −> i n t v a l r o t a t e : t −> char −> char −> u n i t v a l update : t −> s t r i n g −> i n t −> i n t −> u n i tendmodule type STRONG = sig type t v a l to_hex : t −> s t r i n g v a l f i l e : s t r i n g −> t v a l s u b s t r i n g : s t r i n g −> i n t −> i n t −> t v a l w r i t e : out_channel −> t −> u n i t v a l read : i n _ c h a n n e l −> tendmodule SDigest = ( s t r u c t include D i g e s t l e t read i c = I o . r e a d _ s t r i n g i c l e t w r i t e oc t = I o . w r i t e _ s t r i n g oc tend : STRONG) Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  34. 34. About Incubaid Functional Programming @ Incubaid OCaml vs. Haskell OCaml Code Teaser Conclusionopen Hashopen S i g n a t u r emodule Rsync = functor (W:WEAK) −> functor (S :STRONG) −> s t r u c t module MySig = S i g n a t u r e (W) ( S) class d e l t a _ e m i t t e r s i g n a t u r e new_fn h a n d l e r = l e t bs = MySig . b l o c k _ s i z e s i g n a t u r e i n l e t b u f f e r _ s i z e = 8 ∗ bs i n l e t b u f f e r = S t r i n g . create b u f f e r _ s i z e in object ( s e l f ) v a l mutable _read = 0 v a l mutable _ f i r s t _ f r e e = 0 v a l mutable _ n _ f r e e = b u f f e r _ s i z e v a l mutable _ f i r s t _ t o d o = 0 v a l mutable _ p r e v i o u s _ a c t i o n = S t a r t bs v a l mutable _ f i n i s h e d = f a l s e v a l mutable _weak_ok = f a l s e v a l _weak = W. make ( ) method _examine_block b u f f e r o f f s e t l e n g t h = l e t wd = W. d i g e s t _weak i n match MySig . lookup_weak s i g n a t u r e wd with | None −> None | Some bs −> l e t strong = S. substring b u f f e r o f f s e t length in i f s t r o n g = MySig . b s _s t r o n g bs then Some ( MySig . bs_index bs ) else None Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  35. 35. About Incubaid Functional Programming @ Incubaid OCaml ConclusionOutline 1 About Incubaid 2 Functional Programming @ Incubaid Why FP? Current Projects 3 OCaml OCaml vs. Haskell Code Teaser 4 Conclusion Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  36. 36. About Incubaid Functional Programming @ Incubaid OCaml ConclusionIn Retrospect No regrets Productivity increased, less bugs Bugs fixed quickly, features added easily Runtime issues mainly due to improper usage or configuration (the universe is still winning!), except Lwt bugs Limited tooling support compared to e.g. JVM (JMX/JConsole, remote debugging, GC tuning,. . . ) No deal-breaker, because less needed Would be some nice projects! Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  37. 37. About Incubaid Functional Programming @ Incubaid OCaml ConclusionFuture Current focus on OCaml: in general, best known by the team Haskell for “compilable pseudocode”, once in a while Haskell for future projects, why not? Introduce devs, architects,. . . of non-FP projects to the FP concepts and strengths to increase the quality of their work: knowledge of FP concepts changes the way you think about programming, even in non-FP settings Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  38. 38. About Incubaid Functional Programming @ Incubaid OCaml ConclusionInternships & Jobs Interested in working on our software and building future-proof scalable computing platforms & looking for an internship or job? Come join us! recruiting@incubaid.com wehire.be Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid
  39. 39. About Incubaid Functional Programming @ Incubaid OCaml Conclusion Questions?Romain Slootmaekers, Nicolas Trangez Real-World Functional Programming @ Incubaid

×