Successfully reported this slideshow.
Your SlideShare is downloading. ×

Pipeline oriented programming

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 102 Ad

Pipeline oriented programming

Download to read offline

(Video and code at https://fsharpforfunandprofit.com/pipeline/)

Passing data through a pipeline of transformations is an alternative approach to classic OOP. The LINQ methods in .NET are designed around this, but the pipeline approach can be used for so much more than manipulating collections.

In this talk, I'll look at pipeline-oriented programming and how it relates to functional programming, the open-closed principle, unit testing, the onion architecture, and more. I'll finish up by showing how you can build a complete web app using only this approach.

(Video and code at https://fsharpforfunandprofit.com/pipeline/)

Passing data through a pipeline of transformations is an alternative approach to classic OOP. The LINQ methods in .NET are designed around this, but the pipeline approach can be used for so much more than manipulating collections.

In this talk, I'll look at pipeline-oriented programming and how it relates to functional programming, the open-closed principle, unit testing, the onion architecture, and more. I'll finish up by showing how you can build a complete web app using only this approach.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Advertisement

More from Scott Wlaschin (19)

Recently uploaded (20)

Advertisement

Pipeline oriented programming

  1. 1. Pipeline Oriented Programming (DotNext 2021) @ScottWlaschin fsharpforfunandprofit.com/pipeline Code examples will be in C# and F#
  2. 2. What is “Pipeline-oriented Programming"? Part I
  3. 3. Object-oriented Arrows go in all directions
  4. 4. Pipeline-oriented All arrows go left to right
  5. 5. A pipeline-oriented web backend One directional flow, even with branching
  6. 6. What are the benefits of a pipeline-oriented approach?
  7. 7. Benefits • Pipelines encourage composability • Pipelines follow good design principles • Pipelines are easier to maintain • Pipelines make testing easier • Pipelines fit well with modern architectures – E.g. Onion/Clean/Hexagonal, etc
  8. 8. Benefit 1: Pipelines encourage composability Composable == “Like Lego”
  9. 9. Connect two pieces together and get another "piece" that can still be connected You don't need to create a special adapter to make connections.
  10. 10. Component Component Component Pipelines composition Pipelines encourage you to design so that: • Any pieces can be connected • You don’t need special adapters If you do this, you get nice composable components
  11. 11. Bigger Component Sub- Component Sub- Component Sub- Component Sub-components composed into a new bigger unit
  12. 12. Benefit 2: Pipelines follow good design principles Many design patterns work naturally with a pipeline-oriented approach
  13. 13. Single responsibility principle Component Can't get more single responsibility than this!
  14. 14. Open/Closed principle You could be able to add new functionality (“open for extension”) without changing existing code (“closed for modification”) New Behavior Component Component Component Extension Not changed
  15. 15. Strategy Pattern Parameterized Component Component Component Component
  16. 16. Decorator Pattern Decorator Decorator Original Component
  17. 17. Decorated Component Decorator Pattern Decorator Decorator "decorated" component works just like the original one
  18. 18. Benefit 3: Pipelines are easier to maintain
  19. 19. This is easier to maintain This is harder to maintain
  20. 20. Object-oriented: Real-world dependency graph Multiple circular dependencies  Arrows go in all directions
  21. 21. Pipeline-oriented: Real-world dependency graph All arrows go left to right  Same functionality as the OO example
  22. 22. More benefits: • Pipelines make testing easier • Pipelines fit well with modern architectures
  23. 23. Pipeline-oriented Programming in practice Part 2
  24. 24. var count = 0; foreach (var i in list) { var j = i + 2; if (j > 3) { count++; } } return count; Here is a traditional for- loop in C#
  25. 25. return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Here is the same code using LINQ in C#
  26. 26. return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Benefit: Composability The LINQ components have been designed to fit together in many different ways You could write your code this way too!
  27. 27. return list .Select(x => x + 2) .Where(x => x > 3) .Count(); Benefit: Single responsibility Each LINQ component does one thing only. Easier to understand and test
  28. 28. return list .Select(x => x + 2) .Where(x => x > 3) .Where(x => x < 10) .Count(); Benefit: open for extension It is easy to add new steps in the pipeline without touching anything else
  29. 29. list |> List.map (fun x -> x + 2) |> List.filter (fun x -> x > 3) |> List.length Here is the same code using F#
  30. 30. list |> List.map (fun x -> x + 2) |> List.filter (fun x -> x > 3) |> List.length Here is the same code using F#
  31. 31. Immutability and pipelines If you have immutable data you will probably need a pipeline
  32. 32. // Person is immutable var p = new Person("Scott","s@example.com",21); var p2 = p.WithName("Tom"); var p3 = p2.WithEmail("tom@example.com"); var p4 = p3.WithAge(42); When each call returns a new object, it gets repetitive
  33. 33. // Person is immutable var p = new Person("Scott","s@example.com",21); p .WithName("Tom") .WithEmail("tom@example.com") .WithAge(42); Pipelines make it look nicer
  34. 34. Pipes vs. Extension methods Pipes are more general
  35. 35. int Add1(int input) { return input + 1; } int Square(int input) { return input * input; } int Double(int input) { return input * 2; }
  36. 36. int Add1(int input) { return input + 1; } int Square(int input) { return input * input; } int Double(int input) { return input * 2; } int NestedCalls() { return Double(Square(Add1(5))); } How can I make this look nicer? How about a pipeline?
  37. 37. return 5 .Add1 .Square .Double; But this doesn't work, because these are not extension methods.
  38. 38. public static TOut Pipe<TIn, TOut> (this TIn input, Func<TIn, TOut> fn) { return fn(input); } Introducing "Pipe"
  39. 39. public static TOut Pipe<TIn, TOut> (this TIn input, Func<TIn, TOut> fn) { return fn(input); }
  40. 40. public static TOut Pipe<TIn, TOut> (this TIn input, Func<TIn, TOut> fn) { return fn(input); }
  41. 41. return 5 .Pipe(Add1) .Pipe(Square) .Pipe(Double); And now we can create a pipeline out of any* existing functions
  42. 42. int Add(int i, int j) { return i + j; } int Times(int i, int j) { return i * j; } A two parameter function
  43. 43. public static TOut Pipe<TIn, TParam, TOut> (this TIn input, Func<TIn, TParam, TOut> fn, TParam p1) { return fn(input, p1); }
  44. 44. int Add(int i, int j) { return i + j; } int Times(int i, int j) { return i * j; } public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Times, 2); } We can now create a pipeline out of any existing functions with parameters
  45. 45. public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Times, 2); } Why bother? Because now we get all the benefits of a pipeline
  46. 46. public int PipelineWithParams() { return 5 .Pipe(Add, 1) .Pipe(Times, 2) .Pipe(Add, 42) .Pipe(Square); } Why bother? Because now we get all the benefits of a pipeline, such as adding things to the pipeline easily (diffs look nicer too!)
  47. 47. let add x y = x + y let times x y = x * y 5 |> add 1 |> times 2 And here's what the same code looks like in F# F# uses pipelines everywhere!
  48. 48. Pipes vs. Extension methods, Part 2 Extension methods cannot be parameters
  49. 49. int StrategyPattern(... list, ... injectedFn) { return list .Select(x => x + 2) .injectedFn .Where(x => x > 3) .Count(); } We cant use a function parameter as an extension method  My "strategy"
  50. 50. int StrategyPattern(... list, ... injectedFn) { return list .Select(x => x + 2) .Pipe(injectedFn) .Where(x => x > 3) .Count(); } But we can use a function parameter in a pipeline! 
  51. 51. Pipelines in practice Part 3:
  52. 52. Three demonstrations • Pipeline-oriented Roman Numerals • Pipeline-oriented FizzBuzz • Pipeline-oriented web API
  53. 53. Roman Numerals
  54. 54. To Roman Numerals Task: convert an integer to roman numerals • 5 => "V" • 12 => "XII" • 107 => "CVII"
  55. 55. To Roman Numerals
  56. 56. To Roman Numerals • Use the "tally" approach – Start with N copies of "I" – Replace five "I"s with a "V" – Replace two "V"s with a "X" – Replace five "X"s with a "L" – Replace two "L"s with a "C" – etc
  57. 57. To Roman Numerals number etc Replicate "I" Replace_IIIII_V Replace_VV_X Replace_XXXXX_L
  58. 58. string ToRomanNumerals(int n) { return new String('I', n) .Replace("IIIII", "V") .Replace("VV", "X") .Replace("XXXXX", "L") .Replace("LL", "C"); } C# example
  59. 59. string ToRomanNumerals(int n) { return new String('I', n) .Replace("IIIII", "V") .Replace("VV", "X") .Replace("XXXXX", "L") .Replace("LL", "C"); } C# example .Replace("LL", "C") .Replace("VIIII", "IX") .Replace("IIII", "IV") .Replace("LXXXX", "XC") .Replace("XXXX", "XL"); } Extend functionality without touching existing code
  60. 60. let toRomanNumerals n = String.replicate n "I" |> replace "IIIII" "V" |> replace "VV" "X" |> replace "XXXXX" "L" |> replace "LL" "C" // special cases |> replace "VIIII" "IX" |> replace "IIII" "IV" |> replace "LXXXX" "XC" |> replace "XXXX" "XL" F# example
  61. 61. FizzBuzz
  62. 62. FizzBuzz definition • Write a program that prints the numbers from 1 to N • But: – For multiples of three print "Fizz" instead – For multiples of five print "Buzz" instead – For multiples of both three and five print "FizzBuzz" instead.
  63. 63. for (var i = 1; i <= 30; i++) { if (i % 15 == 0) Console.Write("FizzBuzz,"); else if (i % 3 == 0) Console.Write("Fizz,"); else if (i % 5 == 0) Console.Write("Buzz,"); else Console.Write($"{i},"); } C# example
  64. 64. for (var i = 1; i <= 30; i++) { if (i % 15 == 0) Console.Write("FizzBuzz,"); else if (i % 3 == 0) Console.Write("Fizz,"); else if (i % 5 == 0) Console.Write("Buzz,"); else Console.Write($"{i},"); } C# example
  65. 65. Pipeline implementation number Handle 15 case
  66. 66. Pipeline implementation Handle 3 case number Handle 15 case
  67. 67. Pipeline implementation Handle 3 case Handle 5 case number Handle 15 case
  68. 68. Pipeline implementation Handle 3 case Handle 5 case number Answer Handle 15 case Final step
  69. 69. number Handle case Processed (e.g. "Fizz", "Buzz") Unprocessed (e.g. 2, 7, 13) record FizzBuzzData(string Output, int Number);
  70. 70. record FizzBuzzData(string Output, int Number); static FizzBuzzData Handle( this FizzBuzzData data, int divisor, // e.g. 3, 5, etc string output) // e.g. "Fizz", "Buzz", etc { if (data.Output != "") return data; // already processed if (data.Number % divisor != 0) return data; // not applicable return new FizzBuzzData(output, data.Number); }
  71. 71. record FizzBuzzData(string Output, int Number); static FizzBuzzData Handle( this FizzBuzzData data, int divisor, // e.g. 3, 5, etc string output) // e.g. "Fizz", "Buzz", etc { if (data.Output != "") return data; // already processed if (data.Number % divisor != 0) return data; // not applicable return new FizzBuzzData(output, data.Number); } Extension method
  72. 72. static string FizzBuzzPipeline(int i) { return new FizzBuzzData("", i) .Handle(15, "FizzBuzz") .Handle(3, "Fizz") .Handle(5, "Buzz") .FinalStep(); } static void FizzBuzz() { var words = Enumerable.Range(1, 30) .Select(FizzBuzzPipeline); Console.WriteLine(string.Join(",", words)); }
  73. 73. Demo: FizzBuzz in F# Extensions, Decorations and Parallelization
  74. 74. A pipeline-oriented web API
  75. 75. HttpContext Web Component Not Handled Handled
  76. 76. (async) HttpContext null HttpContext
  77. 77. GET Define a component matches verb doesn't match verb HttpContext
  78. 78. route "/hello" Define a component matches route doesn't match route HttpContext
  79. 79. setStatusCode 200 Define a component Always succeeds HttpContext
  80. 80. GET route "/hello" >=> A special pipe for web components setStatusCode 200 >=>
  81. 81. Are you wondering how do we compose these?
  82. 82. Easy!
  83. 83. choose [ ] Picks first component that succeeds Define a new component
  84. 84. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" ] Pick first path that succeeds
  85. 85. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/bad" >=> BAD_REQUEST ] All the benefits of pipeline-oriented programming
  86. 86. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/user" >=> mustBeLoggedIn UNAUTHORIZED >=> requiresRole "Admin" // etc ] All the benefits of pipeline-oriented programming
  87. 87. choose [ GET >=> route "/hello" >=> OK "Hello" GET >=> route "/goodbye" >=> OK "Goodbye" POST >=> route "/user" >=> mustBeLoggedIn UNAUTHORIZED >=> requiresRole "Admin" // etc ] All the benefits of pipeline-oriented programming
  88. 88. Demo: A pipeline oriented web app For more on the web framework I'm using, search the internet for "F# Giraffe"
  89. 89. Testing and architecture Part 4
  90. 90. Benefit 4: Pipelines encourage good architecture
  91. 91. The "onion" architecture Core domain is pure, and all I/O is at the edges Core domain
  92. 92. Domain Logic I/O I/O The "onion" architecture  Core domain is pure, and all I/O is at the edges
  93. 93. In a well-designed pipeline, all I/O is at the edges. Easy to enforce this with a pipeline-oriented approach
  94. 94. Database Database Load data Process it Save it
  95. 95. I/O in the middle of a pipeline Keep them separate
  96. 96. Benefit 5: Easier to test
  97. 97. Deterministic Non-deterministic Non-deterministic
  98. 98. This makes testing easy!
  99. 99. In Conclusion
  100. 100. Why bother? • Reusable components • Understandable – data flows in one direction • Extendable – add new parts without touching old code • Testable – parts can be tested in isolation • A different way of thinking – it's good for your brain to learn new things!
  101. 101. Databases Object-oriented programmng Pipeline-oriented programming Domain-driven design Does pipeline-oriented programming work for every situation? No! But it should be part of your toolkit
  102. 102. Pipeline Oriented Programming – Slides and video will be posted at • fsharpforfunandprofit.com/pipeline Related talks – "The Power of Composition" • fsharpforfunandprofit.com/composition – “Railway Oriented Programming" • fsharpforfunandprofit.com/rop Thanks! Twitter: @ScottWlaschin

×