The document discusses the Pipes and Filters architectural pattern. It defines pipes and filters as a way to divide a processing task into independent and sequential processing steps (filters) connected by channels (pipes). Key aspects include: - Filters transform input data and can run concurrently and independently. Pipes connect filters and transmit data streams. - Examples of implementations include Unix pipelines, Java streams, Akka actors, and servlet filters. The pattern has a long history and is still widely used. - The document provides details on properties of pipes, filters, and implementations using technologies like Akka, Jocote, RabbitMQ, and Flink. It also discusses the author's own Pineapple framework for implementing the

1. Pipes & Filters
Architectural
Pattern
Fredrik Kivi

2. About Me

4. Have you ever end up with a situation
there you doing multiple entity updates
of the status to reflect your application
lifecycle?

5. The Database As Queue
Anti-Pattern
-Mike Hadlow
”When all you have is a hammer,
every problem looks like a nail”

6. The
Pipes & Filters
Architectural Pattern

7. –Wikipedia Pipeline_(Software)
“A pipeline consists of a chain of processing
elements arranged so that the output of each
element is the input of the next...
The information that flows in these pipelines is
often a stream of records, bytes or bits, and the
elements of a pipeline may be called filters; this is
also called the pipes and filters design
pattern.”

8. -Enterprise Integration Patterns
(Gregor Hope & Bobby Woolf)
“Use the Pipes and Filters architectural style
to divide a larger processing task into a sequence
of smaller, independent processing steps (filters)
that are connected by channels (pipes).”

11. Related patterns

12. • Collection Pipeline
• Java Stream API
• Chain of Responsibility (CoR)
• Servlet Filter

13. History

14. • 1972-1974 Unix Pipelines, Pipes & Filters
• 1994-1996 Pipes & Filters Architectural Pattern
• PLoP (Pattern Languages of Programs Conf.)
• Papers
• Books

15. Terminology

16. • Pump
- Component that feeding a pipeline with data
• Filter
- Component that transforming data
• Sink
- Component that collects data and terminates a
pipeline

17. PIPE PROPERTIES
• Connections
• Streams
• Channels
• Represent object status

18. FILTER PROPERTIES
• Transform
• Component
• Concurrent
• Process
• Independent, parallel, decomposed processing

19. Filter implementation

20. Akka Actors
• Independent, parallel, decomposed processing
• Fault tolerant
• Scala up and out
• Akka & Apache Camel?
• Not in my taste ;)

21. Pipe implementation

22. JMS Hello World

25. Jocote

27. @Folksam 2012-2016
KHS

29. • Pumps - Jocote (IBM MQ)
• Pipes - Jocote (RabbitMQ)
• Filters - Akka Actors
• Sinks - LPR
• Scheduling - Akka Scheduler, Quartz
• Enabling/disabling - Akka Cluster Event

30. Retrospective

31. Glad
• RabbitMQ cluster
• Object state represented as pipes
• Akka fault tolerance with queue acknowledgement
• Scale up - Akka router
• Scale out - Independent server nodes consuming
the same queues

32. Sad
• No orchestration
• Hard to follow asynchronous calls as development

33. What if

36. Graphviz
• Short for Graph Visualisation Software
• Open source tools initiated by AT&T Labs
Research
• Graphviz consists of a graph description language
named the DOT language

37. TOOLS
• Eclipse GEF
• www.webgraphviz.com

38. Pineapple
• Based on experience from KHS
• Akka & Jocote
• Wire up actors from model
• Parsed from DOT file
• Created by ModelBuilder
• Discovery API (Spring, JSR 330 or your own)
• Pipes & Filters API

41. D E M O

42. Next
• Connection recovery
• Support for
• Ack/nack
• Backout queues (dead letters)
• Using actor selection as url
• Cron scheduling (ConsumeType)
• JMX
• Enabling/disabling filters, pumps and sinks
• Show model
• Monitor UI

43. Apache Flink
• Apache Flink 0.10.0 - Nov 2015
• Ugli Jocote 1.6.0.0 - Nov 2015
• Looks good and has momentum

44. I been thinking of
• A Jocote Flink Connector
• An option for choosing between Jocote
and Flink in Pineapple
• A Camel-Jocote-Driver

45. • https://github.com/ugli/jocote
• https://github.com/ugli/pineapple
• fredrik.kivi@ugli.se