Hagen Toennies from Gaikai Inc. presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"In this talk we will present how we enable distributed, Unix style programming using Docker and Apache Kafka. We will show how we can take the famous Unix Pipe Pattern and apply it to a Distributed Computing System. We will demonstrate the development of two simple applications with the focus on "Do One Thing and Do It Well." Afterwards we demonstrate how we make these two programs work to together using Apache Kafka. By encapsulating our applications in containers we will also show how that enables us to go from the limited resources of a development machine to cluster of computers in a data center without changing our applications or containers."
Watch the video: http://wp.me/p3RLHQ-goG
Learn more: http://www.hpcadvisorycouncil.com/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
3. Unix Pipe Recap
• […]In Unix-like computer operating systems, a
pipeline is a sequence of processes chained
together by their standard streams, so that the
output of each process (stdout) feeds directly as
input (stdin) to the next one.
3
4. Unix Philosophy
• Write programs that do one thing and do it well.
• Write programs to work together.
• Write programs to handle text streams,
because that is a universal interface.
Peter H. Salus says about the unix philosophy
4
19. Containers
• […] Operating-system-level virtualization is a
server virtualization method in which the kernel of
an operating system allows the existence of
multiple isolated user-space instances
19
31. clj-kstream-cutter
Edmilson Alves 0 Edmilson Alves -LRB- born February
17 , 1976 -RRB- , is a Brazilian midfielder who currently
plays for Roasso Kumamoto in the J. League Division 2 .
[ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …]
31
Input:
Output:
41. Our Distributed Tool
Application Stack in Docker
Edmilson Alves 0 Edmilson Alves -LRB- born February 17 , 1976 -RRB- , is
a Brazilian midfielder who currently plays for Roasso Kumamoto in the J.
League Division 2 .
[ Edmilson, Alves, 0, Edmilson, Alves, LRB, born …]
Alves ~10 Alves ~8
{“name“: “Alves“, “count“: 18, “time“: “January 26th 2017, 17:03:00.000”}
Alves 18
41
44. …to a datacenter setup
$ export DOCKER_HOST=tcp://my.datacenter.de:2576
Build a Docker Swarm
44
45. Current Challenges
• Kafka Streams still at least once
• Persistent and durable Storage
• Still need capacity planning
• Testing / Debugging is still a challenge
• Consistency of the state storage
• Processing Time vs. Event Time
• How about Amdahl’s law
45