Pulsar Virtual Summit North America 2021
Infinitic:
Building a Workflow
Engine on Top of Pulsar
Pulsar Virtual Summit North America 2021
Gilles Barbier
Infinitic Creator
Building workflow engines for 10 years
- Leetix (SAAS) automation marketing
- Zenaton (SAAS) workflow builder for developers
- Infinitic (F/OSS) orchestration engine at scale
Building in public => https://infinitic.substack.com
Pulsar Virtual Summit North America 2021
Today you will learn
● What Infinitic is, and how it can help you build an
infrastructure of distributed microservices
○ providing a event-based RPC mechanism
○ easing the coding of complex business
requirements involving multiple
microservices
○ making your infrastructure scalable, resilient
and observable
● Status and Roadmap
Pulsar Virtual Summit North America 2021
What Is Infinitic?
I. An event-based RPC framework for building distributed microservices
on top of Pulsar
II. A powerful workflow engine
Plugs into existing infrastructure
Seamless integration with Pulsar, requiring no Pulsar knowledge
Pulsar Virtual Summit North America 2021
Microservices
Pulsar Virtual Summit North America 2021
HTTP Architecture
Pulsar Virtual Summit North America 2021
Event-Driven Architecture
Pulsar Virtual Summit North America 2021
Infinitic: Event-Based RPC
Share services interfaces
Pulsar Virtual Summit North America 2021
Infinitic: Client and Worker SDKs
Pulsar Virtual Summit North America 2021
Infinitic Syntax: Asynchronous Call
Client
Service Interface
Pulsar Virtual Summit North America 2021
Infinitic Syntax: Synchronous Calls
Client
Service Interface
Pulsar Virtual Summit North America 2021
Infinitic: Failure Handling
getDurationBeforeRetry method
implements the retry strategy
Service Implementation
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration?
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration?
👎
Pulsar Virtual Summit North America 2021
Infinitic: Event-Based RPC Orchestration
👍
Pulsar Virtual Summit North America 2021
Infinitic: Event-Based RPC Orchestration
Workflow Interface
Client
Workflow Worker
Pulsar Virtual Summit North America 2021
Infinitic: Code Your Workflows
https://infinitic.io/2020/10/13/code-is-the-best-dsl-for-building-workflows/
stubs
Workflow states are
automatically saved
Pulsar Virtual Summit North America 2021
How does it work?
https://infinitic.io/2020/11/16/under-the-hood-of-an-event-driven-workflow-as-code-engine/
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration Features
● Sequential tasks
● Child - workflows
● Events
● Waiting
● Fan-out
● Fan-in
● Tags
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration Features
● Sequential tasks
● Child - workflows
● Events
● Waiting
● Fan-out
● Fan-in
● Tags
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration Features
● Sequential tasks
● Child - workflows
● Events
● Waiting
● Fan-out
● Fan-in
● Tags
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration Features
● Sequential tasks
● Child - workflows
● Events
● Waiting
● Fan-out
● Fan-in
● Tags
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration Features
● Sequential tasks
● Child - workflows
● Events
● Waiting
● Fan-out
● Fan-in
● Tags
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration Features
● Sequential tasks
● Child - workflows
● Events
● Waiting
● Fan-out
● Fan-in
● Tags
Pulsar Virtual Summit North America 2021
Infinitic: Orchestration Features
● Sequential tasks
● Child - workflows
● Events
● Waiting
● Fan-out
● Fan-in
● Tags
Pulsar Virtual Summit North America 2021
Infinitic Into An Existing Infrastructure
Pulsar Virtual Summit North America 2021
Infinitic To Create Your Own Workflow DSL
Possible to reimplement json specification such as:
- Amazon States Language (Step functions)
- serverlessworkflow.io
Pulsar Virtual Summit North America 2021
Infinitic Performance
Preliminary results obtained during last Pulsar
Hackathon (workflow with 6 dummy tasks)
● 20 workflows completed /s on a MacBook Pro
● 150 workflows completed /s on a small
production cluster
Additional optimizations are coming 🚀
Pulsar Virtual Summit North America 2021
Infinitic Resilience
● Pulsar redundancies
● Task Workers redundancies
● Workflow Workers redundancies
● Technical issues -> automatic retry
● Unstoppable workflows:
○ Failed tasks can be handled
○ Timeout tasks can be handled
Pulsar Virtual Summit North America 2021
Current Status
● Current version: v0.6.4
● Ready for PoC and MVP deployment
● Validated by a marketing company to power configurable
automated emails at scale
● Stable version: target end 2021
Pulsar Virtual Summit North America 2021
Roadmap
● Dashboards! (Ideally based on Pulsar SQL)
● Deployed as Stateful Functions
● More workflow features
● Protobuf integration
● More languages (contributors welcome!)
Pulsar Virtual Summit North America 2021
Using Infinitic
● Code: github.com/infiniticio/infinitic
● Documentation: docs.infinitic.io
● Subscribe to: infinitic.substack.com
● Solutions: infinitic.io
Pulsar Virtual Summit North America 2021
Questions?

Infinitic: Building a Workflow Engine on Top of Pulsar - Pulsar Summit NA 2021

  • 1.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Building a Workflow Engine on Top of Pulsar
  • 2.
    Pulsar Virtual SummitNorth America 2021 Gilles Barbier Infinitic Creator Building workflow engines for 10 years - Leetix (SAAS) automation marketing - Zenaton (SAAS) workflow builder for developers - Infinitic (F/OSS) orchestration engine at scale Building in public => https://infinitic.substack.com
  • 3.
    Pulsar Virtual SummitNorth America 2021 Today you will learn ● What Infinitic is, and how it can help you build an infrastructure of distributed microservices ○ providing a event-based RPC mechanism ○ easing the coding of complex business requirements involving multiple microservices ○ making your infrastructure scalable, resilient and observable ● Status and Roadmap
  • 4.
    Pulsar Virtual SummitNorth America 2021 What Is Infinitic? I. An event-based RPC framework for building distributed microservices on top of Pulsar II. A powerful workflow engine Plugs into existing infrastructure Seamless integration with Pulsar, requiring no Pulsar knowledge
  • 5.
    Pulsar Virtual SummitNorth America 2021 Microservices
  • 6.
    Pulsar Virtual SummitNorth America 2021 HTTP Architecture
  • 7.
    Pulsar Virtual SummitNorth America 2021 Event-Driven Architecture
  • 8.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Event-Based RPC Share services interfaces
  • 9.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Client and Worker SDKs
  • 10.
    Pulsar Virtual SummitNorth America 2021 Infinitic Syntax: Asynchronous Call Client Service Interface
  • 11.
    Pulsar Virtual SummitNorth America 2021 Infinitic Syntax: Synchronous Calls Client Service Interface
  • 12.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Failure Handling getDurationBeforeRetry method implements the retry strategy Service Implementation
  • 13.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration?
  • 14.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration? 👎
  • 15.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Event-Based RPC Orchestration 👍
  • 16.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Event-Based RPC Orchestration Workflow Interface Client Workflow Worker
  • 17.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Code Your Workflows https://infinitic.io/2020/10/13/code-is-the-best-dsl-for-building-workflows/ stubs Workflow states are automatically saved
  • 18.
    Pulsar Virtual SummitNorth America 2021 How does it work? https://infinitic.io/2020/11/16/under-the-hood-of-an-event-driven-workflow-as-code-engine/
  • 19.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration Features ● Sequential tasks ● Child - workflows ● Events ● Waiting ● Fan-out ● Fan-in ● Tags
  • 20.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration Features ● Sequential tasks ● Child - workflows ● Events ● Waiting ● Fan-out ● Fan-in ● Tags
  • 21.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration Features ● Sequential tasks ● Child - workflows ● Events ● Waiting ● Fan-out ● Fan-in ● Tags
  • 22.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration Features ● Sequential tasks ● Child - workflows ● Events ● Waiting ● Fan-out ● Fan-in ● Tags
  • 23.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration Features ● Sequential tasks ● Child - workflows ● Events ● Waiting ● Fan-out ● Fan-in ● Tags
  • 24.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration Features ● Sequential tasks ● Child - workflows ● Events ● Waiting ● Fan-out ● Fan-in ● Tags
  • 25.
    Pulsar Virtual SummitNorth America 2021 Infinitic: Orchestration Features ● Sequential tasks ● Child - workflows ● Events ● Waiting ● Fan-out ● Fan-in ● Tags
  • 26.
    Pulsar Virtual SummitNorth America 2021 Infinitic Into An Existing Infrastructure
  • 27.
    Pulsar Virtual SummitNorth America 2021 Infinitic To Create Your Own Workflow DSL Possible to reimplement json specification such as: - Amazon States Language (Step functions) - serverlessworkflow.io
  • 28.
    Pulsar Virtual SummitNorth America 2021 Infinitic Performance Preliminary results obtained during last Pulsar Hackathon (workflow with 6 dummy tasks) ● 20 workflows completed /s on a MacBook Pro ● 150 workflows completed /s on a small production cluster Additional optimizations are coming 🚀
  • 29.
    Pulsar Virtual SummitNorth America 2021 Infinitic Resilience ● Pulsar redundancies ● Task Workers redundancies ● Workflow Workers redundancies ● Technical issues -> automatic retry ● Unstoppable workflows: ○ Failed tasks can be handled ○ Timeout tasks can be handled
  • 30.
    Pulsar Virtual SummitNorth America 2021 Current Status ● Current version: v0.6.4 ● Ready for PoC and MVP deployment ● Validated by a marketing company to power configurable automated emails at scale ● Stable version: target end 2021
  • 31.
    Pulsar Virtual SummitNorth America 2021 Roadmap ● Dashboards! (Ideally based on Pulsar SQL) ● Deployed as Stateful Functions ● More workflow features ● Protobuf integration ● More languages (contributors welcome!)
  • 32.
    Pulsar Virtual SummitNorth America 2021 Using Infinitic ● Code: github.com/infiniticio/infinitic ● Documentation: docs.infinitic.io ● Subscribe to: infinitic.substack.com ● Solutions: infinitic.io
  • 33.
    Pulsar Virtual SummitNorth America 2021 Questions?