Cloud Workflows What's new in serverless orchestration and automation

Cloud Workﬂows: What's new in serverless
orchestration and automation
October 2021 - GDG DevFest 2021 - Doha
Márton Kodok / @martonkodok
Google Developer Expert at REEA.net

● Among the Top 3 romanians on Stackoverflow 195K reputation
● Google Developer Expert on Cloud technologies
● Crafting Web/Mobile backends at REEA.net
● BigQuery + Redis database engine expert
Slideshare: martonkodok
Articles: martonkodok.medium.com
Twitter: @martonkodok
StackOverflow: pentium10
GitHub: pentium10
What's new in serverless orchestration and automation @martonkodok
About me

1. Challenges in connecting services
2. What is Cloud Workflows?
3. Practical use cases
4. What’s new in Workflows?
5. Advanced examples
6. Conclusions
Agenda

● Connectivity - should be easy, but in reality you need to figure out
● Parse the results
● Decisions and conditional step executions
● No built-in error handling, logging
● Struggling with polling, retries, exponential backoff
● Scaling up and down to zero
● Authentication
● Missed capability of capturing callbacks
Challenges in connecting services

MeetCloudWorkﬂows

Workﬂows in Google Cloud portfolio
Introduction
Orchestrate any
1. Google Cloud API
2. SaaS API
3. External or private APIs.
Serverless
Compute
External
API’s
Google
API’s
etc...
Workflows - orchestrate & integrate
SaaS
API’s
Private
API’s
Other
Clouds

Step-Automation-as-a-Service - Serverless HTTP service automation
Declarative workflow language (YAML, JSON)
Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020
Built-in decision and conditional executions expression formulas, operation on var
Subworkflows similar to routine in a programming language with input/return var
Support for external API calls out of the box support outside of Google Cloud
Integrates with any Google Cloud product without worrying about authentication
Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks
What is GCP Workﬂows?

OAuth, OIDC, Secret Manager integration
Enterprise
Security
Keep your workflows secure
X
Authenticated
Invocations
Authenticated
calls to Google Cloud
services
Integration with
Secret Manager
Encryption at rest and
in transit
External API 1
External API 2 External API

Anatomy of a Cloud Workﬂow!

hello.yaml
Developing for Cloud Workﬂow!
workflowRevisionId: 000001-de1
argument: 'null'
startTime: '2020-10-16T20:19:34.448323739Z'
name:
projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c-
4065-a696-940bc7658c33
result: '"Hello, Cloud Workflows!"'
endTime: '2020-10-16T20:19:34.845536315Z'
state: SUCCEEDED
Deploying gcloud workflows deploy hello --source=hello.yaml
Executing gcloud workflows execute hello --data={“var”:”value”}
Describing gcloud workflows executions describe-last

Sample Workﬂows

IT management automation
Combine automation with
scheduler
Wait for service checks
Orchestrate work across
Compute Engine, PubSub,
Stackdriver and other Google
Cloud Products
9 AM trigger
Start a Compute
Engine VM
Log the event
App
Started?
No
Wait 60 seconds
Notify the team

E-commerce invoice generation with Workﬂows
Steps orchestration
Reliable execution, with error
handling and retries
Orchestration microservices
or other API’s
Create an invoice
Generate PDF
Send PDF via email
Receive Order
Cloud
Run
App
Engine
Cloud
Run

Code Examples

1. HTTP calls (GET or POST)
2. Sequence two steps
3. Working with subworkflows
Workﬂow Basics

http_post.yaml
Making an external HTTP POST request
{
"archived":false,
"created_at":"2020-10-16T17:40:17+0000",
"id":"bit.ly/35452TM",
"link":"https://bit.ly/35452TM",
"long_url":"<truncated>",
}

wikipedia.yaml
Sequence two steps to get data from Wikipedia

subworkflow.yaml
Subworkﬂow
call
call
def

What’s new in Workﬂows

1. Functions: string, JSON, map, iterating syntax
2. Cloud logging and Call logging
3. Connectors: built-in authentication, retries, long-operation
4. Callbacks: enables async-executions, or human in the loop
What’s new in Workﬂows

String Functions

Workﬂow Connectors

They are like client libraries for other services handling auth, retries and waiting.
Connect to other Google Cloud APIs, and to integrate other GCP products.
Connectors

Before: Calling Secret Manager traditionally using HTTP Get calls.
Connectors: Replacing GET and POST calls
After: using Connectors to get Secret Manager value

1. Built in Authentication (calling service account needs permissions)
2. Retries (max_retries: 5, max_delay: 60, ability to define custom retry policies)
3. Timeout (default timeout: 30 minutes, can be increased to 1 year)
4. Long-running operations (blocks the workflow execution until the operation completes)
5. Polling (exponential backoff of 1.25 up to 60 seconds, each step is billable)
Connectors details

Long running operation type of Workﬂow Connectors
Product Operation Type Product Operation Type
BigQuery Job Dataflow Job
Cloud Build Operation Firestore Operation
Cloud Functions Operation Document AI Operation
Cloud SQL Operation Cloud Spanner Operation
Compute Engine Operation Workflows Operation
Execution

Workﬂow Callbacks

They are webhook endpoints where you signal that external event has occurred.
Callbacks allow workflows executions to wait until the callback endpoint gets a call.
Callbacks

1. Modern way to avoid sys.sleep, wait, tracking tables, polling
2. Offers a webhook endpoint (secured with authentication)
3. Timeout (default timeout: 12 hours, can be increased to 1 year)
4. Event based (pauses the workflow execution until the call is received on endpoint)
5. Full callback details (callback_request holds headers,query,body,raw,json)
Callback details

Orchestrate and automate the Cloud

Firestore Backups the easy way with Cloud Workﬂows
- Cloud Scheduler
- Cloud Workflows
- Cloud Firestore
- Cloud Storage
Full guide article on:
martonkodok.medium.com
Cloud
Workﬂows
Cloud
Storage
Cloud
Scheduler
Firebase
Cloud
Firestore
backup
Roles and
Permissions
Cloud IAM
authorize

Using Workflows to load Cloud Storage files into BigQuery
- Cloud Workflows
- Cloud Storage
- BigQuery
Cloud
Storage
BigQuery
Cloud
Workflows
Authenticated
Invocations
foreach

Run shell commands and orchestrate Compute Engine VMs
- Cloud Workflows
- Cloud Build
- Compute Engine
- Identity-Aware Proxy
Cloud Workﬂow Steps
Cloud
Workﬂows
Execution
Authorize
Cloud Build
Roles and
Permissions
Cloud IAM
Shell
Service Account
Start VM
Compute
Engine
Submit Stop VM
Compute
Engine
Firewall
Compute Engine
Cloud
IAP
Tunnel
wait wait

Conclusions

Reliable workflow execution - execute workflows for enterprise business apps
Low latency of execution - no cold starts
Built-in error handling out of the box error handling with configurable retry policies
Passing variable built-in JSON parsing and expression-based variable manipulation
Rich runtime iterating through an array, embedded steps for readability
Secret Manager integration out of the box
Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks
Beneﬁts of Cloud Workﬂows

Easy to build/operate
Scales out
Does not lose state
Handles errors/timeouts
Out-of-the-box support of Cloud APIs
Auditable
Enables async-workflows with Callbacks
Developer friendly

The possibilities are endless
Marketing Retail IndustrialandIoT Developer
Event driven marketing
workflow execution
Relay conversions to
customer profiles in external
services
Workflow based emails,
discounts, promotions
Order management
Inventory chain operations
Data gathering and
processing
Synchronize systems
Generate state machines
Verify equipment lifecycle
Workflow based
maintenance needs
Digitalization of internal
policies
Automate the Cloud
Shell-script replacement
Orchestrate devops
workflows
@martonkodok

Thank you. Q&A.
Slides available on:
slideshare.net/martonkodok
Reea.net - Integrated web solutions driven by creativity
to deliver projects.

Cloud Workflows What's new in serverless orchestration and automation

More Related Content

What's hot

Similar to Cloud Workflows What's new in serverless orchestration and automation

More from Márton Kodok

Recently uploaded

Cloud Workflows What's new in serverless orchestration and automation