1. Serverless orchestration and automation
with GCPWorkflows
May 2021 - Devops Pro Europe, Vilnius
Márton Kodok / @martonkodok
Google Developer Expert at REEA.net

2. ● Among the Top 3 romanians on Stackoverflow 190K reputation
● Google Developer Expert on Cloud technologies
● Crafting Web/Mobile backends at REEA.net
● BigQuery + Redis database engine expert
Slideshare: martonkodok
Articles: martonkodok.medium.com
Twitter: @martonkodok
StackOverflow: pentium10
GitHub: pentium10
Serverless orchestration and automation with GCP Workflows @martonkodok
About me

3. 1. Challenges in connecting services
2. What is Workflows? - HTTP based service orchestration and automation
3. Introduction to Workflows - automate complex processes
4. Practical use cases
5. Automate, orchestrate and provide reliable line-of-business automation
6. Conclusions
Agenda
Serverless orchestration and automation with GCP Workflows @martonkodok

4. ● Connectivity - should be easy, but in reality you need to figure out
● Parse the results
● Decisions and conditional step executions
● No built-in error handling, logging
● Struggling with polling, retries, exponential backoff
● Scaling up and down to zero
● Authentication
Challenges in connecting services
Serverless orchestration and automation with GCP Workflows @martonkodok

5. Meet Workflows
https://cloud.google.com/workflows
Serverless orchestration and automation with GCP Workflows @martonkodok

6. Workflows in Google Cloud portfolio
Introduction
Orchestrate any
1. Google Cloud API
2. SaaS API
3. External or private APIs.
Serverless
Compute
External
API’s
Google
API’s
etc...
Workflows - orchestrate & integrate
SaaS
API’s
Private
API’s
Other
Clouds

7. Step-Automation-as-a-Service - Serverless HTTP service automation
Declarative workflow language (YAML, JSON)
Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020
Built-in decision and conditional executions expression formulas, operation on var
Subworkflows similar to routine in a programming language with input/return var
Support for external API calls out of the box support outside of Google Cloud
Integrates with any Google Cloud product without worrying about authentication
What is GCP Workflows?
Serverless orchestration and automation with GCP Workflows @martonkodok

8. OAuth, OIDC, Secret Manager integration
Enterprise
Security
Keep your workflows secure
X
Authenticated
Invocations
Authenticated
calls to Google Cloud
services
Integration with
Secret Manager
Encryption at rest and
in transit
External API 1
External API 2 External API

9. Code example
@martonkodok
- callMyFunction:
call: http.get
args:
url: https://us-central1-project123.cloudfunctions.net/...
query:
metric: NoOfTrs
result: metricResult
- saveResult:
switch:
- condition: ${ metricResult.body.TrNo > 100 }
call: http.post
args:
url: https://mydeployment.notify....
body:
Metric: ${ metricResult.body.TrNo }
getMetric
TrNo>100?
End
Yes
No
{“metric”:”NoOfTrs”}
Notify

10. hello.yaml
Anatomy of a Cloud Workflow!
Serverless orchestration and automation with GCP Workflows @martonkodok
workflowRevisionId: 000001-de1
argument: 'null'
startTime: '2020-10-16T20:19:34.448323739Z'
name:
projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c-
4065-a696-940bc7658c33
result: '"Hello, Cloud Workflows!"'
endTime: '2020-10-16T20:19:34.845536315Z'
state: SUCCEEDED
Deploying gcloud beta workflows deploy hello --source=hello.yaml
Executing gcloud beta workflows execute hello --data={“var”:”value”}
Describing gcloud beta workflows executions describe-last

11. Sample Workflows
Serverless orchestration and automation with GCP Workflows @martonkodok

12. IT management automation
Combine automation with
scheduler
Wait for service checks
Orchestrate work across
Compute Engine, PubSub,
Stackdriver and other Google
Cloud Products
9 AM trigger
Start a Compute
Engine VM
Log the event
App
Started?
No
Wait 60 seconds
Notify the team

13. E-commerce invoice generation with Workflows
Steps orchestration
Reliable execution, with error
handling and retries
Orchestration microservices
or other API’s
Create an invoice
Generate PDF
Send PDF via email
Receive Order
Cloud
Run
App
Engine
Cloud
Run

14. Sending reminders to accounts with overdue payments
Process array elements
Execute steps
conditionally
For each customer:
Get a list of customers
Send reminder
overdue
?
Yes

15. Code Examples
Serverless orchestration and automation with GCP Workflows @martonkodok

16. 1. HTTP Post
2. Sequence two steps
3. HTTP Post combined with Secret Manager credentials
4. Switch block
5. Working with subworkflows
Code Examples
Serverless orchestration and automation with GCP Workflows @martonkodok

17. http_post.yaml
Making an external HTTP POST request
Serverless orchestration and automation with GCP Workflows @martonkodok
{
"archived":false,
"created_at":"2020-10-16T17:40:17+0000",
"id":"bit.ly/35452TM",
"link":"https://bit.ly/35452TM",
"long_url":"<truncated>",
}

18. wikipedia.yaml
Sequence two steps to get data from Wikipedia
Serverless orchestration and automation with GCP Workflows @martonkodok

19. subworkflow.yaml
Subworkflow
Serverless orchestration and automation with GCP Workflows @martonkodok
call
call
def

20. retries.yaml
Retries
Serverless orchestration and automation with GCP Workflows @martonkodok
Retries HTTP status codes [429, 502, 503, 504], connection error, or timeout

21. Orchestrate and automate the Cloud
Serverless orchestration and automation with GCP Workflows @martonkodok

22. Firestore Backups the easy way with Cloud Workflows
- Cloud Scheduler
- Cloud Workflows
- Cloud Firestore
- Cloud Storage
Full guide article on:
martonkodok.medium.com
Cloud
Workflows
Cloud
Storage
Cloud
Scheduler
Firebase
Cloud
Firestore
backup
Roles and
Permissions
Cloud IAM
authorize

23. Using Workflows to load Cloud Storage files into BigQuery
- Cloud Workflows
- Cloud Storage
- BigQuery
Full guide article on:
martonkodok.medium.com
Cloud
Storage
BigQuery
Cloud
Workflows
Authenticated
Invocations
foreach

24. Run shell commands and orchestrate Compute Engine VMs
- Cloud Workflows
- Cloud Build
- Compute Engine
- Identity-Aware Proxy
Full guide article on:
martonkodok.medium.com
Cloud Build
Shell command
Serverless Secure Connect Exec command
Firewall
Compute Engine
Cloud IAP Tunnel
Cloud
Workflows

25. Run shell commands and orchestrate Compute Engine VMs
- Cloud Workflows
- Cloud Build
- Compute Engine
- Identity-Aware Proxy
Full guide article on:
martonkodok.medium.com
Cloud Workflow Steps
Cloud
Workflows
Execution
Authorize
Cloud Build
Roles and
Permissions
Cloud IAM
Shell
Service Account
Start VM
Compute
Engine
Submit Stop VM
Compute
Engine
Firewall
Compute Engine
Cloud
IAP
Tunnel
wait wait

26. Conclusions
Serverless orchestration and automation with GCP Workflows @martonkodok

27. Reliable workflow execution - execute workflows for enterprise business apps
Low latency of execution - no cold starts
Built-in error handling out of the box error handling with configurable retry policies
Passing variable built-in JSON parsing and expression-based variable manipulation
Rich runtime iterating through an array, embedded steps for readability
Secret Manager integration out of the box
Cloud Logging out of the box integration with Cloud Logging
Reading from Firestore read/write an entry using Yaml syntax
Benefits of Cloud Workflows
Serverless orchestration and automation with GCP Workflows @martonkodok

28. Easy to build/operate
Scales out
Does not lose state
Handles errors/timeouts
Out-of-the-box support of Cloud APIs
Auditable
Developer friendly
Serverless orchestration and automation with GCP Workflows @martonkodok

29. The possibilities are endless
Marketing Retail IndustrialandIoT Developer
Event driven marketing
workflow execution
Relay conversions to
customer profiles in external
services
Workflow based emails,
discounts, promotions
Order management
Inventory chain operations
Data gathering and
processing
Synchronize systems
Generate state machines
Verify equipment lifecycle
Workflow based
maintenance needs
Digitalization of internal
policies
Automate the Cloud
Shell-script replacement
Orchestrate devops
workflows
@martonkodok

30. Thank you. Q&A.
Slides available on:
slideshare.net/martonkodok
Reea.net - Integrated web solutions driven by creativity
to deliver projects.