Cloud Workflows What's new in serverless orchestration and automation

Márton Kodok
Márton KodokGoogle Developer Expert/Senior Software Engineer/Team-leader/Mentor at Reea
Cloud Workflows: What's new in serverless
orchestration and automation
October 2021 - GDG DevFest 2021 - Doha
Márton Kodok / @martonkodok
Google Developer Expert at REEA.net
● Among the Top 3 romanians on Stackoverflow 195K reputation
● Google Developer Expert on Cloud technologies
● Crafting Web/Mobile backends at REEA.net
● BigQuery + Redis database engine expert
Slideshare: martonkodok
Articles: martonkodok.medium.com
Twitter: @martonkodok
StackOverflow: pentium10
GitHub: pentium10
What's new in serverless orchestration and automation @martonkodok
About me
1. Challenges in connecting services
2. What is Cloud Workflows?
3. Practical use cases
4. What’s new in Workflows?
5. Advanced examples
6. Conclusions
Agenda
What's new in serverless orchestration and automation @martonkodok
● Connectivity - should be easy, but in reality you need to figure out
● Parse the results
● Decisions and conditional step executions
● No built-in error handling, logging
● Struggling with polling, retries, exponential backoff
● Scaling up and down to zero
● Authentication
● Missed capability of capturing callbacks
Challenges in connecting services
What's new in serverless orchestration and automation @martonkodok
MeetCloudWorkflows
What's new in serverless orchestration and automation @martonkodok
Workflows in Google Cloud portfolio
Introduction
Orchestrate any
1. Google Cloud API
2. SaaS API
3. External or private APIs.
Serverless
Compute
External
API’s
Google
API’s
etc...
Workflows - orchestrate & integrate
SaaS
API’s
Private
API’s
Other
Clouds
Step-Automation-as-a-Service - Serverless HTTP service automation
Declarative workflow language (YAML, JSON)
Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020
Built-in decision and conditional executions expression formulas, operation on var
Subworkflows similar to routine in a programming language with input/return var
Support for external API calls out of the box support outside of Google Cloud
Integrates with any Google Cloud product without worrying about authentication
Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks
What is GCP Workflows?
What's new in serverless orchestration and automation @martonkodok
OAuth, OIDC, Secret Manager integration
Enterprise
Security
Keep your workflows secure
X
Authenticated
Invocations
Authenticated
calls to Google Cloud
services
Integration with
Secret Manager
Encryption at rest and
in transit
External API 1
External API 2 External API
Anatomy of a Cloud Workflow!
What's new in serverless orchestration and automation @martonkodok
hello.yaml
Developing for Cloud Workflow!
What's new in serverless orchestration and automation @martonkodok
workflowRevisionId: 000001-de1
argument: 'null'
startTime: '2020-10-16T20:19:34.448323739Z'
name:
projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c-
4065-a696-940bc7658c33
result: '"Hello, Cloud Workflows!"'
endTime: '2020-10-16T20:19:34.845536315Z'
state: SUCCEEDED
Deploying gcloud workflows deploy hello --source=hello.yaml
Executing gcloud workflows execute hello --data={“var”:”value”}
Describing gcloud workflows executions describe-last
Sample Workflows
What's new in serverless orchestration and automation @martonkodok
IT management automation
Combine automation with
scheduler
Wait for service checks
Orchestrate work across
Compute Engine, PubSub,
Stackdriver and other Google
Cloud Products
9 AM trigger
Start a Compute
Engine VM
Log the event
App
Started?
No
Wait 60 seconds
Notify the team
E-commerce invoice generation with Workflows
Steps orchestration
Reliable execution, with error
handling and retries
Orchestration microservices
or other API’s
Create an invoice
Generate PDF
Send PDF via email
Receive Order
Cloud
Run
App
Engine
Cloud
Run
Code Examples
What's new in serverless orchestration and automation @martonkodok
1. HTTP calls (GET or POST)
2. Sequence two steps
3. Working with subworkflows
Workflow Basics
What's new in serverless orchestration and automation @martonkodok
http_post.yaml
Making an external HTTP POST request
What's new in serverless orchestration and automation @martonkodok
{
"archived":false,
"created_at":"2020-10-16T17:40:17+0000",
"id":"bit.ly/35452TM",
"link":"https://bit.ly/35452TM",
"long_url":"<truncated>",
}
wikipedia.yaml
Sequence two steps to get data from Wikipedia
What's new in serverless orchestration and automation @martonkodok
subworkflow.yaml
Subworkflow
What's new in serverless orchestration and automation @martonkodok
call
call
def
What’s new in Workflows
What's new in serverless orchestration and automation @martonkodok
1. Functions: string, JSON, map, iterating syntax
2. Cloud logging and Call logging
3. Connectors: built-in authentication, retries, long-operation
4. Callbacks: enables async-executions, or human in the loop
What’s new in Workflows
What's new in serverless orchestration and automation @martonkodok
String Functions
What's new in serverless orchestration and automation @martonkodok
Workflow Connectors
What's new in serverless orchestration and automation @martonkodok
They are like client libraries for other services handling auth, retries and waiting.
Connect to other Google Cloud APIs, and to integrate other GCP products.
Connectors
What's new in serverless orchestration and automation @martonkodok
Before: Calling Secret Manager traditionally using HTTP Get calls.
Connectors: Replacing GET and POST calls
What's new in serverless orchestration and automation @martonkodok
After: using Connectors to get Secret Manager value
1. Built in Authentication (calling service account needs permissions)
2. Retries (max_retries: 5, max_delay: 60, ability to define custom retry policies)
3. Timeout (default timeout: 30 minutes, can be increased to 1 year)
4. Long-running operations (blocks the workflow execution until the operation completes)
5. Polling (exponential backoff of 1.25 up to 60 seconds, each step is billable)
Connectors details
What's new in serverless orchestration and automation @martonkodok
Long running operation type of Workflow Connectors
What's new in serverless orchestration and automation @martonkodok
Product Operation Type Product Operation Type
BigQuery Job Dataflow Job
Cloud Build Operation Firestore Operation
Cloud Functions Operation Document AI Operation
Cloud SQL Operation Cloud Spanner Operation
Compute Engine Operation Workflows Operation
Execution
Workflow Callbacks
What's new in serverless orchestration and automation @martonkodok
They are webhook endpoints where you signal that external event has occurred.
Callbacks allow workflows executions to wait until the callback endpoint gets a call.
Callbacks
What's new in serverless orchestration and automation @martonkodok
1. Modern way to avoid sys.sleep, wait, tracking tables, polling
2. Offers a webhook endpoint (secured with authentication)
3. Timeout (default timeout: 12 hours, can be increased to 1 year)
4. Event based (pauses the workflow execution until the call is received on endpoint)
5. Full callback details (callback_request holds headers,query,body,raw,json)
Callback details
What's new in serverless orchestration and automation @martonkodok
Orchestrate and automate the Cloud
What's new in serverless orchestration and automation @martonkodok
Firestore Backups the easy way with Cloud Workflows
- Cloud Scheduler
- Cloud Workflows
- Cloud Firestore
- Cloud Storage
Full guide article on:
martonkodok.medium.com
Cloud
Workflows
Cloud
Storage
Cloud
Scheduler
Firebase
Cloud
Firestore
backup
Roles and
Permissions
Cloud IAM
authorize
Using Workflows to load Cloud Storage files into BigQuery
- Cloud Workflows
- Cloud Storage
- BigQuery
Full guide article on:
martonkodok.medium.com
Cloud
Storage
BigQuery
Cloud
Workflows
Authenticated
Invocations
foreach
Run shell commands and orchestrate Compute Engine VMs
- Cloud Workflows
- Cloud Build
- Compute Engine
- Identity-Aware Proxy
Full guide article on:
martonkodok.medium.com
Cloud Workflow Steps
Cloud
Workflows
Execution
Authorize
Cloud Build
Roles and
Permissions
Cloud IAM
Shell
Service Account
Start VM
Compute
Engine
Submit Stop VM
Compute
Engine
Firewall
Compute Engine
Cloud
IAP
Tunnel
wait wait
Conclusions
What's new in serverless orchestration and automation @martonkodok
Reliable workflow execution - execute workflows for enterprise business apps
Low latency of execution - no cold starts
Built-in error handling out of the box error handling with configurable retry policies
Passing variable built-in JSON parsing and expression-based variable manipulation
Rich runtime iterating through an array, embedded steps for readability
Secret Manager integration out of the box
Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks
Benefits of Cloud Workflows
What's new in serverless orchestration and automation @martonkodok
Easy to build/operate
Scales out
Does not lose state
Handles errors/timeouts
Out-of-the-box support of Cloud APIs
Auditable
Enables async-workflows with Callbacks
Developer friendly
What's new in serverless orchestration and automation @martonkodok
The possibilities are endless
Marketing Retail IndustrialandIoT Developer
Event driven marketing
workflow execution
Relay conversions to
customer profiles in external
services
Workflow based emails,
discounts, promotions
Order management
Inventory chain operations
Data gathering and
processing
Synchronize systems
Generate state machines
Verify equipment lifecycle
Workflow based
maintenance needs
Digitalization of internal
policies
Automate the Cloud
Shell-script replacement
Orchestrate devops
workflows
@martonkodok
Thank you. Q&A.
Slides available on:
slideshare.net/martonkodok
Reea.net - Integrated web solutions driven by creativity
to deliver projects.
1 of 38

Recommended

Serverless orchestration and automation with Cloud Workflows by
Serverless orchestration and automation with Cloud WorkflowsServerless orchestration and automation with Cloud Workflows
Serverless orchestration and automation with Cloud WorkflowsMárton Kodok
268 views30 slides
BigQuery ML - Machine learning at scale using SQL by
BigQuery ML - Machine learning at scale using SQLBigQuery ML - Machine learning at scale using SQL
BigQuery ML - Machine learning at scale using SQLMárton Kodok
305 views42 slides
9 steps to awesome with kubernetes by
9 steps to awesome with kubernetes9 steps to awesome with kubernetes
9 steps to awesome with kubernetesBaraniBuuny
445 views121 slides
Clearing Airflow Obstructions by
Clearing Airflow ObstructionsClearing Airflow Obstructions
Clearing Airflow ObstructionsTatiana Al-Chueyr
226 views70 slides
Full Stack Developer by
Full Stack DeveloperFull Stack Developer
Full Stack DeveloperAkbar Uddin
113 views1 slide
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R... by
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...HostedbyConfluent
506 views16 slides

More Related Content

What's hot

Microservice Plumbing - Glynn Bird - Codemotion Rome 2017 by
Microservice Plumbing  - Glynn Bird - Codemotion Rome 2017Microservice Plumbing  - Glynn Bird - Codemotion Rome 2017
Microservice Plumbing - Glynn Bird - Codemotion Rome 2017Codemotion
585 views29 slides
Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli... by
Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli...Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli...
Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli...Codemotion
485 views27 slides
Containerizing ContentBox CMS by
Containerizing ContentBox CMSContainerizing ContentBox CMS
Containerizing ContentBox CMSGavin Pickin
212 views33 slides
TIAD : Automate everything with Google Cloud by
TIAD : Automate everything with Google CloudTIAD : Automate everything with Google Cloud
TIAD : Automate everything with Google CloudThe Incredible Automation Day
3.4K views41 slides
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R... by
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...HostedbyConfluent
95 views16 slides
Spring Cloud Data Flow Overview by
Spring Cloud Data Flow OverviewSpring Cloud Data Flow Overview
Spring Cloud Data Flow OverviewVMware Tanzu
1.4K views58 slides

What's hot(20)

Microservice Plumbing - Glynn Bird - Codemotion Rome 2017 by Codemotion
Microservice Plumbing  - Glynn Bird - Codemotion Rome 2017Microservice Plumbing  - Glynn Bird - Codemotion Rome 2017
Microservice Plumbing - Glynn Bird - Codemotion Rome 2017
Codemotion585 views
Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli... by Codemotion
Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli...Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli...
Webhooks do's and dont's: what we learned after integrating +100 APIs - Giuli...
Codemotion485 views
Containerizing ContentBox CMS by Gavin Pickin
Containerizing ContentBox CMSContainerizing ContentBox CMS
Containerizing ContentBox CMS
Gavin Pickin212 views
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R... by HostedbyConfluent
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...
Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, R...
Spring Cloud Data Flow Overview by VMware Tanzu
Spring Cloud Data Flow OverviewSpring Cloud Data Flow Overview
Spring Cloud Data Flow Overview
VMware Tanzu1.4K views
Cloud Native Application Development - build fast, low TCO, scalable & agile ... by Lucas Jellema
Cloud Native Application Development - build fast, low TCO, scalable & agile ...Cloud Native Application Development - build fast, low TCO, scalable & agile ...
Cloud Native Application Development - build fast, low TCO, scalable & agile ...
Lucas Jellema157 views
Yannis Zarkadas. Enterprise data science workflows on kubeflow by MarynaHoldaieva
Yannis Zarkadas. Enterprise data science workflows on kubeflowYannis Zarkadas. Enterprise data science workflows on kubeflow
Yannis Zarkadas. Enterprise data science workflows on kubeflow
MarynaHoldaieva70 views
Building a Web Application with Kafka as your Database by confluent
Building a Web Application with Kafka as your DatabaseBuilding a Web Application with Kafka as your Database
Building a Web Application with Kafka as your Database
confluent269 views
Serverless and Servicefull Applications - Where Microservices complements Ser... by Red Hat Developers
Serverless and Servicefull Applications - Where Microservices complements Ser...Serverless and Servicefull Applications - Where Microservices complements Ser...
Serverless and Servicefull Applications - Where Microservices complements Ser...
Red Hat Developers1.9K views
5 lessons learned for successful migration to Confluent cloud | Natan Silinit... by HostedbyConfluent
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
5 lessons learned for successful migration to Confluent cloud | Natan Silinit...
HostedbyConfluent432 views
스타트업을 위한 Confluent 세미나 by confluent
스타트업을 위한 Confluent 세미나스타트업을 위한 Confluent 세미나
스타트업을 위한 Confluent 세미나
confluent328 views
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl... by HostedbyConfluent
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...
HostedbyConfluent1.1K views
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ by HostedbyConfluent
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQCloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ
HostedbyConfluent329 views
GraphQL over REST at Reactathon 2018 by Sashko Stubailo
GraphQL over REST at Reactathon 2018GraphQL over REST at Reactathon 2018
GraphQL over REST at Reactathon 2018
Sashko Stubailo1.4K views
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t by Toshiaki Maki
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1tServerless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Toshiaki Maki5.2K views
Achieving end-to-end visibility into complex event-sourcing transactions usin... by HostedbyConfluent
Achieving end-to-end visibility into complex event-sourcing transactions usin...Achieving end-to-end visibility into complex event-sourcing transactions usin...
Achieving end-to-end visibility into complex event-sourcing transactions usin...
HostedbyConfluent564 views
What is Google Cloud Platform - GDG DevFest 18 Depok by Imre Nagi
What is Google Cloud Platform - GDG DevFest 18 DepokWhat is Google Cloud Platform - GDG DevFest 18 Depok
What is Google Cloud Platform - GDG DevFest 18 Depok
Imre Nagi308 views
Handle insane devices traffic using Google Cloud Platform - Andrea Ulisse - C... by Codemotion
Handle insane devices traffic using Google Cloud Platform - Andrea Ulisse - C...Handle insane devices traffic using Google Cloud Platform - Andrea Ulisse - C...
Handle insane devices traffic using Google Cloud Platform - Andrea Ulisse - C...
Codemotion729 views

Similar to Cloud Workflows What's new in serverless orchestration and automation

Serverless orchestration and automation with Cloud Workflows by
Serverless orchestration and automation with Cloud WorkflowsServerless orchestration and automation with Cloud Workflows
Serverless orchestration and automation with Cloud WorkflowsMárton Kodok
371 views32 slides
Serverless orchestration and automation with Cloud Workflows by
Serverless orchestration and automation with Cloud WorkflowsServerless orchestration and automation with Cloud Workflows
Serverless orchestration and automation with Cloud WorkflowsMárton Kodok
762 views28 slides
Google Cloud Next '22 Recap: Serverless & Data edition by
Google Cloud Next '22 Recap: Serverless & Data editionGoogle Cloud Next '22 Recap: Serverless & Data edition
Google Cloud Next '22 Recap: Serverless & Data editionDaniel Zivkovic
91 views44 slides
GDG DevFest Romania - Architecting for the Google Cloud Platform by
GDG DevFest Romania - Architecting for the Google Cloud PlatformGDG DevFest Romania - Architecting for the Google Cloud Platform
GDG DevFest Romania - Architecting for the Google Cloud PlatformMárton Kodok
462 views55 slides
2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge Luca by
2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge Luca2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge Luca
2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge LucaaOS Community
49 views34 slides
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon by
6. DISZ - Webalkalmazások skálázhatósága  a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága  a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud PlatformonMárton Kodok
133 views44 slides

Similar to Cloud Workflows What's new in serverless orchestration and automation(20)

Serverless orchestration and automation with Cloud Workflows by Márton Kodok
Serverless orchestration and automation with Cloud WorkflowsServerless orchestration and automation with Cloud Workflows
Serverless orchestration and automation with Cloud Workflows
Márton Kodok371 views
Serverless orchestration and automation with Cloud Workflows by Márton Kodok
Serverless orchestration and automation with Cloud WorkflowsServerless orchestration and automation with Cloud Workflows
Serverless orchestration and automation with Cloud Workflows
Márton Kodok762 views
Google Cloud Next '22 Recap: Serverless & Data edition by Daniel Zivkovic
Google Cloud Next '22 Recap: Serverless & Data editionGoogle Cloud Next '22 Recap: Serverless & Data edition
Google Cloud Next '22 Recap: Serverless & Data edition
Daniel Zivkovic91 views
GDG DevFest Romania - Architecting for the Google Cloud Platform by Márton Kodok
GDG DevFest Romania - Architecting for the Google Cloud PlatformGDG DevFest Romania - Architecting for the Google Cloud Platform
GDG DevFest Romania - Architecting for the Google Cloud Platform
Márton Kodok462 views
2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge Luca by aOS Community
2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge Luca2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge Luca
2019-05-16 aOS Luxembourg - 6 - Flow avancé - Serge Luca
aOS Community49 views
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon by Márton Kodok
6. DISZ - Webalkalmazások skálázhatósága  a Google Cloud Platformon6. DISZ - Webalkalmazások skálázhatósága  a Google Cloud Platformon
6. DISZ - Webalkalmazások skálázhatósága a Google Cloud Platformon
Márton Kodok133 views
Doctor Flow: Enterprise Flows best practices - patterns (SharePoint Saturday... by serge luca
Doctor Flow:  Enterprise Flows best practices - patterns (SharePoint Saturday...Doctor Flow:  Enterprise Flows best practices - patterns (SharePoint Saturday...
Doctor Flow: Enterprise Flows best practices - patterns (SharePoint Saturday...
serge luca493 views
Google Cloud Platform (GCP) At a Glance by Cloud Analogy
Google Cloud Platform (GCP)  At a GlanceGoogle Cloud Platform (GCP)  At a Glance
Google Cloud Platform (GCP) At a Glance
Cloud Analogy1.2K views
Automazione serverless con Azure Functions e PowerShell - Marco Obinu - DevOp... by Marco Obinu
Automazione serverless con Azure Functions e PowerShell - Marco Obinu - DevOp...Automazione serverless con Azure Functions e PowerShell - Marco Obinu - DevOp...
Automazione serverless con Azure Functions e PowerShell - Marco Obinu - DevOp...
Marco Obinu231 views
Spring and Pivotal Application Service - SpringOne Tour - Boston by VMware Tanzu
Spring and Pivotal Application Service - SpringOne Tour - BostonSpring and Pivotal Application Service - SpringOne Tour - Boston
Spring and Pivotal Application Service - SpringOne Tour - Boston
VMware Tanzu678 views
GDG Heraklion - Architecting for the Google Cloud Platform by Márton Kodok
GDG Heraklion - Architecting for the Google Cloud PlatformGDG Heraklion - Architecting for the Google Cloud Platform
GDG Heraklion - Architecting for the Google Cloud Platform
Márton Kodok754 views
Why And When Should We Consider Stream Processing In Our Solutions Teqnation ... by Soroosh Khodami
Why And When Should We Consider Stream Processing In Our Solutions Teqnation ...Why And When Should We Consider Stream Processing In Our Solutions Teqnation ...
Why And When Should We Consider Stream Processing In Our Solutions Teqnation ...
Soroosh Khodami115 views
Cloud Run - the rise of serverless and containerization by Márton Kodok
Cloud Run - the rise of serverless and containerizationCloud Run - the rise of serverless and containerization
Cloud Run - the rise of serverless and containerization
Márton Kodok64 views
Continuous Lifecycle London 2018 Event Keynote by Weaveworks
Continuous Lifecycle London 2018 Event KeynoteContinuous Lifecycle London 2018 Event Keynote
Continuous Lifecycle London 2018 Event Keynote
Weaveworks4.1K views
Empowering developers and operators through Gitlab and HashiCorp by Mitchell Pronschinske
Empowering developers and operators through Gitlab and HashiCorpEmpowering developers and operators through Gitlab and HashiCorp
Empowering developers and operators through Gitlab and HashiCorp
Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub... by Lviv Startup Club
Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...
Yannis Zarkadas. Stefano Fioravanzo. Enterprise data science workflows on kub...
Lviv Startup Club136 views
Integrating Google Cloud Dataproc with Alluxio for faster performance in the ... by Alluxio, Inc.
Integrating Google Cloud Dataproc with Alluxio for faster performance in the ...Integrating Google Cloud Dataproc with Alluxio for faster performance in the ...
Integrating Google Cloud Dataproc with Alluxio for faster performance in the ...
Alluxio, Inc.468 views
Google's Infrastructure and Specific IoT Services by Intel® Software
Google's Infrastructure and Specific IoT ServicesGoogle's Infrastructure and Specific IoT Services
Google's Infrastructure and Specific IoT Services
Intel® Software1.6K views

More from Márton Kodok

Gen Apps on Google Cloud PaLM2 and Codey APIs in Action by
Gen Apps on Google Cloud PaLM2 and Codey APIs in ActionGen Apps on Google Cloud PaLM2 and Codey APIs in Action
Gen Apps on Google Cloud PaLM2 and Codey APIs in ActionMárton Kodok
15 views55 slides
DevBCN Vertex AI - Pipelines for your MLOps workflows by
DevBCN Vertex AI - Pipelines for your MLOps workflowsDevBCN Vertex AI - Pipelines for your MLOps workflows
DevBCN Vertex AI - Pipelines for your MLOps workflowsMárton Kodok
67 views57 slides
Discover BigQuery ML, build your own CREATE MODEL statement by
Discover BigQuery ML, build your own CREATE MODEL statementDiscover BigQuery ML, build your own CREATE MODEL statement
Discover BigQuery ML, build your own CREATE MODEL statementMárton Kodok
73 views61 slides
BigQuery best practices and recommendations to reduce costs with BI Engine, S... by
BigQuery best practices and recommendations to reduce costs with BI Engine, S...BigQuery best practices and recommendations to reduce costs with BI Engine, S...
BigQuery best practices and recommendations to reduce costs with BI Engine, S...Márton Kodok
349 views38 slides
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud by
Vertex AI - Unified ML Platform for the entire AI workflow on Google CloudVertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Vertex AI - Unified ML Platform for the entire AI workflow on Google CloudMárton Kodok
1.2K views50 slides
Vertex AI: Pipelines for your MLOps workflows by
Vertex AI: Pipelines for your MLOps workflowsVertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflowsMárton Kodok
789 views31 slides

More from Márton Kodok(20)

Gen Apps on Google Cloud PaLM2 and Codey APIs in Action by Márton Kodok
Gen Apps on Google Cloud PaLM2 and Codey APIs in ActionGen Apps on Google Cloud PaLM2 and Codey APIs in Action
Gen Apps on Google Cloud PaLM2 and Codey APIs in Action
Márton Kodok15 views
DevBCN Vertex AI - Pipelines for your MLOps workflows by Márton Kodok
DevBCN Vertex AI - Pipelines for your MLOps workflowsDevBCN Vertex AI - Pipelines for your MLOps workflows
DevBCN Vertex AI - Pipelines for your MLOps workflows
Márton Kodok67 views
Discover BigQuery ML, build your own CREATE MODEL statement by Márton Kodok
Discover BigQuery ML, build your own CREATE MODEL statementDiscover BigQuery ML, build your own CREATE MODEL statement
Discover BigQuery ML, build your own CREATE MODEL statement
Márton Kodok73 views
BigQuery best practices and recommendations to reduce costs with BI Engine, S... by Márton Kodok
BigQuery best practices and recommendations to reduce costs with BI Engine, S...BigQuery best practices and recommendations to reduce costs with BI Engine, S...
BigQuery best practices and recommendations to reduce costs with BI Engine, S...
Márton Kodok349 views
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud by Márton Kodok
Vertex AI - Unified ML Platform for the entire AI workflow on Google CloudVertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Vertex AI - Unified ML Platform for the entire AI workflow on Google Cloud
Márton Kodok1.2K views
Vertex AI: Pipelines for your MLOps workflows by Márton Kodok
Vertex AI: Pipelines for your MLOps workflowsVertex AI: Pipelines for your MLOps workflows
Vertex AI: Pipelines for your MLOps workflows
Márton Kodok789 views
BigdataConference Europe - BigQuery ML by Márton Kodok
BigdataConference Europe - BigQuery MLBigdataConference Europe - BigQuery ML
BigdataConference Europe - BigQuery ML
Márton Kodok147 views
DevFest Romania 2020 Keynote: Bringing the Cloud to you. by Márton Kodok
DevFest Romania 2020 Keynote: Bringing the Cloud to you.DevFest Romania 2020 Keynote: Bringing the Cloud to you.
DevFest Romania 2020 Keynote: Bringing the Cloud to you.
Márton Kodok66 views
Applying BigQuery ML on e-commerce data analytics by Márton Kodok
Applying BigQuery ML on e-commerce data analyticsApplying BigQuery ML on e-commerce data analytics
Applying BigQuery ML on e-commerce data analytics
Márton Kodok1.5K views
Supercharge your data analytics with BigQuery by Márton Kodok
Supercharge your data analytics with BigQuerySupercharge your data analytics with BigQuery
Supercharge your data analytics with BigQuery
Márton Kodok189 views
Vibe Koli 2019 - Utazás az egyetem padjaitól a Google Developer Expertig by Márton Kodok
Vibe Koli 2019 - Utazás az egyetem padjaitól a Google Developer ExpertigVibe Koli 2019 - Utazás az egyetem padjaitól a Google Developer Expertig
Vibe Koli 2019 - Utazás az egyetem padjaitól a Google Developer Expertig
Márton Kodok150 views
BigQuery ML - Machine learning at scale using SQL by Márton Kodok
BigQuery ML - Machine learning at scale using SQLBigQuery ML - Machine learning at scale using SQL
BigQuery ML - Machine learning at scale using SQL
Márton Kodok1.1K views
Next18 Extended Targu Mures - Bringing the Cloud to you by Márton Kodok
Next18 Extended Targu Mures - Bringing the Cloud to youNext18 Extended Targu Mures - Bringing the Cloud to you
Next18 Extended Targu Mures - Bringing the Cloud to you
Márton Kodok81 views
GCP - A felhőalapú architektúrák és szolgáltatások by Márton Kodok
GCP - A felhőalapú architektúrák és szolgáltatásokGCP - A felhőalapú architektúrák és szolgáltatások
GCP - A felhőalapú architektúrák és szolgáltatások
Márton Kodok157 views
Efikot - Smart City, okos város - a jövőnk kulcsa by Márton Kodok
Efikot - Smart City, okos város - a jövőnk kulcsaEfikot - Smart City, okos város - a jövőnk kulcsa
Efikot - Smart City, okos város - a jövőnk kulcsa
Márton Kodok170 views
CodeCamp Iasi - Creating serverless data analytics system on GCP using BigQuery by Márton Kodok
CodeCamp Iasi - Creating serverless data analytics system on GCP using BigQueryCodeCamp Iasi - Creating serverless data analytics system on GCP using BigQuery
CodeCamp Iasi - Creating serverless data analytics system on GCP using BigQuery
Márton Kodok358 views
Voxxed Days Cluj - Powering interactive data analysis with Google BigQuery by Márton Kodok
Voxxed Days Cluj - Powering interactive data analysis with Google BigQueryVoxxed Days Cluj - Powering interactive data analysis with Google BigQuery
Voxxed Days Cluj - Powering interactive data analysis with Google BigQuery
Márton Kodok417 views
GDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQuery by Márton Kodok
GDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQueryGDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQuery
GDG DevFest Ukraine - Powering Interactive Data Analysis with Google BigQuery
Márton Kodok300 views
Making advanced analytics accessible to more companies by Márton Kodok
Making advanced analytics accessible to more companiesMaking advanced analytics accessible to more companies
Making advanced analytics accessible to more companies
Márton Kodok155 views
DevTalks Keynote Powering interactive data analysis with Google BigQuery by Márton Kodok
DevTalks Keynote Powering interactive data analysis with Google BigQueryDevTalks Keynote Powering interactive data analysis with Google BigQuery
DevTalks Keynote Powering interactive data analysis with Google BigQuery
Márton Kodok258 views

Recently uploaded

Sprint 226 by
Sprint 226Sprint 226
Sprint 226ManageIQ
10 views18 slides
Understanding HTML terminology by
Understanding HTML terminologyUnderstanding HTML terminology
Understanding HTML terminologyartembondar5
6 views8 slides
Copilot Prompting Toolkit_All Resources.pdf by
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdfRiccardo Zamana
16 views4 slides
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI... by
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...Marc Müller
42 views83 slides
SAP FOR CONTRACT MANUFACTURING.pdf by
SAP FOR CONTRACT MANUFACTURING.pdfSAP FOR CONTRACT MANUFACTURING.pdf
SAP FOR CONTRACT MANUFACTURING.pdfVirendra Rai, PMP
13 views2 slides
Programming Field by
Programming FieldProgramming Field
Programming Fieldthehardtechnology
6 views9 slides

Recently uploaded(20)

Sprint 226 by ManageIQ
Sprint 226Sprint 226
Sprint 226
ManageIQ10 views
Understanding HTML terminology by artembondar5
Understanding HTML terminologyUnderstanding HTML terminology
Understanding HTML terminology
artembondar56 views
Copilot Prompting Toolkit_All Resources.pdf by Riccardo Zamana
Copilot Prompting Toolkit_All Resources.pdfCopilot Prompting Toolkit_All Resources.pdf
Copilot Prompting Toolkit_All Resources.pdf
Riccardo Zamana16 views
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI... by Marc Müller
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Dev-Cloud Conference 2023 - Continuous Deployment Showdown: Traditionelles CI...
Marc Müller42 views
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports by Ra'Fat Al-Msie'deen
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug ReportsBushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports
360 graden fabriek by info33492
360 graden fabriek360 graden fabriek
360 graden fabriek
info33492143 views
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated... by TomHalpin9
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
Dev-HRE-Ops - Addressing the _Last Mile DevOps Challenge_ in Highly Regulated...
TomHalpin96 views
JioEngage_Presentation.pptx by admin125455
JioEngage_Presentation.pptxJioEngage_Presentation.pptx
JioEngage_Presentation.pptx
admin1254556 views
Software evolution understanding: Automatic extraction of software identifier... by Ra'Fat Al-Msie'deen
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx by animuscrm
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
animuscrm15 views
Generic or specific? Making sensible software design decisions by Bert Jan Schrijver
Generic or specific? Making sensible software design decisionsGeneric or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisions
Airline Booking Software by SharmiMehta
Airline Booking SoftwareAirline Booking Software
Airline Booking Software
SharmiMehta7 views
Navigating container technology for enhanced security by Niklas Saari by Metosin Oy
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas Saari
Metosin Oy14 views
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra... by Marc Müller
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra....NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
Marc Müller41 views

Cloud Workflows What's new in serverless orchestration and automation

  • 1. Cloud Workflows: What's new in serverless orchestration and automation October 2021 - GDG DevFest 2021 - Doha Márton Kodok / @martonkodok Google Developer Expert at REEA.net
  • 2. ● Among the Top 3 romanians on Stackoverflow 195K reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery + Redis database engine expert Slideshare: martonkodok Articles: martonkodok.medium.com Twitter: @martonkodok StackOverflow: pentium10 GitHub: pentium10 What's new in serverless orchestration and automation @martonkodok About me
  • 3. 1. Challenges in connecting services 2. What is Cloud Workflows? 3. Practical use cases 4. What’s new in Workflows? 5. Advanced examples 6. Conclusions Agenda What's new in serverless orchestration and automation @martonkodok
  • 4. ● Connectivity - should be easy, but in reality you need to figure out ● Parse the results ● Decisions and conditional step executions ● No built-in error handling, logging ● Struggling with polling, retries, exponential backoff ● Scaling up and down to zero ● Authentication ● Missed capability of capturing callbacks Challenges in connecting services What's new in serverless orchestration and automation @martonkodok
  • 5. MeetCloudWorkflows What's new in serverless orchestration and automation @martonkodok
  • 6. Workflows in Google Cloud portfolio Introduction Orchestrate any 1. Google Cloud API 2. SaaS API 3. External or private APIs. Serverless Compute External API’s Google API’s etc... Workflows - orchestrate & integrate SaaS API’s Private API’s Other Clouds
  • 7. Step-Automation-as-a-Service - Serverless HTTP service automation Declarative workflow language (YAML, JSON) Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020 Built-in decision and conditional executions expression formulas, operation on var Subworkflows similar to routine in a programming language with input/return var Support for external API calls out of the box support outside of Google Cloud Integrates with any Google Cloud product without worrying about authentication Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks What is GCP Workflows? What's new in serverless orchestration and automation @martonkodok
  • 8. OAuth, OIDC, Secret Manager integration Enterprise Security Keep your workflows secure X Authenticated Invocations Authenticated calls to Google Cloud services Integration with Secret Manager Encryption at rest and in transit External API 1 External API 2 External API
  • 9. Anatomy of a Cloud Workflow! What's new in serverless orchestration and automation @martonkodok
  • 10. hello.yaml Developing for Cloud Workflow! What's new in serverless orchestration and automation @martonkodok workflowRevisionId: 000001-de1 argument: 'null' startTime: '2020-10-16T20:19:34.448323739Z' name: projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c- 4065-a696-940bc7658c33 result: '"Hello, Cloud Workflows!"' endTime: '2020-10-16T20:19:34.845536315Z' state: SUCCEEDED Deploying gcloud workflows deploy hello --source=hello.yaml Executing gcloud workflows execute hello --data={“var”:”value”} Describing gcloud workflows executions describe-last
  • 11. Sample Workflows What's new in serverless orchestration and automation @martonkodok
  • 12. IT management automation Combine automation with scheduler Wait for service checks Orchestrate work across Compute Engine, PubSub, Stackdriver and other Google Cloud Products 9 AM trigger Start a Compute Engine VM Log the event App Started? No Wait 60 seconds Notify the team
  • 13. E-commerce invoice generation with Workflows Steps orchestration Reliable execution, with error handling and retries Orchestration microservices or other API’s Create an invoice Generate PDF Send PDF via email Receive Order Cloud Run App Engine Cloud Run
  • 14. Code Examples What's new in serverless orchestration and automation @martonkodok
  • 15. 1. HTTP calls (GET or POST) 2. Sequence two steps 3. Working with subworkflows Workflow Basics What's new in serverless orchestration and automation @martonkodok
  • 16. http_post.yaml Making an external HTTP POST request What's new in serverless orchestration and automation @martonkodok { "archived":false, "created_at":"2020-10-16T17:40:17+0000", "id":"bit.ly/35452TM", "link":"https://bit.ly/35452TM", "long_url":"<truncated>", }
  • 17. wikipedia.yaml Sequence two steps to get data from Wikipedia What's new in serverless orchestration and automation @martonkodok
  • 18. subworkflow.yaml Subworkflow What's new in serverless orchestration and automation @martonkodok call call def
  • 19. What’s new in Workflows What's new in serverless orchestration and automation @martonkodok
  • 20. 1. Functions: string, JSON, map, iterating syntax 2. Cloud logging and Call logging 3. Connectors: built-in authentication, retries, long-operation 4. Callbacks: enables async-executions, or human in the loop What’s new in Workflows What's new in serverless orchestration and automation @martonkodok
  • 21. String Functions What's new in serverless orchestration and automation @martonkodok
  • 22. Workflow Connectors What's new in serverless orchestration and automation @martonkodok
  • 23. They are like client libraries for other services handling auth, retries and waiting. Connect to other Google Cloud APIs, and to integrate other GCP products. Connectors What's new in serverless orchestration and automation @martonkodok
  • 24. Before: Calling Secret Manager traditionally using HTTP Get calls. Connectors: Replacing GET and POST calls What's new in serverless orchestration and automation @martonkodok After: using Connectors to get Secret Manager value
  • 25. 1. Built in Authentication (calling service account needs permissions) 2. Retries (max_retries: 5, max_delay: 60, ability to define custom retry policies) 3. Timeout (default timeout: 30 minutes, can be increased to 1 year) 4. Long-running operations (blocks the workflow execution until the operation completes) 5. Polling (exponential backoff of 1.25 up to 60 seconds, each step is billable) Connectors details What's new in serverless orchestration and automation @martonkodok
  • 26. Long running operation type of Workflow Connectors What's new in serverless orchestration and automation @martonkodok Product Operation Type Product Operation Type BigQuery Job Dataflow Job Cloud Build Operation Firestore Operation Cloud Functions Operation Document AI Operation Cloud SQL Operation Cloud Spanner Operation Compute Engine Operation Workflows Operation Execution
  • 27. Workflow Callbacks What's new in serverless orchestration and automation @martonkodok
  • 28. They are webhook endpoints where you signal that external event has occurred. Callbacks allow workflows executions to wait until the callback endpoint gets a call. Callbacks What's new in serverless orchestration and automation @martonkodok
  • 29. 1. Modern way to avoid sys.sleep, wait, tracking tables, polling 2. Offers a webhook endpoint (secured with authentication) 3. Timeout (default timeout: 12 hours, can be increased to 1 year) 4. Event based (pauses the workflow execution until the call is received on endpoint) 5. Full callback details (callback_request holds headers,query,body,raw,json) Callback details What's new in serverless orchestration and automation @martonkodok
  • 30. Orchestrate and automate the Cloud What's new in serverless orchestration and automation @martonkodok
  • 31. Firestore Backups the easy way with Cloud Workflows - Cloud Scheduler - Cloud Workflows - Cloud Firestore - Cloud Storage Full guide article on: martonkodok.medium.com Cloud Workflows Cloud Storage Cloud Scheduler Firebase Cloud Firestore backup Roles and Permissions Cloud IAM authorize
  • 32. Using Workflows to load Cloud Storage files into BigQuery - Cloud Workflows - Cloud Storage - BigQuery Full guide article on: martonkodok.medium.com Cloud Storage BigQuery Cloud Workflows Authenticated Invocations foreach
  • 33. Run shell commands and orchestrate Compute Engine VMs - Cloud Workflows - Cloud Build - Compute Engine - Identity-Aware Proxy Full guide article on: martonkodok.medium.com Cloud Workflow Steps Cloud Workflows Execution Authorize Cloud Build Roles and Permissions Cloud IAM Shell Service Account Start VM Compute Engine Submit Stop VM Compute Engine Firewall Compute Engine Cloud IAP Tunnel wait wait
  • 34. Conclusions What's new in serverless orchestration and automation @martonkodok
  • 35. Reliable workflow execution - execute workflows for enterprise business apps Low latency of execution - no cold starts Built-in error handling out of the box error handling with configurable retry policies Passing variable built-in JSON parsing and expression-based variable manipulation Rich runtime iterating through an array, embedded steps for readability Secret Manager integration out of the box Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks Benefits of Cloud Workflows What's new in serverless orchestration and automation @martonkodok
  • 36. Easy to build/operate Scales out Does not lose state Handles errors/timeouts Out-of-the-box support of Cloud APIs Auditable Enables async-workflows with Callbacks Developer friendly What's new in serverless orchestration and automation @martonkodok
  • 37. The possibilities are endless Marketing Retail IndustrialandIoT Developer Event driven marketing workflow execution Relay conversions to customer profiles in external services Workflow based emails, discounts, promotions Order management Inventory chain operations Data gathering and processing Synchronize systems Generate state machines Verify equipment lifecycle Workflow based maintenance needs Digitalization of internal policies Automate the Cloud Shell-script replacement Orchestrate devops workflows @martonkodok
  • 38. Thank you. Q&A. Slides available on: slideshare.net/martonkodok Reea.net - Integrated web solutions driven by creativity to deliver projects.