Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Cloud Workflows: What's new in serverless
orchestration and automation
October 2021 - GDG DevFest 2021 - Doha
Márton Kodok ...
● Among the Top 3 romanians on Stackoverflow 195K reputation
● Google Developer Expert on Cloud technologies
● Crafting We...
1. Challenges in connecting services
2. What is Cloud Workflows?
3. Practical use cases
4. What’s new in Workflows?
5. Adv...
● Connectivity - should be easy, but in reality you need to figure out
● Parse the results
● Decisions and conditional ste...
MeetCloudWorkflows
What's new in serverless orchestration and automation @martonkodok
Workflows in Google Cloud portfolio
Introduction
Orchestrate any
1. Google Cloud API
2. SaaS API
3. External or private API...
Step-Automation-as-a-Service - Serverless HTTP service automation
Declarative workflow language (YAML, JSON)
Decent pricin...
OAuth, OIDC, Secret Manager integration
Enterprise
Security
Keep your workflows secure
X
Authenticated
Invocations
Authent...
Anatomy of a Cloud Workflow!
What's new in serverless orchestration and automation @martonkodok
hello.yaml
Developing for Cloud Workflow!
What's new in serverless orchestration and automation @martonkodok
workflowRevisi...
Sample Workflows
What's new in serverless orchestration and automation @martonkodok
IT management automation
Combine automation with
scheduler
Wait for service checks
Orchestrate work across
Compute Engine,...
E-commerce invoice generation with Workflows
Steps orchestration
Reliable execution, with error
handling and retries
Orches...
Code Examples
What's new in serverless orchestration and automation @martonkodok
1. HTTP calls (GET or POST)
2. Sequence two steps
3. Working with subworkflows
Workflow Basics
What's new in serverless orc...
http_post.yaml
Making an external HTTP POST request
What's new in serverless orchestration and automation @martonkodok
{
"...
wikipedia.yaml
Sequence two steps to get data from Wikipedia
What's new in serverless orchestration and automation @marton...
subworkflow.yaml
Subworkflow
What's new in serverless orchestration and automation @martonkodok
call
call
def
What’s new in Workflows
What's new in serverless orchestration and automation @martonkodok
1. Functions: string, JSON, map, iterating syntax
2. Cloud logging and Call logging
3. Connectors: built-in authentication...
String Functions
What's new in serverless orchestration and automation @martonkodok
Workflow Connectors
What's new in serverless orchestration and automation @martonkodok
They are like client libraries for other services handling auth, retries and waiting.
Connect to other Google Cloud APIs, ...
Before: Calling Secret Manager traditionally using HTTP Get calls.
Connectors: Replacing GET and POST calls
What's new in ...
1. Built in Authentication (calling service account needs permissions)
2. Retries (max_retries: 5, max_delay: 60, ability ...
Long running operation type of Workflow Connectors
What's new in serverless orchestration and automation @martonkodok
Produ...
Workflow Callbacks
What's new in serverless orchestration and automation @martonkodok
They are webhook endpoints where you signal that external event has occurred.
Callbacks allow workflows executions to wait...
1. Modern way to avoid sys.sleep, wait, tracking tables, polling
2. Offers a webhook endpoint (secured with authentication...
Orchestrate and automate the Cloud
What's new in serverless orchestration and automation @martonkodok
Firestore Backups the easy way with Cloud Workflows
- Cloud Scheduler
- Cloud Workflows
- Cloud Firestore
- Cloud Storage
F...
Using Workflows to load Cloud Storage files into BigQuery
- Cloud Workflows
- Cloud Storage
- BigQuery
Full guide article on...
Run shell commands and orchestrate Compute Engine VMs
- Cloud Workflows
- Cloud Build
- Compute Engine
- Identity-Aware Pr...
Conclusions
What's new in serverless orchestration and automation @martonkodok
Reliable workflow execution - execute workflows for enterprise business apps
Low latency of execution - no cold starts
Bui...
Easy to build/operate
Scales out
Does not lose state
Handles errors/timeouts
Out-of-the-box support of Cloud APIs
Auditabl...
The possibilities are endless
Marketing Retail IndustrialandIoT Developer
Event driven marketing
workflow execution
Relay c...
Thank you. Q&A.
Slides available on:
slideshare.net/martonkodok
Reea.net - Integrated web solutions driven by creativity
t...
Upcoming SlideShare
Loading in …5
×

of

Cloud Workflows What's new in serverless orchestration and automation Slide 1 Cloud Workflows What's new in serverless orchestration and automation Slide 2 Cloud Workflows What's new in serverless orchestration and automation Slide 3 Cloud Workflows What's new in serverless orchestration and automation Slide 4 Cloud Workflows What's new in serverless orchestration and automation Slide 5 Cloud Workflows What's new in serverless orchestration and automation Slide 6 Cloud Workflows What's new in serverless orchestration and automation Slide 7 Cloud Workflows What's new in serverless orchestration and automation Slide 8 Cloud Workflows What's new in serverless orchestration and automation Slide 9 Cloud Workflows What's new in serverless orchestration and automation Slide 10 Cloud Workflows What's new in serverless orchestration and automation Slide 11 Cloud Workflows What's new in serverless orchestration and automation Slide 12 Cloud Workflows What's new in serverless orchestration and automation Slide 13 Cloud Workflows What's new in serverless orchestration and automation Slide 14 Cloud Workflows What's new in serverless orchestration and automation Slide 15 Cloud Workflows What's new in serverless orchestration and automation Slide 16 Cloud Workflows What's new in serverless orchestration and automation Slide 17 Cloud Workflows What's new in serverless orchestration and automation Slide 18 Cloud Workflows What's new in serverless orchestration and automation Slide 19 Cloud Workflows What's new in serverless orchestration and automation Slide 20 Cloud Workflows What's new in serverless orchestration and automation Slide 21 Cloud Workflows What's new in serverless orchestration and automation Slide 22 Cloud Workflows What's new in serverless orchestration and automation Slide 23 Cloud Workflows What's new in serverless orchestration and automation Slide 24 Cloud Workflows What's new in serverless orchestration and automation Slide 25 Cloud Workflows What's new in serverless orchestration and automation Slide 26 Cloud Workflows What's new in serverless orchestration and automation Slide 27 Cloud Workflows What's new in serverless orchestration and automation Slide 28 Cloud Workflows What's new in serverless orchestration and automation Slide 29 Cloud Workflows What's new in serverless orchestration and automation Slide 30 Cloud Workflows What's new in serverless orchestration and automation Slide 31 Cloud Workflows What's new in serverless orchestration and automation Slide 32 Cloud Workflows What's new in serverless orchestration and automation Slide 33 Cloud Workflows What's new in serverless orchestration and automation Slide 34 Cloud Workflows What's new in serverless orchestration and automation Slide 35 Cloud Workflows What's new in serverless orchestration and automation Slide 36 Cloud Workflows What's new in serverless orchestration and automation Slide 37 Cloud Workflows What's new in serverless orchestration and automation Slide 38
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

0 Likes

Share

Download to read offline

Cloud Workflows What's new in serverless orchestration and automation

Download to read offline

understand how Cloud Workflows resolves challenges in connecting services, HTTP based service orchestration and automation. We are going to dive deep how serverless HTTP service automation works to automate step engines. Based on practical examples we will demonstrate the newest features that lets you automate the cloud and integration with any Google Cloud product without worrying about authentication

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Cloud Workflows What's new in serverless orchestration and automation

  1. 1. Cloud Workflows: What's new in serverless orchestration and automation October 2021 - GDG DevFest 2021 - Doha Márton Kodok / @martonkodok Google Developer Expert at REEA.net
  2. 2. ● Among the Top 3 romanians on Stackoverflow 195K reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery + Redis database engine expert Slideshare: martonkodok Articles: martonkodok.medium.com Twitter: @martonkodok StackOverflow: pentium10 GitHub: pentium10 What's new in serverless orchestration and automation @martonkodok About me
  3. 3. 1. Challenges in connecting services 2. What is Cloud Workflows? 3. Practical use cases 4. What’s new in Workflows? 5. Advanced examples 6. Conclusions Agenda What's new in serverless orchestration and automation @martonkodok
  4. 4. ● Connectivity - should be easy, but in reality you need to figure out ● Parse the results ● Decisions and conditional step executions ● No built-in error handling, logging ● Struggling with polling, retries, exponential backoff ● Scaling up and down to zero ● Authentication ● Missed capability of capturing callbacks Challenges in connecting services What's new in serverless orchestration and automation @martonkodok
  5. 5. MeetCloudWorkflows What's new in serverless orchestration and automation @martonkodok
  6. 6. Workflows in Google Cloud portfolio Introduction Orchestrate any 1. Google Cloud API 2. SaaS API 3. External or private APIs. Serverless Compute External API’s Google API’s etc... Workflows - orchestrate & integrate SaaS API’s Private API’s Other Clouds
  7. 7. Step-Automation-as-a-Service - Serverless HTTP service automation Declarative workflow language (YAML, JSON) Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020 Built-in decision and conditional executions expression formulas, operation on var Subworkflows similar to routine in a programming language with input/return var Support for external API calls out of the box support outside of Google Cloud Integrates with any Google Cloud product without worrying about authentication Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks What is GCP Workflows? What's new in serverless orchestration and automation @martonkodok
  8. 8. OAuth, OIDC, Secret Manager integration Enterprise Security Keep your workflows secure X Authenticated Invocations Authenticated calls to Google Cloud services Integration with Secret Manager Encryption at rest and in transit External API 1 External API 2 External API
  9. 9. Anatomy of a Cloud Workflow! What's new in serverless orchestration and automation @martonkodok
  10. 10. hello.yaml Developing for Cloud Workflow! What's new in serverless orchestration and automation @martonkodok workflowRevisionId: 000001-de1 argument: 'null' startTime: '2020-10-16T20:19:34.448323739Z' name: projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c- 4065-a696-940bc7658c33 result: '"Hello, Cloud Workflows!"' endTime: '2020-10-16T20:19:34.845536315Z' state: SUCCEEDED Deploying gcloud workflows deploy hello --source=hello.yaml Executing gcloud workflows execute hello --data={“var”:”value”} Describing gcloud workflows executions describe-last
  11. 11. Sample Workflows What's new in serverless orchestration and automation @martonkodok
  12. 12. IT management automation Combine automation with scheduler Wait for service checks Orchestrate work across Compute Engine, PubSub, Stackdriver and other Google Cloud Products 9 AM trigger Start a Compute Engine VM Log the event App Started? No Wait 60 seconds Notify the team
  13. 13. E-commerce invoice generation with Workflows Steps orchestration Reliable execution, with error handling and retries Orchestration microservices or other API’s Create an invoice Generate PDF Send PDF via email Receive Order Cloud Run App Engine Cloud Run
  14. 14. Code Examples What's new in serverless orchestration and automation @martonkodok
  15. 15. 1. HTTP calls (GET or POST) 2. Sequence two steps 3. Working with subworkflows Workflow Basics What's new in serverless orchestration and automation @martonkodok
  16. 16. http_post.yaml Making an external HTTP POST request What's new in serverless orchestration and automation @martonkodok { "archived":false, "created_at":"2020-10-16T17:40:17+0000", "id":"bit.ly/35452TM", "link":"https://bit.ly/35452TM", "long_url":"<truncated>", }
  17. 17. wikipedia.yaml Sequence two steps to get data from Wikipedia What's new in serverless orchestration and automation @martonkodok
  18. 18. subworkflow.yaml Subworkflow What's new in serverless orchestration and automation @martonkodok call call def
  19. 19. What’s new in Workflows What's new in serverless orchestration and automation @martonkodok
  20. 20. 1. Functions: string, JSON, map, iterating syntax 2. Cloud logging and Call logging 3. Connectors: built-in authentication, retries, long-operation 4. Callbacks: enables async-executions, or human in the loop What’s new in Workflows What's new in serverless orchestration and automation @martonkodok
  21. 21. String Functions What's new in serverless orchestration and automation @martonkodok
  22. 22. Workflow Connectors What's new in serverless orchestration and automation @martonkodok
  23. 23. They are like client libraries for other services handling auth, retries and waiting. Connect to other Google Cloud APIs, and to integrate other GCP products. Connectors What's new in serverless orchestration and automation @martonkodok
  24. 24. Before: Calling Secret Manager traditionally using HTTP Get calls. Connectors: Replacing GET and POST calls What's new in serverless orchestration and automation @martonkodok After: using Connectors to get Secret Manager value
  25. 25. 1. Built in Authentication (calling service account needs permissions) 2. Retries (max_retries: 5, max_delay: 60, ability to define custom retry policies) 3. Timeout (default timeout: 30 minutes, can be increased to 1 year) 4. Long-running operations (blocks the workflow execution until the operation completes) 5. Polling (exponential backoff of 1.25 up to 60 seconds, each step is billable) Connectors details What's new in serverless orchestration and automation @martonkodok
  26. 26. Long running operation type of Workflow Connectors What's new in serverless orchestration and automation @martonkodok Product Operation Type Product Operation Type BigQuery Job Dataflow Job Cloud Build Operation Firestore Operation Cloud Functions Operation Document AI Operation Cloud SQL Operation Cloud Spanner Operation Compute Engine Operation Workflows Operation Execution
  27. 27. Workflow Callbacks What's new in serverless orchestration and automation @martonkodok
  28. 28. They are webhook endpoints where you signal that external event has occurred. Callbacks allow workflows executions to wait until the callback endpoint gets a call. Callbacks What's new in serverless orchestration and automation @martonkodok
  29. 29. 1. Modern way to avoid sys.sleep, wait, tracking tables, polling 2. Offers a webhook endpoint (secured with authentication) 3. Timeout (default timeout: 12 hours, can be increased to 1 year) 4. Event based (pauses the workflow execution until the call is received on endpoint) 5. Full callback details (callback_request holds headers,query,body,raw,json) Callback details What's new in serverless orchestration and automation @martonkodok
  30. 30. Orchestrate and automate the Cloud What's new in serverless orchestration and automation @martonkodok
  31. 31. Firestore Backups the easy way with Cloud Workflows - Cloud Scheduler - Cloud Workflows - Cloud Firestore - Cloud Storage Full guide article on: martonkodok.medium.com Cloud Workflows Cloud Storage Cloud Scheduler Firebase Cloud Firestore backup Roles and Permissions Cloud IAM authorize
  32. 32. Using Workflows to load Cloud Storage files into BigQuery - Cloud Workflows - Cloud Storage - BigQuery Full guide article on: martonkodok.medium.com Cloud Storage BigQuery Cloud Workflows Authenticated Invocations foreach
  33. 33. Run shell commands and orchestrate Compute Engine VMs - Cloud Workflows - Cloud Build - Compute Engine - Identity-Aware Proxy Full guide article on: martonkodok.medium.com Cloud Workflow Steps Cloud Workflows Execution Authorize Cloud Build Roles and Permissions Cloud IAM Shell Service Account Start VM Compute Engine Submit Stop VM Compute Engine Firewall Compute Engine Cloud IAP Tunnel wait wait
  34. 34. Conclusions What's new in serverless orchestration and automation @martonkodok
  35. 35. Reliable workflow execution - execute workflows for enterprise business apps Low latency of execution - no cold starts Built-in error handling out of the box error handling with configurable retry policies Passing variable built-in JSON parsing and expression-based variable manipulation Rich runtime iterating through an array, embedded steps for readability Secret Manager integration out of the box Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks Benefits of Cloud Workflows What's new in serverless orchestration and automation @martonkodok
  36. 36. Easy to build/operate Scales out Does not lose state Handles errors/timeouts Out-of-the-box support of Cloud APIs Auditable Enables async-workflows with Callbacks Developer friendly What's new in serverless orchestration and automation @martonkodok
  37. 37. The possibilities are endless Marketing Retail IndustrialandIoT Developer Event driven marketing workflow execution Relay conversions to customer profiles in external services Workflow based emails, discounts, promotions Order management Inventory chain operations Data gathering and processing Synchronize systems Generate state machines Verify equipment lifecycle Workflow based maintenance needs Digitalization of internal policies Automate the Cloud Shell-script replacement Orchestrate devops workflows @martonkodok
  38. 38. Thank you. Q&A. Slides available on: slideshare.net/martonkodok Reea.net - Integrated web solutions driven by creativity to deliver projects.

understand how Cloud Workflows resolves challenges in connecting services, HTTP based service orchestration and automation. We are going to dive deep how serverless HTTP service automation works to automate step engines. Based on practical examples we will demonstrate the newest features that lets you automate the cloud and integration with any Google Cloud product without worrying about authentication

Views

Total views

75

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

1

Shares

0

Comments

0

Likes

0

×