Successfully reported this slideshow.
Your SlideShare is downloading. ×

Cloud Workflows What's new in serverless orchestration and automation

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 38 Ad

Cloud Workflows What's new in serverless orchestration and automation

Download to read offline

understand how Cloud Workflows resolves challenges in connecting services, HTTP based service orchestration and automation. We are going to dive deep how serverless HTTP service automation works to automate step engines. Based on practical examples we will demonstrate the newest features that lets you automate the cloud and integration with any Google Cloud product without worrying about authentication

understand how Cloud Workflows resolves challenges in connecting services, HTTP based service orchestration and automation. We are going to dive deep how serverless HTTP service automation works to automate step engines. Based on practical examples we will demonstrate the newest features that lets you automate the cloud and integration with any Google Cloud product without worrying about authentication

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to Cloud Workflows What's new in serverless orchestration and automation (20)

Advertisement

More from Márton Kodok (20)

Recently uploaded (20)

Advertisement

Cloud Workflows What's new in serverless orchestration and automation

  1. 1. Cloud Workflows: What's new in serverless orchestration and automation October 2021 - GDG DevFest 2021 - Doha Márton Kodok / @martonkodok Google Developer Expert at REEA.net
  2. 2. ● Among the Top 3 romanians on Stackoverflow 195K reputation ● Google Developer Expert on Cloud technologies ● Crafting Web/Mobile backends at REEA.net ● BigQuery + Redis database engine expert Slideshare: martonkodok Articles: martonkodok.medium.com Twitter: @martonkodok StackOverflow: pentium10 GitHub: pentium10 What's new in serverless orchestration and automation @martonkodok About me
  3. 3. 1. Challenges in connecting services 2. What is Cloud Workflows? 3. Practical use cases 4. What’s new in Workflows? 5. Advanced examples 6. Conclusions Agenda What's new in serverless orchestration and automation @martonkodok
  4. 4. ● Connectivity - should be easy, but in reality you need to figure out ● Parse the results ● Decisions and conditional step executions ● No built-in error handling, logging ● Struggling with polling, retries, exponential backoff ● Scaling up and down to zero ● Authentication ● Missed capability of capturing callbacks Challenges in connecting services What's new in serverless orchestration and automation @martonkodok
  5. 5. MeetCloudWorkflows What's new in serverless orchestration and automation @martonkodok
  6. 6. Workflows in Google Cloud portfolio Introduction Orchestrate any 1. Google Cloud API 2. SaaS API 3. External or private APIs. Serverless Compute External API’s Google API’s etc... Workflows - orchestrate & integrate SaaS API’s Private API’s Other Clouds
  7. 7. Step-Automation-as-a-Service - Serverless HTTP service automation Declarative workflow language (YAML, JSON) Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020 Built-in decision and conditional executions expression formulas, operation on var Subworkflows similar to routine in a programming language with input/return var Support for external API calls out of the box support outside of Google Cloud Integrates with any Google Cloud product without worrying about authentication Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks What is GCP Workflows? What's new in serverless orchestration and automation @martonkodok
  8. 8. OAuth, OIDC, Secret Manager integration Enterprise Security Keep your workflows secure X Authenticated Invocations Authenticated calls to Google Cloud services Integration with Secret Manager Encryption at rest and in transit External API 1 External API 2 External API
  9. 9. Anatomy of a Cloud Workflow! What's new in serverless orchestration and automation @martonkodok
  10. 10. hello.yaml Developing for Cloud Workflow! What's new in serverless orchestration and automation @martonkodok workflowRevisionId: 000001-de1 argument: 'null' startTime: '2020-10-16T20:19:34.448323739Z' name: projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c- 4065-a696-940bc7658c33 result: '"Hello, Cloud Workflows!"' endTime: '2020-10-16T20:19:34.845536315Z' state: SUCCEEDED Deploying gcloud workflows deploy hello --source=hello.yaml Executing gcloud workflows execute hello --data={“var”:”value”} Describing gcloud workflows executions describe-last
  11. 11. Sample Workflows What's new in serverless orchestration and automation @martonkodok
  12. 12. IT management automation Combine automation with scheduler Wait for service checks Orchestrate work across Compute Engine, PubSub, Stackdriver and other Google Cloud Products 9 AM trigger Start a Compute Engine VM Log the event App Started? No Wait 60 seconds Notify the team
  13. 13. E-commerce invoice generation with Workflows Steps orchestration Reliable execution, with error handling and retries Orchestration microservices or other API’s Create an invoice Generate PDF Send PDF via email Receive Order Cloud Run App Engine Cloud Run
  14. 14. Code Examples What's new in serverless orchestration and automation @martonkodok
  15. 15. 1. HTTP calls (GET or POST) 2. Sequence two steps 3. Working with subworkflows Workflow Basics What's new in serverless orchestration and automation @martonkodok
  16. 16. http_post.yaml Making an external HTTP POST request What's new in serverless orchestration and automation @martonkodok { "archived":false, "created_at":"2020-10-16T17:40:17+0000", "id":"bit.ly/35452TM", "link":"https://bit.ly/35452TM", "long_url":"<truncated>", }
  17. 17. wikipedia.yaml Sequence two steps to get data from Wikipedia What's new in serverless orchestration and automation @martonkodok
  18. 18. subworkflow.yaml Subworkflow What's new in serverless orchestration and automation @martonkodok call call def
  19. 19. What’s new in Workflows What's new in serverless orchestration and automation @martonkodok
  20. 20. 1. Functions: string, JSON, map, iterating syntax 2. Cloud logging and Call logging 3. Connectors: built-in authentication, retries, long-operation 4. Callbacks: enables async-executions, or human in the loop What’s new in Workflows What's new in serverless orchestration and automation @martonkodok
  21. 21. String Functions What's new in serverless orchestration and automation @martonkodok
  22. 22. Workflow Connectors What's new in serverless orchestration and automation @martonkodok
  23. 23. They are like client libraries for other services handling auth, retries and waiting. Connect to other Google Cloud APIs, and to integrate other GCP products. Connectors What's new in serverless orchestration and automation @martonkodok
  24. 24. Before: Calling Secret Manager traditionally using HTTP Get calls. Connectors: Replacing GET and POST calls What's new in serverless orchestration and automation @martonkodok After: using Connectors to get Secret Manager value
  25. 25. 1. Built in Authentication (calling service account needs permissions) 2. Retries (max_retries: 5, max_delay: 60, ability to define custom retry policies) 3. Timeout (default timeout: 30 minutes, can be increased to 1 year) 4. Long-running operations (blocks the workflow execution until the operation completes) 5. Polling (exponential backoff of 1.25 up to 60 seconds, each step is billable) Connectors details What's new in serverless orchestration and automation @martonkodok
  26. 26. Long running operation type of Workflow Connectors What's new in serverless orchestration and automation @martonkodok Product Operation Type Product Operation Type BigQuery Job Dataflow Job Cloud Build Operation Firestore Operation Cloud Functions Operation Document AI Operation Cloud SQL Operation Cloud Spanner Operation Compute Engine Operation Workflows Operation Execution
  27. 27. Workflow Callbacks What's new in serverless orchestration and automation @martonkodok
  28. 28. They are webhook endpoints where you signal that external event has occurred. Callbacks allow workflows executions to wait until the callback endpoint gets a call. Callbacks What's new in serverless orchestration and automation @martonkodok
  29. 29. 1. Modern way to avoid sys.sleep, wait, tracking tables, polling 2. Offers a webhook endpoint (secured with authentication) 3. Timeout (default timeout: 12 hours, can be increased to 1 year) 4. Event based (pauses the workflow execution until the call is received on endpoint) 5. Full callback details (callback_request holds headers,query,body,raw,json) Callback details What's new in serverless orchestration and automation @martonkodok
  30. 30. Orchestrate and automate the Cloud What's new in serverless orchestration and automation @martonkodok
  31. 31. Firestore Backups the easy way with Cloud Workflows - Cloud Scheduler - Cloud Workflows - Cloud Firestore - Cloud Storage Full guide article on: martonkodok.medium.com Cloud Workflows Cloud Storage Cloud Scheduler Firebase Cloud Firestore backup Roles and Permissions Cloud IAM authorize
  32. 32. Using Workflows to load Cloud Storage files into BigQuery - Cloud Workflows - Cloud Storage - BigQuery Full guide article on: martonkodok.medium.com Cloud Storage BigQuery Cloud Workflows Authenticated Invocations foreach
  33. 33. Run shell commands and orchestrate Compute Engine VMs - Cloud Workflows - Cloud Build - Compute Engine - Identity-Aware Proxy Full guide article on: martonkodok.medium.com Cloud Workflow Steps Cloud Workflows Execution Authorize Cloud Build Roles and Permissions Cloud IAM Shell Service Account Start VM Compute Engine Submit Stop VM Compute Engine Firewall Compute Engine Cloud IAP Tunnel wait wait
  34. 34. Conclusions What's new in serverless orchestration and automation @martonkodok
  35. 35. Reliable workflow execution - execute workflows for enterprise business apps Low latency of execution - no cold starts Built-in error handling out of the box error handling with configurable retry policies Passing variable built-in JSON parsing and expression-based variable manipulation Rich runtime iterating through an array, embedded steps for readability Secret Manager integration out of the box Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks Benefits of Cloud Workflows What's new in serverless orchestration and automation @martonkodok
  36. 36. Easy to build/operate Scales out Does not lose state Handles errors/timeouts Out-of-the-box support of Cloud APIs Auditable Enables async-workflows with Callbacks Developer friendly What's new in serverless orchestration and automation @martonkodok
  37. 37. The possibilities are endless Marketing Retail IndustrialandIoT Developer Event driven marketing workflow execution Relay conversions to customer profiles in external services Workflow based emails, discounts, promotions Order management Inventory chain operations Data gathering and processing Synchronize systems Generate state machines Verify equipment lifecycle Workflow based maintenance needs Digitalization of internal policies Automate the Cloud Shell-script replacement Orchestrate devops workflows @martonkodok
  38. 38. Thank you. Q&A. Slides available on: slideshare.net/martonkodok Reea.net - Integrated web solutions driven by creativity to deliver projects.

×