Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
Next

0

Share

JS Fest 2019. Peter Braun. DevOps in the Cluster: A deep dive into operators

Operators are a recent concept in the Kubernetes world. They allow you to package, deploy and manage Kubernetes applications. But there is more to it. Operators also let you encode your operational knowledge and apply it from within the cluster. This talk wants to introduce Operators, explain the why, give working examples and go into some best practices for development. We will also look at tasks beyond packaging and deployment that can be performed by an Operator.

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

JS Fest 2019. Peter Braun. DevOps in the Cluster: A deep dive into operators

  1. 1. CONTINUOUS DELIVERY. CONTINUOUS DEVOPS. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINE th
  2. 2. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINE Peter Braun | pbraun@redhat.com | Github: pb82 DevOps in the Cluster: a deep dive into Kubernetes Operators th
  3. 3. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Agenda 1. What is this talk about? 2. The Situation today 3. What is an Operator? 4. Kubernetes Controllers 5. Operator Tooling 6. Tips for developing Operators 7. Demo
  4. 4. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth What this talk is about: Kubernetes Applications and how to manage them.
  5. 5. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth So what is a ‘Kubernetes Application’? A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling.
  6. 6. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth What’s available today? ● Templates ● Helm
  7. 7. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Existing Tools: Templates ● YAML / JSON files ● Kubernetes Resources are purely declarative ● Parameterization is possible ○ Kustomize ○ Openshift Templates ● No concept of dependencies
  8. 8. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Existing Tools: Helm ● The Package Manager for Kubernetes, it can do: ○ Parameterization, ○ Dependency Resolution ○ and even Version Management
  9. 9. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth So what’s missing? ● Templates and Helm help with installation. ● Neither of them allow you to manage the Application.
  10. 10. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth The Solution? Operators!
  11. 11. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Let’s start from zero: what exactly is an Operator? An Operator is a method of packaging, deploying and managing a Kubernetes application.
  12. 12. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Let’s start from zero: what exactly is an Operator? An Operator is a method of packaging, deploying and managing a Kubernetes application using custom resources.
  13. 13. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Why is that distinction important? Operators are Kubernetes Controllers for Custom Resources.
  14. 14. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Kubernetes Controllers ● Kubernetes resources are backed by Controllers ● The purpose of a Controller is to synchronize ○ the cluster state... ○ ...with the desired state (resource definition).
  15. 15. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth An Example: Deployment Controller Desired state Cluster state Analyze Update
  16. 16. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Kubernetes Controllers: Detailed Look Informer (Cache) List Watch ns/name Add/Update/Delete Work QueueAPI Server Reconcile Analyze Update Pop Push back
  17. 17. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operators are Kubernetes Controllers for Custom Resources.
  18. 18. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Custom Resources ● Extension to the Kubernetes API ● Created with a Custom Resource Definition ● Let’s you define your own types
  19. 19. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Custom Resources (Example)
  20. 20. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Custom Resources (Commands) $ kubectl get crds $ kubectl get <CRD name>
  21. 21. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operators are Kubernetes Controllers for Custom Resources.
  22. 22. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operator Characteristics ● Typically written in Golang ● Standalone applications ● Deployed to a namespace ● Come bundled with their Custom Resource Definitions
  23. 23. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth So how to start developing Operators?
  24. 24. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operator Tooling Operator Framework CoreOS Operator Framework: https://coreos.com/operators/
  25. 25. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operator Framework Operator SDK CoreOS Operator Framework: https://coreos.com/operators/
  26. 26. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operator SDK ● CLI Tool to bootstrap new operators ● SDK to abstract controller facilities ● Testing and build
  27. 27. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operator SDK Development Workflow 1. Bootstrap new operator 2. Add API 3. Add Controller 4. Code, run locally, repeat 5. Build & Push image
  28. 28. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operator SDK Development Workflow
  29. 29. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Operator SDK (project structure) Structs to represent the custom resource(s) in Golang Often one controller per custom resource YAML resources (CRD, RBAC)
  30. 30. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Adding an API to the Operator operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService
  31. 31. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Adding a Controller to the Operator operator-sdk add controller --api-version=app.example.com/v1alpha1 --kind=AppService ● Creates a new controller ● Sets up the Informer ● Adds a Reconcile function ● Ready to implement your Logic
  32. 32. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Controller Implementation: Reconcile Function ● Called when resources change ● Only gets the resource name ● Must figure out what changes to the cluster are required ● Return value can be used to reschedule the resource
  33. 33. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Controller Implementation: Creating Resources ● Operator-SDK provides an API ● Programmatically or by parsing templates
  34. 34. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Cleanup and Garbage Collection ● Operators should allow clean uninstallation ● Finalizers and Owner References can help ● Use both with care How to handle deprovision / deletion of the CR?
  35. 35. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Finalizers Finalizer has been set. Resource won’t be deleted until removed.
  36. 36. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Finalizers Resource has Finalizer(s)? Delete Resource No Set Delete Timestamp Reschedule Yes Done Delete Request
  37. 37. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Owner References Resource is owned by another resource.
  38. 38. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Owner References Sets resource owner to the custom resource that triggered it’s creation
  39. 39. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Tips for developing Operators 1. Be careful when setting the Owner Reference 2. Be careful with Finalizers 3. Allow for deletion in any Operator state 4. Use the Phase Pattern 5. Don’t (over)use the Kubernetes API 6. Never rely on local testing only
  40. 40. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Be careful when setting the Owner Reference Why? ● Cascaded Deletion is great ● Best case: no delete logic needed in Operator ● There are cases where it’s problematic Example: ● Backup/Restore ● References an owner that now has a different UID or is not yet created.
  41. 41. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Tips for developing Operators 1. Always set the Owner Reference 2. Be careful with Finalizers 3. Allow for deletion in any Operator state 4. Use the Phase Pattern 5. Don’t (over)use the Kubernetes API 6. Never rely on local testing only
  42. 42. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Be careful with finalizers Why? ● Halts deletion ● Can propagate to parent resources (e.g. namespaces) Example: ● Uninstall script that removes all namespaces ● Blocked by finalizer in one resource ● Needs to be manually resolved now
  43. 43. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Tips for developing Operators 1. Always set the Owner Reference 2. Be careful with Finalizers 3. Allow for deletion in any Operator state 4. Use the Phase Pattern 5. Don’t (over)use the Kubernetes API 6. Never rely on local testing only
  44. 44. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Allow for deletion in any Operator state Why? ● Operators can get stuck ● Give users a chance to make them unstuck Example: ● Operator fails to create a resource (e.g. permissions) ● User requests deprovision ● Operator still in install phase
  45. 45. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Allow for deletion regardless of the Operator Phase How to fix this? ● No problem without finalizers ● Check the ‘deletionTimestamp’ in every state ● Always service finalizers
  46. 46. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Tips for developing Operators 1. Always set the Owner Reference 2. Be careful with Finalizers 3. Allow for deletion in any Operator state 4. Use the Phase Pattern 5. Don’t (over)use the Kubernetes API 6. Never rely on local testing only
  47. 47. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Use the Phase Pattern What is it? ● Operators are state machines ● Every task can be seen as a state ● Clear control flow Install Reconcile Uninstall (service finalizers)
  48. 48. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Use the Phase Pattern Check Phase Take Action Update Phase
  49. 49. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Tips for developing Operators 1. Always set the Owner Reference 2. Be careful with Finalizers 3. Allow for deletion in any Operator state 4. Use the Phase Pattern 5. Don’t (over)use the Kubernetes API 6. Never rely on local testing only
  50. 50. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Don’t (over) use the Kubernetes API Why? ● Bypasses the Cache ● Causes performance issues Example: ● Operator that use get/list excessively ● Should rely on the Informer instead ● Sometimes inevitable, e.g. resource creation
  51. 51. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Tips for developing Operators 1. Always set the Owner Reference 2. Be careful with Finalizers 3. Allow for deletion in any Operator state 4. Use the Phase Pattern 5. Don’t (over)use the Kubernetes API 6. Never rely on local testing only
  52. 52. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Never rely on local testing only Why? ● You can test your Operator without deploying it ● Permissions will be different though Example: ● Operator-sdk’s up local ● Starts the Operator locally without deploying it
  53. 53. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth What else is there? ● Dependency resolution ○ Operators can (and are expected to) deploy other operators ○ An example will be in the Demo
  54. 54. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth What else is there? A better way to do dependency resolution? Operator Lifecycle Manager
  55. 55. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth What else is there? OperatorHub: https://operatorhub.io/
  56. 56. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINEth Demo (from zero to a monitoring stack)
  57. 57. Professional conference on DevOps practices 6APRIL 2019 KYIV, UKRAINE Thank you! th

Operators are a recent concept in the Kubernetes world. They allow you to package, deploy and manage Kubernetes applications. But there is more to it. Operators also let you encode your operational knowledge and apply it from within the cluster. This talk wants to introduce Operators, explain the why, give working examples and go into some best practices for development. We will also look at tasks beyond packaging and deployment that can be performed by an Operator.

Views

Total views

94

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×