Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Operator SDK for K8s using Go


Published on

Sebastien Thomas, System Architect at Coyote Amerique, gave a presentation on operator frameworks. His talk covered how Operator SDK can be used to create Kubernetes Operators with Go.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Operator SDK for K8s using Go

  1. 1. K8s / CNCF Meetup - 2019/02/25 Operator Framework From Github Project : The Operator Framework is an open source toolkit to manage Kubernetes native applications, called Operators, in an effective, automated, and scalable way.
  2. 2. Agenda 1. Overview of Operators 2. Operator SDK usage 3. Operators workflow 4. Hands On 5. Conclusion
  3. 3. COYOTE SYSTEM Who are we ? A leading provider of community-based driving assistant systems Founded in 2005 350 employees 1M daily users on a payed subscription model 2 billion kilometers traveled every month (1,24 billion miles) 50M members declaration analyzed every month Specific know-how in big data and automotive market protected by 13 patents
  4. 4. COYOTE SYSTEM How to be part of the community ? • Available on a range of Products and Apps, • … but also with embedded car solutions COYOTE mini COYOTE S COYOTE NAV+ Smartphone APPs
  5. 5. COYOTE SYSTEM Where does it work ? … almost everywhere in Europe ! › France › Belgium › Netherlands › Luxembourg › Italy › Spain › Germany › Poland › Portugal
  6. 6. Who I am ? 20+ years in Computers / Network / Admin / Devops / Woodworker Work at Coyote as System Architect Github : Blog (sort of) : Coyote Lab Blog (more to come there) : Contact : Sebastien “Prune” THOMAS -
  7. 7. What’s an Operator ? An Operator is an application that deals with the Kubernetes API and Custom Resources to create/operate new Resources. It’s an intelligent piece of software that embed the templating to deploy your resources. The Operator watch events on the K8s API and react (ex : re-create a pod, change Labels, update a Secret, Remove a Service…)
  8. 8. What are Custom Resource Definition CRD are new Resources, like Pods, Deployments, Secrets that you can create. They are managed through the K8s API the same way as official resources kubectl get crd 2019-01-25T15:56:53Z 2019-01-25T15:57:10Z 2019-01-25T16:05:42Z 2019-01-25T16:05:44Z 2019-01-25T16:09:16Z ...
  9. 9. Example Custom Resource apiVersion: kind: CustomResourceDefinition metadata: name: spec: group: names: kind: CertMerge listKind: CertMergeList plural: certmerges singular: certmerge scope: Namespaced version: v1alpha1 apiVersion: kind: CertMerge metadata: name: "test-certmerge-labels" spec: selector: - labelselector: matchLabels: env: "dev" certmerge: "true" namespace: default name: test-cert-labels namespace: default Custom Resource Definition (CRD) Custom Resource Manifest (CR)
  10. 10. Before Operators
  11. 11. With Operators
  12. 12. Operators 1 2 3 4 5 6 7 8 9
  13. 13. Operators
  14. 14. Difference with other tools - Helm / Jsonnet / Ksonnet They are templating tools. Create a template, set some variables, generate the Manifests. Once deployed they have no control (tiller does not count). - StatefulSets / Deployments / Pods They are K8s Resources. Some minimal feedback to scale/restart, no dependency between them, no intelligence in management. - Operators Watch the K8s API and react in real time. Can have a better control to scale/restart/configure the target application, with richer features than Readyness/Liveness Probes
  15. 15. Who needs Operators ? You may need an Operator if : - you need to use many times the same Application. ex : deploying one EtcD cluster in each Namespace - You need to automate some Resource creation. ex : create some SSL Certificates inside Secrets (cert-manager), create Prometheus scraping rules - You need more intelligence in the management. ex : the Etcd-Operator create and manage Pods directly instead of using a Deployment or StatefulSets
  16. 16. Helm Chart to deploy an Operator ? - Operators are usually easy to deploy - use whatever mean you have to deploy them (Helm, Jsonnet, plain manifest from the Operator creator) - Once the Operator is running, use the Custom Resources to trigger its power
  17. 17. Existing Operators ? - Etcd-Operator : - Kafka : - Nats : - Prometheus : - SSL Certificates : - RBAC-Manager : and a lot more, growing... (check
  18. 18. Operator all the thing ? An Operator embed the knowledge and the deployments “templates”. Don’t create an operator : - if your application deployment is not stable ! - to deploy one application per cluster (it’s easier to template it) Create an Operator : - if you have many users in need to use your resource - you have a complicated workflow to handle your resource - you want to (learn to) code in GO (or check other languages operators too)
  19. 19. Operator Creation Operator SDK (Go) : - High level APIs and abstractions to write the operational logic more intuitively - Tools for scaffolding and code generation to bootstrap a new project fast - Extensions to cover common operator use cases - Base on official Kubernetes API packages - Provide common package for leader election for HA Operators
  20. 20. CertMerge Operator -
  21. 21. Install (fast) mkdir -p $GOPATH/src/ cd $GOPATH/src/ git clone cd operator-sdk git checkout v0.4.0 make dep make install operator-sdk --version operator-sdk version v0.4.0+git
  22. 22. Create your operator mkdir -p $GOPATH/src/ cd $GOPATH/src/ operator-sdk new certmerge-operator --cluster-scoped INFO[0000] Create pkg/apis/apis.go INFO[0000] Create pkg/controller/controller.go INFO[0000] Create version/version.go INFO[0000] Create .gitignore INFO[0000] Create Gopkg.toml INFO[0000] Run dep ensure ... INFO[0068] Run dep ensure done INFO[0068] Run git init ... INFO[0074] Run git init done INFO[0074] Project creation complete. INFO[0000] Creating new Go operator 'certmerge-operator'. INFO[0000] Create cmd/manager/main.go INFO[0000] Create build/Dockerfile INFO[0000] Create build/bin/entrypoint INFO[0000] Create build/bin/user_setup INFO[0000] Create deploy/service_account.yaml INFO[0000] Create deploy/role.yaml INFO[0000] Create deploy/role_binding.yaml INFO[0000] Create deploy/operator.yaml
  23. 23. Add API # Add a new API for the custom resource AppService operator-sdk add api --kind=CertMerge This is the basic operation to create the CRD. It creates files in pkg/apis/certmerge/v1alpha1 including certmerge_types.go which holds the definition of the CRD : … type CertMerge struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec CertMergeSpec `json:"spec,omitempty"` Status CertMergeStatus `json:"status,omitempty"` } ...
  24. 24. Add Controler # Add a new controller that watches for AppService operator-sdk add controller --kind=CertMerge Creates files in pkg/controller/certmerge. This is where all your watch and reconcile logic happens Check doc reference at
  25. 25. type CertMerge struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec CertMergeSpec `json:"spec,omitempty"` Status CertMergeStatus `json:"status,omitempty"` } type CertMergeSpec struct { SecretName string `json:"name"` Selector []SecretSelector `json:"selector"` SecretNamespace string `json:"namespace"` SecretList []SecretDefinition `json:"secretlist"` } type SecretSelector struct { LabelSelector metav1.LabelSelector `json:"labelselector"` Namespace string `json:"namespace"` } Types (API)
  26. 26. // add adds a new Controller to mgr with r as the reconcile.Reconciler func add(mgr manager.Manager, r reconcile.Reconciler) error { // Create a new controller c, err := controller.New("certmerge-controller", mgr, controller.Options{Reconciler: r}) if err != nil { return err } // Watch for changes to primary resource CertMerge err = c.Watch(&source.Kind{Type: &certmergev1alpha1.CertMerge{}}, &handler.EnqueueRequestForObject{}) if err != nil { return err } // TODO(user): Modify this to be the types you create that are owned by the primary resource // Watch for changes to secondary resource Pods and requeue the owner CertMerge err = c.Watch(&source.Kind{Type: &corev1.Secret{}}, &handler.EnqueueRequestForOwner{ IsController: true, OwnerType: &certmergev1alpha1.CertMerge{}, }) if err != nil { return err } Watchers
  27. 27. func (r *ReconcileCertMerge) Reconcile(request reconcile.Request) (reconcile.Result, error) { … // Fetch the CertMerge instance that triggered this Reconsile instance := &certmergev1alpha1.CertMerge{} err := r.client.Get(context.TODO(), request.NamespacedName, instance) if err != nil { if errors.IsNotFound(err) { // Request object not found, could have been deleted after reconcile request. // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers. // Return and don't requeue ( by sending `nil` in the error field) return reconcile.Result{}, nil } // Error reading the object - requeue the request. (by sending a non-nil error) return reconcile.Result{}, err } … do some stuff for your operator (see next slide) } Reconcile
  28. 28. // Define a new Secret object secret := newSecretForCR(instance) // create the DATA for the new secret based on the CertMerge request certData := make(map[string][]byte) // Set CertMerge instance as the owner and controller (for garbage collection) if err := controllerutil.SetControllerReference(instance, secret, r.scheme); err != nil { return emptyRes, err } // build the Cert Data from the secret List provided in the CertMerge Custom Resource if len(instance.Spec.SecretList) > 0 { for _, sec := range instance.Spec.SecretList { secContent, err := r.searchSecretByName(ctx, sec.Name, sec.Namespace) ... certData[sec.Name+".crt"] = secContent.Data["tls.crt"] certData[sec.Name+".key"] = secContent.Data["tls.key"] } } // add the Data to the secret secret.Data = certData // create the new secret if err := r.client.Create(ctx, secret); err != nil {...} Reconcile 2
  29. 29. Generate and build # re-generate all the files that depend on the CRD API operator-sdk generate k8s # re-generate the CRD Manifest (rarely used, when you change your API name) operator-sdk generate openapi # build the operator (aka go build) operator-sdk build prune/certmerge-operator:v0.0.1
  30. 30. Operator workflow (easy)
  31. 31. Operator workflow (harder with Predicate)
  32. 32. Hands-on switch to the console / code
  33. 33. Conclusion ● Operator SDK make it really easy ● using K8s primitives (and go-client), not “vendor” dependent ● Operators can be declined in Controlers (admission) ● You need to learn a little bit of the K8s API to get to cool stuff
  34. 34. References ● ● ● ●