Successfully reported this slideshow.
Your SlideShare is downloading. ×

A Series of Fortunate Events: Building an Operator in Java

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 63 Ad

A Series of Fortunate Events: Building an Operator in Java

Download to read offline

SpringOne 2021:
Session Title: A Series of Fortunate Events: Building an Operator in Java
Speakers: Alberto C. Ríos, Staff Engineer at VMware; Bella Bai, Software Engineer at VMware

SpringOne 2021:
Session Title: A Series of Fortunate Events: Building an Operator in Java
Speakers: Alberto C. Ríos, Staff Engineer at VMware; Bella Bai, Software Engineer at VMware

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to A Series of Fortunate Events: Building an Operator in Java (20)

Advertisement

More from VMware Tanzu (20)

Advertisement

A Series of Fortunate Events: Building an Operator in Java

  1. 1. A series of fortunate events Building an operator in Java September 1–2, 2021 springone.io 1 Bella Bai @bellalleb_bai LittleBaiBai Alberto C. Ríos @Albertoimpl Albertoimpl Sample app: https://github.com/building-k8s-operator/kubernetes-java-operator-sample
  2. 2. What is an operator?
  3. 3. Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop. 3 -- Kubernetes Documentation Quote source: https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
  4. 4. Control loop of operators 4 Image source: https://github.com/cncf/tag-app-delivery/blob/master/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md#operator-components-in-kubernetes
  5. 5. An operator is a Kubernetes controller that understands two domains: Kubernetes and something else. By combining knowledge of both domains, it can automate tasks that usually require a human operator that understands both domains. 5 -- Jimmy Zelinskie Quote source: https://github.com/kubeflow/tf-operator/issues/300#issuecomment-357527937
  6. 6. The “business” context in our sample app 6 apiVersion: "operator.example.com/v1alpha1" kind: AdoptionCenter metadata: name: kittens-dream-land apiVersion: "operator.example.com/v1alpha1" kind: CatForAdoption metadata: name: my-precious-fluffy-ball spec: name: Chocobo dateOfBirth: 2014-09-17 description: She is chubby, lazy ... adoptionCenterName: kittens-dream-land The AdoptionCenter icon is made by Freepik: https://www.flaticon.com/
  7. 7. The “business” context in our sample app 7 The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/
  8. 8. When to and when not to?
  9. 9. When do we want to use an operator ● Deploying applications on demand ● Backing up and restoring an application's state ● Automating application upgrades ● Provide a decentralized way to manage centralized resources 9
  10. 10. When NOT to use an operator ● It is just an app with some configuration ● You don't need some special business logic for handling most of the operational work and can deploy it as it is ● When you can just deploy everything with Helm or Kustomize ● No need for any kind of persistence nor status to be backed 10
  11. 11. Scaffolding
  12. 12. Starting from scratch 12 ● Pick an operator library with Spring support ○ Kubernetes Java Client ○ Java Operator SDK ● Generate models based on your CRDs ○ Or generate your CRDs based on your model ○ Maintain single source of truth ● Setup your controller with resource listeners Kubernetes Java Client: https://github.com/kubernetes-client/java Java Operator SDK: https://github.com/java-operator-sdk/java-operator-sdk
  13. 13. @Bean public Controller adoptionCenterController( SharedInformerFactory sharedInformerFactory, AdoptionCenterReconciler reconciler) { return ControllerBuilder .defaultBuilder(sharedInformerFactory) .watch((q) -> ControllerBuilder .controllerWatchBuilder(V1alpha1AdoptionCenter.class, q) .withOnDeleteFilter((resource, cache) -> false) .withOnUpdateFilter((old, new) -> false) .withOnAddFilter((resource) -> true) .build()) .withReconciler(reconciler) .withName(AdoptionCenterReconciler.CONTROLLER_NAME) .build(); } Example controller 13
  14. 14. Starting from scratch 14 ● Pick an operator library with Spring support ○ Kubernetes Java Client ○ Java Operator SDK ● Generate models based on your CRDs ○ Or generate your CRDs based on your model ○ Maintain single source of truth ● Setup your controller with resource listeners Kubernetes Java Client: https://github.com/kubernetes-client/java Java Operator SDK: https://github.com/java-operator-sdk/java-operator-sdk ● Start reconciling 💪
  15. 15. Example reconciler 15 @Override public Result reconcile(Request request) { V1alpha1AdoptionCenter center = adoptionCenterLister.get(request.getName()) try { V1OwnerReference owner = toOwnerReference(center); configMapUpdater.createConfigMap(owner); deploymentEditor.createDeployment(owner); return new Result(false); } catch (Exception e) { return new Result(true); } }
  16. 16. Useful patterns to know
  17. 17. Managing Status
  18. 18. Managing status % kubectl get cats NAME my-precious-fluffy-ball 18
  19. 19. Managing status % kubectl get cats NAME my-precious-fluffy-ball 19
  20. 20. Managing status % kubectl get cats NAME READY REASON my-precious-fluffy-ball True CatAddedToConfigMap 2 0
  21. 21. Managing status schema: openAPIV3Schema : type: object properties : spec: ... status: type: object properties: conditions: type: array items: type: object properties: type: type: string description: The unique identifier of a condition, used to distinguish between other conditions in the resource. status: type: string description: The status of the condition. 21 https://github.com/building-k8s-operator/kubernetes-java-operator-sample/blob/main/ crds/cat-custom-resource-definition.yaml
  22. 22. Managing status apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: catsforadoption.operator.example.com spec: ... versions: - name: v1alpha1 served: true storage: true additionalPrinterColumns: - jsonPath: .status.conditions[?(@.type=="Ready")].status name: Ready type: string - jsonPath: .status.conditions[?(@.type=="Ready")].reason name: Reason type: string schema: … 2 2 https://github.com/building-k8s-operator/kubernetes-java-operator-sample/blob/main/ crds/cat-custom-resource-definition.yaml
  23. 23. Managing status String patch = String.format("{"status": " + "{ "conditions": " + "[{ "type": "%s", "status": "%s", "lastTransitionTime": "%s", "reason": "%s"}]" + "}}", type, status, ZonedDateTime.now(ZoneOffset.UTC), reason); 2 3 PatchUtils.patch( V1alpha1CatForAdoption.class, () -> api .patchNamespacedCatForAdoptionStatusCall( cat.getMetadata().getName(), cat.getMetadata().getNamespace(), new V1Patch(patch), null, null, null, null), V1Patch.PATCH_FORMAT_JSON_MERGE_PATCH, api.getApiClient());
  24. 24. Managing status % kubectl get cats NAME READY REASON my-precious-fluffy-ball True CatAddedToConfigMap 2 4
  25. 25. Garbage Collection
  26. 26. Garbage Collection V1alpha1AdoptionCenter adoptionCenter = lister.namespace(ns).get(name); if(isDeleteRequest(adoptionCenter)) { deploymentService.delete(adoptionCenter); configMapService.delete(adoptionCenter); secretService.delete(adoptionCenter); } 2 6
  27. 27. Garbage Collection % kubectl get deployment my-adoption-center -oyaml apiVersion: apps/v1 kind: Deployment metadata: name: my-adoption-center namespace: animal-rescue ownerReferences: - apiVersion: operator.example.com/v1alpha1 blockOwnerDeletion: true controller: true kind: AdoptionCenter name: my-adoption-center uid: 9d09ca3f-f92f-4745-9eb4-9798096b7408 27
  28. 28. Garbage Collection 2 8 public V1Deployment createDeployment(V1alpha1AdoptionCenter adoptionCenter){ ... deployment .getMetadata() .addOwnerReferencesItem(toOwnerReference(adoptionCenter)); ... } private V1OwnerReference toOwnerReference(V1alpha1AdoptionCenter adoptionCenter) { return new V1OwnerReference().controller(true) .name(adoptionCenter.getMetadata().getName()) .uid(adoptionCenter.getMetadata().getUid()) .kind(adoptionCenter.getKind()) .apiVersion(adoptionCenter.getApiVersion()) .blockOwnerDeletion(true); }
  29. 29. Readiness Gates
  30. 30. Readiness Gates 3 0 kind: Pod ... spec: readinessGates: - conditionType: "example.com/feature-1" status: Conditions: - ... # built in PodConditions - type: "example.com/feature-1" status: "True" lastProbeTime: null lastTransitionTime: 2021-09-01T00:00:00Z Image source: https://www.reddit.com/r/gatekeeping/comments/bafsci/cat_gatekeeping_guests/
  31. 31. Readiness Gates 31 kind: Pod ... spec: readinessGates: - conditionType: "example.com/feature-1" status: Conditions: - ... # built in PodConditions - type: "example.com/feature-1" status: "True" lastProbeTime: null lastTransitionTime: 2021-09-01T00:00:00Z Image source: https://www.reddit.com/r/gatekeeping/comments/bafsci/cat_gatekeeping_guests/
  32. 32. Finalizers
  33. 33. Finalizers are namespaced keys that tell Kubernetes to wait until specific conditions are met before it fully deletes resources marked for deletion. You can use finalizers to control garbage collection of resources. 33 -- Kubernetes Documentation Quote source: https://kubernetes.io/docs/concepts/overview/working-with-objects/finalizers/
  34. 34. Removing CatForAdoption without finalizers 34 The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/ Operator Desired State Actual State
  35. 35. Removing CatForAdoption without finalizers 3 5 The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/ Operator Desired State Actual State Return “Null”
  36. 36. Actual State Desired State Return “Null” Removing CatForAdoption without finalizers 3 6 The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/ Operator
  37. 37. Removing CatForAdoption with finalizers 37 The AdoptionCenter icons are made by Freepik: https://www.flaticon.com/ Operator Desired State Actual State metadata.deletionTimeStamp != null
  38. 38. Add / remove finalizer boolean toDelete = cat.getMetadata().getDeletionTimestamp() != null; if (!toDelete) { boolean notFound = cat.getMetadata().getFinalizers() == null || cat.getMetadata().getFinalizers().isEmpty(); if (notFound) { catFinalizerEditor.add(cat); } upsertCatToAdoptionCenter(cat); } else { removeCatFromAdoptionCenter(cat); catFinalizerEditor.remove(cat); } 38
  39. 39. CatFinalizerEditor code example public V1alpha1CatForAdoption add(V1alpha1CatForAdoption cat) throws ApiException { return PatchUtils.patch( V1alpha1CatForAdoption.class, () -> api.patchNamespacedCatForAdoptionCall( cat.getMetadata().getName(), cat.getMetadata().getNamespace(), new V1Patch("{"metadata":{"finalizers":["" + FINALIZER_STRING + ""]}}"), null, null, null, null), V1Patch.PATCH_FORMAT_JSON_MERGE_PATCH, api.getApiClient()); } 3 9
  40. 40. Watch out when you uninstall 4 0 ● Resources pending delete without operator ● What to do? ○ Pre-delete hook in operator ○ Pre-delete hook in Helm or other packaging options ○ Cancel uninstallation ○ Carefully order the uninstall process ○ Just document as known issue
  41. 41. Custom Resource Validation
  42. 42. Custom Resource Validation 4 2 Create/Update Custom Resource Request Request Decoding & Conversion Validation Storage Conversion etcd Response Encoding & Conversion
  43. 43. Syntactic validation schema: openAPIV3Schema: … name: type: string description: The name of the cat dateOfBirth: type: string format: date description: Date of birth of the cat in the format defined in RFC 3339 description: type: string description: The description of the cat adoptionCenterName: type: string description: Name of the adoption center to register this cat to required: ["adoptionCenterName", "name"] 43
  44. 44. Syntactic validation anyOf: - properties: spec: required: ["propertyA"] - properties: spec: properties: propertyB: items: required: ["propertyC"] 44
  45. 45. Custom Resource Validation 4 5 Create/Update Custom Resource Request Request Decoding & Conversion Validation Storage Conversion etcd Response Encoding & Conversion
  46. 46. Custom Resource Validation 4 6 Create/Update Custom Resource Request Request Decoding & Conversion Admission Storage Conversion etcd Response Encoding & Conversion Mutating webhooks Validation Validating webhooks
  47. 47. Semantic validation public V1beta1AdmissionReviewValidation validate(V1beta1AdmissionReview request) { boolean isAllowed = validate(request.getRequest().getOldObject().get("spec")); V1beta1AdmissionReviewResponse response = new V1beta1AdmissionReviewResponse(); response.setAllowed(isAllowed); return new V1beta1AdmissionReviewValidation(response); } 47
  48. 48. Testing
  49. 49. Testing 4 9 Image source: https://github.com/cncf/tag-app-delivery/blob/master/operator-wg/whitepaper/Operator-WhitePaper_v1-0.md#operator-components-in-kubernetes
  50. 50. Testing ● Unit Tests 5 0
  51. 51. Testing ● Unit Tests ● Component Tests: Closed-box test from the Operator's perspective We do not assert correctness of the deployed application but assert on the values it was deployed with. 51
  52. 52. @WithKubernetesCluster class CatFinalizerEditorComponentTest { ... } Component Tests 5 2 https://kind.sigs.k8s.io
  53. 53. @Autowired private CatFinalizerEditor finalizerEditor; @Autowired private TestK8sClient testK8sClient; @BeforeAll void createCrd() { testK8sClient.createCatCrd();} @Test void add_shouldAddFinalizerToCatResource () throws ApiException { V1alpha1CatForAdoption existing = testK8sClient.createCat(TEST_NAMESPACE, resourceName); V1alpha1CatForAdoption returned = finalizerEditor.add(existing); assertThat(returned.getMetadata().getFinalizers()).contains( FINALIZER_STRING); V1alpha1CatForAdoption catFromApiServer =api.readNamespacedCatForAdoption( resourceName, TEST_NAMESPACE, null, null); assertThat(catFromApiServer.getMetadata().getFinalizers()).contains( FINALIZER_STRING); } Component Tests 5 3
  54. 54. Testing ● Unit Tests ● Component Tests ● Acceptance Tests: Testing the system as a whole Using a real environment and installing the product as a user would. 5 4
  55. 55. @Test void journeyTest() throws Exception { V1alpha1AdoptionCenter adoptionCenter = testK8sClient.createAdoptionCenter(); assertThatSubResourcesAreDeployed(adoptionCenter); assertThatAdoptionCenterHasCat(adoptionCenter.getMetadata().getName(), 0); V1alpha1CatForAdoption cat = testK8sClient.createCat("default", "my-cat"); waitForReconciliationtoTriggerRestart(); assertThatAdoptionCenterHasCat(adoptionCenter.getMetadata().getName(), 1, cat); testK8sClient.deleteCat("default", "my-cat"); waitForReconciliationtoTriggerRestart(); assertThatAdoptionCenterHasCat(adoptionCenter.getMetadata().getName(), 0); testK8sClient.deleteAdoptionCenter(adoptionCenter.getMetadata().getName()); assertThatSubResourcesAreDeleted(adoptionCenter); } Acceptance tests 5 5
  56. 56. Demo Time! https://github.com/building-k8s-operator/kubernetes-java- operator-sample
  57. 57. Closing notes
  58. 58. Closing notes ● Operators are excellent for managing continuous reconciliation 5 8
  59. 59. Closing notes ● Operators are excellent for managing continuous reconciliation 5 9 ● Operators are hard to maintain and develop
  60. 60. Closing notes Java https://github.com/kubernetes-client/java https://github.com/java-operator-sdk/java-operator-sdk Golang https://github.com/operator-framework/operator-sdk https://github.com/kubernetes-sigs/kubebuilder 6 0 ● Operators are excellent for managing continuous reconciliation ● Operators are hard to maintain and develop
  61. 61. Closing notes ● Generate your Java classes from CRD: https://github.com/building-k8s-operator/kubernetes-java-operator-sample#gen erating-java-classes-from-crd ● Use Statuses, Finalizers, Readiness gates and Owner references. ● They can be TDDed! 61
  62. 62. Now it’s your turn to try! https://github.com/building-k8s-operator/kubernetes-java- operator-sample
  63. 63. Any questions? #springone @SpringOne Thank you! Bella Bai @bellalleb_bai Alberto C. Ríos @Albertoimpl

×