Downloaded 22 times
Uploaded byBlueData, Inc.
Introduction to KubeDirector - SF Kubernetes Meetup
Kubedirector is an open-source project designed to manage stateful applications on Kubernetes, focusing on complex big data, AI, and machine learning deployments. It allows application experts and administrators to deploy and manage applications through a custom controller that operates with CRDs, enabling a separation of concerns among users. The framework simplifies the process by being data-driven and requires no app-specific logic in its code, facilitating interactions among different applications.
More Related Content
Similar to Introduction to KubeDirector - SF Kubernetes Meetup
Introduction to KubeDirector - SF Kubernetes Meetup
- 1. KubeDirector:KubeDirector: New Open SourceProjectNew Open Source Project for Stateful Applicationsfor Stateful Applications on Kuberneteson Kubernetes
- 2. Today’s Speakers Joel Baxter @joel_k_baxter DistinguishedEngineer BlueData Tom Phelan @tapbluedata Co-Founder and Chief Architect BlueData
- 3. Agenda • Complex StatefulApplications – What Makes Large-Scale Big Data Analytics, AI, Machine Learning, and Deep Learning Applications Different • BlueK8s • KubeDirector • Demo • Q & A
- 4. Complex Stateful Applications •Big Data / AI / Machine Learning / Deep Learning • What do all these applications have in common? – Require large amounts of data – Use distributed processing, multiple tools / services – When on-prem, typically deployed on bare-metal – Do not have a cloud native architecture • No microservices • Application instance-specific state
- 5. Kubernetes – Pod •Ideally: Each application service could be deployed in its own container running in a Pod – Microservices architecture • Current reality: All services of each node for a complex stateful application must run in the same container – The Pod ordering feature does not help in the ordering of services (which is key for complex stateful apps)
- 6. Kubernetes – Operators •Operator • A way of packaging, deploying, and managing a given application • Operator Framework • A set of developer and runtime tools to help accelerate the writing of a Operator • Operator SDK • An SDK that further hides the complexities of the Kubernetes API Source: https://coreos.com/operators
- 7. Kubernetes – Operators •Still best suited when application is decomposed into independent services – Primarily in the realm of the application vendor or OSS community to change/re- architect apps (e.g. Spark)
- 8. • Application-specific meansa new operator needs to be written for each application Kubernetes – Operators
- 9. BlueK8s and KubeDirector •BlueK8s is an Apache open source initiative focused on bringing enterprise support for complex stateful applications to Kubernetes • A series of projects will be rolled out under the BlueK8s umbrella – The first major project is “KubeDirector” Source: www.bluedata.com/blog/2018/07/operation-stateful-bluek8s-and-kubernetes-director
- 10. KubeDirector Overview • KubeDirector(KD) is a K8s “custom controller” • Watches for custom resources (CRs) to appear/change • Creates/modifies standard K8s resources (StatefulSets etc.) in response, to implement specs from CRs • Differs from normal operator pattern: – No app-specific logic in KubeDirector code – App deployment is data-driven from external app definitions – Supports interactions among different apps + other objects
- 11. Deploy KubeDirector toK8s Learn more at: https://github.com/bluek8s/kubedirector/wiki kubectl create -f kubedirector/deployment.yaml
- 12. Separation of Concerns •Application experts (on-site or elsewhere) – Responsible for making app images/metadata/configscripts – No need to write go code or understand operator concepts • Administrators (on-site) – Select which apps are available to end users – Change app versions independently of KubeDirector upgrade • End users – Pick from menu of applications and config choices
- 13.
- 14. Custom Resource Definitions •Primary CRD: KubeDirectorCluster – Models any kind of application instance launchable by KubeDirector • Other CRDs for related objects, e.g. – App definitions (KubeDirectorApp) – DataTaps and other shared storage – Config sets for AD/LDAP integration for containers – Machine Learning models • This talk will concentrate on KubeDirectorCluster/App
- 15.
- 16. Deployment • Create customresource definitions (CRDs) in your K8s cluster • Deploy KubeDirector – Normally runs in a pod on same K8s cluster – Authenticates to K8s API w/ privileged service account • Configure KubeDirector global settings – E.g. supply app definitions, set types of service & storage
- 17.
- 18. App Definition Metadata •App identifier/description/version • Service endpoints • Available “roles”, and container image per role • Available deploy-time choices, and their effects on services per role • Info for optional runtime setup package • And more!
- 19. App Definition Example1/3 apiVersion: kubedirector.bluedata.io/v1alpha1 kind: KubeDirectorApp metadata: name: spark221e2 spec: label: name: Spark 2.2.1 on centos7x with Jupyter image: repoTag: docker.io/bluedata/sparkbase:2.0 setup_package: import_url: https://s3.amazonaws.com/mybucket/spark221e2/appconfig.tgz
- 20. App Definition Example2/3 - id: spark_master_ui label: name: Spark master (web UI) endpoint: port: 8080 is_dashboard: true url_scheme: http - id: spark_worker_ui label: name: Spark worker (web UI) endpoint: port: 8081 is_dashboard: true url_scheme: http roles: - id: controller cardinality: 1 - id: worker cardinality: 0+ services: - id: spark label: name: Spark master endpoint: port: 7077
- 21. App Definition Example3/3 config: selected_roles: - controller - worker role_services: - role_id: controller service_ids: - spark - spark_master_ui - role_id: worker service_ids: - spark_worker_ui
- 22. Application Setup Package •Optional tgz injected into each container, contains: – Entrypoint script – Standard script functions for reading deployment info – Any artifacts (config file templates etc.) required for setup • Entrypoint script will be invoked at lifecycle events: – This container has just been created – Some other member(s) added to or removed from the cluster
- 23. Setup Script Actions •Perform any setup that requires runtime info – E.g. FQDNs of other member(s) of the cluster • Enable and start appropriate services – Can query the role of current node – Services-to-start depend on role and deploy-time choices • Can use features of KubeDirector Agent, if deployed
- 24.
- 25. Custom Resource Creation apiVersion:"kubedirector.bluedata.io/v1alpha1" kind: "KubeDirectorCluster" metadata: name: "spark-instance" spec: app: spark221e2 roles: - id: controller resources: limits: memory: "4Gi“ - id: worker members: 2
- 26.
- 27.
- 28. Other Operations • Shrink& expand of role member count is handled similarly • All resources are automatically cleaned up if CR is deleted (because CR is their “owner”) • End user can read the CR to see current status, service objects, etc.
- 29.
- 30. Join the KubeDirectorCommunity! Joel Baxter @joel_k_baxter Tom Phelan @tapbluedata https://github.com/bluek8s
Editor's Notes
- #6 Too many interdependencies in current monolithic design In this deployment model, the Pod is equivalent to a single container
- #8 Client (Helm) / Server (Tiller) Functions: package mgmt, app lifecycle mgmt, rendering Functionality permissions are “all or nothing” With Tiller, Helm cannot support per function user privileges Tiller scheduled to be dropped in the next release of Helm Without Tiller, Helm will have difficulty managing long-running tasks and monitoring
- #14 Each container in a deployed cluster will take on a role, which will determine: - the container image - the group of services running on that container - the resources (CPU/GPU/memory/disk) available to the container Functionality provided by optional agent: - Advanced service management - Track and advertise service statuses - Service watchdogging (restart) - Advanced configuration coordination - Token wait/wake between cluster members - Remote file copy within cluster as particular user - Remote command exec within cluster as particular user - Advanced job/script submission - Run on one or multiple nodes - Track status and output; - Advertise status & optionally send output to log collection
- #15 When to model things through CRDs? If an end user can create/edit the thing. The “related objects” CRDs can be used in conjunction with multiple clusters of multiple application types. CRDs are not necessary for administrative configuration of KD settings.
- #16 Largely a standard custom controller deployment so we will skim over this.
- #17 KD authenticates to k8s as a service account - has all privs in its own custom API space - has necessary privs (to deploy app clusters) for k8s resources in some k8s namespace(s) - pods, services, PVCs, etc.
- #19 Setup package is injected into containers. Catalog entry identifies URL for acquiring package, and whether the package’s operation requires that KubeDirector Agent be deployed into the containers. Backup slides have an example catalog entry.
- #20 Somewhat more complicated example on bluek8s github wiki.
- #22 (example of “config choices” not shown)
- #23 Edited/simple example script on bluek8s github wiki.
- #24 Configuration-related functionality provided by optional agent: - Token wait/wake between cluster members - Remote file copy within cluster as particular user - Remote command exec within cluster as particular user
- #26 To deploy an app cluster, the end user creates a CR with their tool of choice (e.g. kubectl). Slightly more complex example on bluek8s github wiki.
- #27 1. An end user submits a CR spec to the K8s API. 2. K8s admission control consults the validator module in KubeDirector. 3. The validator consults the app catalog. If the request is valid for that app (e.g. appropriate choices, number of replicas, minimum memory for role, etc.) then the request is OK'ed. 4. The CR appears in the K8s KV store. 5. KubeDirector notices the CR and starts its reconciliation handler. 6. The handler consults the app catalog for appropriate images, services, persistent storage volumes, etc. It asks the K8s API to create stateful sets, services, etc. and to update the CR object's status properties. 7. Requested new and updated objects appear in the K8s KV store. 8. K8s begins creating pods to comply with stateful set replica counts.
- #28 9. KubeDirector periodically polls the CR and runs the reconciliation handler again. 10. The handler checks on the current state of all related objects in the K8s store. Once all desired pods have become running pods, it proceeds to next steps. 11. The handler injects configuration data into the pods, and then an invocation that will fetch and run the application setup package. 12. Within each pod, the setup package is fetched from an artifact store and executed. 13. Once the application setup completes, the handler updates the CR status properties. 14. New status properties appear on the CR in the K8s KV store.
- #29 See backup slides for more reconciliation logic details.