The document discusses how Kubernetes generates OpenAPI specifications from its resource model. It involves a multi-phase process: (1) A code generator compiles Kubernetes API definitions and documentation comments into a Go file defining the OpenAPI schema; (2) At runtime, additional information like supported verbs is added to construct the full specification; (3) Specs can be merged and filtered as needed. The spec is then used to generate clients and for discovery purposes like caching in kubectl. Future work could involve declaring more validation rules and defaults in the resource definitions.

1. Google Cloud Platform
How Does Kubernetes
Build OpenAPI Specifications?
Gluecon
2018-05-16
Daniel Smith <dbsmith@google.com>
Staff Software Engineer
originalavalamp@ (twitter)
(c) Google LLC

2. Kubernetes Resource Model in 60 seconds
● Allow humans and automated systems to work together
● Standardize all the things!
○ metadata
○ verbs
● Little control loops instead of big state machines
● JSON and proto transport mechanisms
● Clients generated from OpenAPI specs!
● And OpenAPI specs are generated from...

3. Our IDL. ish.
// Deployment enables declarative updates for Pods and ReplicaSets.
type Deployment struct {
metav1.TypeMeta `json:",inline"`
// Standard object metadata.
// +optional
metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
// Specification of the desired behavior of the Deployment.
// +optional
Spec DeploymentSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
// Most recently observed status of the Deployment.
// +optional
Status DeploymentStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

4. Our IDL. ish.
// Deployment enables declarative updates for Pods and ReplicaSets.
type Deployment struct {
metav1.TypeMeta `json:",inline"`
// Standard object metadata.
// +optional
metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
// Specification of the desired behavior of the Deployment.
// +optional
Spec DeploymentSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
// Most recently observed status of the Deployment.
// +optional
Status DeploymentStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}
User-visible documentation
Instructions for OpenAPI generator
Compilable go code

5. Our IDL - Continued
// DeploymentSpec is the specification of the desired behavior of the Deployment.
type DeploymentSpec struct {
// Number of desired pods. This is a pointer to distinguish between explicit
// zero and not specified. Defaults to 1.
// +optional
Replicas *int32 `json:"replicas,omitempty" protobuf:"varint,1,opt,name=replicas"`
// Template describes the pods that will be created.
Template v1.PodTemplateSpec `json:"template" protobuf:"bytes,3,opt,name=template"`
// Indicates that the deployment is paused.
// +optional
Paused bool `json:"paused,omitempty" protobuf:"varint,7,opt,name=paused"`
...
// The maximum time in seconds for a deployment to make progress before it
// is considered to be failed. The deployment controller will continue to
// process failed deployments and a condition with a ProgressDeadlineExceeded
// reason will be surfaced in the deployment status. Note that progress will
// not be estimated during the time a deployment is paused. Defaults to 600s.
ProgressDeadlineSeconds *int32 `json:"progressDeadlineSeconds,omitempty" protobuf:"varint,9,opt,name=progressDeadlineSeconds"`
}

6. ...shouldn’t that be a proto file?

7. ...shouldn’t that be a proto file?
● Maybe

8. (Phase 1) Compile step
● Repo: kubernetes/kube-openapi
● Uses a go parser / code generator library (“gengo”)
● Defines some extension tags...
// This is the comment tag that carries parameters for open API generation.
const tagName = "k8s:openapi-gen"
const tagOptional = "optional"
// Known values for the tag.
const (
tagExtensionPrefix = "x-kubernetes-"
tagPatchStrategy = "patchStrategy"
tagPatchMergeKey = "patchMergeKey"
patchStrategyExtensionName = "patch-strategy"
patchMergeKeyExtensionName = "patch-merge-key"
)

9. Output artifact: another go file
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
...
"k8s.io/api/apps/v1.Deployment": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Deployment enables declarative updates for Pods and ReplicaSets.",
Properties: map[string]spec.Schema{
"kind": {
SchemaProps: spec.SchemaProps{
Description: "Kind is a string value ...",
Type: []string{"string"},
Format: "",
},
},
"apiVersion": {
SchemaProps: spec.SchemaProps{
Description: "APIVersion defines the versioned schema ...",
Type: []string{"string"},
Format: "",
},
},
"metadata": {
SchemaProps: spec.SchemaProps{
Description: "Standard object metadata.",

10. Output artifact: another go file
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
...
"k8s.io/api/apps/v1.Deployment": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Deployment enables declarative updates for Pods and ReplicaSets.",
Properties: map[string]spec.Schema{
"kind": {
SchemaProps: spec.SchemaProps{
Description: "Kind is a string value ...",
Type: []string{"string"},
Format: "",
},
},
"apiVersion": {
SchemaProps: spec.SchemaProps{
Description: "APIVersion defines the versioned schema ...",
Type: []string{"string"},
Format: "",
},
},
"metadata": {
SchemaProps: spec.SchemaProps{
Description: "Standard object metadata.",
from: https://github.com/go-openapi/spec
// OpenAPIDefinition describes single type. Normally
these definitions are auto-generated using gen-openapi.
type OpenAPIDefinition struct {
Schema
spec.Schema
Dependencies []string
}

11. Output artifact: another go file
...
"k8s.io/apimachinery/pkg/util/intstr.IntOrString": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Type: intstr.IntOrString{}.OpenAPISchemaType(),
Format: intstr.IntOrString{}.OpenAPISchemaFormat(),
},
},
},
Define your own!
// OpenAPISchemaType is used by the kube-openapi generator when constructing
// the OpenAPI spec of this type.
//
// See: https://github.com/kubernetes/kube-openapi/tree/master/pkg/generators
func (_ IntOrString) OpenAPISchemaType() []string { return []string{"string"} }
// OpenAPISchemaFormat is used by the kube-openapi generator when constructing
// the OpenAPI spec of this type.
func (_ IntOrString) OpenAPISchemaFormat() string { return "int-or-string" }

12. Phase 2 (runtime): Time to add the verbs...
From here...
case "PUT": // Update a resource.
doc := "replace the specified " + kind
if isSubresource {
doc = "replace " + subresource + " of the specified " + kind
}
handler := metrics.InstrumentRouteFunc(action.Verb, resource, subresource, requestScope,
restfulUpdateResource(updater, reqScope, admit))
route := ws.PUT(action.Path).To(handler).
Doc(doc).
Param(ws.QueryParameter("pretty", "If 'true', then the output is pretty printed.")).
Operation("replace"+namespaced+kind+strings.Title(subresource)+operationSuffix).
Produces(append(storageMeta.ProducesMIMETypes(action.Verb), mediaTypes...)...).
Returns(http.StatusOK, "OK", producedObject).
// TODO: in some cases, the API may return a v1.Status instead of the versioned object
// but currently go-restful can't handle multiple different objects being returned.
Returns(http.StatusCreated, "Created", producedObject).
Reads(defaultVersionedObject).
Writes(producedObject)
addParams(route, action.Params)
routes = append(routes, route)

13. Phase 2 (runtime): Construct the spec
// BuildAndRegisterOpenAPIVersionedService builds the spec and registers a handler to provides access to it.
// Use this method if your OpenAPI spec is static. If you want to update the spec, use BuildOpenAPISpec then
RegisterOpenAPIVersionedService.
func BuildAndRegisterOpenAPIVersionedService(servePath string, webServices []*restful.WebService, config *common.Config, handler
common.PathHandler) (*OpenAPIService, error) { ... }
// BuildOpenAPISpec builds OpenAPI spec given a list of webservices (containing routes) and common.Config to customize it.
func BuildOpenAPISpec(webServices []*restful.WebService, config *common.Config) (*spec.Swagger, error) {
o := openAPI{
config: config,
swagger: &spec.Swagger{
SwaggerProps: spec.SwaggerProps{
Swagger: OpenAPIVersion,
Definitions: spec.Definitions{},
Paths: &spec.Paths{Paths: map[string]spec.PathItem{}},
Info: config.Info,
},
},
}
err := o.init(webServices)
if err != nil {
return nil, err
}
return o.swagger, nil
}

14. Phase 2.5: Aggregation
// MergeSpecs copies paths and definitions from source to dest, rename definitions if needed.
// dest will be mutated, and source will not be changed. It will fail on path conflicts.
func MergeSpecs(dest, source *spec.Swagger) error {
return mergeSpecs(dest, source, true, false)
}
// MergeSpecsIgnorePathConflict is the same as MergeSpecs except it will ignore any path
// conflicts by keeping the paths of destination. It will rename definition conflicts.
func MergeSpecsIgnorePathConflict(dest, source *spec.Swagger) error {
return mergeSpecs(dest, source, true, true)
}
// FilterSpecByPaths removes unnecessary paths and definitions used by those paths.
// i.e. if a Path removed by this function, all definition used by it and not used
// anywhere else will also be removed.
func FilterSpecByPaths(sp *spec.Swagger, keepPathPrefixes []string) {
...
}

15. Phase 3: Client usage
● The spec can change on the fly, so use ETAGs
● kubectl (our CLI) caches discovery information
● It is big, so compress
● It is slow to unmarshal JSON, so use proto
○ We reused the proto format from gnostic
● Many versions
○ OpenAPI version
○ proto encoding version
○ Our API version

16. Future work ideas
● Declare as much as possible in our IDL
○ Validation
○ Defaults
○ Const/immutability marker
○ Which of the standard verbs we support?
○ Subresources?
● OpenAPI -> protobuf definition?
○ `x-proto-tag` extension? (From openapi2proto)
● Custom extension tags
○ We’re moving `kubectl apply` (schema-aware smart update feature) to the control plane
○ (this could be a large rabbit hole)

17. Google Cloud Platform
Thank you!
Sound interesting? We’re hiring...
(c) Google LLC