iTranslated by AI
Ways to Work with Kubernetes Custom Resources in Go
As far as I have researched, there seem to be the following three ways to operate Kubernetes custom resources from Go.
- Using a clientset generated by code-generator
- Using controller-runtime
- Using the dynamic client of client-go
In this article, I will explain the use cases, advantages, and disadvantages of each, along with actual code examples. All code examples are published in the following repository for your reference.
Preparing the verification environment
We will create a Kubernetes cluster in a local environment using kind and apply a CustomResourceDefinition (CRD) for verification. This time, I defined a CRD named MyResource with two properties (field1, field2) in its spec.
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type MyResource struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec MyResourceSpec `json:"spec"`
}
type MyResourceSpec struct {
Field1 string `json:"field1"`
Field2 int32 `json:"field2"`
}
type MyResourceList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata"`
Items [] MyResource `json:"items"`
}
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: myresources.example.com
spec:
group: example.com
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
field1:
type: string
field2:
type: integer
scope: Namespaced
names:
plural: myresources
singular: myresource
kind: MyResource
While these files can be implemented manually, using kubebuilder, which is widely used for Kubernetes controller implementation, allows you to automatically generate a boilerplate for the custom resource structs. It also allows you to automatically generate the CRD manifest from the Go structs.
Run the following commands to create the cluster and apply the CRD.
# Create a kind cluster
kind create cluster --name crd-demo
# Apply the CRD
kubectl apply -f manifests/crd.yaml
Using the clientset generated by code-generator
code-generator is a tool that automatically generates an API client, called a clientset, from the custom resource types. Since the generated clientset API holds the type information of the custom resource, you can implement CRUD operations safely while utilizing compile-time type checking and IDE completion.
On the other hand, as will be explained in detail later, generating a clientset requires tool setup and management of the generated code. Also, if you want to handle multiple custom resources, you will need a clientset for each custom resource. Therefore, I think this method is suitable when an OSS provides a clientset or when the number of custom resources to be handled is small.
Generating the clientset with code-generator
Add the following comments to the file where the custom resource type is defined.
+// +k8s:deepcopy-gen=package
+// +groupName=example.com
package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
+// +genclient
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type MyResource struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
...
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type MyResourceList struct {
metav1.TypeMeta `json:",inline"`
...
The meanings of the comments are as follows:
-
// +k8s:deepcopy-gen=package- Generate DeepCopy methods for all types in the package.
-
// +groupName=example.com- Specify the group name of this API as example.com.
-
// +genclient- Generate code for custom resource operations such as a clientset.
-
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object- Generate a DeepCopy method that satisfies the runtime.Object interface.
By the way, I have not yet found the official specification for the comments, so I referred to the following blog article cited in the official repository.
Next, add code-generator as a project dependency. However, since we don't want to include it in the normal build, prepare a dedicated file with the //go:build tools directive and import code-generator within it.
//go:build tools
package scripts
import _ "k8s.io/code-generator"
Then, referring to the script in the examples directory of code-generator, prepare a shell script for code generation. The file specified with the --boilerplate option is a template for the header comment (license info, etc.) to be inserted at the beginning of the generated Go files. If there is no content to insert, an empty file is fine.
#!/usr/bin/env bash
SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
SCRIPT_ROOT="${SCRIPT_DIR}/.."
CODEGEN_PKG=$(go list -m -f '{{.Dir}}' k8s.io/code-generator)
source "${CODEGEN_PKG}/kube_codegen.sh"
# Path to the Go project
THIS_PKG="github.com/nissy-dev/sandbox/go-crd-operation"
kube::codegen::gen_helpers \
--boilerplate "${SCRIPT_ROOT}/scripts/boilerplate.go.txt" \
"${SCRIPT_ROOT}"
kube::codegen::gen_register \
--boilerplate "${SCRIPT_ROOT}/scripts/boilerplate.go.txt" \
"${SCRIPT_ROOT}"
kube::codegen::gen_client \
--output-dir "${SCRIPT_ROOT}/pkg/generated" \
--output-pkg "${THIS_PKG}/pkg/generated" \
--boilerplate "${SCRIPT_ROOT}/scripts/boilerplate.go.txt" \
"${SCRIPT_ROOT}/pkg/apis"
While kube::codegen::gen_client is used to generate the clientset, you also need to execute the following two commands that the internal implementation of the clientset depends on:
-
kube::codegen::gen_helpers: Generates helper code such as DeepCopy methods -
kube::codegen::gen_register: Generates code to register the custom resource schema with runtime.Scheme
Executing this script will generate code under pkg/generated/.
Implementation Example
Using the generated clientset, you can operate resources in a type-safe manner as follows:
package main
import (
"context"
"fmt"
"path/filepath"
examplev1 "github.com/nissy-dev/sandbox/go-crd-operation/pkg/apis/example.com/v1"
clientset "github.com/nissy-dev/sandbox/go-crd-operation/pkg/generated/clientset/versioned"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/util/homedir"
)
const namespace = "default"
func main() {
ctx := context.Background()
// Create the clientset
kubeconfigPath := filepath.Join(homedir.HomeDir(), ".kube", "config")
cfg, _ := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
client, _ := clientset.NewForConfig(cfg)
resourceClient := client.ExampleV1().MyResources(namespace)
// Create a custom resource
resourceName := "sample-resource-1"
resource := &examplev1.MyResource{
ObjectMeta: metav1.ObjectMeta{
Name: resourceName,
Namespace: namespace,
},
Spec: examplev1.MyResourceSpec{
Field1: "value-1",
Field2: 100,
},
}
_, _ = resourceClient.Create(ctx, resource, metav1.CreateOptions{})
// Retrieve a list of resources
resourceList, _ := resourceClient.List(ctx, metav1.ListOptions{})
for _, item := range resourceList.Items {
fmt.Printf(" - Name: %s, Field1: %s, Field2: %d\\n",
item.Name, item.Spec.Field1, item.Spec.Field2)
}
// Delete the resource
_ = resourceClient.Delete(ctx, resourceName, metav1.DeleteOptions{})
}
Since the arguments for the Create method and others in the client provided by the clientset now accept *examplev1.MyResource, passing other custom resources will result in a compilation error.
Using controller-runtime
controller-runtime is a collection of useful libraries for implementing controllers in Kubernetes.
In Kubebuilder, using controller-runtime is recommended over generating an individual clientset for each custom resource. This is because a single client can handle multiple custom resources with almost no change to the API interface. This is discussed in the following issue:
The client.Object interface handled by controller-runtime
The controller-runtime client handles the client.Object interface.
type Object interface {
metav1.Object
runtime.Object
}
// runtime.Object
type Object interface {
GetObjectKind() schema.ObjectKind
DeepCopyObject() Object
}
In this case, the metav1.Object interface is implemented by embedding metav1.TypeMeta and metav1.ObjectMeta into the custom resource struct.
As for the runtime.Object interface, GetObjectKind is implemented by metav1.TypeMeta. For DeepCopyObject, you need to implement methods like the following in the custom resource struct:
//go:build !ignore_autogenerated
// +build !ignore_autogenerated
// Code generated by deepcopy-gen. DO NOT EDIT.
package v1
import (
runtime "k8s.io/apimachinery/pkg/runtime"
)
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MyResource) DeepCopyInto(out *MyResource) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
out.Spec = in.Spec
out.Status = in.Status
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MyResource.
func (in *MyResource) DeepCopy() *MyResource {
if in == nil {
return nil
}
out := new(MyResource)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *MyResource) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MyResourceList) DeepCopyInto(out *MyResourceList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]MyResource, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MyResourceList.
func (in *MyResourceList) DeepCopy() *MyResourceList {
if in == nil {
return nil
}
out := new(MyResourceList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *MyResourceList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *MyResourceSpec) DeepCopyInto(out *MyResourceSpec) {
*out = *in
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MyResourceSpec.
func (in *MyResourceSpec) DeepCopy() *MyResourceSpec {
if in == nil {
return nil
}
out := new(MyResourceSpec)
in.DeepCopyInto(out)
return out
}
If you are using Kubebuilder's controller-gen, the implementation of this interface is automatically generated, so it should be fine in most cases.
Implementation Example
package main
import (
"context"
"fmt"
"path/filepath"
examplev1 "github.com/nissy-dev/sandbox/go-crd-operation/pkg/apis/example.com/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/util/homedir"
"sigs.k8s.io/controller-runtime/pkg/client"
)
const namespace = "default"
var SchemeGroupVersion = schema.GroupVersion{
Group: "example.com",
Version: "v1",
}
func addMyResourceTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(
SchemeGroupVersion,
&examplev1.MyResource{},
&examplev1.MyResourceList{},
)
metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}
func main() {
ctx := context.Background()
// Create a Scheme (register types)
scheme := runtime.NewScheme()
schemeBuilder := runtime.NewSchemeBuilder(addMyResourceTypes)
schemeBuilder.AddToScheme(scheme)
// Create a controller-runtime client
kubeconfigPath := filepath.Join(homedir.HomeDir(), ".kube", "config")
cfg, _ := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
k8sClient, _ := client.New(cfg, client.Options{Scheme: scheme})
// Create a custom resource
resourceName := "sample-resource-1"
resource := &examplev1.MyResource{
ObjectMeta: metav1.ObjectMeta{
Name: resourceName,
Namespace: namespace,
},
Spec: examplev1.MyResourceSpec{
Field1: "value-1",
Field2: 100,
},
}
_ = k8sClient.Create(ctx, resource)
// Retrieve a list of resources
resourceList := &examplev1.MyResourceList{}
_ = k8sClient.List(ctx, resourceList)
for _, item := range resourceList.Items {
fmt.Printf(" - Name: %s, Field1: %s, Field2: %d\n",
item.Name, item.Spec.Field1, item.Spec.Field2)
}
// Delete the created resource
resource = &examplev1.MyResource{
ObjectMeta: metav1.ObjectMeta{
Name: resourceName,
Namespace: namespace,
},
}
_ = k8sClient.Delete(ctx, resource)
}
In the case of controller-runtime, unlike a clientset, the client requires the registration of a schema. In the code example above, the GVK (Group, Version, Kind) of the custom resource is registered in runtime.Scheme using the addMyResourceTypes function. This registration allows the controller-runtime client to determine which custom resource the passed client.Object corresponds to. When handling multiple custom resources, you pass each registration function to runtime.NewSchemeBuilder.
Using the dynamic client of client-go
The dynamic client in client-go is a versatile client that can operate on any resource dynamically. Controller-runtime also uses the dynamic client internally.
The advantage of the dynamic client is that it allows you to operate on custom resources without having to generate code or implement additional interfaces.
The unstructured type handled by the dynamic client
In the dynamic client, all resources are handled as the unstructured.Unstructured type. The internal implementation of this type is map[string]interface{}, which can represent flexible structures like JSON.
type Unstructured struct {
Object map[string]interface{}
}
Therefore, compile-time type checking does not work, and type assertions are required when accessing fields. On the other hand, it has the flexibility to handle any custom resource even without a prior type definition.
Implementation Example
package main
import (
"context"
"fmt"
"path/filepath"
examplev1 "github.com/nissy-dev/sandbox/go-crd-operation/pkg/apis/example.com/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/client-go/dynamic"
"k8s.io/client-go/tools/clientcmd"
"k8s.io/client-go/util/homedir"
)
const namespace = "default"
var myResourceGVR = schema.GroupVersionResource{
Group: "example.com",
Version: "v1",
Resource: "myresources",
}
func main() {
ctx := context.Background()
// Create the dynamic client
kubeconfigPath := filepath.Join(homedir.HomeDir(), ".kube", "config")
cfg, _ := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
dynamicClient, _ := dynamic.NewForConfig(cfg)
resourceClient := dynamicClient.Resource(myResourceGVR).Namespace(namespace)
// Create a custom resource
resourceName := "sample-resource-1"
resource := &examplev1.MyResource{
TypeMeta: metav1.TypeMeta{
APIVersion: "example.com/v1",
Kind: "MyResource",
},
ObjectMeta: metav1.ObjectMeta{
Name: resourceName,
Namespace: namespace,
},
Spec: examplev1.MyResourceSpec{
Field1: "value-1",
Field2: 100,
},
}
unstructuredObj, _ := runtime.DefaultUnstructuredConverter.ToUnstructured(resource)
_, _ = resourceClient.Create(ctx, &unstructured.Unstructured{Object: unstructuredObj}, metav1.CreateOptions{})
// Retrieve a list of resources
resourceList, _ := resourceClient.List(ctx, metav1.ListOptions{})
for _, item := range resourceList.Items {
var myResource examplev1.MyResource
_ = runtime.DefaultUnstructuredConverter.FromUnstructured(item.Object, &myResource)
fmt.Printf(" - Name: %s, Field1: %s, Field2: %d\n",
myResource.Name, myResource.Spec.Field1, myResource.Spec.Field2)
}
// Delete the created resource
_ = resourceClient.Delete(ctx, resourceName, metav1.DeleteOptions{})
}
As mentioned earlier, since the dynamic client handles the unstructured type, it is necessary to convert between the custom resource type and the unstructured type.
Summary
In this article, I introduced three ways to operate on Kubernetes custom resources in Go. My current thinking is to first consider using controller-runtime, then use a clientset when prioritizing type safety, or the dynamic client when prioritizing flexibility.
In the project where I recently needed to research this, I chose to use the dynamic client because it only handled one custom resource and only needed to retrieve custom resource metadata using the Get method.
In this article, I was only able to compare the basic usage of each method. However, during my research, I also found an article that delves into more advanced topics like resource cache handling and concurrency considerations for each method. If you handle custom resources on a large scale, this article will also be a great reference when choosing a method.
Discussion