kuberneteskubernetes-operatorkubebuilder

Kubebuilder: resource deletion not having expected side effects


I'm creating a custom resource definition (CRD) with an associated controller using kubebuilder. My controller reconcile loop creates a deployment sub-resource and parents it to the custom resource using controllerutil.SetControllerReference(&myResource, deployment, r.Scheme). I've also configured my reconciler so "own" the sub-resource, as follows:

// SetupWithManager sets up the controller with the Manager.
func (r *MyResourceReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&mygroupv1alpha1.MyResource{}).
        Owns(&appsv1.Deployment{}).
        Complete(r)
}

However, when I run my controller locally using make run, I noticed that deleting the my CR (the root object) doesn't cause the Deployment sub-resource to get garbage collected. I also noticed that deleting the Deployment sub-resource doesn't trigger my reconciler to run. Why is this? Is there something I'm not doing or is this possibly a limitation of local development/testing?


Solution

  • Using @coderanger's hint, I could see that the metadata.ownerReferences weren't being set correctly when running the following command:

    kubectl get deployments sample-deployment -o yaml
    

    The problem was my controller's reconcile code. I was calling controllerutil.SetControllerReference(&myResource, deployment, r.Scheme) only after I'd already created and persisted the Deployment.

    Buggy code

    log.Info("Creating a deployment")
    
    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      deploymentName,
            Namespace: myResource.Namespace,
        },
        Spec: deploymentSpec,
    }
    
    if err = r.Create(ctx, deployment); err != nil {
        log.Error(err, "Failed to create deployment")
        if errors.IsInvalid(err) {
            // Don't retry on validation errors
            err = nil
        }
        return ctrl.Result{}, err
    }
    
    // Establish the parent-child relationship between my resource and the deployment
    log.Info("Making my resource a parent of the deployment")
    if err = controllerutil.SetControllerReference(&myResource, deployment, r.Scheme); err != nil {
        log.Error(err, "Failed to set deployment controller reference")
        return ctrl.Result{}, err
    }
    
    

    To fix it, I needed to swap the order of the call to r.Create and controllerutil.SetControllerReference:

    Working code

    log.Info("Creating a deployment")
    
    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      deploymentName,
            Namespace: myResource.Namespace,
        },
        Spec: deploymentSpec,
    }
    
    // Establish the parent-child relationship between my resource and the deployment
    log.Info("Making my resource a parent of the deployment")
    if err = controllerutil.SetControllerReference(&myResource, deployment, r.Scheme); err != nil {
        log.Error(err, "Failed to set deployment controller reference")
        return ctrl.Result{}, err
    }
    
    // Create the deployment with the parent/child relationship configured
    if err = r.Create(ctx, deployment); err != nil {
        log.Error(err, "Failed to create deployment")
        if errors.IsInvalid(err) {
            // Don't retry on validation errors
            err = nil
        }
        return ctrl.Result{}, err
    }
    

    I was able to confirm that this worked by looking at the metadata.ownerReferences YAML data for my created deployment (using the command referenced above).

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "1"
      creationTimestamp: "2021-08-02T16:22:04Z"
      generation: 1
      name: sample-deployment
      namespace: default
      ownerReferences:
      - apiVersion: resources.mydomain.io/v1alpha1
        blockOwnerDeletion: true
        controller: true
        kind: MyResource
        name: myresource-sample
        uid: 6ebb146c-afc7-4601-bd75-58efc29beac9
      resourceVersion: "569913"
      uid: d9a4496f-7418-4831-ab87-4804dcd1f8aa