Kubernetes informer fails with Unauthorized

I'm trying to construct a Kubernetes informer outside of the EKS cluster that it's watching. I'm using aws-iam-authenticator plugin to provide the exec-based credentials to the EKS cluster. For the plugin to work, I'm assuming an IAM role and passing the AWS IAM credentials as environment variables.

The problem is that these credentials expire after an hour and cause the informer to fail with

E0301 23:34:22.167817 582 runtime.go:79] Observed a panic: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"the server has asked for the client to provide credentials (get pods)", Reason:"Unauthorized", Details:(*v1.StatusDetails)(0xc0005b0300), Code:401}} (the server has asked for the client to provide credentials (get pods))

Is there a better way of getting ClientConfig and aws-iam-authenticator to refresh the credentials?

Here's a rough skeleton of my code:

credentialsProvider := aws.NewCredentialsCache(stscreds.NewWebIdentityRoleProvider(...))
creds, err := credentialsProvider.Retrieve(ctx)

config := clientcmdapi.NewConfig()
// ...
config.AuthInfos["eks"] = &clientcmdapi.AuthInfo{
    Exec: &clientcmdapi.ExecConfig{
        Command: "aws-iam-authenticator",
        Args: []string{
            "token",
            "-i",
            clusterName,
        },
        // These env vars are static! :(
        Env: []clientcmdapi.ExecEnvVar{
            {
                Name:  "AWS_ACCESS_KEY_ID",
                Value: creds.AccessKeyID,
            },
            {
                Name:  "AWS_SECRET_ACCESS_KEY",
                Value: creds.SecretAccessKey,
            },
            {
                Name:  "AWS_SESSION_TOKEN",
                Value: creds.SessionToken,
            },
        },
        APIVersion:      "client.authentication.k8s.io/v1beta1",
        InteractiveMode: clientcmdapi.NeverExecInteractiveMode,
    },
}

restConfig, err := config.ClientConfig()
clientset, err = kubernetes.NewForConfig(restConfig)

informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
podInformer := cw.informerFactory.Core().V1().Pods().Informer()

Here are a couple similar threads I found:

• Kubernetes client-go informers getting "Unauthorized" error after 15 mins
• https://github.com/kubernetes/client-go/issues/1189

My solution was to create write the credentials to a file and create a background thread to refresh that file. I can then pass tell aws-iam-authenticator to read the credentials from the file via the AWS_SHARED_CREDENTIALS_FILE environment variable.

This might also be possible using AWS_WEB_IDENTITY_TOKEN_FILE to save some steps, but I didn't look further.

The updated code looks like this

func updateCredentials(ctx context.Context) {
    creds, err := c.credentialsProvider.Retrieve(ctx)
    s := fmt.Sprintf(`[default]
aws_access_key_id=%s
aws_secret_access_key=%s
aws_session_token=%s`, creds.AccessKeyID, creds.SecretAccessKey, creds.SessionToken)
    err = os.WriteFile(credentialsFile.Name(), []byte(s), 0666)
    return nil
}

func updateCredentialsLoop(ctx context.Context) {
    for {
        err := updateCredentials(ctx)
        time.Sleep(5*time.Minute)
    }
}

credentialsProvider := aws.NewCredentialsCache(stscreds.NewWebIdentityRoleProvider(...))

credentialsFile, err := os.CreateTemp("", "credentials")
updateCredentials(ctx)
go updateCredentialsLoop(ctx)

config := clientcmdapi.NewConfig()
// ...
config.AuthInfos["eks"] = &clientcmdapi.AuthInfo{
    Exec: &clientcmdapi.ExecConfig{
        Command: "aws-iam-authenticator",
        Args: []string{
            "token",
            "-i",
            clusterName,
        },
        Env: []clientcmdapi.ExecEnvVar{
            {
                Name:  "AWS_SHARED_CREDENTIALS_FILE",
                Value: credentialsFile.Name(),
            },
        },
        APIVersion:      "client.authentication.k8s.io/v1beta1",
        InteractiveMode: clientcmdapi.NeverExecInteractiveMode,
    },
}

restConfig, err := config.ClientConfig()
clientset, err = kubernetes.NewForConfig(restConfig)

informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
podInformer := cw.informerFactory.Core().V1().Pods().Informer()