istioazure-aksistio-sidecaristio-gatewayistio-operator

how to upgrade Istio in AKS from version 1.7 to 1.8

I am very new to ISTIO and would like to get clarified with my following doubts.

Details

Current AKS version 1.18.14
planning upgrade to AKS 1.19.11
Current istio version 1.7
Planning upgrade to 1.8

We are planning to upgrade Istio version from 1.7 to 1.8 in our AKS cluster 1.18.14 in our production.

But I am not sure about the proper method of upgrade to follow in production since there are multiple methods are given by Istio.

I don't have any clue about the how the current Istio setup is done in my environment and what profile settings we used as it was done long before. Could understand below are the steps followed to install istio..

Istio was installed following way:

  1. Created manifest:

    istioctl manifest generate --set profile=default -f /manifests/overlay/overlay.yaml > $HOME/generated-manifest.yaml

  2. Installed istio:

    istioctl install --set profile=default -f /manifests/overlay/overlay.yaml

  3. Verified istio against the deployed manifest:

    istioctl verify-install -f $HOME/generated-manifest.yaml

Is there any method to export all the existing settings (the one currently running) and do the upgrade?

So I am looking for a production ready approach to upgrade Istio with all existing settings in placed.

Solution

  • Important

    Consider replicating the environment and performing the upgrade on dev/stage first to make sure it works for you and your infrastructure.

    Check what exactly you have installed

    Can be done by getting installed state custom resource and all settings:

    kubectl -n istio-system get IstioOperator installed-state -o yaml > installed-state.yaml

    Below are steps based on official documentation to upgrade using istioctl

    From 1.7.3 to 1.8.6, this will be similar for other versions, however upgrades should be no more than 1 minor version of difference e.g. 1.5 to 1.6.

    Available versions and releases can be checked in Istio Github.

    1 - install istioctl version 1.8.6: Get required binaries:

    curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.8.6 TARGET_ARCH=x86_64 sh -

    and copy istiolctl bin:

    sudo cp bin/istioctl /usr/local/bin/

    2 - run istioctl version to confirm istioctl version and control/data plane versions:

    client version: 1.8.6
control plane version: 1.7.3
data plane version: 1.7.3 (3 proxies)

    3 - run istioctl x precheck to see if revision was set (it may be different if set - see warning at the end of section)

    Istio Revision "" already installed in namespace "istio-system"

    There are two main upgrade strategies:

    Vendor suggests to go with canary as it's more safe and can be tested before final migration.

    4 - Create a backup:

    kubectl get istio-io --all-namespaces -oyaml >  "$HOME"/istio_resource_backup.yaml

    Can be restored with:

    kubectl apply -f "$HOME"/istio_resource_backup.yaml

    5 - Control plane - install Canary version

     istioctl install --set profile=default --set revision=1-8-6

    Check it installed successfully by running following commands:

    kubectl get pods -n istio-system -l app=istiod

    NAME                           READY   STATUS    RESTARTS   AGE
istiod-1-8-6-b855c557b-qq4qd   1/1     Running   0          44s
istiod-54b46bbc58-wzklh        1/1     Running   0          14m

    kubectl get svc -n istio-system -l app=istiod

    NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                         AGE
istiod         ClusterIP   10.109.24.78    <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP,853/TCP   15m
istiod-1-8-6   ClusterIP   10.101.241.85   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP           2m12s

    kubectl get mutatingwebhookconfigurations

    NAME                           WEBHOOKS   AGE
istio-sidecar-injector         1          15m
istio-sidecar-injector-1-8-6   1          98s

    6 - Data plane

    Unlike istiod, Istio gateways do not run revision-specific instances, but are instead in-place upgraded to use the new control plane revision. You can verify that the istio-ingress gateway is using the canary revision by running the following command:

    istioctl proxy-status | grep $(kubectl -n istio-system get pod -l app=istio-ingressgateway -o jsonpath='{.items..metadata.name}') | awk '{print $7}'

    To upgrade the namespace NAME_SPACE, remove the istio-injection label, and add the istio.io/rev label to point to the canary revision. The istio-injection label must be removed because it takes precedence over the istio.io/rev label for backward compatibility:

    kubectl label namespace NAME_SPACE istio-injection- istio.io/rev=1-8-6

    Once namespace(s) updated, pods need to be re-injected. This can be done by restarting them, e.g. with:

    kubectl rollout restart deployment -n NAME_SPACE

    Verify pods are now using canary istiod:

    istioctl proxy-status

    NAME                                                   CDS        LDS        EDS        RDS          ISTIOD                           VERSION
istio-ingressgateway-6664d4b478-j7vhh.istio-system     SYNCED     SYNCED     SYNCED     NOT SENT     istiod-1-8-6-b855c557b-qq4qd     1.8.6
nginx-68748d7f8c-82x8k.default                         SYNCED     SYNCED     SYNCED     SYNCED       istiod-1-8-6-b855c557b-qq4qd     1.8.6
nginx-68748d7f8c-fnhgz.default                         SYNCED     SYNCED     SYNCED     SYNCED       istiod-1-8-6-b855c557b-qq4qd     1.8.6

    7 - Uninstall old control-plane

    Run: istioctl x uninstall -f manifests/profiles/default.yaml

    Check only canary control-plane is running: kubectl get pods -n istio-system -l app=istiod

    NAME                           READY   STATUS    RESTARTS   AGE
istiod-1-8-6-b855c557b-qq4qd   1/1     Running   0          17m

    Other types of istio installation available:

    Please get familiar with istio installation methods' pros and cons.

    Useful links

    Update

    Moving this from comments. There are more challenges with updating from 1.7.3 to 1.8.6 istio versions. For removing current control-plane -f with previous manifest should be used. When applying for the same manifest to 1.8.6 versions, there are errors about policy and telemetry components:

    Error: failed to get profile and enabled components: failed to read profile: unknown field "telemetry" in v1alpha1.IstioComponentSetSpec

    After digging, it appeared, even though api version is used the same - v1alpha1, newer version of istioctl operator can't validate manifest from 1.7.3.

    I took installed-state.yaml as it's described at the beginning of the asnwer from 1.7.3 and 1.8.6 istio installations and got diff between them: policy and telemetry components are completely missing in 1.8.6 which explains the errors. Also there are some changes as well. Github link to the diff file, left is 1.7.3, right is 1.8.6.

    In that case it's probably impossible to upgrade without manual work with manifests:

    1 - check if manifest which was applied is default or has changes. Get a default profile (note! istioctl should be used 1.7.3):

    istioctl profile dump default > default-profile.yaml

    2 - If manifest is default, then safely proceed to install canary with --set profile=default.

    3 - Manifest is not default and has customization. Using istioctl 1.8.6 get a dump of default profile:

    istioctl profile dump default > default-profile-186.yaml

    "Adapt" it to current existing manifest and then proceed to install canary with -f option and adapted manifest.