I am working on a k8s (AKS) Sitecore 10 XM1 deployment, and am seeing two (potentially related) issues in the k8s environment which do not occur with the same container images in Docker for windows. My assumption is that they are probably caused by the ingress controller config, TLS certificates or environment variables since these are the only real differences. I'm having difficulty diagnosing the cause as I'm fairly new to containers, the newish Sitecore identity server and k8s.
I've replaced the actual domain name with 'mydomain', and we have valid TLS certs for the actual domain deployed using certmanager.
The first issue is related to our CD pipeline not being able to login non-interactively (per https://doc.sitecore.com/developers/100/developer-tools/en/configure-a-non-interactive-client-login.html) in order to perform content serialization commands. When my deployment pipeline executes dotnet sitecore login --client-credentials true --authority ${{ parameters.idServerUrl }} --cm ${{ parameters.cmServerUrl }} --allow-write true --client-id "Pipeline_Automation" --client-secret "$(sitecore-identitysecret-pipeline)", I receive the error Keyset is missing. The necessary config files are deployed to our cm and id instances via COPY commands in the environment Dockerfiles (which I can see inside the running containers), and I am able to login via client credentials when the solution is running locally in Docker.
The second issue is that we see Invalid redirect_uri: "https://author.qa.mydomain.com/identity/signin" errors in the id pod logs when attempting to hit https://author.qa.mydomain.com/sitecore at the point where we should be redirected to the sign-in page. I've read that this is typically caused by the actual redirect URL not matching that configured by Sitecore_Sitecore__IdentityServer__Clients__DefaultClient__AllowedCorsOrigins__AllowedCorsOriginsGroup1 environment variable of the k8s spec for the id environment, however the logs indicate that they're identical:
id logs
[13:12:36] IdentityServer4.Validation.AuthorizeRequestValidator [Error] Invalid redirect_uri: "https://author.qa.mydomain.com/identity/signin"
AuthorizeRequestValidationLog {ClientId="Sitecore", ClientName="Sitecore", RedirectUri=null, AllowedRedirectUris=["{AllowedCorsOrigin}/identity/signin", "{AllowedCorsOrigin}/signin-oidc"], SubjectId="anonymous", ResponseType=null, ResponseMode=null, GrantType=null, RequestedScopes="", State=null, UiLocales=null, Nonce=null, AuthenticationContextReferenceClasses=null, DisplayMode=null, PromptMode=null, MaxAge=null, LoginHint=null, SessionId=null, Raw={["client_id"]="Sitecore", ["response_type"]="code id_token token", ["scope"]="openid sitecore.profile", ["state"]="OpenIdConnect.AuthenticationProperties=gradGeDSd71cNE43mbnFr-GuWj0q03FGaBiqGgyhXJW0-7qYt4xi5D5C360rNowTwT0PC9hZCut484hMo7WFSRsHmkseOJ5ud99RUhtxQwHrXuATCqErmKQE0m3UtPBsBhpySWkcs3J1EwkuPeELzplf_uMU7F6lyh77sUfyrQuTZf1bn05wJ2ZnEpGDXmAcSe5nCRQOt5D9hemDBZ4WzA", ["response_mode"]="form_post", ["nonce"]="637527067565470343.M2MwMDkxNGMtNzM0NC00OThkLWI0YmMtZTBiZmJhZDM2OWRmODM4YjViODgtOTdmOC00Nzc5LThhZWMtZjU2MmE2MzY5M2Mz", ["redirect_uri"]="https://author.qa.mydomain.com/identity/signin", ["sc_account_prefix"]="sitecore\\", ["x-client-SKU"]="ID_NET461", ["x-client-ver"]="5.3.0.0"}}
[13:12:36] IdentityServer4.Endpoints.AuthorizeEndpoint [Error] Request validation failed
nginx logs
XXX.XX.XXX.XX - - [31/Mar/2021:12:10:16 +0000] "POST /identity/externallogin?authenticationType=SitecoreIdentityServer&ReturnUrl=%2fidentity%2fexternallogincallback%3fReturnUrl%3d%26sc_site%3dshell%26authenticationSource%3dDefault&sc_site=shell HTTP/2.0" 302 0 "https://author.qa.mydomain.com/identity/login/shell/SitecoreIdentityServer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" 597 0.013 [default-cm-80] [] 10.240.0.83:80 0 0.012 302 435fbe68699d547286aef26f15f4d33f
XXX.XX.XXX.XX - - [31/Mar/2021:12:10:16 +0000] "GET /connect/authorize?client_id=Sitecore&response_type=code%20id_token%20token&scope=openid%20sitecore.profile&state=OpenIdConnect.AuthenticationProperties%3DD9AWnwdC9UV0fjOTg3giJdWs0rdIRftadzkgUYO7c9sGu6nsBnyVCcrXNFJw1EhiXT2bDmJrdpqy3Gf8sMX5UXU1yJmz0u2Rpud_OUTd9YMot-1_OFzTCjPZS7-lrUxBlwkHhJP-eJN5wgDPBKCohsyHpHqmI9KaETYn2p7pnfnsQMh-7nvI9_iy3KFifcHiMYo4zyMkch-FZm7SW_pWsA&response_mode=form_post&nonce=637527894166260277.ZjdmNWIyNjUtZTk2NS00NzhmLTk4MDctMjFlNmU3ZjRjNjE4MjNhMDAwYmEtNGRmYS00YTcxLThmYTYtNTIwNzkzN2M3NGYy&redirect_uri=https%3A%2F%2Fauthor.qa.mydomain.com%2Fidentity%2Fsignin&sc_account_prefix=sitecore%5C&x-client-SKU=ID_NET461&x-client-ver=5.3.0.0 HTTP/2.0" 302 0 "https://author.qa.mydomain.com/identity/login/shell/SitecoreIdentityServer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" 564 0.007 [default-id-80] [] 10.240.0.73:80 0 0.008 302 354a57aea9818b47ce853edc6abee662
XXX.XX.XXX.XX - - [31/Mar/2021:12:10:16 +0000] "GET /home/error?errorId=CfDJ8B12A-SlGFlFuiZCVhQvJ8LQrI1XT9zDZiDk__KCVmMZULmTqb8K9QN9a-lqnM_gl9hxJC3CqPMCOITToOp5QKRb3nP8A8IuiY9Z0v7ofXpvYlzYiMnbe1V03MJs7t_nHh6CS8RkfqlwK1Z1R2V31JcDb7FEXn2Ek6mwWxAonA9LuGcGQe9yhxcy3zmF60DqQkun71_fWMtrgIdIeFHo4Zn1aCYSYYQpuLY84lOB-aiFNdxgS48gV1lASfUshgdBrVTD3y6KuMMdUylKsoOG4JXQRUh5aT67f6J2WCWWwzYiDQ5sos1N7Lxp_Pq7iZq5KQ HTTP/2.0" 404 0 "https://author.qa.mydomain.com/identity/login/shell/SitecoreIdentityServer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0" 295 0.003 [default-id-80] [] 10.240.0.73:80 0 0.000 404 fd27666878ee5cf4affb008674edc693
I believe the issues are related as the identity server behaves normally when the solution runs locally behind its traefik ingress. I am able to login, do not experience the redirect error and can login non-interactively.
I'd like to try using a cli browser such as lynx running in a pod within the cluster to see if I can reproduce the redirect error in order to narrow the cause to the ingress controller, but have not found an easy way to do this.
If anyone could provide some diagnostic advice, I'd be most grateful.
id.yaml k8s spec
apiVersion: v1
kind: Service
metadata:
name: id
spec:
selector:
app: id
ports:
- protocol: TCP
port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: id
labels:
app: id
spec:
replicas: 1
selector:
matchLabels:
app: id
template:
metadata:
labels:
app: id
spec:
nodeSelector:
kubernetes.io/os: windows
containers:
- name: sitecore-xm1-id
image: mydomain.azurecr.io/cms/mydomain-id:latest
ports:
- containerPort: 80
imagePullPolicy: Always
env:
- name: Database_Server
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-databaseservername.txt
- name: Core_Database_Username
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-core-database-username.txt
- name: Core_Database_Password
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-core-database-password.txt
- name: Sitecore_Sitecore__IdentityServer__Clients__PasswordClient__ClientSecrets__ClientSecret1
valueFrom:
secretKeyRef:
name: sitecore-identity
key: sitecore-identitysecret.txt
- name: Sitecore_Sitecore__IdentityServer__CertificateRawData
valueFrom:
secretKeyRef:
name: sitecore-identitycertificate
key: sitecore-identitycertificate.txt
- name: Sitecore_Sitecore__IdentityServer__CertificateRawDataPassword
valueFrom:
secretKeyRef:
name: sitecore-identitycertificate
key: sitecore-identitycertificatepassword.txt
- name: Sitecore_License
valueFrom:
secretKeyRef:
name: sitecore-license
key: sitecore-license.txt
- name: Sitecore_Sitecore__IdentityServer__Clients__CliServerClient__ClientSecrets__ClientSecret1
valueFrom:
secretKeyRef:
name: sitecore-identity
key: sitecore-identitysecret-pipeline.txt
- name: Sitecore_Sitecore__IdentityServer__SitecoreMemberShipOptions__ConnectionString
value: Data Source=$(Database_Server);Initial Catalog=Sitecore.Core;User ID=$(Core_Database_Username);Password=$(Core_Database_Password);
- name: Sitecore_Sitecore__IdentityServer__AccountOptions__PasswordRecoveryUrl
value: https://author.qa.mydomain.com/sitecore/login?rc=1
- name: Sitecore_Sitecore__IdentityServer__Clients__DefaultClient__AllowedCorsOrigins__AllowedCorsOriginsGroup1
value: https://author.qa.mydomain.com
- name: Sitecore_Sitecore__IdentityServer__PublicOrigin
value: https://id.qa.mydomain.com
livenessProbe:
httpGet:
path: /healthz/live
port: 80
httpHeaders:
- name: X-Kubernetes-Probe
value: Liveness
timeoutSeconds: 300
periodSeconds: 30
failureThreshold: 3
startupProbe:
httpGet:
path: /healthz/ready
port: 80
httpHeaders:
- name: X-Kubernetes-Probe
value: Startup
timeoutSeconds: 300
periodSeconds: 30
failureThreshold: 10
imagePullSecrets:
- name: acr-secret
cd.yaml k8s spec
apiVersion: v1
kind: Service
metadata:
name: cm
spec:
selector:
app: cm
ports:
- protocol: TCP
port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cm
labels:
app: cm
spec:
replicas: 1
selector:
matchLabels:
app: cm
template:
metadata:
labels:
app: cm
spec:
nodeSelector:
kubernetes.io/os: windows
containers:
- name: sitecore-xm1-cm
image: mydomain.azurecr.io/cms/mydomain-xm1-cm:latest
ports:
- containerPort: 80
imagePullPolicy: Always
env:
- name: Sitecore_InstanceName
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: Database_Server
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-databaseservername.txt
- name: Master_Database_Username
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-master-database-username.txt
- name: Master_Database_Password
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-master-database-password.txt
- name: Core_Database_Username
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-core-database-username.txt
- name: Core_Database_Password
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-core-database-password.txt
- name: Web_Database_Username
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-web-database-username.txt
- name: Web_Database_Password
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-web-database-password.txt
- name: Forms_Database_Username
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-forms-database-username.txt
- name: Forms_Database_Password
valueFrom:
secretKeyRef:
name: sitecore-database
key: sitecore-forms-database-password.txt
- name: Sitecore_ConnectionStrings_Sitecoreidentity.secret
valueFrom:
secretKeyRef:
name: sitecore-identity
key: sitecore-identitysecret.txt
- name: Sitecore_AppSettings_Telerik.AsyncUpload.ConfigurationEncryptionKey
valueFrom:
secretKeyRef:
name: sitecore-telerik
key: sitecore-telerikencryptionkey.txt
- name: Sitecore_AppSettings_Telerik.Upload.ConfigurationHashKey
valueFrom:
secretKeyRef:
name: sitecore-telerik
key: sitecore-telerikencryptionkey.txt
- name: Sitecore_AppSettings_Telerik.Web.UI.DialogParametersEncryptionKey
valueFrom:
secretKeyRef:
name: sitecore-telerik
key: sitecore-telerikencryptionkey.txt
- name: Sitecore_License
valueFrom:
secretKeyRef:
name: sitecore-license
key: sitecore-license.txt
- name: Sitecore_ConnectionStrings_Core
value: Data Source=$(Database_Server);Initial Catalog=Sitecore.Core;User ID=$(Core_Database_Username);Password=$(Core_Database_Password);
- name: Sitecore_ConnectionStrings_Security
value: Data Source=$(Database_Server);Initial Catalog=Sitecore.Core;User ID=$(Core_Database_Username);Password=$(Core_Database_Password);
- name: Sitecore_ConnectionStrings_Master
value: Data Source=$(Database_Server);Initial Catalog=Sitecore.Master;User ID=$(Master_Database_Username);Password=$(Master_Database_Password);
- name: Sitecore_ConnectionStrings_Web
value: Data Source=$(Database_Server);Initial Catalog=Sitecore.Web;User ID=$(Web_Database_Username);Password=$(Web_Database_Password);
- name: Sitecore_ConnectionStrings_ExperienceForms
value: Data Source=$(Database_Server);Initial Catalog=Sitecore.ExperienceForms;User ID=$(Forms_Database_Username);Password=$(Forms_Database_Password);
- name: Sitecore_ConnectionStrings_Solr.Search
valueFrom:
secretKeyRef:
name: sitecore-solr
key: sitecore-solr-connection-string.txt
- name: Sitecore_Identity_Server_Authority
value: https://id.qa.mydomain.com
- name: Sitecore_Identity_Server_CallbackAuthority
value: https://author.qa.mydomain.com
- name: Sitecore_Identity_Server_InternalAuthority
value: http://id
- name: Sitecore_Identity_Server_Require_Https
value: "false"
- name: SOLR_CORE_PREFIX_NAME
valueFrom:
secretKeyRef:
name: sitecore-solr
key: sitecore-solr-core-prefix-name.txt
livenessProbe:
httpGet:
path: /healthz/live
port: 80
httpHeaders:
- name: X-Kubernetes-Probe
value: Liveness
timeoutSeconds: 300
periodSeconds: 30
failureThreshold: 3
startupProbe:
httpGet:
path: /healthz/ready
port: 80
httpHeaders:
- name: X-Kubernetes-Probe
value: Startup
timeoutSeconds: 300
periodSeconds: 30
failureThreshold: 10
imagePullSecrets:
- name: acr-secret
We resolved this by changing the version of sitecore cli we were using in our deployment pipeline. The powershell script which allows for login and serialization/publishing is:
# Add nuget source & install Sitecore CLI
Write-Host "Installing Sitecore CLI"
dotnet nuget add source ${{ parameters.sitecoreNugetFeed }} --name "Sitecore-Public-Nuget-Feed"
dotnet tool install --version 2.0.0 Sitecore.CLI
# Login to ID Server
Write-Host "Logging into ID Server ${{ parameters.idServerUrl }} for environment ${{ parameters.cmServerUrl }}"
dotnet sitecore login --client-credentials true --authority ${{ parameters.idServerUrl }} --cm ${{ parameters.cmServerUrl }} --allow-write true --client-id "Pipeline_Automation" --client-secret "$(sitecore-identitysecret-pipeline)"
# Deserialize Content
Write-Host "Push Content"
dotnet sitecore ser push
# Publish Database
Write-Host "Publish Database"
dotnet sitecore publish