502 Bad Gateway on Azure Ingress for Kubernetes Service /microservice1/hello (Quarkus Application)

I have a Kubernetes cluster running on Azure with an Ingress configured for multiple services. One of these services, microservice1, is a Quarkus application. Locally, the /microservice1/hello endpoint works perfectly. However, when accessing it through the Azure ingress, I get a 502 Bad Gateway error.

Other services in the cluster include:

quarkus-service accessible at /qs react-service accessible at /web

Ingress Configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/acme-challenge-type: http01
    cert-manager.io/issuer: letsencrypt-prod
    kubernetes.io/ingress.class: azure/application-gateway
  name: simple-frontend-ingress
  namespace: dev
spec:
  rules:
    - host: example.dev.company.com
      http:
        paths:
          - path: /microservice1/
            pathType: Prefix
            backend:
              service:
                name: microservice1-service
                port:
                  number: 80
          - path: /qs
            pathType: Prefix
            backend:
              service:
                name: quarkus-service
                port:
                  number: 80
          - path: /web
            pathType: Prefix
            backend:
              service:
                name: react-service
                port:
                  number: 80
  tls:
    - hosts:
        - example.dev.company.com
      secretName: frontend-tls-prod

Service Configuration:

apiVersion: v1
kind: Service
metadata:
  name: microservice1-service
spec:
  selector:
    app: microservice1
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: ClusterIP

Deployment Configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: microservice1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: microservice1
  template:
    metadata:
      labels:
        app: microservice1
    spec:
      containers:
        - name: microservice1
          image: myregistry.azurecr.io/microservice1:latest
          ports:
            - containerPort: 8080
          env:
            - name: quarkus.http.root-path
              value: /microservice1/

Quarkus application.properties

# Quarkus HTTP Configuration
quarkus.http.host=0.0.0.0
quarkus.http.port=8080
quarkus.http.root-path=/microservice1/

# OIDC Configuration
quarkus.oidc.provider=microsoft
quarkus.oidc.client-id=YOUR_CLIENT_ID
quarkus.oidc.credentials.secret=YOUR_SECRET
quarkus.oidc.auth-server-url=https://example.b2clogin.com/example.onmicrosoft.com/B2C_1_susi/v2.0/

# Logging Configuration
quarkus.log.category."io.quarkus.oidc".min-level=TRACE
quarkus.log.category."io.quarkus.oidc".level=TRACE

# Microservices
quarkus.rest-client.microservice2.url=${MICROSERVICE2}
quarkus.rest-client.microservice3.url=${MICROSERVICE3}

# Database Configuration
quarkus.datasource.db-kind=postgresql
quarkus.datasource.jdbc.url=${DB_URL}
quarkus.datasource.username=${DB_USERNAME}
quarkus.datasource.password=${DB_PASSWORD}
quarkus.datasource.jdbc.max-size=20
quarkus.datasource.jdbc.min-size=2
quarkus.hibernate-orm.database.generation=update

EDIT: added dockerfile

FROM registry.access.redhat.com/ubi8/openjdk-17:1.19

ENV LANGUAGE='en_US:en'


# We make four distinct layers so if there are application changes the library layers can be re-used
COPY --chown=185 build/quarkus-app/lib/ /deployments/lib/
COPY --chown=185 build/quarkus-app/*.jar /deployments/
COPY --chown=185 build/quarkus-app/app/ /deployments/app/
COPY --chown=185 build/quarkus-app/quarkus/ /deployments/quarkus/

# Set environment variables in Dockerfile
ENV DB_URL=""
ENV DB_USERNAME=""
ENV DB_PASSWORD=""

EXPOSE 8080
USER 185
ENV JAVA_OPTS_APPEND="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"

ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ]

Observations:

1. Local testing works fine when pointing to http://localhost:8080/microservice1/hello.
2. The ingress should be correct since i have another quarkus and react application running.
3. Other services (quarkus-service on /qs and react-service on /web are deployed and are working on the suburl

Any insights or troubleshooting tips would be greatly appreciated!

the problem was in the backend settings on azure. the custom probe was not set to the specified service