Expose Elastic APM through Ingress Controller

I have deployed elastic APM server into kubernetes and was trying to expose it through nginx ingress controller. Following is my configuration:

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: elastic
  name: apm-server-config
  labels:
    k8s-app: apm-server
data:
  apm-server.yml: |-
    apm-server:
      host: "0.0.0.0:8200"
    setup.kibana:
      enabled: "true"
      host: "kibana:5601"
    output.elasticsearch:
      hosts: ["elastic:9200"]
---
#Deployment Configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: apm-server
    env: msprod
    state: common
  name: apm-server
  namespace: elastic
spec:
  replicas: 1
  minReadySeconds: 10
  selector:
    matchLabels:
      app: apm-server
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: apm-server
    spec:
      containers:
        - image: docker.elastic.co/apm/apm-server:7.12.1
          imagePullPolicy: Always
          env:
            - name: output.elasticsearch.hosts
              value: "http://elastic:9200"
          name: apm-server              
          ports:
          - name: liveness-port
            containerPort: 8200
          volumeMounts:
          - name: apm-server-config
            mountPath: /usr/share/apm-server/apm-server.yml
            readOnly: true
            subPath: apm-server.yml
          resources:
            limits:
              cpu: 250m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 250Mi
      volumes:
      - name: apm-server-config
        configMap:
          name: apm-server-config
      nodeSelector:
        env: prod
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
#Service Configuration
apiVersion: v1
kind: Service
metadata:
  labels:
    app: apm-server
  name: apm-server
  namespace: elastic
spec:
  ports:
  - port: 8200
    targetPort: 8200
    name: http
    nodePort: 31000
  selector:
    app: apm-server
  sessionAffinity: None
  type: NodePort
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: elastic
  name: gateway-ingress-apm
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
    - host: my.domain.com
      http: 
        paths:
          - path: /apm
            backend:
              serviceName: apm-server
              servicePort: 8200

The pod is running and I am able to hit APM server using kubectl port-forward.

But when I am accessing the apm server with https://my.domain.com/apm then I am getting page not found error in browser and following error in APM pod:

{"log.level":"error","@timestamp":"2021-10-21T06:22:00.198Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":60},"message":"404 page not found","url.original":"/apm","http.request.method":"GET","user_agent.original":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36","source.address":"10.148.7.7","http.request.body.bytes":0,"http.request.id":"9294124a-5356-4b2c-ba8e-c0a589b23571","event.duration":110881,"http.response.status_code":404,"error.message":"404 page not found","ecs.version":"1.6.0"}

The error is coming because there is no context path configured in APM. I have gone through the APM documentation and couldn't find a way to configure context path in the apm server. Please help.

Posting this as answer out of comments.

Initial ingress rule passes the same path /apm to the APM service, which is confirmed by error in APM pod's logs - "message":"404 page not found","url.original":"/apm"

To fix it, nginx ingress has rewrite annotation. The way it works is described in the link with example.

Final ingress.yaml should look like:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  namespace: elastic
  name: gateway-ingress-apm
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2 # adding captured group
spec:
  rules:
    - host: my.domain.com
      http: 
        paths:
          - path: /apm(/|$)(.*) # to have captured group works correctly
            backend:
              serviceName: apm-server
              servicePort: 8200

What happens here is requests sent to my.domain.com/apm goes to the service on / path.

Captured group allows to preserve correct paths, for instance if the request goes to my.domain.com/apm/something, ingress will translate it to /something which will be passed to the service.