kubernetesaws-api-gatewayambassador

Ambassador Gives 403 Trying to Connect to Websocket

Ambassador is throwing a 403 while trying to access a websocket endpoint within the cluster. Here are the steps to recreate the issue:

[Kube Environment: Docker for Desktop on Mac]

Install Ambassador and the Echo Service

  1. Deploy Ambassador with Helm
helm repo add datawire https://getambassador.io
helm install ambassador datawire/ambassador
  1. Deploy the websocket echo deployment/service/mapping
---
apiVersion: v1
kind: Service
metadata:
  name: websocket-echo
  namespace: default
spec:
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app: websocket-echo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: websocket-echo
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: websocket-echo
  template:
    metadata:
      labels:
        app: websocket-echo
    spec:
      containers:
        - name: backend
          image: jmalloc/echo-server
          ports:
            - name: http
              containerPort: 8080
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: websocket-echo
  namespace: default
spec:
  prefix: /websocket/
  service: websocket-echo
  allow_upgrade:
  - websocket

✅ Verify the echo server can be reached from a pod within the cluster

  1. Create a node shell on the cluster
kubectl run my-shell --rm -i --tty --image node:12 -- bash
  1. In the shell, download wscat
npm i -g wscat
  1. In the shell, connect to the service via the cluster service endpoint
wscat -c websocket-echo.default.svc.cluster.local
  1. Verify the connection is established and messages can be echoed back

❌ Verify the echo server cannot be reach from outside the cluster

  1. Install wscat on you local machine
npm i -g wscat
  1. Using wscat, connect to the service using the mapping
wscat -c localhost/websocket-echo/
  1. Verify that the error returned is error: Unexpected server response: 403

Am I missing something simple?

Solution

  • Ambassador will redirect incoming request froms http to https.

    You can verify by

    $ curl localhost/websocket/ -v
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 80 (#0)
> GET /websocket/ HTTP/1.1
> Host: localhost
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< location: https://localhost/websocket/
< date: Fri, 07 Aug 2020 08:25:38 GMT
< server: envoy
< content-length: 0
< 
* Connection #0 to host localhost left intact

    So, you will need to add this config to disable redirection on localhost

    apiVersion: getambassador.io/v2
kind: Host
metadata:
  name: localhost
  namespace: default
spec:
  hostname: localhost
  acmeProvider:
    authority: none
  requestPolicy:
    insecure:
      action: Route

    Reference