I'm trying to put an Envoy proxy in front of my ConnectRPC microservices. I've got the following setup:
static_resources:
listeners:
# Main listener
- name: main
address:
socket_address:
address: 0.0.0.0
port_value: 8001
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: main_routes
virtual_hosts:
- name: all_services
domains: ["*"]
routes:
- match:
prefix: "/healthz"
direct_response:
status: 200
body:
inline_string: "OK"
typed_per_filter_config:
envoy.filters.http.ext_authz:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
disabled: true
- match:
prefix: "/grpc.health.v1.Health/"
response_headers_to_add:
- header:
key: grpc-status
value: '0'
append_action: OVERWRITE_IF_EXISTS_OR_ADD
- header:
key: grpc-message
value: 'OK'
append_action: OVERWRITE_IF_EXISTS_OR_ADD
- header:
key: content-type
value: application/grpc
append_action: OVERWRITE_IF_EXISTS_OR_ADD
direct_response:
status: 200
body:
inline_string: ""
typed_per_filter_config:
envoy.filters.http.ext_authz:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
disabled: true
- match:
prefix: "/services.userd.v1.UserService"
route:
cluster: user_service
- match:
prefix: "/services.searchd.v1.GeolocationService"
route:
cluster: search_service
http_filters:
- name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
http_service:
server_uri:
uri: http://authd.api.local
cluster: auth_service
timeout: 0.25s
path_prefix: "/services.authd.v2.AuthService/Authorize"
include_tls_session: true
encode_raw_headers: false
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
# Auth service
- name: auth_service
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
load_assignment:
cluster_name: auth_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: authd.api.local
port_value: 8001
I've got this sitting behind an ALB that forwards port 443 to port 8001 and I can test that https://myurl.ext/healthz and https://myurl.ext/grpc.health.v1.Health/Check return a 200 response with the expected headers. The issue I'm having is that any other endpoint I request is returning a 403 and my authd service logs aren't showing any requests, which they would be if the requests had been routed as expected. Nor do I see any logs of requests in Envoy (although I'm not sure if I should).
For reference, I've verified that authd is also listening on port 8001 and I don't think it's a security group issue because those normally result in timeouts and these requests are not timing out. What am I doing wrong here?
UPDATE I'm unable to get Envoy to work correctly using
http_service. I added a debugging HTTP endpoint. Regardless of the (ignored)uri, Envoy invokesPOST /services.userd.v1.UserService/GetUser(incorrectly?) against the Auth endpoint when configured to usehttp_service. But, when configured to usegrpc_service, it works correctly (invoking/envoy.service.auth.v3.Authorization/Check.
UPDATE This isn't working correctly. Using the Envoy Authorization service implementation as gRPC (
grpc_service) works andUserService/GetUsermethods are preceded byAuthorization/Checkmethods but, when attempting to use the Authroization service via Connect (http_service), theUserService/GetUsermethod succeeds (200) but I don't see theAuthorization/Checkmethod being invoked.
I think:
I'm not intimately familiar with Envoy but I was able to get UserService/GetUser returning 200 (OK) and am confident that External Authorization is succeeding.
UserService/GetUser:
curl \
--verbose \
--header 'Accept: application/json' \
--header 'Connect-Protocol-Version: 1' \
--header 'Connect-Timeout-Ms: ' \
--header 'Content-Type: application/json' \
--data '{}' \
--url http://127.0.0.1:8001/services.userd.v1.UserService/GetUser
* Trying 127.0.0.1:8001...
* Connected to 127.0.0.1 (127.0.0.1) port 8001
* using HTTP/1.x
> POST /services.userd.v1.UserService/GetUser HTTP/1.1
> Host: 127.0.0.1:8001
> User-Agent: curl/8.12.1
> Accept: application/json
> Connect-Protocol-Version: 1
> Content-Type: application/json
> Content-Length: 2
>
* upload completely sent off: 2 bytes
< HTTP/1.1 200 OK
< content-type: application/json
< vary: Origin
< content-length: 2
< date: Tue, 23 Sep 2025 20:27:59 GMT
< x-envoy-upstream-service-time: 0
< server: envoy
<
* Connection #0 to host 127.0.0.1 left intact
And grep'ing Envoy's stats endpoint:
curl \
--silent \
http://localhost:9901/stats \
| grep ext_authz
cluster.fauxrpc.ext_authz.ok: 2 <--------------------- Success
http.ingress_http.ext_authz.denied: 0
http.ingress_http.ext_authz.disabled: 0
http.ingress_http.ext_authz.error: 0
http.ingress_http.ext_authz.failure_mode_allowed: 0
http.ingress_http.ext_authz.filter_state_name_collision: 0
http.ingress_http.ext_authz.ignored_dynamic_metadata: 0
http.ingress_http.ext_authz.invalid: 0
http.ingress_http.ext_authz.ok: 2 <------------------- Success
I'm running Envoy with trace debugging on ext_authz component:
podman run \
--interactive --tty --rm \
--name=envoy \
--net=host \
--volume=${PWD}/envoy.yaml:/etc/envoy/envoy.yaml \
${IMAGE_ENVOY} \
--config-path /etc/envoy/envoy.yaml \
--component-log-level ext_authz:trace
And I'm using FauxRPC to simulate your (mostly undocumented) gRPC services:
fauxrpc run \
--schema=./services.binpb \
--dashboard
Where:
.
├── envoy.yaml
├── protos
│ ├── envoy
│ │ └── service
│ │ └── auth
│ │ └── v3
│ │ └── external_auth.proto
│ └── services
│ ├── authd
│ │ └── v2
│ │ └── auth_service.proto
│ ├── searchd
│ │ └── v1
│ │ └── search_service.proto
│ └── userd
│ └── v1
│ └── user_service.proto
└── services.binpb
NOTE The Envoy-specific External Authorization service (
external_auth.proto
Your services.authd.v2.AuthService is not used because it does not (appear) to match the Envoy service's required schema.
I tweaked Envoy's CheckResponse message to always return OkHttpResponse messages so that this works consistently with FauxRPC:
// Always return OkHttpResponse
message CheckResponse {
oneof http_response {
// DeniedHttpResponse denied_response = 2;
OkHttpResponse ok_response = 3;
}
}
services.binpb:
PROTOS="${PWD}/protos"
protoc \
--proto_path=${PROTOS} \
--include_imports \
--include_source_info \
--descriptor_set_out=services.binpb \
${PROTOS}/envoy/service/auth/v3/external_auth.proto \
${PROTOS}/services/authd/v2/auth_service.proto \
${PROTOS}/services/searchd/v1/search_service.proto \
${PROTOS}/services/userd/v1/user_service.proto
And the following envoy.yaml:
admin:
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
# Main listener
- name: main
address:
socket_address:
address: 0.0.0.0
port_value: 8001
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
codec_type: AUTO
route_config:
name: main_routes
virtual_hosts:
- name: all_services
domains: ["*"]
routes:
- match:
prefix: "/healthz"
direct_response:
status: 200
body:
inline_string: "OK"
typed_per_filter_config:
envoy.filters.http.ext_authz:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
disabled: true
- match:
prefix: "/grpc.health.v1.Health/"
response_headers_to_add:
- header:
key: grpc-status
value: '0'
append_action: OVERWRITE_IF_EXISTS_OR_ADD
- header:
key: grpc-message
value: 'OK'
append_action: OVERWRITE_IF_EXISTS_OR_ADD
- header:
key: content-type
value: application/grpc
append_action: OVERWRITE_IF_EXISTS_OR_ADD
direct_response:
status: 200
body:
inline_string: ""
typed_per_filter_config:
envoy.filters.http.ext_authz:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
disabled: true
# Using Envoy ExtAuthz service instead
# - match:
# prefix: "/services.authd.v2.AuthService"
# route:
# cluster: fauxrpc
# typed_per_filter_config:
# envoy.filters.http.ext_authz:
# "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
# disabled: true
- match:
prefix: "/services.userd.v1.UserService"
route:
cluster: fauxrpc
- match:
prefix: "/services.searchd.v1.GeolocationService"
route:
cluster: fauxrpc
http_filters:
- name: envoy.filters.http.ext_authz
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
# Either Connect protocol
# Requiring additional request headers
http_service:
server_uri:
uri: http://localhost:6660
cluster: fauxrpc
timeout: 0.25s
authorization_request:
headers_to_add:
- key: "Connect-Protocol-Version"
value: "1"
- key: "Accept"
value: "application/json"
- key: "Content-Type"
value: "application/json"
# Or gRPC
# grpc_service:
# envoy_grpc:
# cluster_name: fauxrpc
# timeout: 0.25s
include_tls_session: true
encode_raw_headers: false
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
# Auth service
- name: fauxrpc
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
load_assignment:
cluster_name: fauxrpc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
# localhost defaulted to IPv6 which was problematic
# Force IPv4 with IPv4 address
address: 127.0.0.1
port_value: 6660
And Envoy logging output:
[2025-09-23 20:27:55.183][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:367] [Tags: "ConnectionId":"0","StreamId":"11200526026145856258"] ext_authz filter calling authorization server.
[2025-09-23 20:27:55.187][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:652] [Tags: "ConnectionId":"0","StreamId":"11200526026145856258"] ext_authz filter added header(s) to the request:
[2025-09-23 20:27:55.187][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:730] [Tags: "ConnectionId":"0","StreamId":"11200526026145856258"] ext_authz filter removed header(s) from the request:
[2025-09-23 20:27:55.190][15][trace][ext_authz] [source/extensions/filters/http/ext_authz/ext_authz.cc:477] [Tags: "ConnectionId":"0","StreamId":"11200526026145856258"] ext_authz filter has 0 response header(s) to add and 0 response header(s) to set to the encoded response: