Spring Cloud Gateway with Kubernetes discovery client for http and grpc requests

I'm trying to implement gateway service using Spring Cloud Gateway with k8s client dicovery that will redirect both http/1.1 and http/2(GRPC) requests

I set the following configurations:

server:
  port: 9995
  shutdown: graceful
  http2:
    enabled: true

logging:
  level:
    org.springframework.cloud.gateway: TRACE

spring:
  application:
    name: api-gateway
  lifecycle:
    timeout-per-shutdown-phase: 30s
  cloud:
    gateway:
      discovery:
        locator:
          enabled: true
          include-expression: "metadata['gateway.enabled']=='true'"
          predicates:
            - name: Path
              args:
                pattern: "'/'+serviceId+'/**'"
          filters:
            - name: RewritePathAndPort
              args:
                grpcPort: 9090
                httpPort: 8080
    kubernetes:
      discovery:
        enabled: true
        primary-port-name: app-port
      loadbalancer:
        enabled: true
        mode: service
#        port-name: grpc-port

management:
  endpoint:
    health:
      enabled: true
      show-details: always
    gateway:
      enabled: true
  endpoints:
    web:
      exposure:
        include: '*'

I alsom implemented the custom filter RewritePathAndPort:

@Component
class RewritePathAndPortGatewayFilterFactory :
  AbstractGatewayFilterFactory<RewritePathAndPortGatewayFilterFactory.Config>(Config::class.java) {
  val log = KotlinLogging.logger {}

  class Config {
    var grpcPort: Int = 9090
    var httpPort: Int = 8080
  }

  override fun apply(config: Config): GatewayFilter =
    GatewayFilter { exchange, chain ->
      val req = exchange.request
      ServerWebExchangeUtils.addOriginalRequestUrl(exchange, req.uri)
      val path = req.uri.rawPath

      val pathSegments = path.split("/")
      log.info { "pathSegments: $pathSegments" }
      val host = pathSegments[1]
      log.info { "new host: $host" }

      val (newPath, newPort) =
        if (path.contains("/grpc/")) {
          stripPathSegments(pathSegments = pathSegments, segmentsToStrip = 3) to config.grpcPort
        } else {
          path to config.httpPort
        }

      log.info { "new port: $newPort" }

      log.info { "new path: $newPath" }

      val newUri =
        UriComponentsBuilder
          .fromUri(req.uri)
          .scheme("lb")
          .host(host)
          .port(newPort) // Set the new port
          .replacePath(newPath) // Set the new path
          .build(true) // true for encoding the path
          .toUri()
      log.info { "new Uri: $newUri" }
      val request =
        req
          .mutate()
          .uri(newUri)
          .path(newPath)
          .build()

      log.info { "new request: $request" }

      exchange.attributes[ServerWebExchangeUtils.GATEWAY_REQUEST_URL_ATTR] = request.uri
      log.info { "new exchange: $exchange" }
      chain.filter(exchange.mutate().request(request).build())
    }

  override fun shortcutFieldOrder(): List<String> = listOf("regexp", "replacement", "port")

  private fun stripPathSegments(
    pathSegments: List<String>,
    segmentsToStrip: Int,
  ): String =
    pathSegments
      .subList(segmentsToStrip, pathSegments.size)
      .joinToString("/")
      .let { "/$it" }
}

explination:

if it's an http request:

• override the host by the first segment of the path
• leave the path as is
• use port 8080

For example request with path: /other-service/api/v1/hello the uri will be lb://other-service:8080/other-service/api/v1/hello

if it's a grpc request:

• override the host by the first segment of the path
• strip the first 2 segments from the path
• use port 9090

For example: /other-grpc-service/grpc/HelloService/sayHello the uri will be lb://other-grpc-service:9090/HelloService/sayHello

With the configuration above, http requests are working and grpc don't (fail on Connection prematurely closed BEFORE response) because the load balancer uses the default http port configured on the http service which is 8080

if I uncomment port-name: grpc-port which tells the load balancer to use port grpc-port, the grpc requests are working but http requests fail on same error. This happens because the load balancer overrides the uri port with grpc-port that configured on the grpc service which 9090

What causes this is: ReactiveLoadBalancerClientFilter.

Is there any way to work around this?

edit: I found a workaround after hours of tries. I basically copies ReactiveLoadBalancerClientFilter and changed the port according to the content-type header and it worked, but this is less then ideal

I was able to solve it by using GlobalFilter and Ordered instead of AbstractGatewayFilterFactory:

@Component
class EnhanceReactiveLoadBalancerClientFilter(
  private val gatewayProperties: GatewayProperties,
) : GlobalFilter,
  Ordered {
  private val log = KotlinLogging.logger { }

  override fun filter(
    exchange: ServerWebExchange,
    chain: GatewayFilterChain,
  ): Mono<Void> {
    val attributeUri = exchange.attributes[GATEWAY_REQUEST_URL_ATTR] as URI
    val originalPath = exchange.request.path.value()
    val newPath = originalPath.substringAfter(GRPC_PATH_INDICATOR)

    val newPort =
      if (originalPath == newPath) {
        gatewayProperties.ports.http
      } else {
        gatewayProperties.ports.grpc
      }

    val newAttributeUri =
      UriComponentsBuilder
        .fromUri(attributeUri)
        .port(newPort)
        .replacePath(newPath)
        .build()
        .toUri()

    log.debug { "reconstructed new attribute uri: $newAttributeUri" }
    exchange.attributes[GATEWAY_REQUEST_URL_ATTR] = newAttributeUri

    return chain.filter(exchange)
  }

  override fun getOrder(): Int = ReactiveLoadBalancerClientFilter.LOAD_BALANCER_CLIENT_FILTER_ORDER + 3
}

explanation: I couldn't find a way to order RewritePathAndPortGatewayFilterFactory after ReactiveLoadBalancerClientFilter by creating an ordered global filter, I was able to set it right after it and also change the port and path the way since ReactiveLoadBalancerClientFilter takes the port from the k8s service itself.

I still wish I could to do it with AbstractGatewayFilterFactory since it feels like the right solution. but the above is good enough

edit: changing only GATEWAY_REQUEST_URL_ATTR attribute and not the request uri is working because spring gateway's filter that responsible for actually sending the request (/org/springframework/cloud/gateway/filter/NettyRoutingFilter.java) is looking at this attriubte and not the request URI