apache-sparkjettyembedded-jettyjetty-9spark-ui

How to increase Jetty's header buffer size in the Spark UI reverse proxy


I'm getting "HTTP ERROR 502 Bad Gateway" when I click on a worker link in my standalone Spark UI. Looking at the master logs I can see a corresponding message...

HttpSenderOverHTTP.java:219 Generated headers (4096 bytes), chunk (-1 bytes), content (0 bytes) - HEADER_OVERFLOW/HttpGenerator@231f022d{s=START}

The network infrastructure in front of my Spark UI does indeed generate a header that is bigger than 4096 bytes, and the Spark reverse proxy is attempting to pass that to the worker UI. If I bypass that infrastructure the UI works as it should.

After digging into the Spark UI code I believe that the requestBufferSize init parameter of the Jetty ProxyServlet controls this.

Can this be increased at run-time via (say) a Java property? For example, something like...

SPARK_MASTER_OPTS=-Dorg.eclipse.jetty.proxy.ProxyServlet.requestBufferSize=8192 ...

I've tried the above without success -- I'm not familiar enough with Jetty or Servlets in general to know if that's even close to valid. Obviously I'm also looking into ways of reducing the header size but that involves systems that I have much less control over.

(Spark v3.0.2 / Jetty 9.4)


Solution

  • Here's the workaround that I was forced to use -- Putting a proxy in front of the Spark UI that strips the headers.

    I used NGINX with this in the default.conf...

    server { 
      listen 8080;
      location / {
        proxy_pass http://my-spark-master:8080/;
        proxy_pass_request_headers off;
        proxy_redirect off;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
    }