.netsslcaddycaddyfile

.NET 8 api with Caddy reverse proxy not working for SSL


I'm trying to set up SSL for a domain that is pointing to a simple api. I'm running .net 8 and have a Caddy server set up as a reverse proxy, all set up using docker compose. I initially had nginx running using certbot/letsencrypt but couldn't get that working so switched to Caddy because I heard it works out of the box. I thought my issue was the web server but I think the issue is with the API or kesterel handing the SSL?

This is my Caddy file:

example.com {

  redir /api /api/
  handle_path /api/* {
    reverse_proxy backend:8000
  }

  handle {
    root * /usr/share/caddy
    file_server
    try_files {path} /index.html
  }

  log {
    output stdout
  }
}

my docker compose:

name: myapp

services:
  backend:
    container_name: mycontainer
    image: myimage
    ports:
      - 8000:8080
  caddy:
    container_name: caddy
    image: caddy
    restart: always
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - caddy-config:/config
      - caddy-data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./index.html:/usr/share/caddy/index.html

volumes:
  caddy-config:
  caddy-data:

this is when I start everythign with docker compose up:

[+] Running 3/3
 ✔ Network myapp_default  Created                                                                                             0.1s
 ✔ Container caddy         Created                                                                                             0.0s
 ✔ Container mycontainer     Created                                                                                             0.1s
Attaching to caddy, mycontainer
caddy      | {"level":"info","ts":1725743811.7998998,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
caddy      | {"level":"info","ts":1725743811.8028176,"msg":"adapted config to JSON","adapter":"caddyfile"}
caddy      | {"level":"warn","ts":1725743811.803073,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy      | {"level":"info","ts":1725743811.8087811,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy      | {"level":"info","ts":1725743811.8092213,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy      | {"level":"info","ts":1725743811.8094192,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy      | {"level":"info","ts":1725743811.8110456,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy      | {"level":"info","ts":1725743811.811443,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy      | {"level":"info","ts":1725743811.8148913,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy      | {"level":"info","ts":1725743811.815138,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy      | {"level":"info","ts":1725743811.8152907,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["example.com"]}
caddy      | {"level":"info","ts":1725743811.8163688,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy      | {"level":"info","ts":1725743811.8165393,"msg":"serving initial configuration"}
caddy      | {"level":"info","ts":1725743811.818198,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000942200"}
caddy      | {"level":"info","ts":1725743811.8239324,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
caddy      | {"level":"info","ts":1725743811.8250542,"logger":"tls","msg":"finished cleaning storage units"}
caddy      | {"level":"info","ts":1725743812.0530305,"logger":"http.acme_client","msg":"got renewal info","names":["example.com"],"window_start":1730607560,"window_end":1730780360,"selected_time":1730768979,"recheck_after":1725765412.053012,"explanation_url":""}
caddy      | {"level":"info","ts":1725743812.0540433,"logger":"tls","msg":"updated ACME renewal information","identifiers":["example.com"],"cert_hash":"5a638e82e1d8085182c5w44372dae49ac0c0425a171d3b9a7147df","ari_unique_id":"kydGmAOpUWiOmNbEQkjbI7I.BG3jz_njzgdss2X27mGXu","cert_expiry":1733284790,"selected_time":1730764641,"next_update":1725765412.053012,"explanation_url":""}

mycontainer| info: Microsoft.Hosting.Lifetime[14]
mycontainer|       Now listening on: http://[::]:8080
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer|       Application started. Press Ctrl+C to shut down.
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer|       Hosting environment: Production
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer|       Content root path: /App

when I hit my domain with curl on the non https url, it works:

curl -v -L http://example.com:8000/api/heartbeat

*   Trying 2600:3c03::f03c:92ff:fe92:c4a9...
* TCP_NODELAY set
* Expire in 149994 ms for 3 (transfer 0x558166564f50)
* Expire in 200 ms for 4 (transfer 0x558166564f50)
* Connected to example.com (2600:3c03::f03c:92ff:fe92:c4a9) port 8000 (#0)
> GET /api/heartbeat HTTP/1.1
> Host: example.com.com:8000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 07 Sep 2024 21:48:29 GMT
< Server: Kestrel
< Transfer-Encoding: chunked
<
* Connection #0 to host example.com left intact
hello

When I try https, it doesn't:

curl -v -L https://example.com:8000/api/heartbeat

* TCP_NODELAY set
* Expire in 149998 ms for 3 (transfer 0x55ee5dca7f50)
* Expire in 200 ms for 4 (transfer 0x55ee5dca7f50)
* Connected to example.com port 8000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

and when I try without the port number on https:

curl -v -L https://example.com/api/heartbeat

 TCP_NODELAY set
* Expire in 149992 ms for 3 (transfer 0x563d5ed06f50)
* Expire in 200 ms for 4 (transfer 0x563d5ed06f50)
* Connected to example.com (2600:3c03::f03c:92ff:fe92:c4a9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=example.com
*  start date: Sep  5 03:59:51 2024 GMT
*  expire date: Dec  4 03:59:50 2024 GMT
*  subjectAltName: host "example.com" matched cert's "example.com"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x563d5ed06f50)
> GET /api/heartbeat HTTP/2
> Host: example.com
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Sat, 07 Sep 2024 21:51:12 GMT
<
* Connection #0 to host example.com left intact

and caddy logs this:

caddy      | {"level":"error","ts":1725745872.6222372,"logger":"http.log.error.log0","msg":"dial tcp 172.27.0.2:8000: connect: connection refused","request":{"remote_ip":"172.27.0.1","remote_port":"33794","client_ip":"172.27.0.1","proto":"HTTP/2.0","method":"GET","host":"example.com","uri":"/api/heartbeat","headers":{"User-Agent":["curl/7.64.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"example.com"}},"duration":0.00286854,"status":502,"err_id":"psdgnenq6","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
caddy      | {"level":"error","ts":1725745872.6229415,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.27.0.1","remote_port":"33794","client_ip":"172.27.0.1","proto":"HTTP/2.0","method":"GET","host":"example.com","uri":"/api/heartbeat","headers":{"User-Agent":["curl/7.64.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"example.com"}},"bytes_read":0,"user_id":"","duration":0.00286854,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

In .NET, I've tried a few things. I tried both using UseHttpsRedirection and not using it and also setting the https port in my appsettings file and not setting it. Currently, I removed setting the https port from my app settings and this is what my code looks like:

builder.Services.AddControllers();
builder.Services.AddCors(options => options.AddPolicy("CorsPolicy",
builder =>
{
    builder.AllowAnyMethod()
           .AllowAnyHeader()
           .AllowAnyOrigin();
}));
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
//    app.UseForwardedHeaders();
}
else
{
    app.UseExceptionHandler("/Error");
  //  app.UseForwardedHeaders();
  //  app.UseHsts();
}

app.UseCors("CorsPolicy");
//app.UseHttpsRedirection();

app.UseAuthorization();

app.MapControllers();

app.Run();


[Route("api/[controller]")]
[ApiController]
public class HeartBeatController : ControllerBase
{
    [HttpGet]
    public IActionResult Get()
    {
        return Ok("hello");
    }
}

So I've been trying to solve this for about a week now and can't figure out what's wrong. Anything stand out or suggestions on what to do to debug further?


Solution

  • So I figured this out and there were two issues. I had to change the port in my Caddy file from 8000 to 8080.

    The other issue is was the handle_path directive in the Caddy file. Apparently, this will match and then strip out the text, so instead of the request being http://example.com/api/heartbeat what was being sent to the api was http://example.com/heartbeat. So, I changed my Caddy file to:

     handle /api/* {
          reverse_proxy http://backend:8080
        }
    

    So I'm using handle instead of handle_path, which doesn't strip out the "api" from the URL.