I'm trying to set up SSL for a domain that is pointing to a simple api. I'm running .net 8 and have a Caddy server set up as a reverse proxy, all set up using docker compose. I initially had nginx running using certbot/letsencrypt but couldn't get that working so switched to Caddy because I heard it works out of the box. I thought my issue was the web server but I think the issue is with the API or kesterel handing the SSL?
This is my Caddy file:
example.com {
redir /api /api/
handle_path /api/* {
reverse_proxy backend:8000
}
handle {
root * /usr/share/caddy
file_server
try_files {path} /index.html
}
log {
output stdout
}
}
my docker compose:
name: myapp
services:
backend:
container_name: mycontainer
image: myimage
ports:
- 8000:8080
caddy:
container_name: caddy
image: caddy
restart: always
ports:
- '80:80'
- '443:443'
volumes:
- caddy-config:/config
- caddy-data:/data
- ./Caddyfile:/etc/caddy/Caddyfile
- ./index.html:/usr/share/caddy/index.html
volumes:
caddy-config:
caddy-data:
this is when I start everythign with docker compose up:
[+] Running 3/3
✔ Network myapp_default Created 0.1s
✔ Container caddy Created 0.0s
✔ Container mycontainer Created 0.1s
Attaching to caddy, mycontainer
caddy | {"level":"info","ts":1725743811.7998998,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
caddy | {"level":"info","ts":1725743811.8028176,"msg":"adapted config to JSON","adapter":"caddyfile"}
caddy | {"level":"warn","ts":1725743811.803073,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy | {"level":"info","ts":1725743811.8087811,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy | {"level":"info","ts":1725743811.8092213,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy | {"level":"info","ts":1725743811.8094192,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy | {"level":"info","ts":1725743811.8110456,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy | {"level":"info","ts":1725743811.811443,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy | {"level":"info","ts":1725743811.8148913,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy | {"level":"info","ts":1725743811.815138,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy | {"level":"info","ts":1725743811.8152907,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["example.com"]}
caddy | {"level":"info","ts":1725743811.8163688,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy | {"level":"info","ts":1725743811.8165393,"msg":"serving initial configuration"}
caddy | {"level":"info","ts":1725743811.818198,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000942200"}
caddy | {"level":"info","ts":1725743811.8239324,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
caddy | {"level":"info","ts":1725743811.8250542,"logger":"tls","msg":"finished cleaning storage units"}
caddy | {"level":"info","ts":1725743812.0530305,"logger":"http.acme_client","msg":"got renewal info","names":["example.com"],"window_start":1730607560,"window_end":1730780360,"selected_time":1730768979,"recheck_after":1725765412.053012,"explanation_url":""}
caddy | {"level":"info","ts":1725743812.0540433,"logger":"tls","msg":"updated ACME renewal information","identifiers":["example.com"],"cert_hash":"5a638e82e1d8085182c5w44372dae49ac0c0425a171d3b9a7147df","ari_unique_id":"kydGmAOpUWiOmNbEQkjbI7I.BG3jz_njzgdss2X27mGXu","cert_expiry":1733284790,"selected_time":1730764641,"next_update":1725765412.053012,"explanation_url":""}
mycontainer| info: Microsoft.Hosting.Lifetime[14]
mycontainer| Now listening on: http://[::]:8080
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer| Application started. Press Ctrl+C to shut down.
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer| Hosting environment: Production
mycontainer| info: Microsoft.Hosting.Lifetime[0]
mycontainer| Content root path: /App
when I hit my domain with curl on the non https url, it works:
curl -v -L http://example.com:8000/api/heartbeat
* Trying 2600:3c03::f03c:92ff:fe92:c4a9...
* TCP_NODELAY set
* Expire in 149994 ms for 3 (transfer 0x558166564f50)
* Expire in 200 ms for 4 (transfer 0x558166564f50)
* Connected to example.com (2600:3c03::f03c:92ff:fe92:c4a9) port 8000 (#0)
> GET /api/heartbeat HTTP/1.1
> Host: example.com.com:8000
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Date: Sat, 07 Sep 2024 21:48:29 GMT
< Server: Kestrel
< Transfer-Encoding: chunked
<
* Connection #0 to host example.com left intact
hello
When I try https, it doesn't:
curl -v -L https://example.com:8000/api/heartbeat
* TCP_NODELAY set
* Expire in 149998 ms for 3 (transfer 0x55ee5dca7f50)
* Expire in 200 ms for 4 (transfer 0x55ee5dca7f50)
* Connected to example.com port 8000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
and when I try without the port number on https:
curl -v -L https://example.com/api/heartbeat
TCP_NODELAY set
* Expire in 149992 ms for 3 (transfer 0x563d5ed06f50)
* Expire in 200 ms for 4 (transfer 0x563d5ed06f50)
* Connected to example.com (2600:3c03::f03c:92ff:fe92:c4a9) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=example.com
* start date: Sep 5 03:59:51 2024 GMT
* expire date: Dec 4 03:59:50 2024 GMT
* subjectAltName: host "example.com" matched cert's "example.com"
* issuer: C=US; O=Let's Encrypt; CN=E6
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x563d5ed06f50)
> GET /api/heartbeat HTTP/2
> Host: example.com
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 502
< alt-svc: h3=":443"; ma=2592000
< server: Caddy
< content-length: 0
< date: Sat, 07 Sep 2024 21:51:12 GMT
<
* Connection #0 to host example.com left intact
and caddy logs this:
caddy | {"level":"error","ts":1725745872.6222372,"logger":"http.log.error.log0","msg":"dial tcp 172.27.0.2:8000: connect: connection refused","request":{"remote_ip":"172.27.0.1","remote_port":"33794","client_ip":"172.27.0.1","proto":"HTTP/2.0","method":"GET","host":"example.com","uri":"/api/heartbeat","headers":{"User-Agent":["curl/7.64.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"example.com"}},"duration":0.00286854,"status":502,"err_id":"psdgnenq6","err_trace":"reverseproxy.statusError (reverseproxy.go:1269)"}
caddy | {"level":"error","ts":1725745872.6229415,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"172.27.0.1","remote_port":"33794","client_ip":"172.27.0.1","proto":"HTTP/2.0","method":"GET","host":"example.com","uri":"/api/heartbeat","headers":{"User-Agent":["curl/7.64.0"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"example.com"}},"bytes_read":0,"user_id":"","duration":0.00286854,"size":0,"status":502,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
In .NET, I've tried a few things. I tried both using UseHttpsRedirection and not using it and also setting the https port in my appsettings file and not setting it. Currently, I removed setting the https port from my app settings and this is what my code looks like:
builder.Services.AddControllers();
builder.Services.AddCors(options => options.AddPolicy("CorsPolicy",
builder =>
{
builder.AllowAnyMethod()
.AllowAnyHeader()
.AllowAnyOrigin();
}));
// Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();
var app = builder.Build();
// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
app.UseSwagger();
app.UseSwaggerUI();
// app.UseForwardedHeaders();
}
else
{
app.UseExceptionHandler("/Error");
// app.UseForwardedHeaders();
// app.UseHsts();
}
app.UseCors("CorsPolicy");
//app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();
app.Run();
[Route("api/[controller]")]
[ApiController]
public class HeartBeatController : ControllerBase
{
[HttpGet]
public IActionResult Get()
{
return Ok("hello");
}
}
So I've been trying to solve this for about a week now and can't figure out what's wrong. Anything stand out or suggestions on what to do to debug further?
So I figured this out and there were two issues. I had to change the port in my Caddy file from 8000 to 8080.
The other issue is was the handle_path directive in the Caddy file. Apparently, this will match and then strip out the text, so instead of the request being http://example.com/api/heartbeat what was being sent to the api was http://example.com/heartbeat. So, I changed my Caddy file to:
handle /api/* {
reverse_proxy http://backend:8080
}
So I'm using handle instead of handle_path, which doesn't strip out the "api" from the URL.