I have a web app sitting in an Linux Container, The LXD version is LXD 3.13
Web traffic comes in on the host vm (Ubuntu 16.04) and is pushed into the LXC using NGinx, nginx is version 1.10.3
Inside the container I have Kong receiving the traffic, and forwarding it to my app. Kong is version 1.1.2. The container itself is also Ubuntu 16.04.
INTERNET -> NGINX -> LXD -> KONG(runs NGINX) -> MY APP
My goal is to have Kong terminate SSL traffic and provide whitelist IP-restrictions. Without the whitelist I have everything working. But my issue is that Kong is seeing the LXD Base IP and not the original Client IP that calls NGinx. I think I need to use "Proxy Protocol" configurations to solve this but if I have other options I'm all ears.
My NGinx configuration is pretty simple. I have to use stream instead of html so that I don't have to terminate the SSL. This means I can't set the X-Real-IP header, which would've been an easier solution I think. So instead I'm attempting to use the proxy protocol.
stream {
server {
listen 80;
proxy_pass 10.214.23.104:8080;
proxy_protocol on;
}
server {
listen 443;
proxy_pass 10.214.23.104:8443;
proxy_protocol on;
}
}
10.214.23.104 in this case is just the IP my LXD assigned for this container. 8080 and 8443 are the Kong ports for http/https traffic respectively. When I try adding the 'proxy_protocol on;' lines instead of seeing a wrong IP response from Kong I'm getting a 'Unsupported or unrecognized SSL message' instead. and Kong doesn't appear to be processing it.
This is my NGinx error logs in this case:
epoll: fd:8 ev:0
001 d:00007F170E93E0F8
post event 0000558DBC67B8E0
timer delta: 300041
posted event 0000558DBC67B8E0
delete posted event 0000558DBC67B8E0
accept on 0.0.0.0:443, ready: 0
posix_memalign: 0000558DBC650650:256 @16
***.***.***.***:52412 fd:12
*5 client ***.***.***.***:52412 connected to 0.0.0.0:443
*5 tcp_nodelay
*5 posix_memalign: 0000558DBC650760:256 @16
*5 proxy connection handler
*5 malloc: 0000558DBC650870:328
*5 malloc: 0000558DBC654C00:16384
*5 stream proxy send PROXY protocol header
*5 get rr peer, try: 1
*5 stream socket 13
*5 epoll add connection: fd:13 ev:80002005
*5 connect to 10.214.23.104:8443, fd:13 #6
*5 proxy connect: -2
*5 event timer add: 13: 60000:1560459025106
worker cycle
accept mutex locked
epoll timer: 60000
epoll: fd:13 ev:0004 d:00007F170E93E3B0
*5 post event 0000558DBC68DA10
timer delta: 0
posted event 0000558DBC68DA10
*5 delete posted event 0000558DBC68DA10
*5 event timer del: 13: 1560459025106
*5 stream proxy connect upstream
*5 tcp_nodelay
*5 proxy 10.214.23.1:50946 connected to 10.214.23.104:8443
*5 malloc: 0000558DBC69F8A0:16384
*5 send: fd:13 44 of 44
*5 epoll add event: fd:12 op:1 ev:80002001
*5 event timer add: 12: 600000:1560459565106
worker cycle
accept mutex locked
epoll timer: 600000
epoll: fd:13 ev:0005 d:00007F170E93E3B0
*5 post event 0000558DBC67BA00
*5 post event 0000558DBC68DA10
timer delta: 1
posted event 0000558DBC67BA00
*5 delete posted event 0000558DBC67BA00
*5 recv: fd:13 177 of 16384
*5 send: fd:12 177 of 177
*5 recv: fd:13 0 of 16384
*5 upstream disconnected, bytes from/to client:0/177, bytes from/to upstream:177/44
*5 finalize stream proxy: 0
*5 free rr peer 1 0
*5 close stream proxy upstream connection: 13
*5 delete posted event 0000558DBC68DA10
*5 reusable connection: 0
*5 close stream connection: 12
*5 event timer del: 12: 1560459565106
*5 reusable connection: 0
*5 free: 0000558DBC69F8A0
*5 free: 0000558DBC654C00
*5 free: 0000558DBC650870
*5 free: 0000558DBC650650, unused: 0
*5 free: 0000558DBC650760, unused: 40
I'm not quite sure how to troubleshoot LXD yet. there is nothing in the lxc.log file for my container.
On the Kong side I have configured the Logging and IP-Restriction plugins, as well as set up a SSL Certificate for the appropriate routes. I've added the real_ip_header as proxy_protocol; my understanding is that this is all that's necessary to read the IP when using proxy protocol. My configuration for Kong looks like this:
trusted_ips = 0.0.0.0/0
admin_listen = 0.0.0.0:8001
proxy_listen = 0.0.0.0:8080, 0.0.0.0:8443 ssl
...db stuff...
plugins = bundled,session
real_ip_header = proxy_protocol
Kong's Access Log does look like its writing out that it recieved a call but it is not processing it and nothing is appearing in the error log or the logging plugin's log for that service.
10.214.23.1 - - [14/Jun/2019:05:23:46 -0900] "PROXY TCP4 72.38.194.90 10.1.1.4 37604 443" 400 12 "-" "-"
I realized that the proxy_protocol value for real_ip_header isn't processed if I have already specified the proxy_listen configuration. We had previously changed the listening port due to conflicts, So I can't use the default proxy_listen value.
So my solution was to change the proxy_listen line to:
proxy_listen = 0.0.0.0:8080 proxy_protocol, 0.0.0.0:8443 ssl proxy_protocol
the real_ip_header = proxy_protocol line is still required.