docker-compose: route traffic through vpn except for connections to other services

Consider this docker-compose configuration:

# docker-compose.yml
version: "3.7"

services:
  app:
    build: ./app
    depends_on:
      - db
      - vpn
    ports:
      - "3001:3000"
  db:
    image: postgres
  vpn:
    build: ./vpn
    cap_add:
      - NET_ADMIN

Description

The app is accessed from the docker host via http://localhost:3001.
The app needs to connect to a postgres db, which is the second container.
Also, the app needs to connect to an api, which is only available though a vpn. This is why a third container, vpn, establishes the required vpn connection.

Goal

The app container should should be able to reach the other services within this docker-compose environment, i.e. the db, and route the rest of its traffic through the vpn container, such that it can access the api behind the vpn tunnel.

What I've tried

I have tried to set the network_mode of the app:
```
services:
  app:
    network_mode: "service:vpn"
```
This routes all traffic of the app container through the vpn. With this, I can reach the api behind the vpn tunnel from the app container. But this is not compatible with ports: - "3001:3000". Also, the db container cannot be reached from the app anymore: ping: bad address 'db'.
I also have tried to link the db container from the vpn container, hoping that this would make the db service available to the app.
```
services:
  app:
    network_mode: "service:vpn"
  vpn:
    links:
      - db
```
But still db cannot be found by app.
If I link the db container from the vpn container but do not establish the vpn connection within the vpn container, the db container can be reached from the app.
And I've experimented with adding 127.0.0.1 db to the /etc/hosts of the app container, vaguely hoping that I could reach the db port directly. But this also does not work.

Does anyone have a clue how to achieve this?

Solution

I've finally found a solution, but it required three steps:

Step 1: `network_mode: service`

In order to route all traffic of the app container through the vpn, set network_mode on the app container:

services:
  app:
    network_mode: "service:vpn"

Step 2: DNS servers

In order to resolve both the host names behind the vpn tunnel as well as the local docker services, the vpn container needs to talk to both DNS servers: the DNS server behind the tunnel as well as the docker-compose DNS server.

The docker-compose DNS server is always 127.0.0.11 as far as I understood.

To find out the remote DNS server behind the tunnel, establish the tunnel and then run cat /etc/resolv.conf. This will list the DNS server behind the tunnel in the line commented with "by strongSwan".

In the startup script of the vpn container, add both DNS servers to the resolv.conf of the vpn container:

# vpn-container startup script:
echo "nameserver <remote dns server ip>" >> /etc/resolv.conf
echo "nameserver 127.0.0.11" >> /etc/resolv.conf

To test this, log into the vpn container and try to ping a remote ip and the db container:

docker-compose run vpn /bin/bash
ping db  # should work
ping <some-ip-behind-the-vpn-tunnel>  # should also work

Step 3: Expose the port

With network_mode: "service:vpn" on the app container, the app container cannot expose its ports to the host anymore as far as I understood. Instead, the app container and the vpn container appear as the same machine to the docker host, now. Therefore, one can expose the desired ports on the vpn container instead.

services:
  vpn:
    ports:
      - "3001:3000"

The app (!) is then reachable through http://localhost:3001 on the docker host.

Bringing all together: Final `docker-compose.yml`

# docker-compose.yml
version: "3.7"

services:
  app:
    build: ./app
    depends_on:
      - db
      - vpn
    network_mode: "service:vpn"
  db:
    image: postgres
  vpn:
    build: ./vpn
    cap_add:
      - NET_ADMIN
    ports:
      - "3001:3000"
    command: >
      bash -c "echo 'nameserver <remote dns server ip>' >> /etc/resolv.conf
      && echo 'nameserver 127.0.0.11' >> /etc/resolv.conf
      && ..."