Visual Studio Docker orchestrator: cannot debug when dockerfile isn't in project directory

Visual Studio does not show a debug option when using a Docker compose project (.dcproj) and the Dockerfile is not in the project folder but up a couple of levels.

I have the following existing file structure:

root
|- docker-compose.yml
|- docker-compose.dcproj
|-- DockerFiles\
|   |- Dockerfile
|-- TestForDocker\
|   |- TestForDocker.sln
|   |-- TestForDocker\
|       |- TestForDocker.csproj

This causes that there's not an StartDebugging option in the "Manage Docker Compose Launch Settings"

Placing the Dockerfile in the .csproj directory and changing the path in the docker-compose.yml makes the option to StartDebugging visible:

How can I have it so that the Dockerfile doesn't have to be in the same folder as the .csproj?

I've tried to add

<DockerfileFile>..\..\DockerFiles\Dockerfile</DockerfileFile>

to the .csproj, but to no avail, this especially in combination with

<DockerComposeProjectPath>..\..\docker-compose.dcproj</DockerComposeProjectPath>

For reference, here's the docker-compose.yml:

services:
  testfordocker:
    image: ${DOCKER_REGISTRY-}testfordocker
    build:
      context: .
      dockerfile: DockerFiles/Dockerfile
# The StartDebugging becomes visible whith the line below uncommented instead of the one above.
#      dockerfile: TestForDocker/TestForDocker/Dockerfile

It is possible to add a label to your docker-compose.yml or docker-compose.override.yml to associate the service with the project.

build: 
  blah..blah
labels:
  com.microsoft.visual-studio.project-name: "TestforDocker"

See Issue on github.

The label com.microsoft.visual-studio.project-name unfortunately isn't defined in the online documentation.
Credits go to dbreshears on github