How to add a directory to classpath from a gradle plugin?

I have a task which creates a directory within build-directory. There is one file within the directory. I am trying to add that directory to classpath from within a gradle plugin written in java.

public class Myplugin implements Plugin<Project> {
    @Override
    public void apply(Project target) {
        SourceSetContainer sourceSetContainer = target.getExtensions().getByType(SourceSetContainer.class);
        SourceSet main = sourceSetContainer.findByName("main");
        assert main != null;
        // Next line is adding 'my-directory' to the classpath
        main.setRuntimeClasspath(main.getRuntimeClasspath().plus(target.getLayout().getBuildDirectory().dir("my-directory").get().getAsFileTree()));
    }
}

This results in classpath like this (notice the last line):

/Users/MYUSER/git/my-repo/build/classes/java/main, 
/Users/MYUSER/git/my-repo/build/classes/kotlin/main, 
/Users/MYUSER/git/my-repo/build/classes/groovy/main, 
/Users/MYUSER/git/my-repo/build/tmp/kapt3/classes/main, 
/Users/MYUSER/git/my-repo/build/resources/main, 
... Bunch of JAR  files ...
/Users/MYUSER/git/my-repo/build/my-directory/my-file.yml   <-----

However I want just want the directory to appear in the classpath, not the files within it. Something like this:

/Users/MYUSER/git/my-repo/build/classes/java/main, 
/Users/MYUSER/git/my-repo/build/classes/kotlin/main, 
/Users/MYUSER/git/my-repo/build/classes/groovy/main, 
/Users/MYUSER/git/my-repo/build/tmp/kapt3/classes/main, 
/Users/MYUSER/git/my-repo/build/resources/main, 
... Bunch of JAR  files ...
/Users/MYUSER/git/my-repo/build/my-directory   <-----

How to achieve this?

Below is the code, to add /build/my-directory/ as a directory entry to the classpath.

main.setRuntimeClasspath(
    main.getRuntimeClasspath().plus(
        target.files(target.getLayout().getBuildDirectory().dir("my-directory"))
    )
);

The issue with getAsFileTree() in the problem is, it adds each file in the directory to the classpath.