Programs such as Truecrypt or Dropbox offer a comfortable way for file handling. At least under Windows Truecrypt mounts a new drive which feels natural, like a physical drive. Dropbox creates a new folder that can automatically upload/download data.
If I want to implement something similar, is then Java’s FileSystemProvider the right way to do it?
What I want is that a new folder appears and lists (remote) files which are not present on the user’s system. But when the user tries to open such a file or copy it to some other path then this should work as expected.
To create a virtual filesystem visible to all (or chosen) applications in the system you need to create a filesystem driver on Windows (on Linux and MacOS there exist FUSE and OSXFUSE respectively). This is a slow and painful process.
Java's classes, as @keshlam wrote in the comment, are not suitable for creation of the system-visible virtual disk.
Again for Windows you can check our Callback File System product that lets you write the business logic in user mode and provides a pre-created filesystem kernel-mode driver. It includes Java API so you can use it in Java. Due to differences in filesystem architecture in Unix-based OS and Windows the APIs are different from those found in FUSE though.