Restrict communication between Java programs to same user

We want to build a Java app associated with files and URLs, but only one instance should run at a time. If a second file is opened, it should be handled by an already running instance, if available. (Think of this as a file viewer: a user clicking a link in a browser opens the app, clicking a second link brings the same app to the front and opens the second URL in a new view.)

To that end, my plan was:

• User calls app file1.ext. This starts the first instance.
• This instance detects it is the only instance running, continues launch and opens file1.ext.
• User calls app file2.ext. This starts the second instance.
• Second instance detects it is the second instance and sends a command to open file2.ext to the first instance.
• First instance receives that command from second instance, opens file2.ext, and confirms.
• Second instance aborts launch and exits.

I can think of a variety of technologies for implementing the communication channel, but all of them open a port on the local machine. That is fine, but I want to make sure only the current user can use this port. I can probably prevent a second user from sending commands to the first user's app accidentally, but how do I enforce that a port or whatever is used for the inter-process communication is available only to the user who initiates the first app launch?

The solution should be as platform independent as possible.

You can use a lock/pid file, the program should check if the lock file exists to find if there is an existing process. The lock file should contain the port number it's listening on, the server key, and a session key. You'll need to set the file permission so it's only readable by the current user. All messages to the initial process must contain a matching session key. The session key is the proof that the originating process has the permission to read the lock file, so your open port inherits the lock file's permission.

For security, you'll need to be careful about the order that you open the port and write the lock file. You'll need to make sure that you have the port open before writing the lock file, or a malicious program could outrace the server and receive messages that it shouldn't have been able to read. Also, the second process should check the owner of the lock file, to make sure that it's created by the current user. Also, before the second processes start sending in data, you should also check the server return a server key in its handshake, to make sure that the currently running server actually also have read access to the lock file, as the server could have been long dead, replaced by the malicious program. Finally, make sure that the program binds only to local connections, unless you actually want to allow requests from the network.

If this is a Linux/Unix/Mac only program, then you have another alternative of opening a Unix domain socket. You should set the file permission for the domain socket so that it's only readable/writable by the current user. If you use domain socket, you don't need a session key or all these handshake things since a domain socket's permission is enforced by the domain socket's file permission.