To my understanding, java.util.concurrent.Semaphore allows me to specify how many threads can use a resource at once. A thread can use Semaphore.acquireUninterruptibly() to consume the limited number of "slots" in the Semaphore. Once the thread is done, a call should be made to Semaphore.release() in order to give back the slot, so that another thread that is waiting (acquireUninterruptibly() makes the thread wait) can grab the new slot. I am aware of the fairness policy, and ordering of the threads is irrelevant for my purposes. What matters is that all of the threads execute.
And that's my problem -- only one of the threads executes. Here is my code.
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Semaphore;
public class Main
{
private static final Path rootbarFolder = Path.of("C:", "Users");
private boolean REATTEMPT_UPON_FAILURE = true;
public static void main(final String[] args) throws Exception
{
new Main();
}
private Main() throws Exception
{
javax.swing.SwingUtilities
.invokeLater
(
() ->
{
javax.swing.JOptionPane.showMessageDialog(null, "Close this popup window to make the program exit on failure");
this.REATTEMPT_UPON_FAILURE = false;
}
)
;
final List<Path> fooPathList = this.getListOfbarfooPaths();
final List<Thread> fooPathThreads = new ArrayList<>();
final Semaphore semaphore = new Semaphore(1, true);
KICK_OFF_THREADS:
for (final Path fooPath : fooPathList)
{
final Thread fooPathThread = this.createThread(semaphore, fooPath);
fooPathThreads.add(fooPathThread);
semaphore.acquireUninterruptibly();
fooPathThread.start();
}
JOIN_THREADS:
for (final Thread fooPathThread : fooPathThreads)
{
fooPathThread.join();
}
}
private Thread createThread(final Semaphore semaphore, final Path fooPath)
{
return
Thread
.ofPlatform()
.unstarted
(
() ->
{
try
{
int exitCode = -1;
while (exitCode != 0 && this.REATTEMPT_UPON_FAILURE)
{
System.out.println("\n\nAttempting " + fooPath);
final Process fooProcess = this.createzilklaquo(fooPath);
exitCode = //Tells us the status of the run
fooProcess.waitFor(); //Don't close down the JVM before this process finishes running!
System.out.println(fooPath + " -- fooProcess exitCode = " + exitCode);
Thread.sleep(10_000);
}
}
catch (final Exception exception)
{
throw new RuntimeException(exception);
}
finally
{
semaphore.release();
}
}
)
;
}
private Process createzilklaquo(final Path fooPath)
{
try
{
final ProcessBuilder fooProcessBuilder =
new
ProcessBuilder
(
"cmd",
"/C",
"THIS_COMMAND_WILL_FAIL"
)
.directory
(
rootbarFolder //
.resolve(fooPath) //
.toFile() //
)
.inheritIO()
;
fooProcessBuilder
.environment()
.put("SUB_FOLDER", fooPath.getFileName().toString())
;
final Process fooProcess =
fooProcessBuilder
.start() //Kicks off the newly created Process
;
// final int exitCode = //Tells us the status of the run
// fooProcess.waitFor(); //Don't close down the JVM before this process finishes running!
//
// System.out.println("fooProcess exitCode = " + exitCode);
return fooProcess;
}
catch (final Exception e)
{
throw new RuntimeException(e);
}
}
private List<Path> getListOfbarfooPaths() throws Exception
{
final Process fooListProcess =
new
ProcessBuilder
(
"cmd", //Since this is Windows, CMD is the easiest way to accomplish what we want
"/C", //Starts an instance of CMD, does the below commands, outputs/pipes them, then immediately closes
"dir", //Lists all contents in the folder
"/A:D", //Filters the contents down to only directories
"/B" //Removes extra metadata -- just the names
)
.directory
(
rootbarFolder //Perform the action in the root bar folder
.toFile() //Wish I could give a Path instead of a File
)
//.inheritIO() //Forward all Input and Output to the same as Java's (commented out because it drowns out my logs)
.start() //Kicks off the newly created Process
;
final int exitCode = //Tells us the status of the run
fooListProcess.waitFor(); //Don't close down the JVM before this process finishes running!
System.out.println("fooListProcess exitCode = " + exitCode);
final String fooListRawOutput = //The raw output from the newly created process
new
String
(
fooListProcess //Now that the process has finished, we can pull from it
.getInputStream() //The way that you quo the OUTPUT of the process is to call getINPUTStream -- very unintuitive
.readAllBytes() //Let's quo all of it
)
;
final List<Path> fooList = //The list of foos that we will be working with
fooListRawOutput //We will be extracting it from the raw output
.lines() //It's a new-line-separated list, so split by line
.map(rootbarFolder::resolve) //Turn it into a Path that is the rootbarFolder resolved to the sub-folder -- root -> root/subFolder
.toList() //Finally, put the contents into a list
;
fooList.forEach(System.out::println);
return fooList;
}
}
Again, this is a fake example based off a real one that I cannot show because of company policy.
I have a folder with many subfolders in it. For example's sake, I am pointing it at the Users folder in the Windows C:/ drive. If you are Linux or something else, feel free to change that Path variable at the top to point to a different directory on your machine. But have it be a directory with multiple items in that directory.
So, on my machine, here are the folders in my C:/Users folder.
C:\Users\All Users
C:\Users\david
C:\Users\Default
C:\Users\Default User
C:\Users\Public
But when I run this program, it will only make a print statement for C:\Users\All Users, not for any of the other lines. That implies to me that this program is not attempting the other threads.
This is what it prints.
$ java Main.java
fooListProcess exitCode = 0
C:\Users\All Users
C:\Users\david
C:\Users\Default
C:\Users\Default User
C:\Users\Public
Attempting C:\Users\All Users
'THIS_COMMAND_WILL_FAIL' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\All Users -- fooProcess exitCode = 1
So, we can see that it attempted the All Users folder, but then it just stopped.
I was expecting something more like this.
java Main.java
fooListProcess exitCode = 0
C:\Users\All Users
C:\Users\david
C:\Users\Default
C:\Users\Default User
C:\Users\Public
Attempting C:\Users\All Users
'THIS_COMMAND_WILL_FAIL' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\All Users -- fooProcess exitCode = 1
Attempting C:\Users\david
'THIS_COMMAND_WILL_FAIL' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\david -- fooProcess exitCode = 1
...repeat for all other folders
But anyways, I want to do multithreading to do some work on these subfolders. The work I am doing fails often, so I have a while loop listening to the exitCode, and reattempting. However, for reasons that I cannot say, I need to have an escape hatch that says "From this point in time forward, any future failures should end the threads execution". That is what the JOptionPane is for. I know it is a terrible example, but the point is that it switches a boolean flag, which is what I need it to do.
Here's my problem though. When I switch the flag, the subsequent threads which should be waiting on the Semaphore to have a free slot don't get kicked off.
As for what the above code is literally doing, it attempts a command which is guaranteed to fail, triggering the while loop mechanic I mentioned. Then, the thread sleeps for 10 seconds, and tries again.
I have a popup that allows me to set the flag when I choose by pressing OK. When I do, the first thread completes, but the next thread never starts.
Now, I know the JVM can take multithreading "shortcuts" (for lack of a better term), and that's most likely what is happening here. But I don't know if that is the cause.
Why aren't the rest of my threads getting kicked off?
EDIT - I'm noticing some close votes, claiming that my question is not relevant for StackOverflow. Unfortunately, that is too broad for me to know how to improve my question.
My entire question is this -- I am having trouble understanding why basic libraries from the Java Standard library aren't doing what I think they should. I am clearly communicating my understanding of them, I showed my attempt, and my example is simple, minimal, and reproducible (as far as my computer can tell, anyways). If there is something further I should do, or I failed to do something, please let me know specifically so that I can make the change.
EDIT 2 - Oh, apparently, if you click on the close votes, and then click on the selection, they will go into more detail. Not very intuitive, but I understand now.
I have added all the requested info. Embarrassingly enough, that also gave me a push in the right direction and I was able to solve it myself. I have posted and accepted my answer below -- the solution was embarrassingly simple.
Silly me, look at that while condition.
while (exitCode != 0 && this.REATTEMPT_UPON_FAILURE)
&&
This means that once we set the flag to false, nothing else will even ENTER the while loop. A potential solution would be a do-while loop instead.
Everything worked perfectly once I switched it to a do-while loop. Here is my final solution.
int exitCode = -1;
do
{
System.out.println("\n\nAttempting " + fooPath);
final Process fooProcess = this.createzilklaquo(fooPath);
exitCode = //Tells us the status of the run.
fooProcess.waitFor(); //Don't close down the JVM before this
//process finishes running!
System.out.println(fooPath + " -- fooProcess exitCode = " + exitCode);
Thread.sleep(10_000);
}
while (exitCode != 0 && this.REATTEMPT_UPON_FAILURE)
;