javaspring-bootprocessbuilder

Running a ProcessBuilder process from inside a SpringBoot application regularly hangs and does not execute the command


First off, this is a SpringBoot application running inside a Docker container.

The application will perform a Liquibase update for each configured tenant. In this case, 3 tenants.

What I find is that it might successfully execute for tenant 001 (or it may hang), and then hang on the 2nd or 3rd tenant at the same point listed before.

I have set the ProcessBuilder to redirectErrorStream(true) and inheritIO() but neither make any difference to the reliability of the API/command.

What am I doing wrong here?

The same command that is success for 1 tenant may hang on another, even though there is no difference in tenant state or db .. all tenants are equal at the minute.

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Arrays;
import java.util.List;
import org.springframework.stereotype.Component;

@Component
public class MyComponent {
  public MyComponent() throws Exception {
    
    System.out.println("RB");

    List<String> tenants = Arrays.asList("001", "002schema", "003_schema");

    for (String tenant : tenants) {

      ProcessBuilder processBuilder = new ProcessBuilder();
      processBuilder.redirectErrorStream(true);
      processBuilder.inheritIO();

      System.out.println("Liquibase update: " + tenant);

      processBuilder.command(
          "/home/liquibase/liquibase",
          "--url=jdbc:postgresql://localhost:5432/postgres?currentSchema=" + tenant,
          "--changeLogFile=config/liquibase/changelog/root-changelog.xml",
          "--username=postgres",
          "--password=password",
          "--search-path=/home/",
          "--liquibase-schema-name=" + tenant,
          "--preserveSchemaCase=true",
          "update");

      System.out.println(processBuilder);

      Process process = processBuilder.start();

      System.out.println("Started");

      StringBuilder output = new StringBuilder();
      BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
      System.out.println("Buffer created");

      String line;
      while ((line = reader.readLine()) != null) {
        output.append(line + "\n");
      }
      System.out.println("Read lines from buffer");

      int exitVal = process.waitFor();
      System.out.println("Waited for process");
      if (exitVal == 0) {
        System.out.println("Success!");
        System.out.println(output);
      } else {
        System.out.println("Error!");
        System.out.println(output);
      }

      reader.close();
      process.destroy();
    }
  }
}

This is the output from the logs:

2023-03-24 22:39:00.515  INFO 30 --- [           main] c.c.c.d.migration.SpringBootApplication  : Starting SpringBootApplication using Java 17.0.6 on docker-desktop with PID 30 (/home/app-0.1.0-SNAPSHOT.jar started by user in /home)
2023-03-24 22:39:00.528  INFO 30 --- [           main] c.c.c.d.migration.SpringBootApplication  : No active profile set, falling back to 1 default profile: "default"
RB
Liquibase update: 001
java.lang.ProcessBuilder@d35dea7
Started
Buffer created
Read lines from buffer
####################################################
##   _     _             _ _                      ##
##  | |   (_)           (_) |                     ##
##  | |    _  __ _ _   _ _| |__   __ _ ___  ___   ##
##  | |   | |/ _` | | | | | '_ \ / _` / __|/ _ \  ##
##  | |___| | (_| | |_| | | |_) | (_| \__ \  __/  ##
##  \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___|  ##
##              | |                               ##
##              |_|                               ##
##                                                ## 
##  Get documentation at docs.liquibase.com       ##
##  Get certified courses at learn.liquibase.com  ## 
##  Free schema change activity reports at        ##
##      https://hub.liquibase.com                 ##
##                                                ##
####################################################
Starting Liquibase at 22:39:12 (version 4.20.0 #7837 built at 2023-03-07 16:25+0000)
Liquibase Version: 4.20.0
Liquibase Open Source 4.20.0 by Liquibase
Database is up to date, no changesets to execute
Liquibase command 'update' was executed successfully.
Waited for process
Success!

Liquibase update: 002schema
java.lang.ProcessBuilder@5fbe4146
Started
Buffer created
Read lines from buffer


Solution

  • You appear to have the right calls for ProcessBuilder but they aren't in the right order to help debug your issue.

    You should print output before call to waitFor as the message may tell you the issue.

    System.out.println("Read lines from buffer");
    System.out.println(output);
    int exitVal = process.waitFor();
    

    Sometimes certain sub-processes wait on their STDIN eg "bash" / "cmd.exe /c" so adding explicit close to STDIN may help:

    Process process = processBuilder.start();
    process.getOutputStream().close();
    

    You can also simplify the logic using StringWriter and try finally blocks. You shouldn't need inheritIO. Consider using file redirection processBuilder.redirectOutput(File) in case the sub-process writes a hugh mount of data to avoid OOM issues.

      ProcessBuilder processBuilder = new ProcessBuilder();
      processBuilder.redirectErrorStream(true);
      // processBuilder.inheritIO();
    
      System.out.println("Liquibase update: " + tenant);
    
      processBuilder.command(
          "/home/liquibase/liquibase",
          "--url=jdbc:postgresql://localhost:5432/postgres?currentSchema=" + tenant,
          "--changeLogFile=config/liquibase/changelog/root-changelog.xml",
          "--username=postgres",
          "--password=password",
          "--search-path=/home/",
          "--liquibase-schema-name=" + tenant,
          "--preserveSchemaCase=true",
          "update");
    
      System.out.println(processBuilder);
    
      Process process = processBuilder.start();
    
      System.out.println("Started");
    
      // If your app does not read it's STDIN then close it to signal end of input
      process.getOutputStream().close();
    
      StringWriter output = new StringWriter();
      try(BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
          System.out.println("Buffer created");
    
          reader.transferTo(output);
      }
      System.out.println("Read lines from buffer");
      System.out.println("=".repeat(80));
      System.out.println(output);
      System.out.println("=".repeat(80));
    
      int exitVal = process.waitFor();
      System.out.println("Waited for process exitVal = "+exitVal+ (exitVal == 0 ? " Success": " ERROR"));