linuxbashdockershell

How to use a function from first stage into second stage in Docker multistage build?

I am trying to build a Docker image that preprocesses MRI files. This requires several steps: skull-stripping the MRI, creating masks for the MRIs, etc. Each of these functions come from different packages (different Docker images), so my Docker image is just a wrapper that uses each of these functions iteratively so that clinicians can use it in Docker Desktop without using any line of code (just putting all original MRI files in an input directory and having the fully preprocessed MRI files in an output directory, just entering those directories as input and output in Docker desktop).

My problem is that I am not sure how to pass a function from the first stage to the second stage. I understand the overall idea of a Docker multistage build, but:

  1. I do not know what directories and files to copy from the first stage to the second stage for the function to work (I do not know how to figure out what files and directories the function needs to work) and I do not know how to call the function from the first stage in the second stage.

  2. I tried to copy all files and directories from the first stage in the second stage and putting them into a folder in the second stage, but then I do not know how to call that function in the second stage.

Dockerfile with the abstract idea of what I am trying to do:

#################################################FIRST IMAGE#################################################
# Pull first image from its docker hub
FROM itsdockerhub/firstimage AS base
#################################################FIRST IMAGE#################################################


#################################################SECOND IMAGE#################################################
# Import second image from its dockerhub
FROM itsdockerhub/secondimage

# Copy the iterative loop script (which is the entrypoint script I want to run)
COPY iterative_loop.sh .

# Create a new directory in the second image to save all files and directories from the first image
RUN mkdir ./allfilesandfoldersfromfirstimage

# Copy all the files and folders from first image
COPY --from=base . ./allfilesandfoldersfromfirstimage

#################################################FIRST IMAGE#################################################

# Call the iterative loop script
ENTRYPOINT ["./iterative_loop.sh"]

Iterative sh script with the abstract idea of what I am trying to do:

#!/bin/bash


# Create directory to save the first changes to the MRIs
mkdir ./MRIs_first_modifications

#################################################WORK WITH FUNCTIONS FROM FIRST IMAGE#################################################


# Loop over the input MRI files and do the first changes using the function from the first image
for FILE in /input/*; do
  namepart=$(basename "$FILE")
  filename="${namepart%%.*}" 
  extension="${namepart#*.}" 
  modified_name="${filename}_skull_stripped.${extension}"
  # Move into the folder with all files and folders from first image
  cd ./allfilesandfoldersfromfirstimage
  # Call the function from first image
  functionfromfirstimage -i $FILE -o ./MRIs_first_modifications/$modified_name
  # Return to the working directory of the second image
  cd ..
done
cp ./MRIs_first_modifications/* /output
#################################################WORK WITH FUNCTIONS FROM FIRST IMAGE#################################################


#################################################WORK WITH FUNCTIONS FROM SECOND IMAGE#################################################

# Loop over the modified MRI files and do the next change with a function from the second image
for FILE in ./MRIs_first_modifications/*; do
  namepart="$(basename "$FILE")"
  filename="${namepart%%.*}" 
  extension="${namepart#*.}" 
  modified_name="${filename}_mask.${extension}"
  functionfromsecondimage $FILE /output/$modified_name
done
#################################################WORK WITH FUNCTIONS FROM SECOND IMAGE#################################################

The real first function I am trying to use is synthstrip from freesurfer/synthstrip Docker image and the real second function I am trying to use is fslmaths from vistalab/fsl-v5.0 Docker image

And now the real Dockerfile that I tried

#################################################SYNTHSTRIP#################################################
# Import synthstrip image from freesufer
FROM freesurfer/synthstrip AS skull_strip_with_synthstrip
#################################################SYNTHSTRIP#################################################


#################################################FSL#################################################
# Import fsl image from vistalab
FROM vistalab/fsl-v5.0

# Copy the iterative loop script
COPY iterative_loop.sh .

# Make sure that the iterative loop script is executable
RUN chmod ugo+rwx iterative_loop.sh

RUN mkdir ./oldsynthstrip

# Copy all the initial files from synthstrip
COPY --from=skull_strip_with_synthstrip . ./oldsynthstrip


#################################################FSL#################################################

# Call the iterative loop script
ENTRYPOINT ["./iterative_loop.sh"]

And the real iterative_loop sh script that I tried

#!/bin/bash

#################################################SYNTHSTRIP#################################################
#################################################SYNTHSTRIP#################################################
# Create directory to save the skull-stripped MRIs
mkdir ./skull_stripped


# Loop over the input MRI files and skull-strip them with synthstrip
for FILE in /input/*; do
  namepart=$(basename "$FILE")
  if [[ $namepart == *".nii.gz"* ]]  ; then
      filename="${namepart%%.*}" 
      extension="${namepart#*.}"                    
      modified_name="${filename}_skull_stripped.${extension}" ; else
      filename="${namepart%.*}" 
      extension="${namepart##*.}"                    
      modified_name="${filename}_skull_stripped.${extension}"     
  fi
  echo
  echo
  echo
  echo
  echo -e "\033[1m....SKULL STRIPPING $namepart .... PLEASE BE PATIENT .... EACH MRI CAN TAKE A FEW MINUTES TO PROCESS ....\033[0m"
  echo
  echo
  echo
  echo
  cd ./oldsynthstrip  
  mri_synthstrip -i $FILE -o ./skull_stripped/$modified_name
  cd ..
done
cp ./skull_stripped/* /output
#################################################SYNTHSTRIP#################################################
#################################################SYNTHSTRIP#################################################



#################################################FSL#################################################
#################################################FSL#################################################
# Loop over the skull-stripped MRI files and create the masks with fsl
for FILE in ./skull_stripped/*; do
  namepart="$(basename "$FILE")"
  if [[ $namepart == *".nii.gz"* ]]  ; then
      filename="${namepart%%.*}" 
      extension="${namepart#*.}"                    
      modified_name="${filename}_mask.${extension}" ; else
      filename="${namepart%.*}" 
      extension="${namepart##*.}"                    
      modified_name="${filename}_mask.${extension}"   
  fi  
  echo
  echo
  echo
  echo
  echo -e "\033[1m....CREATING MASK FOR $namepart .... EACH MRI SHOULD TAKE ONLY A FEW SECONDS TO PROCESS ....\033[0m"
  echo
  echo
  echo
  echo
  fslmaths $FILE -abs -bin /output/$modified_name
done
#################################################FSL#################################################
#################################################FSL#################################################

It does not find the function synthstrip from the first image with the error ./iterative_loop.sh: line 33: ./oldsynthstrip/mri_synthstrip: No such file or directory

I also changed

  cd ./oldsynthstrip  
  mri_synthstrip -i $FILE -o ./skull_stripped/$modified_name
  cd ..

to

./oldsynthstrip/mri_synthstrip -i $FILE -o ./skull_stripped/$modified_name

which gives the same error

I guess my question is:

  1. How to copy all files and folders from first image so that it works in the next image
  2. I know that that is extremely inefficient, if there is a way to figure out what part of the first image I should copy for the function to work?

I apologize if this is straightforward, but I could not find an appropriate answer in related posts and I am new to Docker.

Any help appreciated.

Thank you very much

Solution

  • I'd approach this problem by writing a single shell script that runs on the host, that invokes the various Docker images. You can use the docker run -v option to make parts of the host filesystem appear in the container so that the tools can use them. The overall shell script would look like any of the scripts you have in the question, except that you have a docker run --rm ... command instead of directly invoking the tool on the host.

    Reading those two images' documentation, a basic wrapper script could look like:

    #!/bin/sh

# Create the intermediate/output directories if needed
mkdir -p MRIs_first_modifications output

# Transform the input files
for f in input/*.nii.gz; do
  # Rewrite the filename
  local_name="${f#input/}"
  output_name="${local_name%.nii.gz}_skull_stripped.nii.gz"

  # Run the tool
  docker run --rm \
    -v "$PWD/input:/input" \
    -v "$PWD/MRIs_first_modifications:/output" \
    -u "$(id -u)" \
    freesurfer/synthstrip \
    -i "/input/$local_name" -o "/output/$output_name"
done

# Second stage, using the next image
for f in MRIs_first_modifications/*_skull_stripped.nii.gz; do
  local_name="${f#MRIs_first_modifications/}"
  output_name="${local_name%_skull_stripped.nii.gz}_mask.nii.gz"

  docker run --rm \
    -v "$PWD/MRIs_first_modifications:/input" \
    -v "$PWD/output:/output" \
    -u "$(id -u)" \
    vistalab/fsl-v5.0 \
    bet2 -i "/input/$local_name" -o "/output/$output_name"
done

    The exact way you run the two images is slightly different and is image-dependent. In both cases the source directory (whatever it is) is mounted on /input in the container, and the result directory is on /output, and you tell the embedded command-line tool to read from /input and write to /output. The docker run --rm option cleans up the container when it's done, and the docker run -u option uses the host's numeric user ID so that file permissions are correct.

    This approach doesn't try to use a Dockerfile as an orchestration tool. You could distribute the script to your users and they could just run it as-is, without requiring any specific knowledge of what it does and without requiring any special tools beyond an ordinary Bourne shell plus Docker.

    (This would also fit well with a Makefile-based setup, which would include some advantages like being able to do incremental updates and having a simpler approach to matching the input and output filenames.)

    A Docker image does not have "functions" per se. Copying an executable from one image to another is often possible, but fraught with potential problems such as shared-library dependencies (the second image you've linked to here is extremely old by container-image standards). Inside an image you can't directly invoke another image or run a program that only exists inside another image. That's a major part of what leads me to recommend a host-based shell script here.

    A Docker image also doesn't seem like the right final output to me as you've described this problem. You wouldn't usually try to keep your data in an image. In principle you could use a multi-stage build and COPY the results from one stage to the next one, but you'd still have a challenge to get the final results out (the docker create && docker cp && docker rm invocation isn't something a casual user knows off hand).