ubuntu-18.04 singularity-container squashfs

squashfs error when running singularity after Ubuntu upgrade

Overview:

I am working with code in a singularity image on a machine running Ubuntu 18.04. It was working (well, I was debugging the code) as of Tuesday evening on machine A. I ran an update and as of the next day any attempt to run any singularity container gave me the following error:

ERROR  : Failed to mount squashfs image in (read only): Invalid argument
ABORT  : Retval = 255

After confirming that It wasn't due to my code (by simply running singularity exec <my_image_file>.sif and getting the same error, I remembered I have a nearly identical setup on another machine (B). I was able to run the image there with no issues.

Unfortunately, I forgot I had "unattended updates" selected on that machine and rebooted. It now returns the same error.

My Attempts:

I'm a little empty here. All I have done is convince myself that it's not directly related to my code and is almost certainly related to something that updated. I don't really know much at all about squashfs and am relatively new to singularity. I have not found anything online with this exact error (the "Invalid argument" part). As a Hail Mary I reinstalled singularity on machine A and it did not change.

I am not convinced that this is an issue with singularity itself and I am using an old version, so I don't think this is a valid github issue there. I am using version 2.5.2 because that is the version being used on the cluster this code is meant for.

Details

-vv log from singularity (note: identical feedback no matter what args I pass):

Increasing verbosity level (4)
Singularity version: 2.5.2-dist
Exec'ing: /usr/local/libexec/singularity/cli/exec.exec
Evaluating args: '<my_image_file>.sif <my_program> <program_arg>'
VERBOSE: Set messagelevel to: 4
VERBOSE: Initialize configuration file: /usr/local/etc/singularity/singularity.conf
VERBOSE: Got config key allow setuid = 'yes'
VERBOSE: Got config key max loop devices = '256'
VERBOSE: Got config key allow pid ns = 'yes'
VERBOSE: Got config key config passwd = 'yes'
VERBOSE: Got config key config group = 'yes'
VERBOSE: Got config key config resolv_conf = 'yes'
VERBOSE: Got config key mount proc = 'yes'
VERBOSE: Got config key mount sys = 'yes'
VERBOSE: Got config key mount dev = 'yes'
VERBOSE: Got config key mount devpts = 'yes'
VERBOSE: Got config key mount home = 'yes'
VERBOSE: Got config key mount tmp = 'yes'
VERBOSE: Got config key mount hostfs = 'no'
VERBOSE: Got config key bind path = '/etc/localtime'
VERBOSE: Got config key bind path = '/etc/hosts'
VERBOSE: Got config key user bind control = 'yes'
VERBOSE: Got config key enable overlay = 'try'
VERBOSE: Got config key mount slave = 'yes'
VERBOSE: Got config key sessiondir max size = '16'
VERBOSE: Got config key allow container squashfs = 'yes'
VERBOSE: Got config key allow container extfs = 'yes'
VERBOSE: Got config key allow container dir = 'yes'
VERBOSE: Initializing Singularity Registry
VERBOSE: Adding value to registry: 'COMMAND' = 'exec'
VERBOSE: Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc'
VERBOSE: Adding value to registry: 'VERSION' = '2.5.2-dist'
VERBOSE: Adding value to registry: 'IMAGE' = '<my_image_file>.sif'
VERBOSE: Adding value to registry: 'MESSAGELEVEL' = '4'
VERBOSE: Adding value to registry: 'BINDIR' = '/usr/local/bin'
VERBOSE: Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var'
VERBOSE: Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec'
VERBOSE: Set home (via getpwuid()) to: <my_home_directory>
VERBOSE: Running SUID program workflow
VERBOSE: Checking program has appropriate permissions
VERBOSE: Checking configuration file is properly owned by root
VERBOSE: Checking if singularity.conf allows us to run as suid
VERBOSE: Invoking the user namespace
VERBOSE: Not virtualizing USER namespace: running as SUID
VERBOSE: No autofs bug path in configuration, skipping
VERBOSE: Instantiating read only container image object
VERBOSE: Checking that file pointer is a Singularity image
VERBOSE: File is a valid SquashFS image
VERBOSE: Adding value to registry: 'CLEANUPD_FD' = '-1'
VERBOSE: Not virtualizing IPC namespace on user request
VERBOSE: Not virtualizing PID namespace on user request
VERBOSE: Not virtualizing network namespace on user request
VERBOSE: Using session directory: /usr/local/var/singularity/mnt/session
VERBOSE: Adding value to registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session'
VERBOSE: Found available loop device: /dev/loop18
VERBOSE: Using loop device: /dev/loop18
VERBOSE: Mounting squashfs image: /dev/loop18 -> /usr/local/var/singularity/mnt/container
ERROR  : Failed to mount squashfs image in (read only): Invalid argument
ABORT  : Retval = 255

grep for 'upgrade' in dpkg.log:

2020-07-23 06:10:36 upgrade evolution-data-server-common:all 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:10:41 upgrade linux-libc-dev:amd64 4.15.0-111.112 4.15.0-112.113
2020-07-23 06:10:46 upgrade python-pil:amd64 5.1.0-1ubuntu0.2 5.1.0-1ubuntu0.3
2020-07-23 06:10:51 upgrade python3-pil:amd64 5.1.0-1ubuntu0.2 5.1.0-1ubuntu0.3
2020-07-23 06:10:56 upgrade python2.7:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:57 upgrade libpython2.7:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:57 upgrade libpython2.7-stdlib:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:58 upgrade python2.7-minimal:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:10:58 upgrade libpython2.7-minimal:amd64 2.7.17-1~18.04ubuntu1 2.7.17-1~18.04ubuntu1.1
2020-07-23 06:11:06 upgrade python3.6-venv:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:06 upgrade python3.6-dev:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:06 upgrade libpython3.6-dev:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:11 upgrade libpython3.6:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:11 upgrade python3.6:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:11 upgrade libpython3.6-stdlib:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:12 upgrade python3.6-minimal:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:12 upgrade libpython3.6-minimal:amd64 3.6.9-1~18.04ubuntu1 3.6.9-1~18.04ubuntu1.1
2020-07-23 06:11:29 upgrade linux-generic-hwe-18.04:amd64 5.3.0.62.115 5.4.0.42.46~18.04.35
2020-07-23 06:11:29 upgrade linux-image-generic-hwe-18.04:amd64 5.3.0.62.115 5.4.0.42.46~18.04.35
2020-07-23 06:11:41 upgrade linux-headers-generic-hwe-18.04:amd64 5.3.0.62.115 5.4.0.42.46~18.04.35
2020-07-23 06:12:37 upgrade evolution-data-server:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:37 upgrade libebook-1.2-19:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libedata-book-1.2-25:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libedata-cal-1.2-28:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libebook-contacts-1.2-2:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libebackend-1.2-10:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libecal-1.2-19:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:38 upgrade libedataserver-1.2-23:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:39 upgrade libcamel-1.2-61:amd64 3.28.5-0ubuntu0.18.04.2 3.28.5-0ubuntu0.18.04.3
2020-07-23 06:12:44 upgrade libavformat-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:44 upgrade libavcodec-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswresample-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswresample2:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswscale-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:45 upgrade libswscale4:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavresample-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavresample3:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavutil-dev:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavutil55:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:46 upgrade libavcodec57:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:47 upgrade libavformat57:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:52 upgrade libpostproc54:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:12:57 upgrade libavfilter6:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:13:02 upgrade libavdevice57:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2
2020-07-23 06:13:06 upgrade ffmpeg:amd64 7:3.4.6-0ubuntu0.18.04.1 7:3.4.8-0ubuntu0.2

The image is built from a Docker image with a few extra steps. I'm not supplying any info on that since the behavior doesn't seem to depend on when I built the image.

Solution

I had this exact problem and I managed to fix it in Ubuntu 18.04 with singularity version 2.6.1 (I don't think the solution will be much different for you in 2.5.2 though).

Steps to take

Uninstall earlier versions of Singularity if needed.
Download tarball of the correct Singularity version by opening a Terminal and running the following commands:

# Go to the place you want to work in
cd ~
VERSION=2.6.1  # Set an environment variable just for convenience, could replace with another version.
# Download
wget https://github.com/singularityware/singularity/releases/download/$VERSION/singularity-$VERSION.tar.gz
# Extract
tar xvf singularity-$VERSION.tar.gz

Now, there should be a folder called singularity-2.6.1 (for me it was placed in Home (~). Then open the file mount.c placed in singularity-2.6.1/src/lib/image/squashfs/mount.c (I am not sure if this is the same for v2.5.2 but I assume not that much has changed. We might be out of luck though)
In this file find the line of code

if ( singularity_mount(loop_dev, mount_point, "squashfs", MS_NOSUID|MS_RDONLY|MS_NODEV, "errors=remount-ro") < 0 )

Change it to

if ( singularity_mount(loop_dev, mount_point, "squashfs", MS_NOSUID|MS_RDONLY|MS_NODEV, "") < 0 )

Don't forget to save!

Return to the Terminal to install Singularity from the edited installation files by running the commands

cd singularity-$VERSION
./configure --prefix=/usr/local
make
sudo make install

For me, the problem was now fixed!

EDIT

As pointed out by aguadopd, the make command may not work if you do not have libarchive-dev installed. If this is the case, then first fix it by running in a terminal

sudo apt-get update -y
sudo apt-get install -y libarchive-dev

Then in a new terminal

singularity shell -p $HOME/Downloads/hrwros-09.simg

Problem rationale

Apparently, there used to be a bug in squashfs such that it simply ignored arguments if it didn't know it. When this was fixed earlier this year, it started giving errors when encountering an invalid argument, as happened in our case. (Also explained here https://www.mail-archive.com/debian-kernel@lists.debian.org/msg117435.html) Therefore, removing the unknown argument "errors=remount-ro" fixed the problem.

Since I'm honestly 100% new to this, I'm not sure if it will induce any new errors at other places. However, this is also how this error was basically fixed in a later release of Singularity (https://github.com/hpcng/singularity/pull/4802/files/3f3136be150d9a8f63e58a0b4e39dc8487f1c0dc).

Hope it helped!