filesystem are unmounted before the services are stopped in systemd

I am debugging systemd shutdown issue. Here the problem is some of the filesystems are unmounted while the services are still running.

In general, we want systemd to shutdown the services first and then umount the mount points.

But here, umount and stopping the services are happening in parallel. (See below). Also unmounting of root filesystem at first.

#        Unmounting /root...
         Unmounting /var/lib/ntp...
         Unmounting /etc/cron.d/local...
[  OK  ] Stopped Apply Kernel Variables.
         Unmounting /proc/fs/nfsd...
         Unmounting /tmp/rshell/trace...
         Stopping Availability of block devices...
         Unmounting /etc/timestamp...
         Unmounting /var/lib/nfs/rpc_pipefs...
         Unmounting /etc/sysconfig/clock...
[  OK  ] Removed slice system-getty.slice.
[  OK  ] Stopped Load Kernel Modules.
         Unmounting /etc/ssh/ssh_external_host_rsa_key...
[  OK  ] Stopped Create Static Device Nodes in /dev.
         Unmounting /mnt/log...
[  OK  ] Stopped Resets System Activity Logs.
         Stopping Crond Periodic Command Scheduler...
[  OK  ] Stopped Mount Restricted SFTP Jail Folders.
[  OK  ] Stopped Mount Restricted Shell Folders.
         Stopping Runs processtat...
         Unmounting /etc/ssh/ssh_external_host_ecdsa_key.pub...
[  OK  ] Stopped target RPC Port Mapper.
         Unmounting /boot...
         Unmounting /srv...
         Stopping Initializes network console logging...
[  OK  ] Stopped Crond Periodic Command Scheduler.
[FAILED] Failed unmounting /root.
[  OK  ] Unmounted /var/lib/ntp.
[  OK  ] Unmounted /etc/cron.d/local.
**[FAILED] Failed unmounting /mnt/sysimg.**
[  OK  ] Unmounted /etc/sysconfig/clock.
[  OK  ] Unmounted /usr/share/cracklib.
**[FAILED] Failed unmounting /run/netns.**
[  OK  ] Unmounted /tftpboot/state.
**[FAILED] Failed unmounting /tmp/ldap.**

How do we synchronise this in systemd?

In general, systemd-reboot.service depends on final.target, shutdown.target and umount.target.

Here looks like, umount.target and shutdown.target are executed in parallel.

# cat /usr/lib/systemd/system/systemd-reboot.service 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Reboot
Documentation=man:systemd-halt.service(8)
DefaultDependencies=no
Requires=shutdown.target umount.target final.target
After=shutdown.target umount.target final.target

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl --force reboot

I tried, umount.target being dependent on shutdown.target but that did not help. Always these umount and service shutdown seems to be happening in parallel. If my understanding is wrong please correct.

Please give some tips/suggestions to properly shutdown the services first and then unmount the mount points.

In your service unit, try the following

BindsTo=mymount.mount
After=mymount.mount

mymount.mount must already exist. In my case I let systemd generate /var/run/systemd/generator/mymount.mount based on /etc/fstab contents.

BindsTo= means the mount must be started for the service to start and the service must be stopped if the mount is stopped. But both units can still be started and stopped (almost) in parallel. You need another constraint. The mount must reach active state before the service can start. You'll achieve this with After=. As with other systemd directives, what After= does during start is inverted during stop: the service will have to be fully stopped before the mount is stopped.

The relevant documentation with emphasis added. The doc does not mention what happens during the stop but I confirmed it works as we expect..

BindsTo=

Configures requirement dependencies, very similar in style to Requires=. However, this dependency type is stronger: in addition to the effect of Requires= it declares that if the unit bound to is stopped, this unit will be stopped too. This means a unit bound to another unit that suddenly enters inactive state will be stopped too. Units can suddenly, unexpectedly enter inactive state for different reasons: the main process of a service unit might terminate on its own choice, the backing device of a device unit might be unplugged or the mount point of a mount unit might be unmounted without involvement of the system and service manager.

When used in conjunction with After= on the same unit the behaviour of BindsTo= is even stronger. In this case, the unit bound to strictly has to be in active state for this unit to also be in active state. This not only means a unit bound to another unit that suddenly enters inactive state, but also one that is bound to another unit that gets skipped due to a failed condition check (such as ConditionPathExists=, ConditionPathIsSymbolicLink=, … — see below) will be stopped, should it be running. Hence, in many cases it is best to combine BindsTo= with After=.

When BindsTo=b.service is used on a.service, this dependency will show as BoundBy=a.service in property listing of b.service. BoundBy= dependency cannot be specified directly.