• cryptmount and mount not playing nicely together

    From William Unruh@2:250/1 to All on Saturday, December 02, 2023 20:30:53
    I have two a regular mount point /fastlocal, which is a partition on an
    SSD. On top of various items in /fastlocal, I cryptmount a variety of
    encrypted partitions. But this is leading to trouble. After mounting the encrypted partition, I ran mount -a. Now, it should NOT have remounted /fastlocal, since it was already mounted. But it did. Mount now says
    that there are two mounts of fastmount
    /dev/sda7 on /fastlocal type ext4 (rw,noatime)
    /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
    And now the
    cryptmount partitions were buried. Ie, all I find in /fastlocal is the
    empty mountpoints for the various cryptmount partitions. I try to
    unmount the cryptmount partions but they all tell me that they are in
    use and cannot be umounted. I try to unmount /fastlocal, and it tells me
    that cannot do so as it is in use. So I am stuck.
    If I were not manymiles away from the machine, I could reboot, but
    somehow a usb dive got left in the machine, and when I try to reboot, it
    tries to reboot from the usbdrive (it is a copy of the Mga8 installation
    disk). And my arm is not long enough to reach it. Is there any way out
    of this predicament.
    And this appears to be a bug in mount and/or cryptmount. Ie, the second
    mount of /fastlocal covers up the cryptmount mounts.



    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)
  • From William Unruh@2:250/1 to All on Saturday, December 02, 2023 21:46:47
    On 2023-12-02, William Unruh <unruh@invalid.ca> wrote:
    I have two a regular mount point /fastlocal, which is a partition on an
    SSD. On top of various items in /fastlocal, I cryptmount a variety of encrypted partitions. But this is leading to trouble. After mounting the encrypted partition, I ran mount -a. Now, it should NOT have remounted /fastlocal, since it was already mounted. But it did. Mount now says
    that there are two mounts of fastmount
    /dev/sda7 on /fastlocal type ext4 (rw,noatime)
    /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
    And now the
    cryptmount partitions were buried. Ie, all I find in /fastlocal is the
    empty mountpoints for the various cryptmount partitions. I try to
    unmount the cryptmount partions but they all tell me that they are in
    use and cannot be umounted. I try to unmount /fastlocal, and it tells me
    that cannot do so as it is in use. So I am stuck.
    If I were not manymiles away from the machine, I could reboot, but
    somehow a usb dive got left in the machine, and when I try to reboot, it tries to reboot from the usbdrive (it is a copy of the Mga8 installation disk). And my arm is not long enough to reach it. Is there any way out
    of this predicament.
    And this appears to be a bug in mount and/or cryptmount. Ie, the second
    mount of /fastlocal covers up the cryptmount mounts.



    It is even stupider than I thought.
    When I installed the SSD, a few years ago, Ihad a homedirectory on /disk9/home/unruh
    When I installed the SSD, I installed my home directory onto the SSD
    whcih was mounted on /fastlocal.
    This worked fine but just in case there were some old references in some software script to /disk9/home/unruh, I made a soft link to
    /fastlocal/unruh. Then sometime, I put in a bind mount from
    /fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
    by hand after the system had booted up, this did not matter, but if I
    did a mount -a, that mount from /fastlocal/unruh on top of
    /fastlocal/unruh covered up the cryptmount.
    Now the bug in the system meant that if I tried to umount
    /fastlocal/unruh, I think it tried to do that first mount, instead of
    the last mount. and the cryptmounts sat on top of it. and stopped the
    unmount. But the crytpmounts were now covered by the last
    /fastlocal/unruh mount and the system was all tied up and unfixable.

    Or is there some way I can get out of this mess without rebooting?



    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)
  • From jim whitby@2:250/1 to All on Saturday, December 02, 2023 22:42:29
    On 12/2/23 16:46, William Unruh wrote:
    On 2023-12-02, William Unruh <unruh@invalid.ca> wrote:
    I have two a regular mount point /fastlocal, which is a partition on an
    SSD. On top of various items in /fastlocal, I cryptmount a variety of
    encrypted partitions. But this is leading to trouble. After mounting the
    encrypted partition, I ran mount -a. Now, it should NOT have remounted
    /fastlocal, since it was already mounted. But it did. Mount now says
    that there are two mounts of fastmount
    /dev/sda7 on /fastlocal type ext4 (rw,noatime)
    /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
    And now the
    cryptmount partitions were buried. Ie, all I find in /fastlocal is the
    empty mountpoints for the various cryptmount partitions. I try to
    unmount the cryptmount partions but they all tell me that they are in
    use and cannot be umounted. I try to unmount /fastlocal, and it tells me
    that cannot do so as it is in use. So I am stuck.
    If I were not manymiles away from the machine, I could reboot, but
    somehow a usb dive got left in the machine, and when I try to reboot, it
    tries to reboot from the usbdrive (it is a copy of the Mga8 installation
    disk). And my arm is not long enough to reach it. Is there any way out
    of this predicament.
    And this appears to be a bug in mount and/or cryptmount. Ie, the second
    mount of /fastlocal covers up the cryptmount mounts.



    It is even stupider than I thought.
    When I installed the SSD, a few years ago, Ihad a homedirectory on /disk9/home/unruh
    When I installed the SSD, I installed my home directory onto the SSD
    whcih was mounted on /fastlocal.
    This worked fine but just in case there were some old references in some software script to /disk9/home/unruh, I made a soft link to
    /fastlocal/unruh. Then sometime, I put in a bind mount from
    /fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
    by hand after the system had booted up, this did not matter, but if I
    did a mount -a, that mount from /fastlocal/unruh on top of
    /fastlocal/unruh covered up the cryptmount.
    Now the bug in the system meant that if I tried to umount
    /fastlocal/unruh, I think it tried to do that first mount, instead of
    the last mount. and the cryptmounts sat on top of it. and stopped the unmount. But the crytpmounts were now covered by the last
    /fastlocal/unruh mount and the system was all tied up and unfixable.

    Or is there some way I can get out of this mess without rebooting?


    Can you unmount by using using umount /dev/sd???


    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: XSUsenet (2:250/1@fidonet)
  • From David W. Hodgins@2:250/1 to All on Saturday, December 02, 2023 23:10:53
    On Sat, 02 Dec 2023 15:30:53 -0500, William Unruh <unruh@invalid.ca> wrote:

    I have two a regular mount point /fastlocal, which is a partition on an
    SSD. On top of various items in /fastlocal, I cryptmount a variety of encrypted partitions. But this is leading to trouble. After mounting the encrypted partition, I ran mount -a. Now, it should NOT have remounted /fastlocal, since it was already mounted. But it did. Mount now says
    that there are two mounts of fastmount
    /dev/sda7 on /fastlocal type ext4 (rw,noatime)
    /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
    And now the
    cryptmount partitions were buried. Ie, all I find in /fastlocal is the
    empty mountpoints for the various cryptmount partitions. I try to
    unmount the cryptmount partions but they all tell me that they are in
    use and cannot be umounted. I try to unmount /fastlocal, and it tells me
    that cannot do so as it is in use. So I am stuck.
    If I were not manymiles away from the machine, I could reboot, but
    somehow a usb dive got left in the machine, and when I try to reboot, it tries to reboot from the usbdrive (it is a copy of the Mga8 installation disk). And my arm is not long enough to reach it. Is there any way out
    of this predicament.
    And this appears to be a bug in mount and/or cryptmount. Ie, the second
    mount of /fastlocal covers up the cryptmount mounts.

    Mounting something on a directory does hide everything under it, including prior mounts on that directory.

    To unhide those things, unmount the second mount, so in this case
    "umount /fastlocal". It will only umount the most recent mount.

    Regards, Dave Hodgins

    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)
  • From William Unruh@2:250/1 to All on Sunday, December 03, 2023 15:41:16
    On 2023-12-02, jim whitby <jim@afraid.org> wrote:
    On 12/2/23 16:46, William Unruh wrote:
    On 2023-12-02, William Unruh <unruh@invalid.ca> wrote:
    I have two a regular mount point /fastlocal, which is a partition on an
    SSD. On top of various items in /fastlocal, I cryptmount a variety of
    encrypted partitions. But this is leading to trouble. After mounting the >>> encrypted partition, I ran mount -a. Now, it should NOT have remounted
    /fastlocal, since it was already mounted. But it did. Mount now says
    that there are two mounts of fastmount
    /dev/sda7 on /fastlocal type ext4 (rw,noatime)
    /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
    And now the
    cryptmount partitions were buried. Ie, all I find in /fastlocal is the
    empty mountpoints for the various cryptmount partitions. I try to
    unmount the cryptmount partions but they all tell me that they are in
    use and cannot be umounted. I try to unmount /fastlocal, and it tells me >>> that cannot do so as it is in use. So I am stuck.
    If I were not manymiles away from the machine, I could reboot, but
    somehow a usb dive got left in the machine, and when I try to reboot, it >>> tries to reboot from the usbdrive (it is a copy of the Mga8 installation >>> disk). And my arm is not long enough to reach it. Is there any way out
    of this predicament.
    And this appears to be a bug in mount and/or cryptmount. Ie, the second
    mount of /fastlocal covers up the cryptmount mounts.



    It is even stupider than I thought.
    When I installed the SSD, a few years ago, Ihad a homedirectory on
    /disk9/home/unruh
    When I installed the SSD, I installed my home directory onto the SSD
    whcih was mounted on /fastlocal.
    This worked fine but just in case there were some old references in some
    software script to /disk9/home/unruh, I made a soft link to
    /fastlocal/unruh. Then sometime, I put in a bind mount from
    /fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
    by hand after the system had booted up, this did not matter, but if I
    did a mount -a, that mount from /fastlocal/unruh on top of
    /fastlocal/unruh covered up the cryptmount.
    Now the bug in the system meant that if I tried to umount
    /fastlocal/unruh, I think it tried to do that first mount, instead of
    the last mount. and the cryptmounts sat on top of it. and stopped the
    unmount. But the crytpmounts were now covered by the last
    /fastlocal/unruh mount and the system was all tied up and unfixable.

    Or is there some way I can get out of this mess without rebooting?


    Can you unmount by using using umount /dev/sd???


    Nope, does not work. Complains "Target is busy".
    Not sure what it thinks the target is since in /etc/fstab, the target is /disk9/home/unruh
    which is a softlink to /fastlocal/unruh, which seems to have resulted in /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
    mounting a directory on top of itself should be illegal and not work.
    What is interesting is that other mounts onto directories inside /fastlocal/unruh get hidden by the mount of the directory on top of
    itself, and then make the directory busy.



    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)
  • From David W. Hodgins@2:250/1 to All on Sunday, December 03, 2023 19:58:41
    On Sun, 03 Dec 2023 10:41:16 -0500, William Unruh <unruh@invalid.ca> wrote:
    Nope, does not work. Complains "Target is busy".
    Not sure what it thinks the target is since in /etc/fstab, the target is /disk9/home/unruh
    which is a softlink to /fastlocal/unruh, which seems to have resulted in /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
    mounting a directory on top of itself should be illegal and not work.
    What is interesting is that other mounts onto directories inside /fastlocal/unruh get hidden by the mount of the directory on top of
    itself, and then make the directory busy.

    Don't try to unmount the device, as that's used by both mounts (directly or indirectly). umount the directory used in the second mount, to reveal the directory and it's contents used in the first mount.

    If the second mount is busy, something started after the second mount is using a directory or file inside of that second mount. It could be something such
    as the current directory inside of a terminal in a konsole tab, or a program such as tracker monitoring the file system for changes to index.

    The program lsof can help identify those programs.

    Regards, Dave Hodgins

    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)
  • From William Unruh@2:250/1 to All on Monday, December 04, 2023 00:03:19
    On 2023-12-03, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
    On Sun, 03 Dec 2023 10:41:16 -0500, William Unruh <unruh@invalid.ca> wrote:
    Nope, does not work. Complains "Target is busy".
    Not sure what it thinks the target is since in /etc/fstab, the target is
    /disk9/home/unruh
    which is a softlink to /fastlocal/unruh, which seems to have resulted in
    /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
    mounting a directory on top of itself should be illegal and not work.
    What is interesting is that other mounts onto directories inside
    /fastlocal/unruh get hidden by the mount of the directory on top of
    itself, and then make the directory busy.

    Don't try to unmount the device, as that's used by both mounts (directly or indirectly). umount the directory used in the second mount, to reveal the directory and it's contents used in the first mount.

    If the second mount is busy, something started after the second mount is using
    a directory or file inside of that second mount. It could be something such as the current directory inside of a terminal in a konsole tab, or a program such as tracker monitoring the file system for changes to index.

    The program lsof can help identify those programs.

    Unfortunately neither lsof nor fuser indicate that anything is using
    that directory.
    Also it is not mounted on any remote nfs client. findmnt indicates that
    the various cryptmounts are mounted, but mount sees none of them and
    cryptmount -u says they are not mounted, presumably because they are
    hidden under the second mount of /fastlocal/unruh on top of itself.
    Presumably it is also because the second mount occured via the link.
    (ie it was in fstab
    /fastlocal/unruh /disk9/home/unruh none bind,nofail 0 0

    Regards, Dave Hodgins

    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)
  • From jim whitby@2:250/1 to All on Monday, December 04, 2023 14:50:49
    On 12/3/23 10:41, William Unruh wrote:
    <snip>>>>
    Can you unmount by using using umount /dev/sd???


    Nope, does not work. Complains "Target is busy".
    Not sure what it thinks the target is since in /etc/fstab, the target is /disk9/home/unruh
    which is a softlink to /fastlocal/unruh, which seems to have resulted in /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
    mounting a directory on top of itself should be illegal and not work.
    What is interesting is that other mounts onto directories inside /fastlocal/unruh get hidden by the mount of the directory on top of
    itself, and then make the directory busy.


    Part of man umount
    ....

    -f, --force
    Force an unmount (in case of an unreachable NFS system).

    Note that this option does not guarantee that umount command
    does not hang. It’s strongly
    recommended to use absolute paths without symlinks to avoid unwanted readlink(2) and stat(2) system
    calls on unreachable NFS in umount.
    ....

    -l, --lazy
    Lazy unmount. Detach the filesystem from the file hierarchy
    now, and clean up all references to this
    filesystem as soon as it is not busy anymore.

    A system reboot would be expected in near future if you’re
    going to use this option for network
    filesystem or local filesystem with submounts. The
    recommended use-case for umount -l is to prevent
    hangs on shutdown due to an unreachable network share where
    a normal umount will hang due to a
    downed server or a network partition. Remounts of the share
    will not be possible.


    Try the -l first

    Jim

    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: XSUsenet (2:250/1@fidonet)
  • From William Unruh@2:250/1 to All on Tuesday, December 05, 2023 19:22:14
    On 2023-12-04, jim whitby <jim@afraid.org> wrote:
    On 12/3/23 10:41, William Unruh wrote:
    <snip>>>>
    Can you unmount by using using umount /dev/sd???


    Nope, does not work. Complains "Target is busy".
    Not sure what it thinks the target is since in /etc/fstab, the target is /disk9/home/unruh
    which is a softlink to /fastlocal/unruh, which seems to have resulted in /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
    mounting a directory on top of itself should be illegal and not work.
    What is interesting is that other mounts onto directories inside /fastlocal/unruh get hidden by the mount of the directory on top of
    itself, and then make the directory busy.


    Part of man umount
    ...

    -f, --force
    Force an unmount (in case of an unreachable NFS system).

    Note that this option does not guarantee that umount command does not hang. It’s strongly
    recommended to use absolute paths without symlinks to avoid unwanted readlink(2) and stat(2) system
    calls on unreachable NFS in umount.

    None of these are nfs filesystems. they are all local. I have also found
    -f not to work even for nfs mounted filesystems.

    ...

    -l, --lazy
    Lazy unmount. Detach the filesystem from the file hierarchy
    now, and clean up all references to this
    filesystem as soon as it is not busy anymore.

    The problem is that I have no idea what it making them busy. In this
    case I suspect it is the fact that cryptmount filesystems are mounted on
    the first mounting /fastlocal/unruh, and the second one hides all of he cryptmounts. So I cannot unmount cryptmount partitions, and the system
    things that there is only the second mount and the cryptmounts are
    making the second one busy.
    If I do
    findmts
    I see only the one mount onto /fastlocal/unruh, not two of them.


    I finally "solved" the problem by getting someone who was able to get
    into the locked server room to unplug the usb stick from the machine
    which was stopping the reboot. Then I could reboot the machine making
    sure I had removed the indirect mount onto the /fastlocal/unruh via the
    solft link.
    (Ie, the second mount was a mount to /disk9/home/unruh which was a
    solftlink to /fastlocal/unruh, which the mount commend interpreted as a
    mount onto /fastlocal/unruh, even though a mount of that same partition
    already existed via a direct mount onto /fastlocal/unruh. Ie, my fstab
    had lines like
    /dev/sdb3 /fastlocal/unruh
    and
    /dev/sdb3 /disk9/home/unruh bind,....
    and disk9/home/unruh was a softlink to /fastlocal/unruh
    That second one did not occur during reboot, but did occur after I
    mounted the crytmounts onto some interior directories in
    /fastlocal/unruh and then ran mount -a
    If I tried to mount /dev/sdb3 directly onto /fastlocal/unruh, the mount
    command would complain that /dev/sdb3 was already mounted onto /fastlocal/unruh, but did not complain, I guess because I told it to
    mount onto the link /disk9/home/unruh. Ie, the error trap failed to
    complete the link and realise I was trying a double mount.

    Ie, mount should not allow bin mounts onto links, or should test that
    the link does not result in a double mount from the same partition onto
    the same mountpoint


    A system reboot would be expected in near future if you’re going to use this option for network
    filesystem or local filesystem with submounts. The
    recommended use-case for umount -l is to prevent
    hangs on shutdown due to an unreachable network share where
    a normal umount will hang due to a
    downed server or a network partition. Remounts of the share
    will not be possible.


    Try the -l first

    Jim

    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)
  • From Kirk_Rockstein@2:250/1 to All on Wednesday, December 06, 2023 18:04:18

    On 2023-12-05, William Unruh <unruh@invalid.ca> wrote:
    On 2023-12-04, jim whitby <jim@afraid.org> wrote:
    On 12/3/23 10:41, William Unruh wrote:
    <snip>>>>
    Can you unmount by using using umount /dev/sd???


    Nope, does not work. Complains "Target is busy".
    Not sure what it thinks the target is since in /etc/fstab, the target is >> > /disk9/home/unruh
    which is a softlink to /fastlocal/unruh, which seems to have resulted in >> > /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
    mounting a directory on top of itself should be illegal and not work.
    What is interesting is that other mounts onto directories inside
    /fastlocal/unruh get hidden by the mount of the directory on top of
    itself, and then make the directory busy.


    Part of man umount
    ...

    -f, --force
    Force an unmount (in case of an unreachable NFS system).

    Note that this option does not guarantee that umount command
    does not hang. It’s strongly
    recommended to use absolute paths without symlinks to avoid
    unwanted readlink(2) and stat(2) system
    calls on unreachable NFS in umount.

    None of these are nfs filesystems. they are all local. I have also found
    -f not to work even for nfs mounted filesystems.

    ...

    -l, --lazy
    Lazy unmount. Detach the filesystem from the file hierarchy
    now, and clean up all references to this
    filesystem as soon as it is not busy anymore.

    The problem is that I have no idea what it making them busy. In this
    case I suspect it is the fact that cryptmount filesystems are mounted on
    the first mounting /fastlocal/unruh, and the second one hides all of he cryptmounts. So I cannot unmount cryptmount partitions, and the system
    things that there is only the second mount and the cryptmounts are
    making the second one busy.
    If I do
    findmts
    I see only the one mount onto /fastlocal/unruh, not two of them.


    I finally "solved" the problem by getting someone who was able to get
    into the locked server room to unplug the usb stick from the machine
    which was stopping the reboot. Then I could reboot the machine making
    sure I had removed the indirect mount onto the /fastlocal/unruh via the
    solft link.
    (Ie, the second mount was a mount to /disk9/home/unruh which was a
    solftlink to /fastlocal/unruh, which the mount commend interpreted as a
    mount onto /fastlocal/unruh, even though a mount of that same partition already existed via a direct mount onto /fastlocal/unruh. Ie, my fstab
    had lines like
    /dev/sdb3 /fastlocal/unruh
    and
    /dev/sdb3 /disk9/home/unruh bind,....
    and disk9/home/unruh was a softlink to /fastlocal/unruh
    That second one did not occur during reboot, but did occur after I
    mounted the crytmounts onto some interior directories in
    /fastlocal/unruh and then ran mount -a
    If I tried to mount /dev/sdb3 directly onto /fastlocal/unruh, the mount command would complain that /dev/sdb3 was already mounted onto /fastlocal/unruh, but did not complain, I guess because I told it to
    mount onto the link /disk9/home/unruh. Ie, the error trap failed to
    complete the link and realise I was trying a double mount.

    Ie, mount should not allow bin mounts onto links, or should test that
    the link does not result in a double mount from the same partition onto
    the same mountpoint


    A system reboot would be expected in near future if you’re
    going to use this option for network
    filesystem or local filesystem with submounts. The
    recommended use-case for umount -l is to prevent
    hangs on shutdown due to an unreachable network share where
    a normal umount will hang due to a
    downed server or a network partition. Remounts of the share
    will not be possible.


    Try the -l first

    Jim

    Maybe Running:
    find /proc -name mounts | xargs grep /dev/sdb3
    then kill -9 the PID's of found offending processes/mounts.


    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: Air Applewood, The Linux Gateway to the UK & Eire (2:250/1@fidonet)
  • From David W. Hodgins@2:250/1 to All on Wednesday, December 06, 2023 22:37:53
    On Wed, 06 Dec 2023 13:04:18 -0500, Kirk_Rockstein <Kirk_Rockstein@nowhere.invalid> wrote:
    Maybe Running:
    find /proc -name mounts | xargs grep /dev/sdb3
    then kill -9 the PID's of found offending processes/mounts.

    That can be shortened to "grep /dev/sdb3 /proc/*/mounts", which shows what processes have been started after sdb3 was mounted, or mounted it.

    It does not help indicate which of those processes have open files within
    that mount.

    Regards, Dave Hodgins

    --- MBSE BBS v1.0.8.4 (Linux-x86_64)
    * Origin: A noiseless patient Spider (2:250/1@fidonet)