On Fri, Jul 16, 2021 at 1:59 PM Vojtech Juranek <vjuranek@redhat.com> wrote:
On Friday, 16 July 2021 12:31:34 CEST Gianluca Cecchi wrote:
> On Fri, Jul 16, 2021 at 11:15 AM Vojtech Juranek <vjuranek@redhat.com>
> wrote:
> > > What to do to crosscheck what is using the device and so preventing the
> > > "-f" to complete?
> >
> > can you try
> >
> >     dmsetup info  /dev/mapper/36090a0d800851c9d2195d5b837c9e328
> >
> > and check "Open count" filed to see if there is still anything open?
> >
> > Also, you can try
> >
> >     fuser /dev/dm-2
> >
> > to see which process is using the device
> [root@ov301 ~]# dmsetup info  /dev/mapper/36090a0d800851c9d2195d5b837c9e328
> Name:              36090a0d800851c9d2195d5b837c9e328
> State:             ACTIVE
> Read Ahead:        256
> Tables present:    LIVE
> Open count:        1

This means there's some open connection. As lsof or fuser doesn't show
anything I wonder how this could happen.

Theoretically (not tested as I actually don't know how to reproduce this) and
on your own risk:-), you can try

    dmsetup suspend /dev/mapper/36090a0d800851c9d2195d5b837c9e328
    dmsetup clear /dev/mapper/36090a0d800851c9d2195d5b837c9e328
    dmsetup wipe_table /dev/mapper/36090a0d800851c9d2195d5b837c9e328

which should remove any stale connection. After that dmsetup info should show
Open count 0 and multipath -f 36090a0d800851c9d2195d5b837c9e328 should work

The host doesn't see the storage any more, and anyway it's a test system where I try with oVirt, before going with oVirt itself or RHV in production.

[root@ov301 ~]# dmsetup suspend /dev/mapper/36090a0d800851c9d2195d5b837c9e328
[root@ov301 ~]# dmsetup clear /dev/mapper/36090a0d800851c9d2195d5b837c9e328
[root@ov301 ~]#  dmsetup wipe_table /dev/mapper/36090a0d800851c9d2195d5b837c9e328

But still
[root@ov301 ~]# dmsetup info  /dev/mapper/36090a0d800851c9d2195d5b837c9e328
Name:              36090a0d800851c9d2195d5b837c9e328
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 2
Number of targets: 1
UUID: mpath-36090a0d800851c9d2195d5b837c9e328

Anyway the removal operation now goes ok:
[root@ov301 ~]# multipath -f 36090a0d800851c9d2195d5b837c9e328
[root@ov301 ~]# echo $?

and no multipath device in my output

[root@ov301 ~]# multipath -l
364817197c52f98316900666e8c2b0b2b dm-14 EQLOGIC,100E-00
size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 16:0:0:0 sde 8:64 active undef running
  `- 17:0:0:0 sdf 8:80 active undef running
[root@ov301 ~]#
In /var/log/messages, during the sequence of commands above I see:

Jul 16 14:08:20 ov301 multipathd[1580]: 36090a0d800851c9d2195d5b837c9e328: removing map by alias
Jul 16 14:08:20 ov301 multipath[2229532]: dm-2 is not a multipath map
Jul 16 14:09:03 ov301 multipathd[1580]: 36090a0d800851c9d2195d5b837c9e328: remove map (operator)
Jul 16 14:09:03 ov301 multipathd[1580]: 36090a0d800851c9d2195d5b837c9e328: devmap not registered, can't remove

Thanks for the moment...

I'm going to do similar storage moving and decommissioning of the old one for 4 other storage domains (two of them iSCSI -> iSCSI, two of them iSCSI -> FC) belonging to RHV environments (4.4.6 at the moment) in the next weeks, so in case I'm going to open a case for them if I find the same strange behavior.