Correct..  on wwid  

 
I do want to make clear here.  that to geta around the error you must ADD  (not remove ) drives to /etc/lvm/lvm.conf  so oVirt Gluster can complete setup of drives.

[root@thor log]# cat /etc/lvm/lvm.conf |grep filter
# Broken for gluster in oVirt
#filter = ["a|^/dev/disk/by-id/lvm-pv-uuid-AAHPao-R62q-8aac-410x-ZdA7-UL4i-Bh2bwJ$|", "a|^/dev/disk/by-id/lvm-pv-uuid-bSnFU3-jtUj-AGds-07sw-zdYC-52fM-mujuvC$|", "r|.*|"]
# working for gluster wizard in oVirt
filter = ["a|^/dev/disk/by-id/lvm-pv-uuid-AAHPao-R62q-8aac-410x-ZdA7-UL4i-Bh2bwJ$|", "a|^/dev/disk/by-id/lvm-pv-uuid-bSnFU3-jtUj-AGds-07sw-zdYC-52fM-mujuvC$|", "a|^/dev/disk/by-id/wwn-0x5001b448b847be41$|", "r|.*|"]



On Tue, Sep 22, 2020 at 3:57 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
Obtaining the wwid is not exactly correct.
You can identify them via:

multipath -v4 | grep 'got wwid of'

Short example: 
[root@ovirt conf.d]# multipath -v4 | grep 'got wwid of'
Sep 22 22:55:58 | nvme0n1: got wwid of 'nvme.1cc1-324a31313230303131343036-414441544120535838323030504e50-00000001'
Sep 22 22:55:58 | sda: got wwid of 'TOSHIBA-TR200_Z7KB600SK46S'
Sep 22 22:55:58 | sdb: got wwid of 'ST500NM0011_Z1M00LM7'
Sep 22 22:55:58 | sdc: got wwid of 'WDC_WD5003ABYX-01WERA0_WD-WMAYP2303189'
Sep 22 22:55:58 | sdd: got wwid of 'WDC_WD15EADS-00P8B0_WD-WMAVU0115133'

Of course if you are planing to use only gluster it could be far easier to set:

[root@ovirt conf.d]# cat /etc/multipath/conf.d/blacklist.conf 
blacklist {
        devnode "*"
}



Best Regards,
Strahil Nikolov

В вторник, 22 септември 2020 г., 22:12:21 Гринуич+3, Nir Soffer <nsoffer@redhat.com> написа:





On Tue, Sep 22, 2020 at 1:50 AM Jeremey Wise <jeremey.wise@gmail.com> wrote:
>
>
> Agree about an NVMe Card being put under mpath control.

NVMe can be used via multipath, this is a new feature added in RHEL 8.1:
https://bugzilla.redhat.com/1498546

Of course when the NVMe device is local there is no point to use it
via multipath.
To avoid this, you need to blacklist the devices like this:

1. Find the device wwid

For NVMe, you need the device ID_WWN:

    $ udevadm info -q property /dev/nvme0n1 | grep ID_WWN
    ID_WWN=eui.5cd2e42a81a11f69

2. Add local blacklist file:

    $ mkdir /etc/multipath/conf.d
    $ cat /etc/multipath/conf.d/local.conf
    blacklist {
        wwid "eui.5cd2e42a81a11f69"
    }

3. Reconfigure multipath

    $ multipathd reconfigure

Gluster should do this for you automatically during installation, but
it does not
you can do this manually.

> I have not even gotten to that volume / issue.  My guess is something weird in CentOS / 4.18.0-193.19.1.el8_2.x86_64  kernel with NVMe block devices.
>
> I will post once I cross bridge of getting standard SSD volumes working
>
> On Mon, Sep 21, 2020 at 4:12 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
>>
>> Why is your NVME under multipath ? That doesn't make sense at all .
>> I have modified my multipath.conf to block all local disks . Also ,don't forget the '# VDSM PRIVATE' line somewhere in the top of the file.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>>
>>
>>
>>
>>
>> В понеделник, 21 септември 2020 г., 09:04:28 Гринуич+3, Jeremey Wise <jeremey.wise@gmail.com> написа:
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> vdo: ERROR - Device /dev/sdc excluded by a filter
>>
>>
>>
>>
>> Other server
>> vdo: ERROR - Device /dev/mapper/nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001p1 excluded by a filter.
>>
>>
>> All systems when I go to create VDO volume on blank drives.. I get this filter error.  All disk outside of the HCI wizard setup are now blocked from creating new Gluster volume group.
>>
>> Here is what I see in /dev/lvm/lvm.conf |grep filter
>> [root@odin ~]# cat /etc/lvm/lvm.conf |grep filter
>> filter = ["a|^/dev/disk/by-id/lvm-pv-uuid-e1fvwo-kEfX-v3lT-SKBp-cgze-TwsO-PtyvmC$|", "a|^/dev/disk/by-id/lvm-pv-uuid-mr9awW-oQH5-F4IX-CbEO-RgJZ-x4jK-e4YZS1$|", "r|.*|"]
>>
>> [root@odin ~]# ls -al /dev/disk/by-id/
>> total 0
>> drwxr-xr-x. 2 root root 1220 Sep 18 14:32 .
>> drwxr-xr-x. 6 root root  120 Sep 18 14:32 ..
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 ata-INTEL_SSDSC2BB080G4_BTWL40350DXP080KGN -> ../../sda
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 ata-INTEL_SSDSC2BB080G4_BTWL40350DXP080KGN-part1 -> ../../sda1
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 ata-INTEL_SSDSC2BB080G4_BTWL40350DXP080KGN-part2 -> ../../sda2
>> lrwxrwxrwx. 1 root root    9 Sep 18 14:32 ata-Micron_1100_MTFDDAV512TBN_17401F699137 -> ../../sdb
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 ata-WDC_WDS100T2B0B-00YS70_183533804564 -> ../../sdc
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-name-cl-home -> ../../dm-2
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-name-cl-root -> ../../dm-0
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-name-cl-swap -> ../../dm-1
>> lrwxrwxrwx. 1 root root  11 Sep 18 16:40 dm-name-gluster_vg_sdb-gluster_lv_data -> ../../dm-11
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-name-gluster_vg_sdb-gluster_lv_engine -> ../../dm-6
>> lrwxrwxrwx. 1 root root  11 Sep 18 16:40 dm-name-gluster_vg_sdb-gluster_lv_vmstore -> ../../dm-12
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:35 dm-name-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001 -> ../../dm-3
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:49 dm-name-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001p1 -> ../../dm-4
>> lrwxrwxrwx. 1 root root  10 Sep 18 14:32 dm-name-vdo_sdb -> ../../dm-5
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-uuid-LVM-GpvYIuypEfrR7nEDn5uHPenKwjrsn4ADc49gc6PWLRBCoJ2B3JC9tDJejyx5eDPT -> ../../dm-1
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-uuid-LVM-GpvYIuypEfrR7nEDn5uHPenKwjrsn4ADOMNJfgcat9ZLOpcNO7FyG8ixcl5s93TU -> ../../dm-2
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-uuid-LVM-GpvYIuypEfrR7nEDn5uHPenKwjrsn4ADzqPGk0yTQ19FIqgoAfsCxWg7cDMtl71r -> ../../dm-0
>> lrwxrwxrwx. 1 root root  10 Sep 18 16:40 dm-uuid-LVM-ikNfztYY7KGT1SI2WYXPz4DhM2cyTelOq6Om5comvRFWJDbtVZAKtE5YGl4jciP9 -> ../../dm-6
>> lrwxrwxrwx. 1 root root  11 Sep 18 16:40 dm-uuid-LVM-ikNfztYY7KGT1SI2WYXPz4DhM2cyTelOqVheASEgerWSEIkjM1BR3us3D9ekHt0L -> ../../dm-11
>> lrwxrwxrwx. 1 root root  11 Sep 18 16:40 dm-uuid-LVM-ikNfztYY7KGT1SI2WYXPz4DhM2cyTelOQz6vXuivIfup6cquKAjPof8wIGOSe4Vz -> ../../dm-12
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:35 dm-uuid-mpath-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001 -> ../../dm-3
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:49 dm-uuid-part1-mpath-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001 -> ../../dm-4
>> lrwxrwxrwx. 1 root root  10 Sep 18 14:32 dm-uuid-VDO-472035cc-8d2b-40ac-afe9-fa60b62a887f -> ../../dm-5
>> lrwxrwxrwx. 1 root root  10 Sep 18 14:32 lvm-pv-uuid-e1fvwo-kEfX-v3lT-SKBp-cgze-TwsO-PtyvmC -> ../../dm-5
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 lvm-pv-uuid-mr9awW-oQH5-F4IX-CbEO-RgJZ-x4jK-e4YZS1 -> ../../sda2
>> lrwxrwxrwx. 1 root root  13 Sep 18 14:32 nvme-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001 -> ../../nvme0n1
>> lrwxrwxrwx. 1 root root  15 Sep 18 14:32 nvme-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001-part1 -> ../../nvme0n1p1
>> lrwxrwxrwx. 1 root root  13 Sep 18 14:32 nvme-SPCC_M.2_PCIe_SSD_AA000000000000002458 -> ../../nvme0n1
>> lrwxrwxrwx. 1 root root  15 Sep 18 14:32 nvme-SPCC_M.2_PCIe_SSD_AA000000000000002458-part1 -> ../../nvme0n1p1
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-0ATA_INTEL_SSDSC2BB08_BTWL40350DXP080KGN -> ../../sda
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-0ATA_INTEL_SSDSC2BB08_BTWL40350DXP080KGN-part1 -> ../../sda1
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-0ATA_INTEL_SSDSC2BB08_BTWL40350DXP080KGN-part2 -> ../../sda2
>> lrwxrwxrwx. 1 root root    9 Sep 18 14:32 scsi-0ATA_Micron_1100_MTFD_17401F699137 -> ../../sdb
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-0ATA_WDC_WDS100T2B0B-_183533804564 -> ../../sdc
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-1ATA_INTEL_SSDSC2BB080G4_BTWL40350DXP080KGN -> ../../sda
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-1ATA_INTEL_SSDSC2BB080G4_BTWL40350DXP080KGN-part1 -> ../../sda1
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-1ATA_INTEL_SSDSC2BB080G4_BTWL40350DXP080KGN-part2 -> ../../sda2
>> lrwxrwxrwx. 1 root root    9 Sep 18 14:32 scsi-1ATA_Micron_1100_MTFDDAV512TBN_17401F699137 -> ../../sdb
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-1ATA_WDC_WDS100T2B0B-00YS70_183533804564 -> ../../sdc
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-35001b448b9608d90 -> ../../sdc
>> lrwxrwxrwx. 1 root root    9 Sep 18 14:32 scsi-3500a07511f699137 -> ../../sdb
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-355cd2e404b581cc0 -> ../../sda
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-355cd2e404b581cc0-part1 -> ../../sda1
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-355cd2e404b581cc0-part2 -> ../../sda2
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:35 scsi-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001 -> ../../dm-3
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:49 scsi-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001-part1 -> ../../dm-4
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-SATA_INTEL_SSDSC2BB08_BTWL40350DXP080KGN -> ../../sda
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-SATA_INTEL_SSDSC2BB08_BTWL40350DXP080KGN-part1 -> ../../sda1
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 scsi-SATA_INTEL_SSDSC2BB08_BTWL40350DXP080KGN-part2 -> ../../sda2
>> lrwxrwxrwx. 1 root root    9 Sep 18 14:32 scsi-SATA_Micron_1100_MTFD_17401F699137 -> ../../sdb
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 scsi-SATA_WDC_WDS100T2B0B-_183533804564 -> ../../sdc
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 wwn-0x5001b448b9608d90 -> ../../sdc
>> lrwxrwxrwx. 1 root root    9 Sep 18 14:32 wwn-0x500a07511f699137 -> ../../sdb
>> lrwxrwxrwx. 1 root root    9 Sep 18 22:40 wwn-0x55cd2e404b581cc0 -> ../../sda
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 wwn-0x55cd2e404b581cc0-part1 -> ../../sda1
>> lrwxrwxrwx. 1 root root  10 Sep 18 22:40 wwn-0x55cd2e404b581cc0-part2 -> ../../sda2
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:35 wwn-0xvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001 -> ../../dm-3
>> lrwxrwxrwx. 1 root root  10 Sep 18 23:49 wwn-0xvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001-part1 -> ../../dm-4
>> lrwxrwxrwx. 1 root root  15 Sep 18 14:32 wwn-nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001-part1 -> ../../nvme0n1p1
>> [root@odin ~]# ls -al /dev/disk/by-id/
>>
>> So filter notes three objects:
>> lvm-pv-uuid-e1fvwo.... -> dm-5  ->vdo_sdb  (used by HCI for all the three gluster base volumes )
>> lvm-pv-uuid-mr9awW... -> sda2 -> boot volume
>>
>> [root@odin ~]# lsblk
>> NAME                                                                                                MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
>> sda                                                                                                    8:0    0  74.5G  0 disk
>> ├─sda1                                                                                                8:1    0    1G  0 part  /boot
>> └─sda2                                                                                                8:2    0  73.5G  0 part
>>  ├─cl-root                                                                                          253:0    0  44.4G  0 lvm  /
>>  ├─cl-swap                                                                                          253:1    0  7.5G  0 lvm  [SWAP]
>>  └─cl-home                                                                                          253:2    0  21.7G  0 lvm  /home
>> sdb                                                                                                    8:16  0  477G  0 disk
>> └─vdo_sdb                                                                                            253:5    0  2.1T  0 vdo
>>  ├─gluster_vg_sdb-gluster_lv_engine                                                                253:6    0  100G  0 lvm  /gluster_bricks/engine
>>  ├─gluster_vg_sdb-gluster_thinpool_gluster_vg_sdb_tmeta                                            253:7    0    1G  0 lvm
>>  │ └─gluster_vg_sdb-gluster_thinpool_gluster_vg_sdb-tpool                                          253:9    0    2T  0 lvm
>>  │  ├─gluster_vg_sdb-gluster_thinpool_gluster_vg_sdb                                              253:10  0    2T  1 lvm
>>  │  ├─gluster_vg_sdb-gluster_lv_data                                                              253:11  0  1000G  0 lvm  /gluster_bricks/data
>>  │  └─gluster_vg_sdb-gluster_lv_vmstore                                                            253:12  0  1000G  0 lvm  /gluster_bricks/vmstore
>>  └─gluster_vg_sdb-gluster_thinpool_gluster_vg_sdb_tdata                                            253:8    0    2T  0 lvm
>>    └─gluster_vg_sdb-gluster_thinpool_gluster_vg_sdb-tpool                                          253:9    0    2T  0 lvm
>>      ├─gluster_vg_sdb-gluster_thinpool_gluster_vg_sdb                                              253:10  0    2T  1 lvm
>>      ├─gluster_vg_sdb-gluster_lv_data                                                              253:11  0  1000G  0 lvm  /gluster_bricks/data
>>      └─gluster_vg_sdb-gluster_lv_vmstore                                                            253:12  0  1000G  0 lvm  /gluster_bricks/vmstore
>> sdc                                                                                                    8:32  0 931.5G  0 disk
>> nvme0n1                                                                                              259:0    0 953.9G  0 disk
>> ├─nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001    253:3    0 953.9G  0 mpath
>> │ └─nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001p1 253:4    0 953.9G  0 part
>> └─nvme0n1p1
>>
>> So I don't think this is LVM filtering things
>>
>> Multipath showing weird treatment of the NVMe drive.. but that is outside this converstation
>> [root@odin ~]# multipath -l
>> nvme.126f-4141303030303030303030303030303032343538-53504343204d2e32205043496520535344-00000001 dm-3 NVME,SPCC M.2 PCIe SSD
>> size=954G features='1 queue_if_no_path' hwhandler='0' wp=rw
>> `-+- policy='service-time 0' prio=0 status=active
>>  `- 0:1:1:1 nvme0n1 259:0 active undef running
>> [root@odin ~]#
>>
>> Where is getting this filter.
>> I have done gdisk /dev/sdc ( new 1TB Drive) and shows no partition.  I even did a full dd if=/dev/zero  and no change.
>>
>> I reloaded OS on system to get through wizard setup.  Now that all three nodes are in the HCI cluster..  all six drives (2 x 1TB in each server) are now locked from any use due to this error about filter.
>>
>> Ideas?
>>
>> --
>> jeremey.wise@gmail.com
>> _______________________________________________
>> Users mailing list -- users@ovirt.org
>> To unsubscribe send an email to users-leave@ovirt.org
>> Privacy Statement: https://www.ovirt.org/privacy-policy.html
>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JIG3DCS72QCYYYSWF5XTQIMQSQVXDMSR/
>
>
>
> --
> jeremey.wise@gmail.com
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2NNMZUSENFJAQJYOCDSBBXLBHL3S6S2W/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/62X4ELFAMPUIZYXYSGKMBGOEJA4SCM3N/



--
jeremey.wise@gmail.com