Usually I would blacklist the Gluster devices by creating the necessary stanzas in /etc/multipath/conf.d/blacklist.conf<div><br></div><div>This way you will keep the situation simple.</div><div><br></div><div><br></div><div>For your problem, it's hard to odentify the problem based on the e-mails.What are your symptoms ?</div><div><br></div><div>To debug GlusterFS, it is good to start from the brick logs (/var/log/gluster/bricks) and the current heal status. On a 3-way replica volume heals should be resolved by GlusterFS - if not, there is a bug.</div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov</div><div><br></div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov<br> <br> <blockquote style="margin: 0 0 20px 0;"> <div style="font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Tue, May 31, 2022 at 16:32, jb</div><div>&lt;jonbae77@gmail.com&gt; wrote:</div> </div> <div style="padding: 10px 0 0 20px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> I still have the same problems, but it look like that the errors comes a <br clear="none">bit less often.<br clear="none"><br clear="none">I'm starting now to migrate the disk images to a NFS storage. When there <br clear="none">is no other way, I would recreate the glusterFS cluster.<br clear="none"><br clear="none">The problem I have is, that I don't know where is the root of this <br clear="none">problem and if recreating would fix the issue in longer terms.<br clear="none"><div class="yqt9023665641" id="yqtfd94695"><br clear="none">Am 29.05.22 um 20:26 schrieb Nir Soffer:<br clear="none">&gt; On Sun, May 29, 2022 at 9:03 PM Jonathan Baecker &lt;<a shape="rect" ymailto="mailto:jonbae77@gmail.com" href="mailto:jonbae77@gmail.com">jonbae77@gmail.com</a>&gt; wrote:<br clear="none">&gt;&gt; Am 29.05.22 um 19:24 schrieb Nir Soffer:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; On Sun, May 29, 2022 at 7:50 PM Jonathan Baecker &lt;<a shape="rect" ymailto="mailto:jonbae77@gmail.com" href="mailto:jonbae77@gmail.com">jonbae77@gmail.com</a>&gt; wrote:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Hello everybody,<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; we run a 3 node self hosted cluster with GlusterFS. I had a lot of problem upgrading ovirt from 4.4.10 to 4.5.0.2 and now we have cluster instability.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; First I will write down the problems I had with upgrading, so you get a bigger picture:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; engine update when fine<br clear="none">&gt;&gt; But nodes I could not update because of wrong version of imgbase, so I did a manual update to 4.5.0.1 and later to 4.5.0.2. First time after updating it was still booting into 4.4.10, so I did a reinstall.<br clear="none">&gt;&gt; Then after second reboot I ended up in the emergency mode. After a long searching I figure out that lvm.conf using use_devicesfile now but there it uses the wrong filters. So I comment out this and add the old filters back. This procedure I have done on all 3 nodes.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; When use_devicesfile (default in 4.5) is enabled, lvm filter is not<br clear="none">&gt;&gt; used. During installation<br clear="none">&gt;&gt; the old lvm filter is removed.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Can you share more info on why it does not work for you?<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; The problem was, that the node could not mount the gluster volumes anymore and ended up in emergency mode.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; - output of lsblk<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; NAME&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  MAJ:MIN RM&nbsp;  SIZE RO TYPE&nbsp; MOUNTPOINT<br clear="none">&gt;&gt; sda&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 8:0&nbsp; &nbsp; 0&nbsp;  1.8T&nbsp; 0 disk<br clear="none">&gt;&gt; `-XA1920LE10063_HKS028AV&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:0&nbsp; &nbsp; 0&nbsp;  1.8T&nbsp; 0 mpath<br clear="none">&gt;&gt;&nbsp; &nbsp; |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tmeta&nbsp;  253:16&nbsp;  0&nbsp; &nbsp;  9G&nbsp; 0 lvm<br clear="none">&gt;&gt;&nbsp; &nbsp; | `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18&nbsp;  0&nbsp;  1.7T&nbsp; 0 lvm<br clear="none">&gt;&gt;&nbsp; &nbsp; |&nbsp;  |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda&nbsp; &nbsp;  253:19&nbsp;  0&nbsp;  1.7T&nbsp; 1 lvm<br clear="none">&gt;&gt;&nbsp; &nbsp; |&nbsp;  |-gluster_vg_sda-gluster_lv_data&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:20&nbsp;  0&nbsp;  100G&nbsp; 0 lvm&nbsp;  /gluster_bricks/data<br clear="none">&gt;&gt;&nbsp; &nbsp; |&nbsp;  `-gluster_vg_sda-gluster_lv_vmstore&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:21&nbsp;  0&nbsp;  1.6T&nbsp; 0 lvm&nbsp;  /gluster_bricks/vmstore<br clear="none">&gt;&gt;&nbsp; &nbsp; `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tdata&nbsp;  253:17&nbsp;  0&nbsp;  1.7T&nbsp; 0 lvm<br clear="none">&gt;&gt;&nbsp; &nbsp; &nbsp; `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18&nbsp;  0&nbsp;  1.7T&nbsp; 0 lvm<br clear="none">&gt;&gt;&nbsp; &nbsp; &nbsp; &nbsp; |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda&nbsp; &nbsp;  253:19&nbsp;  0&nbsp;  1.7T&nbsp; 1 lvm<br clear="none">&gt;&gt;&nbsp; &nbsp; &nbsp; &nbsp; |-gluster_vg_sda-gluster_lv_data&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:20&nbsp;  0&nbsp;  100G&nbsp; 0 lvm&nbsp;  /gluster_bricks/data<br clear="none">&gt;&gt;&nbsp; &nbsp; &nbsp; &nbsp; `-gluster_vg_sda-gluster_lv_vmstore&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:21&nbsp;  0&nbsp;  1.6T&nbsp; 0 lvm&nbsp;  /gluster_bricks/vmstore<br clear="none">&gt;&gt; sr0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  11:0&nbsp; &nbsp; 1&nbsp; 1024M&nbsp; 0 rom<br clear="none">&gt;&gt; nvme0n1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 259:0&nbsp; &nbsp; 0 238.5G&nbsp; 0 disk<br clear="none">&gt;&gt; |-nvme0n1p1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 259:1&nbsp; &nbsp; 0&nbsp; &nbsp;  1G&nbsp; 0 part&nbsp; /boot<br clear="none">&gt;&gt; |-nvme0n1p2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 259:2&nbsp; &nbsp; 0&nbsp;  134G&nbsp; 0 part<br clear="none">&gt;&gt; | |-onn-pool00_tmeta&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:1&nbsp; &nbsp; 0&nbsp; &nbsp;  1G&nbsp; 0 lvm<br clear="none">&gt;&gt; | | `-onn-pool00-tpool&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:3&nbsp; &nbsp; 0&nbsp; &nbsp; 87G&nbsp; 0 lvm<br clear="none">&gt;&gt; | |&nbsp;  |-onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1&nbsp; &nbsp; &nbsp;  253:4&nbsp; &nbsp; 0&nbsp; &nbsp; 50G&nbsp; 0 lvm&nbsp;  /<br clear="none">&gt;&gt; | |&nbsp;  |-onn-pool00&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:7&nbsp; &nbsp; 0&nbsp; &nbsp; 87G&nbsp; 1 lvm<br clear="none">&gt;&gt; | |&nbsp;  |-onn-home&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:8&nbsp; &nbsp; 0&nbsp; &nbsp;  1G&nbsp; 0 lvm&nbsp;  /home<br clear="none">&gt;&gt; | |&nbsp;  |-onn-tmp&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:9&nbsp; &nbsp; 0&nbsp; &nbsp;  1G&nbsp; 0 lvm&nbsp;  /tmp<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:10&nbsp;  0&nbsp; &nbsp; 15G&nbsp; 0 lvm&nbsp;  /var<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var_crash&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:11&nbsp;  0&nbsp; &nbsp; 10G&nbsp; 0 lvm&nbsp;  /var/crash<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var_log&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:12&nbsp;  0&nbsp; &nbsp;  8G&nbsp; 0 lvm&nbsp;  /var/log<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var_log_audit&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:13&nbsp;  0&nbsp; &nbsp;  2G&nbsp; 0 lvm&nbsp;  /var/log/audit<br clear="none">&gt;&gt; | |&nbsp;  |-onn-ovirt--node--ng--4.5.0.1--0.20220511.0+1&nbsp; &nbsp; &nbsp;  253:14&nbsp;  0&nbsp; &nbsp; 50G&nbsp; 0 lvm<br clear="none">&gt;&gt; | |&nbsp;  `-onn-var_tmp&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:15&nbsp;  0&nbsp; &nbsp; 10G&nbsp; 0 lvm&nbsp;  /var/tmp<br clear="none">&gt;&gt; | |-onn-pool00_tdata&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:2&nbsp; &nbsp; 0&nbsp; &nbsp; 87G&nbsp; 0 lvm<br clear="none">&gt;&gt; | | `-onn-pool00-tpool&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:3&nbsp; &nbsp; 0&nbsp; &nbsp; 87G&nbsp; 0 lvm<br clear="none">&gt;&gt; | |&nbsp;  |-onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1&nbsp; &nbsp; &nbsp;  253:4&nbsp; &nbsp; 0&nbsp; &nbsp; 50G&nbsp; 0 lvm&nbsp;  /<br clear="none">&gt;&gt; | |&nbsp;  |-onn-pool00&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:7&nbsp; &nbsp; 0&nbsp; &nbsp; 87G&nbsp; 1 lvm<br clear="none">&gt;&gt; | |&nbsp;  |-onn-home&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:8&nbsp; &nbsp; 0&nbsp; &nbsp;  1G&nbsp; 0 lvm&nbsp;  /home<br clear="none">&gt;&gt; | |&nbsp;  |-onn-tmp&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:9&nbsp; &nbsp; 0&nbsp; &nbsp;  1G&nbsp; 0 lvm&nbsp;  /tmp<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:10&nbsp;  0&nbsp; &nbsp; 15G&nbsp; 0 lvm&nbsp;  /var<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var_crash&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:11&nbsp;  0&nbsp; &nbsp; 10G&nbsp; 0 lvm&nbsp;  /var/crash<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var_log&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:12&nbsp;  0&nbsp; &nbsp;  8G&nbsp; 0 lvm&nbsp;  /var/log<br clear="none">&gt;&gt; | |&nbsp;  |-onn-var_log_audit&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:13&nbsp;  0&nbsp; &nbsp;  2G&nbsp; 0 lvm&nbsp;  /var/log/audit<br clear="none">&gt;&gt; | |&nbsp;  |-onn-ovirt--node--ng--4.5.0.1--0.20220511.0+1&nbsp; &nbsp; &nbsp;  253:14&nbsp;  0&nbsp; &nbsp; 50G&nbsp; 0 lvm<br clear="none">&gt;&gt; | |&nbsp;  `-onn-var_tmp&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 253:15&nbsp;  0&nbsp; &nbsp; 10G&nbsp; 0 lvm&nbsp;  /var/tmp<br clear="none">&gt;&gt; | `-onn-swap&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:5&nbsp; &nbsp; 0&nbsp; &nbsp; 20G&nbsp; 0 lvm&nbsp;  [SWAP]<br clear="none">&gt;&gt; `-nvme0n1p3&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 259:3&nbsp; &nbsp; 0&nbsp; &nbsp; 95G&nbsp; 0 part<br clear="none">&gt;&gt;&nbsp; &nbsp; `-gluster_vg_nvme0n1p3-gluster_lv_engine&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  253:6&nbsp; &nbsp; 0&nbsp; &nbsp; 94G&nbsp; 0 lvm&nbsp;  /gluster_bricks/engine<br clear="none">&gt;&nbsp;  &gt;<br clear="none">&gt;&gt; - The old lvm filter used, and why it was needed<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; filter = ["a|^/dev/disk/by-id/lvm-pv-uuid-Nn7tZl-TFdY-BujO-VZG5-EaGW-5YFd-Lo5pwa$|", "a|^/dev/disk/by-id/lvm-pv-uuid-Wcbxnx-2RhC-s1Re-s148-nLj9-Tr3f-jj4VvE$|", "a|^/dev/disk/by-id/lvm-pv-uuid-lX51wm-H7V4-3CTn-qYob-Rkpx-Tptd-t94jNL$|", "r|.*|"]<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; I don't remember exactly any more why it was needed, but without the node was not working correctly. I think I even used vdsm-tool config-lvm-filter.<br clear="none">&gt; I think that if&nbsp; you list the devices in this filter:<br clear="none">&gt;<br clear="none">&gt;&nbsp; &nbsp; &nbsp; ls -lh /dev/disk/by-id/lvm-pv-uuid-Nn7tZl-TFdY-BujO-VZG5-EaGW-5YFd-Lo5pwa \<br clear="none">&gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  /dev/disk/by-id/lvm-pv-uuid-Wcbxnx-2RhC-s1Re-s148-nLj9-Tr3f-jj4VvE<br clear="none">&gt; \<br clear="none">&gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; /dev/disk/by-id/lvm-pv-uuid-lX51wm-H7V4-3CTn-qYob-Rkpx-Tptd-t94jNL<br clear="none">&gt;<br clear="none">&gt; You will see that these are the devices used by these vgs:<br clear="none">&gt;<br clear="none">&gt;&nbsp; &nbsp; &nbsp; gluster_vg_sda, gluster_vg_nvme0n1p3, onn<br clear="none">&gt;<br clear="none">&gt;&gt; - output of vdsm-tool config-lvm-filter<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Analyzing host...<br clear="none">&gt;&gt; Found these mounted logical volumes on this host:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/gluster_vg_nvme0n1p3-gluster_lv_engine<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /gluster_bricks/engine<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p3<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/gluster_vg_sda-gluster_lv_data<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /gluster_bricks/data<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/mapper/XA1920LE10063_HKS028AV<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/gluster_vg_sda-gluster_lv_vmstore<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /gluster_bricks/vmstore<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/mapper/XA1920LE10063_HKS028AV<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-home<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /home<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-swap<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; [SWAP]<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-tmp<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /tmp<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-var<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /var<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-var_crash<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /var/crash<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-var_log<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /var/log<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-var_log_audit<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /var/log/audit<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; logical volume:&nbsp; /dev/mapper/onn-var_tmp<br clear="none">&gt;&gt;&nbsp; &nbsp; mountpoint:&nbsp; &nbsp; &nbsp; /var/tmp<br clear="none">&gt;&gt;&nbsp; &nbsp; devices:&nbsp; &nbsp; &nbsp; &nbsp;  /dev/nvme0n1p2<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Configuring LVM system.devices.<br clear="none">&gt;&gt; Devices for following VGs will be imported:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp;  gluster_vg_sda, gluster_vg_nvme0n1p3, onn<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; To properly configure the host, we need to add multipath<br clear="none">&gt;&gt; blacklist in /etc/multipath/conf.d/vdsm_blacklist.conf:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; blacklist {<br clear="none">&gt;&gt;&nbsp; &nbsp; &nbsp; &nbsp; wwid "eui.0025388901b1e26f"<br clear="none">&gt;&gt;&nbsp; &nbsp; }<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Configure host? [yes,NO]<br clear="none">&gt; If you run "vdsm-tool config-lvm-filter" and confirm with "yes", I<br clear="none">&gt; think all the vgs<br clear="none">&gt; will be imported properly into lvm devices file.<br clear="none">&gt;<br clear="none">&gt; I don't think it will solve the storage issues you have since Feb<br clear="none">&gt; 2022, but at least<br clear="none">&gt; you will have a standard configuration and the next upgrade will not revert your<br clear="none">&gt; local settings.<br clear="none">&gt;<br clear="none">&gt;&gt; If using lvm devices does not work for you, you can enable the lvm<br clear="none">&gt;&gt; filter in vdsm configuration<br clear="none">&gt;&gt; by adding a drop-in file:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ cat /etc/vdsm/vdsm.conf.d/99-local.conf<br clear="none">&gt;&gt; [lvm]<br clear="none">&gt;&gt; config_method = filter<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; And run:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt;&nbsp; &nbsp; &nbsp; vdsm-tool config-lvm-filter<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; to configure the lvm filter in the best way for vdsm. If this does not create<br clear="none">&gt;&gt; the right filter we would like to know why, but in general you should use<br clear="none">&gt;&gt; lvm devices since it avoids the trouble of maintaining the filter and dealing<br clear="none">&gt;&gt; with upgrades and user edited lvm filter.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; If&nbsp; you disable use_devicesfile, the next vdsm upgrade will enable it<br clear="none">&gt;&gt; back unless<br clear="none">&gt;&gt; you change the configuration.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; I would be happy to just use the default, when there is a way to make use_devicesfile to wok.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Also even if you disable use_devicesfile in lvm.conf, vdsm still use<br clear="none">&gt;&gt; --devices instead<br clear="none">&gt;&gt; of filter when running lvm commands, and lvm commands run by vdsm ignore your<br clear="none">&gt;&gt; lvm filter since the --devices option overrides the system settings.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; ...<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; I notice some unsync volume warning, but because I had this in the past to, after upgrading, I though after some time they will disappear. The next day there still where there, so I decided to put the nodes again in the maintenance mode and restart the glusterd service. After some time the sync warnings where gone.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Not clear what these warnings are, I guess Gluster warning?<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Yes was Gluster warnings under Storage -&gt; Volumes it was saying that some entries are unsync.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; So now the actual problem:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Since this time the cluster is unstable. I get different errors and warning, like:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; VM [name] is not responding<br clear="none">&gt;&gt; out of nothing HA VM gets migrated<br clear="none">&gt;&gt; VM migration can fail<br clear="none">&gt;&gt; VM backup with snapshoting and export take very long<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; How do you backup the vms? do you sue a backup application? how is it<br clear="none">&gt;&gt; configured?<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; I use a self made plython script, which uses the rest api. I create a snapshot from the VM, build a new VM from that snapshot and move the new one to the export domain.<br clear="none">&gt; This is not very efficient - this copy the entire vm at the point of<br clear="none">&gt; time of the snapshot<br clear="none">&gt; and then copy it again to the export domain.<br clear="none">&gt;<br clear="none">&gt; If you use a backup application supporting the incremental backup API,<br clear="none">&gt; the first full backup<br clear="none">&gt; will copy the entire vm once, but later incremental backup will copy<br clear="none">&gt; only the changes<br clear="none">&gt; since the last backup.<br clear="none">&gt;<br clear="none">&gt;&gt; VMs are getting very slow some times<br clear="none">&gt;&gt; Storage domain vmstore experienced a high latency of 9.14251<br clear="none">&gt;&gt; ovs|00001|db_ctl_base|ERR|no key "dpdk-init" in Open_vSwitch record "." column other_config<br clear="none">&gt;&gt; 489279 [1064359]: s8 renewal error -202 delta_length 10 last_success 489249<br clear="none">&gt;&gt; 444853 [2243175]: s27 delta_renew read timeout 10 sec offset 0 /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/ids<br clear="none">&gt;&gt; 471099 [2243175]: s27 delta_renew read timeout 10 sec offset 0 /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/ids<br clear="none">&gt;&gt; many of: 424035 [2243175]: s27 delta_renew long write time XX sec<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; All these issues tell use that your storage is not working correctly.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; sanlock.log is full of renewal errors form May:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep 2022-05- sanlock.log | wc -l<br clear="none">&gt;&gt; 4844<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep 2022-05- sanlock.log | grep 'renewal error' | wc -l<br clear="none">&gt;&gt; 631<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; But there is lot of trouble from earlier months:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep 2022-04- sanlock.log | wc -l<br clear="none">&gt;&gt; 844<br clear="none">&gt;&gt; $ grep 2022-04- sanlock.log | grep 'renewal error' | wc -l<br clear="none">&gt;&gt; 29<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep 2022-03- sanlock.log | wc -l<br clear="none">&gt;&gt; 1609<br clear="none">&gt;&gt; $ grep 2022-03- sanlock.log | grep 'renewal error' | wc -l<br clear="none">&gt;&gt; 483<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep 2022-02- sanlock.log | wc -l<br clear="none">&gt;&gt; 826<br clear="none">&gt;&gt; $ grep 2022-02- sanlock.log | grep 'renewal error' | wc -l<br clear="none">&gt;&gt; 242<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Here sanlock log looks healthy:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep 2022-01- sanlock.log | wc -l<br clear="none">&gt;&gt; 3<br clear="none">&gt;&gt; $ grep 2022-01- sanlock.log | grep 'renewal error' | wc -l<br clear="none">&gt;&gt; 0<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep 2021-12- sanlock.log | wc -l<br clear="none">&gt;&gt; 48<br clear="none">&gt;&gt; $ grep 2021-12- sanlock.log | grep 'renewal error' | wc -l<br clear="none">&gt;&gt; 0<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; vdsm log shows that 2 domains are not accessible:<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; $ grep ERROR vdsm.log<br clear="none">&gt;&gt; 2022-05-29 15:07:19,048+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt; 2022-05-29 16:33:59,049+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt; 2022-05-29 16:34:39,049+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt; 2022-05-29 17:21:39,050+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt; 2022-05-29 17:55:59,712+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt; 2022-05-29 17:56:19,711+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt; 2022-05-29 17:56:39,050+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt; 2022-05-29 17:56:39,711+0200 ERROR (check/loop) [storage.monitor]<br clear="none">&gt;&gt; Error checking path<br clear="none">&gt;&gt; /rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata<br clear="none">&gt;&gt; (monitor:511)<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; You need to find what is the issue with your Gluster storage.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; I hope that Ritesh can help debug the issue with Gluster.<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; Nir<br clear="none">&gt;&gt;<br clear="none">&gt;&gt; I'm worry that I do something, that it makes it even more worst, and I hove not idea what's the problem. To me it looks not exactly like a problem with data inconsistencies.<br clear="none">&gt; The problem is that your Gluster storage is not healthy, and reading<br clear="none">&gt; and writing to it times out.<br clear="none">&gt;<br clear="none">&gt; Please keep <a shape="rect" ymailto="mailto:users@ovirt.org" href="mailto:users@ovirt.org">users@ovirt.org</a> CC when you reply. Gluster storage is very<br clear="none">&gt; popular in this mailing list<br clear="none">&gt; and you may get useful help from other users.<br clear="none">&gt;<br clear="none">&gt; Nir<br clear="none">&gt;<br clear="none">_______________________________________________<br clear="none">Users mailing list -- <a shape="rect" ymailto="mailto:users@ovirt.org" href="mailto:users@ovirt.org">users@ovirt.org</a><br clear="none">To unsubscribe send an email to <a shape="rect" ymailto="mailto:users-leave@ovirt.org" href="mailto:users-leave@ovirt.org">users-leave@ovirt.org</a><br clear="none">Privacy Statement: <a shape="rect" href="https://www.ovirt.org/privacy-policy.html" target="_blank">https://www.ovirt.org/privacy-policy.html</a><br clear="none">oVirt Code of Conduct: <a shape="rect" href="https://www.ovirt.org/community/about/community-guidelines/" target="_blank">https://www.ovirt.org/community/about/community-guidelines/</a><br clear="none">List Archives: </div><a shape="rect" href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/VBYRVRQPXXDZTDFG46LEECHLRDWDWZ37/" target="_blank">https://lists.ovirt.org/archives/list/users@ovirt.org/message/VBYRVRQPXXDZTDFG46LEECHLRDWDWZ37/</a><div class="yqt9023665641" id="yqtfd04678"><br clear="none"></div> </div> </blockquote></div>