9:30 a.m.
I still have the same problems, but it look like that the errors comes a
bit less often.
I'm starting now to migrate the disk images to a NFS storage. When there
is no other way, I would recreate the glusterFS cluster.
The problem I have is, that I don't know where is the root of this
problem and if recreating would fix the issue in longer terms.
Am 29.05.22 um 20:26 schrieb Nir Soffer:
> On Sun, May 29, 2022 at 9:03 PM Jonathan Baecker <jonbae77(a)gmail.com> wrote:
>> Am 29.05.22 um 19:24 schrieb Nir Soffer:
>>
>> On Sun, May 29, 2022 at 7:50 PM Jonathan Baecker <jonbae77(a)gmail.com>
wrote:
>>
>> Hello everybody,
>>
>> we run a 3 node self hosted cluster with GlusterFS. I had a lot of problem
upgrading ovirt from 4.4.10 to 4.5.0.2 and now we have cluster instability.
>>
>> First I will write down the problems I had with upgrading, so you get a bigger
picture:
>>
>> engine update when fine
>> But nodes I could not update because of wrong version of imgbase, so I did a
manual update to 4.5.0.1 and later to 4.5.0.2. First time after updating it was still
booting into 4.4.10, so I did a reinstall.
>> Then after second reboot I ended up in the emergency mode. After a long searching
I figure out that lvm.conf using use_devicesfile now but there it uses the wrong filters.
So I comment out this and add the old filters back. This procedure I have done on all 3
nodes.
>>
>> When use_devicesfile (default in 4.5) is enabled, lvm filter is not
>> used. During installation
>> the old lvm filter is removed.
>>
>> Can you share more info on why it does not work for you?
>>
>> The problem was, that the node could not mount the gluster volumes anymore and
ended up in emergency mode.
>>
>> - output of lsblk
>>
>> NAME MAJ:MIN RM SIZE RO
TYPE MOUNTPOINT
>> sda 8:0 0 1.8T 0
disk
>> `-XA1920LE10063_HKS028AV 253:0 0 1.8T 0
mpath
>> |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tmeta 253:16 0 9G 0
lvm
>> | `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18 0 1.7T 0
lvm
>> | |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda 253:19 0 1.7T 1
lvm
>> | |-gluster_vg_sda-gluster_lv_data 253:20 0 100G 0
lvm /gluster_bricks/data
>> | `-gluster_vg_sda-gluster_lv_vmstore 253:21 0 1.6T 0
lvm /gluster_bricks/vmstore
>> `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda_tdata 253:17 0 1.7T 0
lvm
>> `-gluster_vg_sda-gluster_thinpool_gluster_vg_sda-tpool 253:18 0 1.7T 0
lvm
>> |-gluster_vg_sda-gluster_thinpool_gluster_vg_sda 253:19 0 1.7T 1
lvm
>> |-gluster_vg_sda-gluster_lv_data 253:20 0 100G 0
lvm /gluster_bricks/data
>> `-gluster_vg_sda-gluster_lv_vmstore 253:21 0 1.6T 0
lvm /gluster_bricks/vmstore
>> sr0 11:0 1 1024M 0
rom
>> nvme0n1 259:0 0 238.5G 0
disk
>> |-nvme0n1p1 259:1 0 1G 0
part /boot
>> |-nvme0n1p2 259:2 0 134G 0
part
>> | |-onn-pool00_tmeta 253:1 0 1G 0
lvm
>> | | `-onn-pool00-tpool 253:3 0 87G 0
lvm
>> | | |-onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1 253:4 0 50G 0
lvm /
>> | | |-onn-pool00 253:7 0 87G 1
lvm
>> | | |-onn-home 253:8 0 1G 0
lvm /home
>> | | |-onn-tmp 253:9 0 1G 0
lvm /tmp
>> | | |-onn-var 253:10 0 15G 0
lvm /var
>> | | |-onn-var_crash 253:11 0 10G 0
lvm /var/crash
>> | | |-onn-var_log 253:12 0 8G 0
lvm /var/log
>> | | |-onn-var_log_audit 253:13 0 2G 0
lvm /var/log/audit
>> | | |-onn-ovirt--node--ng--4.5.0.1--0.20220511.0+1 253:14 0 50G 0
lvm
>> | | `-onn-var_tmp 253:15 0 10G 0
lvm /var/tmp
>> | |-onn-pool00_tdata 253:2 0 87G 0
lvm
>> | | `-onn-pool00-tpool 253:3 0 87G 0
lvm
>> | | |-onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1 253:4 0 50G 0
lvm /
>> | | |-onn-pool00 253:7 0 87G 1
lvm
>> | | |-onn-home 253:8 0 1G 0
lvm /home
>> | | |-onn-tmp 253:9 0 1G 0
lvm /tmp
>> | | |-onn-var 253:10 0 15G 0
lvm /var
>> | | |-onn-var_crash 253:11 0 10G 0
lvm /var/crash
>> | | |-onn-var_log 253:12 0 8G 0
lvm /var/log
>> | | |-onn-var_log_audit 253:13 0 2G 0
lvm /var/log/audit
>> | | |-onn-ovirt--node--ng--4.5.0.1--0.20220511.0+1 253:14 0 50G 0
lvm
>> | | `-onn-var_tmp 253:15 0 10G 0
lvm /var/tmp
>> | `-onn-swap 253:5 0 20G 0
lvm [SWAP]
>> `-nvme0n1p3 259:3 0 95G 0
part
>> `-gluster_vg_nvme0n1p3-gluster_lv_engine 253:6 0 94G 0
lvm /gluster_bricks/engine
> >
>> - The old lvm filter used, and why it was needed
>>
>> filter =
["a|^/dev/disk/by-id/lvm-pv-uuid-Nn7tZl-TFdY-BujO-VZG5-EaGW-5YFd-Lo5pwa$|",
"a|^/dev/disk/by-id/lvm-pv-uuid-Wcbxnx-2RhC-s1Re-s148-nLj9-Tr3f-jj4VvE$|",
"a|^/dev/disk/by-id/lvm-pv-uuid-lX51wm-H7V4-3CTn-qYob-Rkpx-Tptd-t94jNL$|",
"r|.*|"]
>>
>> I don't remember exactly any more why it was needed, but without the node was
not working correctly. I think I even used vdsm-tool config-lvm-filter.
> I think that if you list the devices in this filter:
>
> ls -lh /dev/disk/by-id/lvm-pv-uuid-Nn7tZl-TFdY-BujO-VZG5-EaGW-5YFd-Lo5pwa \
> /dev/disk/by-id/lvm-pv-uuid-Wcbxnx-2RhC-s1Re-s148-nLj9-Tr3f-jj4VvE
> \
> /dev/disk/by-id/lvm-pv-uuid-lX51wm-H7V4-3CTn-qYob-Rkpx-Tptd-t94jNL
>
> You will see that these are the devices used by these vgs:
>
> gluster_vg_sda, gluster_vg_nvme0n1p3, onn
>
>> - output of vdsm-tool config-lvm-filter
>>
>> Analyzing host...
>> Found these mounted logical volumes on this host:
>>
>> logical volume: /dev/mapper/gluster_vg_nvme0n1p3-gluster_lv_engine
>> mountpoint: /gluster_bricks/engine
>> devices: /dev/nvme0n1p3
>>
>> logical volume: /dev/mapper/gluster_vg_sda-gluster_lv_data
>> mountpoint: /gluster_bricks/data
>> devices: /dev/mapper/XA1920LE10063_HKS028AV
>>
>> logical volume: /dev/mapper/gluster_vg_sda-gluster_lv_vmstore
>> mountpoint: /gluster_bricks/vmstore
>> devices: /dev/mapper/XA1920LE10063_HKS028AV
>>
>> logical volume: /dev/mapper/onn-home
>> mountpoint: /home
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-ovirt--node--ng--4.5.0.2--0.20220513.0+1
>> mountpoint: /
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-swap
>> mountpoint: [SWAP]
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-tmp
>> mountpoint: /tmp
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-var
>> mountpoint: /var
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-var_crash
>> mountpoint: /var/crash
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-var_log
>> mountpoint: /var/log
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-var_log_audit
>> mountpoint: /var/log/audit
>> devices: /dev/nvme0n1p2
>>
>> logical volume: /dev/mapper/onn-var_tmp
>> mountpoint: /var/tmp
>> devices: /dev/nvme0n1p2
>>
>> Configuring LVM system.devices.
>> Devices for following VGs will be imported:
>>
>> gluster_vg_sda, gluster_vg_nvme0n1p3, onn
>>
>> To properly configure the host, we need to add multipath
>> blacklist in /etc/multipath/conf.d/vdsm_blacklist.conf:
>>
>> blacklist {
>> wwid "eui.0025388901b1e26f"
>> }
>>
>>
>> Configure host? [yes,NO]
> If you run "vdsm-tool config-lvm-filter" and confirm with "yes",
I
> think all the vgs
> will be imported properly into lvm devices file.
>
> I don't think it will solve the storage issues you have since Feb
> 2022, but at least
> you will have a standard configuration and the next upgrade will not revert your
> local settings.
>
>> If using lvm devices does not work for you, you can enable the lvm
>> filter in vdsm configuration
>> by adding a drop-in file:
>>
>> $ cat /etc/vdsm/vdsm.conf.d/99-local.conf
>> [lvm]
>> config_method = filter
>>
>> And run:
>>
>> vdsm-tool config-lvm-filter
>>
>> to configure the lvm filter in the best way for vdsm. If this does not create
>> the right filter we would like to know why, but in general you should use
>> lvm devices since it avoids the trouble of maintaining the filter and dealing
>> with upgrades and user edited lvm filter.
>>
>> If you disable use_devicesfile, the next vdsm upgrade will enable it
>> back unless
>> you change the configuration.
>>
>> I would be happy to just use the default, when there is a way to make
use_devicesfile to wok.
>>
>> Also even if you disable use_devicesfile in lvm.conf, vdsm still use
>> --devices instead
>> of filter when running lvm commands, and lvm commands run by vdsm ignore your
>> lvm filter since the --devices option overrides the system settings.
>>
>> ...
>>
>> I notice some unsync volume warning, but because I had this in the past to, after
upgrading, I though after some time they will disappear. The next day there still where
there, so I decided to put the nodes again in the maintenance mode and restart the
glusterd service. After some time the sync warnings where gone.
>>
>> Not clear what these warnings are, I guess Gluster warning?
>>
>> Yes was Gluster warnings under Storage -> Volumes it was saying that some
entries are unsync.
>>
>> So now the actual problem:
>>
>> Since this time the cluster is unstable. I get different errors and warning,
like:
>>
>> VM [name] is not responding
>> out of nothing HA VM gets migrated
>> VM migration can fail
>> VM backup with snapshoting and export take very long
>>
>> How do you backup the vms? do you sue a backup application? how is it
>> configured?
>>
>> I use a self made plython script, which uses the rest api. I create a snapshot
from the VM, build a new VM from that snapshot and move the new one to the export domain.
> This is not very efficient - this copy the entire vm at the point of
> time of the snapshot
> and then copy it again to the export domain.
>
> If you use a backup application supporting the incremental backup API,
> the first full backup
> will copy the entire vm once, but later incremental backup will copy
> only the changes
> since the last backup.
>
>> VMs are getting very slow some times
>> Storage domain vmstore experienced a high latency of 9.14251
>> ovs|00001|db_ctl_base|ERR|no key "dpdk-init" in Open_vSwitch record
"." column other_config
>> 489279 [1064359]: s8 renewal error -202 delta_length 10 last_success 489249
>> 444853 [2243175]: s27 delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/ids
>> 471099 [2243175]: s27 delta_renew read timeout 10 sec offset 0
/rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/ids
>> many of: 424035 [2243175]: s27 delta_renew long write time XX sec
>>
>> All these issues tell use that your storage is not working correctly.
>>
>> sanlock.log is full of renewal errors form May:
>>
>> $ grep 2022-05- sanlock.log | wc -l
>> 4844
>>
>> $ grep 2022-05- sanlock.log | grep 'renewal error' | wc -l
>> 631
>>
>> But there is lot of trouble from earlier months:
>>
>> $ grep 2022-04- sanlock.log | wc -l
>> 844
>> $ grep 2022-04- sanlock.log | grep 'renewal error' | wc -l
>> 29
>>
>> $ grep 2022-03- sanlock.log | wc -l
>> 1609
>> $ grep 2022-03- sanlock.log | grep 'renewal error' | wc -l
>> 483
>>
>> $ grep 2022-02- sanlock.log | wc -l
>> 826
>> $ grep 2022-02- sanlock.log | grep 'renewal error' | wc -l
>> 242
>>
>> Here sanlock log looks healthy:
>>
>> $ grep 2022-01- sanlock.log | wc -l
>> 3
>> $ grep 2022-01- sanlock.log | grep 'renewal error' | wc -l
>> 0
>>
>> $ grep 2021-12- sanlock.log | wc -l
>> 48
>> $ grep 2021-12- sanlock.log | grep 'renewal error' | wc -l
>> 0
>>
>> vdsm log shows that 2 domains are not accessible:
>>
>> $ grep ERROR vdsm.log
>> 2022-05-29 15:07:19,048+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata
>> (monitor:511)
>> 2022-05-29 16:33:59,049+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata
>> (monitor:511)
>> 2022-05-29 16:34:39,049+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata
>> (monitor:511)
>> 2022-05-29 17:21:39,050+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata
>> (monitor:511)
>> 2022-05-29 17:55:59,712+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata
>> (monitor:511)
>> 2022-05-29 17:56:19,711+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata
>> (monitor:511)
>> 2022-05-29 17:56:39,050+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_data/de5f4123-0fac-4238-abcf-a329c142bd47/dom_md/metadata
>> (monitor:511)
>> 2022-05-29 17:56:39,711+0200 ERROR (check/loop) [storage.monitor]
>> Error checking path
>>
/rhev/data-center/mnt/glusterSD/onode1.example.org:_vmstore/3cf83851-1cc8-4f97-8960-08a60b9e25db/dom_md/metadata
>> (monitor:511)
>>
>> You need to find what is the issue with your Gluster storage.
>>
>> I hope that Ritesh can help debug the issue with Gluster.
>>
>> Nir
>>
>> I'm worry that I do something, that it makes it even more worst, and I hove
not idea what's the problem. To me it looks not exactly like a problem with data
inconsistencies.
> The problem is that your Gluster storage is not healthy, and reading
> and writing to it times out.
>
> Please keep users(a)ovirt.org CC when you reply. Gluster storage is very
> popular in this mailing list
> and you may get useful help from other users.
>
> Nir
>