- Users - oVirt List Archives

After restored hosted engine as a new vms, all the existing & live hosts and storage domain are showing offline
by dhanaraj.ramesh＠yahoo.com 11 Jan '22

11 Jan '22

Hi Team due to some reasons one of our slef hosted engine went down, hence we provisioned new centos vm and restored the hosted engine backup file. After restoring we could see all the Datacenter, cluster, storage, hosts and vm entities in restored ovirt engine but they all are offline. please help us how to we connect all the running hosts and storage domain, vms back to restored hosted engine. do we need to get a down time for all the vms to restore the hosts? how do I activate all the existing storage domains as is.

2 2

Gluster Hook differences between fresh and old clusters
by Strahil Nikolov 11 Jan '22

11 Jan '22

Hi All, recently I have migrated from 4.3.10 to 4.4.9 and it seems something odd is happening. Symptoms: - A lot of warnings for Gluster hook discrepancies - Trying to refresh the hooks via the sync button fails (engine error: https://justpaste.it/827zo ) - Existing "Default" cluster tracks more hooks than a fresh new cluster New cluster hooks: http://i.imgur.com/FEL2Z1D.png Migrated cluster: https://i.imgur.com/L8dWYZY.png What can I do to resolve the issue ? I've tried to resync the hooks, move away /var/lib/glusterd/hooks/1/ and reinstall gluster packages, try to resolve via the "Resolve Conflicts" in the UI and nothing helped so far. Best Regards, Strahil Nikolov

2 4

Error while removing snapshot: Unable to get volume info
by francesco＠shellrent.com 10 Jan '22

10 Jan '22

Hi all, I'm trying to remove a snapshot from a HA VM in a setup with glusterfs (2 nodes C8 stream oVirt 4.4 + 1 arbiter C8). The error that appears in the vdsm log of the host is: 2022-01-10 09:33:03,003+0100 ERROR (jsonrpc/4) [api] FINISH merge error=Merge failed: {'top': '441354e7-c234-4079-b494-53fa99cdce6f', 'base': 'fdf38f20-3416-4d75-a159-2a341b1ed637', 'job': '50206e3a-8018-4ea8-b191-e4bc859ae0c7', 'reason': 'Unable to get volume info for domain 574a3cd1-5617-4742-8de9-4732be4f27e0 volume 441354e7-c234-4079-b494-53fa99cdce6f'} (api:131) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/livemerge.py", line 285, in merge drive.domainID, drive.poolID, drive.imageID, job.top) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 5988, in getVolumeInfo (domainID, volumeID)) vdsm.virt.errors.StorageUnavailableError: Unable to get volume info for domain 574a3cd1-5617-4742-8de9-4732be4f27e0 volume 441354e7-c234-4079-b494-53fa99cdce6f During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 124, in method ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/API.py", line 776, in merge drive, baseVolUUID, topVolUUID, bandwidth, jobUUID) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 5833, in merge driveSpec, baseVolUUID, topVolUUID, bandwidth, jobUUID) File "/usr/lib/python3.6/site-packages/vdsm/virt/livemerge.py", line 288, in merge str(e), top=top, base=job.base, job=job_id) The volume list in the host differs from the engine one: HOST: vdsm-tool dump-volume-chains 574a3cd1-5617-4742-8de9-4732be4f27e0 | grep -A10 0b995271-e7f3-41b3-aff7-b5ad7942c10d image: 0b995271-e7f3-41b3-aff7-b5ad7942c10d - fdf38f20-3416-4d75-a159-2a341b1ed637 status: OK, voltype: INTERNAL, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 44255387648 - 10df3adb-38f4-41d1-be84-b8b5b86e92cc status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE, capacity: 53687091200, truesize: 7335407616 ls -1 0b995271-e7f3-41b3-aff7-b5ad7942c10d 10df3adb-38f4-41d1-be84-b8b5b86e92cc 10df3adb-38f4-41d1-be84-b8b5b86e92cc.lease 10df3adb-38f4-41d1-be84-b8b5b86e92cc.meta fdf38f20-3416-4d75-a159-2a341b1ed637 fdf38f20-3416-4d75-a159-2a341b1ed637.lease fdf38f20-3416-4d75-a159-2a341b1ed637.meta ENGINE: engine=# select * from images where image_group_id='0b995271-e7f3-41b3-aff7-b5ad7942c10d'; -[ RECORD 1 ]---------+------------------------------------- image_guid | 10df3adb-38f4-41d1-be84-b8b5b86e92cc creation_date | 2022-01-07 11:23:43+01 size | 53687091200 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 441354e7-c234-4079-b494-53fa99cdce6f imagestatus | 1 lastmodified | 2022-01-07 11:23:39.951+01 vm_snapshot_id | bd2291a4-8018-4874-a400-8d044a95347d volume_type | 2 volume_format | 4 image_group_id | 0b995271-e7f3-41b3-aff7-b5ad7942c10d _create_date | 2022-01-07 11:23:41.448463+01 _update_date | 2022-01-07 11:24:10.414777+01 active | t volume_classification | 0 qcow_compat | 2 -[ RECORD 2 ]---------+------------------------------------- image_guid | 441354e7-c234-4079-b494-53fa99cdce6f creation_date | 2021-12-15 07:16:31.647+01 size | 53687091200 it_guid | 00000000-0000-0000-0000-000000000000 parentid | fdf38f20-3416-4d75-a159-2a341b1ed637 imagestatus | 1 lastmodified | 2022-01-07 11:23:41.448+01 vm_snapshot_id | 2d610958-59e3-4685-b209-139b4266012f volume_type | 2 volume_format | 4 image_group_id | 0b995271-e7f3-41b3-aff7-b5ad7942c10d _create_date | 2021-12-15 07:16:32.37005+01 _update_date | 2022-01-07 11:23:41.448463+01 active | f volume_classification | 1 qcow_compat | 0 -[ RECORD 3 ]---------+------------------------------------- image_guid | fdf38f20-3416-4d75-a159-2a341b1ed637 creation_date | 2020-08-12 17:16:07+02 size | 53687091200 it_guid | 00000000-0000-0000-0000-000000000000 parentid | 00000000-0000-0000-0000-000000000000 imagestatus | 4 lastmodified | 2021-12-15 07:16:32.369+01 vm_snapshot_id | 603811ba-3cdd-4388-a971-05e300ced0c3 volume_type | 2 volume_format | 4 image_group_id | 0b995271-e7f3-41b3-aff7-b5ad7942c10d _create_date | 2020-08-12 17:16:07.506823+02 _update_date | 2021-12-15 07:16:32.37005+01 active | f volume_classification | 1 qcow_compat | 2 However in the engine gui I see only two snapshots ID: 1- 10df3adb-38f4-41d1-be84-b8b5b86e92cc (status ok) 2- 441354e7-c234-4079-b494-53fa99cdce6f (Disk status illegal) So the situation is: - on the host I see two volumes, status OK - on the engine GUI I see two volumes, one OK and the other one, disk in illegal status - on the engine DB I see three volumes, that are the combo of the two previous situations I would avoid restarting the VM, any advice for fixing this messy situation? I can attach engine log/vdsm.log Thank you for your time and help, Francesco

3 2

Re: How to debug "Non Operational" host
by Gervais de Montbrun 10 Jan '22

10 Jan '22

Usually, it looks like this: No more information than Non Operational. I have yet been able to mount vmstore. Are there any tricks that I can try to accomplish this? It seems to me that it is something in vdsm that is not working. Cheers, Gervais > On Nov 24, 2021, at 5:42 PM, Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk> wrote: > > > > That's where the ! mark would show any detected issues. > It's always safer to restart when in maintenance mode. > > the unsynched entries won't heal without the endpoints connected. > > Have you managed to get the vmstore mounted on ovirt1? > > Regard, > > Paul S. > > > From: Gervais de Montbrun <gervais(a)demontbrun.com <mailto:gervais@demontbrun.com>> > Sent: 24 November 2021 21:06 > To: Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk <mailto:P.Staniforth@leedsbeckett.ac.uk>> > Cc: users(a)ovirt.org <mailto:users@ovirt.org> <users(a)ovirt.org <mailto:users@ovirt.org>> > Subject: Re: [ovirt-users] How to debug "Non Operational" host > > Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. > Hi Paul (and anyone else who wants to chime in), > > By "host's section of webadmin," do you mean here? > <PastedGraphic-6.png> > > If so, then there is no indication that gluster is not working properly (even if I take it out of maintenance mode and it is showing non-operational). > > re: "gluster volume heal vmstore info summary" > This is what I see: > ovirt1: > [root(a)ovirt1.dgi <mailto:root@ovirt1.dgi> ~]# gluster volume heal vmstore info summary > Brick ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Connected > Total Number of entries: 27 > Number of entries in heal pending: 27 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Transport endpoint is not connected > Total Number of entries: - > Number of entries in heal pending: - > Number of entries in split-brain: - > Number of entries possibly healing: - > > Brick ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Transport endpoint is not connected > Total Number of entries: - > Number of entries in heal pending: - > Number of entries in split-brain: - > Number of entries possibly healing: - > > ovirt2: > [root(a)ovirt2.dgi <mailto:root@ovirt2.dgi> ~]# gluster volume heal vmstore info summary > Brick ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Connected > Total Number of entries: 46 > Number of entries in heal pending: 46 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Transport endpoint is not connected > Total Number of entries: - > Number of entries in heal pending: - > Number of entries in split-brain: - > Number of entries possibly healing: - > > Brick ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Transport endpoint is not connected > Total Number of entries: - > Number of entries in heal pending: - > Number of entries in split-brain: - > Number of entries possibly healing: - > > ovirt3: > [root(a)ovirt3.dgi <mailto:root@ovirt3.dgi> ~]# gluster volume heal vmstore info summary > Brick ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Connected > Total Number of entries: 46 > Number of entries in heal pending: 46 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Transport endpoint is not connected > Total Number of entries: - > Number of entries in heal pending: - > Number of entries in split-brain: - > Number of entries possibly healing: - > > Brick ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore > Status: Transport endpoint is not connected > Total Number of entries: - > Number of entries in heal pending: - > Number of entries in split-brain: - > Number of entries possibly healing: > > Forgive me for pasting all this in. My experience with gluster has been installing it as part of my hypervised ovirt install. It's generally just worked ok without any intervention. > > I see similar things for the engine brick also. Example: > ovirt2: > [root(a)ovirt2.dgi <mailto:root@ovirt2.dgi> ~]# gluster volume heal engine info summary > Brick ovirt1-storage.dgi:/gluster_bricks/engine/engine > Status: Connected > Total Number of entries: 1 > Number of entries in heal pending: 1 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Brick ovirt2-storage.dgi:/gluster_bricks/engine/engine > Status: Transport endpoint is not connected > Total Number of entries: - > Number of entries in heal pending: - > Number of entries in split-brain: - > Number of entries possibly healing: - > > Brick ovirt3-storage.dgi:/gluster_bricks/engine/engine > Status: Connected > Total Number of entries: 1 > Number of entries in heal pending: 1 > Number of entries in split-brain: 0 > Number of entries possibly healing: 0 > > Is it safe to restart glusterd on all the servers? So far, I only did a restart on ovirt1 as it was in maintenance mode anyway. > > Cheers, > Gervais > > > >> On Nov 24, 2021, at 4:44 PM, Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk <mailto:P.Staniforth@leedsbeckett.ac.uk>> wrote: >> >> Hi Gervais, >> >> The other mounts are the storage domains, as ovirt1 only has 1 storage domain mounted this is probably why the host is showing as non-operational. >> >> In the host's section of webadmin does it indicate any gluster issues? sometimes it detects these and gives you the option to restart gluster in the general tab of the host. >> >> Also, on any of the hosts to check all the gluster nodes are connected you can run >> >> "gluster volume heal vmstore info summary" >> >> then you can manually restart the glusterd service. >> >> e.g. "systemctl restart glusterd.service" >> >> Regards, >> >> Paul S. >> From: Gervais de Montbrun <gervais(a)demontbrun.com <mailto:gervais@demontbrun.com>> >> Sent: 24 November 2021 18:15 >> To: Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk <mailto:P.Staniforth@leedsbeckett.ac.uk>> >> Cc: users(a)ovirt.org <mailto:users@ovirt.org> <users(a)ovirt.org <mailto:users@ovirt.org>> >> Subject: Re: [ovirt-users] How to debug "Non Operational" host >> >> Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. >> Hi Paul. >> >> I've updated the /etc/hosts file on the engine and restarted the hosted-engine. It seems that the "Could not associate brick" errors have stopped. Thank you! >> No change in my issue though ☹️. >> >> It looks like Glusterd is starting up OK. I did force start the bricks on ovirt1. >> [root@ <mailto:root@ovirt1.dgi>ovirt1.dgi <mailto:root@ovirt1.dgi> ~]# systemctl status glusterd >> ● glusterd.service - GlusterFS, a clustered file-system server >> Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) >> Drop-In: /etc/systemd/system/glusterd.service.d >> └─99-cpu.conf >> Active: active (running) since Wed 2021-11-24 16:19:25 UTC; 1h 42min ago >> Docs: man:glusterd(8) >> Process: 2321 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) >> Main PID: 2354 (glusterd) >> Tasks: 92 (limit: 1648201) >> Memory: 63.7G >> CPU: 1h 17min 42.666s >> CGroup: /glusterfs.slice/glusterd.service >> ├─2354 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO >> ├─3247 /usr/sbin/glusterfsd -s ovirt1-storage.dgi --volfile-id vmstore.ovirt1-storage.dgi.gluster_bricks-vmstore-vmstore -p /var/run/gluster/vols/vmstore/ovirt1-storage.dgi-gluster_bricks-vmstore-vmstore.pid -S /var/run/gluster/fb93ffff591764c8.socket --brick-name /gluster_bricks/vmstore/vmstore -l /var/log/glusterfs/bricks/gluster_b> >> ├─3275 /usr/sbin/glusterfsd -s ovirt1-storage.dgi --volfile-id engine.ovirt1-storage.dgi.gluster_bricks-engine-engine -p /var/run/gluster/vols/engine/ovirt1-storage.dgi-gluster_bricks-engine-engine.pid -S /var/run/gluster/66ebd47080b528d1.socket --brick-name /gluster_bricks/engine/engine -l /var/log/glusterfs/bricks/gluster_bricks-en> >> └─3287 /usr/sbin/glusterfs -s localhost --volfile-id shd/engine -p /var/run/gluster/shd/engine/engine-shd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c9b8692f3e532562.socket --xlator-option *replicate*.node-uuid=fdf2cf13-c2c5-4afa-8d73-76c50c69122a --process-name glustershd --client-pid=-6 >> >> Nov 24 16:19:22 ovirt1.dgi systemd[1]: Starting GlusterFS, a clustered file-system server... >> Nov 24 16:19:25 ovirt1.dgi systemd[1]: Started GlusterFS, a clustered file-system server. >> Nov 24 16:19:28 ovirt1.dgi glusterd[2354]: [2021-11-24 16:19:28.909836] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume engine. Stopping local bricks. >> Nov 24 16:19:28 ovirt1.dgi glusterd[2354]: [2021-11-24 16:19:28.910745] C [MSGID: 106002] [glusterd-server-quorum.c:355:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume vmstore. Stopping local bricks. >> Nov 24 16:19:31 ovirt1.dgi glusterd[2354]: [2021-11-24 16:19:31.925206] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume engine. Starting local bricks. >> Nov 24 16:19:31 ovirt1.dgi glusterd[2354]: [2021-11-24 16:19:31.938507] C [MSGID: 106003] [glusterd-server-quorum.c:348:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume vmstore. Starting local bricks. >> >> The bricks are showing all green at least in the UI, but they never seem to catch up and not show any unsynced entries: >> <PastedGraphic-5.png> >> >> As for mounting the bricks, they are mounting based on what is in /etc/fstab. >> [root@ <mailto:root@ovirt1.dgi>ovirt1.dgi <mailto:root@ovirt1.dgi> ~]# cat /etc/fstab >> # >> # /etc/fstab >> # Created by anaconda on Wed Feb 17 20:17:28 2021 >> # >> # Accessible filesystems, by reference, are maintained under '/dev/disk/'. >> # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info. >> # >> # After editing this file, run 'systemctl daemon-reload' to update systemd >> # units generated from this file. >> # >> /dev/onn/ovirt-node-ng-4.4.9-0.20211026.0+1 / xfs defaults,discard 0 0 >> UUID=07e9dfea-9710-483d-bc63-aea7dbb5801d /boot xfs defaults 0 0 >> /dev/mapper/onn-home /home xfs defaults,discard 0 0 >> /dev/mapper/onn-tmp /tmp xfs defaults,discard 0 0 >> /dev/mapper/onn-var /var xfs defaults,discard 0 0 >> /dev/mapper/onn-var_log /var/log xfs defaults,discard 0 0 >> /dev/mapper/onn-var_log_audit /var/log/audit xfs defaults,discard 0 0 >> /dev/mapper/onn-swap none swap defaults 0 0 >> UUID=4e2c88e4-2bae-4b41-bb62-631820435845 /gluster_bricks/engine xfs inode64,noatime,nodiratime 0 0 >> UUID=ad938b6e-44d2-492a-a313-c4d0c0608e09 /gluster_bricks/vmstore xfs inode64,noatime,nodiratime 0 0 >> >> [root@ <mailto:root@ovirt1.dgi>ovirt1.dgi <mailto:root@ovirt1.dgi> ~]# mount |grep gluster_bricks >> /dev/mapper/gluster_vg_sdb-gluster_lv_engine on /gluster_bricks/engine type xfs (rw,noatime,nodiratime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=3072,noquota) >> /dev/mapper/gluster_vg_sdb-gluster_lv_vmstore on /gluster_bricks/vmstore type xfs (rw,noatime,nodiratime,seclabel,attr2,inode64,logbufs=8,logbsize=256k,sunit=512,swidth=3072,noquota) >> >> What does this "extra" mount? >> [root@ <mailto:root@ovirt1.dgi>ovirt1.dgi <mailto:root@ovirt1.dgi> ~]# mount | grep storage >> ovirt1-storage.dgi:/engine on /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) >> >> I've noticed that on my working servers, I see this output: >> [root@ <mailto:root@ovirt2.dgi>ovirt2.dgi <mailto:root@ovirt2.dgi> ~]# mount | grep storage >> ovirt1-storage.dgi:/engine on /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) >> ovirt1-storage.dgi:/vmstore on /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_vmstore type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) >> >> There is obviously something not mounting properly on ovirt1. I don't see how this can be a network issue as the storage for the hosted engine is working ok. >> >> I truly appreciate the help. Any other ideas or logs/places to check? >> >> Cheers, >> Gervais >> >> >> >>> On Nov 24, 2021, at 4:03 AM, Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk <mailto:P.Staniforth@leedsbeckett.ac.uk>> wrote: >>> >>> >>> Hi Gervais, >>> >>> The engine doesn't need to be able to ping the IP address, just needs to know what it is so adding them to the /etc/hosts file should work. >>> >>> Also, I would check ovirt1, is it mounting the brick, what does "systemctl status glusterd" show, what are the logs in /var/log/gluster ? >>> >>> >>> Regards, >>> >>> Paul S. >>> From: Gervais de Montbrun <gervais(a)demontbrun.com <mailto:gervais@demontbrun.com>> >>> Sent: 24 November 2021 01:16 >>> To: Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk <mailto:P.Staniforth@leedsbeckett.ac.uk>> >>> Cc: Vojtech Juranek <vjuranek(a)redhat.com <mailto:vjuranek@redhat.com>>; users(a)ovirt.org <mailto:users@ovirt.org> <users(a)ovirt.org <mailto:users@ovirt.org>> >>> Subject: Re: [ovirt-users] How to debug "Non Operational" host >>> >>> Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. >>> Hi Paul, >>> >>> I don't quite get what you mean by this: >>> >>>> assuming you have a storage network for the gluster nodes the engine needs to resolve be able to resolve the host addresses >>> >>> >>> The storage network is on 10GB network cards and plugged into a stand-alone switch. The hosted-engine is not on the same network at all and can not ping the IP's associated with those cards. Are you saying that it needs access to that network, or that is needs to be able to resolve the IP's. I can add them to the /etc/hosts file on the ovirt-engine or do I need to reconfigure my setup? It was working as it currently configured before applying the update. >>> >>> I have no idea why the ovirt1 server is not showing up with the fqdn. I set up all the servers the same way. It's been like that since I set things up. I have looked for where this might be corrected, but can't find it. Ideas? >>> >>> The yellow bricks... I can force start them (and I have in the past), but now it turns green for a few minutes and then returns to red. >>> >>> Cheers, >>> Gervais >>> >>> >>> >>>> On Nov 23, 2021, at 12:57 PM, Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk <mailto:P.Staniforth@leedsbeckett.ac.uk>> wrote: >>>> >>>> Hello Gervais, >>>> >>>> is the brick mounted on ovirt1 ? can you mount it using the settings in /etc/fstab ? >>>> >>>> The hostname is not using a FQDN for ovirt1 >>>> >>>> assuming you have a storage network for the gluster nodes the engine needs to resolve be able to resolve the host addresses >>>> ovirt1-storage.dgi >>>> ovirt2-storage.dgi >>>> ovirt3-storage.dgi >>>> >>>> So that it can assign them to the correct network. >>>> >>>> When the volume is showing yellow you can force restart them again from the GUI. >>>> >>>> Regards, >>>> >>>> Paul S. >>>> From: Gervais de Montbrun <gervais(a)demontbrun.com <mailto:gervais@demontbrun.com>> >>>> Sent: 23 November 2021 13:42 >>>> To: Vojtech Juranek <vjuranek(a)redhat.com <mailto:vjuranek@redhat.com>> >>>> Cc: users(a)ovirt.org <mailto:users@ovirt.org> <users(a)ovirt.org <mailto:users@ovirt.org>> >>>> Subject: [ovirt-users] Re: How to debug "Non Operational" host >>>> >>>> Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. >>>> >>>> Hi Vojta, >>>> >>>> Thanks for the help. >>>> >>>> I tried to activate my server this morning and captured the logs from vdsm.log and engine.log. They are attached. >>>> >>>> Something went awry with my gluster (I think) as it is showing that the bricks on the affected server (ovirt1) are not mounted: >>>> <PastedGraphic-2.png> >>>> >>>> <PastedGraphic-3.png> >>>> >>>> <PastedGraphic-4.png> >>>> >>>> >>>> The networking looks fine. >>>> >>>> Cheers, >>>> Gervais >>>> >>>> >>>> >>>> > On Nov 23, 2021, at 3:37 AM, Vojtech Juranek <vjuranek(a)redhat.com <mailto:vjuranek@redhat.com>> wrote: >>>> > >>>> > On Tuesday, 23 November 2021 03:36:07 CET Gervais de Montbrun wrote: >>>> >> Hi Folks, >>>> >> >>>> >> I did a minor upgrade on the first host in my cluster and now it is >>>> >> reporting "Non Operational" >>>> >> >>>> >> This is what yum showed as updatable. However, I did the update through the >>>> >> ovirt-engine web interface. >>>> >> >>>> >> ovirt-node-ng-image-update.noarch >>>> >> 4.4.9-1.el8 >>>> >> ovirt-4.4 Obsoleting Packages >>>> >> ovirt-node-ng-image-update.noarch >>>> >> 4.4.9-1.el8 >>>> >> ovirt-4.4 ovirt-node-ng-image-update.noarch >>>> >> 4.4.8.3-1.el8 >>>> >> @System ovirt-node-ng-image-update.noarch >>>> >> 4.4.9-1.el8 >>>> >> ovirt-4.4 >>>> >> ovirt-node-ng-image-update-placeholder.noarch >>>> >> 4.4.8.3-1.el8 >>>> >> @System >>>> >> >>>> >> How do I start to debug this issue? >>>> > >>>> > Check engine log in /var/log/ovirt-engine/engine.log on the machine where >>>> > engine runs >>>> > >>>> >> >>>> >> >>>> >> Also, it looks like the vmstore brick is not mounting on that host. I only >>>> >> see the engine mounted. >>>> > >>>> > >>>> > Could you also attach relevant part of vdsm log (/var/log/vdsm/vdsm.log) from >>>> > the machine where mount failed? You should see some mount related error there. >>>> > This could be also a reason why hosts become non-operational. >>>> > >>>> > Thanks >>>> > Vojta >>>> > >>>> >> Broken server: >>>> >> root(a)ovirt1.dgi <mailto:root@ovirt1.dgi> log]# mount | grep storage >>>> >> ovirt1-storage.dgi:/engine on >>>> >> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type >>>> >> fuse.glusterfs >>>> >> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read= >>>> >> 131072) Working server: >>>> >> [root(a)ovirt2.dgi <mailto:root@ovirt2.dgi> ~]# mount | grep storage >>>> >> ovirt1-storage.dgi:/engine on >>>> >> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type >>>> >> fuse.glusterfs >>>> >> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read= >>>> >> 131072) ovirt1-storage.dgi:/vmstore on >>>> >> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_vmstore type >>>> >> fuse.glusterfs >>>> >> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read= >>>> >> 131072) >>>> >> >>>> >> >>>> >> I tried putting the server into maintenance mode and running a reinstall on >>>> >> it. No change. I'de really appreciate some help sorting this our. >>>> >> >>>> >> Cheers, >>>> >> Gervais >>>> > >>>> > _______________________________________________ >>>> > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >>>> > To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >>>> > Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>>> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>>> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/S6C7R6LUTJXFM… <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >>>> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/AWSWTXS6CEAYS… <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> >>>> To view the terms under which this email is distributed, please go to:- >>>> https://leedsbeckett.ac.uk/disclaimer/email <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fleedsbeck…>_______________________________________________ >>>> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >>>> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4YYEHIXZCSQCN… <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> >>> To view the terms under which this email is distributed, please go to:- >>> https://leedsbeckett.ac.uk/disclaimer/email <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fleedsbeck…>_______________________________________________ >>> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >>> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >>> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/H7SLDPRW6FJ5S… <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> >> To view the terms under which this email is distributed, please go to:- >> https://leedsbeckett.ac.uk/disclaimer/email <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fleedsbeck…>_______________________________________________ >> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FCXTUGI2G2GXN… <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> > To view the terms under which this email is distributed, please go to:- > https://leedsbeckett.ac.uk/disclaimer/email <https://leedsbeckett.ac.uk/disclaimer/email>_______________________________________________ > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html <https://www.ovirt.org/privacy-policy.html> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ <https://www.ovirt.org/community/about/community-guidelines/> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OZSTLXDSHK6AW… <https://lists.ovirt.org/archives/list/users@ovirt.org/message/OZSTLXDSHK6AW…>

2 7

did 4.3.9 reset bug https://bugzilla.redhat.com/show_bug.cgi?id=1590266
by kelley bryan 10 Jan '22

10 Jan '22

I am experiencing the error message in the ovirt-hosted-engine-setup-ansible-create_target_vm log {2020-05-06 14:15:30,024-0500 ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u"Fail if Engine IP is different from engine's he_fqdn resolved IP", 'ansible_result': u'type: <type \'dict\'>\nstr: {\'msg\': u"Engine VM IP address is while the engine\'s he_fqdn ovirt1-engine.kelleykars.org resolves to 192.168.122.2. If you are using DHCP, check your DHCP reservation configuration", \'changed\': False, \'_ansible_no_log\': False}', 'task_duration': 1, 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml'}}:Q! The bug 1590266 says it should report the engine VM IP address xxx.xxx.xxx.xxx while the Engines he_fqdn is xxxxxxxxx I need to see what it thins is wrong as both dig fqdn engine name and dig -x ip return the correct information. Now this bug looks like it may play but I don't see the failed rediness check in the this log https://access.redhat.com/solutions/4462431 or is it because the vm fails or dies or ???

3 7

Help installing oVirt on single machine, without cockpit
by Cameronsplaze222＠gmail.com 10 Jan '22

10 Jan '22

I'm trying to install both the oVirt engine (so I have the API), and hypervisor on the same machine. I'm hoping I don't need cockpit, and I can write some custom scripts to manage oVirt through the API. I tried a couple ways, and both were blocked: # on CentOS 9 stream, installing oVirt engine: On a fresh CentOS install, tried following the directions [here](https://www.ovirt.org/documentation/installing_ovirt_as_a_standalone_…. They failed at step 3.2, when adding the modules (missing groups or modules). Tried skipping it, but step 3.3.2 failed when running `dnf install ovirt-engine`, package couldn't be found. The closest package I see is `ovirt-hosted-engine-setup-2.5.4-1.el9.noarch`. Is that what I want? # On oVirt node, installing engine: Couldn't get the networking working, even though I added it during the install process. Also tried wired connection, and no luck. Blocked early on, but read this was SUPER minimal anyway, so maybe this wasn't the right path. I'm just trying to have a working API, along side the hypervisor to run vm's, both on the same machine. I'm down to go a third way too, if anyone has any ideas. Thanks!

7 17

After upgrade to vdsm-4.40.90.4-1.el8 - Internal JSON-RPC error - how to fix?
by John Mortensen 09 Jan '22

09 Jan '22

Hi, After we upgraded to vdsm-4.40.90.4-1.el8 on our two node cluster two things has happned: 1. First node that was upgraded now continuously logs this error: VDSM <node name> command Get Host Statistics failed: Internal JSON-RPC error: {'reason': "'str' object has no attribute 'decode'"} 2. During import of two virtual machines from VMware (has done multible before upgrade) the import seems to never finish - currently running on day 2-3... any clues how to fix this? /John

3 4

Re: [External] : Unrecoverable NMI error on HP Gen8 hosts.
by Diggy Mc 09 Jan '22

09 Jan '22

> Hi Diggy, > > I'm not sure if it's an oVirt issue, but it can be a network or firewall issue. > Did you test the connection between oVirt hosts and the iLO interfaces? > Simple tests like ping to ensure one host can reach others iLO interfaces and ipmitool to > ensure you can connect to the management interfaces? > > Marcos > It is not network or firewall related. There is no firewall between the oVirt hosts/engine and the iLO interfaces. When I configured oVirt power management, the offered test passed. I'm not running IPMI. On a side note, the problem exists both before and after enabling oVirt's power management feature.

2 1

Instability after update
by Andrea Chierici 09 Jan '22

09 Jan '22

Cheers, I am in trouble... I am running an ovirt manager v4.4.9.5-1.el8. This week I updated all the hosts to latest release (starting from repo ovirt-release44-4.4.9.3-1.el8.noarch) since I hadn't done that when I upgraded the manager. Before that the system was rock solid. Unfortunately, after the upgrade, I get frequent errors on running VMs: VM XXXXXXXX is down with error. Exit message: Lost connection with qemu process. I can't get any hint from the logs, I wonder if someone has any idea of what is going on. Thanks, Andrea -- Andrea Chierici - INFN-CNAF Viale Berti Pichat 6/2, 40127 BOLOGNA Office Tel: +39 051 2095463 SkypeID ataruz --

3 7

Unrecoverable NMI error on HP Gen8 hosts.
by Diggy Mc 08 Jan '22

08 Jan '22

I have oVirt Node v4.4.8.3 running on several HP ProLiant Gen8 servers. I receive the following error under certain circumstances: "An Unrecoverable System Error (NMI) has occurred (iLO application watchdog timeout NMI, Service Information: 0x0000002B, 0x00000000)" When a host starts taking a load (but nowhere near a threshold), I encounter the above iLO-logged error and the host locks-up. I have had to grossly under-utilize my hosts to avoid this problem. I'm hoping for a better fix or work-around. I've had the same problem beginning with my oVirt 4.3.x hosts, so it isn't oVirt version specific. The little information I could find on the error wasn't helpful. Red Hat acknowledges the issue, but limited to shutdown/reboot operations; not during "normal" operations. Anyone else experienced this problem? How did you fix it or work around it? I'd like to better utilize my servers if possible. In advance, thank you to anyone and everyone who offers help.

5 6