- Users - oVirt List Archives

Unable start self hosted engine after accidental shut down
by Eugène Ngontang 19 May '21

19 May '21

Hi, Our self hosted engine has been accidentally shut down by a teammate and now I'm trying hard to get it back up without success. I've tried the --vm-start command but it says the VM is in WaitForLaunch status. I've set the global maintenance mode but it does nothing. root@milhouse-main ~]# hosted-engine --vm-start VM exists and is down, cleaning up and restarting VM in WaitForLaunch [root@milhouse-main ~]# hosted-engine --set-maintenance --mode=global [root@milhouse-main ~]# hosted-engine --vm-status !! Cluster is in GLOBAL MAINTENANCE mode !! --== Host milhouse-main.envrmnt.local (id: 1) status ==-- conf_on_shared_storage : True Status up-to-date : False Hostname : milhouse-main.envrmnt.local Host ID : 1 Engine status : unknown stale-data Score : 3400 stopped : False Local maintenance : False crc32 : 931b2db9 local_conf_timestamp : 1642052 Host timestamp : 1642052 Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1642052 (Tue May 18 19:52:59 2021) host-id=1 score=3400 vm_conf_refresh_time=1642052 (Tue May 18 19:53:00 2021) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False !! Cluster is in GLOBAL MAINTENANCE mode !! You have new mail in /var/spool/mail/root [root@milhouse-main ~]# hosted-engine --vm-start VM exists and is down, cleaning up and restarting VM in WaitForLaunch [root@milhouse-main ~]# And when I list all vms, I can see the hosted engine is in the Shut Off status and the managed vms are all paused [root@milhouse-main ~]# virsh -r list --all setlocale: No such file or directory Id Name State ---------------------------------------------------- 2 hp_gpu-node11 paused 3 fp_gpu-node5 paused 4 hp_gpu-node10 paused 5 hp_gpu-node7 paused 6 cpu-node3 paused 7 hp_gpu-node5 paused 8 fp_gpu-node1 paused 9 fp_gpu-node0 paused 10 cpu-node1 paused 11 fp_gpu-node6 paused 12 hp_gpu-node8 paused 13 fp_gpu-node10 paused 14 fp_gpu-node4 paused 15 fp_gpu-node9 paused 16 hp_gpu-node4 paused 17 fp_gpu-node15 paused 18 fp_gpu-node8 paused 19 hp_gpu-node0 paused 20 fp_gpu-node14 paused 21 fp_gpu-node2 paused 22 fp_gpu-node11 paused 23 hp_gpu-node9 paused 24 cpu-node2 paused 25 hp_gpu-node1 paused 26 hp_gpu-node2 paused 27 fp_gpu-node12 paused 28 hp_gpu-node3 paused 29 hp_gpu-node6 paused 30 infra-vm paused 31 cpu-node0 paused 32 fp_gpu-node3 paused 33 fp_gpu-node7 paused 34 fp_gpu-node13 paused 35 bigip-16.1x-milhouse paused - HostedEngine shut off [root@milhouse-main ~]# I don't want to reboot the host server, cause I could loose all my VMs. Can someone help here please? Thanks. Regards, Eugène NG -- LesCDN <http://lescdn.com> engontang(a)lescdn.com ------------------------------------------------------------ *Aux hommes il faut un chef, et au* * chef il faut des hommes!L'habit ne fait pas le moine, mais lorsqu'on te voit on te juge!*

4 4

Error Deployment New Hosted Engine on 4.4.1 node
by Dominique D 19 May '21

19 May '21

I have a setup with 3 servers production environment with version 4.4.1 that works very well (gluster and hosted-engine) I prepared a lab with the same setup (with the same ISO) to test a update to (4.4.6) The deployment of the Hyperconverge works but when I want to do hosted-engine --deploy or in the cokpit, at some time the host displays this message and reboot and the engine is not installed. I tried with the latest ISO 4.4.6 (Fresh Install) and I have the same message. Is there a network change with version 8.3 of CentOS? [ 33.785588] bondscan-yhEyFK: option fail_over_mac: invalid value (3) [ 33.787183] bondscan-yhEyFK: option arp_all_targets: invalid value (2) [ 33.804832] bondscan-yhEyFK: option arp_validate: invalid value (7) [ 33.807568] bondscan-yhEyFK: option xmit_hash_policy: invalid value (6) [ 33.807884] bondscan-yhEyFK: option lacp_rate: mode dependency failed, not supported in mode broadcast(3) [ 33.809858] bondscan-yhEyFK: option ad_select: invalid value (3) [ 33.811003] bondscan-yhEyFK: option primary_reselect: invalid value (3) [ 33.813490] bondscan-yhEyFK: option fail_over_mac: invalid value (3) [ 33.815175] bondscan-yhEyFK: option arp_all_targets: invalid value (2) [ 33.839036] bondscan-yhEyFK: option arp_validate: mode dependency failed, not supported in mode 802.3ad(4) [ 33.841905] bondscan-yhEyFK: option xmit_hash_policy: invalid value (6) [ 33.842628] bondscan-yhEyFK: option lacp_rate: invalid value (2) [ 33.844573] bondscan-yhEyFK: option ad_select: invalid value (3) [ 33.846293] bondscan-yhEyFK: option primary_reselect: invalid value (3) [ 33.847492] bondscan-yhEyFK: option fail_over_mac: invalid value (3) [ 33.848978] bondscan-yhEyFK: option arp_all_targets: invalid value (2) [ 33.871465] bondscan-yhEyFK: option arp_validate: mode dependency failed, not supported in mode balance-tlb(5) [ 33.874149] bondscan-yhEyFK: option xmit_hash_policy: invalid value (6) [ 33.874475] bondscan-yhEyFK: option lacp_rate: mode dependency failed, not supported in mode balance-tlb(5) [ 33.876514] bondscan-yhEyFK: option ad_select: invalid value (3) [ 33.877381] bondscan-yhEyFK: option primary_reselect: invalid value (3) [ 33.878554] bondscan-yhEyFK: option fail_over_mac: invalid value (3) [ 33.879991] bondscan-yhEyFK: option arp_all_targets: invalid value (2) [ 33.909461] bondscan-yhEyFK: option arp_validate: mode dependency failed, not supported in mode balance-alb(6) [ 33.914593] bondscan-yhEyFK: option xmit_hash_policy: invalid value (6) [ 33.916657] bondscan-yhEyFK: option lacp_rate: mode dependency failed, not supported in mode balance-alb(6) [ 33.920665] bondscan-yhEyFK: option ad_select: invalid value (3) [ 33.927749] bondscan-yhEyFK: option primary_reselect: invalid value (3) [ 33.932404] bondscan-yhEyFK: option fail_over_mac: invalid value (3) [ 33.935589] bondscan-yhEyFK: option arp_all_targets: invalid value(2)

3 5

Gluster volumes not healing (perhaps after host maintenance?)
by David White 18 May '21

18 May '21

I discovered that the servers I purchased did not come with 10Gbps network cards, like I thought they did. So my storage network has been running on a 1Gbps connection for the past week, since I deployed the servers into the datacenter a little over a week ago. I purchased 10Gbps cards, and put one of my hosts into maintenance mode yesterday, prior to replacing the daughter card. It is now back online running fine on the 10Gbps card. All VMs seem to be working, even when I migrate them onto cha2, which is the host I did maintenance on yesterday morning. The other two hosts are still running on the 1Gbps connection, but I plan to do maintenance on them next week. The oVirt manager shows that all 3 hosts are up, and that all of my volumes - and all of my bricks - are up. However, every time I look at the storage, it appears that the self-heal info for 1 of the volumes is 10 minutes, and the self-heal info for another volume is 50+ minutes. This morning is the first time in the last couple of days that I've paid close attention to the numbers, but I don't see them going down. When I log into each of the hosts, I do see everything is connected in gluster. It is interesting to me, in this particular case, though that gluster on cha3 notices the hostname of 10.1.0.10 to be the IP address, and not the hostname (cha1). The host that I did the maintenance on is cha2. [root@cha3-storage dwhite]# gluster peer statusNumber of Peers: 2Hostname: 10.1.0.10Uuid: 87a4f344-321a-48b9-adfb-e3d2b56b8e7bState: Peer in Cluster (Connected)Hostname: cha2-storage.mgt.barredowlweb.comUuid: 93e12dee-c37d-43aa-a9e9-f4740b9cab14State: Peer in Cluster (Connected) When I run `gluster volume heal data`, I see the following: [root@cha3-storage dwhite]# gluster volume heal data Launching heal operation to perform index self heal on volume data has been unsuccessful: Commit failed on cha2-storage.mgt.barredowlweb.com. Please check log file for details. I get the same results if I run the command on cha2, for any volume: [root@cha2-storage dwhite]# gluster volume heal data Launching heal operation to perform index self heal on volume data has been unsuccessful: Glusterd Syncop Mgmt brick op 'Heal' failed. Please check glustershd log file for details. [root@cha2-storage dwhite]# gluster volume heal vmstore Launching heal operation to perform index self heal on volume vmstore has been unsuccessful: Glusterd Syncop Mgmt brick op 'Heal' failed. Please check glustershd log file for details. I see a lot of stuff like this on cha2 /var/log/glusterfs/glustershd.log: [2021-04-24 11:33:01.319888] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 2-engine-client-0: changing port to 49153 (from 0)[2021-04-24 11:33:01.329463] I [MSGID: 114057] [client-handshake.c:1128:select_server_supported_programs] 2-engine-client-0: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}][2021-04-24 11:33:01.330075] W [MSGID: 114043] [client-handshake.c:727:client_setvolume_cbk] 2-engine-client-0: failed to set the volume [{errno=2}, {error=No such file or directory}][2021-04-24 11:33:01.330116] W [MSGID: 114007] [client-handshake.c:752:client_setvolume_cbk] 2-engine-client-0: failed to get from reply dict [{process-uuid}, {errno=22}, {error=Invalid argument}][2021-04-24 11:33:01.330140] E [MSGID: 114044] [client-handshake.c:757:client_setvolume_cbk] 2-engine-client-0: SETVOLUME on remote-host failed [{remote-error=Brick not found}, {errno=2}, {error=No such file or directory}][2021-04-24 11:33:01.330155] I [MSGID: 114051] [client-handshake.c:879:client_setvolume_cbk] 2-engine-client-0: sending CHILD_CONNECTING event [][2021-04-24 11:33:01.640480] I [rpc-clnt.c:1975:rpc_clnt_reconfig] 3-vmstore-client-0: changing port to 49154 (from 0)The message "W [MSGID: 114007] [client-handshake.c:752:client_setvolume_cbk] 3-vmstore-client-0: failed to get from reply dict [{process-uuid}, {errno=22}, {error=Invalid argument}]" repeated 4 times between [2021-04-24 11:32:49.602164] and [2021-04-24 11:33:01.649850][2021-04-24 11:33:01.649867] E [MSGID: 114044] [client-handshake.c:757:client_setvolume_cbk] 3-vmstore-client-0: SETVOLUME on remote-host failed [{remote-error=Brick not found}, {errno=2}, {error=No such file or directory}][2021-04-24 11:33:01.649969] I [MSGID: 114051] [client-handshake.c:879:client_setvolume_cbk] 3-vmstore-client-0: sending CHILD_CONNECTING event [][2021-04-24 11:33:01.650095] I [MSGID: 114018] [client.c:2225:client_rpc_notify] 3-vmstore-client-0: disconnected from client, process will keep trying to connect glusterd until brick's port is available [{conn-name=vmstore-client-0}] How do I further troubleshoot? Sent with ProtonMail Secure Email.

3 5

Re: Gluster Geo-Replication Fails
by Strahil Nikolov 18 May '21

18 May '21

Now to make it perfect , leave it running and analyze the AVCs with semanage. In the end SELINUX will remain working and geo-rep should be running also. I've previously tried the rpm generated from https://github.com/gluster/glusterfs-selinux but it didn't help at that time. If possible, give it a try. Best Regards,Strahil Nikolov On Tue, May 18, 2021 at 16:09, Simon Scott<simon(a)justconnect.ie> wrote: _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GB23MBKEQKZNK…

1 0

Re: oVirt deploy new HE Host problem
by Yedidyah Bar David 18 May '21

18 May '21

On Mon, May 17, 2021 at 10:34 AM Marko Vrgotic <M.Vrgotic(a)activevideo.com> wrote: > > Hi gentleman, > > > > Hope you had a great weekend. > > Can I assume that you will be able to look into log files this week ? > > > > As per Yedidyah’s comment, I stopped troubleshooting . > > > > Kindly awaiting your reply. Hi Marko, Please upload somewhere all of /var/log from all hosts and the engine, and share a link. Thanks. In particular, you didn't include 'hosted-engine --deploy' logs from /var/log/ovirt-hosted-engine-setup. Also: the attached ovirt-host-deploy log indicates this is on 4.3, which is EOL and unsupported. Best regards, -- Didi

2 1

Gluster Geo-Replication Fails
by simon＠justconnect.ie 18 May '21

18 May '21

Hi All, I have to sites each with 2 x 3 Node clusters. oVirt 4.4.5 with GlusterFS 8.4. On Site A I have a Replica 3 + Arbiter volume that I want to geo-replicate to Site B. I've setup a Replica 3 + Arbiter volume on Site B. All password-less ssh requirements are in place and the Geo-Replication Session has been created successfully. I issue the 'Start' command successfully but the status switches between 'Initializing', 'Active' & 'Failed' within 20 seconds constantly. Can anyone point me in the direction to get this up and running as it is our proposed DR solution. Kind Regards Shimme

3 3

poweroff and reboot with ovirt_vm ansible module
by Nathanaël Blanchet 18 May '21

18 May '21

Hello, is there a way to poweroff or reboot (without stopped and running state) a vm with the ovirt_vm ansible module? -- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

7 11

[ANN] Async release for oVirt 4.4.6
by Lev Veyde 18 May '21

18 May '21

oVirt 4.4.6 Async update #3 On May 18th 2021 the oVirt project released an async update to the following packages: - Vdsm 4.40.60.7 - oVirt Node 4.4.6.3 Fixing the following bugs: - Bug 1959945 <https://bugzilla.redhat.com/show_bug.cgi?id=1959945> - [NBDE] RHVH 4.4.6 host fails to startup, without prompting for passphrase - Bug 1955571 <https://bugzilla.redhat.com/show_bug.cgi?id=1955571> - Verify if we still need to omit ifcfg and clevis dracut modules for properly working bridged network - Bug 1950209 <https://bugzilla.redhat.com/show_bug.cgi?id=1950209> - Leaf images used by the VM is deleted by the engine during snapshot merge oVirt Node Changes: - Consume above oVirt updates - Updated to Gluster 8.5 <https://docs.gluster.org/en/latest/release-notes/8.5/> Full diff list: --- ovirt-node-ng-image-4.4.6.2.manifest-rpm 2021-05-14 08:58:12.581488678 +0200 +++ ovirt-node-ng-image-4.4.6.3.manifest-rpm 2021-05-18 13:09:07.858527812 +0200 @@ -220,7 +220,7 @@ -glusterfs-8.4-1.el8.x86_64 -glusterfs-cli-8.4-1.el8.x86_64 -glusterfs-client-xlators-8.4-1.el8.x86_64 -glusterfs-events-8.4-1.el8.x86_64 -glusterfs-fuse-8.4-1.el8.x86_64 -glusterfs-geo-replication-8.4-1.el8.x86_64 -glusterfs-server-8.4-1.el8.x86_64 +glusterfs-8.5-1.el8.x86_64 +glusterfs-cli-8.5-1.el8.x86_64 +glusterfs-client-xlators-8.5-1.el8.x86_64 +glusterfs-events-8.5-1.el8.x86_64 +glusterfs-fuse-8.5-1.el8.x86_64 +glusterfs-geo-replication-8.5-1.el8.x86_64 +glusterfs-server-8.5-1.el8.x86_64 @@ -383,6 +383,6 @@ -libgfapi0-8.4-1.el8.x86_64 -libgfchangelog0-8.4-1.el8.x86_64 -libgfrpc0-8.4-1.el8.x86_64 -libgfxdr0-8.4-1.el8.x86_64 -libglusterd0-8.4-1.el8.x86_64 -libglusterfs0-8.4-1.el8.x86_64 +libgfapi0-8.5-1.el8.x86_64 +libgfchangelog0-8.5-1.el8.x86_64 +libgfrpc0-8.5-1.el8.x86_64 +libgfxdr0-8.5-1.el8.x86_64 +libglusterd0-8.5-1.el8.x86_64 +libglusterfs0-8.5-1.el8.x86_64 @@ -643 +643 @@ -ovirt-node-ng-image-update-placeholder-4.4.6.2-1.el8.noarch +ovirt-node-ng-image-update-placeholder-4.4.6.3-1.el8.noarch @@ -651,2 +651,2 @@ -ovirt-release-host-node-4.4.6.2-1.el8.noarch -ovirt-release44-4.4.6.2-1.el8.noarch +ovirt-release-host-node-4.4.6.3-1.el8.noarch +ovirt-release44-4.4.6.3-1.el8.noarch @@ -754 +754 @@ -python3-gluster-8.4-1.el8.x86_64 +python3-gluster-8.5-1.el8.x86_64 @@ -940,15 +940,15 @@ -vdsm-4.40.60.6-1.el8.x86_64 -vdsm-api-4.40.60.6-1.el8.noarch -vdsm-client-4.40.60.6-1.el8.noarch -vdsm-common-4.40.60.6-1.el8.noarch -vdsm-gluster-4.40.60.6-1.el8.x86_64 -vdsm-hook-ethtool-options-4.40.60.6-1.el8.noarch -vdsm-hook-fcoe-4.40.60.6-1.el8.noarch -vdsm-hook-openstacknet-4.40.60.6-1.el8.noarch -vdsm-hook-vhostmd-4.40.60.6-1.el8.noarch -vdsm-hook-vmfex-dev-4.40.60.6-1.el8.noarch -vdsm-http-4.40.60.6-1.el8.noarch -vdsm-jsonrpc-4.40.60.6-1.el8.noarch -vdsm-network-4.40.60.6-1.el8.x86_64 -vdsm-python-4.40.60.6-1.el8.noarch -vdsm-yajsonrpc-4.40.60.6-1.el8.noarch +vdsm-4.40.60.7-1.el8.x86_64 +vdsm-api-4.40.60.7-1.el8.noarch +vdsm-client-4.40.60.7-1.el8.noarch +vdsm-common-4.40.60.7-1.el8.noarch +vdsm-gluster-4.40.60.7-1.el8.x86_64 +vdsm-hook-ethtool-options-4.40.60.7-1.el8.noarch +vdsm-hook-fcoe-4.40.60.7-1.el8.noarch +vdsm-hook-openstacknet-4.40.60.7-1.el8.noarch +vdsm-hook-vhostmd-4.40.60.7-1.el8.noarch +vdsm-hook-vmfex-dev-4.40.60.7-1.el8.noarch +vdsm-http-4.40.60.7-1.el8.noarch +vdsm-jsonrpc-4.40.60.7-1.el8.noarch +vdsm-network-4.40.60.7-1.el8.x86_64 +vdsm-python-4.40.60.7-1.el8.noarch +vdsm-yajsonrpc-4.40.60.7-1.el8.noarch -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

1 0

Ovirt Engine -- Connection Refused to all hosts
by Nick Polites 18 May '21

18 May '21

Hi All, I am not sure if my original post is being reviewed before posting but trying again in case it failed to send. I tried logging in this morning to oVrit and see that all of my hosts are unresponsive. I am seeing a connection refused error in the engine logs. I am able to SSH and ping the host from the engine. Any help would be appreciated. 2021-05-15 15:19:21,041Z ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] Command 'GetCapabilitiesAsyn cVDSCommand(HostName = hlkvm03, VdsIdAndVdsVDSCommandParametersBase:{hostId='2186eca7-4d9d-482f-b1b7-b63ac46b96aa', vds='Host[hlkvm03,2186eca7-4d9d-482f-b1b7 -b63ac46b96aa]'})' execution failed: java.net.ConnectException: Connection refused ~

3 2

Migrating VMs with templates from 4.2.8 to new 4.4.6 instance
by Pavel Strzinek 17 May '21

17 May '21

Hello, I am having hard time migrating vms based on templates from existing 4.2.8 instance to newly installed 4.4.6 HCI with different storage. When I export VMs and corresponding templates to export NFS storage, detach it from source DC and attach the export storage to a new oVirt environment, the import of templates is failing with error "Failed to import Template XXX to Data Center YYY, Cluster ZZZ". I cannot find anything more specific about the error in logs. Am I missing something? Exporting to OVA and importing back does work, but I want to make use of template thin provisioning.

2 3