- Users - oVirt List Archives

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 05 May '21

05 May '21

Status Update: During migration between Host1 and Host3, links are being created and I see no Errors as before in the agent/broker logs. Having all links in place, and having waited 24hours, for potential Engine updates, I tried to deploy HE on Host2: Part1: * From oVirt UI – HostedEngine Deploy * Imeddiately on Host2 started seeing messages like “ Is the HostedEngine deployed ?” * hosted-engine.conf got populated only with host_id and ca_path * it failed Part2: * I noticed that during deployment no links in /var/rund/vdsm/storage/<hosted_storage_id> were created * Copied hosted-engine.conf from Host1 to Host2 * Replace the host_id with correct value * Reran the deployment * Noticed that /var/rund/vdsm/storage/<hosted_storage_id> two links got created, one of them being the link to metadata_image * Host1 and Host3 hosted-engine –vm-status was showing Host2 but with status unknown/stale data * Deployment failed Part3: * Since link to conf_image was not created in first phase, I added it manually * Populated the hosted-engine.conf * Reran the deployment * Same result * Hosted-engine.conf on Host2 would end up with only host_id and ca_path values and deployment would fail At this point, I cleaned up all hosted-engie remains from Host2 using ovirt-hosted-engine-cleanup and removed metadata of Host2 from host1 and Host3 I am out of ideas. It seems that Host1 and Host3 are happily operating, but I am unable to add any other hosts to HE pool. Please assist if you have any ideas. I am open to executing restore – but I would much rather like to at least discover where or what is the problem, before moving to planning restore. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Wednesday, 5 May 2021 at 12:16 To: Strahil Nikolov <hunter86_bg(a)yahoo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Both hosts have same agent.conf: ❯ cat <(ssh root(a)ovirt-sj-01.ictv.com 'cat /etc/ovirt-hosted-engine-ha/agent.conf') [score] # NOTE: These values must be the same for all hosts in the HA cluster! base-score=3400 network-score-penalty=1600 not-uptodate-config-penalty=1000 mgmt-bridge-score-penalty=600 free-memory-score-penalty=400 cpu-load-score-penalty=1000 engine-retry-score-penalty=50 cpu-load-penalty-min=0.4 cpu-load-penalty-max=0.9 ❯ cat <(ssh root(a)ovirt-sj-03.ictv.com 'cat /etc/ovirt-hosted-engine-ha/agent.conf') [score] # NOTE: These values must be the same for all hosts in the HA cluster! base-score=3400 network-score-penalty=1600 not-uptodate-config-penalty=1000 mgmt-bridge-score-penalty=600 free-memory-score-penalty=400 cpu-load-score-penalty=1000 engine-retry-score-penalty=50 cpu-load-penalty-min=0.4 cpu-load-penalty-max=0.9 And its same on the good platform: ❯ ssh root(a)ovirt-hv-01.avinity.tv 'cat /etc/ovirt-hosted-engine-ha/agent.conf' [score] # NOTE: These values must be the same for all hosts in the HA cluster! base-score=3400 network-score-penalty=1600 not-uptodate-config-penalty=1000 mgmt-bridge-score-penalty=600 free-memory-score-penalty=400 cpu-load-score-penalty=1000 engine-retry-score-penalty=50 cpu-load-penalty-min=0.4 cpu-load-penalty-max=0.9 ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Strahil Nikolov <hunter86_bg(a)yahoo.com> Date: Wednesday, 5 May 2021 at 00:08 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, Marko Vrgotic <M.Vrgotic(a)activevideo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** Yes, /etc/ovirt-hosted-engine-ha/agent.conf should be enough. Best Regards, Strahil Nikolov

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 05 May '21

05 May '21

Yes,/etc/ovirt-hosted-engine-ha/agent.conf should be enough. Best Regards,Strahil Nikolov

2 1

oVirt 4.4.7 development
by Michal Skrivanek 05 May '21

05 May '21

Hi all, we are starting to work on oVirt 4.4.7 and as you probably noticed we are slowly transitioning away from CentOS. We added CentOS Stream option for regular hosts, and the current 4.4.6 Node and Appliance are already built from Stream. We also have some initial patches for alternative RHEL clones other than CentOS, however we can’t really support them until they provide the required dependencies. We are not ditching CentOS just yet, but we can no longer accommodate the long delay it takes CentOS team to produce a next minor version, hence for oVirt 4.4.7 we will require the operating system to be at the RHEL 8.4 level - that’s a CentOS 8.4 if that gets released in time or the actual RHEL 8.4 or any 8.4 clone like Alma, Rocky, OL with the right dependencies (probably the ones from Stream) or CentOS Stream. In the meantime we're going to be moving the existing CI resources to test with CentOS Stream to unblock development of new features. If you would want to help with testing other OSes patches are welcome to add support for them Thanks, michal

2 2

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 04 May '21

04 May '21

Ok – These are the symlinks in place: 1. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_conf_image> 2. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_metadata_image> 3. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_lockspace_image> 4. /var/run/vdsm/storage/<hosted_storage_id>/<path_to_vmdisk_id_image> - this one exist only on the current Host 5. /var/run/vdsm/storage/<hosted_storage_id>/<one more_link_I_have_no_clue what_it_is_for> Migration issue – seem to be solved at part 3: Part1: * Try to migrate HE from Host1 to Host3 – migration fails * Analysis showed that Host3 was missing the symlink num4 – which should be created during migration * I created symlink num4 and that migration works but HE on donor host has status “down_unexpected” instead of “down” Part2: * Next thing I noticed was that symlink num1 was missing on both Hosts * Added symlink num1 to both Hosts and reran the migration * Migration done successfully from Host1 to Host3 and HE on donor host had status “down” Part3: * Migrating back from Host3 to Host1 * Noticed that at beginning of migration the symlink num4 gets created on destination Host (Host1 in this case) * This time the symlink num4 on destination was not deleted and migration went through Part4: * Repeated migration from Host1 to Host3 * All went as expected. New HE Host deploy issue: * Current conclusion is that the new HE Host deploy was failing as the symlink num1 was not existing on running Hosts ( 1 and 3) so it was reported as missing/non-existent * Since the link is now added and it persists – I will try to deploy new HA Host I will keep you posted. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Tuesday, 4 May 2021 at 13:47 To: Strahil Nikolov <hunter86_bg(a)yahoo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil, Creating link towards conf_image on shared storage, did help in regards to HE migrating with donor not ending up with down_expected but simply down. Regarding “Can you paste the contents of the ha daemons' config where the shared storage should be defined?”, are you referring to /etc/ovirt-hosted-engine-ha/agent.conf, var/lib/ovirt-hosted-engine-ha/broker.conf and ha.conf ? If not, please give me specific files, so I can provide configs. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Strahil Nikolov <hunter86_bg(a)yahoo.com> Date: Tuesday, 4 May 2021 at 09:21 To: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, Yedidyah Bar David <didi(a)redhat.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** It's quite interesting why the link is not created when it should ... Can you paste the contents of the ha daemons' config where the shared storage should be defined? You can create the link if needed, but it won't solve the problem. Best Regards, Strahil Nikolov

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 04 May '21

04 May '21

It's quite interesting why the link is not created when it should ...Can you paste the contents of the ha daemons' config where the shared storage should be defined? You can create the link if needed, but it won't solve the problem. Best Regards,Strahil Nikolov

2 1

OLVM: error deleting snapshot live
by Renevey Christian 04 May '21

04 May '21

Hi, With last OLVM update installed, we are unable to delete snapshot without stopping the vm (live merge error) Vdsm.log: 2021-05-04 09:44:28,389+0200 ERROR (jsonrpc/6) [virt.vm] (vmId='dba27e34-f8f3-4d9b-a6fd-95ba92c2970e') Live merge failed (job: 36f515b5-fb9e-4e29-b5f8-5b7c062cef5d) (vm:5957) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5955, in merge bandwidth, flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 719, in blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirtError: internal error: child reported (status=125): Requested operation is not valid: Setting different SELinux label on /rhev/data-center/mnt/blockSD/47c2336e-1936-4b28-b759-efcfe8a19b58/images/269f76c4-2a63-4cdf-a2da-0f56f4acc464/c317c53b-ad79-421f-9263-f1f19c81a9a7 which is already in use 2021-05-04 09:44:28,412+0200 INFO (jsonrpc/6) [api.virt] FINISH merge return={'status': {'message': 'Merge failed', 'code': 52}} from=::ffff:192.168.1.110,38834, flow_id=390f0b25-f364-4adc-b7b3-b732ee0e0213, vmId=dba27e34-f8f3-4d9b-a6fd-95ba92c2970e (api:54) 2021-05-04 09:44:28,413+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 52) in 0.58 seconds (__init__:312) 2021-05-04 09:44:29,124+0200 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.1.110,38834 (api:48) Any idea whats wrong ? Rgds

2 1

How to use Ovirt node 4.4.5.0 with local storage
by danilosevilla＠gmail.com 04 May '21

04 May '21

Hi, I'm totally new with Ovirt. I have a production server in which I planned to install Ovirt. I need to use their local storage for Virtualization, but it seems that only have chance to storage in NAS or other network or fiber storage. Can I use the local storage for virtualization?. If yes, how can I do that. Thanks.

2 1

error id 980
by ozmen62＠hotmail.com 04 May '21

04 May '21

Hi, First of all, thanks for this great list and the people who try to help eachother. This time my problem is about engine OVF file or engine storage setting. How can i came with this idea, because every 2-3 hours engine try to migrate other host(From host1 to host2) "Invalid status on Data Center B300. Setting status to Non Responsive." when i check hosts log , see these ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM stopped on localhost ovirt-ha-agent ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore ERROR Unable to extract HEVM OVF i believe there is some kind of fc storage path or config problem. My storage setting is good. there is no error about them Is there anyone who experienced about that to solve it thanks

2 5

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

Investigation so far has showed, that symlink for conf_volume/conf_image is missing on Hosts 1 and 3. According to the ovirt-engine deep dive, connection/access to conf_volume is required for adding new HE Hosts as it holds its configuration template. This is most likely the reason why I could not add new Host and why the Host1 and Host3 were occasionally complaining the the conf_image_uuid is missing. I also checked the DB tb images – the conf_image and conf_volume entries exist. Question: Considering DB entries seem correct, am I “safe” to manually create the symlink on Host1 and Host3 , from global maintenance mode, set correct permissions and than exit maintenance mode? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 12:44 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Dear Strahil and Yedidyah, Is it possible that HostedEngine DEPLOY on Host2 is failing due to “inability” to access the conf_volume_UUID “c518f937-60fe-4fed-a54c-db11328bb507” I do not see that error message on Host2, but I see it on Host1 and Host3: 2021-05-03 02:43:32,438-0700 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='b4bba796-b62b-41cb-8d91-6637c5fb9705') Unexpected error (task:875) 2021-05-03 02:43:32,439-0700 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH prepareImage error=Volume does not exist: (u'c518f937-60fe-4fed-a54c-db11328bb507',) (dispatcher:83) Could this be result of hosted-engine.conf staying empty and deployment not working? If I check the /var/log/ovirt-hosted-engine-setup/ folder is empty. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:38 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

Dear Strahil and Yedidyah, Is it possible that HostedEngine DEPLOY on Host2 is failing due to “inability” to access the conf_volume_UUID “c518f937-60fe-4fed-a54c-db11328bb507” I do not see that error message on Host2, but I see it on Host1 and Host3: 2021-05-03 02:43:32,438-0700 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='b4bba796-b62b-41cb-8d91-6637c5fb9705') Unexpected error (task:875) 2021-05-03 02:43:32,439-0700 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH prepareImage error=Volume does not exist: (u'c518f937-60fe-4fed-a54c-db11328bb507',) (dispatcher:83) Could this be result of hosted-engine.conf staying empty and deployment not working? If I check the /var/log/ovirt-hosted-engine-setup/ folder is empty. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:38 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Cluster level 4.5
by Don Dupuis 03 May '21

03 May '21

What version of libvirt is required for a host to be put in this cluster level? I am using CentOS 8.3 and cpu is Cascade Lake Server. It says that my host is only compatible with cluster version 4.2,4.3 and 4.4. I am doing a new install of Ovirt 4.4.5. I have tried to update libvirt version but have run into issues. Currently installed libvirt is libvirt-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64. Don

2 1

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 03 May '21

03 May '21

As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing.Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. Best Regards,Strahil Nikolov On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic<M.Vrgotic(a)activevideo.com> wrote:  Dear oVirt, I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: On Engine: 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) On source Host: 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run time.time(), migrationParams, machineParams File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration self._migration_flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: operation aborted: migration out job: canceled by client I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. This is production system, so I cannot just move on to upgrading/deploying to 4.4. Additionally – : - Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? - What is the effect or reinitialize-lockspace? Kindly awaiting your reply. Happy to provide any additional information needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com w: www.activevideo.com ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FRTRNTSGPLQOS…

3 2

new engine restore from backup
by ozmen62＠hotmail.com 03 May '21

03 May '21

Hi, I've built e new engine and tried to restore from backup after run engine-setup it returns; [ ERROR ] Failed to execute stage 'Misc configuration': function getdwhhistorytimekeepingbyvarname(unknown) does not exist LINE 2: select * from GetDwhHistoryTimekeepingByVarName( redhat page says seploy new engine, but already done it. old and new engine are same version Do you have any idea how can i restore engine. There is no another backup

2 4

Trouble migrating the websocket-proxy / Trying to make VM-Portal publicly available
by tivon.haeberlein＠secges.de 02 May '21

02 May '21

Hi community, I hope you can help me on this one. My oVirt cluster is deployed on an internal network (10.1.30.0/24). At the moment I am trying to make the VM-Portal publicly available so people can administer their vms with the portal and noVNC. I have read the oVirt Documentation (as well as trying a couple of things) and found that the best way to accomplish this is by installing a proxy server which proxies HTTP and HTTPS to the engine. To get the noVNC working I have to install the ovirt-engine-websocket-proxy on the proxy host as well and set it up. For the proxy host I'm using centos 8. For the HTTP and HTTPS proxy I'm using haproxy (which is working perfectly already). For the noVNC Proxy I'm trying to follow the documentation (https://www.ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engin…) but am facing a couple of issues. Issues start with removing the websocket proxy from the engine machine. I put the cluster into global-maintenance-mode and run "engine-cleanup" as told in the docs: [root@engine ~]# engine-cleanup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-remove-20210502182713-ne56jy.log Version: otopi-1.9.4 (otopi-1.9.4-1.el8) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ INFO ] Hosted Engine HA is in Global Maintenance mode. All the installed ovirt components are about to be removed, data will be lost (OK, Cancel) [Cancel]: [ ERROR ] Failed to execute stage 'Setup validation': Aborted by user [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20210502182713-ne56jy.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210502182730-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of cleanup failed As you can see the output differs a bit from the docs. Instead of being asked "Do you want to remove all components? (Yes, No) [Yes]: No" I'm being notified that all data will be lost when continuing. Of course I do not want to loose all data so I abort the cleanup. I then tried to just install the websocket-proxy on the proxy server and ran into the next issue. On the proxy server I am also following the docs and running "yum install ovirt-engine-websocket-proxy" followed by "engine-setup": [root@proxy ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20210502183039-ydl26v.log Version: otopi-1.9.4 (otopi-1.9.4-1.el8) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- Configure WebSocket Proxy on this host (Yes, No) [Yes]: --== PACKAGES ==-- [ INFO ] Checking for product updates... [ INFO ] DNF No matches found for the following disable plugin patterns: versionlock [ INFO ] No product updates found ### I've cut a bit of the output to keep it shorter ### [ INFO ] Stage: Transaction setup [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ ERROR ] Failed to execute stage 'Misc configuration': 'OVESETUP_SYSTEM/selinuxBooleans' [ INFO ] DNF Performing DNF transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210502183039-ydl26v.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210502183159-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed As you can see it failed at stage "'Misc configuration': 'OVESETUP_SYSTEM/selinuxBooleans'". Here I did try to disable SElinux as well as enforcing it (as set at default) without any success. Am I on the right approach to make the VM-Portal publicly avalible by doing this or is there a better / more supported way? Can you help me out on getting this running or pointing me in the right direction (if this is the right approach)? Thank you guys! Best regards, Tivon I'm new to the community but will help to keep oVirt growing and support where I can.

1 0

How to assign a server disk and nic profile via REST API?
by ovirt.org＠nevim.eu 02 May '21

02 May '21

Hello to everybody, for about an hour I have stuck my nose in the ovirt and rhev documentation, but I still can't understand how to use the REST API to change the disk profile and nothing the server profile. Ideally, how to set it for them when setting up a VM. Thank you so much for the advice.

2 7

Slow VM replication
by David Johnson 01 May '21

01 May '21

Hi everyone, When cloning a VM, I discovered that the time to close appears excessive based on the underlying platform I am using. It appears that the clone operation is not making efficient use of the network. I have seen up to 4 GBits sustained throughput by applications on VM's on the cluster. Is there a configuration I might be missing? *System specifics:* Backing store: NFS on TrueNAS running zRaid 3 on 11 spinning disks Ovirt Controller: I7 desktop General network: 1 GBit Ethernet Ovirt Host: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 2x32 cores, 256 GB RAM Data Network: 10 GBit 10GBase-Twinax, dedicated to this Host and the TrueNAS *Operation:* Copying the 60GB partition, the copy operation never exceeds 40 megabytes per second (less than 0.5 GBit), even though the dedicated 10 Gigabit data network is not otherwise busy. [image: image.png] *David Johnson* *Director of Development, Maxis Technology* 844.696.2947 ext 702 (o) | 479.531.3590 (c) <https://www.linkedin.com/in/pojoguy/> <https://maxistechnology.com/wp-content/uploads/vcards/vcard-David_Johnson.v…> <https://maxistechnology.com/> *Follow us:* <https://www.linkedin.com/company/maxis-tech-inc/>

3 5

Re%3A how to enable HA for all vms and entire cluster at global settings&In-Reply-To=<20210429060341.18703.30460%40mail.ovirt.org>
by Tivon Häberlein 30 Apr '21

30 Apr '21

Hi, I haven't found a global option to set this. You'll have to do it vm after vm, which isn't a bad thing in my opinion as you'll have to reboot every vm after enabling HA anyways. For new VMs depending on your workflow I'd create a template or an instance type with HA enabled. You can also enable HA in the preset instance types via "Administration" > "Configure" > "Instance Types". Keep in mind that you will have to enable advanced options to see the High Avalibitity tab. Best regards, Tivon

1 0

Unable to migrate Engine to another HE Host
by Marko Vrgotic 30 Apr '21

30 Apr '21

Dear oVirt, I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: On Engine: 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) On source Host: 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run time.time(), migrationParams, machineParams File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration self._migration_flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: operation aborted: migration out job: canceled by client I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. This is production system, so I cannot just move on to upgrading/deploying to 4.4. Additionally – : * Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? * What is the effect or reinitialize-lockspace? Kindly awaiting your reply. Happy to provide any additional information needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message.

1 0

how to override host in console.vv
by Enrico Becchetti 30 Apr '21

30 Apr '21

Hi all, I use ovirt 4.3.10 with an engine running in a standalone virtual machine. The operating system of this engine and the hypervisors is CentOS 7. In the initial setup I chose a name for the engine, ovirt.local, then I added a second public IP address named ovirt.mydomain in order to allow the user portal also from the Internet. Now here's my problem. Users are able to connect to the VM portal but they cannot access the console of their virtual machines. Checking the console.vv file I noticed that there are two parts, the first [virt-viewer] which contains the hypervisor on which the virtual machine runs, and the second in this configuration [ovirt] inside ovirt I've got "host = ovirt.local: 443". Now I would like to change the value of the "host" parameter with the FQDN. Can you tell me how I can set an override of this variable ? Thanks a lot. Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Skype:enrico_becchetti Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________

1 1

Re: Unable to provide network acces to second cluster in DC
by Miguel Garcia 29 Apr '21

29 Apr '21

Browsing at the path above I see prodvlan and havlan listed and manage network settings is the same as Public1. Compute > Clusters > Public2 > Logical Network > Manage Networks The problem is that when I try to create a vm for cluster Public2 on the nic sections no nic profile are listed, it remains <empty> option only.

2 1

[ANN] oVirt 4.4.6 Sixth Release Candidate is now available for testing
by Lev Veyde 29 Apr '21

29 Apr '21

oVirt 4.4.6 Sixth Release Candidate is now available for testing The oVirt Project is pleased to announce the availability of oVirt 4.4.6 Sixth Release Candidate for testing, as of April 29th, 2021. This update is the sixth in a series of stabilization updates to the 4.4 series. How to prevent hosts entering emergency mode after upgrade from oVirt 4.4.1 Note: Upgrading from 4.4.2 GA or later should not require re-doing these steps, if already performed while upgrading from 4.4.1 to 4.4.2 GA. These are only required to be done once. Due to Bug 1837864 <https://bugzilla.redhat.com/show_bug.cgi?id=1837864> - Host enter emergency mode after upgrading to latest build If you have your root file system on a multipath device on your hosts you should be aware that after upgrading from 4.4.1 to 4.4.6 you may get your host entering emergency mode. In order to prevent this be sure to upgrade oVirt Engine first, then on your hosts: 1. Remove the current lvm filter while still on 4.4.1, or in emergency mode (if rebooted). 2. Reboot. 3. Upgrade to 4.4.6 (redeploy in case of already being on 4.4.6). 4. Run vdsm-tool config-lvm-filter to confirm there is a new filter in place. 5. Only if not using oVirt Node: - run "dracut --force --add multipath” to rebuild initramfs with the correct filter configuration 6. Reboot. Documentation - If you want to try oVirt as quickly as possible, follow the instructions on the Download <https://ovirt.org/download/> page. - For complete installation, administration, and usage instructions, see the oVirt Documentation <https://ovirt.org/documentation/>. - For upgrading from a previous version, see the oVirt Upgrade Guide <https://ovirt.org/documentation/upgrade_guide/>. - For a general overview of oVirt, see About oVirt <https://ovirt.org/community/about.html>. Important notes before you try it Please note this is a pre-release build. The oVirt Project makes no guarantees as to its suitability or usefulness. This pre-release must not be used in production. Installation instructions For installation instructions and additional information please refer to: https://ovirt.org/documentation/ This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 8.3 or newer * CentOS Linux (or similar) 8.3 or newer * CentOS Stream 8 This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 8.3 or newer * CentOS Linux (or similar) 8.3 or newer * CentOS Stream 8 * oVirt Node 4.4 based on CentOS Stream 8 (available for x86_64 only) See the release notes [1] for installation instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available based on CentOS Stream 8 - oVirt Node NG is already available based on CentOS Stream 8 Additional Resources: * Read more about the oVirt 4.4.6 release highlights: http://www.ovirt.org/release/4.4.6/ * Get more oVirt project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.4.6/ [2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/ -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

1 0

how to enable HA for all vms and entire cluster at global settings
by dhanaraj.ramesh＠yahoo.com 29 Apr '21

29 Apr '21

Hi Team instead of enabling HA for each vm one by one, I want to enable HA for all vms that is already running in the cluster and also for the future vms that going to be installed. I already did configure HA reservation at the cluster level, any other settings or commands I should configure? how I can achieve the above

1 0

Unable to provide network acces to second cluster in DC
by Miguel Garcia 28 Apr '21

28 Apr '21

I have created the following DC and cluster structure in my ovirt server - Data Center Public - - cluster-public1 - - cluster-public2 Also created a couple Logical Network for DC-Public - - prodvlan - - havlan And added 4 host (two for each cluster because processor type) and each host has ports added to both logical networks (besides ovirtmgmt) When I create a new vm in cluster public1 and add nics I'm able to list prodvlan and havlan but when I try to create the vm for cluster public2 no logical networks are listed. Any idea how to get logical networks listed for cluster public2? Thanks in advance

2 1

Restoring hosted engine from backup fails on new FC storage domain creation
by Juhani Rautiainen 28 Apr '21

28 Apr '21

Hi! I started the upgrade from 4.3->4.4. Now I'm stuck as restoring the backup fails to create the correct storage domain for Hosted Engine. How can I create one? Error from ansible task is: [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Domain format is different from master storage domain format]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[Domain format is different from master storage domain format]\". HTTP response code is 400."} And from UI I can see that what has been created is a data domain. Not data (master) domain as old one was. Now I'm stuck here. This is a fibre channel system where I'm trying to do this if it is relevant. What I could find from the logs is this: 2021-04-27 09:36:06,925+0300 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:105 storage_domain_details: {'changed': False, 'ovirt_storage_domains': [{'href': '/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4', 'comment': '', 'description': '', 'id': 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': 'hosted_storage', 'available': 531502202880, 'backup': False, 'block_size': 512, 'committed': 0, 'critical_space_action_blocker': 5, 'discard_after_delete': True, 'disk_profiles': [], 'disk_snapshots': [], 'disks': [], 'external_status': 'ok', 'master': False, 'permissions': [], 'status': 'unattached', 'storage': {'type': 'fcp', 'volume_group': {'id': 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit', 'logical_units': [{'discard_max_size': 268435456, 'discard_zeroes_data': False, 'id': '36000d31005b4f6000000000000000029', 'lun_mapping': 3, 'paths': 0, 'product_id': 'Compellent Vol', 'serial': 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000, 'storage_domain_id': 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'vendor_id': 'COMPELNT', 'volume_group_id': 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections': [], 'storage_format': 'v5', 'supports_discard': True, 'supports_discard_zeroes_data': False, 'templates': [], 'type': 'data', 'used': 4294967296, 'vms': [], 'warning_low_space_indicator': 10, 'wipe_after_delete': False}], 'failed': False} 'master': False? I'm not sure if this creation or check. I tried this opration twice. I removed the new hosted_storage and remove also old hosted_engine domain on second try to make sure that it doesn't prevent creation of another master. No luck with that. Thanks, Juhani

3 14

Import VM with Xen Based in Ovirt
by hemaa＠emiratesnbd.com 28 Apr '21

28 Apr '21

Hi All, I am trying to migrate a VM from a Xen Based virtualization to Ovirt with the import option in GUI But I am not able to list any stopped VMs on the Xen host. So what I did is I started the VM and got it listed. Then selected the VM and moved to right hand section. Clicked on next button. But then it gave an error "All chosen VMs couldn't be retrieved by the external system and therefore have been filtered. Please see log for details." When I check the libvirt logs in xen hypervisor I see error as below. There is no other errors in engine logs or vdsm logs 2021-04-22 08:52:36.796+0000: 30219: error : do_connect:116 : internal error: failed to connect to xend 2021-04-22 08:52:36.830+0000: 30219: error : virNodeSuspendSupportsTarget:332 : internal error: Cannot probe for supported suspend types 2021-04-22 08:52:36.830+0000: 30219: warning : xenHypervisorMakeCapabilities:2550 : Failed to get host power management capabilities 2021-04-22 08:52:36.832+0000: 30220: error : virConnectListAllDomains:6803 : this function is not supported by the connection driver: virConnectListAllDomains This is the same error I got when I tried importing a running VM as well. Any idea on what could be a possible reason behind this?

5 5

Faild ovirt-websocket-proxy.
by jesado74＠gmail.com 28 Apr '21

28 Apr '21

Hello again. After getting the hosted-engine vm back, now I have the following problem. I can't use the graphic console. The websocket-proy fails me. Surely when I update the certificates I have missed some intermediate step, but I do not know how to follow. Anyone know of a link to follow the certificate change procedure in the websocket-proxy service. Thank you. PS: Here I leave the output of the service: ● ovirt-websocket-proxy.service - oVirt Engine websockets proxy Loaded: loaded (/usr/lib/systemd/system/ovirt-websocket-proxy.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Wed 2021-04-28 02:03:54 UTC; 7h ago Process: 1694 ExecStart = / usr / share / ovirt-engine / services / ovirt-websocket-proxy / ovirt-websocket-proxy.py --systemd = notify $ EXTRA_ARGS start (code = exited, status = 1 / FAILURE) Main PID: 1694 (code = exited, status = 1 / FAILURE) Apr 28 02:03:53 ovirt.myhostedengine.com systemd [1]: Starting oVirt Engine websockets proxy ... Apr 28 02:03:54 ovirt.myhostedengine.com ovirt-websocket [1694]: 2021-04-28 02: 03: 54,614 + 0000 ovirt-websocket-proxy: ERROR run: 554 Error: SSL_ONLY is set but SSL_CERTIFICATE or SSL_KEY file not found. Apr 28 02:03:54ovirt.hostedengine.com ovirt-websocket-proxy.py [1694]: ovirt-websocket-proxy [1694] ERROR run: 554 Error: SSL_ONLY is set but SSL_CERTIFICATE or SSL_KEY file not found. Apr 28 02:03:54 ovirt.myhostedengine.com systemd [1]: ovirt-websocket-proxy.service: main process exited, code = exited, status = 1 / FAILURE Apr 28 02:03:54 ovirt.myhostedengine.com systemd [1]: Failed to start oVirt Engine websockets proxy. Apr 28 02:03:54 ovirt.myhostedengine.com systemd [1]: Unit ovirt-websocket-proxy.service entered failed state. Apr 28 02:03:54 ovirt.myhostedengine.com systemd [1]: ovirt-websocket-proxy.service failed. Sorry if I say any inconsistencies, but I'm new to this whole ovirt topic. Thanks again.

2 1

How i can set vnic while Create VM
by ovirt.org＠nevim.eu 28 Apr '21

28 Apr '21

How i can set vnic while Create VM via rest-api? I mean this (on screenshot), how to via rest-api? https://prnt.sc/126trb8 Thanks very much.

1 0

Ovirt-engine Setup Error-Failed to execute stage 'Misc configuration
by rajkumar madhu 28 Apr '21

28 Apr '21

Hi Team, Unable to install Ovirt in my centos server.while executing Engine-setup command ,I am getting below error.Please help me on this . Please confirm installation settings (OK, Cancel) [OK]: [ INFO ] Stage: Transaction setup [ INFO ] Stopping engine service [ INFO ] Stopping ovirt-fence-kdump-listener service [ INFO ] Stopping dwh service [ INFO ] Stopping vmconsole-proxy service [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ ERROR ] Failed to execute stage 'Misc configuration': [Errno 2] No such file or directory: '/etc/ovirt-engine/extensions.d' [ INFO ] DNF Performing DNF transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210427090149-r0i80v.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210427090311-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed Regards, Rajkumar M. 9176772077

2 4

Ovirt 4.4.6.5-1 engine-setup failed
by Don Dupuis 28 Apr '21

28 Apr '21

I am installing Ovirt 4.4.6.5-1 engine on CentOS Stream 8 on a dedicated vm. Towards the end of the engine-setup session I have a failure: [ INFO ] Stage: Transaction setup [ INFO ] Stopping engine service [ INFO ] Stopping ovirt-fence-kdump-listener service [ INFO ] Stopping dwh service [ INFO ] Stopping vmconsole-proxy service [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Upgrading CA [ INFO ] Initializing PostgreSQL [ INFO ] Creating PostgreSQL 'engine' database [ INFO ] Configuring PostgreSQL [ INFO ] Creating PostgreSQL 'ovirt_engine_history' database [ INFO ] Configuring PostgreSQL [ INFO ] Creating CA: /etc/pki/ovirt-engine/ca.pem [ INFO ] Creating CA: /etc/pki/ovirt-engine/qemu-ca.pem [ INFO ] Updating OVN SSL configuration [ INFO ] Updating OVN timeout configuration [ INFO ] Creating/refreshing DWH database schema [ INFO ] Setting up ovirt-vmconsole proxy helper PKI artifacts [ INFO ] Setting up ovirt-vmconsole SSH PKI artifacts [ INFO ] Configuring WebSocket Proxy [ INFO ] Creating/refreshing Engine database schema [ INFO ] Creating a user for Grafana [ INFO ] Creating/refreshing Engine 'internal' domain database schema [ INFO ] Creating default mac pool range [ INFO ] Adding default OVN provider to database [ INFO ] Adding OVN provider secret to database [ INFO ] Setting a password for internal user admin [ INFO ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil [ INFO ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up [ INFO ] Starting engine service [ INFO ] Starting dwh service [ INFO ] Starting Grafana service [ ERROR ] Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio' [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210426111225-g31wa5.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210426111501-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed Below is output of /var/log/ovirt-imageio/daemon.log: 2021-04-26 11:15:01,481 ERROR (MainThread) [server] Server failed Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 46, in main server = Server(cfg) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 110, in __init__ self.remote_service = services.RemoteService(self.config, self.auth) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/services.py", line 73, in __init__ self._server = http.Server((config.remote.host, port), http.Connection) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 91, in __init__ self.create_socket(prefer_ipv4) File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 123, in create_socket self.socket = socket.socket(self.address_family, self.socket_type) File "/usr/lib64/python3.6/socket.py", line 144, in __init__ _socket.socket.__init__(self, family, type, proto, fileno) OSError: [Errno 97] Address family not supported by protocol I am not seeing errors in /var/log/ovirt-engine/setup/setup.log. What other information do you need that is revelant to this issue? Thanks Don

3 8

Brick didn't come up after planned Poweroutage
by Marcel d'Heureuse 27 Apr '21

27 Apr '21

Hi Guys, We hat a planned power outage and we have stopped all VMS and try to keep the servers alive as long as possible. Some Servers made a Reboot by itself but the IPMI interface was not reachable for the engine. The some hosts have made a restart. Now all systems running and all 180 VMs too. But one Brick of the hosted engine didn't come up. Did you have an idea. Please find attached some screenshots. Our Hosted Engine has 6 bricks on 6 different Servers. A restart of the Host didn't work. The Server Restart of glusterd didn't work. I hope you have some more Ideas? Thanks Br Marcel

2 1

Re: [ceph-users] osd nearfull is not detected
by Konstantin Shalygin 27 Apr '21

27 Apr '21

Create tracker for this issue [1] [1] https://tracker.ceph.com/issues/50533 <https://tracker.ceph.com/issues/50533> k > On 21 Apr 2021, at 21:21, Dan van der Ster <dan(a)vanderster.com <mailto:dan@vanderster.com>> wrote: > > Are you currently doing IO on the relevant pool? Maybe nearfull isn't > reported until some pgstats are reported. > > Otherwise sorry I haven't seen this.

1 0

Re: Attempting to detach a storage domain
by Vojtech Juranek 27 Apr '21

27 Apr '21

On Tuesday, 27 April 2021 02:18:18 CEST Matthew.Stier(a)fujitsu.com wrote: > There is no /var/log/vdsm directory on the hosted-engine. > > This is from the current host, hosting the engine: aaa.bbb.ccc.100 is the > hosted-engine > 2021-04-26 19:12:42,760-0500 INFO (jsonrpc/2) [api.host] START > getCapabilities() from=::ffff:aaa.bbb.ccc.100,37812 (api:48) 2021-04-26 > 19:12:42,763-0500 INFO (jsonrpc/2) [api.host] FINISH getCapabilities > error=internal error: Duplicate key from=::ffff:168.127.151.100,37812 > (api:52) 2021-04-26 19:12:42,763-0500 ERROR (jsonrpc/2) > [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback > (most recent call last): > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, > in _handle_request res = method(**params) > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in > _dynamicMethod result = fn(*methodArgs) > File "<string>", line 2, in getCapabilities > File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in > method ret = func(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1371, in > getCapabilities c = caps.get() > File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 95, in > get machinetype.compatible_cpu_models()) > File "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 43, in > __call__ value = self.func(*args) > File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 142, in > compatible_cpu_models all_models = domain_cpu_models(c, arch, cpu_mode) > File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 97, in > domain_cpu_models domcaps = conn.getDomainCapabilities(None, arch, None, > virt_type, 0) File > "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line > 131, in wrapper ret = f(*args, **kwargs) > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, > in wrapper return func(inst, *args, **kwargs) > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3844, in > getDomainCapabilities if ret is None: raise libvirtError > ('virConnectGetDomainCapabilities() failed', conn=self) libvirtError: > internal error: Duplicate key looks like https://bugzilla.redhat.com/show_bug.cgi?id=1791790 The fix should be contained in recent oVirt relases. > 2021-04-26 19:12:42,764-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC > call Host.getCapabilities failed (error -32603) in 0.00 seconds > (__init__:312) > > -----Original Message----- > From: Nir Soffer <nsoffer(a)redhat.com> > Sent: Monday, April 26, 2021 5:12 PM > To: Stier, Matthew <Matthew.Stier(a)fujitsu.com> > Cc: Benny Zlotnik <bzlotnik(a)redhat.com>; users(a)ovirt.org > Subject: Re: [ovirt-users] Re: Attempting to detach a storage domain > > On Tue, Apr 27, 2021 at 12:48 AM Matthew.Stier(a)fujitsu.com > <Matthew.Stier(a)fujitsu.com> wrote: > > > > > > I'm getting tons of these in the > > hosted-engine:/var/log/ovirt-engine/engine.log > > > > > > > > 2021-04-26 16:40:40,260-05 ERROR > > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > > (EE-ManagedThreadFactory-engineScheduled-Thread-8) [] EVENT_ID: > > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM xxxx.yyyy.zzz command Get > > Host Capabilities failed: Internal JSON-RPC error: {'reason': > > 'internal error: Duplicate key'} > > > This looks like bad response from vdsm. Can you share the vdsm log from the > host xxxx.yyyy.zzz? > /var/log/vdsm/vdsm.log > > Nir > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z2GOGEZ2DZX34 > ADZ5L2BATIINYGVWYJD/

1 0

Cannot delete snapshot
by Giulio Casella 27 Apr '21

27 Apr '21

Hi, I'm having (another) issue working with snapshots. Since a few days my backup system (storware vprotect) cannot delete a snapshot. Message reported by SPM is: HSMGetAllTasksStatusesVDS failed: value=Volume does not exist: ('6180c2e8-141a-4126-bcb1-5af5caa98175',) abortedcode=201 Other snapshots from other VMs are working fine (created and deleted as needed). I verified it, I can reach on SPM filesystem other snapshots from the same VM, but that one doesn't exist. I suspect that during normal backup management that snapshot has been correctly removed from file system, but still live in ovirt database. I'm pretty sure vProtect use ovirt API to operate, it doesn't directly interact with ovirt database. That's why I'm talking to you and not to storware guys. Is there a (safe) way to get rid of that snapshot? TIA Regards, Giulio Casella

5 11

Re: Ansible ovirt.ovirt_vm nics
by Matthew.Stier＠fujitsu.com 27 Apr '21

27 Apr '21

I believe I have identified the issue, variable inheritance. The inventory I include to define the hosts, includes several classes. ‘virtualmachines’ which the script uses to get the list of VMs it is going to process. All VM defaults are defined within its ‘:vars’ section. I also have groups of VMs where I want to override variables. Unfortunately, what is being defined in ‘virtualmachines’ is not being override in the class level, because they are considered sibling classes. I have found the ‘ansible_group_priority’ variable, which can be assigned at each class. I assigned the entry in [virtualmachines:vars] with the value of 2 (default is 1) and then in the other classes, I assigned a value of 3. Now the variable in [class:vars] will override the variables in [virtualmachines:vars]. Of course the host level can still override all. From: Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> Sent: Thursday, April 22, 2021 2:01 PM To: Martin Perina <mperina(a)redhat.com> Cc: users(a)ovirt.org; Martin Necas <mnecas(a)redhat.com> Subject: [ovirt-users] Re: Ansible ovirt.ovirt_vm nics It’s basically the one listed in the URL. The changes I’ve made is to add some addition ‘defaults’, and the code I mentioned below, to select different nic profiles for vms that are on different networks. I’m running an Oracle Linux 7u9 system, and the version of Ansible is 2.9.18 In my environment, I have created different networks, and labeled them based upon their vlan tag number (vlan20, vlan21, vlan22, …) A VM can be created on any of these vlans, so I need to be able to select the nic profile within ‘ini’ file holding all the configuration information. (hostname, fqdn, IP address, etc…) From: Martin Perina <mperina(a)redhat.com<mailto:mperina@redhat.com>> Sent: Thursday, April 22, 2021 2:51 AM To: Stier, Matthew <Matthew.Stier(a)fujitsu.com<mailto:Matthew.Stier@fujitsu.com>> Cc: users(a)ovirt.org<mailto:users@ovirt.org>; Martin Necas <mnecas(a)redhat.com<mailto:mnecas@redhat.com>> Subject: Re: [ovirt-users] Ansible ovirt.ovirt_vm nics Hi Matthew, Could you please share with us your playbook? Which ansible version are you using? Are you using ovirt_vm module from oVirt Ansible Collection which contains newer versions than the ovirt_vm module included in Ansible 2.9? https://docs.ansible.com/ansible/latest/collections/ovirt/ovirt/ovirt_vm_mo… Thanks, Martin On Thu, Apr 22, 2021 at 9:28 AM Matthew.Stier(a)fujitsu.com<mailto:Matthew.Stier@fujitsu.com> <Matthew.Stier(a)fujitsu.com<mailto:Matthew.Stier@fujitsu.com>> wrote: The ‘nics’ section of ovirt_vm is vague and with nearly no examples. My playbook is based upon https://blogs.oracle.com/scoter/ansible-with-oracle-linux-virtualization-ma… I’ve made several modifications, (more default) and added a few lines, which I believe is supposed to assign vnic profiles to the primary vnic (nic1), based upon the definition of ‘vm_nic1_profile’ defined in an included ‘ini’ file. (the profile defaulting to blank if it is not defined in the ini file.) It isn’t doing its job. nics: - name: “nic1” profile_name: "{{ hostvars[item]['vm_nic1_profile'] | default('') }}” The playbook runs without complaint. If I run it with option ‘-vvv’ part of the output lists the variables, but the ‘nics’ variable is an empty list (nics[]) Any hints on what I’m doing wrong? I’ve checked the forum, but it tends to strip leading spaces, which is bad for indent sensitive code. _______________________________________________ Users mailing list -- users(a)ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave(a)ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MEZLXGSSRCRJR… -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

1 0

HCI - oVirt for CEPH
by penguin pages 27 Apr '21

27 Apr '21

I have been building out HCI stack with KVM/RHEV + oVirt with the HCI deployment process. This is very nice for small / remote site use cases, but with Gluster being anounced as EOL in 18 months, what is the replacement plan? Are their working projects and plans to replace Gluster with CEPH? Are their deployment plans to get an HCI stack onto a supported file system? I liked gluster for the control plan for the oVirt engine and smaller utility VMs as each system has a full copy, I can retrieve /extract a copy of the VM without having all bricks back... it was just "easy" to use. CEPH just means more complexity.. and though it scales better and has better features, it means that repair means having critical mass of nodes up before you can extra data (vs any disk can be pulled out of a gluster node, plugged into my laptop and I can at least extract the data). I guess I am not trying to debate shifting to CEPH.. it does not matter.. that ship sailed... What I am asking is when / what are the plans for replacement of Gluster for HCI. Because right now, for small sites for HCI, when Gluster is no longer supported.. and CEPH does not make it... is to go VMWare and vSAN or some other total different stack.

4 9

Re: Attempting to detach a storage domain
by Matthew.Stier＠fujitsu.com 27 Apr '21

27 Apr '21

Problem solved. I used Ansible and ran 'grep Duplicate /var/log/vdsm/vdsm.log' on all 121 of my hosts. Only the log of the host currently running the hosted-engine had entries. I migrated the hosted-engine, and all other VMs to other hosts, and then put the affected host into maintenance. I then undeployed it; then patched it; and finally deployed it. Its back up and running and taking VMs, and no more 'Duplicate keys' errors. -----Original Message----- From: Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> Sent: Monday, April 26, 2021 7:18 PM To: Nir Soffer <nsoffer(a)redhat.com> Cc: users(a)ovirt.org Subject: [ovirt-users] Re: Attempting to detach a storage domain There is no /var/log/vdsm directory on the hosted-engine. This is from the current host, hosting the engine: aaa.bbb.ccc.100 is the hosted-engine 2021-04-26 19:12:42,760-0500 INFO (jsonrpc/2) [api.host] START getCapabilities() from=::ffff:aaa.bbb.ccc.100,37812 (api:48) 2021-04-26 19:12:42,763-0500 INFO (jsonrpc/2) [api.host] FINISH getCapabilities error=internal error: Duplicate key from=::ffff:168.127.151.100,37812 (api:52) 2021-04-26 19:12:42,763-0500 ERROR (jsonrpc/2) [jsonrpc.JsonRpcServer] Internal server error (__init__:350) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1371, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 95, in get machinetype.compatible_cpu_models()) File "/usr/lib/python2.7/site-packages/vdsm/common/cache.py", line 43, in __call__ value = self.func(*args) File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 142, in compatible_cpu_models all_models = domain_cpu_models(c, arch, cpu_mode) File "/usr/lib/python2.7/site-packages/vdsm/machinetype.py", line 97, in domain_cpu_models domcaps = conn.getDomainCapabilities(None, arch, None, virt_type, 0) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3844, in getDomainCapabilities if ret is None: raise libvirtError ('virConnectGetDomainCapabilities() failed', conn=self) libvirtError: internal error: Duplicate key 2021-04-26 19:12:42,764-0500 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.getCapabilities failed (error -32603) in 0.00 seconds (__init__:312) -----Original Message----- From: Nir Soffer <nsoffer(a)redhat.com> Sent: Monday, April 26, 2021 5:12 PM To: Stier, Matthew <Matthew.Stier(a)fujitsu.com> Cc: Benny Zlotnik <bzlotnik(a)redhat.com>; users(a)ovirt.org Subject: Re: [ovirt-users] Re: Attempting to detach a storage domain On Tue, Apr 27, 2021 at 12:48 AM Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> wrote: > > I'm getting tons of these in the > hosted-engine:/var/log/ovirt-engine/engine.log > > 2021-04-26 16:40:40,260-05 ERROR > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] > (EE-ManagedThreadFactory-engineScheduled-Thread-8) [] EVENT_ID: > VDS_BROKER_COMMAND_FAILURE(10,802), VDSM xxxx.yyyy.zzz command Get > Host Capabilities failed: Internal JSON-RPC error: {'reason': > 'internal error: Duplicate key'} This looks like bad response from vdsm. Can you share the vdsm log from the host xxxx.yyyy.zzz? /var/log/vdsm/vdsm.log Nir _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z2GOGEZ2DZX34…

1 0

Re: Attempting to detach a storage domain
by Nir Soffer 27 Apr '21

27 Apr '21

On Tue, Apr 27, 2021 at 12:48 AM Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> wrote: > > I'm getting tons of these in the hosted-engine:/var/log/ovirt-engine/engine.log > > 2021-04-26 16:40:40,260-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-8) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM xxxx.yyyy.zzz command Get Host Capabilities failed: Internal JSON-RPC error: {'reason': 'internal error: Duplicate key'} This looks like bad response from vdsm. Can you share the vdsm log from the host xxxx.yyyy.zzz? /var/log/vdsm/vdsm.log Nir

2 1

Re: Attempting to detach a storage domain
by Benny Zlotnik 26 Apr '21

26 Apr '21

I think engine.log would be a good place to start, it would likely tell us on which host there's a problem (if there is one) On Sun, Apr 25, 2021 at 10:22 AM Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> wrote: > > Which logs, on which hosts should I be looking through? > > I have a hint of that from my research, but all 121 hosts are up and running. > > -----Original Message----- > From: Benny Zlotnik <bzlotnik(a)redhat.com> > Sent: Sunday, April 25, 2021 1:53 AM > To: Stier, Matthew <Matthew.Stier(a)fujitsu.com> > Cc: users(a)ovirt.org > Subject: Re: [ovirt-users] Attempting to detach a storage domain > > What do the logs say? > This usually means that not all hosts were able to disconnect from it > > On Sun, Apr 25, 2021 at 9:45 AM Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> wrote: > > > > Ovirt: 4.3.10 > > > > Storage: iSCSI > > > > > > > > Problem: Attempting to place storage domain into ‘maintenance’ in preparation for detachment and destruction, has left it hung in a ‘Preparation for maintenance’ state. > > > > > > > > I have three storage domain I need to put into maintenance, detach and delete. When I placed place the first, and smallest (100GB) into maintenance mode, it switched to ‘Preparing for Maintenance’, and has stuck there for hours. > > > > > > > > Early on, I was able to re-activate it, but I do want to remove for use somewhere else, and I want make sure I can remove it, before I try to do the same with two 11TB storage domains. > > > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org Privacy > > Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/HY3LWX7H > > MN56SDHNQWB3MDXLLU7GNLGX/ >

2 1

How to reduce load during create VM
by ovirt.org＠nevim.eu 26 Apr '21

26 Apr '21

Hi everyone, please, can you limit the "Create VM" process somewhere so that setting up quota does not load the server so much, especially storage when duplicating the template disk?

3 3

oVirt on GCP
by mario.gandara＠gmail.com 26 Apr '21

26 Apr '21

Hello, guys! I'm trying to deploy oVirt on GCP for the last two weeks. Someone did it on GCP using nested virtualization? My scenario is: One standalone oVirt Engine and two hosts. In the future I imagine to scale this solution. Thank you. Mário Gândara

2 1

How do I share a disk across multiple VMs?
by David White 26 Apr '21

26 Apr '21

I need to mount a partition across 3 different VMs. How do I attach a disk to multiple VMs? This looks like fairly old documentation-not-documentation: https://www.ovirt.org/develop/release-management/features/storage/sharedraw… Sent with ProtonMail Secure Email.

8 24

Problem with ISCSI multipathing identifying faild path to a LUN every few seconds.
by Gal Villaret 25 Apr '21

25 Apr '21

Hi All, In the past few days, we have been having problems with hosts becoming unmanageable due to multipathd identifying false failed paths and VDSM crashing because of it. We were running version 4.4.1 and upgrading to 4.4.5(engine) and 4.4.6(nodes) seems to have resolved the VDSM. However, currently, we see that the multipathing events continue. From what we have observed, the events start in correlation to the host reporting on low swap space. The low swap space seems to be related to Commvault backup operation running. By running top on the host while a backup operation is running I can see swap being consumed to 100% although there is plenty of RAM available. After the multipath events start happening the only means of stopping it was to reboot the host. This is the warning in oVirt UI: Apr 24, 2021, 9:14:07 PM - Available swap memory of host Ovirt-Node2 [953 MB] is under defined threshold [1024 MB]. This is the first appearance of the multipath event in /var/log/messages: Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=15s Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 Sense Key : Aborted Command [current] Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 <<vendor>>ASC=0xc1 ASCQ=0x1 Apr 24 21:14:58 ovirt-node2 kernel: sd 21:0:0:9: [sdgi] tag#77 CDB: Read(16) 88 00 00 00 00 00 79 ac 75 a0 00 00 06 00 00 00 Apr 24 21:14:58 ovirt-node2 kernel: blk_update_request: I/O error, dev sdgi, sector 2041345440 op 0x0:(READ) flags 0x4200 phys_seg 192 prio class 0 Apr 24 21:14:58 ovirt-node2 kernel: device-mapper: multipath: 253:20: Failing path 131:224. Apr 24 21:14:58 ovirt-node2 multipathd[3044]: sdgi: mark as failed Apr 24 21:14:58 ovirt-node2 multipathd[3044]: 3600000e00d2c0000002cb4a8000b0000: remaining active paths: 11 Apr 24 21:15:03 ovirt-node2 multipathd[3044]: 3600000e00d2c0000002cb4a8000b0000: sdgi - tur checker reports path is up Apr 24 21:15:03 ovirt-node2 multipathd[3044]: 131:224: reinstated Apr 24 21:15:03 ovirt-node2 multipathd[3044]: 3600000e00d2c0000002cb4a8000b0000: remaining active paths: 12 Apr 24 21:15:03 ovirt-node2 kernel: device-mapper: multipath: 253:20: Reinstating path 131:224. Apr 24 21:15:03 ovirt-node2 kernel: sd 21:0:0:9: alua: port group 8091 state A preferred supports toluSNA Apr 24 21:15:03 ovirt-node2 kernel: sd 21:0:0:9: alua: port group 8091 state A preferred supports toluSNA Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=15s Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 Sense Key : Aborted Command [current] Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 <<vendor>>ASC=0xc1 ASCQ=0x1 Apr 24 21:15:13 ovirt-node2 kernel: sd 13:0:0:9: [sdau] tag#25 CDB: Read(16) 88 00 00 00 00 00 79 ac 75 a0 00 00 06 00 00 00 Apr 24 21:15:13 ovirt-node2 kernel: blk_update_request: I/O error, dev sdau, sector 2041345440 op 0x0:(READ) flags 0x4200 phys_seg 192 prio class 0 Apr 24 21:15:13 ovirt-node2 kernel: device-mapper: multipath: 253:20: Failing path 66:224. Apr 24 21:15:13 ovirt-node2 multipathd[3044]: sdau: mark as failed Underlying storage is Fujitsu DX200 S5 with all SSD drives. Each host has two 10Gbit network adapters dedicated to ISCSI. Any help with this would be highly appreciated. Thanks, Gal Villaret

2 1

Migration VM and Storage Domains
by LS CHENG 25 Apr '21

25 Apr '21

Hi I have a question about migrating VM's. If I have 2 VM and each VM is stored in separate Storage Domains can the VM migrate between physical hosts? Or the VM must be stored in the same Storage Domain? For example VM1 is stored in SD1 and runs in HOST-A VM2 is stored in SD2 and runs in HOST-B Can VM2 migrated to HOST-A? Thanks

2 2

Re: Attempting to detach a storage domain
by Benny Zlotnik 25 Apr '21

25 Apr '21

What do the logs say? This usually means that not all hosts were able to disconnect from it On Sun, Apr 25, 2021 at 9:45 AM Matthew.Stier(a)fujitsu.com <Matthew.Stier(a)fujitsu.com> wrote: > > Ovirt: 4.3.10 > > Storage: iSCSI > > > > Problem: Attempting to place storage domain into ‘maintenance’ in preparation for detachment and destruction, has left it hung in a ‘Preparation for maintenance’ state. > > > > I have three storage domain I need to put into maintenance, detach and delete. When I placed place the first, and smallest (100GB) into maintenance mode, it switched to ‘Preparing for Maintenance’, and has stuck there for hours. > > > > Early on, I was able to re-activate it, but I do want to remove for use somewhere else, and I want make sure I can remove it, before I try to do the same with two 11TB storage domains. > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/HY3LWX7HMN56S…

2 1

Attempting to detach a storage domain
by Matthew.Stier＠fujitsu.com 25 Apr '21

25 Apr '21

Ovirt: 4.3.10 Storage: iSCSI Problem: Attempting to place storage domain into 'maintenance' in preparation for detachment and destruction, has left it hung in a 'Preparation for maintenance' state. I have three storage domain I need to put into maintenance, detach and delete. When I placed place the first, and smallest (100GB) into maintenance mode, it switched to 'Preparing for Maintenance', and has stuck there for hours. Early on, I was able to re-activate it, but I do want to remove for use somewhere else, and I want make sure I can remove it, before I try to do the same with two 11TB storage domains.

1 0

Can we remove Export and ISO Domain from KVM ?.
by smuthyala＠helenoftroy.com 23 Apr '21

23 Apr '21

I need to migrate both Export and ISO Domains to newer storage. Can i remove them and recreate. Or is there a way to export and import into a newer storage. TIA

1 0

ovsdb-server tlsv1 unknown ca
by ozmen62＠hotmail.com 23 Apr '21

23 Apr '21

1 0

Re: Ansible ovirt.ovirt_vm nics
by Martin Perina 22 Apr '21

22 Apr '21

Hi Matthew, Could you please share with us your playbook? Which ansible version are you using? Are you using ovirt_vm module from oVirt Ansible Collection which contains newer versions than the ovirt_vm module included in Ansible 2.9? https://docs.ansible.com/ansible/latest/collections/ovirt/ovirt/ovirt_vm_mo… Thanks, Martin On Thu, Apr 22, 2021 at 9:28 AM Matthew.Stier(a)fujitsu.com < Matthew.Stier(a)fujitsu.com> wrote: > The ‘nics’ section of ovirt_vm is vague and with nearly no examples. > > > > My playbook is based upon > https://blogs.oracle.com/scoter/ansible-with-oracle-linux-virtualization-ma… > > > > I’ve made several modifications, (more default) and added a few lines, > which I believe is supposed to assign vnic profiles to the primary vnic > (nic1), based upon the definition of ‘vm_nic1_profile’ defined in an > included ‘ini’ file. (the profile defaulting to blank if it is not defined > in the ini file.) > > > > It isn’t doing its job. > > > > nics: > > - name: “nic1” > > profile_name: "{{ hostvars[item]['vm_nic1_profile'] | default('') }}” > > > > > > The playbook runs without complaint. If I run it with option ‘-vvv’ part > of the output lists the variables, but the ‘nics’ variable is an empty list > (nics[]) > > > > Any hints on what I’m doing wrong? I’ve checked the forum, but it tends > to strip leading spaces, which is bad for indent sensitive code. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/MEZLXGSSRCRJR… > -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

2 1