- Users - oVirt List Archives

error id 980
by ozmen62＠hotmail.com 04 May '21

04 May '21

Hi, First of all, thanks for this great list and the people who try to help eachother. This time my problem is about engine OVF file or engine storage setting. How can i came with this idea, because every 2-3 hours engine try to migrate other host(From host1 to host2) "Invalid status on Data Center B300. Setting status to Non Responsive." when i check hosts log , see these ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM stopped on localhost ovirt-ha-agent ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore ERROR Unable to extract HEVM OVF i believe there is some kind of fc storage path or config problem. My storage setting is good. there is no error about them Is there anyone who experienced about that to solve it thanks

2 5

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

Investigation so far has showed, that symlink for conf_volume/conf_image is missing on Hosts 1 and 3. According to the ovirt-engine deep dive, connection/access to conf_volume is required for adding new HE Hosts as it holds its configuration template. This is most likely the reason why I could not add new Host and why the Host1 and Host3 were occasionally complaining the the conf_image_uuid is missing. I also checked the DB tb images – the conf_image and conf_volume entries exist. Question: Considering DB entries seem correct, am I “safe” to manually create the symlink on Host1 and Host3 , from global maintenance mode, set correct permissions and than exit maintenance mode? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 12:44 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Dear Strahil and Yedidyah, Is it possible that HostedEngine DEPLOY on Host2 is failing due to “inability” to access the conf_volume_UUID “c518f937-60fe-4fed-a54c-db11328bb507” I do not see that error message on Host2, but I see it on Host1 and Host3: 2021-05-03 02:43:32,438-0700 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='b4bba796-b62b-41cb-8d91-6637c5fb9705') Unexpected error (task:875) 2021-05-03 02:43:32,439-0700 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH prepareImage error=Volume does not exist: (u'c518f937-60fe-4fed-a54c-db11328bb507',) (dispatcher:83) Could this be result of hosted-engine.conf staying empty and deployment not working? If I check the /var/log/ovirt-hosted-engine-setup/ folder is empty. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:38 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

Dear Strahil and Yedidyah, Is it possible that HostedEngine DEPLOY on Host2 is failing due to “inability” to access the conf_volume_UUID “c518f937-60fe-4fed-a54c-db11328bb507” I do not see that error message on Host2, but I see it on Host1 and Host3: 2021-05-03 02:43:32,438-0700 ERROR (jsonrpc/5) [storage.TaskManager.Task] (Task='b4bba796-b62b-41cb-8d91-6637c5fb9705') Unexpected error (task:875) 2021-05-03 02:43:32,439-0700 ERROR (jsonrpc/5) [storage.Dispatcher] FINISH prepareImage error=Volume does not exist: (u'c518f937-60fe-4fed-a54c-db11328bb507',) (dispatcher:83) Could this be result of hosted-engine.conf staying empty and deployment not working? If I check the /var/log/ovirt-hosted-engine-setup/ folder is empty. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:38 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Cluster level 4.5
by Don Dupuis 03 May '21

03 May '21

What version of libvirt is required for a host to be put in this cluster level? I am using CentOS 8.3 and cpu is Cascade Lake Server. It says that my host is only compatible with cluster version 4.2,4.3 and 4.4. I am doing a new install of Ovirt 4.4.5. I have tried to update libvirt version but have run into issues. Currently installed libvirt is libvirt-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64. Don

2 1

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

This is ERROR i caught on Host2 during HE Deploy: 2021-05-03 02:36:07,982-0700 ERROR (periodic/0) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 02:36:07,985-0700 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:71) 2021-05-03 02:36:07,987-0700 ERROR (ioprocess/4934) [IOProcessClient] (054c43fc-1924-4106-9f80-0f2ac62b9886) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:07,992-0700 ERROR (ioprocess/4948) [IOProcessClient] (ec0dc13d-8721-42a1-a5c3-762a3e235f76) Communication thread failed (__init__:160) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 114, in _communicate raise Exception("FD closed") Exception: FD closed 2021-05-03 02:36:08,009-0700 INFO (MainThread) [jsonrpc.JsonRpcServer] Stopping JsonRPC Server (__init__:442) ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 11:34 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Marko Vrgotic 03 May '21

03 May '21

I executed ovirt-hosted-engine-cleanup on Host2 and it seems as it went ok: [root@ovirt-sj-02 log]# ovirt-hosted-engine-cleanup This will de-configure the host to run ovirt-hosted-engine-setup from scratch. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Destroy hosted-engine VM ===- You must run deploy first error: failed to get domain 'HostedEngine' error: Domain not found: no domain with matching name 'HostedEngine' -=== Stop HA services ===- -=== Shutdown sanlock ===- shutdown force 1 wait 0 shutdown done 0 -=== Disconnecting the hosted-engine storage domain ===- You must run deploy first -=== De-configure VDSM networks ===- ovirtmgmt ovirtmgmt A previously configured management bridge has been found on the system, this will try to de-configure it. Under certain circumstances you can loose network connection. Caution, this operation should be used with care.Are you sure you want to proceed? [y/n] y -=== Stop other services ===- -=== De-configure external daemons ===- Removing database file /var/lib/vdsm/storage/managedvolume.db -=== Removing configuration files ===- ? /etc/init/libvirtd.conf already missing - removing /etc/libvirt/nwfilter/vdsm-no-mac-spoofing.xml ? /etc/ovirt-hosted-engine/answers.conf already missing - removing /etc/ovirt-hosted-engine/hosted-engine.conf - removing /etc/vdsm/vdsm.conf - removing /etc/pki/vdsm/certs/cacert.pem - removing /etc/pki/vdsm/certs/vdsmcert.pem - removing /etc/pki/vdsm/keys/vdsmkey.pem - removing /etc/pki/vdsm/libvirt-spice/ca-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-cert.pem - removing /etc/pki/vdsm/libvirt-spice/server-key.pem - removing /etc/pki/vdsm/libvirt-vnc/ca-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-cert.pem - removing /etc/pki/vdsm/libvirt-vnc/server-key.pem - removing /etc/pki/CA/cacert.pem - removing /etc/pki/libvirt/clientcert.pem - removing /etc/pki/libvirt/private/clientkey.pem ? /etc/pki/ovirt-vmconsole/*.pem already missing - removing /var/cache/libvirt/qemu ? /var/run/ovirt-hosted-engine-ha/* already missing -=== Removing IP Rules ===- Reran the HE Deploy on Host2 and from logs I see exactly same situation as in my previous email. This is AGENT|BROKER logs with DEBUG: MainThread::INFO::2021-05-03 02:32:47,007::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::DEBUG::2021-05-03 02:32:47,007::agent::72::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Running agent MainThread::DEBUG::2021-05-03 02:32:47,007::hosted_engine::220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Executing: openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject MainThread::DEBUG::2021-05-03 02:32:47,064::hosted_engine::230::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Certificate subject: subject= /O=ictv.com/CN=ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 02:32:47,065::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::568::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::DEBUG::2021-05-03 02:32:47,066::hosted_engine::600::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) Checking vdsmd status MainThread::DEBUG::2021-05-03 02:32:47,130::hosted_engine::605::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_check_service) vdsmd running MainThread::DEBUG::2021-05-03 02:32:47,130::util::384::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(__log_debug) Creating a new json-rpc connection to VDSM Client localhost:54321::DEBUG::2021-05-03 02:32:47,144::concurrent::258::root::(run) START thread <Thread(Client localhost:54321, started daemon 140520355350272)> (func=<bound method Reactor.process_requests of <yajsonrpc.betterAsyncore.Reactor object at 0x7fcd71fe9590>>, args=(), kwargs={}) Client localhost:54321::DEBUG::2021-05-03 02:32:47,151::stompclient::138::yajsonrpc.protocols.stomp.AsyncClient::(_process_connected) Stomp connection established MainThread::DEBUG::2021-05-03 02:32:47,161::stompclient::294::jsonrpc.AsyncoreClient::(send) Sending response MainThread::INFO::2021-05-03 02:32:47,192::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 02:32:47,195::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 02:32:47,195::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 02:32:47,196::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 02:32:47,196::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 02:32:47,196::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Marko Vrgotic <M.Vrgotic(a)activevideo.com> Date: Monday, 3 May 2021 at 10:19 To: Yedidyah Bar David <didi(a)redhat.com>, Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host Hi Strahil and Yedidyah, First of all I want to thank you for helping me. So, this is what I did: * Executed steps as in https://access.redhat.com/solutions/2212601 and https://access.redhat.com/solutions/3319891 on working Hosts (Host1 and Host3) – to clean up failed deployment * Verified on Host1 and Host3 that metadata is cleaned by running : [root@ovirt-sj-03 ~]# hosted-engine --vm-status | grep -B2 "Host ID" Status up-to-date : True Hostname : ovirt-sj-01.ictv.com Host ID : 1 -- Status up-to-date : True Hostname : ovirt-sj-03.ictv.com Host ID : 3 * Reran hosted-engine deploy from UI on Host2 From the start of deployment, I could see following ERROR messages on Host2: VDSM: 2021-05-03 07:58:21,205+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:36,257+0000 ERROR (periodic/3) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) 2021-05-03 07:58:46,093+0000 ERROR (jsonrpc/7) [root] failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? (api:196) OVIRT-HA-AGENT|BROKER: MainThread::ERROR::2021-05-03 08:12:57,690::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:12:57,691::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2021-05-03 08:13:08,005::agent::67::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.3.6 started MainThread::INFO::2021-05-03 08:13:08,061::hosted_engine::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-sj-02.ictv.com MainThread::INFO::2021-05-03 08:13:08,187::hosted_engine::543::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2021-05-03 08:13:08,189::brokerlink::80::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor network, options {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None} MainThread::ERROR::2021-05-03 08:13:08,190::hosted_engine::559::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2021-05-03 08:13:08,190::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper return he.start_monitoring() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring self._initialize_broker() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker m.get('options', {})) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 89, in start_monitor ).format(t=type, o=options, e=e) RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': None, 'network_test': None, 'tcp_t_port': None, 'addr': None}] MainThread::ERROR::2021-05-03 08:13:08,191::agent::145::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent MainThread::INFO::2021-05-03 08:13:08,191::agent::89::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down And the Hosted-Engine.conf file looks like this: [root@ovirt-sj-02 ~]# cat /etc/ovirt-hosted-engine/hosted-engine.conf ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem host_id=2 Ovirt hosted_engine network path is mounted. What could be the reason hosted-engine.conf is missing all other entries? Do I copy the hosted-engine.conf from host1 or host3, making sure host_id is unique (host_id=2) on Host2, and try to re-deploy? How does the hosted-engine.conf gets loaded or from where? Am I missing a mount point? Is the deployment failing because hosted-engine.conf is empty or it’s the other way around? ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com<mailto:m.vrgotic@activevideo.com> w: www.activevideo.com<http://www.activevideo.com> ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. From: Yedidyah Bar David <didi(a)redhat.com> Date: Sunday, 2 May 2021 at 08:44 To: Strahil Nikolov <hunter86_bg(a)yahoo.com> Cc: Marko Vrgotic <M.Vrgotic(a)activevideo.com>, users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: Unable to migrate Engine to another HE Host ***CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender!!!*** On Sat, May 1, 2021 at 6:46 PM Strahil Nikolov via Users <users(a)ovirt.org> wrote: > > As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing. > Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. > > Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. > > > I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. 'engine-cleanup' is a utility you run on the engine machine, which does basically the opposite of 'engine-setup', and in any case cleans up the _engine_, not any particular host. If you run it, your engine is gone, likely forever (unless you kept backups). What you might were looking for is 'ovirt-hosted-engine-cleanup'. This one cleans up a host from a hosted-engine deployment. It's generally intended to be used after a failed deployment attempt. I think it can work well also in your case, but would first try to fix using other means. > > If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. > > Best Regards, > Strahil Nikolov > > On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic > <M.Vrgotic(a)activevideo.com> wrote: > > Dear oVirt, > > > > I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. > > > > I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: > > > > 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) > > > > Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: > > > > On Engine: > > 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) > > > > On source Host: > > 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) > > Traceback (most recent call last): > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run > > time.time(), migrationParams, machineParams > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration > > self._perform_with_conv_schedule(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule > > self._perform_migration(duri, muri) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration > > self._migration_flags) > > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f > > ret = attr(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper > > ret = f(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper > > return func(inst, *args, **kwargs) > > File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 > > if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) > > libvirtError: operation aborted: migration out job: canceled by client > > > > > > I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. > > My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. > > > > This is production system, so I cannot just move on to upgrading/deploying to 4.4. > > > > Additionally – : > > Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? > What is the effect or reinitialize-lockspace? > > > > Kindly awaiting your reply. Happy to provide any additional information needed. > > > > > > > > ----- > > kind regards/met vriendelijke groeten > > > > Marko Vrgotic > Sr. System Engineer @ System Administration > > > ActiveVideo > > o: +31 (35) 6774131 > > m: +31 (65) 5734174 > > e: m.vrgotic(a)activevideo.com > w: www.activevideo.com<http://www.activevideo.com> > > > > ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. > > > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > oVirt Code of Conduct: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt… > List Archives: https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi… -- Didi

1 0

Re: Unable to migrate Engine to another HE Host
by Strahil Nikolov 03 May '21

03 May '21

As we have tested off the list, it seems that the symbolic link in /var/lib/vdsm that ovirt-ha-agent/broker create was missing.Yet migration succeeds, but the donor host looses score as the vm 'died unexpectedly'. Try to cleanup the host2's metadata and try to provision it , so you can proceed with the fix of host1 & host3. I have no clue if engine-cleanup will affect the shared syorage, but it's possible - so use as last resort. If you fail to add host2 , you can always reinstall it as host4 and try to add it fresh. Best Regards,Strahil Nikolov On Fri, Apr 30, 2021 at 16:27, Marko Vrgotic<M.Vrgotic(a)activevideo.com> wrote:  Dear oVirt, I have already reached out twice regarding the issues that occurred, due to power outage, but noticed only when upgrading engine to latest 4.3. version. I am unable to redeploy engine on Host2, the hosted-engine file stays empty and VDSM on Hosts1 and 3 is reporting, even though I cleared the metadata for the Host2, on Host 1 and Host3: 2021-04-30 05:57:58,454-0700 ERROR (jsonrpc/7) [ovirt_hosted_engine_ha.client.client.HAClient] Malformed metadata for host 2: received 0 of 512 expected bytes (client:137) Today I tried to migrate HE from Host 3 to Host 1 and it fails each time with following message: On Engine: 2021-04-30 12:57:56,961Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1233892) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: HostedEngine, Source: ovirt-sj-03.ictv.com, Destination: ovirt-sj-01.ictv.com) On source Host: 2021-04-30 05:57:56,705-0700 ERROR (migsrc/66b6d489) [virt.vm] (vmId='66b6d489-ceb8-486a-951a-355e21f13627') Failed to migrate (migration:450) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 431, in _regular_run time.time(), migrationParams, machineParams File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 505, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 591, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 525, in _perform_migration self._migration_flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1781, in migrateToURI3 if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self) libvirtError: operation aborted: migration out job: canceled by client I know that this version is end of life – but I would very much appreciate if someone could help me asses if this means corruption in DB or the overall damage, simply to know how to plan further actions. My impression was that I still had to functional HE Hosts in the pool, but after seeing migration failure, it’s pretty much down to single host. This is production system, so I cannot just move on to upgrading/deploying to 4.4. Additionally – : - Is the effect of the engine-cleanup on HE Host local or it affects all HE Hosts? Could that help bringing the Host back to state so that HE can be re-deployed? - What is the effect or reinitialize-lockspace? Kindly awaiting your reply. Happy to provide any additional information needed. ----- kind regards/met vriendelijke groeten Marko Vrgotic Sr. System Engineer @ System Administration ActiveVideo o: +31 (35) 6774131 m: +31 (65) 5734174 e: m.vrgotic(a)activevideo.com w: www.activevideo.com ActiveVideo Networks BV. Mediacentrum 3745 Joop van den Endeplein 1.1217 WJ Hilversum, The Netherlands. The information contained in this message may be legally privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or ActiveVideo Networks, LLC by telephone at +1 408.931.9200 and delete or destroy any copy of this message. _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FRTRNTSGPLQOS…

3 2

new engine restore from backup
by ozmen62＠hotmail.com 03 May '21

03 May '21

Hi, I've built e new engine and tried to restore from backup after run engine-setup it returns; [ ERROR ] Failed to execute stage 'Misc configuration': function getdwhhistorytimekeepingbyvarname(unknown) does not exist LINE 2: select * from GetDwhHistoryTimekeepingByVarName( redhat page says seploy new engine, but already done it. old and new engine are same version Do you have any idea how can i restore engine. There is no another backup

2 4

Trouble migrating the websocket-proxy / Trying to make VM-Portal publicly available
by tivon.haeberlein＠secges.de 02 May '21

02 May '21

Hi community, I hope you can help me on this one. My oVirt cluster is deployed on an internal network (10.1.30.0/24). At the moment I am trying to make the VM-Portal publicly available so people can administer their vms with the portal and noVNC. I have read the oVirt Documentation (as well as trying a couple of things) and found that the best way to accomplish this is by installing a proxy server which proxies HTTP and HTTPS to the engine. To get the noVNC working I have to install the ovirt-engine-websocket-proxy on the proxy host as well and set it up. For the proxy host I'm using centos 8. For the HTTP and HTTPS proxy I'm using haproxy (which is working perfectly already). For the noVNC Proxy I'm trying to follow the documentation (https://www.ovirt.org/documentation/installing_ovirt_as_a_self-hosted_engin…) but am facing a couple of issues. Issues start with removing the websocket proxy from the engine machine. I put the cluster into global-maintenance-mode and run "engine-cleanup" as told in the docs: [root@engine ~]# engine-cleanup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-remove-20210502182713-ne56jy.log Version: otopi-1.9.4 (otopi-1.9.4-1.el8) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- [ INFO ] Stage: Setup validation During execution engine service will be stopped (OK, Cancel) [OK]: [ INFO ] Hosted Engine HA is in Global Maintenance mode. All the installed ovirt components are about to be removed, data will be lost (OK, Cancel) [Cancel]: [ ERROR ] Failed to execute stage 'Setup validation': Aborted by user [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-remove-20210502182713-ne56jy.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210502182730-cleanup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of cleanup failed As you can see the output differs a bit from the docs. Instead of being asked "Do you want to remove all components? (Yes, No) [Yes]: No" I'm being notified that all data will be lost when continuing. Of course I do not want to loose all data so I abort the cleanup. I then tried to just install the websocket-proxy on the proxy server and ran into the next issue. On the proxy server I am also following the docs and running "yum install ovirt-engine-websocket-proxy" followed by "engine-setup": [root@proxy ~]# engine-setup [ INFO ] Stage: Initializing [ INFO ] Stage: Environment setup Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20210502183039-ydl26v.log Version: otopi-1.9.4 (otopi-1.9.4-1.el8) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- Configure WebSocket Proxy on this host (Yes, No) [Yes]: --== PACKAGES ==-- [ INFO ] Checking for product updates... [ INFO ] DNF No matches found for the following disable plugin patterns: versionlock [ INFO ] No product updates found ### I've cut a bit of the output to keep it shorter ### [ INFO ] Stage: Transaction setup [ INFO ] Stopping websocket-proxy service [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ ERROR ] Failed to execute stage 'Misc configuration': 'OVESETUP_SYSTEM/selinuxBooleans' [ INFO ] DNF Performing DNF transaction rollback [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210502183039-ydl26v.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210502183159-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed As you can see it failed at stage "'Misc configuration': 'OVESETUP_SYSTEM/selinuxBooleans'". Here I did try to disable SElinux as well as enforcing it (as set at default) without any success. Am I on the right approach to make the VM-Portal publicly avalible by doing this or is there a better / more supported way? Can you help me out on getting this running or pointing me in the right direction (if this is the right approach)? Thank you guys! Best regards, Tivon I'm new to the community but will help to keep oVirt growing and support where I can.

1 0

How to assign a server disk and nic profile via REST API?
by ovirt.org＠nevim.eu 02 May '21

02 May '21

Hello to everybody, for about an hour I have stuck my nose in the ovirt and rhev documentation, but I still can't understand how to use the REST API to change the disk profile and nothing the server profile. Ideally, how to set it for them when setting up a VM. Thank you so much for the advice.

2 7