- Users - oVirt List Archives

the virtual machine crushed and I cant shutdown the vm successfully
by zhouhao＠vip.friendtimes.net 04 Sep '19

04 Sep '19

Dear ovirt developers: The trouble bothered me for a long time;Some times my vm will crush and I cant shutdown it,I had to reboot the ovirt-node to slove it; the vm's error below The ovirt-node's error below The vm's threading on the ovirt-node, the IO ratio is 100% and the vm's process change to defunct I can not kill it,every time I had to shutdown the ovirt-node on the engine website ,the vm's status always in the way to shutdown ,even if I wait for it for hours; It either failed or is shutting down and the "power off" cant shutdown the vm too. ---------------------------------------------------------------- The other infomation about my ovirt-node node-version： node hardware zhouhao(a)vip.friendtimes.net

1 0

the virtual machine crushed and I cant shutdown the vm successfully
by zhouhao＠vip.friendtimes.net 04 Sep '19

04 Sep '19

Dear ovirt developers: The trouble bothered me for a long time;Some times my vm will crush and I cant shutdown it,I had to reboot the ovirt-node to slove it; the vm's error below The ovirt-node's error below The vm's threading on the ovirt-node, the IO ratio is 100% and the vm's process change to defunct I can not kill it,every time I had to shutdown the ovirt-node on the engine website ,the vm's status always in the way to shutdown ,even if I wait for it for hours; It either failed or is shutting down and the "power off" cant shutdown the vm too. ---------------------------------------------------------------- The other infomation about my ovirt-node node-version： node hardware zhouhao(a)vip.friendtimes.net

1 0

iSCSI Multipath/multiple gateways
by Dan Poltawski 03 Sep '19

03 Sep '19

My iSCSI target (ceph-based) has multiple gateways, I’d like to ensure my hosted storage is aware of these such that each gateway can be rebooted for maintance without impacting service. What is the appropriate way to configure this so that each host knows of the multipath configuration? I do not have multiple network paths, just multiple gateways on the same network, so I think the multipath UI tool isn’t the way to do this. Any help/docs pointers appreciated. Regards, Dan ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

1 0

engine.log flooded with "Field 'foo' can not be updated when status is 'Up'"
by Matthias Leopold 02 Sep '19

02 Sep '19

Hi, I updated my production oVirt environment from 4.3.3 to 4.3.5 today. Everything went fine so far, but there's one annoying phenomenon: When I log into the "Administration Portal" and request the VM list ("/ovirt-engine/webadmin/?locale=en_US#vms") engine.log is flooded with lines like WARN [org.ovirt.engine.core.utils.ObjectIdentityChecker] (default task-10618) [54d8c375-aa72-42f8-876e-8777d9d1a08a] Field 'balloonEnabled' can not be updated when status is 'Up' "Field", task and UUID vary and the flood stops after a while. Also listing or trying to edit other entities seems to trigger this "storm" or loop over and over again to a point that log file size is becoming an issue and interface is becoming sluggish. I can also see that CPU usage of engine java process goes up. When I log out everything is quiet and "VM Portal" is not affected at all. I have seen lines like that before and know that they are usually OK (when changing VM properties), but these logs used to be linked to singular events. I suspect that the present behaviour might be linked to VMs that have "Pending Virtual Machine changes", which are in most cases "Custom Compatibility Version" changes that still stem from the upgrade to Cluster Version 4.3. I can't be sure and I can't resolve all these pending changes now, but these should not be causing such annoying behaviour in the first place. I resorted to setting engine log level to "ERROR" right now to at least stop the log file from growing, but this not a solution. I can still see CPU load going up when using the interface. I very much hope that someone can explain whats happening and tell me how to resolve this. thanks a lot Matthias

4 8

Intel Xeon X5460
by Marcello Gentile 02 Sep '19

02 Sep '19

Is this procesor supported? https://ark.intel.com/content/www/us/en/ark/products/33087/intel-xeon-proce… Seems to meet all the requirements but when I install the self hosted engine it fails with the error message below: [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 156: Host Ovirt-Node1.adsdata.ca<http://Ovirt-Node1.adsdata.ca> moved to Non-Operational state as host CPU type is not supported in this cluster compatibility version or is not supported at all, code 9000: Failed to verify Power Management configuration for Host Ovirt-Node1.adsdata.ca<http://Ovirt-Node1.adsdata.ca>., fix accordingly and re-deploy."} Thanks for any help

3 3

How to modify DC's default route setting?
by gotoh＠eis.co.jp 01 Sep '19

01 Sep '19

We have to change default route on a production Ovirt system ( ovirt4.3 : engine + 9 hosts + glusterFS subsystem, approx 90 virtual machines.) , due to network enhancement We changed engine's default gateway and could permanently change host's default route by modifying the persistent setting. However, DC's logical network's default route remains unchanged. Consequently, the logical network became out of sync. We would be glad if anyone could instruct us how to change logical network's default route of DC. Thanks in advance, Hideo Goto

1 0

Error during self-hosted install using cockpit
by Marcello Gentile 01 Sep '19

01 Sep '19

Install fails with ERROR ansible failed {'status': 'FAILED', 'ansible_type': 'task', 'ansible_task': u'Fail with error description', 'ansible_result': u"type: <type 'dict'>\nstr: {'msg': u'The host has been set in non_operational status, deployment errors: code 156: Host Ovirt-Node1.adsdata.ca<http://Ovirt-Node1.adsdata.ca> moved to Non-Operational state as host CPU type is not supported in this cluster compatibility version or is not supported at all, code 9000: Failed to verify Power Management con", 'task_duration': 1, 'ansible_host': u'localhost', 'ansible_playbook': u'/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml'}

2 1

Engine died while creating VM
by Erick Perez - Quadrian Enterprises 01 Sep '19

01 Sep '19

Hi, I was just creating a VM on a freshly installed 2-node selfhosted ovirt setup with a NFS backend. NFS connections are v4.2 for VM data domain What other log is needed? Thanks, in advance. Here is the output of the last 300 lines of /var/log/messages [root@hvm002 ~]# [root@hvm002 ~]# tail /var/log/messages -n 300 Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:34:56 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:06 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:06 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:06 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:06 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:07 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:15 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:15 hvm002 systemd: Starting Cockpit Web Service... Aug 26 04:35:15 hvm002 systemd: Started Cockpit Web Service. Aug 26 04:35:15 hvm002 cockpit-ws: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert Aug 26 04:35:15 hvm002 cockpit-ws: couldn't read from connection: Peer sent fatal TLS alert: Certificate is bad Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:17 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:17 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:17 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:17 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:27 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:27 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:27 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:27 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:28 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/5 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0> timeout=60, duration=60.00 at 0x7f091c0aadd0> task#=4 at 0x7f091c0c6e10>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:35:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:38 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:38 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:38 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:38 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:48 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:48 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:48 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:48 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:49 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:52 hvm002 systemd: Created slice User Slice of root. Aug 26 04:35:52 hvm002 systemd-logind: New session 2 of user root. Aug 26 04:35:52 hvm002 systemd: Started Session 2 of user root. Aug 26 04:35:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:59 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:59 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:59 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:59 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:36:09 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:09 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:09 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:09 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:10 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:20 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:20 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:20 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:20 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:36:30 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:30 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:30 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:30 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:31 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/5 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0> timeout=60, duration=120.00 at 0x7f091c0aadd0> task#=4 at 0x7f091c0c6e10>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:36:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:36:41 hvm002 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Aug 26 04:36:41 hvm002 abrt-server: Duplicate: core backtrace Aug 26 04:36:41 hvm002 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2019-08-26-04:32:28-3432 Aug 26 04:36:41 hvm002 abrt-server: Deleting problem directory Python-2019-08-26-04:36:41-16985 (dup of Python-2019-08-26-04:32:28-3432) Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:41 hvm002 dbus[3444]: [system] Activating service name='org.freedesktop.problems' (using servicehelper) Aug 26 04:36:41 hvm002 dbus[3444]: [system] Successfully activated service 'org.freedesktop.problems' Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-broker.service entered failed state. Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service failed. Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:36:42 hvm002 systemd: Started Session c15 of user root. Aug 26 04:36:42 hvm002 systemd: Started Session c16 of user root. Aug 26 04:36:51 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:51 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:51 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:51 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:52 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:02 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:02 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:02 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:02 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:12 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:12 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:12 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:12 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:13 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:23 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:23 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:23 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:23 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:33 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:33 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:33 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:33 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:34 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650> timeout=60, duration=60.00 at 0x7f08f8259750> task#=8 at 0x7f091c110c50>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:44 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:44 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:44 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:44 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:54 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:54 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:54 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:54 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:55 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:05 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:05 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:05 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:05 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:15 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:15 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:15 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:15 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:16 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:26 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:26 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:26 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:26 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:36 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:36 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:36 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:36 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:37 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650> timeout=60, duration=120.00 at 0x7f08f8259750> task#=8 at 0x7f091c110c50>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:47 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:47 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:47 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:47 hvm002 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Aug 26 04:38:47 hvm002 abrt-server: Duplicate: core backtrace Aug 26 04:38:47 hvm002 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2019-08-26-04:32:28-3432 Aug 26 04:38:47 hvm002 abrt-server: Deleting problem directory Python-2019-08-26-04:38:47-17322 (dup of Python-2019-08-26-04:32:28-3432) Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-broker.service entered failed state. Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service failed. Aug 26 04:38:48 hvm002 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Aug 26 04:38:48 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:48 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:38:48 hvm002 systemd: Started oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:38:48 hvm002 systemd: Started Session c17 of user root. Aug 26 04:38:48 hvm002 systemd: Started Session c18 of user root. Aug 26 04:38:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:57 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:57 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:57 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:57 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:58 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:39:08 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:39:08 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:39:08 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:39:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:39:08 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service failed. [root@hvm002 ~]# --------------------- Erick Perez ---------------------

1 2

Error while trying to import Xenserver OVA to Ovirt 4.3.5
by Erick Perez - Quadrian Enterprises 01 Sep '19

01 Sep '19

Hi there, I have a Xen Server 6.2 that we want to migrate its VMs to ovrt 4.3.5 on Centos.hosts. I exported the VMs from Xen to an OVA file and placed the OVA file in one of the Ovirt nodes. While using the import OVA web interface I get this error: VDSM hvm001.unachi.ac.pa command GetOvaInfoVDS failed: Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.getExternalVmFromOva of <vdsm.API.Global object at 0x7f558c050d10>> with arguments: (u'/opt/UNACHI VMs2.ova',) error: not all arguments converted during string formatting"} What will the correct procedure be? thanks, --------------------- Erick Perez ---------------------

1 0

4.3.5 VM Guest filesystem corruption while deploying from template and root owned files problem.
by levin＠mydream.com.hk 31 Aug '19

31 Aug '19

Just upgraded from 4.2.7 to 4.3.5 with preserving 4.2 compatibility in cluster level. Everything is smooth and keep up and running over two weeks. However when we just want to deploy from existing Centos Linux VM Template, the VM shown "not a correct XFS inode." "file /grub2/i386-pc/normal.mod not found" error. Also tried another RHEL and OEL template, they all have same problem. So I tried to install a new Linux and converting to template to see whether it is existing disk problem, but it appear the same xfs error after template deployment. Have been tried on qcow2 and raw format. no different. The interesting finding is Windows Server Template doesn't suffer this problem. So is it from image tools bug? Secondly, I noticed there are intermittent issue where ovirt node create image objects owned by root user, and that cause VM fail to start. It happens likely when creating VM image snapshot and after restore snapshot. Anyone can point me out where should I looks into. Thank you!

1 0

Re: VMs inaccessible from Ubuntu/Debian-based OS
by Strahil 30 Aug '19

30 Aug '19

You need virt-viewer installed on your ubuntu or change console type to spice. Best Regards, Strahil NikolovOn Aug 30, 2019 15:35, souvaliotimaria(a)mail.com wrote: > > Hello all, > > I'm having an issue the past couple of days. I have tried anything I could find to solve this but with no success. > > I have a three node ovirt installation, glustered, hyperconverged and there i have a few VMs. The installation is for experimental reasons before we merge it in our DC. Anyway, though I can connect to the VMs' console from Fedora and Windows, I can't from Ubuntu. > > I have tried installing the browser-spice-plugin that's the corresponding package to spice-xpi and I got no results. > > I purged the browser-spice-plugin and then I installed the spice-client package, downloaded the spice-xpi from Fedora (FC19) (as instructed in https://www.ovirt.org/develop/infra/testing/spice.html) copied the libnsISpicec.so to /usr/lib/mozilla/plugins and made sure that xserver-xorg-video-qxl and spice-vdagent were installed and in their latest version, and still nothing. > > No matter what I have tried, I can't gain access to the console. No matter the browser I use, the message I get is "Unable to connect to the graphic server /tmp/mozilla_msouval0/console-1.vv". > > I have checked the logs and couldn't find anything useful, maybe I'm not checking the right logs? > I run tcpdump on both the node the VM is being hosted and the Ubuntu machine I'm using and though on the UBuntu I capture a few packets (6) on both sides but on the Ubuntu side there were 15 packets received by the filter. > > Could you please guide towards the right way to solve this? Could this be an ntp problem? > > ovirt node version: 4.2.3.1 > Workstation: Jessie > > (I thought maybe Jessie was too old, and I run the same steps from Ubuntu 14.04 and Ubuntu 16.04, but the problem still remains) > > Thank you in advance for any help. > Maria > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4VMMYEJNSHGUH…

2 1

VMs inaccessible from Ubuntu/Debian-based OS
by souvaliotimaria＠mail.com 30 Aug '19

30 Aug '19

Hello all, I'm having an issue the past couple of days. I have tried anything I could find to solve this but with no success. I have a three node ovirt installation, glustered, hyperconverged and there i have a few VMs. The installation is for experimental reasons before we merge it in our DC. Anyway, though I can connect to the VMs' console from Fedora and Windows, I can't from Ubuntu. I have tried installing the browser-spice-plugin that's the corresponding package to spice-xpi and I got no results. I purged the browser-spice-plugin and then I installed the spice-client package, downloaded the spice-xpi from Fedora (FC19) (as instructed in https://www.ovirt.org/develop/infra/testing/spice.html) copied the libnsISpicec.so to /usr/lib/mozilla/plugins and made sure that xserver-xorg-video-qxl and spice-vdagent were installed and in their latest version, and still nothing. No matter what I have tried, I can't gain access to the console. No matter the browser I use, the message I get is "Unable to connect to the graphic server /tmp/mozilla_msouval0/console-1.vv". I have checked the logs and couldn't find anything useful, maybe I'm not checking the right logs? I run tcpdump on both the node the VM is being hosted and the Ubuntu machine I'm using and though on the UBuntu I capture a few packets (6) on both sides but on the Ubuntu side there were 15 packets received by the filter. Could you please guide towards the right way to solve this? Could this be an ntp problem? ovirt node version: 4.2.3.1 Workstation: Jessie (I thought maybe Jessie was too old, and I run the same steps from Ubuntu 14.04 and Ubuntu 16.04, but the problem still remains) Thank you in advance for any help. Maria

1 0

Hosted-engine inaccessible
by Tau Makgaile 29 Aug '19

29 Aug '19

Hi, I am currently experiencing a problem with my Hosted-engine. Hosted-engine disconnected after increasing / partition. The increase went well but after some time the hosted-enigine VM disconnected and has since been giving alerts such as* re-initializingFSM*. Though VMs underneth are running, Hosted-engine --vm-status: *"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail"* There is no backup to restore at the moment. I am looking for a way to bring it up without redeploying the hosted engine. Thanks in advance for your help. Kind regards, Tau

2 1

python sdk create a windows machine with initial run sysprep
by Harry Conings 29 Aug '19

29 Aug '19

Hi i am trying to create virtual machines from a template but i cannot seem to find how to ad the inital run sysprep option how should i give that option ? my code <code> NEW_VM_NAME = 'W10MYMACINE' TEMPLATE_NAME = 'W10TEMP' CLUSTER_NAME= 'MYCLUSTER' try: api.vms.add(params.VM(name=NEW_VM_NAME, cluster=api.clusters.get(CLUSTER_NAME), template=api.templates.get(TEMPLATE_NAME))) print 'VM was created from Template successfully' print 'Waiting for VM to reach Down status' while api.vms.get(NEW_VM_NAME).status.state != 'down': sleep(1) except Exception as e: print 'Failed to create VM from Template:\n%s' % str(e) </code>

2 3

Error preparing image
by Paul Wright 29 Aug '19

29 Aug '19

Hi, I have just taken over an Ovirt cluster running 4.1.2.2, it comprises of three hosts of which, two can run the HE. This message floods /var/log/messages on the hosts that can run the HE. Aug 28 17:39:56 ovirt-03 journal: ovirt-ha-agent ovirt_hosted_engine_ha.lib.image.Image ERROR Error preparing image - storagepoolID: 00000000-0000-0000-0000-000000000000 - storagedomainID: 0ea4f86e-3d20-422f-baf0-bb0f6ca3d0b9 - imageID: 3019b1a4-4776-42d9-a3d3-43930acfe681 - volumeID: 28ebf041-7222-4954-af2d-0889cb5f0cd0: Cannot prepare illegal volume: (u'28ebf041-7222-4954-af2d-0889cb5f0cd0',) What does it mean? Many thanks, Paul -- <http://www.freefolk.com/> See our latest work <http://freefolk.com/advertising> Freefolk 140 Wardour Street, London W1F 8ZT freefolk.com <http://www.freefolk.com/> / +44 (0)20 7734 9110 @freefolkstudios <https://www.twitter.com/freefolkstudios> / facebook/freefolkstudios <https://www.facebook.com/freefolkstudios> / instagram/freefolkstudios <https://www.instagram.com/freefolkstudios> / linkedin <https://www.linkedin.com/company/finish> <http://freefolk.com/film-and-tv/>

1 0

[ANN] oVirt 4.3.6 Fourth Release Candidate is now available for testing
by Sandro Bonazzola 29 Aug '19

29 Aug '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.6 Fourth Release Candidate for testing, as of August 29th, 2019. This update is a release candidate of the sixth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) will be made available when CentOS 7.7 will be released. See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is not yet available, pending CentOS 7.7 release to be available Additional Resources: * Read more about the oVirt 4.3.6 release highlights: http://www.ovirt.org/release/4.3.6/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.6/ [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: Installing new oVirt node fails
by Lucie Leistnerova 29 Aug '19

29 Aug '19

Hi, I'm sorry for my late response. On 8/13/19 12:21 PM, Winfried de Heiden wrote: > I guess installation will fail, most likeley beacause of some > "strange" characters in the ssh pub-keys > Installation is incomplete and therfor vdsm will start complaining. > > In the past, I was able to reproduce oVirt installation(issues) using > nested kvm. For some reason the old method > (https://ovirt.org/develop/developer-guide/vdsm/developers.html) does > notwork anymore. No more nested kvm anymore? I'm prety shure my > installtion issues are ssh-pubkey related, but I need a virtual > hypervisor to reproduce... > I'm QE and I never installed vdsm other than from provided rpms. So I can't help with that guide. I can just check the deploy logs from the engine, whether I can get more information what was wrong. > Winfried > > > Op 12-08-19 om 13:19 schreef Lucie Leistnerova: >> On 8/12/19 10:34 AM, Winfried de Heiden wrote: >>> Hi all, >>> >>> No, the certificate was OK, no UTF characters found. I think I found >>> the problem however; the public keys as found in >>> /root/.ssh/authorized_keys. >>> >>> One or two keys did contain a "@" in the description. After removing >>> these keys, installation went well. I'll try to reproduce using a >>> virtual hypervisor, smells like a bug. >>> >> That sounds strange, @ is 0x40. Otopi complains about 0x3c, that >> characters start with À. >> Please send the host deploy log. >> >> >>> Winfried >>> >>> Op 12-08-19 om 07:59 schreef Lucie Leistnerova: >>>> >>>> Hi Winfried, >>>> >>>> On 8/11/19 7:20 PM, Winfried de Heiden wrote: >>>>> Hi all, >>>>> >>>>> Installing/regestering a host to ovirt (ovirt: 4.3.5.5-1.el7 + >>>>> oVirt Node 4.3.5.2)) will fail with, seems lang something using >>>>> yum goes wrong. Installing will fail... >>>>> >>>> You may hit this issue >>>> >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1733438 >>>> >>>> Check your certificates for utf characters. >>>> >>>>> 019-08-11 18:40:44,408+0200 DEBUG otopi.plugins.otopi.dialog.machine dialog.__logString:204 DIALOG:SEND **%EventStart STAGE misc METHOD otopi.plugins.otopi.network.ssh.Plugin._append_key (None) >>>>> 2019-08-11 18:40:44,409+0200 DEBUG otopi.context context._executeMethod:145 method exception >>>>> Traceback (most recent call last): >>>>> File "/tmp/ovirt-sMdZ7tvESy/pythonlib/otopi/context.py", line 132, in _executeMethod >>>>> method['method']() >>>>> File "/tmp/ovirt-sMdZ7tvESy/otopi-plugins/otopi/network/ssh.py", line 152, in _append_key >>>>> constants.CoreEnv.MODIFIED_FILES >>>>> File "/tmp/ovirt-sMdZ7tvESy/pythonlib/otopi/filetransaction.py", line 151, in __init__ >>>>> self._content = self._content.encode("utf-8") >>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 750: ordinal not in range(128) >>>>> 2019-08-11 18:40:44,409+0200ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Misc configuration': 'ascii' codec can't decode byte 0xc3 in position 750: ordinal not in range(128) >>>>> 2019-08-11 18:40:44,410+0200 DEBUG otopi.transaction transaction.abort:119 aborting 'Yum Transaction' >>>>> 2019-08-11 18:40:44,410+0200 INFO otopi.plugins.otopi.packagers.yumpackager yumpackager.info:80 Yum Performing yum transaction rollback >>>>> Loaded plugins: fastestmirror, product-id, subscription-manager, versionlock >>>>> >>>>> Afterwards, oVirt Node will complain: >>>>> >>>>> aug 11 19:13:32 bigvirt.blabla.bla vdsm[2342]: ERROR ssl handshake: SSLError, address: ::ffff:10.19.69.3 >>>>> aug 11 19:13:55 bigvirt.blabla.bla vdsm[2342]: ERROR ssl handshake: SSLError, address: ::ffff:10.19.69.3 >>>>> aug 11 19:14:18 bigvirt.blabla.bla vdsm[2342]: ERROR ssl handshake: SSLError, address: ::ffff:10.19.69.3 >>>>> aug 11 19:14:41 bigvirt.blabla.bla vdsm[2342]: ERROR ssl handshake: SSLError, address: ::ffff:10.19.69.3 >>>>> >>>>> Anyone? >>>>> >>>>> Winfried >>>>> >>>>> _______________________________________________ >>>>> Users mailing list --users(a)ovirt.org >>>>> To unsubscribe send an email tousers-leave(a)ovirt.org >>>>> Privacy Statement:https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/2A4Q… >>>> Best regards, >>>> >>>> Lucie >>>> >>>> -- >>>> Lucie Leistnerova >>>> Senior Quality Engineer, QE Cloud, RHVM >>>> Red Hat EMEA >>>> >>>> IRC: lleistne @ #rhev-qe >>> >> -- >> Lucie Leistnerova >> Senior Quality Engineer, QE Cloud, RHVM >> Red Hat EMEA >> >> IRC: lleistne @ #rhev-qe > -- Lucie Leistnerova Senior Quality Engineer, QE Cloud, RHVM Red Hat EMEA IRC: lleistne @ #rhev-qe

2 1

ovirt api snapshot date format?
by Bill James 29 Aug '19

29 Aug '19

I'm trying to use the ovirt REST api to read snapshot date but the number I'm getting back doesn't appear to be in epoch time. What format is it? Example: "date":1565299725928,"snapshot_status":"ok" Should be "Aug 8, 2019", epoch conversion put it at Tue May 16 16:25:28 51572. (accessed via https://$HOSTNAME/ovirt-engine/api/vms/$value/snapshots)

2 1

Removing Unknown VM's on Dead Cluster
by Ross, Travis 28 Aug '19

28 Aug '19

I've clearly seen this is a pretty common issue with many different methods for cleaning up dead clusters. I'm having issues with unknown VM's not allowing my host into Maintenance Mode. Version: ovirt-release36-3.6.7-1.noarch Situation: Another site repurposed a host entirely without properly cleaning it's status in my Engine. No option to delete host. I've attempted the "Confirm Host is Rebooted" -> Maintenance Mode. However the error is: Cannot Switch the following Hosts to Maintenance mode: ... One or more running VMs are indicated as non-migratable. The status of all the listed VM's is "UNKNOWN" w/ "?" next to them. If I look at the settings for these VM's they already have "Do not allow migration" checked. Also I'm unable to modify the VM since the Host doesn't exist. Can't just delete the Domain because the Host exists. Can't delete Storage because host is not in Maintenance Mode. Can't set Maintenance Mode because VM's can't migrate. At this point all I want to do is upgrade to latest ovirt, however I need to get this orphaned Domain out of my single node cluster, which still runs a ~10 VM's hosted on Gluster Storage. Thoughts on whether I could just backup these VM's, re-install ovirt fresh then import the VM's? Thanks, Travis Ross W 512.231.4235 M 512.547.9437 [cid:image003.png@01D55D96.2A607620] KLA-Tencor is now KLA! Learn more at kla.com <https://www.kla.com/>

1 0

Update single node environment from 4.3.3 to 4.3.5 problem
by Gianluca Cecchi 28 Aug '19

28 Aug '19

Hello, after updating hosted engine from 4.3.3 to 4.3.5 and then the only host composing the environment (plain CentOS 7.6) it seems it is not able to start vdsm daemons kernel installed with update is kernel-3.10.0-957.27.2.el7.x86_64 Same problem also if using previous running kernel 3.10.0-957.12.2.el7.x86_64 [root@ovirt01 vdsm]# uptime 00:50:08 up 25 min, 3 users, load average: 0.60, 0.67, 0.60 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# systemctl status vdsmd -l ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/etc/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit) since Fri 2019-08-23 00:37:27 CEST; 7s ago Process: 25810 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=1/FAILURE) Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered failed state. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service holdoff time over, scheduling restart. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Stopped Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: start request repeated too quickly for vdsmd.service Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Failed to start Virtual Desktop Server Manager. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: Unit vdsmd.service entered failed state. Aug 23 00:37:27 ovirt01.mydomain systemd[1]: vdsmd.service failed. [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# pwd /var/log/vdsm [root@ovirt01 vdsm]# ll -t | head total 118972 -rw-r--r--. 1 root root 3406465 Aug 23 00:25 supervdsm.log -rw-r--r--. 1 root root 73621 Aug 23 00:25 upgrade.log -rw-r--r--. 1 vdsm kvm 0 Aug 23 00:01 vdsm.log -rw-r--r--. 1 vdsm kvm 538480 Aug 22 23:46 vdsm.log.1.xz -rw-r--r--. 1 vdsm kvm 187486 Aug 22 23:46 mom.log -rw-r--r--. 1 vdsm kvm 621320 Aug 22 22:01 vdsm.log.2.xz -rw-r--r--. 1 root root 374464 Aug 22 22:00 supervdsm.log.1.xz -rw-r--r--. 1 vdsm kvm 2097122 Aug 22 21:53 mom.log.1 -rw-r--r--. 1 vdsm kvm 636212 Aug 22 20:01 vdsm.log.3.xz [root@ovirt01 vdsm]# link to upgrade.log contents here: https://drive.google.com/file/d/17jtX36oH1hlbNUAiVhdBkVDbd28QegXG/view?usp=… link to supervdsm.log (in gzip format) here: https://drive.google.com/file/d/1l61ePU-eFS_xVHEAHnJthzTTnTyzu0MP/view?usp=… It seems that since update I get these kind of lines inside it... restore-net::DEBUG::2019-08-22 23:56:38,591::cmdutils::133::root::(exec_cmd) /sbin/tc filter del dev eth0 pref 5000 (cwd None) restore-net::DEBUG::2019-08-22 23:56:38,595::cmdutils::141::root::(exec_cmd) FAILED: <err> = 'RTNETLINK answers: Invalid argument\nWe have an error talking to the kernel\n'; <rc> = 2 [root@ovirt01 vdsm]# systemctl status supervdsmd -l ● supervdsmd.service - Auxiliary vdsm service for running helper functions as root Loaded: loaded (/usr/lib/systemd/system/supervdsmd.service; static; vendor preset: enabled) Active: active (running) since Fri 2019-08-23 00:25:17 CEST; 23min ago Main PID: 4540 (supervdsmd) Tasks: 3 CGroup: /system.slice/supervdsmd.service └─4540 /usr/bin/python2 /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock Aug 23 00:25:17 ovirt01.mydomain systemd[1]: Started Auxiliary vdsm service for running helper functions as root. [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master ovirtmgmt state UP group default qlen 1000 link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff 3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 00:c2:c6:a4:18:c5 brd ff:ff:ff:ff:ff:ff 4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 36:21:c1:5e:70:aa brd ff:ff:ff:ff:ff:ff 5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 46:d8:db:81:41:4e brd ff:ff:ff:ff:ff:ff 22: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether b8:ae:ed:7f:17:11 brd ff:ff:ff:ff:ff:ff inet 192.168.1.211/24 brd 192.168.1.255 scope global ovirtmgmt valid_lft forever preferred_lft forever [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# ip route show default via 192.168.1.1 dev ovirtmgmt 192.168.1.0/24 dev ovirtmgmt proto kernel scope link src 192.168.1.211 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# brctl show bridge name bridge id STP enabled interfaces ovirtmgmt 8000.b8aeed7f1711 no eth0 [root@ovirt01 vdsm]# [root@ovirt01 vdsm]# systemctl status openvswitch ● openvswitch.service - Open vSwitch Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled; vendor preset: disabled) Active: active (exited) since Fri 2019-08-23 00:25:09 CEST; 26min ago Process: 3894 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 3894 (code=exited, status=0/SUCCESS) Tasks: 0 CGroup: /system.slice/openvswitch.service Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Starting Open vSwitch... Aug 23 00:25:09 ovirt01.mydomain systemd[1]: Started Open vSwitch. [root@ovirt01 vdsm]# ovs-vsctl show 02539902-1788-4796-9cdf-cf11ce8436bb Bridge br-int fail_mode: secure Port br-int Interface br-int type: internal ovs_version: "2.11.0" [root@ovirt01 vdsm]# any hints? Thanks Gianluca

4 16

Evaluate oVirt to remove VMware
by dsalade＠gmail.com 28 Aug '19

28 Aug '19

Howdy All!!! Somewhat new user to Linux and having some issues with oVirt node networking. We are mainly a Windows/VMware medium shop but have a pretty good network. Our VMware setup has redundant switches using multiple VLANs. This was fairly easy to setup with VMware. We never had an issue with VMware, add 2 NICs to the ESXi host and then adding VLANs into the Virtual Switch. Biggest issue I am having, should I be bonding, teaming and VLANing on top of that? Tried setting up node with Engine inside which fails to deploy because of networking. Then I tried a standalone Engine and could not add node to it - I believe networking as well as Power Management agent was the problem. Would like the nodes/engine to be on one subnet (lets say 10.10.11.x) and VMs to be allowed to run there are well as 10 other VLANs. I guess welcome to the wonderful world of Linux networking and learning some new technologies is in store for me. I hope I am not being vague, like I said fairly new to Linux. Thanks for any responses! Allen

4 6

VDSM Hooks during migration
by Vrgotic, Marko 27 Aug '19

27 Aug '19

Dear oVIrt, Would you be so kind to help me/tell me or point me how to find which Hooks, and in which order, are triggered when VM is being migrated? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic

2 6

Re: VM --- is not responding.
by Strahil 26 Aug '19

26 Aug '19

Hm... It was supposes to show controller status. Maybe the hpssacli you have is not supporting your raid cards.Check for newer version on HPE's support page. Best Regards, Strahil Nikolov Best Regards, Strahil NikolovOn Aug 14, 2019 11:40, Edoardo Mazza <edo7411(a)gmail.com> wrote: > > I installed hpssacli-2.40-13.0.x86_64.rpm and the result of "hpssacli ctrl all show status" is: > Error: No controllers detected. Possible causes:..... > The s.o. run on sd cards and the vm runs on array on traditional disk > thanks > Edoardo > > Il giorno lun 12 ago 2019 alle ore 05:59 Strahil <hunter86_bg(a)yahoo.com> ha scritto: >> >> Would you check the health status of the controllers : >> hpssacli ctrl all show status >> >> Best Regards, >> Strahil Nikolov >> >> On Aug 11, 2019 09:55, Edoardo Mazza <edo7411(a)gmail.com> wrote: >>> >>> The hosts are 3 ProLiant DL380 Gen10, 2 hosts with HPE Smart Array P816i-a SR Gen10 like controller and the other host with >>> HPE Smart Array P408i-a SR Gen10. The storage for ovirt enviroment is gluster and the last host is the arbiter in the gluster enviroment. >>> The S.M.A.R.T. healt status is ok for all host >>> Edoardo >>> >>> >>> >>> >>> >>> Il giorno gio 8 ago 2019 alle ore 16:19 Sandro Bonazzola <sbonazzo(a)redhat.com> ha scritto: >>>> >>>> >>>> >>>> Il giorno gio 8 ago 2019 alle ore 11:19 Edoardo Mazza <edo7411(a)gmail.com> ha scritto: >>>>> >>>>> Hi all, >>>>> It is more days that for same vm I received this error, but I don't underdand why. >>>>> The traffic of the virtual machine is not excessive, cpu and ram to, but for few minutes the vm is not responding. and in the messages log file of the vm I received the error under, yo can help me? >>>>> thanks >>>> >>>> >>>> can you check the S.M.A.R.T. health status of the disks? >>>> >>>> >>>>> >>>>> Edoardo >>>>> kernel: NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [kworker/2:0:26227] >>>>> Aug 8 02:51:11 vmmysql kernel: Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ >>>>> ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_ >>>>> nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_con >>>>> ntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_c >>>>> odec_generic iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel snd_hda_intel snd_hda_codec aesni_intel snd_hda_core lrw gf128mul >>>>> glue_helper ablk_helper snd_hwdep cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore virtio_rng sg virtio_balloon >>>>> i2c_piix4 parport_pc parport joydev pcspkr ip_tables xfs libcrc32c sd_mod >>>>> Aug 8 02:51:14 vmmysql kernel: crc_t10dif crct10dif_generic sr_mod cdrom virtio_net virtio_console virtio_scsi ata_generic p >>>>> ata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw qxl floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod >>>>> Aug 8 02:51:14 vmmysql kernel: CPU: 2 PID: 26227 Comm: kworker/2:0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.12.1.el7.x86_64 #1 >>>>> Aug 8 02:51:14 vmmysql kernel: Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014 >>>>> Aug 8 02:51:14 vmmysql kernel: Workqueue: events_freezable disk_events_workfn >>>>> Aug 8 02:51:14 vmmysql kernel: task: ffff9e25b6609040 ti: ffff9e27b1610000 task.ti: ffff9e27b1610000 >>>>> Aug 8 02:51:14 vmmysql kernel: RIP: 00

2 1

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

It seems that according to https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.0/ht… the ports of interest are: 16514/TCP 49152 - 49216/TCP Best Regards, Strahil NikolovOn Aug 25, 2019 08:48, Strahil <hunter86_bg(a)yahoo.com> wrote: > > Curtis, > > Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? > > Best Regards, > Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > > > Creating a small VM from a Cinder template and running it gave me a test VM. > > > > When I migrated from swm-01 to swm-02, swm-01 immediately became > > unresponsive to pings, SSH'es, and to the ovirt interface which marked > > it as "NonResponsive" soon after the VM finished. The VM did finish > > migrating, however I'm unsure if that's a good migration or not. > > > > Thank you, Strahil. > > > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > > > What is your bandwidth threshold for the network used for VM migration ? > > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > > > Do you see disconnects ? > > > > > > If no, try a little bit up (the threshold) and check again. > > > > > > Best Regards, > > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > > > It took a while for my servers to come back on the network this time. > > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > > > So I disabled and stopped the lldpad service. > > > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > > again. No ping, SSH hung, same as before - almost as soon as the > > > > migration started. > > > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > > requesting the port configurations. I've already added the port > > > > numbers and switch name that I got from CDP. > > > > > > > > Thanks again, I really appreciate the help! > > > > cecjr > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > >> > > > > >> > > > > >> > > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > > >>> > > > > >>> This little cluster isn't in production or anything like that yet. > > > > >>> > > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > > >>> migrate around at random. > > > > >>> > > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > > >>> > > > > >>> Any other ideas? > > > > >>> > > > > >> > > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > > >> This is a strong indication that the root cause is related to the switch. > > > > >> Maybe you can get some information about the switch config by > > > > >> 'lldptool get-tlv -n -i em1' > > > > >> > > > > > > > > > > Another guess: > > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > > 'systemctl stop lldpad' > > > > > another try to migrate. > > > > > > > > > > > > > > >> > > > > >> > > > > >>> > > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > > >>> >> > > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > > >>> >> response and usually the response doesn't help. Then

2 1

attach untagged vlan internally on vm
by ernestclydeachua＠gmail.com 25 Aug '19

25 Aug '19

good day currently i am testing oVirt on a single box and setup some tagged vms and non tagged vm. the non tagged vm is a firewall but it has limitations on the number of nic so i cannot attach tagged vnic and wish to handdle vlan tagging on it is it possible to pass untaged franes internally?

6 7

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

Curtis, Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? Best Regards, Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > Creating a small VM from a Cinder template and running it gave me a test VM. > > When I migrated from swm-01 to swm-02, swm-01 immediately became > unresponsive to pings, SSH'es, and to the ovirt interface which marked > it as "NonResponsive" soon after the VM finished. The VM did finish > migrating, however I'm unsure if that's a good migration or not. > > Thank you, Strahil. > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > What is your bandwidth threshold for the network used for VM migration ? > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > Do you see disconnects ? > > > > If no, try a little bit up (the threshold) and check again. > > > > Best Regards, > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > It took a while for my servers to come back on the network this time. > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > So I disabled and stopped the lldpad service. > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > again. No ping, SSH hung, same as before - almost as soon as the > > > migration started. > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > requesting the port configurations. I've already added the port > > > numbers and switch name that I got from CDP. > > > > > > Thanks again, I really appreciate the help! > > > cecjr > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >> > > > >> > > > >> > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> > > > >>> This little cluster isn't in production or anything like that yet. > > > >>> > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > >>> migrate around at random. > > > >>> > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > >>> > > > >>> Any other ideas? > > > >>> > > > >> > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > >> This is a strong indication that the root cause is related to the switch. > > > >> Maybe you can get some information about the switch config by > > > >> 'lldptool get-tlv -n -i em1' > > > >> > > > > > > > > Another guess: > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > 'systemctl stop lldpad' > > > > another try to migrate. > > > > > > > > > > > >> > > > >> > > > >>> > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >>> > > > > >>> > > > > >>> > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> >> > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > >>> >> response and usually the response doesn't help. Then

1 0

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

Curtis, Do you have enough space to run tcpdump (port not 22) on both hosts and on the small VM you have done previously - and then start the migration? Best Regards, Strahil NikolovOn Aug 24, 2019 22:15, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > I applied a 90Mbs QOS Rate Limit with 10 set for the shares to both > interfaces of 2 of the hosts. My hosts names are swm-01 and swm-02. > > Creating a small VM from a Cinder template and running it gave me a test VM. > > When I migrated from swm-01 to swm-02, swm-01 immediately became > unresponsive to pings, SSH'es, and to the ovirt interface which marked > it as "NonResponsive" soon after the VM finished. The VM did finish > migrating, however I'm unsure if that's a good migration or not. > > Thank you, Strahil. > > On Sat, Aug 24, 2019 at 12:39 PM Strahil <hunter86_bg(a)yahoo.com> wrote: > > > > What is your bandwidth threshold for the network used for VM migration ? > > Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? > > > > Do you see disconnects ? > > > > If no, try a little bit up (the threshold) and check again. > > > > Best Regards, > > Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > > > > > It took a while for my servers to come back on the network this time. > > > I think it's due to ovirt continuing to try to migrate the VMs around > > > like I requested. The 3 servers' names are "swm-01, swm-02 and > > > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > > > > > So I disabled and stopped the lldpad service. > > > > > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > > > again. No ping, SSH hung, same as before - almost as soon as the > > > migration started. > > > > > > If you wall have any ideas what switch-level setting might be enabled, > > > let me know, cause I'm stumped. I can add it to the ticket that's > > > requesting the port configurations. I've already added the port > > > numbers and switch name that I got from CDP. > > > > > > Thanks again, I really appreciate the help! > > > cecjr > > > > > > > > > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >> > > > >> > > > >> > > > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> > > > >>> This little cluster isn't in production or anything like that yet. > > > >>> > > > >>> So, I went ahead and used your ethtool commands to disable pause > > > >>> frames on both interfaces of each server. I then, chose a few VMs to > > > >>> migrate around at random. > > > >>> > > > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > > > >>> ssh, and the SSH session that I had open was unresponsive. > > > >>> > > > >>> Any other ideas? > > > >>> > > > >> > > > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > > > >> This is a strong indication that the root cause is related to the switch. > > > >> Maybe you can get some information about the switch config by > > > >> 'lldptool get-tlv -n -i em1' > > > >> > > > > > > > > Another guess: > > > > After the optional 'lldptool get-tlv -n -i em1' > > > > 'systemctl stop lldpad' > > > > another try to migrate. > > > > > > > > > > > >> > > > >> > > > >>> > > > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > >>> > > > > >>> > > > > >>> > > > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > > > >>> >> > > > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > > > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > > > >>> >> the port configuration. They just get ignored - but that's par for the > > > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > > > >>> >> response and usually the response doesn't help. Then

1 0

Re: Need to enable STP on ovirt bridges
by Strahil 25 Aug '19

25 Aug '19

What is your bandwidth threshold for the network used for VM migration ? Can you set a 90 mbit/s threshold (yes, less than 100mbit/s) and try to migrate a small (1 GB RAM) VM ? Do you see disconnects ? If no, try a little bit up (the threshold) and check again. Best Regards, Strahil NikolovOn Aug 23, 2019 23:19, "Curtis E. Combs Jr." <ej.albany(a)gmail.com> wrote: > > It took a while for my servers to come back on the network this time. > I think it's due to ovirt continuing to try to migrate the VMs around > like I requested. The 3 servers' names are "swm-01, swm-02 and > swm-03". Eventually (about 2-3 minutes ago) they all came back online. > > So I disabled and stopped the lldpad service. > > Nope. Started some more migrations and swm-02 and swm-03 disappeared > again. No ping, SSH hung, same as before - almost as soon as the > migration started. > > If you wall have any ideas what switch-level setting might be enabled, > let me know, cause I'm stumped. I can add it to the ticket that's > requesting the port configurations. I've already added the port > numbers and switch name that I got from CDP. > > Thanks again, I really appreciate the help! > cecjr > > > > On Fri, Aug 23, 2019 at 3:28 PM Dominik Holler <dholler(a)redhat.com> wrote: > > > > > > > > On Fri, Aug 23, 2019 at 9:19 PM Dominik Holler <dholler(a)redhat.com> wrote: > >> > >> > >> > >> On Fri, Aug 23, 2019 at 8:03 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> > >>> This little cluster isn't in production or anything like that yet. > >>> > >>> So, I went ahead and used your ethtool commands to disable pause > >>> frames on both interfaces of each server. I then, chose a few VMs to > >>> migrate around at random. > >>> > >>> swm-02 and swm-03 both went out again. Unreachable. Can't ping, can't > >>> ssh, and the SSH session that I had open was unresponsive. > >>> > >>> Any other ideas? > >>> > >> > >> Sorry, no. Looks like two different NICs with different drivers and frimware goes down together. > >> This is a strong indication that the root cause is related to the switch. > >> Maybe you can get some information about the switch config by > >> 'lldptool get-tlv -n -i em1' > >> > > > > Another guess: > > After the optional 'lldptool get-tlv -n -i em1' > > 'systemctl stop lldpad' > > another try to migrate. > > > > > >> > >> > >>> > >>> On Fri, Aug 23, 2019 at 1:50 PM Dominik Holler <dholler(a)redhat.com> wrote: > >>> > > >>> > > >>> > > >>> > On Fri, Aug 23, 2019 at 6:45 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> >> > >>> >> Unfortunately, I can't check on the switch. Trust me, I've tried. > >>> >> These servers are in a Co-Lo and I've put 5 tickets in asking about > >>> >> the port configuration. They just get ignored - but that's par for the > >>> >> coarse for IT here. Only about 2 out of 10 of our tickets get any > >>> >> response and usually the response doesn't help. Then the system they > >>> >> use auto-closes the ticket. That was why I was suspecting STP before. > >>> >> > >>> >> I can do ethtool. I do have root on these servers, though. Are you > >>> >> trying to get me to turn off link-speed auto-negotiation? Would you > >>> >> like me to try that? > >>> >> > >>> > > >>> > It is just a suspicion, that the reason is pause frames. > >>> > Let's start on a NIC which is not used for ovirtmgmt, I guess em1. > >>> > Does 'ethtool -S em1 | grep pause' show something? > >>> > Does 'ethtool em1 | grep pause' indicates support for pause? > >>> > The current config is shown by 'ethtool -a em1'. > >>> > '-A autoneg' "Specifies whether pause autonegotiation should be enabled." according to ethtool doc. > >>> > Assuming flow control is enabled by default, I would try to disable it via > >>> > 'ethtool -A em1 autoneg off rx off tx off' > >>> > and check if it is applied via > >>> > 'ethtool -a em1' > >>> > and check if the behavior under load changes. > >>> > > >>> > > >>> > > >>> >> > >>> >> On Fri, Aug 23, 2019 at 12:24 PM Dominik Holler <dholler(a)redhat.com> wrote: > >>> >> > > >>> >> > > >>> >> > > >>> >> > On Fri, Aug 23, 2019 at 5:49 PM Curtis E. Combs Jr. <ej.albany(a)gmail.com> wrote: > >>> >> >> > >>> >> >> Sure! Right now, I only have a 5

3 2

Moving ovirt engine disk to another storage volume
by Erick Perez - Quadrian Enterprises 25 Aug '19

25 Aug '19

Good morning, I am running Ovirt 4.3.5 in Centos 7.6 with one virt node an a NFS storage node. I did the self-hosted engine setup and i plan to add a second virt host in a few days. I need to do heavy maintenance on the storage node (VDO and mdadm things) and would like to know how (or a link to an article) can I move the ovirt engine disk to another storage. Currentl the NFS storage has two volumens (volA,volB) and the physical host have spare space too. Virtual machines are in VolB and the engine is in VolA. I would like to move the engine disk from VolA to VolB or to local storage. BTW I am not sure if I should say "move the engine" or should I say move "hosted_storage domain" domain thanks in advance. --------------------- Erick Perez

3 6

TASK [ovirt.hosted_engine_setup : Get local VM IP] Problem
by Emre Özkan 24 Aug '19

24 Aug '19

hi guys, ı waiting this part when I try to install with hosted-engine --deploy command.have you got any knowledge about that.My environment(hyper-v running nested vm on azure and rhev running on hyper-v) [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Get local VM IP]

1 0

numa pinning and reserved hugepages (1G) Bug in scheduler calculation or decision ?
by Ralf Schenk 24 Aug '19

24 Aug '19

Hello List, i ran into problems using numa-pinning and reserved hugepages. - My EPYC 7281 based Servers (Dual Socket) have 8 Numa-Nodes each having 32 GB of memory for a total of 256 GB System Memory - I'm using 192 x 1 GB hugepages reserved on the kernel cmdline default_hugepagesz=1G hugepagesz=1G hugepages=192 This reserves 24 hugepages on each numa-node. I wanted to pin a MariaDB VM using 32 GB (Custom Property hugepages=1048576) to numa-nodes 0-3 of CPU-Socket 1. Pinning in GUI etc. no problem. When trying to start the vm this can't be done since ovirt claims that the host can't fullfill the memory requirements - which is simply not correct since there were > 164 hugepages free. It should have taken 8 hugepages from each numa node 0-3 to fullfill the 32 GB Memory requirement. I also freed the system completely from other VM's but that didn't work either. Is it possible that the scheduler only takes into account the "free memory" (as seen in numactl -H below) *not reserved* by hugepages for its decisions ? Since the host has only < 8 GB of free mem per numa-node I can understand that VM was not able to start under that condition. VM is runnig and using 32 hugepages without pinning but a warning states "VM dbserver01b does not fit to a single NUMA node on host myhost.mydomain.de. This may negatively impact its performance. Consider using vNUMA and NUMA pinning for this VM." This is the numa Hardware Layout and hugepages usage now with other VM's running: from cat /proc/meminfo HugePages_Total: 192 HugePages_Free: 160 HugePages_Rsvd: 0 HugePages_Surp: 0 I can confirm that also under the condition of running other VM's there are at least 8 hugepages free for each numa-node 0-3: grep "" /sys/devices/system/node/*/hugepages/hugepages-1048576kB/free_hugepages /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/free_hugepages:8 /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/free_hugepages:23 /sys/devices/system/node/node2/hugepages/hugepages-1048576kB/free_hugepages:20 /sys/devices/system/node/node3/hugepages/hugepages-1048576kB/free_hugepages:22 /sys/devices/system/node/node4/hugepages/hugepages-1048576kB/free_hugepages:16 /sys/devices/system/node/node5/hugepages/hugepages-1048576kB/free_hugepages:5 /sys/devices/system/node/node6/hugepages/hugepages-1048576kB/free_hugepages:19 /sys/devices/system/node/node7/hugepages/hugepages-1048576kB/free_hugepages:24 numactl -h: available: 8 nodes (0-7) node 0 cpus: 0 1 2 3 32 33 34 35 node 0 size: 32673 MB node 0 free: 3779 MB node 1 cpus: 4 5 6 7 36 37 38 39 node 1 size: 32767 MB node 1 free: 6162 MB node 2 cpus: 8 9 10 11 40 41 42 43 node 2 size: 32767 MB node 2 free: 6698 MB node 3 cpus: 12 13 14 15 44 45 46 47 node 3 size: 32767 MB node 3 free: 1589 MB node 4 cpus: 16 17 18 19 48 49 50 51 node 4 size: 32767 MB node 4 free: 2630 MB node 5 cpus: 20 21 22 23 52 53 54 55 node 5 size: 32767 MB node 5 free: 2487 MB node 6 cpus: 24 25 26 27 56 57 58 59 node 6 size: 32767 MB node 6 free: 3279 MB node 7 cpus: 28 29 30 31 60 61 62 63 node 7 size: 32767 MB node 7 free: 5513 MB node distances: node 0 1 2 3 4 5 6 7 0: 10 16 16 16 32 32 32 32 1: 16 10 16 16 32 32 32 32 2: 16 16 10 16 32 32 32 32 3: 16 16 16 10 32 32 32 32 4: 32 32 32 32 10 16 16 16 5: 32 32 32 32 16 10 16 16 6: 32 32 32 32 16 16 10 16 7: 32 32 32 32 16 16 16 10 -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs(a)databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------

2 2

Need to enable STP on ovirt bridges
by ej.albany＠gmail.com 24 Aug '19

24 Aug '19

Hello. I have been trying to figure out an issue for a very long time. That issue relates to the ethernet and 10gb fc links that I have on my cluster being disabled any time a migration occurs. I believe this is because I need to have STP turned on in order to participate with the switch. However, there does not seem to be any way to tell oVirt to stop turning it off! Very frustrating. After entering a cronjob that enables stp on all bridges every 1 minute, the migration issue disappears.... Is there any way at all to do without this cronjob and set STP to be ON without having to resort to such a silly solution? Here are some details about my systems, if you need it. selinux is disabled. [root@swm-02 ~]# rpm -qa | grep ovirt ovirt-imageio-common-1.5.1-0.el7.x86_64 ovirt-release43-4.3.5.2-1.el7.noarch ovirt-imageio-daemon-1.5.1-0.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch ovirt-hosted-engine-setup-2.3.11-1.el7.noarch ovirt-ansible-hosted-engine-setup-1.0.26-1.el7.noarch python2-ovirt-host-deploy-1.8.0-1.el7.noarch ovirt-ansible-engine-setup-1.1.9-1.el7.noarch python2-ovirt-setup-lib-1.2.0-1.el7.noarch cockpit-machines-ovirt-195.1-1.el7.noarch ovirt-hosted-engine-ha-2.3.3-1.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch cockpit-ovirt-dashboard-0.13.5-1.el7.noarch ovirt-provider-ovn-driver-1.2.22-1.el7.noarch ovirt-host-deploy-common-1.8.0-1.el7.noarch ovirt-host-4.3.4-1.el7.x86_64 python-ovirt-engine-sdk4-4.3.2-2.el7.x86_64 ovirt-host-dependencies-4.3.4-1.el7.x86_64 ovirt-ansible-repositories-1.1.5-1.el7.noarch [root@swm-02 ~]# cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) [root@swm-02 ~]# uname -r 3.10.0-957.27.2.el7.x86_64 You have new mail in /var/spool/mail/root [root@swm-02 ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master test state UP group default qlen 1000 link/ether d4:ae:52:8d:50:48 brd ff:ff:ff:ff:ff:ff 3: em2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether d4:ae:52:8d:50:49 brd ff:ff:ff:ff:ff:ff 4: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UP group default qlen 1000 link/ether 90:e2:ba:1e:14:80 brd ff:ff:ff:ff:ff:ff 5: p1p2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 90:e2:ba:1e:14:81 brd ff:ff:ff:ff:ff:ff 6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether a2:b8:d6:e8:b3:d8 brd ff:ff:ff:ff:ff:ff 7: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 96:a0:c1:4a:45:4b brd ff:ff:ff:ff:ff:ff 25: test: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d4:ae:52:8d:50:48 brd ff:ff:ff:ff:ff:ff inet 10.15.11.21/24 brd 10.15.11.255 scope global test valid_lft forever preferred_lft forever 26: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 90:e2:ba:1e:14:80 brd ff:ff:ff:ff:ff:ff inet 10.15.28.31/24 brd 10.15.28.255 scope global ovirtmgmt valid_lft forever preferred_lft forever 27: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 62:e5:e5:07:99:eb brd ff:ff:ff:ff:ff:ff 29: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UNKNOWN group default qlen 1000 link/ether fe:6f:9c:95:00:02 brd ff:ff:ff:ff:ff:ff [root@swm-02 ~]# free -m total used free shared buff/cache available Mem: 64413 1873 61804 9 735 62062 Swap: 16383 0 16383 [root@swm-02 ~]# free -h total used free shared buff/cache available Mem: 62G 1.8G 60G 9.5M 735M 60G Swap: 15G 0B 15G [root@swm-02 ~]# ls ls lsb_release lshw lslocks lsmod lspci lssubsys lsusb.py lsattr lscgroup lsinitrd lslogins lsns lss16toppm lstopo-no-graphics lsblk lscpu lsipc lsmem lsof lsscsi lsusb [root@swm-02 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 44 Model name: Intel(R) Xeon(R) CPU X5672 @ 3.20GHz Stepping: 2 CPU MHz: 3192.064 BogoMIPS: 6384.12 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 12288K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat spec_ctrl intel_stibp flush_l1d [root@swm-02 ~]#

6 34

Migration from bare metal engine to hosted doesn't seem to work at all in 4.3.2
by Clinton Goudie-Nice 24 Aug '19

24 Aug '19

Hi all, I'm trying to migrate from an ovirt bare metal instance to the hosted appliance. The instructions here: https://www.ovirt.org/documentation/self-hosted/chap-Migrating_from_Bare_Me… are at best murky, and at worst quite wrong, and I'm looking for some help. The instructions say "You must answer No to the following question so that you can restore the BareMetal-Engine backup file on HostedEngine-VM before running engine-setup. Automatically execute engine-setup on the engine appliance on first boot (Yes, No)[Yes]? No" Makes complete sense, except this question is not asked at all. Some research on the internet says we should be able to do this via append-answers: [environment:default] OVEHOSTED_VM/cloudinitExecuteEngineSetup=bool:False Except this doesn't work either. When the appliance deploys the engine is setup and already running. How do I take a backup file, and a backup log, and deploy them into a hosted engine? Clint

1 0

Creating hosts via the REST API using SSH Public Key authentication
by schill-julian＠gmx.de 23 Aug '19

23 Aug '19

In the UI one can create hosts using two authentication methods: 'Password' and 'SSH Public Key'. I have only found the Password authentication in the API Docs (/ovirt-engine/apidoc/#/services/hosts/methods/add). My question is: How can i create hosts using SSH Public Key authentication via the REST API? I would appreciate an example POST request!

3 2

nodectl on plain CentOS hypervisors
by Gianluca Cecchi 23 Aug '19

23 Aug '19

Does it make sense to install nodectl utility on plain CentOS 7.x nodes? Or any other alternative for plain OS nodes vs ovirt-node-ng ones? On my updated CentOS 7.6 oVirt node I have not the command; I think it is provided by the package ovirt-node-ng-nodectl, that is one of the available ones if I run "yum search" on the system. Thanks Gianluca

2 2

oVirt 3.6: Node went into Error state while migrations happening
by Christopher Cox 23 Aug '19

23 Aug '19

On the node in question, the metadata isn't coming across (state) wise. It shows VMs being in an unknown state (some are up and some are down), some show as migrating and there are 9 forever hung migrating tasks. We tried to bring up some of the down VMs that had a state of Down, but that ended up getting them the state of "Wait for Lauch", though those VMs are actually started. Right now, my plan is attempt a restart of vdsmd on the node in question. Just trying to get the node to a working state again. There a total of 9 nodes in our cluster, but we can't manage any VMs on the affected node right now. Is there a way in 3.6 to cancel the hung tasks? I'm worried that if vdsmd is restarted on the node, the tasks might be "attempted"... I really need them to be forgotten if possible. Ideally want all "Unknown" to return to either an "up" or "down" state (depending if the VM is up or down) and for "Wait for Launch" for those, to go to "up" and for all the "Migrating" to go to "up" or "down" (I think only one is actually down). I'm concerned that any attempt manually maniplate the state in the ovirt mgmt head db will be moot because the node will be queried for state and that state will be taken and override anything I attempt to do. Thoughts??

2 1

When I create a new domain in storage , it report : VDSM command ActivateStorageDomainVDS failed: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842', )
by wangyu13476969128＠126.com 23 Aug '19

23 Aug '19

The version of ovirt-engine is 4.2.8 The version of ovirt-node is 4.2.8 When I create a new domain in storage , the storage type is NFS , it report : VDSM command ActivateStorageDomainVDS failed: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) The error of vdsm.log is : 2019-08-23 11:02:14,740+0800 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'id': u'c6893a09-ab28-4328-b186-d2f88f2320d4', u'connection': u'172.16.10.74:/ovirt-data', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'protocol_version': u'auto', u'password': '********', u'port': u''}], options=None) from=::ffff:172.16.90.10,52962, flow_id=5d720a32, task_id=2af924ff-37a6-46a1-b79f-4251d21d5ff9 (api:46) 2019-08-23 11:02:14,743+0800 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'status': 0, 'id': u'c6893a09-ab28-4328-b186-d2f88f2320d4'}]} from=::ffff:172.16.90.10,52962, flow_id=5d720a32, task_id=2af924ff-37a6-46a1-b79f-4251d21d5ff9 (api:52) 2019-08-23 11:02:14,743+0800 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer succeeded in 0.00 seconds (__init__:573) 2019-08-23 11:02:14,751+0800 INFO (jsonrpc/6) [vdsm.api] START activateStorageDomain(sdUUID=u'3bcdc32c-040e-4a4c-90fb-de950f54f1b4', spUUID=u'b87012a1-8f7a-4af5-8884-e0fb8002e842', options=None) from=::ffff:172.16.90.10,53080, flow_id=709ee722, task_id=9b8f32af-5fdd-4ffa-9520-c474af03db70 (api:46) 2019-08-23 11:02:14,752+0800 INFO (jsonrpc/6) [vdsm.api] FINISH activateStorageDomain error=Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) from=::ffff:172.16.90.10,53080, flow_id=709ee722, task_id=9b8f32af-5fdd-4ffa-9520-c474af03db70 (api:50) 2019-08-23 11:02:14,752+0800 ERROR (jsonrpc/6) [storage.TaskManager.Task] (Task='9b8f32af-5fdd-4ffa-9520-c474af03db70') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "<string>", line 2, in activateStorageDomain File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1262, in activateStorageDomain pool = self.getPool(spUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 350, in getPool raise se.StoragePoolUnknown(spUUID) StoragePoolUnknown: Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) 2019-08-23 11:02:14,752+0800 INFO (jsonrpc/6) [storage.TaskManager.Task] (Task='9b8f32af-5fdd-4ffa-9520-c474af03db70') aborting: Task is aborted: "Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',)" - code 309 (task:1181) 2019-08-23 11:02:14,752+0800 ERROR (jsonrpc/6) [storage.Dispatcher] FINISH activateStorageDomain error=Unknown pool id, pool not connected: (u'b87012a1-8f7a-4af5-8884-e0fb8002e842',) (dispatcher:82) How can I solve this problem ?

2 1

large size OVA import fails
by jason.l.cox＠l3harris.com 22 Aug '19

22 Aug '19

I have a fairly large OVA (~200GB) that was exported from oVirt4.3.5. I'm trying to import it into a new cluster, also oVirt4.3.5. The import starts fine but fails again and again. Everything I can find online appears to be outdated, mentioning incorrect log file locations and saying virt-v2v does the import. On the engine in /var/log/ovirt-engine/engine.log I can see where it is doing the CreateImageVDSCommand, then a few outputs concerning adding the disk, which end with USER_ADD_DISK_TO_VM_FINISHED_SUCCESS, then the ansible command: 2019-08-20 15:40:38,653-04 Executing Ansible command: /usr/bin/ansible-playbook --ssh-common-args=-F /var/lib/ovirt-engine/.ssh/config -v --private-key=/etc/pki/ovirt-engine/keys/engine_id_rsa --inventory=/tmp/ansible-inventory8416464991088315694 --extra-vars=ovirt_import_ova_path="/mnt/vm_backups/myvm.ova" --extra-vars=ovirt_import_ova_disks="['/rhev/data-center/mnt/glusterSD/myhost.mydomain.com:_vmstore/59502c8b-fd1e-482b-bff7-39c699c196b3/images/886a3313-19a9-435d-aeac-64c2d507bb54/465ce2ba-8883-4378-bae7-e231047ea09d']" --extra-vars=ovirt_import_ova_image_mappings="{}" /usr/share/ovirt-engine/playbooks/ovirt-ova-import.yml [Logfile: /var/log/ovirt-engine/ova/ovirt-import-ova-ansible-20190820154038-myhost.mydomain.com-25f6ac6f-9bdc-4301-b896-d357712dbf01.log] Then nothing about the import until: 2019-08-20 16:11:08,859-04 INFO [org.ovirt.engine.core.bll.exportimport.ImportVmFromOvaCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [3321d4f6] Lock freed to object 'EngineLock:{exclusiveLocks='[myvm=VM_NAME, 464a25ba-8f0a-421d-a6ab-13eff67b4c96=VM]', sharedLocks=''}' 2019-08-20 16:11:08,894-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-88) [3321d4f6] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm myvm to Data Center Default, Cluster Default I've found the import logs on the engine, in /var/log/ovirt-engine/ova, but the ovirt-import-ova-ansible*.logs for the imports of concern only contain: 2019-08-20 19:59:48,799 p=44701 u=ovirt | Using /usr/share/ovirt-engine/playbooks/ansible.cfg as config file 2019-08-20 19:59:49,271 p=44701 u=ovirt | PLAY [all] ********************************************************************* 2019-08-20 19:59:49,280 p=44701 u=ovirt | TASK [ovirt-ova-extract : Run extraction script] ******************************* Watching the host selected for the import, I can see the qemu-img convert process running, but then the engine frees the lock on the VM and reports the import as having failed. However, the qemu-img process continues to run on the host. I don't know where else to look to try and find out what's going on and I cannot see anything that says why the import failed. Since the qemu-img process on the host is still running after the engine log shows the lock has been freed and import failed, I'm guessing what's happening is on the engine side. Looking at the time between the start of the ansible command and when the lock is freed it is consistently around 30 minutes. # first try 2019-08-20 15:40:38,653-04 ansible command start 2019-08-20 16:11:08,859-04 lock freed 31 minutes # second try 2019-08-20 19:59:48,463-04 ansible command start 2019-08-20 20:30:21,697-04 lock freed 30 minutes, 33 seconds # third try 2019-08-21 09:16:42,706-04 ansible command start 2019-08-21 09:46:47,103-04 lock freed 30 minutes, 45 seconds With that in mind, I took a look at the available configuration keys from engine-config --list. After getting each the only one set to ~30 minutes and looks like it could be the problem was SSHInactivityHardTimeoutSeconds (set to 1800 seconds). I set it to 3600 and tried the import again, but it still failed at ~30 minutes, so that's apparently not the correct key. Also, just FYI, I also tried to import the ova using virt-v2v, but that fails immediately: virt-v2v: error: expecting XML expression to return an integer (expression: rasd:Parent/text(), matching string: 00000000-0000-0000-0000-000000000000) If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Does virt-v2v not support OVAs created by the oVirt 'export to ova' option? So my main question is, Is there a timeout for VM imports through the engine web UI? And if so, is if configurable? Thanks in advance.

2 1

ovn networking
by Staniforth, Paul 22 Aug '19

22 Aug '19

Hello I the latest release of the engine 4.3.5.5-1.el7 ovn-nbctl show ovn-sbctl show both seem to work but produce the errors net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5) PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4) should libibverbs be a dependency? Paul S. To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/

3 5

VM won't migrate back to original node
by Alexander Schichow 22 Aug '19

22 Aug '19

I set up a high available VM which is using a shareable LUN. When migrating said VM from the original Node everything is fine. When I try to migrate it back, I get an error. If I suspend or shutdown the VM, it returns to the original host. Then it repeats. I'm really out of ideas so maybe any of you happen to have any? I would really appreciate that. Thanks

2 1

Custum driver in oVirt node image?
by thomas＠hoberg.net 22 Aug '19

22 Aug '19

Using 3 Mini-ITX Atoms J5005 boards to build a three node HC test environment (actually there is a fourth to simulate a remote DC) These boards have a RealTek 8169 Gbit controller, which for hardware/firmware or driver reasons comes up just fine on cold boots, but struggles on warm-boots or whenever the interface is re-configured, as is the case when the OVS overlay is configured, ... * unless link-speed autonegotiation is enabled after which it works just fine. So for a normal system, I can just put a proper ETHTOOL_OPTS="autoneg on" into /etc/sysconfig/network-scrkipts/ifcfg-<device-name> to avoid trouble on warm boots, but once the overlay network is put on top, the network manager no longer controls the device and the standard "ignore" setting of driver/hardware evidently causes the link to fail: dmesg reports the interface toggling and ovs isn't happy either. The Ethernet switch is unmanaged, so there is nothing I can do there to speed up or eliminate negotiations, and I currently see three options: 1. Find/be told where to renable Ethernet link speed autonegotiation with the OVS/VDSM/?? so the RTL8169 is happy again 2. Use a known USB3 GBit NIC (no space in the chassis to put e.g. an Intel NIC) 3. Make an USB3 2.5Gbit NIC based on RealTek's 8156 work, but currently that requires compiling from source and loading it at boot Option 3 works just fine with a CentOS base, but I got tons of problem making oVirt hosted engine work with that. Option 3 for the oVirt node image seems to class with the read-only nature of the oVirt node imge: While I can load a pre-compile driver interactively, I can't make it stick with dracut -f. I haven't immediately found a way to patch/fix/build derivative oVirt node images with that driver included and am somewhat to impatient to wait until it's mainstream, so any help would be appreciated. I've verified option 2 with a single USB3 GBit adapter natively supported by the kernel that I already had available, but I then opted to jump the gun and buy the slightly more expensive 2.5GBit variant as a better match to a Gluster environment: A mistake in this context as it turned out.

2 2

[ANN] oVirt 4.3.6 Third Release Candidate is now available for testing
by Sandro Bonazzola 22 Aug '19

22 Aug '19

The oVirt Project is pleased to announce the availability of the oVirt 4.3.6 Third Release Candidate for testing, as of August 22nd, 2019. This update is a release candidate of the sixth in a series of stabilization updates to the 4.3 series. This is pre-release software. This pre-release should not to be used in production. This release is available now on x86_64 architecture for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for: * Red Hat Enterprise Linux 7.7 or later (but <8) * CentOS Linux (or similar) 7.7 or later (but <8) * oVirt Node 4.3 (available for x86_64 only) will be made available when CentOS 7.7 will be released. See the release notes [1] for known issues, new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Node is not yet available, pending CentOS 7.7 release to be available Additional Resources: * Read more about the oVirt 4.3.6 release highlights: http://www.ovirt.org/release/4.3.6/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] http://www.ovirt.org/release/4.3.6/ -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 0

Re: Where to configure iscsi initiator name?
by Gianluca Cecchi 22 Aug '19

22 Aug '19

On Wed, Aug 21, 2019 at 7:07 PM Dan Poltawski <dan.poltawski(a)tnp.net.uk> wrote: > When I added the first node a 'random' initiator name was generated of > form: > > # cat /etc/iscsi/initiatorname.iscsi > InitiatorName=iqn.1994-05.com.redhat:[RANDOM] > > Having attempted to add another node, this node has another initiator name > generated and can't access the storage. Is there a way to configure this > initiator name to a static value which will be configured when new nodes > get added to the cluster? Or is there some reason for this i'm missing? > > thanks, > > Dan > oVirt nodes are iSCSI clients and each node needs to have a different InitiatorName value. Eventually you can modify initiatorname of a node before running discovery against oVirt and rebooting it (or restarting iscsi services). You can edit the file or use the iscsi-iname command. It must be named with an iqn name in the format of iqn.YYYY-MM.reverse.domain.name:OptionalIdentifier Typically you configure your iSCSI storage array giving access to the lun to be used as a storage domain to all oVirt hosts initiators. Or you can use chap authentication or both Se also here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht… https://www.certdepot.net/rhel7-configure-iscsi-target-initiator-persistent… HIH, Gianluca

1 0

Data Center Contending/Non responsive - No SPM
by Matthew B 21 Aug '19

21 Aug '19

Hello, I'm having a problem with oVirt - we're running 4.3 on CentOS 7: [root@ovirt ~]# rpm -q ovirt-engine; uname -r ovirt-engine-4.3.4.3-1.el7.noarch 3.10.0-957.21.2.el7.x86_64 Currently the Data Center alternates between Non responsive and contending status, and SPM selection fails. The error in the events tab is: VDSM compute5.domain command HSMGetTaskStatusVDS failed: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',) Full error: 2019-08-21 13:53:53,507-0700 ERROR (tasks/8) [storage.TaskManager.Task] (Task='3076bb8c-7462-4573-a832-337da478ae0e') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 333, in startSpm self._upgradePool(expectedDomVersion, __securityOverride=True) File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 484, in _upgradePool str(targetDomVersion)) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1108, in _convertDomain targetFormat) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 447, in convert converter(repoPath, hostId, imageRepo, isMsd) File "/usr/lib/python2.7/site-packages/vdsm/storage/formatconverter.py", line 405, in v5DomainConverter domain.convert_volumes_metadata(target_version) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileSD.py", line 813, in convert_volumes_metadata for vol in self.iter_volumes(): File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 764, in iter_volumes yield self.produceVolume(img_id, vol_id) File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 846, in produceVolume volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/glusterVolume.py", line 45, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 817, in __init__ self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID, volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 71, in __init__ volUUID) File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 86, in __init__ self.validate() File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line 112, in validate self.validateVolumePath() File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py", line 131, in validateVolumePath raise se.VolumeDoesNotExist(self.volUUID) VolumeDoesNotExist: Volume does not exist: (u'2bffb8d0-dfb5-4b08-9c6b-716e11f280c2',) Most of our VMs are still running, but we can't start or restart any VMs. All of the domains show as down since SPM selection fails. Any thoughts? Thanks, -Matthew

1 0

Re: Where to configure iscsi initiator name?
by Vincent Royer 21 Aug '19

21 Aug '19

You have to add all the initiators for each node to the iscsi LUN. At least thats how I got it working. Then I think it's recommended to setup multipath. However take this with a grain because iscsi did not work well for me. On Wed, Aug 21, 2019, 10:08 AM Dan Poltawski <dan.poltawski(a)tnp.net.uk> wrote: > When I added the first node a 'random' initiator name was generated of > form: > > # cat /etc/iscsi/initiatorname.iscsi > InitiatorName=iqn.1994-05.com.redhat:[RANDOM] > > Having attempted to add another node, this node has another initiator name > generated and can't access the storage. Is there a way to configure this > initiator name to a static value which will be configured when new nodes > get added to the cluster? Or is there some reason for this i'm missing? > > thanks, > > Dan > ------------------------------ > > The Networking People (TNP) Limited. Registered office: Network House, > Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company > number: 07667393 > > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify the system manager. > This message contains confidential information and is intended only for the > individual named. If you are not the named addressee you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately by e-mail if you have received this e-mail by mistake and > delete this e-mail from your system. If you are not the intended recipient > you are notified that disclosing, copying, distributing or taking any > action in reliance on the contents of this information is strictly > prohibited. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UKNN77NZJ4ZTB… >

1 0

hosted engine setup, iSCSI no LUNs shown
by billyburly＠gmail.com 21 Aug '19

21 Aug '19

I'm trying to setup the hosted engine on top of iSCSI storage. It successfully logs in and gets the target, however the process errors out claiming there are no LUNs. But if you look on the host, the disks were added to the system. [ INFO ] TASK [ovirt.hosted_engine_setup : iSCSI discover with REST API] [ INFO ] ok: [localhost] The following targets have been found: [1] iqn.2001-04.com.billdurr.durrnet.vm-int:vmdata TPGT: 1, portals: 192.168.47.10:3260 Please select a target (1) [1]: 1 [ INFO ] Getting iSCSI LUNs list ... [ INFO ] TASK [ovirt.hosted_engine_setup : Get iSCSI LUNs] [ INFO ] ok: [localhost] [ ERROR ] Cannot find any LUN on the selected target [ ERROR ] Unable to get target list Here's what the config in targetcli looks like [root@vm1 ~]# targetcli ls o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 2] | | o- p_iscsi_lun1 .............. [/dev/drbd0 (62.0GiB) write-thru activated] | | | o- alua ............................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ................... [ALUA state: Active/optimized] | | o- p_iscsi_lun2 ............. [/dev/drbd1 (310.6GiB) write-thru activated] | | o- alua ............................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ................... [ALUA state: Active/optimized] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 1] | o- iqn.2001-04.com.billdurr.durrnet.vm-int:vmdata ................ [TPGs: 1] | o- tpg1 .............................................. [gen-acls, no-auth] | o- acls ...................................................... [ACLs: 0] | o- luns ...................................................... [LUNs: 2] | | o- lun0 ......... [block/p_iscsi_lun1 (/dev/drbd0) (default_tg_pt_gp)] | | o- lun1 ......... [block/p_iscsi_lun2 (/dev/drbd1) (default_tg_pt_gp)] | o- portals ................................................ [Portals: 1] | o- 192.168.47.10:3260 ........................................... [OK] o- loopback ..................................................... [Targets: 0] o- srpt ......................................................... [Targets: 0] The two LUNs show up on the host after the hosted engine setup tries to enumerate the LUNs for the target [root@vm1 ~]# lsscsi [0:0:0:0] storage HP P420i 8.32 - [0:1:0:0] disk HP LOGICAL VOLUME 8.32 /dev/sda [0:1:0:1] disk HP LOGICAL VOLUME 8.32 /dev/sdb [0:1:0:2] disk HP LOGICAL VOLUME 8.32 /dev/sdc [11:0:0:0] disk LIO-ORG p_iscsi_lun1 4.0 /dev/sdd [11:0:0:1] disk LIO-ORG p_iscsi_lun2 4.0 /dev/sde

3 2

Reaction and timeouts to APD (All Paths Down)
by Gianluca Cecchi 21 Aug '19

21 Aug '19

Hello, similar questions/arguments took already place in the past, but I think it could be a good point to dig more if possible. I focus on block based storage and my environments have iSCSI based with multipath, connected to Equallogic. Currently I have set no_path_retry to 4, so I have a 20 seconds timeout (polling_interval=5). Sometimes some network planned activity (not under my domain) that should have no impact "doesn't go so well" and I could have longer delays (also 60-70 seconds) and so oVirt reactions where I see soft fencing, VMs going into paused state or "question mark" state... Comparing to vSphere, the same events don't apparently cause anything (and I see same lost paths events in datastore and ESXi host monitoring--> Events pane). This basically depends on APD default timeout that seems to be 140 seconds. This difference of behavior has the effect of showing an apparent better SLA of vSphere during these short time outages and a sort of show stopper for extending oVirt implementation.... Interesting vSphere KBs here: Storage device has entered the All Paths Down state (2032934) https://kb.vmware.com/s/article/2032934 Containing: "Note: By default, the APD timeout is set to 140 seconds." All Paths Down timeout for a storage device has expired (2032940) https://kb.vmware.com/s/article/2032940 Path redundancy to the storage device is degraded (1009555) https://kb.vmware.com/s/article/1009555 Storage device has recovered from the APD state (2032945) https://kb.vmware.com/s/article/2032945 So the question is what kind of real risks I have if I simulate that behavior and set a no_path_retry value so that polling_interval x no_path_retry = 140 (in default config of polling_interval=5 it would mean no_path_retry = 28) BTW: I also have an environment based on RHV 4.3.5 and iSCSI and in parallel I opened a case (02452597) for asking clarifications/chances remaining in supported configuration, so for logs and so on it could help going into it for Red Hat developers Thanks, Gianluca

1 0

Where to configure iscsi initiator name?
by Dan Poltawski 21 Aug '19

21 Aug '19

When I added the first node a 'random' initiator name was generated of form: # cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:[RANDOM] Having attempted to add another node, this node has another initiator name generated and can't access the storage. Is there a way to configure this initiator name to a static value which will be configured when new nodes get added to the cluster? Or is there some reason for this i'm missing? thanks, Dan ________________________________ The Networking People (TNP) Limited. Registered office: Network House, Caton Rd, Lancaster, LA1 3PE. Registered in England & Wales with company number: 07667393 This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

1 0

engine-setup is failing in fedora30
by Gobinda Das 21 Aug '19

21 Aug '19

Hi, I am facing issue to do engine-setup in fedora30. [godas@godas ovirt-engine]$ $HOME/ovirt-engine/bin/engine-setup [ INFO ] Stage: Initializing Setup was run under unprivileged user this will produce development installation do you wish to proceed? (Yes, No) [No]: Yes [ INFO ] Stage: Environment setup Configuration files: ['/home/gobinda/ovirt-engine/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'] Log file: /home/gobinda/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20190821121001-bkc6we.log Version: otopi-1.8.1_master (otopi-1.8.1-0.0.master.20190228094531.git1953bdc.fc30) [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup (late) [ INFO ] Stage: Environment customization --== PRODUCT OPTIONS ==-- --== PACKAGES ==-- --== NETWORK CONFIGURATION ==-- [WARNING] Failed to resolve godas.lab.eng.blr.redhat.com using DNS, it can be resolved only locally --== DATABASE CONFIGURATION ==-- --== OVIRT ENGINE CONFIGURATION ==-- Perform full vacuum on the engine database engine@localhost? This operation may take a while depending on this setup health and the configuration of the db vacuum process. See https://www.postgresql.org/docs/10/sql-vacuum.html (Yes, No) [No]: --== STORAGE CONFIGURATION ==-- --== PKI CONFIGURATION ==-- --== APACHE CONFIGURATION ==-- --== SYSTEM CONFIGURATION ==-- --== MISC CONFIGURATION ==-- --== END OF CONFIGURATION ==-- [ INFO ] Stage: Setup validation --== CONFIGURATION PREVIEW ==-- Default SAN wipe after delete : False Host FQDN : godas.lab.eng.blr.redhat.com Set up Cinderlib integration : False Engine database host : localhost Engine database port : 5432 Engine database secured connection : False Engine database host name validation : False Engine database name : engine Engine database user name : engine Engine installation : True PKI organization : lab.eng.blr.redhat.com Set up ovirt-provider-ovn : False Configure WebSocket Proxy : True Configure VMConsole Proxy : True Please confirm installation settings (OK, Cancel) [OK]: OK [ INFO ] Checking the Engine database consistency [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration (early) [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Upgrading CA [ INFO ] Configuring WebSocket Proxy [ INFO ] Backing up database localhost:engine to '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/backups/engine-20190821121010.0iip56rw.dump'. [ INFO ] Creating/refreshing Engine database schema Unregistering existing client registration info. [ INFO ] Generating post install configuration file '/home/gobinda/ovirt-engine/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf' [ ERROR ] Failed to execute stage 'Misc configuration': a bytes-like object is required, not 'str' [ INFO ] Rolling back database schema [ INFO ] Clearing Engine database engine [ INFO ] Restoring Engine database engine [ INFO ] Restoring file '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/backups/engine-20190821121010.0iip56rw.dump' to database localhost:engine. [ ERROR ] Engine database rollback failed: cannot use a string pattern on a bytes-like object [ INFO ] Stage: Clean up Log file is located at /home/gobinda/ovirt-engine/var/log/ovirt-engine/setup/ovirt-engine-setup-20190821121001-bkc6we.log [ INFO ] Generating answer file '/home/gobinda/ovirt-engine/var/lib/ovirt-engine/setup/answers/20190821121039-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed I*t was working before but the moment i pulled latest master code the issue started.* -- Thanks, Gobinda

2 1

When I added the ovirt-node to the ovirt-engine through the web management interface, I get the following error: Host 172.16.41.100 installation failed. Failed to configure management network on the host. Host 172.16.41.100 does not comply with the cluster cluster-kvm networks, the following networks are missing on host: 'ovirtmgmt'
by wangyu13476969128＠126.com 21 Aug '19

21 Aug '19

When I added the ovirt-node to the ovirt-engine through the web management interface, I get the following error: Host 172.16.41.100 installation failed. Failed to configure management network on the host. Host 172.16.41.100 does not comply with the cluster cluster-kvm networks, the following networks are missing on host: 'ovirtmgmt' How can I solve this problem ?

3 12