Rebooted the hosted engine VM

Migration now works

Guess that's needed after changing cluster CPU type

Thanks,

Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Weill Cornell Medicine
1300 York - LC-502
E: doug@med.cornell.edu
O: 212-746-6305
F: 212-746-8690


On Thu, Aug 23, 2018 at 8:46 PM, Douglas Duckworth <dod2014@med.cornell.edu> wrote:
Hi 

I am unable to migrate to hypervisor that's sandy bridge CPU.

The cluster supports CPUs that old.

This is the message on host when migration fails:

Aug 23 20:40:14 lyon vdsm[9648]: WARN Worker blocked: <Worker name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7f2f8054b310> at 0x7f2f8054b350> timeout=15, duration=15 at 0x7f2f8048d290> task#=79 at 0x7f2f80540490>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012  self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012  self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012  self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run#012  ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012  self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012  task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012  self._callable()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in __call__#012  self._func()#012File: "/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in __call__#012  stats = hostapi.get_stats(self._cif, self._samples.stats())#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats#012  ret['haStats'] = _getHaInfo()#012File: "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo#012  stats = instance.get_all_stats()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 94, in get_all_stats#012  stats = broker.get_stats_from_storage()#012File: "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 135, in get_stats_from_storage#012  result = self._proxy.get_stats()#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__#012  return self.__send(self.__name, args)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request#012  verbose=self.__verbose#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request#012  return self.single_request(host, handler, request_body, verbose)#012File: "/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request#012  response = h.getresponse(buffering=True)#012File: "/usr/lib64/python2.7/httplib.py", line 1113, in getresponse#012  response.begin()#012File: "/usr/lib64/python2.7/httplib.py", line 444, in begin#012  version, status, reason = self._read_status()#012File: "/usr/lib64/python2.7/httplib.py", line 400, in _read_status#012  line = self.fp.readline(_MAXLINE + 1)#012File: "/usr/lib64/python2.7/socket.py", line 476, in readline#012  data = self._sock.recv(self._rbufsize)
Aug 23 20:43:45 lyon vdsm[9648]: WARN ping was deprecated in favor of ping2 and confirmConnectivity
Aug 23 20:43:45 lyon vdsm[9648]: WARN Attempting to add an existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b
Aug 23 20:43:46 lyon lldpad: recvfrom(Event interface): No buffer space available
Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking state
Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state
Aug 23 20:43:46 lyon kernel: device vnet0 entered promiscuous mode
Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered blocking state
Aug 23 20:43:46 lyon kernel: ovirtmgmt: port 2(vnet0) entered forwarding state
Aug 23 20:43:46 lyon NetworkManager[1658]: <info>  [1535071426.4326] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/57)
Aug 23 20:43:46 lyon NetworkManager[1658]: <info>  [1535071426.4336] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Aug 23 20:43:46 lyon NetworkManager[1658]: <info>  [1535071426.4341] device (vnet0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external')
Aug 23 20:43:46 lyon dbus[1396]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.machine1.service'
Aug 23 20:43:46 lyon systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 23 20:43:46 lyon systemd: Starting Virtual Machine and Container Registration Service...
Aug 23 20:43:46 lyon dbus[1396]: [system] Successfully activated service 'org.freedesktop.machine1'
Aug 23 20:43:46 lyon systemd: Started Virtual Machine and Container Registration Service.
Aug 23 20:43:46 lyon systemd-machined: New machine qemu-2-HostedEngine.
Aug 23 20:43:46 lyon systemd: Started Virtual Machine qemu-2-HostedEngine.
Aug 23 20:43:46 lyon systemd: Starting Virtual Machine qemu-2-HostedEngine.
Aug 23 20:43:46 lyon kvm: 1 guest now active
Aug 23 20:43:46 lyon libvirtd: 2018-08-24 00:43:46.940+0000: 9195: error : virCPUx86UpdateLive:2742 : operation failed: guest CPU doesn't match specification: missing features: fma,movbe,f16c,rdrand,fsgsbase,bmi1,hle,avx2,smep,bmi2,erms,invpcid,rtm,mpx,rdseed,adx,smap,xsavec,xgetbv1,abm,3dnowprefetch
Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state
Aug 23 20:43:47 lyon kernel: device vnet0 left promiscuous mode
Aug 23 20:43:47 lyon kernel: ovirtmgmt: port 2(vnet0) entered disabled state
Aug 23 20:43:47 lyon avahi-daemon[1300]: Withdrawing workstation service for vnet0.
Aug 23 20:43:47 lyon NetworkManager[1658]: <info>  [1535071427.2753] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed')
Aug 23 20:43:47 lyon NetworkManager[1658]: <info>  [1535071427.2753] device (vnet0): released from master device ovirtmgmt
Aug 23 20:43:47 lyon lldpad: recvfrom(Event interface): No buffer space available
Aug 23 20:43:47 lyon kvm: 0 guests now active
Aug 23 20:43:47 lyon systemd-machined: Machine qemu-2-HostedEngine terminated.
Aug 23 20:43:47 lyon vdsm[9648]: WARN File: /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.ovirt-guest-agent.0 already removed
Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non existing network: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b
Aug 23 20:43:47 lyon vdsm[9648]: WARN Attempting to remove a non existing net user: ovirtmgmt/adf14389-1563-4b1a-9af6-4b40370a825b
Aug 23 20:43:47 lyon vdsm[9648]: WARN File: /var/lib/libvirt/qemu/channels/adf14389-1563-4b1a-9af6-4b40370a825b.org.qemu.guest_agent.0 already removed
Aug 23 20:43:51 lyon lldpad: setsockopt nearest_bridge: Invalid argument
Aug 23 20:43:52 lyon lldpad: setsockopt nearest_bridge: Invalid argument

Can I fix this somehow?

Thanks,

Douglas Duckworth, MSc, LFCS
HPC System Administrator
Scientific Computing Unit
Weill Cornell Medicine
1300 York - LC-502
E: doug@med.cornell.edu
O: 212-746-6305
F: 212-746-8690