Two Problems On New Install
Hi Everyone, I recently installed oVirt 4.0 on two servers with an iSCSI backstore, and I'm having two issues I'm having trouble tracking down. First, the initial datastore that the hosted engine is running on won't connect. It just says "Unattached" in the GUI, and I can't find an error message anywhere. Not a huge deal, it just means I can't see the hosted engine itself in the GUI. More importantly: I can't migrate VMs. They fail with errors like this in the vdsm.log of the destination host: Thread-60381::ERROR::2017-02-14 21:55:47,921::vm::767::virt.vm::(_startUnderlyingVm) vmId=`59f235cc-3ab2-4dae-b460-62be850e3d5d`::Failed to start a migration destination vm Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm self._completeIncomingMigration() File "/usr/share/vdsm/virt/vm.py", line 3113, in _completeIncomingMigration self._incomingMigrationFinished.isSet(), usedTimeout) File "/usr/share/vdsm/virt/vm.py", line 3196, in _attachLibvirtDomainAfterMigration raise MigrationError(e.get_error_message()) MigrationError: Domain not found: no domain with matching uuid '59f235cc-3ab2-4dae-b460-62be850e3d5d' If I shut the VM down and start it on the host it works fine, it just won't migrate. I notice I'm also seeing a lot (like every 7 seconds) of this error: JsonRpc (StompReactor)::ERROR::2017-02-14 21:57:33,984::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof Which I can't seem to diagnose. Any ideas?
Hi Charles, I am not sure about the migration issue, we need more logs [1] to be able to tell, but the SSL issue is probably just annoying and not a functionality bug (see https://bugzilla.redhat.com/show_bug.cgi?id=1412583). Best regards [1] libvirt log for the domain, the VM process might be crashing for some reason -- Martin Sivak SLA / oVirt On Wed, Feb 15, 2017 at 3:45 AM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Everyone,
I recently installed oVirt 4.0 on two servers with an iSCSI backstore, and I'm having two issues I'm having trouble tracking down. First, the initial datastore that the hosted engine is running on won't connect. It just says "Unattached" in the GUI, and I can't find an error message anywhere. Not a huge deal, it just means I can't see the hosted engine itself in the GUI.
More importantly: I can't migrate VMs. They fail with errors like this in the vdsm.log of the destination host:
Thread-60381::ERROR::2017-02-14 21:55:47,921::vm::767::virt.vm::(_startUnderlyingVm) vmId=`59f235cc-3ab2-4dae-b460-62be850e3d5d`::Failed to start a migration destination vm Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm self._completeIncomingMigration() File "/usr/share/vdsm/virt/vm.py", line 3113, in _completeIncomingMigration self._incomingMigrationFinished.isSet(), usedTimeout) File "/usr/share/vdsm/virt/vm.py", line 3196, in _attachLibvirtDomainAfterMigration raise MigrationError(e.get_error_message()) MigrationError: Domain not found: no domain with matching uuid '59f235cc-3ab2-4dae-b460-62be850e3d5d'
If I shut the VM down and start it on the host it works fine, it just won't migrate. I notice I'm also seeing a lot (like every 7 seconds) of this error:
JsonRpc (StompReactor)::ERROR::2017-02-14 21:57:33,984::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof
Which I can't seem to diagnose. Any ideas?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Martin, Thanks for responding, and sorry for the long delay in getting back to you. Where would I find the libvirt log? I checked on the host and I have individual log files in /var/log/libvirt/qemu/<vmName>.log, but nothing for the guest I'm trying to migrate. I did some more digging and found these errors, which seem to point to some sort of issue communicating with the JsonRpcServer module. I've put the full vdsm.log output from the migration attempt up at http://krissy.islandadmin.ca/public/vdsm.txt but the interesting parts seems to be: jsonrpc.Executor/3::DEBUG::2017-02-24 01:01:36,415::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getAllVmIoTunePolicies' in bridge with {} jsonrpc.Executor/3::ERROR::2017-02-24 01:01:36,436::__init__::550::jsonrpc.JsonRpcServer::(_handle_request) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 545, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 195, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1358, in getAllVmIoTunePolicies io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() File "/usr/share/vdsm/clientIF.py", line 447, in getAllVmIoTunePolicies vm_io_tune_policies[v.id] = {'policy': v.getIoTunePolicy(), File "/usr/share/vdsm/virt/vm.py", line 2772, in getIoTunePolicy qos = self._getVmPolicy() File "/usr/share/vdsm/virt/vm.py", line 2746, in _getVmPolicy metadata_xml = self._dom.metadata( File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM u'3390a529-e0c3-4f1f-9c8f-7595b2b3f38f' was not started yet or was shut down jsonrpc.Executor/3::INFO::2017-02-24 01:01:36,437::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getAllVmIoTunePolicies failed (error -32603) in 0.03 seconds Thread-351015::DEBUG::2017-02-24 01:01:36,471::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 Thread-351015::DEBUG::2017-02-24 01:01:36,471::fileUtils::209::Storage.fileUtils::(createdir) Creating directory: /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/a8da74c5-24d8-46c1-9a70-92e3045a3c18 Thread-351015::DEBUG::2017-02-24 01:01:36,472::blockSD::1186::Storage.StorageDomain::(createImageLinks) Creating symlink from /dev/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/2e00f0ba-5d93-4c3b-af37-b53dcc68a43f to /v Thread-351015::DEBUG::2017-02-24 01:01:36,472::blockSD::1150::Storage.StorageDomain::(linkBCImage) Creating symlink from /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/a8da74c5-24d8-46c1-9a70-92e304 ... jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.b22b3666-f8d6-4beb-ac19-f2a6c8f49271`ReqID=`108242ef-f04e-4b9b-8a08-8f9ed1fae9 jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`a9e9fcd9-65fe-4652-96d1-8afc2cf47269`::_resourcesAcquired: Storage.b22b3666-f8d6-4beb-ac19-f2a6c jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::task::995::Storage.TaskManager.Task::(_decref) Task=`a9e9fcd9-65fe-4652-96d1-8afc2cf47269`::ref 1 aborting False jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::fileUtils::190::Storage.fileUtils::(cleanupdir) Removing directory: /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/9cdfe4ab-c23e-42cd-8f45-2ca573a jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,646::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { preferred_names = ["^/dev/mapper Thread-351015::ERROR::2017-02-24 01:01:37,656::vm::767::virt.vm::(_startUnderlyingVm) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Failed to start a migration destination vm Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm self._completeIncomingMigration() File "/usr/share/vdsm/virt/vm.py", line 3113, in _completeIncomingMigration self._incomingMigrationFinished.isSet(), usedTimeout) File "/usr/share/vdsm/virt/vm.py", line 3196, in _attachLibvirtDomainAfterMigration raise MigrationError(e.get_error_message()) MigrationError: Domain not found: no domain with matching uuid '3390a529-e0c3-4f1f-9c8f-7595b2b3f38f' Thread-351015::INFO::2017-02-24 01:01:37,658::vm::1330::virt.vm::(setDownStatus) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Changed state to Down: VM failed to migrate (code=8) Thread-351015::INFO::2017-02-24 01:01:37,658::guestagent::430::virt.vm::(stop) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Stopping connection Thread-351015::DEBUG::2017-02-24 01:01:37,660::__init__::209::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 5250206560, "3390a529-e0c3-4f1f-9c8f-7595b2b3f38f": {"status": "Down", "timeOf Thread-351015::DEBUG::2017-02-24 01:01:37,660::vm::777::virt.vm::(_startUnderlyingVm) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Releasing incoming migration semaphore On 2017-02-15 05:16 AM, Martin Sivak wrote:
Hi Charles,
I am not sure about the migration issue, we need more logs [1] to be able to tell, but the SSL issue is probably just annoying and not a functionality bug (see https://bugzilla.redhat.com/show_bug.cgi?id=1412583).
Best regards
[1] libvirt log for the domain, the VM process might be crashing for some reason
-- Martin Sivak SLA / oVirt
On Wed, Feb 15, 2017 at 3:45 AM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Everyone,
I recently installed oVirt 4.0 on two servers with an iSCSI backstore, and I'm having two issues I'm having trouble tracking down. First, the initial datastore that the hosted engine is running on won't connect. It just says "Unattached" in the GUI, and I can't find an error message anywhere. Not a huge deal, it just means I can't see the hosted engine itself in the GUI.
More importantly: I can't migrate VMs. They fail with errors like this in the vdsm.log of the destination host:
Thread-60381::ERROR::2017-02-14 21:55:47,921::vm::767::virt.vm::(_startUnderlyingVm) vmId=`59f235cc-3ab2-4dae-b460-62be850e3d5d`::Failed to start a migration destination vm Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm self._completeIncomingMigration() File "/usr/share/vdsm/virt/vm.py", line 3113, in _completeIncomingMigration self._incomingMigrationFinished.isSet(), usedTimeout) File "/usr/share/vdsm/virt/vm.py", line 3196, in _attachLibvirtDomainAfterMigration raise MigrationError(e.get_error_message()) MigrationError: Domain not found: no domain with matching uuid '59f235cc-3ab2-4dae-b460-62be850e3d5d'
If I shut the VM down and start it on the host it works fine, it just won't migrate. I notice I'm also seeing a lot (like every 7 seconds) of this error:
JsonRpc (StompReactor)::ERROR::2017-02-14 21:57:33,984::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof
Which I can't seem to diagnose. Any ideas?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi, yeah both snippets are actually telling you that there was something really unexpected. The first one comes from a call by MOM who is trying to get some IO limiting information about the VM. VDSM responded with "I know about the VM, but I can't find it in libvirt". The second part is something similar.. VDSM is expecting a VM to be known to libvirt, but there is no sign of it. I am adding some vdsm people who might know more about how to debug this. Best regards Martin Sivak On Fri, Feb 24, 2017 at 6:25 AM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Martin,
Thanks for responding, and sorry for the long delay in getting back to you. Where would I find the libvirt log? I checked on the host and I have individual log files in /var/log/libvirt/qemu/<vmName>.log, but nothing for the guest I'm trying to migrate.
I did some more digging and found these errors, which seem to point to some sort of issue communicating with the JsonRpcServer module. I've put the full vdsm.log output from the migration attempt up at http://krissy.islandadmin.ca/public/vdsm.txt but the interesting parts seems to be:
jsonrpc.Executor/3::DEBUG::2017-02-24 01:01:36,415::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) Calling 'Host.getAllVmIoTunePolicies' in bridge with {} jsonrpc.Executor/3::ERROR::2017-02-24 01:01:36,436::__init__::550::jsonrpc.JsonRpcServer::(_handle_request) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 545, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 195, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1358, in getAllVmIoTunePolicies io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() File "/usr/share/vdsm/clientIF.py", line 447, in getAllVmIoTunePolicies vm_io_tune_policies[v.id] = {'policy': v.getIoTunePolicy(), File "/usr/share/vdsm/virt/vm.py", line 2772, in getIoTunePolicy qos = self._getVmPolicy() File "/usr/share/vdsm/virt/vm.py", line 2746, in _getVmPolicy metadata_xml = self._dom.metadata( File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, in __getattr__ % self.vmid) NotConnectedError: VM u'3390a529-e0c3-4f1f-9c8f-7595b2b3f38f' was not started yet or was shut down jsonrpc.Executor/3::INFO::2017-02-24 01:01:36,437::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call Host.getAllVmIoTunePolicies failed (error -32603) in 0.03 seconds Thread-351015::DEBUG::2017-02-24 01:01:36,471::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 Thread-351015::DEBUG::2017-02-24 01:01:36,471::fileUtils::209::Storage.fileUtils::(createdir) Creating directory: /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/a8da74c5-24d8-46c1-9a70-92e3045a3c18 Thread-351015::DEBUG::2017-02-24 01:01:36,472::blockSD::1186::Storage.StorageDomain::(createImageLinks) Creating symlink from /dev/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/2e00f0ba-5d93-4c3b-af37-b53dcc68a43f to /v Thread-351015::DEBUG::2017-02-24 01:01:36,472::blockSD::1150::Storage.StorageDomain::(linkBCImage) Creating symlink from /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/a8da74c5-24d8-46c1-9a70-92e304
...
jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::resourceManager::241::Storage.ResourceManager.Request::(grant) ResName=`Storage.b22b3666-f8d6-4beb-ac19-f2a6c8f49271`ReqID=`108242ef-f04e-4b9b-8a08-8f9ed1fae9 jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::task::829::Storage.TaskManager.Task::(resourceAcquired) Task=`a9e9fcd9-65fe-4652-96d1-8afc2cf47269`::_resourcesAcquired: Storage.b22b3666-f8d6-4beb-ac19-f2a6c jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::task::995::Storage.TaskManager.Task::(_decref) Task=`a9e9fcd9-65fe-4652-96d1-8afc2cf47269`::ref 1 aborting False jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,645::fileUtils::190::Storage.fileUtils::(cleanupdir) Removing directory: /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/9cdfe4ab-c23e-42cd-8f45-2ca573a jsonrpc.Executor/6::DEBUG::2017-02-24 01:01:37,646::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { preferred_names = ["^/dev/mapper Thread-351015::ERROR::2017-02-24 01:01:37,656::vm::767::virt.vm::(_startUnderlyingVm) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Failed to start a migration destination vm Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm self._completeIncomingMigration() File "/usr/share/vdsm/virt/vm.py", line 3113, in _completeIncomingMigration self._incomingMigrationFinished.isSet(), usedTimeout) File "/usr/share/vdsm/virt/vm.py", line 3196, in _attachLibvirtDomainAfterMigration raise MigrationError(e.get_error_message()) MigrationError: Domain not found: no domain with matching uuid '3390a529-e0c3-4f1f-9c8f-7595b2b3f38f' Thread-351015::INFO::2017-02-24 01:01:37,658::vm::1330::virt.vm::(setDownStatus) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Changed state to Down: VM failed to migrate (code=8) Thread-351015::INFO::2017-02-24 01:01:37,658::guestagent::430::virt.vm::(stop) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Stopping connection Thread-351015::DEBUG::2017-02-24 01:01:37,660::__init__::209::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 5250206560, "3390a529-e0c3-4f1f-9c8f-7595b2b3f38f": {"status": "Down", "timeOf Thread-351015::DEBUG::2017-02-24 01:01:37,660::vm::777::virt.vm::(_startUnderlyingVm) vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Releasing incoming migration semaphore
On 2017-02-15 05:16 AM, Martin Sivak wrote:
Hi Charles,
I am not sure about the migration issue, we need more logs [1] to be able to tell, but the SSL issue is probably just annoying and not a functionality bug (see https://bugzilla.redhat.com/show_bug.cgi?id=1412583).
Best regards
[1] libvirt log for the domain, the VM process might be crashing for some reason
-- Martin Sivak SLA / oVirt
On Wed, Feb 15, 2017 at 3:45 AM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Everyone,
I recently installed oVirt 4.0 on two servers with an iSCSI backstore, and I'm having two issues I'm having trouble tracking down. First, the initial datastore that the hosted engine is running on won't connect. It just says "Unattached" in the GUI, and I can't find an error message anywhere. Not a huge deal, it just means I can't see the hosted engine itself in the GUI.
More importantly: I can't migrate VMs. They fail with errors like this in the vdsm.log of the destination host:
Thread-60381::ERROR::2017-02-14 21:55:47,921::vm::767::virt.vm::(_startUnderlyingVm) vmId=`59f235cc-3ab2-4dae-b460-62be850e3d5d`::Failed to start a migration destination vm Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm self._completeIncomingMigration() File "/usr/share/vdsm/virt/vm.py", line 3113, in _completeIncomingMigration self._incomingMigrationFinished.isSet(), usedTimeout) File "/usr/share/vdsm/virt/vm.py", line 3196, in _attachLibvirtDomainAfterMigration raise MigrationError(e.get_error_message()) MigrationError: Domain not found: no domain with matching uuid '59f235cc-3ab2-4dae-b460-62be850e3d5d'
If I shut the VM down and start it on the host it works fine, it just won't migrate. I notice I'm also seeing a lot (like every 7 seconds) of this error:
JsonRpc (StompReactor)::ERROR::2017-02-14 21:57:33,984::betterAsyncore::113::vds.dispatcher::(recv) SSL error during reading data: unexpected eof
Which I can't seem to diagnose. Any ideas?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (2)
-
Charles Tassell -
Martin Sivak