[ovirt-users] vmware import hangs after click load button on 3.5 rc5

Thu Sep 24 14:24:18 UTC 2015

Hi Nir,

https://gerrit.ovirt.org/46634/ has improved the situation, even tho the “Import Virtual Machine(s)” window still hangs. I queried all 3 vCenter Hosts then I run:

grep -e v2v -e libvirtE -e ERROR /var/log/vdsm/vdsm.log

I get the following much more helpful information:

Thread-19193::ERROR::2015-09-24 13:05:20,542::v2v::158::root::(get_external_vms) error getting domain xml for vm 'devsquid': internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'
Thread-19193::ERROR::2015-09-24 13:05:23,975::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HD2012': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19193::ERROR::2015-09-24 13:05:41,102::v2v::158::root::(get_external_vms) error getting domain xml for vm 'QA11 2012 R2': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19193::ERROR::2015-09-24 13:05:43,643::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDWin8': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19193::ERROR::2015-09-24 13:05:45,861::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDwin10-2': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19193::ERROR::2015-09-24 13:05:51,421::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDWin10': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19193::ERROR::2015-09-24 13:06:05,973::v2v::158::root::(get_external_vms) error getting domain xml for vm 'wsus-ash-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19193::ERROR::2015-09-24 13:06:06,841::v2v::158::root::(get_external_vms) error getting domain xml for vm 'Hactar2': internal error: Expecting VMX entry 'numvcpus' to be an unsigned integer (1 or a multiple of 2) but found 3
Thread-19193::ERROR::2015-09-24 13:06:21,972::v2v::158::root::(get_external_vms) error getting domain xml for vm 'squid3.2': internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'
Thread-19382::ERROR::2015-09-24 13:06:45,322::v2v::158::root::(get_external_vms) error getting domain xml for vm 'sequoia-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19382::ERROR::2015-09-24 13:06:47,515::v2v::158::root::(get_external_vms) error getting domain xml for vm 'sales-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19382::ERROR::2015-09-24 13:06:50,535::v2v::158::root::(get_external_vms) error getting domain xml for vm 'rdpgw-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
Thread-19382::ERROR::2015-09-24 13:06:54,202::v2v::688::root::(_add_disk_info) Error getting disk size
  File "/usr/share/vdsm/v2v.py", line 686, in _add_disk_info
    if ret is None: raise libvirtError ('virStorageVolGetInfo() failed', vol=self)
libvirtError: internal error: Could not search in datastore 'MD3220i-LUN0': FileNotFound - File [MD3220i-LUN0] isostore was not found

Looks like it is continuing with some errors but still getting tripped on others. (vdsm.log attached)
Sadly none of these errors are showing in the events in the front end which would be more helpful.

I have been playing divide and conquer with my 80 VMs and it seems there are still things that libvirt doesn’t like about some of those VMs. Is there a way I can run these commands from the command line so I can pin the problems down and alert the libvirt guys directly? If so where would I find what commands are being run? Should I still create BZ against ovirt for these (one for each different problem or one for them all)?

Thanks again for your help

Ian

From: Nir Soffer [mailto:nsoffer at redhat.com]
Sent: 23 September 2015 19:15
To: Ian Fraser <Ian.Fraser at asm.org.uk>; users <users at ovirt.org>; Shahar Havivi <shavivi at redhat.com>
Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5

On Wed, Sep 23, 2015 at 8:26 PM, Nir Soffer <nsoffer at redhat.com<mailto:nsoffer at redhat.com>> wrote:
On Wed, Sep 23, 2015 at 8:21 PM, Nir Soffer <nsoffer at redhat.com<mailto:nsoffer at redhat.com>> wrote:
On Wed, Sep 23, 2015 at 12:40 PM, Ian Fraser <Ian.Fraser at asm.org.uk<mailto:Ian.Fraser at asm.org.uk>> wrote:
Hi Nir,

I have changed the v2v.py file as per your request, it has now changed the behaviour. The popup window still hangs and I get the following two events:

Failed to retrieve VMs information from external server vpx://username%40domain@vcenter.server/datacentre_name/hostname?no_verify=1

VDSM ovirt-host-02 command failed: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

Fixing the first error, we see now the real error; libvirt cannot handle this vm configuration. We will ask one of the libvirt guys to look into this.

In vdsm.log, we see that one vm was parsed successfully (it failed when geting disk info):

Thread-1076::ERROR::2015-09-23 10:18:51,778::v2v::682::root::(_add_disk_info) Error getting disk size
Traceback (most recent call last):
  File "/usr/share/vdsm/v2v.py", line 679, in _add_disk_info
    vol = conn.storageVolLookupByPath(disk['alias'])
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4308, in storageVolLookupByPath
    if ret is None:raise libvirtError('virStorageVolLookupByPath() failed', conn=self)
libvirtError: An error occurred, but the cause is unknown

Little later, libvirt failed to return the xml for another vm:

Thread-1076::ERROR::2015-09-23 10:18:52,604::__init__::526::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 521, in _serveRequest
    res = method(**params)
  File "/usr/share/vdsm/rpc/Bridge.py", line 273, in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/share/vdsm/API.py", line 1436, in getExternalVMs
    return v2v.get_external_vms(uri, username, password)
  File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms
    root = ET.fromstring(vm.XMLDesc(0))
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in XMLDesc
    if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed', dom=self)
libvirtError: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

It is possible that the failing vm was the one you want to import, but it is also possible that the vm you want to import was fine.

The following patch improve error handling, so we are not aborting the entire request if libvirt fail to return the domain xml for one of the vms.
https://gerrit.ovirt.org/46634/

Please test the patch and report if it works for you.

Nir

You may try to disable the cdrom device on that vm, which is probably useless now.

I have attached the vdsm.log to this email, should I also attach to the BZ I opened?

Yes, please attach it.

Hmm, this is a new issue, as the original issue is solved by the patch.

Please open a new bug for the libvirt error:

internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

Thanks,
Nir

(Adding back users at ovirt.org<mailto:users at ovirt.org>, since this thread may help others with same issue.)

Many thanks

Ian

From: Nir Soffer [mailto:nsoffer at redhat.com<mailto:nsoffer at redhat.com>]
Sent: 23 September 2015 00:04
To: Ian Fraser <Ian.Fraser at asm.org.uk<mailto:Ian.Fraser at asm.org.uk>>; Shahar Havivi <shavivi at redhat.com<mailto:shavivi at redhat.com>>
Cc: users at ovirt.org<mailto:users at ovirt.org>
Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5

Hi Ian,

Your import failed because looking up some disk failed. Unfortunately, we don't have enough information in the log abut this failure.

Because of incorrect error handling, this error failed the entire request, failing your import.

Patch [1] fixes the second problem. If is possible that with this patch listing the external vms will work and you will be able to import the vm, but it is also possible that the first error was significant and will fail the import later.

It would be useful if you test this patch and report if it works for you.

Would you open an ovirt bug for this issue, attaching the vdsm log?

[1] https://gerrit.ovirt.org/46540

Nir

On Tue, Sep 22, 2015 at 9:09 AM, Ian Fraser <Ian.Fraser at asm.org.uk<mailto:Ian.Fraser at asm.org.uk>> wrote:
Thanks Nir,

File attached.

From: Nir Soffer [mailto:nsoffer at redhat.com<mailto:nsoffer at redhat.com>]
Sent: 21 September 2015 23:16
To: Ian Fraser <Ian.Fraser at asm.org.uk<mailto:Ian.Fraser at asm.org.uk>>
Cc: users at ovirt.org<mailto:users at ovirt.org>
Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5

On Tue, Sep 22, 2015 at 12:14 AM, Ian Fraser <Ian.Fraser at asm.org.uk<mailto:Ian.Fraser at asm.org.uk>> wrote:
I did get a “VDSM <hostname> command failed: local variable 'capacity' referenced before assignment” error in the events I have just noticed, does that shed any more light?

This shed some light. Can you share the vdsm.log containing this error?

Look in /var/log/vdsm/vdsm.log*

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________
________________________________________

The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments.

For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender.

Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced.
Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
Registered in England No.2053849

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

________________________________

The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments.

For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender.

Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced.
Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
Registered in England No.2053849

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

________________________________

The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments.

For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender.

Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced.
Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
Registered in England No.2053849

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150924/74b2f686/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: vdsm.zip
Type: application/x-zip-compressed
Size: 460528 bytes
Desc: vdsm.zip
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150924/74b2f686/attachment-0001.bin>