Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5

Hi Nir, https://gerrit.ovirt.org/46634/ has improved the situation, even tho the “Import Virtual Machine(s)” window still hangs. I queried all 3 vCenter Hosts then I run: grep -e v2v -e libvirtE -e ERROR /var/log/vdsm/vdsm.log I get the following much more helpful information: Thread-19193::ERROR::2015-09-24 13:05:20,542::v2v::158::root::(get_external_vms) error getting domain xml for vm 'devsquid': internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image' Thread-19193::ERROR::2015-09-24 13:05:23,975::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HD2012': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19193::ERROR::2015-09-24 13:05:41,102::v2v::158::root::(get_external_vms) error getting domain xml for vm 'QA11 2012 R2': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19193::ERROR::2015-09-24 13:05:43,643::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDWin8': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19193::ERROR::2015-09-24 13:05:45,861::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDwin10-2': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19193::ERROR::2015-09-24 13:05:51,421::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDWin10': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19193::ERROR::2015-09-24 13:06:05,973::v2v::158::root::(get_external_vms) error getting domain xml for vm 'wsus-ash-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19193::ERROR::2015-09-24 13:06:06,841::v2v::158::root::(get_external_vms) error getting domain xml for vm 'Hactar2': internal error: Expecting VMX entry 'numvcpus' to be an unsigned integer (1 or a multiple of 2) but found 3 Thread-19193::ERROR::2015-09-24 13:06:21,972::v2v::158::root::(get_external_vms) error getting domain xml for vm 'squid3.2': internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image' Thread-19382::ERROR::2015-09-24 13:06:45,322::v2v::158::root::(get_external_vms) error getting domain xml for vm 'sequoia-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19382::ERROR::2015-09-24 13:06:47,515::v2v::158::root::(get_external_vms) error getting domain xml for vm 'sales-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19382::ERROR::2015-09-24 13:06:50,535::v2v::158::root::(get_external_vms) error getting domain xml for vm 'rdpgw-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' Thread-19382::ERROR::2015-09-24 13:06:54,202::v2v::688::root::(_add_disk_info) Error getting disk size File "/usr/share/vdsm/v2v.py", line 686, in _add_disk_info if ret is None: raise libvirtError ('virStorageVolGetInfo() failed', vol=self) libvirtError: internal error: Could not search in datastore 'MD3220i-LUN0': FileNotFound - File [MD3220i-LUN0] isostore was not found

Looks like it is continuing with some errors but still getting tripped on others. (vdsm.log attached)

Sadly none of these errors are showing in the events in the front end which would be more helpful.

I have been playing divide and conquer with my 80 VMs and it seems there are still things that libvirt doesn’t like about some of those VMs. Is there a way I can run these commands from the command line so I can pin the problems down and alert the libvirt guys directly?

If so where would I find what commands are being run? Should I still create BZ against ovirt for these (one for each different problem or one for them all)?

Ian *From:* Nir Soffer [mailto:nsoffer@redhat.com] *Sent:* 23 September 2015 19:15 *To:* Ian Fraser <Ian.Fraser(a)asm.org.uk>; users <users(a)ovirt.org>; Shahar Havivi <shavivi(a)redhat.com&gt; *Subject:* Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5 On Wed, Sep 23, 2015 at 8:26 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: On Wed, Sep 23, 2015 at 8:21 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: On Wed, Sep 23, 2015 at 12:40 PM, Ian Fraser <Ian.Fraser(a)asm.org.uk&gt; wrote: Hi Nir, I have changed the v2v.py file as per your request, it has now changed the behaviour. The popup window still hangs and I get the following two events: Failed to retrieve VMs information from external server vpx://username%40domain@vcenter.server /datacentre_name/hostname?no_verify=1 VDSM ovirt-host-02 command failed: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image' Fixing the first error, we see now the real error; libvirt cannot handle this vm configuration. We will ask one of the libvirt guys to look into this. In vdsm.log, we see that one vm was parsed successfully (it failed when geting disk info): Thread-1076::ERROR::2015-09-23 10:18:51,778::v2v::682::root::(_add_disk_info) Error getting disk size Traceback (most recent call last): File "/usr/share/vdsm/v2v.py", line 679, in _add_disk_info vol = conn.storageVolLookupByPath(disk['alias']) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4308, in storageVolLookupByPath if ret is None:raise libvirtError('virStorageVolLookupByPath() failed', conn=self) libvirtError: An error occurred, but the cause is unknown Little later, libvirt failed to return the xml for another vm: Thread-1076::ERROR::2015-09-23 10:18:52,604::__init__::526::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 521, in _serveRequest res = method(**params) File "/usr/share/vdsm/rpc/Bridge.py", line 273, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1436, in getExternalVMs return v2v.get_external_vms(uri, username, password) File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms root = ET.fromstring(vm.XMLDesc(0)) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in XMLDesc if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed', dom=self) libvirtError: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image' It is possible that the failing vm was the one you want to import, but it is also possible that the vm you want to import was fine. The following patch improve error handling, so we are not aborting the entire request if libvirt fail to return the domain xml for one of the vms. https://gerrit.ovirt.org/46634/ Please test the patch and report if it works for you. Nir You may try to disable the cdrom device on that vm, which is probably useless now. I have attached the vdsm.log to this email, should I also attach to the BZ I opened? Yes, please attach it. Hmm, this is a new issue, as the original issue is solved by the patch. Please open a new bug for the libvirt error: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image' Thanks, Nir (Adding back users(a)ovirt.org, since this thread may help others with same issue.) Many thanks Ian From: Nir Soffer [mailto:nsoffer@redhat.com] Sent: 23 September 2015 00:04 To: Ian Fraser <Ian.Fraser(a)asm.org.uk>; Shahar Havivi <shavivi(a)redhat.com&gt; Cc: users(a)ovirt.org Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5 Hi Ian, Your import failed because looking up some disk failed. Unfortunately, we don't have enough information in the log abut this failure. Because of incorrect error handling, this error failed the entire request, failing your import. Patch [1] fixes the second problem. If is possible that with this patch listing the external vms will work and you will be able to import the vm, but it is also possible that the first error was significant and will fail the import later. It would be useful if you test this patch and report if it works for you. Would you open an ovirt bug for this issue, attaching the vdsm log? [1] https://gerrit.ovirt.org/46540 Nir On Tue, Sep 22, 2015 at 9:09 AM, Ian Fraser <Ian.Fraser(a)asm.org.uk&gt; wrote: Thanks Nir, File attached. From: Nir Soffer [mailto:nsoffer@redhat.com] Sent: 21 September 2015 23:16 To: Ian Fraser <Ian.Fraser(a)asm.org.uk&gt; Cc: users(a)ovirt.org Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5 On Tue, Sep 22, 2015 at 12:14 AM, Ian Fraser <Ian.Fraser(a)asm.org.uk&gt; wrote: I did get a “VDSM <hostname> command failed: local variable 'capacity' referenced before assignment” error in the events I have just noticed, does that shed any more light? This shed some light. Can you share the vdsm.log containing this error? Look in /var/log/vdsm/vdsm.log* ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ________________________________________ The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments. For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender. Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced. Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ Registered in England No.2053849 ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ________________________________ The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments. For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender. Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced. Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ Registered in England No.2053849 ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________ ------------------------------ The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments. For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender. Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced. Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ Registered in England No.2053849 ______________________________________________________________________ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com ______________________________________________________________________