[ovirt-users] vmware import hangs after click load button on 3.5 rc5

[Nir Soffer]

On Thu, Sep 24, 2015 at 5:24 PM, Ian Fraser <Ian.Fraser at asm.org.uk> wrote:

> Hi Nir,
>
>
>
> https://gerrit.ovirt.org/46634/ has improved the situation, even tho the
> “Import Virtual Machine(s)” window still hangs. I queried all 3 vCenter
> Hosts then I run:
>
>
>
> grep -e v2v -e libvirtE -e ERROR /var/log/vdsm/vdsm.log
>
>
>
> I get the following much more helpful information:
>
>
>
> Thread-19193::ERROR::2015-09-24
> 13:05:20,542::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'devsquid': internal error: Invalid or not yet handled value
> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
> 'cdrom-image'
>
> Thread-19193::ERROR::2015-09-24
> 13:05:23,975::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'HD2012': internal error: Expecting VMX entry 'ethernet0.virtualDev'
> to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
>
> Thread-19193::ERROR::2015-09-24
> 13:05:41,102::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'QA11 2012 R2': internal error: Expecting VMX entry
> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
> but found 'e1000e'
>
> Thread-19193::ERROR::2015-09-24
> 13:05:43,643::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'HDWin8': internal error: Expecting VMX entry 'ethernet0.virtualDev'
> to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
>
> Thread-19193::ERROR::2015-09-24
> 13:05:45,861::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'HDwin10-2': internal error: Expecting VMX entry
> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
> but found 'e1000e'
>
> Thread-19193::ERROR::2015-09-24
> 13:05:51,421::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'HDWin10': internal error: Expecting VMX entry
> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
> but found 'e1000e'
>
> Thread-19193::ERROR::2015-09-24
> 13:06:05,973::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'wsus-ash-01': internal error: Expecting VMX entry
> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
> but found 'e1000e'
>
> Thread-19193::ERROR::2015-09-24
> 13:06:06,841::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'Hactar2': internal error: Expecting VMX entry 'numvcpus' to be an
> unsigned integer (1 or a multiple of 2) but found 3
>
> Thread-19193::ERROR::2015-09-24
> 13:06:21,972::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'squid3.2': internal error: Invalid or not yet handled value
> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
> 'cdrom-image'
>
> Thread-19382::ERROR::2015-09-24
> 13:06:45,322::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'sequoia-demo-01': internal error: Expecting VMX entry
> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
> but found 'e1000e'
>
> Thread-19382::ERROR::2015-09-24
> 13:06:47,515::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'sales-demo-01': internal error: Expecting VMX entry
> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
> but found 'e1000e'
>
> Thread-19382::ERROR::2015-09-24
> 13:06:50,535::v2v::158::root::(get_external_vms) error getting domain xml
> for vm 'rdpgw-demo-01': internal error: Expecting VMX entry
> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
> but found 'e1000e'
>
> Thread-19382::ERROR::2015-09-24
> 13:06:54,202::v2v::688::root::(_add_disk_info) Error getting disk size
>
>   File "/usr/share/vdsm/v2v.py", line 686, in _add_disk_info
>
>     if ret is None: raise libvirtError ('virStorageVolGetInfo() failed',
> vol=self)
>
> libvirtError: internal error: Could not search in datastore
> 'MD3220i-LUN0': FileNotFound - File [MD3220i-LUN0] isostore was not found
>

Thanks for testing the patch. This info should be very useful to libvirt
guys.


> Looks like it is continuing with some errors but still getting tripped on
> others. (vdsm.log attached)
>

I will look at the log later - the fact that the engine front end still
hangs mean there is another bug hiding in the vdsm.


> Sadly none of these errors are showing in the events in the front end
> which would be more helpful.
>

Please open a bug about that - the ui should display every vm found, with
error information for those cannot be imported.


>
>
> I have been playing divide and conquer with my 80 VMs and it seems there
> are still things that libvirt doesn’t like about some of those VMs. Is
> there a way I can run these commands from the command line so I can pin the
> problems down and alert the libvirt guys directly?
>

Yes, use:

vdsClient -s 0 getExternalVMs <uri> <username> <password>

You should see the uri format sent by the engine in vdsm logs.


> If so where would I find what commands are being run? Should I still
> create BZ against ovirt for these (one for each different problem or one
> for them all)?
>

Yes, please open a libvirt bug for these errors.

I talked with some libvirty guys, and they suggested to post about this on
the libvirt users list. This is most likely to get the attention of
developers that have worked on the vmware driver in the past.

Thanks again for your help
>
>
>
> Ian
>
>
>
> *From:* Nir Soffer [mailto:nsoffer at redhat.com]
> *Sent:* 23 September 2015 19:15
> *To:* Ian Fraser <Ian.Fraser at asm.org.uk>; users <users at ovirt.org>; Shahar
> Havivi <shavivi at redhat.com>
>
> *Subject:* Re: [ovirt-users] vmware import hangs after click load button
> on 3.5 rc5
>
>
>
> On Wed, Sep 23, 2015 at 8:26 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>
> On Wed, Sep 23, 2015 at 8:21 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>
> On Wed, Sep 23, 2015 at 12:40 PM, Ian Fraser <Ian.Fraser at asm.org.uk>
> wrote:
>
> Hi Nir,
>
> I have changed the v2v.py file as per your request, it has now changed the
> behaviour. The popup window still hangs and I get the following two events:
>
> Failed to retrieve VMs information from external server
> vpx://username%40domain@vcenter.server
> /datacentre_name/hostname?no_verify=1
>
> VDSM ovirt-host-02 command failed: internal error: Invalid or not yet
> handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for
> device type 'cdrom-image'
>
>
>
> Fixing the first error, we see now the real error; libvirt cannot handle
> this vm configuration. We will ask one of the libvirt guys to look into
> this.
>
>
>
> In vdsm.log, we see that one vm was parsed successfully (it failed when
> geting disk info):
>
>
>
> Thread-1076::ERROR::2015-09-23
> 10:18:51,778::v2v::682::root::(_add_disk_info) Error getting disk size
>
> Traceback (most recent call last):
>
>   File "/usr/share/vdsm/v2v.py", line 679, in _add_disk_info
>
>     vol = conn.storageVolLookupByPath(disk['alias'])
>
>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4308, in
> storageVolLookupByPath
>
>     if ret is None:raise libvirtError('virStorageVolLookupByPath()
> failed', conn=self)
>
> libvirtError: An error occurred, but the cause is unknown
>
>
>
> Little later, libvirt failed to return the xml for another vm:
>
>
>
> Thread-1076::ERROR::2015-09-23
> 10:18:52,604::__init__::526::jsonrpc.JsonRpcServer::(_serveRequest)
> Internal server error
>
> Traceback (most recent call last):
>
>   File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 521,
> in _serveRequest
>
>     res = method(**params)
>
>   File "/usr/share/vdsm/rpc/Bridge.py", line 273, in _dynamicMethod
>
>     result = fn(*methodArgs)
>
>   File "/usr/share/vdsm/API.py", line 1436, in getExternalVMs
>
>     return v2v.get_external_vms(uri, username, password)
>
>   File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms
>
>     root = ET.fromstring(vm.XMLDesc(0))
>
>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in
> XMLDesc
>
>     if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed',
> dom=self)
>
> libvirtError: internal error: Invalid or not yet handled value
> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
> 'cdrom-image'
>
>
>
> It is possible that the failing vm was the one you want to import, but it
> is also possible that the vm you want to import was fine.
>
>
>
> The following patch improve error handling, so we are not aborting the
> entire request if libvirt fail to return the domain xml for one of the vms.
>
> https://gerrit.ovirt.org/46634/
>
>
>
> Please test the patch and report if it works for you.
>
>
>
> Nir
>
>
>
>
>
> You may try to disable the cdrom device on that vm, which is probably
> useless now.
>
>
>
>
> I have attached the vdsm.log to this email, should I also attach to the BZ
> I opened?
>
>
>
> Yes, please attach it.
>
>
>
> Hmm, this is a new issue, as the original issue is solved by the patch.
>
>
>
> Please open a new bug for the libvirt error:
>
>
>
> internal error: Invalid or not yet handled value 'emptyBackingString' for
> VMX entry 'ide1:0.fileName' for device type 'cdrom-image'
>
>
>
> Thanks,
>
> Nir
>
>
>
>
>
> (Adding back users at ovirt.org, since this thread may help others with same
> issue.)
>
>
>
>
> Many thanks
>
> Ian
>
> From: Nir Soffer [mailto:nsoffer at redhat.com]
> Sent: 23 September 2015 00:04
> To: Ian Fraser <Ian.Fraser at asm.org.uk>; Shahar Havivi <shavivi at redhat.com>
>
> Cc: users at ovirt.org
> Subject: Re: [ovirt-users] vmware import hangs after click load button on
> 3.5 rc5
>
> Hi Ian,
>
> Your import failed because looking up some disk failed. Unfortunately, we
> don't have enough information in the log abut this failure.
>
> Because of incorrect error handling, this error failed the entire request,
> failing your import.
>
> Patch [1] fixes the second problem. If is possible that with this patch
> listing the external vms will work and you will be able to import the vm,
> but it is also possible that the first error was significant and will fail
> the import later.
>
> It would be useful if you test this patch and report if it works for you.
>
> Would you open an ovirt bug for this issue, attaching the vdsm log?
>
> [1] https://gerrit.ovirt.org/46540
>
> Nir
>
> On Tue, Sep 22, 2015 at 9:09 AM, Ian Fraser <Ian.Fraser at asm.org.uk> wrote:
> Thanks Nir,
>
> File attached.
>
> From: Nir Soffer [mailto:nsoffer at redhat.com]
> Sent: 21 September 2015 23:16
> To: Ian Fraser <Ian.Fraser at asm.org.uk>
> Cc: users at ovirt.org
> Subject: Re: [ovirt-users] vmware import hangs after click load button on
> 3.5 rc5
>
> On Tue, Sep 22, 2015 at 12:14 AM, Ian Fraser <Ian.Fraser at asm.org.uk>
> wrote:
> I did get a “VDSM <hostname> command failed: local variable 'capacity'
> referenced before assignment” error in the events I have just noticed, does
> that shed any more light?
>
> This shed some light. Can you share the vdsm.log containing this error?
>
> Look in /var/log/vdsm/vdsm.log*
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
> ________________________________________
>
>
> The information in this message and any attachment is intended for the
> addressee and is confidential. If you are not that addressee, no action
> should be taken in reliance on the information and you should please reply
> to this message immediately to inform us of incorrect receipt and destroy
> this message and any attachments.
>
> For the purposes of internet level email security incoming and outgoing
> emails may be read by personnel other than the named recipient or sender.
>
> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that
> emails and attachments are virus free or compatible with your systems. You
> should make your own checks and ASM (UK) Ltd does not accept liability in
> respect of viruses or computer problems experienced.
> Registered address: Agency Sector Management (UK) Ltd. Ashford House,
> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
> Registered in England No.2053849
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
> ________________________________
>
> The information in this message and any attachment is intended for the
> addressee and is confidential. If you are not that addressee, no action
> should be taken in reliance on the information and you should please reply
> to this message immediately to inform us of incorrect receipt and destroy
> this message and any attachments.
>
> For the purposes of internet level email security incoming and outgoing
> emails may be read by personnel other than the named recipient or sender.
>
> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that
> emails and attachments are virus free or compatible with your systems. You
> should make your own checks and ASM (UK) Ltd does not accept liability in
> respect of viruses or computer problems experienced.
> Registered address: Agency Sector Management (UK) Ltd. Ashford House,
> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
> Registered in England No.2053849
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
>
>
>
>
>
>
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
> ------------------------------
>
> The information in this message and any attachment is intended for the
> addressee and is confidential. If you are not that addressee, no action
> should be taken in reliance on the information and you should please reply
> to this message immediately to inform us of incorrect receipt and destroy
> this message and any attachments.
>
> For the purposes of internet level email security incoming and outgoing
> emails may be read by personnel other than the named recipient or sender.
>
> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that
> emails and attachments are virus free or compatible with your systems. You
> should make your own checks and ASM (UK) Ltd does not accept liability in
> respect of viruses or computer problems experienced.
> Registered address: Agency Sector Management (UK) Ltd. Ashford House,
> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
> Registered in England No.2053849
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150924/52a69548/attachment-0001.html>