[ovirt-users] vmware import hangs after click load button on 3.5 rc5

[Nir Soffer]

On Thu, Sep 24, 2015 at 6:26 PM, Nir Soffer <nsoffer at redhat.com> wrote:

> On Thu, Sep 24, 2015 at 5:24 PM, Ian Fraser <Ian.Fraser at asm.org.uk> wrote:
>
>> Hi Nir,
>>
>>
>>
>> https://gerrit.ovirt.org/46634/ has improved the situation, even tho the
>> “Import Virtual Machine(s)” window still hangs. I queried all 3 vCenter
>> Hosts then I run:
>>
>>
>>
>> grep -e v2v -e libvirtE -e ERROR /var/log/vdsm/vdsm.log
>>
>>
>>
>> I get the following much more helpful information:
>>
>>
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:05:20,542::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'devsquid': internal error: Invalid or not yet handled value
>> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
>> 'cdrom-image'
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:05:23,975::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'HD2012': internal error: Expecting VMX entry 'ethernet0.virtualDev'
>> to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:05:41,102::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'QA11 2012 R2': internal error: Expecting VMX entry
>> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
>> but found 'e1000e'
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:05:43,643::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'HDWin8': internal error: Expecting VMX entry 'ethernet0.virtualDev'
>> to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:05:45,861::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'HDwin10-2': internal error: Expecting VMX entry
>> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
>> but found 'e1000e'
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:05:51,421::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'HDWin10': internal error: Expecting VMX entry
>> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
>> but found 'e1000e'
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:06:05,973::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'wsus-ash-01': internal error: Expecting VMX entry
>> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
>> but found 'e1000e'
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:06:06,841::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'Hactar2': internal error: Expecting VMX entry 'numvcpus' to be an
>> unsigned integer (1 or a multiple of 2) but found 3
>>
>> Thread-19193::ERROR::2015-09-24
>> 13:06:21,972::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'squid3.2': internal error: Invalid or not yet handled value
>> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
>> 'cdrom-image'
>>
>> Thread-19382::ERROR::2015-09-24
>> 13:06:45,322::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'sequoia-demo-01': internal error: Expecting VMX entry
>> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
>> but found 'e1000e'
>>
>> Thread-19382::ERROR::2015-09-24
>> 13:06:47,515::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'sales-demo-01': internal error: Expecting VMX entry
>> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
>> but found 'e1000e'
>>
>> Thread-19382::ERROR::2015-09-24
>> 13:06:50,535::v2v::158::root::(get_external_vms) error getting domain xml
>> for vm 'rdpgw-demo-01': internal error: Expecting VMX entry
>> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000'
>> but found 'e1000e'
>>
>> Thread-19382::ERROR::2015-09-24
>> 13:06:54,202::v2v::688::root::(_add_disk_info) Error getting disk size
>>
>>   File "/usr/share/vdsm/v2v.py", line 686, in _add_disk_info
>>
>>     if ret is None: raise libvirtError ('virStorageVolGetInfo() failed',
>> vol=self)
>>
>> libvirtError: internal error: Could not search in datastore
>> 'MD3220i-LUN0': FileNotFound - File [MD3220i-LUN0] isostore was not found
>>
>
> Thanks for testing the patch. This info should be very useful to libvirt
> guys.
>
>
>> Looks like it is continuing with some errors but still getting tripped on
>> others. (vdsm.log attached)
>>
>
> I will look at the log later - the fact that the engine front end still
> hangs mean there is another bug hiding in the vdsm.
>

Looking in the log, we see:

1. Request started

Thread-19382::DEBUG::2015-09-24
13:06:41,079::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling
'Host.getExternalVMs' in bridge with {u'username': u'ian_admin at asm.org.uk',
u'password': '********', u'
uri': u'vpx://
ian_admin%40asm.org.uk at vcenter.asm.org.uk/Ashford_House/vh-ash-03.asm.org.uk?no_verify=1
'}

2. (Lot of errors...)

3. Request ends

Thread-19382::DEBUG::2015-09-24
13:06:57,237::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return
'Host.getExternalVMs' in bridge with [{'status': 'Up', 'arch': 'i686',
'disks': [{'allocation': '16096690176', 'alias': '[CS215]
remotesupport/remotesupport.vmdk', 'capacity': '16106127360', 'type':
'disk', 'dev': 'sda'}], 'vmId': '4204d4fc-60d9-ea0f-1130-48d05cc81abb',
'smp': 1, 'memSize': 256, 'vmName': 'remotesupport', 'networks':
[{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr':
'00:50:56:84:32:29', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'i686',
'disks': [{'allocation': '80492888064', 'alias': "[CS215] sharon's
Desktop/sharon's Desktop.vmdk", 'capacity': '80530636800', 'type': 'disk',
'dev': 'sda'}], 'vmId': '4204abe1-c816-ba3c-126d-6129941a2d6c', 'smp': 2,
'memSize': 4096, 'vmName': "sharon's Desktop", 'networks': [{'bridge': 'ASM
Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:31:c6',
'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks':
[{'allocation': '64364740608', 'alias': '[CS215] QA8/QA8.vmdk', 'capacity':
'64436843520', 'type': 'disk', 'dev': 'sda'}], 'vmId':
'42043e44-82f2-7e5a-2cb3-59d2d52b0cda', 'smp': 2, 'memSize': 8192,
'vmName': 'QA8', 'networks': [{'bridge': 'ASM Internal Network', 'model':
'vmxnet3', 'macAddr': '00:50:56:84:00:14', 'type': 'bridge'}, {'bridge':
'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:00:15',
'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks':
[{'allocation': '27928821760', 'alias': '[CS215]
Fenchurch/Fenchurch_3.vmdk', 'capacity': '37580963840', 'type': 'disk',
'dev': 'sda'}, {'allocation': '51758759936', 'alias': '[CS215]
Fenchurch/Fenchurch_1.vmdk', 'capacity': '107389255680', 'type': 'disk',
'dev': 'sdb'}, {'allocation': '10485760', 'alias': '[CS215]
Fenchurch/Fenchurch_2.vmdk', 'capacity': '10485760', 'type': 'disk', 'dev':
'sdq'}], 'vmId': '4204de6a-185e-33b2-de96-d7938a66a2af', 'smp': 1,
'memSize': 2048, 'vmName': 'Fenchurch', 'networks': [{'bridge': 'ASM
Internal Network', 'model': 'e1000', 'macAddr': '00:50:56:84:00:12',
'type': 'bridge'}]}, {'status': 'Up', 'arch': 'i686', 'disks':
[{'allocation': '32211206144', 'alias': '[CS215] QA7/QA7.vmdk', 'capacity':
'32212254720', 'type': 'disk', 'dev': 'sda'}, {'allocation': '0', 'alias':
'[CS215] QA7/QA7_1.vmdk', 'capacity': '21474836480', 'type': 'disk', 'dev':
'sdb'}], 'vmId': '42040c13-0291-ad9f-9a3e-53c3b897a0a8', 'smp': 1,
'memSize': 1024, 'vmName': 'QA7', 'networks': [{'bridge': 'ASM Internal
Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:50:89', 'type':
'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation':
'8154775552', 'alias': '[CS215] QA9/QA9-000002.vmdk', 'capacity':
'42949672960', 'type': 'disk', 'dev': 'sda'}, {'allocation': '588251136',
'alias': '[CS215] QA9/QA9_7-000001.vmdk', 'capacity': '42949672960',
'type': 'disk', 'dev': 'sdb'}, {'alias': '[MD3220i-LUN0]
isostore/trinity-rescue-kit.3.4-build-372.iso', 'type': 'cdrom', 'dev':
'hdc'}], 'vmId': '42045216-fadd-96dc-84d3-586ac3a8c173', 'smp': 2,
'memSize': 2048, 'vmName': 'QA9', 'networks': [{'bridge': 'ASM Internal
Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:1d:01', 'type':
'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation':
'19641925632', 'alias': '[CS215] QADB/QADB.vmdk', 'capacity':
'26843545600', 'type': 'disk', 'dev': 'sda'}, {'allocation':
'107373133824', 'alias': '[CS215] QADB/QADB_1.vmdk', 'capacity':
'107374182400', 'type': 'disk', 'dev': 'sdb'}], 'vmId':
'42042f12-6654-3e58-fae2-914f355747fa', 'smp': 2, 'memSize': 4096,
'vmName': 'QADB', 'networks': [{'bridge': 'ASM Internal Network', 'model':
'e1000', 'macAddr': '00:50:56:84:6f:bd', 'type': 'bridge'}]}, {'status':
'Up', 'arch': 'x86_64', 'disks': [{'allocation': '42601545728', 'alias':
'[CS215] QA10/QA10.vmdk', 'capacity': '42949672960', 'type': 'disk', 'dev':
'sda'}], 'vmId': '4204307f-4c05-c8ef-351d-ec3885b3341f', 'smp': 2,
'memSize': 4096, 'vmName': 'QA10', 'networks': [{'bridge': 'ASM Internal
Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:50:80', 'type':
'bridge'}]}]

So vdsm was successful, returning list of vms to engine - do you see the vm
you want to import in this list?

If engine is ui is stuck, there is probably be a bug in the engine side, or
maybe vdsm and engine have different views on what should be returned.

You can open an ovirt-engine bug with this log and engine log from the same
timeframe. I guess you will find there big traceback which can explain why
the ui get stuck.

Arik, can you take a look at this?


>
>> Sadly none of these errors are showing in the events in the front end
>> which would be more helpful.
>>
>
> Please open a bug about that - the ui should display every vm found, with
> error information for those cannot be imported.
>
>
>>
>>
>> I have been playing divide and conquer with my 80 VMs and it seems there
>> are still things that libvirt doesn’t like about some of those VMs. Is
>> there a way I can run these commands from the command line so I can pin the
>> problems down and alert the libvirt guys directly?
>>
>
> Yes, use:
>
> vdsClient -s 0 getExternalVMs <uri> <username> <password>
>
> You should see the uri format sent by the engine in vdsm logs.
>
>
>> If so where would I find what commands are being run? Should I still
>> create BZ against ovirt for these (one for each different problem or one
>> for them all)?
>>
>
> Yes, please open a libvirt bug for these errors.
>
> I talked with some libvirty guys, and they suggested to post about this on
> the libvirt users list. This is most likely to get the attention of
> developers that have worked on the vmware driver in the past.
>
> Thanks again for your help
>>
>>
>>
>> Ian
>>
>>
>>
>> *From:* Nir Soffer [mailto:nsoffer at redhat.com]
>> *Sent:* 23 September 2015 19:15
>> *To:* Ian Fraser <Ian.Fraser at asm.org.uk>; users <users at ovirt.org>;
>> Shahar Havivi <shavivi at redhat.com>
>>
>> *Subject:* Re: [ovirt-users] vmware import hangs after click load button
>> on 3.5 rc5
>>
>>
>>
>> On Wed, Sep 23, 2015 at 8:26 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>
>> On Wed, Sep 23, 2015 at 8:21 PM, Nir Soffer <nsoffer at redhat.com> wrote:
>>
>> On Wed, Sep 23, 2015 at 12:40 PM, Ian Fraser <Ian.Fraser at asm.org.uk>
>> wrote:
>>
>> Hi Nir,
>>
>> I have changed the v2v.py file as per your request, it has now changed
>> the behaviour. The popup window still hangs and I get the following two
>> events:
>>
>> Failed to retrieve VMs information from external server
>> vpx://username%40domain@vcenter.server
>> /datacentre_name/hostname?no_verify=1
>>
>> VDSM ovirt-host-02 command failed: internal error: Invalid or not yet
>> handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for
>> device type 'cdrom-image'
>>
>>
>>
>> Fixing the first error, we see now the real error; libvirt cannot handle
>> this vm configuration. We will ask one of the libvirt guys to look into
>> this.
>>
>>
>>
>> In vdsm.log, we see that one vm was parsed successfully (it failed when
>> geting disk info):
>>
>>
>>
>> Thread-1076::ERROR::2015-09-23
>> 10:18:51,778::v2v::682::root::(_add_disk_info) Error getting disk size
>>
>> Traceback (most recent call last):
>>
>>   File "/usr/share/vdsm/v2v.py", line 679, in _add_disk_info
>>
>>     vol = conn.storageVolLookupByPath(disk['alias'])
>>
>>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4308, in
>> storageVolLookupByPath
>>
>>     if ret is None:raise libvirtError('virStorageVolLookupByPath()
>> failed', conn=self)
>>
>> libvirtError: An error occurred, but the cause is unknown
>>
>>
>>
>> Little later, libvirt failed to return the xml for another vm:
>>
>>
>>
>> Thread-1076::ERROR::2015-09-23
>> 10:18:52,604::__init__::526::jsonrpc.JsonRpcServer::(_serveRequest)
>> Internal server error
>>
>> Traceback (most recent call last):
>>
>>   File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line
>> 521, in _serveRequest
>>
>>     res = method(**params)
>>
>>   File "/usr/share/vdsm/rpc/Bridge.py", line 273, in _dynamicMethod
>>
>>     result = fn(*methodArgs)
>>
>>   File "/usr/share/vdsm/API.py", line 1436, in getExternalVMs
>>
>>     return v2v.get_external_vms(uri, username, password)
>>
>>   File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms
>>
>>     root = ET.fromstring(vm.XMLDesc(0))
>>
>>   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in
>> XMLDesc
>>
>>     if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed',
>> dom=self)
>>
>> libvirtError: internal error: Invalid or not yet handled value
>> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type
>> 'cdrom-image'
>>
>>
>>
>> It is possible that the failing vm was the one you want to import, but it
>> is also possible that the vm you want to import was fine.
>>
>>
>>
>> The following patch improve error handling, so we are not aborting the
>> entire request if libvirt fail to return the domain xml for one of the vms.
>>
>> https://gerrit.ovirt.org/46634/
>>
>>
>>
>> Please test the patch and report if it works for you.
>>
>>
>>
>> Nir
>>
>>
>>
>>
>>
>> You may try to disable the cdrom device on that vm, which is probably
>> useless now.
>>
>>
>>
>>
>> I have attached the vdsm.log to this email, should I also attach to the
>> BZ I opened?
>>
>>
>>
>> Yes, please attach it.
>>
>>
>>
>> Hmm, this is a new issue, as the original issue is solved by the patch.
>>
>>
>>
>> Please open a new bug for the libvirt error:
>>
>>
>>
>> internal error: Invalid or not yet handled value 'emptyBackingString' for
>> VMX entry 'ide1:0.fileName' for device type 'cdrom-image'
>>
>>
>>
>> Thanks,
>>
>> Nir
>>
>>
>>
>>
>>
>> (Adding back users at ovirt.org, since this thread may help others with
>> same issue.)
>>
>>
>>
>>
>> Many thanks
>>
>> Ian
>>
>> From: Nir Soffer [mailto:nsoffer at redhat.com]
>> Sent: 23 September 2015 00:04
>> To: Ian Fraser <Ian.Fraser at asm.org.uk>; Shahar Havivi <shavivi at redhat.com
>> >
>>
>> Cc: users at ovirt.org
>> Subject: Re: [ovirt-users] vmware import hangs after click load button on
>> 3.5 rc5
>>
>> Hi Ian,
>>
>> Your import failed because looking up some disk failed. Unfortunately, we
>> don't have enough information in the log abut this failure.
>>
>> Because of incorrect error handling, this error failed the entire
>> request, failing your import.
>>
>> Patch [1] fixes the second problem. If is possible that with this patch
>> listing the external vms will work and you will be able to import the vm,
>> but it is also possible that the first error was significant and will fail
>> the import later.
>>
>> It would be useful if you test this patch and report if it works for you.
>>
>> Would you open an ovirt bug for this issue, attaching the vdsm log?
>>
>> [1] https://gerrit.ovirt.org/46540
>>
>> Nir
>>
>> On Tue, Sep 22, 2015 at 9:09 AM, Ian Fraser <Ian.Fraser at asm.org.uk>
>> wrote:
>> Thanks Nir,
>>
>> File attached.
>>
>> From: Nir Soffer [mailto:nsoffer at redhat.com]
>> Sent: 21 September 2015 23:16
>> To: Ian Fraser <Ian.Fraser at asm.org.uk>
>> Cc: users at ovirt.org
>> Subject: Re: [ovirt-users] vmware import hangs after click load button on
>> 3.5 rc5
>>
>> On Tue, Sep 22, 2015 at 12:14 AM, Ian Fraser <Ian.Fraser at asm.org.uk>
>> wrote:
>> I did get a “VDSM <hostname> command failed: local variable 'capacity'
>> referenced before assignment” error in the events I have just noticed, does
>> that shed any more light?
>>
>> This shed some light. Can you share the vdsm.log containing this error?
>>
>> Look in /var/log/vdsm/vdsm.log*
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>> ________________________________________
>>
>>
>> The information in this message and any attachment is intended for the
>> addressee and is confidential. If you are not that addressee, no action
>> should be taken in reliance on the information and you should please reply
>> to this message immediately to inform us of incorrect receipt and destroy
>> this message and any attachments.
>>
>> For the purposes of internet level email security incoming and outgoing
>> emails may be read by personnel other than the named recipient or sender.
>>
>> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee
>> that emails and attachments are virus free or compatible with your systems.
>> You should make your own checks and ASM (UK) Ltd does not accept liability
>> in respect of viruses or computer problems experienced.
>> Registered address: Agency Sector Management (UK) Ltd. Ashford House,
>> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
>> Registered in England No.2053849
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>> ________________________________
>>
>> The information in this message and any attachment is intended for the
>> addressee and is confidential. If you are not that addressee, no action
>> should be taken in reliance on the information and you should please reply
>> to this message immediately to inform us of incorrect receipt and destroy
>> this message and any attachments.
>>
>> For the purposes of internet level email security incoming and outgoing
>> emails may be read by personnel other than the named recipient or sender.
>>
>> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee
>> that emails and attachments are virus free or compatible with your systems.
>> You should make your own checks and ASM (UK) Ltd does not accept liability
>> in respect of viruses or computer problems experienced.
>> Registered address: Agency Sector Management (UK) Ltd. Ashford House,
>> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
>> Registered in England No.2053849
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>>
>>
>>
>>
>>
>>
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>> ------------------------------
>>
>> The information in this message and any attachment is intended for the
>> addressee and is confidential. If you are not that addressee, no action
>> should be taken in reliance on the information and you should please reply
>> to this message immediately to inform us of incorrect receipt and destroy
>> this message and any attachments.
>>
>> For the purposes of internet level email security incoming and outgoing
>> emails may be read by personnel other than the named recipient or sender.
>>
>> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee
>> that emails and attachments are virus free or compatible with your systems.
>> You should make your own checks and ASM (UK) Ltd does not accept liability
>> in respect of viruses or computer problems experienced.
>> Registered address: Agency Sector Management (UK) Ltd. Ashford House,
>> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
>> Registered in England No.2053849
>>
>> ______________________________________________________________________
>> This email has been scanned by the Symantec Email Security.cloud service.
>> For more information please visit http://www.symanteccloud.com
>> ______________________________________________________________________
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150924/25c931d0/attachment-0001.html>