On Thu, Sep 24, 2015 at 6:26 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Thu, Sep 24, 2015 at 5:24 PM, Ian Fraser <Ian.Fraser@asm.org.uk> wrote:

Hi Nir,

 

https://gerrit.ovirt.org/46634/ has improved the situation, even tho the “Import Virtual Machine(s)” window still hangs. I queried all 3 vCenter Hosts then I run:

 

grep -e v2v -e libvirtE -e ERROR /var/log/vdsm/vdsm.log

 

I get the following much more helpful information:

 

Thread-19193::ERROR::2015-09-24 13:05:20,542::v2v::158::root::(get_external_vms) error getting domain xml for vm 'devsquid': internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

Thread-19193::ERROR::2015-09-24 13:05:23,975::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HD2012': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19193::ERROR::2015-09-24 13:05:41,102::v2v::158::root::(get_external_vms) error getting domain xml for vm 'QA11 2012 R2': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19193::ERROR::2015-09-24 13:05:43,643::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDWin8': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19193::ERROR::2015-09-24 13:05:45,861::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDwin10-2': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19193::ERROR::2015-09-24 13:05:51,421::v2v::158::root::(get_external_vms) error getting domain xml for vm 'HDWin10': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19193::ERROR::2015-09-24 13:06:05,973::v2v::158::root::(get_external_vms) error getting domain xml for vm 'wsus-ash-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19193::ERROR::2015-09-24 13:06:06,841::v2v::158::root::(get_external_vms) error getting domain xml for vm 'Hactar2': internal error: Expecting VMX entry 'numvcpus' to be an unsigned integer (1 or a multiple of 2) but found 3

Thread-19193::ERROR::2015-09-24 13:06:21,972::v2v::158::root::(get_external_vms) error getting domain xml for vm 'squid3.2': internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

Thread-19382::ERROR::2015-09-24 13:06:45,322::v2v::158::root::(get_external_vms) error getting domain xml for vm 'sequoia-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19382::ERROR::2015-09-24 13:06:47,515::v2v::158::root::(get_external_vms) error getting domain xml for vm 'sales-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19382::ERROR::2015-09-24 13:06:50,535::v2v::158::root::(get_external_vms) error getting domain xml for vm 'rdpgw-demo-01': internal error: Expecting VMX entry 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e'

Thread-19382::ERROR::2015-09-24 13:06:54,202::v2v::688::root::(_add_disk_info) Error getting disk size

  File "/usr/share/vdsm/v2v.py", line 686, in _add_disk_info

    if ret is None: raise libvirtError ('virStorageVolGetInfo() failed', vol=self)

libvirtError: internal error: Could not search in datastore 'MD3220i-LUN0': FileNotFound - File [MD3220i-LUN0] isostore was not found


Thanks for testing the patch. This info should be very useful to libvirt guys.
 

Looks like it is continuing with some errors but still getting tripped on others. (vdsm.log attached)


I will look at the log later - the fact that the engine front end still hangs mean there is another bug hiding in the vdsm.

Looking in the log, we see:

1. Request started

Thread-19382::DEBUG::2015-09-24 13:06:41,079::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getExternalVMs' in bridge with {u'username': u'ian_admin@asm.org.uk', u'password': '********', u'
uri': u'vpx://ian_admin%40asm.org.uk@vcenter.asm.org.uk/Ashford_House/vh-ash-03.asm.org.uk?no_verify=1'}

2. (Lot of errors...)

3. Request ends

Thread-19382::DEBUG::2015-09-24 13:06:57,237::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getExternalVMs' in bridge with [{'status': 'Up', 'arch': 'i686', 'disks': [{'allocation': '16096690176', 'alias': '[CS215] remotesupport/remotesupport.vmdk', 'capacity': '16106127360', 'type': 'disk', 'dev': 'sda'}], 'vmId': '4204d4fc-60d9-ea0f-1130-48d05cc81abb', 'smp': 1, 'memSize': 256, 'vmName': 'remotesupport', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:32:29', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'i686', 'disks': [{'allocation': '80492888064', 'alias': "[CS215] sharon's Desktop/sharon's Desktop.vmdk", 'capacity': '80530636800', 'type': 'disk', 'dev': 'sda'}], 'vmId': '4204abe1-c816-ba3c-126d-6129941a2d6c', 'smp': 2, 'memSize': 4096, 'vmName': "sharon's Desktop", 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:31:c6', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '64364740608', 'alias': '[CS215] QA8/QA8.vmdk', 'capacity': '64436843520', 'type': 'disk', 'dev': 'sda'}], 'vmId': '42043e44-82f2-7e5a-2cb3-59d2d52b0cda', 'smp': 2, 'memSize': 8192, 'vmName': 'QA8', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:00:14', 'type': 'bridge'}, {'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:00:15', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '27928821760', 'alias': '[CS215] Fenchurch/Fenchurch_3.vmdk', 'capacity': '37580963840', 'type': 'disk', 'dev': 'sda'}, {'allocation': '51758759936', 'alias': '[CS215] Fenchurch/Fenchurch_1.vmdk', 'capacity': '107389255680', 'type': 'disk', 'dev': 'sdb'}, {'allocation': '10485760', 'alias': '[CS215] Fenchurch/Fenchurch_2.vmdk', 'capacity': '10485760', 'type': 'disk', 'dev': 'sdq'}], 'vmId': '4204de6a-185e-33b2-de96-d7938a66a2af', 'smp': 1, 'memSize': 2048, 'vmName': 'Fenchurch', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'e1000', 'macAddr': '00:50:56:84:00:12', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'i686', 'disks': [{'allocation': '32211206144', 'alias': '[CS215] QA7/QA7.vmdk', 'capacity': '32212254720', 'type': 'disk', 'dev': 'sda'}, {'allocation': '0', 'alias': '[CS215] QA7/QA7_1.vmdk', 'capacity': '21474836480', 'type': 'disk', 'dev': 'sdb'}], 'vmId': '42040c13-0291-ad9f-9a3e-53c3b897a0a8', 'smp': 1, 'memSize': 1024, 'vmName': 'QA7', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:50:89', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '8154775552', 'alias': '[CS215] QA9/QA9-000002.vmdk', 'capacity': '42949672960', 'type': 'disk', 'dev': 'sda'}, {'allocation': '588251136', 'alias': '[CS215] QA9/QA9_7-000001.vmdk', 'capacity': '42949672960', 'type': 'disk', 'dev': 'sdb'}, {'alias': '[MD3220i-LUN0] isostore/trinity-rescue-kit.3.4-build-372.iso', 'type': 'cdrom', 'dev': 'hdc'}], 'vmId': '42045216-fadd-96dc-84d3-586ac3a8c173', 'smp': 2, 'memSize': 2048, 'vmName': 'QA9', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:1d:01', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '19641925632', 'alias': '[CS215] QADB/QADB.vmdk', 'capacity': '26843545600', 'type': 'disk', 'dev': 'sda'}, {'allocation': '107373133824', 'alias': '[CS215] QADB/QADB_1.vmdk', 'capacity': '107374182400', 'type': 'disk', 'dev': 'sdb'}], 'vmId': '42042f12-6654-3e58-fae2-914f355747fa', 'smp': 2, 'memSize': 4096, 'vmName': 'QADB', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'e1000', 'macAddr': '00:50:56:84:6f:bd', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '42601545728', 'alias': '[CS215] QA10/QA10.vmdk', 'capacity': '42949672960', 'type': 'disk', 'dev': 'sda'}], 'vmId': '4204307f-4c05-c8ef-351d-ec3885b3341f', 'smp': 2, 'memSize': 4096, 'vmName': 'QA10', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:50:80', 'type': 'bridge'}]}]
 
So vdsm was successful, returning list of vms to engine - do you see the vm you want to import in this list?

If engine is ui is stuck, there is probably be a bug in the engine side, or maybe vdsm and engine have different views on what should be returned.

You can open an ovirt-engine bug with this log and engine log from the same timeframe. I guess you will find there big traceback which can explain why the ui get stuck.

Arik, can you take a look at this?

 

Sadly none of these errors are showing in the events in the front end which would be more helpful.


Please open a bug about that - the ui should display every vm found, with error information for those cannot be imported.
 

 

I have been playing divide and conquer with my 80 VMs and it seems there are still things that libvirt doesn’t like about some of those VMs. Is there a way I can run these commands from the command line so I can pin the problems down and alert the libvirt guys directly?


Yes, use:

vdsClient -s 0 getExternalVMs <uri> <username> <password>

You should see the uri format sent by the engine in vdsm logs.
 

If so where would I find what commands are being run? Should I still create BZ against ovirt for these (one for each different problem or one for them all)?


Yes, please open a libvirt bug for these errors.

I talked with some libvirty guys, and they suggested to post about this on the libvirt users list. This is most likely to get the attention of developers that have worked on the vmware driver in the past.

Thanks again for your help

 

Ian

 

From: Nir Soffer [mailto:nsoffer@redhat.com]
Sent: 23 September 2015 19:15
To: Ian Fraser <Ian.Fraser@asm.org.uk>; users <users@ovirt.org>; Shahar Havivi <shavivi@redhat.com>


Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5

 

On Wed, Sep 23, 2015 at 8:26 PM, Nir Soffer <nsoffer@redhat.com> wrote:

On Wed, Sep 23, 2015 at 8:21 PM, Nir Soffer <nsoffer@redhat.com> wrote:

On Wed, Sep 23, 2015 at 12:40 PM, Ian Fraser <Ian.Fraser@asm.org.uk> wrote:

Hi Nir,

I have changed the v2v.py file as per your request, it has now changed the behaviour. The popup window still hangs and I get the following two events:

Failed to retrieve VMs information from external server vpx://username%40domain@vcenter.server/datacentre_name/hostname?no_verify=1

VDSM ovirt-host-02 command failed: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

 

Fixing the first error, we see now the real error; libvirt cannot handle this vm configuration. We will ask one of the libvirt guys to look into this.

 

In vdsm.log, we see that one vm was parsed successfully (it failed when geting disk info):

 

Thread-1076::ERROR::2015-09-23 10:18:51,778::v2v::682::root::(_add_disk_info) Error getting disk size

Traceback (most recent call last):

  File "/usr/share/vdsm/v2v.py", line 679, in _add_disk_info

    vol = conn.storageVolLookupByPath(disk['alias'])

  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4308, in storageVolLookupByPath

    if ret is None:raise libvirtError('virStorageVolLookupByPath() failed', conn=self)

libvirtError: An error occurred, but the cause is unknown

 

Little later, libvirt failed to return the xml for another vm:

 

Thread-1076::ERROR::2015-09-23 10:18:52,604::__init__::526::jsonrpc.JsonRpcServer::(_serveRequest) Internal server error

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 521, in _serveRequest

    res = method(**params)

  File "/usr/share/vdsm/rpc/Bridge.py", line 273, in _dynamicMethod

    result = fn(*methodArgs)

  File "/usr/share/vdsm/API.py", line 1436, in getExternalVMs

    return v2v.get_external_vms(uri, username, password)

  File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms

    root = ET.fromstring(vm.XMLDesc(0))

  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in XMLDesc

    if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed', dom=self)

libvirtError: internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

 

It is possible that the failing vm was the one you want to import, but it is also possible that the vm you want to import was fine.

 

The following patch improve error handling, so we are not aborting the entire request if libvirt fail to return the domain xml for one of the vms.

 

Please test the patch and report if it works for you.

 

Nir

 

 

You may try to disable the cdrom device on that vm, which is probably useless now.

 


I have attached the vdsm.log to this email, should I also attach to the BZ I opened?

 

Yes, please attach it.

 

Hmm, this is a new issue, as the original issue is solved by the patch.

 

Please open a new bug for the libvirt error:

 

internal error: Invalid or not yet handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type 'cdrom-image'

 

Thanks,

Nir

 

 

(Adding back users@ovirt.org, since this thread may help others with same issue.)

 


Many thanks

Ian

From: Nir Soffer [mailto:nsoffer@redhat.com]
Sent: 23 September 2015 00:04
To: Ian Fraser <Ian.Fraser@asm.org.uk>; Shahar Havivi <shavivi@redhat.com>

Cc: users@ovirt.org
Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5

Hi Ian,

Your import failed because looking up some disk failed. Unfortunately, we don't have enough information in the log abut this failure.

Because of incorrect error handling, this error failed the entire request, failing your import.

Patch [1] fixes the second problem. If is possible that with this patch listing the external vms will work and you will be able to import the vm, but it is also possible that the first error was significant and will fail the import later.

It would be useful if you test this patch and report if it works for you.

Would you open an ovirt bug for this issue, attaching the vdsm log?

[1] https://gerrit.ovirt.org/46540

Nir

On Tue, Sep 22, 2015 at 9:09 AM, Ian Fraser <Ian.Fraser@asm.org.uk> wrote:
Thanks Nir,

File attached.

From: Nir Soffer [mailto:nsoffer@redhat.com]
Sent: 21 September 2015 23:16
To: Ian Fraser <Ian.Fraser@asm.org.uk>
Cc: users@ovirt.org
Subject: Re: [ovirt-users] vmware import hangs after click load button on 3.5 rc5

On Tue, Sep 22, 2015 at 12:14 AM, Ian Fraser <Ian.Fraser@asm.org.uk> wrote:
I did get a “VDSM <hostname> command failed: local variable 'capacity' referenced before assignment” error in the events I have just noticed, does that shed any more light?

This shed some light. Can you share the vdsm.log containing this error?

Look in /var/log/vdsm/vdsm.log*

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

________________________________________


The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments.

For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender.

Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced.
Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
Registered in England No.2053849

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

________________________________

The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments.

For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender.

Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced.
Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
Registered in England No.2053849

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________

 

 

 


______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________



The information in this message and any attachment is intended for the addressee and is confidential. If you are not that addressee, no action should be taken in reliance on the information and you should please reply to this message immediately to inform us of incorrect receipt and destroy this message and any attachments.

For the purposes of internet level email security incoming and outgoing emails may be read by personnel other than the named recipient or sender.

Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee that emails and attachments are virus free or compatible with your systems. You should make your own checks and ASM (UK) Ltd does not accept liability in respect of viruses or computer problems experienced.
Registered address: Agency Sector Management (UK) Ltd. Ashford House, 41-45 Church Road, Ashford, Middlesex, TW15 2TQ
Registered in England No.2053849
______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________