Dear Benny and others,

So it seems I wasn’t being patient with GDB and it does show me some output. This error of qemu-img convert even is failing and preventing updating ovirt-node version from 4.3.2 to 4.3.3.1. I get a feeling this is an unrelated error, but I thought I’d be complete:

Excuse any typos, im having to type this manually from a remote session, but the error:

[733272.427922] hid-generic 0003:0624:0249.0001: usb_submit_urb(ctrl) failed: -19

If this bug is preventing even a local yum updatei can’t see how it’s any issue other than somehow involved with the hardware of the hypervisor, our network and storage configuration must be irrelevant to this fact at this stage?

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 11 Apr 2019, at 12:00, Callum Smith <callum@well.ox.ac.uk> wrote:

Without the sudo and running in a  dir where the root has access to, gdb has zero output: 

<Screenshot 2019-04-11 at 12.00.27.png>
Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 11 Apr 2019, at 11:54, Callum Smith <callum@well.ox.ac.uk> wrote:

Some more information:

running qemu-img convert manually having captured the failed attempt from the previous:

sudo -u vdsm /usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/mnt/10.141.15.248:_export_instruct_vm__storage/0e01f014-530b-4067-aa1d-4e9378626a9d/images/6597eede-9fa0-4451-84fc-9f9c070cb5f3/765fa48b-2e77-4637-b4ca-e1affcd71e48 -O raw /rhev/data-center/mnt/10.141.15.248:_export_instruct_vm__storage/0e01f014-530b-4067-aa1d-4e9378626a9d/images/9cc99110-70a2-477f-b3ef-1031a912d12b/c2776107-4579-43a6-9d60-93a5ea9c64c5 -W

Added the -W flag just to see what would happen:

gdb -p 79913 -batch -ex "t a a bt"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f528d7661f0 in __poll_nocancel () from /lib64/libc.so.6

Thread 1 (Thread 0x7f528e6bb840 (LWP 79913)):
#0  0x00007f528d7661f0 in __poll_nocancel () from /lib64/libc.so.6
#1  0x00007f528dc510fb in sudo_ev_scan_impl () from /usr/libexec/sudo/libsudo_util.so.0
#2  0x00007f528dc49b44 in sudo_ev_loop_v1 () from /usr/libexec/sudo/libsudo_util.so.0
#3  0x000055e94aa0e271 in exec_nopty ()
#4  0x000055e94aa0afda in sudo_execute ()
#5  0x000055e94aa18a12 in run_command ()
#6  0x000055e94aa0969e in main ()


And without -W:

gdb -p 85235 -batch -ex "t a a bt"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007fc0cc69b1f0 in __poll_nocancel () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc0cd5f0840 (LWP 85235)):
#0  0x00007fc0cc69b1f0 in __poll_nocancel () from /lib64/libc.so.6
#1  0x00007fc0ccb860fb in sudo_ev_scan_impl () from /usr/libexec/sudo/libsudo_util.so.0
#2  0x00007fc0ccb7eb44 in sudo_ev_loop_v1 () from /usr/libexec/sudo/libsudo_util.so.0
#3  0x00005610f4397271 in exec_nopty ()
#4  0x00005610f4393fda in sudo_execute ()
#5  0x00005610f43a1a12 in run_command ()
#6  0x00005610f439269e in main ()

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 11 Apr 2019, at 09:57, Callum Smith <callum@well.ox.ac.uk> wrote:

Dear Benny,

It would seem that even cloning a VM is failing, creating a VM works on the same storage. This is the only error i could find:

ERROR Internal server error
                                                              Traceback (most recent call last):
                                                                File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request
                                                                  res = method(**params)
                                                                File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 194, in _dynamicMethod
                                                                  result = fn(*methodArgs)
                                                                File "<string>", line 2, in getAllVmStats
                                                                File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method
                                                                  ret = func(*args, **kwargs)
                                                                File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1388, in getAllVmStats
                                                                  statsList = self._cif.getAllVmStats()
                                                                File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 567, in getAllVmStats
                                                                  return [v.getStats() for v in self.vmContainer.values()]
                                                                File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1766, in getStats
                                                                  oga_stats = self._getGuestStats()
                                                                File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1967, in _getGuestStats
                                                                  stats = self.guestAgent.getGuestInfo()
                                                                File "/usr/lib/python2.7/site-packages/vdsm/virt/guestagent.py", line 505, in getGuestInfo
                                                                  del qga['appsList']
                                                              KeyError: 'appsList'


It's the qemu-img convert for sure that's just failing to do anything, this is the command from the clone:
/usr/bin/qemu-img convert -p -t none -T none -f raw /rhev/data-center/mnt/10.141.15.248:_export_instruct_vm__storage/0e01f014-530b-4067-aa1d-4e9378626a9d/images/6597eede-9fa0-4451-84fc-9f9c070cb5f3/765fa48b-2e77-4637-b4ca-e1affcd71e48 -O raw /rhev/data-center/mnt/10.141.15.248:_export_instruct_vm__storage/0e01f014-530b-4067-aa1d-4e9378626a9d/images/f0700631-e60b-4c2a-a6f5-a6c818ae7651/d4fb05ec-7c78-4d89-9a66-614c093c6e16

gdb has a blank output for this though. This means 4.3.2 is fairly unusable for us, so two questions, can I downgrade to 4.2, and is there a fix coming in 4.3.3 for this?

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 10 Apr 2019, at 11:22, Callum Smith <callum@well.ox.ac.uk> wrote:

Creating a disk on the target share works fine. - This seems to specifically be an issue to do with moving a disk to/from a share.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 10 Apr 2019, at 09:53, Callum Smith <callum@well.ox.ac.uk> wrote:

gdb -p $(pidof qemu-img convert) -batch -ex "t a a bt"
289444: No such file or directory.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 10 Apr 2019, at 09:36, Benny Zlotnik <bzlotnik@redhat.com> wrote:

Can you run:
$ gdb -p $(pidof qemu-img convert) -batch -ex "t a a bt"


On Wed, Apr 10, 2019 at 11:26 AM Callum Smith <callum@well.ox.ac.uk> wrote:

Dear All,

Further to this, I can't migrate a disk to different storage using the GUI. Both disks are configured identically and on the same physical NFS provider.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 9 Apr 2019, at 12:12, Callum Smith <callum@well.ox.ac.uk> wrote:

Dear All,

It would seem this is a bug in 4.3.? - As upgrading the old oVirt HE to 4.3 (from 4.2.latest) now means that the export of VMs to export domain no longer works.

Again qemu-img convert is using some cpu, but no network. Progress is 0.

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

On 8 Apr 2019, at 15:42, Callum Smith <callum@well.ox.ac.uk> wrote:

Dear All,

We've exported some VMs from our old oVirt infrastructure and want to import them into the new one, but qemu-img appears to be failing. We have mounted an export domain populated from the old oVirt in the new hosted engine and are using the GUI to import the VM. Manually running the command sits at 16% CPU, 0% network usage and no progress. It appears to lock the NFS mount and ls and lsof both hang.

sudo -u vdsm /usr/bin/qemu-img convert -p -t none -T none -f raw <path to source> -O raw <path to dest>

Conversely a simple cp will work (ruling out file permissions errors):
sudo -u vddm cp <path to source> <path to test>

What might we be doing wrong?

Regards,
Callum

--

Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZUBKWQIGGSJETPVRWN42R4J7COPFV6GS/


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6EE6X43GTAJ6L4QBH2XQJ4LVPIXCZC3T/


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GD2XCKEIPNCXXQWMR4OP5IKGCORJQEGB/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/SV3V34DQJZ7A3MH4CQMMIZ4GPPHWFB7E/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MICXCNEDJL5NOFUNS7R7W5CYDNEFINVK/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KRV6NPWUCOAMRXOV5S7PIHBMJ3ZOCAAD/


_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R53H5T4ZHFGTSWPJAOORMTAZYD355SBF/