[JIRA] (OVIRT-2024) virt-sparsify fails silently
by Dafna Ron (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2024?page=com.atlassian.jir... ]
Dafna Ron reassigned OVIRT-2024:
--------------------------------
Assignee: Dafna Ron (was: infra)
> virt-sparsify fails silently
> -----------------------------
>
> Key: OVIRT-2024
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2024
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Dafna Ron
> Assignee: Dafna Ron
> Labels: ost_failures
>
> virt-sparsify fails silently (intentionally silently).
> you can see the error in the host vdsm log (even on successful builds)
> 018-05-14 07:15:02,271-0400 ERROR (tasks/6) [root] Job u'796dd36e-0095-4adc-afb9-8b011f6b8ced' failed (jobs:221)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/jobs.py", line 157, in run
> self._run()
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sdm/api/sparsify_volume.py", line 56, in _run
> virtsparsify.sparsify_inplace(self._vol_info.path)
> File "/usr/lib/python2.7/site-packages/vdsm/virtsparsify.py", line 71, in sparsify_inplace
> raise cmdutils.Error(cmd, rc, out, err)
> Error: Command ['/usr/bin/virt-sparsify', '--machine-readable', '--in-place', u'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/96157fda-d60a-49d1-adb8-f306e402a0cf/images/156eaf32-5219-4be7-b61e-b5104f2b6c4d/77eb6602-13ac-46aa-a4
> 39-af9e51e3c2cb'] failed with rc=1 out=['3/12'] err=['virt-sparsify: error: libguestfs error: guestfs_launch failed.', 'This usually means the libguestfs appliance failed to start or crashed.', 'Do:', ' export LIBGUESTFS_DEBUG=1 LIBGUEST
> FS_TRACE=1', 'and run the command again. For further information, read:', ' http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs', "You can also run 'libguestfs-test-tool' and post the *complete* output", 'into a bug report or
> message to the libguestfs mailing list.', '', 'If reporting bugs, run virt-sparsify with debugging enabled and include the ', 'complete output:', '', ' virt-sparsify -v -x [...]']
> 2018-05-14 07:15:02,273-0400 INFO (tasks/6) [root] Job u'796dd36e-0095-4adc-afb9-8b011f6b8ced' will be deleted in 3600 seconds (jobs:249)
> 2018-05-14 07:15:02,273-0400 DEBUG (tasks/6) [jsonrpc.Notification] Sending event {"params": {"status": "failed", "description": "sparsify_volume", "job_type": "storage", "error": {"message": "General Exception: ('Command [\\'/usr/bin/vir
> t-sparsify\\', \\'--machine-readable\\', \\'--in-place\\', u\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/96157fda-d60a-49d1-adb8-f306e402a0cf/images/156eaf32-5219-4be7-b61e-b5104f2b6c4d/77eb6602-13ac-46aa-a439-af9e51e3c2cb\\
> '] failed with rc=1 out=[\\'3/12\\'] err=[\\'virt-sparsify: error: libguestfs error: guestfs_launch failed.\\', \\'This usually means the libguestfs appliance failed to start or crashed.\\', \\'Do:\\', \\' export LIBGUESTFS_DEBUG=1 LIBGU
> ESTFS_TRACE=1\\', \\'and run the command again. For further information, read:\\', \\' http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\', \"You can also run \\'libguestfs-test-tool\\' and post the *complete* output\", \\'
> into a bug report or message to the libguestfs mailing list.\\', \\'\\', \\'If reporting bugs, run virt-sparsify with debugging enabled and include the \\', \\'complete output:\\', \\'\\', \\' virt-sparsify -v -x [...]\\']',)", "code": 1
> 00}, "notify_time": 4295843980, "id": "796dd36e-0095-4adc-afb9-8b011f6b8ced"}, "jsonrpc": "2.0", "method": "|jobs|status|796dd36e-0095-4adc-afb9-8b011f6b8ced"} (__init__:180)
> 2018-05-14 07:15:02,273-0400 DEBUG (tasks/6) [vds] Sending notification |jobs|status|796dd36e-0095-4adc-afb9-8b011f6b8ced with params {'status': 'failed', 'description': 'sparsify_volume', 'job_type': 'storage', 'error': {'message': 'Gene
> ral Exception: (\'Command [\\\'/usr/bin/virt-sparsify\\\', \\\'--machine-readable\\\', \\\'--in-place\\\', u\\\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_share1/96157fda-d60a-49d1-adb8-f306e402a0cf/images/156eaf32-5219-4be7-b61e-b5
> 104f2b6c4d/77eb6602-13ac-46aa-a439-af9e51e3c2cb\\\'] failed with rc=1 out=[\\\'3/12\\\'] err=[\\\'virt-sparsify: error: libguestfs error: guestfs_launch failed.\\\', \\\'This usually means the libguestfs appliance failed to start or crash
> ed.\\\', \\\'Do:\\\', \\\' export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1\\\', \\\'and run the command again. For further information, read:\\\', \\\' http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs\\\', "You can also run \
> \\'libguestfs-test-tool\\\' and post the *complete* output", \\\'into a bug report or message to the libguestfs mailing list.\\\', \\\'\\\', \\\'If reporting bugs, run virt-sparsify with debugging enabled and include the \\\', \\\'complet
> e output:\\\', \\\'\\\', \\\' virt-sparsify -v -x [...]\\\']\',)', 'code': 100}, 'notify_time': 4295843980, 'id': u'796dd36e-0095-4adc-afb9-8b011f6b8ced'} (clientIF:225)
> 2018-05-14 07:15:02,274-0400 INFO (tasks/6) [storage.ThreadPool.WorkerThread] FINISH task 041a7423-8c05-44b2-b3c9-6d5cf8710edc (threadPool:210)
> 2018-05-14 07:15:04,229-0400 DEBUG (qgapoller/1) [vds] Not sending QEMU-GA command 'guest-get-users' to vm_id='e1317264-a572-47e0-98b4-d492239e285e', command is not supported (qemuguestagent:192)
> 2018-05-14 07:15:04,230-0400 DEBUG (qgapoller/2) [vds] Not sending QEMU-GA command 'guest-get-users' to vm_id='29a56e04-51c7-4f25-b3c8-6df5489b2445', command is not supported (qemuguestagent:192)
> 2018-05-14 07:15:05,186-0400 DEBUG (vmchannels) [vds] Timeout on fileno 63. (vmchannels:126)
> Shmuel is investigating and we need to follow up with him.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100085)
6 years, 7 months
[JIRA] (OVIRT-2026) CQ job killed on time-out - need to see why
by Dafna Ron (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-2026?page=com.atlassian.jir... ]
Dafna Ron reassigned OVIRT-2026:
--------------------------------
Assignee: Dafna Ron (was: infra)
> CQ job killed on time-out - need to see why
> -------------------------------------------
>
> Key: OVIRT-2026
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2026
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Dafna Ron
> Assignee: Dafna Ron
> Labels: ost_failures
>
> [CQ]: 84817,6 (ovirt-engine) failed "ovirt-master" system tests
> For further details about the change see:
> https://gerrit.ovirt.org/#/c/84817/6
> For failed test results see:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/7601/
> we can see a gap in times and we can see the exit was on check-connectivity for network but need to investigate further:
> 07:16:44 [upgrade-from-release-suit] Running in /home/jenkins/workspace/ovirt-master_change-queue-tester/exported-artifacts/upgrade-from-release-suit-master-el7
> [Pipeline] [upgrade-from-release-suit] {
> [Pipeline] [upgrade-from-release-suit] unstash
> [Pipeline] [upgrade-from-release-suit] }
> [Pipeline] [upgrade-from-release-suit] // dir
> [Pipeline] [upgrade-from-release-suit] }
> 08:12:19 [basic-suit] Error while running thread
> 08:12:19 [basic-suit] Traceback (most recent call last):
> 08:12:19 [basic-suit] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
> 08:12:19 [basic-suit] queue.put({'return': func()})
> 08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_basic_networking.py", line 194, in _bond_nics
> 08:12:19 [basic-suit] [bond])
> 08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test_utils/network_utils_v4.py", line 44, in attach_network_to_host
> 08:12:19 [basic-suit] check_connectivity=True)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37188, in setup_networks
> 08:12:19 [basic-suit] return self._internal_action(action, 'setupnetworks', None, headers, query, wait)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action
> 08:12:19 [basic-suit] return future.wait() if wait else future
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
> 08:12:19 [basic-suit] return self._code(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback
> 08:12:19 [basic-suit] self._check_fault(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
> 08:12:19 [basic-suit] body = self._internal_read_body(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
> 08:12:19 [basic-suit] self._connection.check_xml_content_type(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type
> 08:12:19 [basic-suit] response.headers
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
> 08:12:19 [basic-suit] raise Error(msg)
> 08:12:19 [basic-suit] Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML
> 08:12:19 [basic-suit] Error while running thread
> 08:12:19 [basic-suit] Traceback (most recent call last):
> 08:12:19 [basic-suit] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
> 08:12:19 [basic-suit] queue.put({'return': func()})
> 08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_basic_networking.py", line 194, in _bond_nics
> 08:12:19 [basic-suit] [bond])
> 08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test_utils/network_utils_v4.py", line 44, in attach_network_to_host
> 08:12:19 [basic-suit] check_connectivity=True)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37188, in setup_networks
> 08:12:19 [basic-suit] return self._internal_action(action, 'setupnetworks', None, headers, query, wait)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action
> 08:12:19 [basic-suit] return future.wait() if wait else future
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
> 08:12:19 [basic-suit] return self._code(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback
> 08:12:19 [basic-suit] self._check_fault(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
> 08:12:19 [basic-suit] body = self._internal_read_body(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
> 08:12:19 [basic-suit] self._connection.check_xml_content_type(response)
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type
> 08:12:19 [basic-suit] response.headers
> 08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
> 08:12:19 [basic-suit] raise Error(msg)
> 08:12:19 [basic-suit] Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML
> 08:41:12 [basic-suit] ./automation/basic_suite_master.sh: line 152: 183236 Killed timeout --kill-after 5m 100m ./run_suite.sh -o "$run_path" "${extra_cmds[@]}" "$SUITE"
> 08:41:12 [basic-suit] + res=137
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100085)
6 years, 7 months
[JIRA] (OVIRT-2026) CQ job killed on time-out - need to see why
by Dafna Ron (oVirt JIRA)
Dafna Ron created OVIRT-2026:
--------------------------------
Summary: CQ job killed on time-out - need to see why
Key: OVIRT-2026
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2026
Project: oVirt - virtualization made easy
Issue Type: Bug
Reporter: Dafna Ron
Assignee: infra
[CQ]: 84817,6 (ovirt-engine) failed "ovirt-master" system tests
For further details about the change see:
https://gerrit.ovirt.org/#/c/84817/6
For failed test results see:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/7601/
we can see a gap in times and we can see the exit was on check-connectivity for network but need to investigate further:
07:16:44 [upgrade-from-release-suit] Running in /home/jenkins/workspace/ovirt-master_change-queue-tester/exported-artifacts/upgrade-from-release-suit-master-el7
[Pipeline] [upgrade-from-release-suit] {
[Pipeline] [upgrade-from-release-suit] unstash
[Pipeline] [upgrade-from-release-suit] }
[Pipeline] [upgrade-from-release-suit] // dir
[Pipeline] [upgrade-from-release-suit] }
08:12:19 [basic-suit] Error while running thread
08:12:19 [basic-suit] Traceback (most recent call last):
08:12:19 [basic-suit] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
08:12:19 [basic-suit] queue.put({'return': func()})
08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_basic_networking.py", line 194, in _bond_nics
08:12:19 [basic-suit] [bond])
08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test_utils/network_utils_v4.py", line 44, in attach_network_to_host
08:12:19 [basic-suit] check_connectivity=True)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37188, in setup_networks
08:12:19 [basic-suit] return self._internal_action(action, 'setupnetworks', None, headers, query, wait)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action
08:12:19 [basic-suit] return future.wait() if wait else future
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
08:12:19 [basic-suit] return self._code(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback
08:12:19 [basic-suit] self._check_fault(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
08:12:19 [basic-suit] body = self._internal_read_body(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
08:12:19 [basic-suit] self._connection.check_xml_content_type(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type
08:12:19 [basic-suit] response.headers
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
08:12:19 [basic-suit] raise Error(msg)
08:12:19 [basic-suit] Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML
08:12:19 [basic-suit] Error while running thread
08:12:19 [basic-suit] Traceback (most recent call last):
08:12:19 [basic-suit] File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue
08:12:19 [basic-suit] queue.put({'return': func()})
08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test-scenarios/003_basic_networking.py", line 194, in _bond_nics
08:12:19 [basic-suit] [bond])
08:12:19 [basic-suit] File "/home/jenkins/workspace/ovirt-master_change-queue-tester/ovirt-system-tests/basic-suite-master/test_utils/network_utils_v4.py", line 44, in attach_network_to_host
08:12:19 [basic-suit] check_connectivity=True)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37188, in setup_networks
08:12:19 [basic-suit] return self._internal_action(action, 'setupnetworks', None, headers, query, wait)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action
08:12:19 [basic-suit] return future.wait() if wait else future
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
08:12:19 [basic-suit] return self._code(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback
08:12:19 [basic-suit] self._check_fault(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
08:12:19 [basic-suit] body = self._internal_read_body(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
08:12:19 [basic-suit] self._connection.check_xml_content_type(response)
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type
08:12:19 [basic-suit] response.headers
08:12:19 [basic-suit] File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
08:12:19 [basic-suit] raise Error(msg)
08:12:19 [basic-suit] Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML
08:41:12 [basic-suit] ./automation/basic_suite_master.sh: line 152: 183236 Killed timeout --kill-after 5m 100m ./run_suite.sh -o "$run_path" "${extra_cmds[@]}" "$SUITE"
08:41:12 [basic-suit] + res=137
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100085)
6 years, 7 months