Hi all,
OST failed to run due to the following error in test_003_00_metrics_bootstrap
-
ost_utils.ansible.module_mappers.AnsibleExecutionError: Error running
ansible: rc=2, stdout=/usr/lib/python3.6/site-packages/requests/__init__.py:91:
RequestsDependencyWarning: urllib3 (1.26.4) or chardet (3.0.4) doesn't
match a supported version!
RequestsDependencyWarning)
lago-basic-suite-master-engine | FAILED | rc=1 >>
This command will collect system configuration and diagnostic
information from this system.
The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before
being passed to any third party.
No changes will be made to system configuration.
Use the -h option to see usage.
DEBUG: Configuration:
DEBUG: command: collect
DEBUG: Traceback (most recent call last):
DEBUG: File "/usr/lib/python3.6/site-packages/ovirt_log_collector/__main__.py",
line 2067, in <module>
DEBUG: '%s directory is not empty.' % (conf["local_tmp_dir"])
DEBUG: Exception: /dev/shm/iSX3ZN directory is not empty.ERROR:
/dev/shm/iSX3ZN directory is not empty.non-zero return code
Stacktrace
ansible_engine = <ost_utils.ansible.module_mappers.ModuleMapper object
at 0x7f67b85bb048>
ansible_hosts = <ost_utils.ansible.module_mappers.ModuleMapper object
at 0x7f67b85bbb38>
def test_metrics_and_log_collector(ansible_engine, ansible_hosts):
vt = utils.VectorThread(
[
functools.partial(configure_metrics, ansible_engine,
ansible_hosts),
functools.partial(run_log_collector, ansible_engine),
],
)
vt.start_all()
> vt.join_all()
basic-suite-master/test-scenarios/test_003_00_metrics_bootstrap.py:96:
Logs can be found in -
https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/16188…
Can someone have a look?
--
Regards,
Eyal Shenitzky
Hi all,
Can we please somehow have a single location one can check when
searching where its source code is, etc., and update all the others to
point there? We currently have at least:
https://github.com/lago-project/lagohttp://lago.readthedocs.io/https://www.ovirt.org/develop/infra/testing/lago/https://gerrit.ovirt.org/admin/repos/lagohttps://gerrit.ovirt.org/admin/repos/lago-ost
It seems like the last one is the current "source of truth", but I
didn't see this announced anywhere. Also, coming to the next point,
its name is confusing with "lago-ost-plugin".
In addition to above, we also have these, for lago-ost-plugin:
https://github.com/lago-project/lago-ost-pluginhttps://lago-ost-plugin.readthedocs.io/
It's not clear if we still need it or not. It would be nice to mention
this in such a home page, even after we do not need it anymore (just
the fact that it's not needed, to clear potential confusion for people
still finding it).
For all of above git repos that are not in use anymore, it would also
be nice to push a patch adding a short note to their README saying
they are not used anymore and linking at a home page.
Last but not least, I do support the new name "lago-ost" - "lago"
itself appears in far too many other unrelated places on the net.
Thanks and best regards,
--
Didi
Dear Ladies and Gentlemen!
I am currently working with the java-sdk and I encountered a problem.
If I would like to retrieve the disk details, I get the following error:
Disk currDisk = ovirtConnection.followLink(diskAttachment.disk());
The Error is occurring in this line:
[cid:image001.png@01D44537.AF127FD0]
The getResponst looks quiet ok. (I inspected: [cid:image002.png@01D44537.AF127FD0] and it looks ok).
Error:
wrong number of arguments
The code is quiet similar to what you published on github (https://github.com/oVirt/ovirt-engine-sdk-java/blob/master/sdk/src/test/jav… ).
Can you confirm the defect?
Best regards
Patrick
*Raising issue with pipeline:*
*11:47:53* Traceback (most recent call last):*11:47:53* File
"/home/cloud-user/workspace/standard-enqueue/jenkins/stdci_tools/usrc.py",
line 15, in <module>*11:47:53* from xdg.BaseDirectory import
xdg_cache_home*11:47:53* ImportError: No module named
xdg.BaseDirectory
Seen on
http://ovirt-devops-jenkins.upshift.rdu2.redhat.com/job/standard-enqueue/34…
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
On Tuesday, 23 March 2021 04:25:25 CET Shubha Kulkarni wrote:
> Hello,
>
> I am investigating an issue with deleting snapshot in oVirt 4.3.10.
> Basically the delete snapshot operation fails and I am seeing following
> error in VDSM(vdsm-4.30.46) log -
>
> ============================================================================
> ================================================================= 2021-03-17
> 21:38:01,346-0400 INFO (jsonrpc/1) [virt.vm]
> (vmId='4b04639c-386e-463c-8a8a-dfd3bc46d306') Starting merge with
> jobUUID=u'e6206d9f-6899-40b7-857a-3be3dd42d77d', original
> chain=ac065664-ad6c-4ad6-aea2-9558df71d41d <
> 210900dd-ff19-4a00-8706-926cb192b0db < 273fc001-1c18-4234-b2b6-f0485e5d13ef
> < e1e10354-ae43-456f-9341-1bb81c5bf960 <
> 7985a853-4d47-428b-bb5b-b34a38cc57e0 (top), disk='sda', base='sda[2]',
> top='sda[1]', bandwidth=0, flags=8 (vm:5954) 2021-03-17 21:38:01,354-0400
> ERROR (jsonrpc/1) [virt.vm] (vmId='4b04639c-386e-463c-8a8a-dfd3bc46d306')
> Live merge failed (job: e6206d9f-6899-40b7-857a-3be3dd42d77d) (vm:5960)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5958, in
> merge bandwidth, flags)
> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100,
> in f ret = attr(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py",
> line 131, in wrapper ret = f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94,
> in wrapper return func(inst, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 719, in
> blockCommit if ret == -1: raise libvirtError ('virDomainBlockCommit()
> failed', dom=self) libvirtError: internal error: qemu block name
> 'json:{"backing": {"backing": {"backing": {"driver": "raw", "file":
> {"driver": "file", "filename":
> "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4
> 743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/ac065664-a
> d6c-4ad6-aea2-9558df71d41d"}}, "driver": "qcow2", "file": {"driver": "file",
> "filename":
> "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4
> 743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/210900dd-f
> f19-4a00-8706-926cb192b0db"}}, "driver": "qcow2", "file": {"driver": "file",
> "filename":
> "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4
> 743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/273fc001-1
> c18-4234-b2b6-f0485e5d13ef"}}, "driver": "qcow2", "file": {"driver": "file",
> "filename":
> "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4
> 743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/e1e10354-a
> e43-456f-9341-1bb81c5bf960"}}' doesn 2021-03-17 21:38:01,368-0400 INFO
> (jsonrpc/1) [api.virt] FINISH merge return={'status': {'message': 'Merge
> failed', 'code': 52}}
> ===========================================================================
> =================================================================
>
> I found out that there is an issue logged here
> https://bugzilla.redhat.com/show_bug.cgi?id=1785939. However I am not able
> to see how it was fixed
see https://bugzilla.redhat.com/show_bug.cgi?id=1785939#c10
> and whether it was merged in oVirt. Where can I
> find the relevant commit (in vdsm and/or libvirt)?
>
> Thanks
Hello,
I am investigating an issue with deleting snapshot in oVirt 4.3.10. Basically the delete snapshot operation fails and I am seeing following error in VDSM(vdsm-4.30.46) log -
=============================================================================================================================================
2021-03-17 21:38:01,346-0400 INFO (jsonrpc/1) [virt.vm] (vmId='4b04639c-386e-463c-8a8a-dfd3bc46d306') Starting merge with jobUUID=u'e6206d9f-6899-40b7-857a-3be3dd42d77d', original chain=ac065664-ad6c-4ad6-aea2-9558df71d41d < 210900dd-ff19-4a00-8706-926cb192b0db < 273fc001-1c18-4234-b2b6-f0485e5d13ef < e1e10354-ae43-456f-9341-1bb81c5bf960 < 7985a853-4d47-428b-bb5b-b34a38cc57e0 (top), disk='sda', base='sda[2]', top='sda[1]', bandwidth=0, flags=8 (vm:5954)
2021-03-17 21:38:01,354-0400 ERROR (jsonrpc/1) [virt.vm] (vmId='4b04639c-386e-463c-8a8a-dfd3bc46d306') Live merge failed (job: e6206d9f-6899-40b7-857a-3be3dd42d77d) (vm:5960)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5958, in merge
bandwidth, flags)
File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f
ret = attr(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 719, in blockCommit
if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
libvirtError: internal error: qemu block name 'json:{"backing": {"backing": {"backing": {"driver": "raw", "file": {"driver": "file", "filename": "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/ac065664-ad6c-4ad6-aea2-9558df71d41d"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/210900dd-ff19-4a00-8706-926cb192b0db"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/273fc001-1c18-4234-b2b6-f0485e5d13ef"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/rhev/data-center/mnt/nash-nfs7:_nfs_shubha__ovirt__2_data/8bce5be7-aede-4743-b3fd-1c199880892f/images/eb3cb8f2-9544-472f-946b-646eab9c621f/e1e10354-ae43-456f-9341-1bb81c5bf960"}}' doesn
2021-03-17 21:38:01,368-0400 INFO (jsonrpc/1) [api.virt] FINISH merge return={'status': {'message': 'Merge failed', 'code': 52}}
============================================================================================================================================
I found out that there is an issue logged here https://bugzilla.redhat.com/show_bug.cgi?id=1785939. However I am not able to see how it was fixed and whether it was merged in oVirt. Where can I find the relevant commit (in vdsm and/or libvirt)?
Thanks
On Tue, Mar 16, 2021 at 10:47 PM Greg King <greg.king(a)oracle.com> wrote:
> I am new to vdsm and trying to understand the architecture/internals much
> better
>
Welcome to vdsm Greg!
The ovirt documentation for architecture I have found so far seems to be
> relatively high level
>
And it is mostly outdated, but we don't have anything better.
> My effort to understand the architecture by walking through the vdsm code
> using pdb/rpdb is slow and probably not all that efficient
>
>
>
> Does anyone have pointers to documentation that might explain the vdsm
> modules, classes and internals a little more in depth?
>
I don't think we have more detailed documentation, but there are lot of
talks and slide decks that give more info on specific topics, and are
usually
are more updated:
https://www.ovirt.org/community/archived_conferences_presentations.html
There is also lot of content on youtube, here some example that I could
find easily:
- [oVirt 3.6 deep dive] - live storage migration between mixed domains
https://www.youtube.com/watch?v=BPy29Q__VV4
- oVirt 4.1 deep dive - VM leases
https://www.youtube.com/watch?v=MVa-4fQo2V8
- Back to the future – incremental backup in oVirt
https://www.youtube.com/watch?v=X-xHD9ddN6s
- oVirt 4k - teaching an old dog new tricks
https://www.youtube.com/watch?v=Q1VQxjYEzDY
>
> I’d also like to understand where I might be able to add rpdb.set_trace()
> so I can step through functions being called in libvirt.py
>
I don't think using a debugger is very helpful with vdsm, since vdsm is not
designed for stopping a thread for unlimited time. In some cases the system
will log warning and traceback every 60 seconds about blocked worker.
In other cases monitoring code may fail to update stats, which may cause
engines to deactivate a host or migrate vms or other trouble.
The best way to debug and understand vdsm is to follow the logs, and add
move logs when needed. The main advantage compared with a debugger is
that the time spent with the logs will pay back when you have to debug real
issues in user setup, when logs are the only available resource.
Having said that, being able to follow the entire flow by printing a
traceback
is a great way to understand how the system works.
You can use vdsm.common.concurrent.format_traceback:
https://github.com/oVirt/vdsm/blob/114121ab122a0cd5e529807b938b3506f247f42b…
To print traceback at interesting points. For tracing function from the
libvirt
python binginding, you can modify libvirtconnection.py:
https://github.com/oVirt/vdsm/blob/114121ab122a0cd5e529807b938b3506f247f42b…
This module creates a connection, and wraps libvirt.virDomain with a wrapper
that panics on fatal errors. You can modify the wrapper to log a traceback
for all or some of libvirt.virDomain functions.
Another option it to modify the virDomain wrapper to log a traceback:
https://github.com/oVirt/vdsm/blob/114121ab122a0cd5e529807b938b3506f247f42b…
For example here:
https://github.com/oVirt/vdsm/blob/114121ab122a0cd5e529807b938b3506f247f42b…
Good luck with your vdsm ride!
Nir
