[ovirt-devel] Re: [ OST Failure Report ] [ oVirt master (ovirt-engine) ] [ 22-03-2019 ] [004_basic_sanity.hotplug_cpu ]

On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry(a)redhat.com&gt; wrote: > On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > I added in > > https://gerrit.ovirt.org/#/c/98925/ > > a ping directly before the ssh. > > The ping succeeds, but the ssh fails. > > > > > > On Tue, 26 Mar 2019 17:07:45 +0100 > > Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > > > Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry(a)redhat.com&gt; > ha > > > scritto: > > > > > > > +1 from me > > > > > > > > > > Merged. I have 2 patches constantly failing on it, rebased them, you > can > > > follow on: > > > https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862 > > > > > > > still failing on jenkins, but at least one succeeds locally for me > > Succeeds locally for me also. > > Dafna, are we sure there's not an infra issue? > I think since its a race ( and we've seen failures on this test in the past, also a race I think ), its probably hard to reproduce locally. Also, we probably need to make sure the same Libvirt version is used. The upstream servers are quite old, it can also be local run ends up being faster and not hitting the same issues ( as we've seen in the past ) Could it be a bug in the ssh client ( paramiko? )

Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can cause this? ( other than slow servers ) > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler(a)redhat.com&gt; > > > > wrote: > > > > > > > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > > > > Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > > > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > > > > Dafna Ron <dron(a)redhat.com&gt; wrote: > > > > > > > > > > > > > This is still failing randomly > > > > > > > > > > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to > understand > > > > > > which action is crashing the guest. > > > > > > > > > > > > > > > > I was not able to reproduce the failure with the change above. > > > > > We could merge the change to have better information on the next > > > > > failure. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < > dholler(a)redhat.com&gt; > > > > wrote: > > > > > > > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > > > > Ryan Barry <rbarry(a)redhat.com&gt; wrote: > > > > > > > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > > > > > > > I'm very suspicious of this happening immediately after > > > > hotplugging a > > > > > > > > NIC, > > > > > > > > > especially since the bug attached to > > > > https://gerrit.ovirt.org/#/c/98765/ > > > > > > > > > talks about dropping packets. Dominik, did anything else > change > > > > here? > > > > > > > > > > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > > > > > > > Is there already a pattern in the failed runs detected, or > does it > > > > fail > > > > > > > > randomly? > > > > > > > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > > > > amarchuk(a)redhat.com&gt; > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should > open a > > > > bug in > > > > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < > nsoffer(a)redhat.com&gt; > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < > dron(a)redhat.com&gt; > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > > > > looking at the logs, we can see that the for some > reason, > > > > libvirt > > > > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > > > > display="on" for > > > > > > > > mdevs, > > > > > > > > > > use nodisplay to override > > > > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may > be > > > > netwrok > > > > > > > > > > related. > > > > > > > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... > > > > > > > > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... > > > > > > > > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and > the > > > > problem is > > > > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > > > > qemuDomainObjTaint:7521 : > > > > > > > > > > Domain id=3 name='vm0' > > > > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > > > > virProcessRunInMountNamespace:1159 : internal error: > child > > > > reported: > > > > > > > > unable > > > > > > > > > > to set security context > 'system_u:object_r:virt_content_t:s0' > > > > on > > > > > > > > > > > > > > > > > > > > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > > > > > > > > No such file or directory > > > > > > > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > > > > files in > > > > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > > > > namespace and adding there the devices used by the VM. > Since > > > > libvirt > > > > > > > > > > create the devices in its namespace > > > > > > > > > > > it should not complain about missing paths in > > > > /rhev/data-center. > > > > > > > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > responding: > > > > QEMU > > > > > > > > guest > > > > > > > > > > agent is not connected > > > > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > responding: > > > > QEMU > > > > > > > > guest > > > > > > > > > > agent is not connected > > > > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > responding: > > > > QEMU > > > > > > > > guest > > > > > > > > > > agent is not connected > > > > > > > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Anton Marchukov > > > > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > Infra mailing list -- infra(a)ovirt.org > > > > > > > > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > > > > > > > > Privacy Statement: > https://www.ovirt.org/site/privacy-policy/ > > > > > > > > > > oVirt Code of Conduct: > > > > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > > > List Archives: > > > > > > > > > > > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Devel mailing list -- devel(a)ovirt.org > > > > > > > > To unsubscribe send an email to devel-leave(a)ovirt.org > > > > > > > > Privacy Statement: > https://www.ovirt.org/site/privacy-policy/ > > > > > > > > oVirt Code of Conduct: > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > List Archives: > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHR... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Ryan Barry > > > > > > > > Associate Manager - RHV Virt/SLA > > > > > > > > rbarry(a)redhat.com M: +16518159306 IM: rbarry > > > > _______________________________________________ > > > > Infra mailing list -- infra(a)ovirt.org > > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > oVirt Code of Conduct: > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > List Archives: > > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLN... > > > > > > > > > > > > > > > -- > > Ryan Barry > > Associate Manager - RHV Virt/SLA > > rbarry(a)redhat.com M: +16518159306 IM: rbarry > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7... >