On Wed, 27 Mar 2019 10:07:16 +0200
Eyal Edri <eedri@redhat.com> wrote:

> On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry@redhat.com> wrote:
>
> > On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
> > >
> > > I added in
> > > https://gerrit.ovirt.org/#/c/98925/
> > > a ping directly before the ssh.
> > > The ping succeeds, but the ssh fails.
> > >
> > >
> > > On Tue, 26 Mar 2019 17:07:45 +0100
> > > Sandro Bonazzola <sbonazzo@redhat.com> wrote:
> > >
> > > > Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry@redhat.com>
> > ha
> > > > scritto:
> > > >
> > > > > +1 from me
> > > > >
> > > >
> > > > Merged. I have 2 patches constantly failing on it, rebased them, you
> > can
> > > > follow on:
> > > > https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
> > > >
> > >
> > > still failing on jenkins, but at least one succeeds locally for me
> >
> > Succeeds locally for me also.
> >
> > Dafna, are we sure there's not an infra issue?
> >
>
> I think since its a race ( and we've seen failures on this test in the
> past, also a race I think ), its probably hard to reproduce locally.
> Also, we probably need to make sure the same Libvirt version is used.
> The upstream servers are quite old, it can also be local run ends up being
> faster and not hitting the same issues ( as we've seen in the past )
>
> Could it be a bug in the ssh client ( paramiko? )
>


Probably wrong idea, but worth to ask:
Any ideas which  ssh_timeout is used or how to modify?

If 100 tries including a time.sleep(1) takes 100 seconds,
either the timeout is not the expected 10 seconds, or the guest refuses
the connection.

2019-03-27 07:35:23,801::log_utils.py::end_log_task::670::nose::INFO::  # hotplug_nic:  [32mSuccess [0m (in 0:00:05)
2019-03-27 07:35:23,801::log_utils.py::start_log_task::655::nose::INFO::  # hotplug_cpu:  [0m [0m

> Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can
> cause this? ( other than slow servers )
>
>
> >
> > >
> > > >
> > > >
> > > > >
> > > > > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler@redhat.com>
> > > > > wrote:
> > > > > >
> > > > > > On Tue, 26 Mar 2019 12:31:36 +0100
> > > > > > Dominik Holler <dholler@redhat.com> wrote:
> > > > > >
> > > > > > > On Tue, 26 Mar 2019 10:58:22 +0000
> > > > > > > Dafna Ron <dron@redhat.com> wrote:
> > > > > > >
> > > > > > > > This is still failing randomly
> > > > > > > >
> > > > > > >
> > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to
> > understand
> > > > > > > which action is crashing the guest.
> > > > > > >
> > > > > >
> > > > > > I was not able to reproduce the failure with the change above.
> > > > > > We could merge the change to have better information on the next
> > > > > > failure.
> > > > > >
> > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <
> > dholler@redhat.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400
> > > > > > > > > Ryan Barry <rbarry@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > > It may be virt, but I'm looking...
> > > > > > > > > >
> > > > > > > > > > I'm very suspicious of this happening immediately after
> > > > > hotplugging a
> > > > > > > > > NIC,
> > > > > > > > > > especially since the bug attached to
> > > > > https://gerrit.ovirt.org/#/c/98765/
> > > > > > > > > > talks about dropping packets. Dominik, did anything else
> > change
> > > > > here?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > No, nothing I am aware of.
> > > > > > > > >
> > > > > > > > > Is there already a pattern in the failed runs detected, or
> > does it
> > > > > fail
> > > > > > > > > randomly?
> > > > > > > > >
> > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <
> > > > > amarchuk@redhat.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Which team is it? Is it Virt? Just checking who should
> > open a
> > > > > bug in
> > > > > > > > > > > libvirt as suggested.
> > > > > > > > > > >
> > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer <
> > nsoffer@redhat.com>
> > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <
> > dron@redhat.com>
> > > > > wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > We are failing ovirt-engine master on test
> > > > > > > > > 004_basic_sanity.hotplug_cpu
> > > > > > > > > > > > looking at the logs, we can see that the for some
> > reason,
> > > > > libvirt
> > > > > > > > > > > reports a vm as none responsive which fails the test.
> > > > > > > > > > > >
> > > > > > > > > > > > CQ first failure was for patch:
> > > > > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add
> > > > > display="on" for
> > > > > > > > > mdevs,
> > > > > > > > > > > use nodisplay to override
> > > > > > > > > > > > But I do not think this is the cause of failure.
> > > > > > > > > > > >
> > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may
> > be
> > > > > netwrok
> > > > > > > > > > > related.
> > > > > > > > > > > >
> > > > > > > > > > > > You can see the libvirt log here:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-1/_var_log/libvirt.log
> > > > > > > > > > > >
> > > > > > > > > > > > you can see the full logs here:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-master/post-004_basic_sanity.py/
> > > > > > > > > > > >
> > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and
> > the
> > > > > problem is
> > > > > > > > > > > ssh connection to the internal vm
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Dafna
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > error:
> > > > > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning :
> > > > > > > > > qemuDomainObjTaint:7521 :
> > > > > > > > > > > Domain id=3 name='vm0'
> > > > > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is
> > > > > > > > > > > tainted: hook-script
> > > > > > > > > > > >
> > > > > > > > > > > > Why our vm is tainted?
> > > > > > > > > > > >
> > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error :
> > > > > > > > > > > virProcessRunInMountNamespace:1159 : internal error:
> > child
> > > > > reported:
> > > > > > > > > unable
> > > > > > > > > > > to set security context
> > 'system_u:object_r:virt_content_t:s0'
> > > > > on
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
> > > > > > > > > > > No such file or directory
> > > > > > > > > > > >
> > > > > > > > > > > > This should not happen, libvirt is not adding labels to
> > > > > files in
> > > > > > > > > > > /rhev/data-center. It is using using its own mount
> > > > > > > > > > > > namespace and adding there the devices used by the VM.
> > Since
> > > > > libvirt
> > > > > > > > > > > create the devices in its namespace
> > > > > > > > > > > > it should not complain about missing paths in
> > > > > /rhev/data-center.
> > > > > > > > > > > >
> > > > > > > > > > > > I think we should file a libvirt bug for this.
> > > > > > > > > > > >
> > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error :
> > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not
> > responding:
> > > > > QEMU
> > > > > > > > > guest
> > > > > > > > > > > agent is not connected
> > > > > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error :
> > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not
> > responding:
> > > > > QEMU
> > > > > > > > > guest
> > > > > > > > > > > agent is not connected
> > > > > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error :
> > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not
> > responding:
> > > > > QEMU
> > > > > > > > > guest
> > > > > > > > > > > agent is not connected
> > > > > > > > > > > >
> > > > > > > > > > > > Do we have guest agent in the test VMs?
> > > > > > > > > > > >
> > > > > > > > > > > > Nir
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Anton Marchukov
> > > > > > > > > > > Associate Manager - RHV DevOps - Red Hat
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > _______________________________________________
> > > > > > > > > > > Infra mailing list -- infra@ovirt.org
> > > > > > > > > > > To unsubscribe send an email to infra-leave@ovirt.org
> > > > > > > > > > > Privacy Statement:
> > https://www.ovirt.org/site/privacy-policy/
> > > > > > > > > > > oVirt Code of Conduct:
> > > > > > > > > > >
> > https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > > > > > List Archives:
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW4IDWZAS3RYMAFQ56VG/
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > _______________________________________________
> > > > > > > > > Devel mailing list -- devel@ovirt.org
> > > > > > > > > To unsubscribe send an email to devel-leave@ovirt.org
> > > > > > > > > Privacy Statement:
> > https://www.ovirt.org/site/privacy-policy/
> > > > > > > > > oVirt Code of Conduct:
> > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > > > List Archives:
> > > > > > > > >
> > > > >
> > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI53QDC24TY6J2ZL2JWSH/
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Ryan Barry
> > > > >
> > > > > Associate Manager - RHV Virt/SLA
> > > > >
> > > > > rbarry@redhat.com    M: +16518159306     IM: rbarry
> > > > > _______________________________________________
> > > > > Infra mailing list -- infra@ovirt.org
> > > > > To unsubscribe send an email to infra-leave@ovirt.org
> > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > > > > oVirt Code of Conduct:
> > > > > https://www.ovirt.org/community/about/community-guidelines/
> > > > > List Archives:
> > > > >
> > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURGWBOSISFMAECNC2AXXY/
> > > > >
> > > >
> > > >
> > >
> >
> >
> > --
> >
> > Ryan Barry
> >
> > Associate Manager - RHV Virt/SLA
> >
> > rbarry@redhat.com    M: +16518159306     IM: rbarry
> > _______________________________________________
> > Devel mailing list -- devel@ovirt.org
> > To unsubscribe send an email to devel-leave@ovirt.org
> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7OSQKIVFNATQTESPXAWQV5/
> >
>
>
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCD6OCOWXSHZYMXWNRA7EQQZVVJBRIUC/