Hi, We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test. CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure. Adding Marcin, Milan and Dan as well as I think it may be netwrok related. You can see the libvirt log here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... you can see the full logs here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm Thanks, Dafna error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script Why our vm is tainted?

This should not happen, libvirt is not adding labels to files in

2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Do we have guest agent in the test VMs?

On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron(a)redhat.com&gt; wrote: Hi, We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test. CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure. Adding Marcin, Milan and Dan as well as I think it may be netwrok related. You can see the libvirt log here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... you can see the full logs here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm Thanks, Dafna error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script Why our vm is tainted? 2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory This should not happen, libvirt is not adding labels to files in /rhev/data-center. It is using using its own mount namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center. I think we should file a libvirt bug for this. 2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Do we have guest agent in the test VMs? Nir

It may be virt, but I'm looking... I'm very suspicious of this happening immediately after hotplugging a NIC, especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here?

On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk(a)redhat.com&gt; wrote: > Which team is it? Is it Virt? Just checking who should open a bug in > libvirt as suggested. > > > On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > > Hi, > > > > We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu > > looking at the logs, we can see that the for some reason, libvirt > reports a vm as none responsive which fails the test. > > > > CQ first failure was for patch: > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, > use nodisplay to override > > But I do not think this is the cause of failure. > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > related. > > > > You can see the libvirt log here: > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... > > > > you can see the full logs here: > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... > > > > Evgheni and I confirmed this is not an infra issue and the problem is > ssh connection to the internal vm > > > > Thanks, > > Dafna > > > > > > error: > > 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > tainted: hook-script > > > > Why our vm is tainted? > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > virProcessRunInMountNamespace:1159 : internal error: child reported: unable > to set security context 'system_u:object_r:virt_content_t:s0' on > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > No such file or directory > > > > This should not happen, libvirt is not adding labels to files in > /rhev/data-center. It is using using its own mount > > namespace and adding there the devices used by the VM. Since libvirt > create the devices in its namespace > > it should not complain about missing paths in /rhev/data-center. > > > > I think we should file a libvirt bug for this. > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > agent is not connected > > 2019-03-22 15:08:58.193+0000: 22070: error : > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > agent is not connected > > 2019-03-22 15:13:58.179+0000: 22071: error : > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > agent is not connected > > > > Do we have guest agent in the test VMs? > > > > Nir > > -- > Anton Marchukov > Associate Manager - RHV DevOps - Red Hat > > > > > > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... >

This is still failing randomly

On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler(a)redhat.com&gt; wrote: > On Mon, 25 Mar 2019 17:30:53 -0400 > Ryan Barry <rbarry(a)redhat.com&gt; wrote: > > > It may be virt, but I'm looking... > > > > I'm very suspicious of this happening immediately after hotplugging a > NIC, > > especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ > > talks about dropping packets. Dominik, did anything else change here? > > > > No, nothing I am aware of. > > Is there already a pattern in the failed runs detected, or does it fail > randomly? > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk(a)redhat.com&gt; > > wrote: > > > > > Which team is it? Is it Virt? Just checking who should open a bug in > > > libvirt as suggested. > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > > > > Hi, > > > > > > > > We are failing ovirt-engine master on test > 004_basic_sanity.hotplug_cpu > > > > looking at the logs, we can see that the for some reason, libvirt > > > reports a vm as none responsive which fails the test. > > > > > > > > CQ first failure was for patch: > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for > mdevs, > > > use nodisplay to override > > > > But I do not think this is the cause of failure. > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > > > related. > > > > > > > > You can see the libvirt log here: > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... > > > > > > > > you can see the full logs here: > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... > > > > > > > > Evgheni and I confirmed this is not an infra issue and the problem is > > > ssh connection to the internal vm > > > > > > > > Thanks, > > > > Dafna > > > > > > > > > > > > error: > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > qemuDomainObjTaint:7521 : > > > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > tainted: hook-script > > > > > > > > Why our vm is tainted? > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > virProcessRunInMountNamespace:1159 : internal error: child reported: > unable > > > to set security context 'system_u:object_r:virt_content_t:s0' on > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > No such file or directory > > > > > > > > This should not happen, libvirt is not adding labels to files in > > > /rhev/data-center. It is using using its own mount > > > > namespace and adding there the devices used by the VM. Since libvirt > > > create the devices in its namespace > > > > it should not complain about missing paths in /rhev/data-center. > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > guest > > > agent is not connected > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > guest > > > agent is not connected > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > guest > > > agent is not connected > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > Nir > > > > > > -- > > > Anton Marchukov > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Infra mailing list -- infra(a)ovirt.org > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > oVirt Code of Conduct: > > > https://www.ovirt.org/community/about/community-guidelines/ > > > List Archives: > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... > > > > > > > > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHR... >

On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler(a)redhat.com&gt; wrote: > On Tue, 26 Mar 2019 10:58:22 +0000 > Dafna Ron <dron(a)redhat.com&gt; wrote: > > > This is still failing randomly > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to understand > which action is crashing the guest. > I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure. > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > Ryan Barry <rbarry(a)redhat.com&gt; wrote: > > > > > > > It may be virt, but I'm looking... > > > > > > > > I'm very suspicious of this happening immediately after hotplugging a > > > NIC, > > > > especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ > > > > talks about dropping packets. Dominik, did anything else change here? > > > > > > > > > > No, nothing I am aware of. > > > > > > Is there already a pattern in the failed runs detected, or does it fail > > > randomly? > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk(a)redhat.com&gt; > > > > wrote: > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a bug in > > > > > libvirt as suggested. > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > > > > > > Hi, > > > > > > > > > > > > We are failing ovirt-engine master on test > > > 004_basic_sanity.hotplug_cpu > > > > > > looking at the logs, we can see that the for some reason, libvirt > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > CQ first failure was for patch: > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for > > > mdevs, > > > > > use nodisplay to override > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > > > > > related. > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and the problem is > > > > > ssh connection to the internal vm > > > > > > > > > > > > Thanks, > > > > > > Dafna > > > > > > > > > > > > > > > > > > error: > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > qemuDomainObjTaint:7521 : > > > > > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > tainted: hook-script > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > virProcessRunInMountNamespace:1159 : internal error: child reported: > > > unable > > > > > to set security context 'system_u:object_r:virt_content_t:s0' on > > > > > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > > > No such file or directory > > > > > > > > > > > > This should not happen, libvirt is not adding labels to files in > > > > > /rhev/data-center. It is using using its own mount > > > > > > namespace and adding there the devices used by the VM. Since libvirt > > > > > create the devices in its namespace > > > > > > it should not complain about missing paths in /rhev/data-center. > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > guest > > > > > agent is not connected > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > guest > > > > > agent is not connected > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > guest > > > > > agent is not connected > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > Nir > > > > > > > > > > -- > > > > > Anton Marchukov > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Infra mailing list -- infra(a)ovirt.org > > > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > oVirt Code of Conduct: > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > List Archives: > > > > > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... > > > > > > > > > > > > > > > > _______________________________________________ > > > Devel mailing list -- devel(a)ovirt.org > > > To unsubscribe send an email to devel-leave(a)ovirt.org > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > oVirt Code of Conduct: > > > https://www.ovirt.org/community/about/community-guidelines/ > > > List Archives: > > > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHR... > > > >

Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry(a)redhat.com&gt; ha scritto: > +1 from me > Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862

> > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler(a)redhat.com&gt; > wrote: > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > Dafna Ron <dron(a)redhat.com&gt; wrote: > > > > > > > This is still failing randomly > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to understand > > > which action is crashing the guest. > > > > > > > I was not able to reproduce the failure with the change above. > > We could merge the change to have better information on the next > > failure. > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler(a)redhat.com&gt; > wrote: > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > Ryan Barry <rbarry(a)redhat.com&gt; wrote: > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > I'm very suspicious of this happening immediately after > hotplugging a > > > > > NIC, > > > > > > especially since the bug attached to > https://gerrit.ovirt.org/#/c/98765/ > > > > > > talks about dropping packets. Dominik, did anything else change > here? > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > Is there already a pattern in the failed runs detected, or does it > fail > > > > > randomly? > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > amarchuk(a)redhat.com&gt; > > > > > > wrote: > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a > bug in > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer(a)redhat.com&gt; > wrote: > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron(a)redhat.com&gt; > wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > looking at the logs, we can see that the for some reason, > libvirt > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > display="on" for > > > > > mdevs, > > > > > > > use nodisplay to override > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be > netwrok > > > > > > > related. > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... > > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... > > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and the > problem is > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > qemuDomainObjTaint:7521 : > > > > > > > Domain id=3 name='vm0' > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > virProcessRunInMountNamespace:1159 : internal error: child > reported: > > > > > unable > > > > > > > to set security context 'system_u:object_r:virt_content_t:s0' > on > > > > > > > > > > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > > > > > No such file or directory > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > files in > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > namespace and adding there the devices used by the VM. Since > libvirt > > > > > > > create the devices in its namespace > > > > > > > > it should not complain about missing paths in > /rhev/data-center. > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > -- > > > > > > > Anton Marchukov > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Infra mailing list -- infra(a)ovirt.org > > > > > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > > oVirt Code of Conduct: > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > List Archives: > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Devel mailing list -- devel(a)ovirt.org > > > > > To unsubscribe send an email to devel-leave(a)ovirt.org > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > oVirt Code of Conduct: > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > List Archives: > > > > > > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHR... > > > > > > > > > > > > > -- > > Ryan Barry > > Associate Manager - RHV Virt/SLA > > rbarry(a)redhat.com M: +16518159306 IM: rbarry > _______________________________________________ > Infra mailing list -- infra(a)ovirt.org > To unsubscribe send an email to infra-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLN... >

On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler(a)redhat.com&gt; wrote: > > I added in > https://gerrit.ovirt.org/#/c/98925/ > a ping directly before the ssh. > The ping succeeds, but the ssh fails. > > > On Tue, 26 Mar 2019 17:07:45 +0100 > Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry(a)redhat.com&gt; ha > > scritto: > > > > > +1 from me > > > > > > > Merged. I have 2 patches constantly failing on it, rebased them, you can > > follow on: > > https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862 > > > > still failing on jenkins, but at least one succeeds locally for me Succeeds locally for me also. Dafna, are we sure there's not an infra issue?

> > > > > > > > > > > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler(a)redhat.com&gt; > > > wrote: > > > > > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > > > Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > > > Dafna Ron <dron(a)redhat.com&gt; wrote: > > > > > > > > > > > This is still failing randomly > > > > > > > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to understand > > > > > which action is crashing the guest. > > > > > > > > > > > > > I was not able to reproduce the failure with the change above. > > > > We could merge the change to have better information on the next > > > > failure. > > > > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < dholler(a)redhat.com&gt; > > > wrote: > > > > > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > > > Ryan Barry <rbarry(a)redhat.com&gt; wrote: > > > > > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > > > > > I'm very suspicious of this happening immediately after > > > hotplugging a > > > > > > > NIC, > > > > > > > > especially since the bug attached to > > > https://gerrit.ovirt.org/#/c/98765/ > > > > > > > > talks about dropping packets. Dominik, did anything else change > > > here? > > > > > > > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > > > > > Is there already a pattern in the failed runs detected, or does it > > > fail > > > > > > > randomly? > > > > > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > > > amarchuk(a)redhat.com&gt; > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a > > > bug in > > > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < nsoffer(a)redhat.com&gt; > > > wrote: > > > > > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < dron(a)redhat.com&gt; > > > wrote: > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > > > looking at the logs, we can see that the for some reason, > > > libvirt > > > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > > > display="on" for > > > > > > > mdevs, > > > > > > > > > use nodisplay to override > > > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be > > > netwrok > > > > > > > > > related. > > > > > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... > > > > > > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... > > > > > > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and the > > > problem is > > > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > > > qemuDomainObjTaint:7521 : > > > > > > > > > Domain id=3 name='vm0' > > > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > > > virProcessRunInMountNamespace:1159 : internal error: child > > > reported: > > > > > > > unable > > > > > > > > > to set security context 'system_u:object_r:virt_content_t:s0' > > > on > > > > > > > > > > > > > > > > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > > > > > > > No such file or directory > > > > > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > > > files in > > > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > > > namespace and adding there the devices used by the VM. Since > > > libvirt > > > > > > > > > create the devices in its namespace > > > > > > > > > > it should not complain about missing paths in > > > /rhev/data-center. > > > > > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > > > QEMU > > > > > > > guest > > > > > > > > > agent is not connected > > > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > > > QEMU > > > > > > > guest > > > > > > > > > agent is not connected > > > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > > > QEMU > > > > > > > guest > > > > > > > > > agent is not connected > > > > > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Anton Marchukov > > > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > Infra mailing list -- infra(a)ovirt.org > > > > > > > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > > > > oVirt Code of Conduct: > > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > > List Archives: > > > > > > > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Devel mailing list -- devel(a)ovirt.org > > > > > > > To unsubscribe send an email to devel-leave(a)ovirt.org > > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > > oVirt Code of Conduct: > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > List Archives: > > > > > > > > > > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHR... > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Ryan Barry > > > > > > Associate Manager - RHV Virt/SLA > > > > > > rbarry(a)redhat.com M: +16518159306 IM: rbarry > > > _______________________________________________ > > > Infra mailing list -- infra(a)ovirt.org > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > oVirt Code of Conduct: > > > https://www.ovirt.org/community/about/community-guidelines/ > > > List Archives: > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLN... > > > > > > > > -- Ryan Barry Associate Manager - RHV Virt/SLA rbarry(a)redhat.com M: +16518159306 IM: rbarry _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7...

On Wed, 27 Mar 2019 10:07:16 +0200 Eyal Edri <eedri(a)redhat.com&gt; wrote: > On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry(a)redhat.com&gt; wrote: > > > On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > > I added in > > > https://gerrit.ovirt.org/#/c/98925/ > > > a ping directly before the ssh. > > > The ping succeeds, but the ssh fails. > > > > > > > > > On Tue, 26 Mar 2019 17:07:45 +0100 > > > Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > > > > > Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry < rbarry(a)redhat.com&gt; > > ha > > > > scritto: > > > > > > > > > +1 from me > > > > > > > > > > > > > Merged. I have 2 patches constantly failing on it, rebased them, you > > can > > > > follow on: > > > > https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862 > > > > > > > > > > still failing on jenkins, but at least one succeeds locally for me > > > > Succeeds locally for me also. > > > > Dafna, are we sure there's not an infra issue? > > > > I think since its a race ( and we've seen failures on this test in the > past, also a race I think ), its probably hard to reproduce locally. > Also, we probably need to make sure the same Libvirt version is used. > The upstream servers are quite old, it can also be local run ends up being > faster and not hitting the same issues ( as we've seen in the past ) > > Could it be a bug in the ssh client ( paramiko? ) > Probably wrong idea, but worth to ask: Any ideas which ssh_timeout is used or how to modify? If 100 tries including a time.sleep(1) takes 100 seconds, either the timeout is not the expected 10 seconds, or the guest refuses the connection.

> Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can > cause this? ( other than slow servers ) > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler < dholler(a)redhat.com&gt; > > > > > wrote: > > > > > > > > > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > > > > > Dominik Holler <dholler(a)redhat.com&gt; wrote: > > > > > > > > > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > > > > > Dafna Ron <dron(a)redhat.com&gt; wrote: > > > > > > > > > > > > > > > This is still failing randomly > > > > > > > > > > > > > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to > > understand > > > > > > > which action is crashing the guest. > > > > > > > > > > > > > > > > > > > I was not able to reproduce the failure with the change above. > > > > > > We could merge the change to have better information on the next > > > > > > failure. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < > > dholler(a)redhat.com&gt; > > > > > wrote: > > > > > > > > > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > > > > > Ryan Barry <rbarry(a)redhat.com&gt; wrote: > > > > > > > > > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > > > > > > > > > I'm very suspicious of this happening immediately after > > > > > hotplugging a > > > > > > > > > NIC, > > > > > > > > > > especially since the bug attached to > > > > > https://gerrit.ovirt.org/#/c/98765/ > > > > > > > > > > talks about dropping packets. Dominik, did anything else > > change > > > > > here? > > > > > > > > > > > > > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > > > > > > > > > Is there already a pattern in the failed runs detected, or > > does it > > > > > fail > > > > > > > > > randomly? > > > > > > > > > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > > > > > amarchuk(a)redhat.com&gt; > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should > > open a > > > > > bug in > > > > > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < > > nsoffer(a)redhat.com&gt; > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < > > dron(a)redhat.com&gt; > > > > > wrote: > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > > > > > looking at the logs, we can see that the for some > > reason, > > > > > libvirt > > > > > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > > > > > display="on" for > > > > > > > > > mdevs, > > > > > > > > > > > use nodisplay to override > > > > > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may > > be > > > > > netwrok > > > > > > > > > > > related. > > > > > > > > > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... > > > > > > > > > > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... > > > > > > > > > > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and > > the > > > > > problem is > > > > > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > > > > > qemuDomainObjTaint:7521 : > > > > > > > > > > > Domain id=3 name='vm0' > > > > > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > > > > > virProcessRunInMountNamespace:1159 : internal error: > > child > > > > > reported: > > > > > > > > > unable > > > > > > > > > > > to set security context > > 'system_u:object_r:virt_content_t:s0' > > > > > on > > > > > > > > > > > > > > > > > > > > > > > > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > > > > > > > > > No such file or directory > > > > > > > > > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > > > > > files in > > > > > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > > > > > namespace and adding there the devices used by the VM. > > Since > > > > > libvirt > > > > > > > > > > > create the devices in its namespace > > > > > > > > > > > > it should not complain about missing paths in > > > > > /rhev/data-center. > > > > > > > > > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > > responding: > > > > > QEMU > > > > > > > > > guest > > > > > > > > > > > agent is not connected > > > > > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > > responding: > > > > > QEMU > > > > > > > > > guest > > > > > > > > > > > agent is not connected > > > > > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not > > responding: > > > > > QEMU > > > > > > > > > guest > > > > > > > > > > > agent is not connected > > > > > > > > > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > Anton Marchukov > > > > > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > Infra mailing list -- infra(a)ovirt.org > > > > > > > > > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > > > > > > > > > Privacy Statement: > > https://www.ovirt.org/site/privacy-policy/ > > > > > > > > > > > oVirt Code of Conduct: > > > > > > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > > > > List Archives: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > Devel mailing list -- devel(a)ovirt.org > > > > > > > > > To unsubscribe send an email to devel-leave(a)ovirt.org > > > > > > > > > Privacy Statement: > > https://www.ovirt.org/site/privacy-policy/ > > > > > > > > > oVirt Code of Conduct: > > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > > List Archives: > > > > > > > > > > > > > > > > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHR... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Ryan Barry > > > > > > > > > > Associate Manager - RHV Virt/SLA > > > > > > > > > > rbarry(a)redhat.com M: +16518159306 IM: rbarry > > > > > _______________________________________________ > > > > > Infra mailing list -- infra(a)ovirt.org > > > > > To unsubscribe send an email to infra-leave(a)ovirt.org > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > oVirt Code of Conduct: > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > List Archives: > > > > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLN... > > > > > > > > > > > > > > > > > > > > > > -- > > > > Ryan Barry > > > > Associate Manager - RHV Virt/SLA > > > > rbarry(a)redhat.com M: +16518159306 IM: rbarry > > _______________________________________________ > > Devel mailing list -- devel(a)ovirt.org > > To unsubscribe send an email to devel-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7... > > > > _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCD6OCOWXSH...

Hi I had a failure on my laptop when running in mock on hotplug cpu test. (when testing a change, got the same error on the jenkins) Dominik requested me to make a video of the vm0 dmesg command. https://drive.google.com/file/d/1Kr6r4SMhnVsWBvWD6E6JIZI4ddivo2pv/view?us...

Regards, Galit On Wed, Mar 27, 2019 at 2:39 PM Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > Il giorno mer 27 mar 2019 alle ore 09:54 Dominik Holler < > dholler(a)redhat.com&gt; ha scritto: > >> On Wed, 27 Mar 2019 10:07:16 +0200 >> Eyal Edri <eedri(a)redhat.com&gt; wrote: >> >> > On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry(a)redhat.com&gt; wrote: >> > >> > > On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler(a)redhat.com&gt; >> wrote: >> > > > >> > > > I added in >> > > > https://gerrit.ovirt.org/#/c/98925/ >> > > > a ping directly before the ssh. >> > > > The ping succeeds, but the ssh fails. >> > > > >> > > > >> > > > On Tue, 26 Mar 2019 17:07:45 +0100 >> > > > Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: >> > > > >> > > > > Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry < >> rbarry(a)redhat.com&gt; >> > > ha >> > > > > scritto: >> > > > > >> > > > > > +1 from me >> > > > > > >> > > > > >> > > > > Merged. I have 2 patches constantly failing on it, rebased them, >> you >> > > can >> > > > > follow on: >> > > > > https://gerrit.ovirt.org/#/c/98863/ and >> https://gerrit.ovirt.org/98862 >> > > > > >> > > > >> > > > still failing on jenkins, but at least one succeeds locally for me >> > > >> > > Succeeds locally for me also. >> > > >> > > Dafna, are we sure there's not an infra issue? >> > > >> > >> > I think since its a race ( and we've seen failures on this test in the >> > past, also a race I think ), its probably hard to reproduce locally. >> > Also, we probably need to make sure the same Libvirt version is used. >> > The upstream servers are quite old, it can also be local run ends up >> being >> > faster and not hitting the same issues ( as we've seen in the past ) >> > >> > Could it be a bug in the ssh client ( paramiko? ) >> > >> >> >> Probably wrong idea, but worth to ask: >> Any ideas which ssh_timeout is used or how to modify? >> >> If 100 tries including a time.sleep(1) takes 100 seconds, >> either the timeout is not the expected 10 seconds, or the guest refuses >> the connection. >> >> > I'm looking into a similar failure and found this on host1 logs at the > time of the ssh failure: > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/390... > > Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: > port 2(vnet1) entered blocking state > Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: > port 2(vnet1) entered disabled state > Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: device vnet1 > entered promiscuous mode > Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: > port 2(vnet1) entered blocking state > Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: > port 2(vnet1) entered forwarding state > Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: > <info> [1553672120.9133] manager: (vnet1): new Tun device > (/org/freedesktop/NetworkManager/Devices/44) > Mar 27 03:35:20 lago-basic-suite-master-host-1 lldpad: recvfrom(Event > interface): No buffer space available > Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: > <info> [1553672120.9277] device (vnet1): state change: unmanaged -> > unavailable (reason 'connection-assumed', sys-iface-state: 'external') > Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: > <info> [1553672120.9297] device (vnet1): state change: unavailable -> > disconnected (reason 'none', sys-iface-state: 'external') > Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i > vnet1 -j libvirt-J-vnet1' failed: Illegal target name 'libvirt-J-vnet1'. > Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet1 -j libvirt-P-vnet1' failed: Illegal target name 'libvirt-P-vnet1'. > Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet1' > failed: Chain 'libvirt-J-vnet1' doesn't exist. > Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' > failed: Chain 'libvirt-P-vnet1' doesn't exist. > Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet1' > failed: Chain 'libvirt-J-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet1' > failed: Chain 'libvirt-J-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet1' > failed: Chain 'libvirt-P-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet1' > failed: Chain 'libvirt-P-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-mac' > failed: Chain 'J-vnet1-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-mac' > failed: Chain 'J-vnet1-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-arp-mac' > failed: Chain 'J-vnet1-arp-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-arp-mac' > failed: Chain 'J-vnet1-arp-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: iptables > v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or > 'iptables --help' for more information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet1 -g FI-vnet1' failed: iptables v1.4.21: goto 'FI-vnet1' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet1 -g HI-vnet1' failed: iptables v1.4.21: goto 'HI-vnet1' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet1' failed: iptables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet1' failed: iptables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet1' failed: iptables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet1' failed: iptables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet1' failed: iptables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet1' failed: iptables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet1 FO-vnet1' failed: > iptables: No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet1 FI-vnet1' failed: > iptables: No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet1 HI-vnet1' failed: > iptables: No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: ip6tables > v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or > 'ip6tables --help' for more information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet1 -g FI-vnet1' failed: ip6tables v1.4.21: goto 'FI-vnet1' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet1 -g HI-vnet1' failed: ip6tables v1.4.21: goto 'HI-vnet1' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet1' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet1' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet1' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet1' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet1' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet1' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet1 FO-vnet1' failed: > ip6tables: No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet1 FI-vnet1' failed: > ip6tables: No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet1 HI-vnet1' failed: > ip6tables: No chain/target/match by that name. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i > vnet1 -j libvirt-I-vnet1' failed: Illegal target name 'libvirt-I-vnet1'. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet1 -j libvirt-O-vnet1' failed: Illegal target name 'libvirt-O-vnet1'. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet1' > failed: Chain 'libvirt-I-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet1' > failed: Chain 'libvirt-O-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet1' > failed: Chain 'libvirt-I-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet1' > failed: Chain 'libvirt-I-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet1' > failed: Chain 'libvirt-O-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet1' > failed: Chain 'libvirt-O-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' > failed: Chain 'libvirt-P-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet1 > libvirt-O-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-mac' > failed: Chain 'I-vnet1-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-mac' > failed: Chain 'I-vnet1-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-arp-mac' > failed: Chain 'I-vnet1-arp-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-arp-mac' > failed: Chain 'I-vnet1-arp-mac' doesn't exist. > Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event > interface): No buffer space available > Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event > interface): No buffer space available > Mar 27 03:35:28 lago-basic-suite-master-host-1 systemd: Created slice User > Slice of root. > > On the lago log at corresponding time frame: > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/390... > > 2019-03-27 07:35:23,801::log_utils.py::end_log_task::670::nose::INFO:: # hotplug_nic: [32mSuccess [0m (in 0:00:05) > 2019-03-27 07:35:23,801::log_utils.py::start_log_task::655::nose::INFO:: # hotplug_cpu: [0m [0m > > > Around the same time on host 0: > https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/390... > > Mar 27 03:33:06 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port > 2(vnet0) entered disabled state > Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: > <info> [1553671986.5243] device (vnet0): state change: disconnected -> > unmanaged (reason 'unmanaged', sys-iface-state: 'removed') > Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: > <info> [1553671986.5251] device (vnet0): released from master device > ovirtmgmt > Mar 27 03:33:06 lago-basic-suite-master-host-0 lldpad: recvfrom(Event > interface): No buffer space available > Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd-machined: Machine > qemu-2-vm2 terminated. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables > v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or > 'iptables --help' for more information. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables > v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or > 'ip6tables --help' for more information. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i > vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables > v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or > 'iptables --help' for more information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev > --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching > rule exist in that chain?). > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables > v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or > 'ip6tables --help' for more information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Created slice User > Slice of root. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Started Session > c453 of user root. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev > --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching > rule exist in that chain?). > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove > map (uevent) > Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: devmap > not registered, can't remove > Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove > map (uevent) > Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. > Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User > Slice of root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session > c454 of user root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User > Slice of root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session > c455 of user root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove > map (uevent) > Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: devmap > not registered, can't remove > Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove > map (uevent) > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User > Slice of root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session > c456 of user root. > Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: > /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 > already removed > Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: > /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 > already removed > Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port > 2(vnet0) entered blocking state > Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port > 2(vnet0) entered disabled state > Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: device vnet0 > entered promiscuous mode > Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port > 2(vnet0) entered blocking state > Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port > 2(vnet0) entered forwarding state > Mar 27 03:33:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event > interface): No buffer space available > Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: > <info> [1553671996.4807] manager: (vnet0): new Tun device > (/org/freedesktop/NetworkManager/Devices/49) > Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: > <info> [1553671996.4970] device (vnet0): state change: unmanaged -> > unavailable (reason 'connection-assumed', sys-iface-state: 'external') > Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: > <info> [1553671996.4997] device (vnet0): state change: unavailable -> > disconnected (reason 'none', sys-iface-state: 'external') > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i > vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' > failed: Chain 'J-vnet0-mac' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' > failed: Chain 'J-vnet0-mac' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' > failed: Chain 'J-vnet0-arp-mac' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' > failed: Chain 'J-vnet0-arp-mac' doesn't exist. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables > v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or > 'iptables --help' for more information. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: > iptables: No chain/target/match by that name. > Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: > iptables: No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: > iptables: No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables > v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or > 'ip6tables --help' for more information. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: > ip6tables: No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: > ip6tables: No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: > ip6tables: No chain/target/match by that name. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i > vnet0 -j libvirt-I-vnet0' failed: Illegal target name 'libvirt-I-vnet0'. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet0' > failed: Chain 'libvirt-I-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet0' > failed: Chain 'libvirt-I-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet0' > failed: Chain 'libvirt-I-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet0 > libvirt-O-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-mac' > failed: Chain 'I-vnet0-mac' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-mac' > failed: Chain 'I-vnet0-mac' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-arp-mac' > failed: Chain 'I-vnet0-arp-mac' doesn't exist. > Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-arp-mac' > failed: Chain 'I-vnet0-arp-mac' doesn't exist. > Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd-machined: New > machine qemu-3-vm2. > Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd: Started Virtual > Machine qemu-3-vm2. > Mar 27 03:33:18 lago-basic-suite-master-host-0 kvm: 1 guest now active > Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Created slice User > Slice of root. > Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Started Session > c457 of user root. > Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Created slice User > Slice of root. > Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Started Session > c458 of user root. > Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Created slice User > Slice of root. > Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Started Session > c459 of user root. > Mar 27 03:33:36 lago-basic-suite-master-host-0 systemd: Removed slice User > Slice of root. > Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port > 2(vnet0) entered disabled state > Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: device vnet0 left > promiscuous mode > Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port > 2(vnet0) entered disabled state > Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: > <info> [1553672056.8430] device (vnet0): state change: disconnected -> > unmanaged (reason 'unmanaged', sys-iface-state: 'removed') > Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: > <info> [1553672056.8437] device (vnet0): released from master device > ovirtmgmt > Mar 27 03:34:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event > interface): No buffer space available > Mar 27 03:34:16 lago-basic-suite-master-host-0 kvm: 0 guests now active > Mar 27 03:34:16 lago-basic-suite-master-host-0 systemd-machined: Machine > qemu-3-vm2 terminated. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables > v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or > 'iptables --help' for more information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables > v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or > 'ip6tables --help' for more information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i > vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' > failed: Chain 'libvirt-J-vnet0' doesn't exist. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' > failed: Chain 'libvirt-P-vnet0' doesn't exist. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables > v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or > 'iptables --help' for more information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' > is not a chain#012#012Try `iptables -h' or 'iptables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev > --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching > rule exist in that chain?). > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables > v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or > 'ip6tables --help' for more information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev > --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev > --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev > --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' > is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more > information. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev > --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching > rule exist in that chain?). > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: > No chain/target/match by that name. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o > vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. > Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: > COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' > failed: Chain 'libvirt-O-vnet0' doesn't exist. > Mar 27 03:34:18 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor > state: count=5 workers=set([<Worker name=periodic/0 waiting task#=207 at > 0x7f8fac0b5310>, <Worker name=periodic/3 waiting task#=194 at > 0x7f8fac0b5b10>, <Worker name=periodic/4 waiting task#=0 at > 0x7f8f5c412190>, <Worker name=periodic/1 waiting task#=200 at > 0x7f8fac0b5550>, <Worker name=periodic/2 running <Task discardable > <Operation action=<VmDispatcher operation=<class > 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at > 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c46f650> discarded > task#=214 at 0x7f8fac0b5810>]) > Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: > /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 > already removed > Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: > /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 > already removed > Mar 27 03:34:20 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor > state: count=6 workers=set([<Worker name=periodic/2 running <Task > discardable <Operation action=<VmDispatcher operation=<class > 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at > 0x7f8fac0c6290> timeout=1.0, duration=3.01 at 0x7f8f5c46f650> discarded > task#=214 at 0x7f8fac0b5810>, <Worker name=periodic/3 running <Task > discardable <Operation action=<VmDispatcher operation=<class > 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at > 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c35ba90> discarded > task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/0 waiting task#=208 at > 0x7f8fac0b5310>, <Worker name=periodic/1 running <Task discardable > <Operation action=<VmDispatcher operation=<class > 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7f8fac0c61d0> at 0x7f8fac0c6210> > timeout=7.5, duration=0.44 at 0x7f8f5c2dd250> task#=200 at 0x7f8fac0b5550>, > <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker > name=periodic/5 waiting task#=0 at 0x7f8f8c13dfd0>]) > > > > >> >> > Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can >> > cause this? ( other than slow servers ) >> > >> > >> > > >> > > > >> > > > > >> > > > > >> > > > > > >> > > > > > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler < >> dholler(a)redhat.com&gt; >> > > > > > wrote: >> > > > > > > >> > > > > > > On Tue, 26 Mar 2019 12:31:36 +0100 >> > > > > > > Dominik Holler <dholler(a)redhat.com&gt; wrote: >> > > > > > > >> > > > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 >> > > > > > > > Dafna Ron <dron(a)redhat.com&gt; wrote: >> > > > > > > > >> > > > > > > > > This is still failing randomly >> > > > > > > > > >> > > > > > > > >> > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to >> > > understand >> > > > > > > > which action is crashing the guest. >> > > > > > > > >> > > > > > > >> > > > > > > I was not able to reproduce the failure with the change above. >> > > > > > > We could merge the change to have better information on the >> next >> > > > > > > failure. >> > > > > > > >> > > > > > > >> > > > > > > > > >> > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < >> > > dholler(a)redhat.com&gt; >> > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 >> > > > > > > > > > Ryan Barry <rbarry(a)redhat.com&gt; wrote: >> > > > > > > > > > >> > > > > > > > > > > It may be virt, but I'm looking... >> > > > > > > > > > > >> > > > > > > > > > > I'm very suspicious of this happening immediately >> after >> > > > > > hotplugging a >> > > > > > > > > > NIC, >> > > > > > > > > > > especially since the bug attached to >> > > > > > https://gerrit.ovirt.org/#/c/98765/ >> > > > > > > > > > > talks about dropping packets. Dominik, did anything >> else >> > > change >> > > > > > here? >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > No, nothing I am aware of. >> > > > > > > > > > >> > > > > > > > > > Is there already a pattern in the failed runs detected, >> or >> > > does it >> > > > > > fail >> > > > > > > > > > randomly? >> > > > > > > > > > >> > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < >> > > > > > amarchuk(a)redhat.com&gt; >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Which team is it? Is it Virt? Just checking who >> should >> > > open a >> > > > > > bug in >> > > > > > > > > > > > libvirt as suggested. >> > > > > > > > > > > > >> > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < >> > > nsoffer(a)redhat.com&gt; >> > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < >> > > dron(a)redhat.com&gt; >> > > > > > wrote: >> > > > > > > > > > > > > Hi, >> > > > > > > > > > > > > >> > > > > > > > > > > > > We are failing ovirt-engine master on test >> > > > > > > > > > 004_basic_sanity.hotplug_cpu >> > > > > > > > > > > > > looking at the logs, we can see that the for some >> > > reason, >> > > > > > libvirt >> > > > > > > > > > > > reports a vm as none responsive which fails the >> test. >> > > > > > > > > > > > > >> > > > > > > > > > > > > CQ first failure was for patch: >> > > > > > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add >> > > > > > display="on" for >> > > > > > > > > > mdevs, >> > > > > > > > > > > > use nodisplay to override >> > > > > > > > > > > > > But I do not think this is the cause of failure. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think >> it may >> > > be >> > > > > > netwrok >> > > > > > > > > > > > related. >> > > > > > > > > > > > > >> > > > > > > > > > > > > You can see the libvirt log here: >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > >> > > >> https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/arti... >> > > > > > > > > > > > > >> > > > > > > > > > > > > you can see the full logs here: >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > >> > > >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artif... >> > > > > > > > > > > > > >> > > > > > > > > > > > > Evgheni and I confirmed this is not an infra >> issue and >> > > the >> > > > > > problem is >> > > > > > > > > > > > ssh connection to the internal vm >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > Dafna >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > error: >> > > > > > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : >> > > > > > > > > > qemuDomainObjTaint:7521 : >> > > > > > > > > > > > Domain id=3 name='vm0' >> > > > > > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is >> > > > > > > > > > > > tainted: hook-script >> > > > > > > > > > > > > >> > > > > > > > > > > > > Why our vm is tainted? >> > > > > > > > > > > > > >> > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : >> > > > > > > > > > > > virProcessRunInMountNamespace:1159 : internal error: >> > > child >> > > > > > reported: >> > > > > > > > > > unable >> > > > > > > > > > > > to set security context >> > > 'system_u:object_r:virt_content_t:s0' >> > > > > > on >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > >> > > >> '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': >> > > > > > > > > > > > No such file or directory >> > > > > > > > > > > > > >> > > > > > > > > > > > > This should not happen, libvirt is not adding >> labels to >> > > > > > files in >> > > > > > > > > > > > /rhev/data-center. It is using using its own mount >> > > > > > > > > > > > > namespace and adding there the devices used by >> the VM. >> > > Since >> > > > > > libvirt >> > > > > > > > > > > > create the devices in its namespace >> > > > > > > > > > > > > it should not complain about missing paths in >> > > > > > /rhev/data-center. >> > > > > > > > > > > > > >> > > > > > > > > > > > > I think we should file a libvirt bug for this. >> > > > > > > > > > > > > >> > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : >> > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not >> > > responding: >> > > > > > QEMU >> > > > > > > > > > guest >> > > > > > > > > > > > agent is not connected >> > > > > > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : >> > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not >> > > responding: >> > > > > > QEMU >> > > > > > > > > > guest >> > > > > > > > > > > > agent is not connected >> > > > > > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : >> > > > > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not >> > > responding: >> > > > > > QEMU >> > > > > > > > > > guest >> > > > > > > > > > > > agent is not connected >> > > > > > > > > > > > > >> > > > > > > > > > > > > Do we have guest agent in the test VMs? >> > > > > > > > > > > > > >> > > > > > > > > > > > > Nir >> > > > > > > > > > > > >> > > > > > > > > > > > -- >> > > > > > > > > > > > Anton Marchukov >> > > > > > > > > > > > Associate Manager - RHV DevOps - Red Hat >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > _______________________________________________ >> > > > > > > > > > > > Infra mailing list -- infra(a)ovirt.org >> > > > > > > > > > > > To unsubscribe send an email to >> infra-leave(a)ovirt.org >> > > > > > > > > > > > Privacy Statement: >> > > https://www.ovirt.org/site/privacy-policy/ >> > > > > > > > > > > > oVirt Code of Conduct: >> > > > > > > > > > > > >> > > https://www.ovirt.org/community/about/community-guidelines/ >> > > > > > > > > > > > List Archives: >> > > > > > > > > > > > >> > > > > > > > > > >> > > > > > >> > > >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JU... >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > _______________________________________________ >> > > > > > > > > > Devel mailing list -- devel(a)ovirt.org >> > > > > > > > > > To unsubscribe send an email to devel-leave(a)ovirt.org >> > > > > > > > > > Privacy Statement: >> > > https://www.ovirt.org/site/privacy-policy/ >> > > > > > > > > > oVirt Code of Conduct: >> > > > > > > > > > >> https://www.ovirt.org/community/about/community-guidelines/ >> > > > > > > > > > List Archives: >> > > > > > > > > > >> > > > > > >> > > >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHR... >> > > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > >> > > > > > Ryan Barry >> > > > > > >> > > > > > Associate Manager - RHV Virt/SLA >> > > > > > >> > > > > > rbarry(a)redhat.com M: +16518159306 IM: rbarry >> > > > > > _______________________________________________ >> > > > > > Infra mailing list -- infra(a)ovirt.org >> > > > > > To unsubscribe send an email to infra-leave(a)ovirt.org >> > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> > > > > > oVirt Code of Conduct: >> > > > > > https://www.ovirt.org/community/about/community-guidelines/ >> > > > > > List Archives: >> > > > > > >> > > >> https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLN... >> > > > > > >> > > > > >> > > > > >> > > > >> > > >> > > >> > > -- >> > > >> > > Ryan Barry >> > > >> > > Associate Manager - RHV Virt/SLA >> > > >> > > rbarry(a)redhat.com M: +16518159306 IM: rbarry >> > > _______________________________________________ >> > > Devel mailing list -- devel(a)ovirt.org >> > > To unsubscribe send an email to devel-leave(a)ovirt.org >> > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> > > oVirt Code of Conduct: >> > > https://www.ovirt.org/community/about/community-guidelines/ >> > > List Archives: >> > > >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7... >> > > >> > >> > >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCD6OCOWXSH... >> > > > -- > > SANDRO BONAZZOLA > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA <https://www.redhat.com/> > > sbonazzo(a)redhat.com > <https://red.ht/sig> >