[ OST Failure Report ] [ oVirt master (ovirt-engine) ] [ 22-03-2019 ] [004_basic_sanity.hotplug_cpu ]
Hi, We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test. CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure. Adding Marcin, Milan and Dan as well as I think it may be netwrok related. You can see the libvirt log here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac... you can see the full logs here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact... Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm Thanks, Dafna error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script 2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory 2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote:
Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm
Thanks, Dafna
error:
2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory
This should not happen, libvirt is not adding labels to files in
/rhev/data-center. It is using using its own mount namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center. I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
we have more failures with this error. I checked latest failure and guest-agent is installed. logs can be found here for latest failed job: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/371/artifact/basi... 2019-03-25 14:57:05,176::INFO::repoman.common.stores.RPM::Adding package /var/lib/lago/ovirt-4.3-snapshot-static-el7/noarch/ovirt-guest-agent-common-1.0.16-1.el7.noarch.rpm to repo Non persistent RPMStore On Fri, Mar 22, 2019 at 7:52 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote:
Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm
Thanks, Dafna
error:
2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory
This should not happen, libvirt is not adding labels to files in
/rhev/data-center. It is using using its own mount namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center.
I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
Which team is it? Is it Virt? Just checking who should open a bug in libvirt as suggested.
On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm
Thanks, Dafna
error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory
This should not happen, libvirt is not adding labels to files in /rhev/data-center. It is using using its own mount namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center.
I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
It may be virt, but I'm looking... I'm very suspicious of this happening immediately after hotplugging a NIC, especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here? On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Which team is it? Is it Virt? Just checking who should open a bug in libvirt as suggested.
On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is
ssh connection to the internal vm
Thanks, Dafna
error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 :
Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error :
virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory
This should not happen, libvirt is not adding labels to files in
/rhev/data-center. It is using using its own mount
namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center.
I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
-- Ryan Barry Associate Manager - RHV Virt/SLA rbarry@redhat.com M: +16518159306 IM: rbarry <https://red.ht/sig>
On Mon, 25 Mar 2019 17:30:53 -0400 Ryan Barry <rbarry@redhat.com> wrote:
It may be virt, but I'm looking...
I'm very suspicious of this happening immediately after hotplugging a NIC, especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here?
No, nothing I am aware of. Is there already a pattern in the failed runs detected, or does it fail randomly?
On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Which team is it? Is it Virt? Just checking who should open a bug in libvirt as suggested.
On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is
ssh connection to the internal vm
Thanks, Dafna
error: 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 :
Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error :
virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory
This should not happen, libvirt is not adding labels to files in
/rhev/data-center. It is using using its own mount
namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center.
I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
This is still failing randomly On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler@redhat.com> wrote:
On Mon, 25 Mar 2019 17:30:53 -0400 Ryan Barry <rbarry@redhat.com> wrote:
It may be virt, but I'm looking...
I'm very suspicious of this happening immediately after hotplugging a NIC, especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here?
No, nothing I am aware of.
Is there already a pattern in the failed runs detected, or does it fail randomly?
On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Which team is it? Is it Virt? Just checking who should open a bug in libvirt as suggested.
On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is
ssh connection to the internal vm
Thanks, Dafna
error: 2019-03-22 15:08:22.658+0000: 22068: warning :
qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error :
virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
No such file or directory
This should not happen, libvirt is not adding labels to files in
/rhev/data-center. It is using using its own mount
namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center.
I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
On Tue, 26 Mar 2019 10:58:22 +0000 Dafna Ron <dron@redhat.com> wrote:
This is still failing randomly
I created https://gerrit.ovirt.org/#/c/98906/ to help to understand which action is crashing the guest.
On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler@redhat.com> wrote:
On Mon, 25 Mar 2019 17:30:53 -0400 Ryan Barry <rbarry@redhat.com> wrote:
It may be virt, but I'm looking...
I'm very suspicious of this happening immediately after hotplugging a NIC, especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here?
No, nothing I am aware of.
Is there already a pattern in the failed runs detected, or does it fail randomly?
On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Which team is it? Is it Virt? Just checking who should open a bug in libvirt as suggested.
On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is
ssh connection to the internal vm
Thanks, Dafna
error: 2019-03-22 15:08:22.658+0000: 22068: warning :
qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error :
virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
No such file or directory
This should not happen, libvirt is not adding labels to files in
/rhev/data-center. It is using using its own mount
namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center.
I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 10:58:22 +0000 Dafna Ron <dron@redhat.com> wrote:
This is still failing randomly
I created https://gerrit.ovirt.org/#/c/98906/ to help to understand which action is crashing the guest.
I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure.
On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler@redhat.com> wrote:
On Mon, 25 Mar 2019 17:30:53 -0400 Ryan Barry <rbarry@redhat.com> wrote:
It may be virt, but I'm looking...
I'm very suspicious of this happening immediately after hotplugging a NIC, especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here?
No, nothing I am aware of.
Is there already a pattern in the failed runs detected, or does it fail randomly?
On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Which team is it? Is it Virt? Just checking who should open a bug in libvirt as suggested.
On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: Hi,
We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.
CQ first failure was for patch: https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override But I do not think this is the cause of failure.
Adding Marcin, Milan and Dan as well as I think it may be netwrok related.
You can see the libvirt log here:
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
you can see the full logs here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
Evgheni and I confirmed this is not an infra issue and the problem is
ssh connection to the internal vm
Thanks, Dafna
error: 2019-03-22 15:08:22.658+0000: 22068: warning :
qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?
2019-03-22 15:08:22.693+0000: 22068: error :
virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
No such file or directory
This should not happen, libvirt is not adding labels to files in
/rhev/data-center. It is using using its own mount
namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace it should not complain about missing paths in /rhev/data-center.
I think we should file a libvirt bug for this.
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?
Nir
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
+1 from me On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 10:58:22 +0000 Dafna Ron <dron@redhat.com> wrote:
This is still failing randomly
I created https://gerrit.ovirt.org/#/c/98906/ to help to understand which action is crashing the guest.
I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure.
On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler@redhat.com> wrote:
On Mon, 25 Mar 2019 17:30:53 -0400 Ryan Barry <rbarry@redhat.com> wrote:
It may be virt, but I'm looking...
I'm very suspicious of this happening immediately after hotplugging a NIC, especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here?
No, nothing I am aware of.
Is there already a pattern in the failed runs detected, or does it fail randomly?
On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov <amarchuk@redhat.com> wrote:
Which team is it? Is it Virt? Just checking who should open a bug in libvirt as suggested.
> On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote: > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: > Hi, > > We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu > looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test. > > CQ first failure was for patch: > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override > But I do not think this is the cause of failure. > > Adding Marcin, Milan and Dan as well as I think it may be netwrok related. > > You can see the libvirt log here: >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > you can see the full logs here: > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm > > Thanks, > Dafna > > > error: > 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script > > Why our vm is tainted? > > 2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
No such file or directory > > This should not happen, libvirt is not adding labels to files in /rhev/data-center. It is using using its own mount > namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace > it should not complain about missing paths in /rhev/data-center. > > I think we should file a libvirt bug for this. > > 2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected > 2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected > 2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected > > Do we have guest agent in the test VMs? > > Nir
-- Anton Marchukov Associate Manager - RHV DevOps - Red Hat
_______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
-- Ryan Barry Associate Manager - RHV Virt/SLA rbarry@redhat.com M: +16518159306 IM: rbarry
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry@redhat.com> ha scritto:
+1 from me
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 10:58:22 +0000 Dafna Ron <dron@redhat.com> wrote:
This is still failing randomly
I created https://gerrit.ovirt.org/#/c/98906/ to help to understand which action is crashing the guest.
I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure.
On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler@redhat.com>
On Mon, 25 Mar 2019 17:30:53 -0400 Ryan Barry <rbarry@redhat.com> wrote:
It may be virt, but I'm looking...
I'm very suspicious of this happening immediately after
hotplugging a
NIC,
especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ talks about dropping packets. Dominik, did anything else change here?
No, nothing I am aware of.
Is there already a pattern in the failed runs detected, or does it fail randomly?
On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < amarchuk@redhat.com> wrote:
> Which team is it? Is it Virt? Just checking who should open a bug in > libvirt as suggested. > > > On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote: > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: > > Hi, > > > > We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu > > looking at the logs, we can see that the for some reason,
> reports a vm as none responsive which fails the test. > > > > CQ first failure was for patch: > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, > use nodisplay to override > > But I do not think this is the cause of failure. > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > related. > > > > You can see the libvirt log here: > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > you can see the full logs here: > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > Evgheni and I confirmed this is not an infra issue and the
> ssh connection to the internal vm > > > > Thanks, > > Dafna > > > > > > error: > > 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > tainted: hook-script > > > > Why our vm is tainted? > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > virProcessRunInMountNamespace:1159 : internal error: child reported: unable > to set security context 'system_u:object_r:virt_content_t:s0' on >
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
> No such file or directory > > > > This should not happen, libvirt is not adding labels to files in > /rhev/data-center. It is using using its own mount > > namespace and adding there the devices used by the VM. Since
wrote: libvirt problem is libvirt
> create the devices in its namespace > > it should not complain about missing paths in /rhev/data-center. > > > > I think we should file a libvirt bug for this. > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > agent is not connected > > 2019-03-22 15:08:58.193+0000: 22070: error : > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > agent is not connected > > 2019-03-22 15:13:58.179+0000: 22071: error : > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > agent is not connected > > > > Do we have guest agent in the test VMs? > > > > Nir > > -- > Anton Marchukov > Associate Manager - RHV DevOps - Red Hat > > > > > > _______________________________________________ > Infra mailing list -- infra@ovirt.org > To unsubscribe send an email to infra-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
>
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
-- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig>
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails. On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry@redhat.com> ha scritto:
+1 from me
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
still failing on jenkins, but at least one succeeds locally for me
On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 10:58:22 +0000 Dafna Ron <dron@redhat.com> wrote:
This is still failing randomly
I created https://gerrit.ovirt.org/#/c/98906/ to help to understand which action is crashing the guest.
I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure.
On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler@redhat.com>
On Mon, 25 Mar 2019 17:30:53 -0400 Ryan Barry <rbarry@redhat.com> wrote:
> It may be virt, but I'm looking... > > I'm very suspicious of this happening immediately after
hotplugging a
NIC, > especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ > talks about dropping packets. Dominik, did anything else change here? >
No, nothing I am aware of.
Is there already a pattern in the failed runs detected, or does it fail randomly?
> On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < amarchuk@redhat.com> > wrote: > > > Which team is it? Is it Virt? Just checking who should open a bug in > > libvirt as suggested. > > > > > On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote: > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: > > > Hi, > > > > > > We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu > > > looking at the logs, we can see that the for some reason,
> > reports a vm as none responsive which fails the test. > > > > > > CQ first failure was for patch: > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, > > use nodisplay to override > > > But I do not think this is the cause of failure. > > > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > > related. > > > > > > You can see the libvirt log here: > > > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > > > you can see the full logs here: > > > > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > > > Evgheni and I confirmed this is not an infra issue and the
> > ssh connection to the internal vm > > > > > > Thanks, > > > Dafna > > > > > > > > > error: > > > 2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : > > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > tainted: hook-script > > > > > > Why our vm is tainted? > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > virProcessRunInMountNamespace:1159 : internal error: child reported: unable > > to set security context 'system_u:object_r:virt_content_t:s0' on > >
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
> > No such file or directory > > > > > > This should not happen, libvirt is not adding labels to files in > > /rhev/data-center. It is using using its own mount > > > namespace and adding there the devices used by the VM. Since
wrote: libvirt problem is libvirt
> > create the devices in its namespace > > > it should not complain about missing paths in /rhev/data-center. > > > > > > I think we should file a libvirt bug for this. > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > > agent is not connected > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > > agent is not connected > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest > > agent is not connected > > > > > > Do we have guest agent in the test VMs? > > > > > > Nir > > > > -- > > Anton Marchukov > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > _______________________________________________ > > Infra mailing list -- infra@ovirt.org > > To unsubscribe send an email to infra-leave@ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
> > > > _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails.
On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry@redhat.com> ha scritto:
+1 from me
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
still failing on jenkins, but at least one succeeds locally for me
Succeeds locally for me also. Dafna, are we sure there's not an infra issue?
On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 10:58:22 +0000 Dafna Ron <dron@redhat.com> wrote:
This is still failing randomly
I created https://gerrit.ovirt.org/#/c/98906/ to help to understand which action is crashing the guest.
I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure.
On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler <dholler@redhat.com>
> On Mon, 25 Mar 2019 17:30:53 -0400 > Ryan Barry <rbarry@redhat.com> wrote: > > > It may be virt, but I'm looking... > > > > I'm very suspicious of this happening immediately after
hotplugging a
> NIC, > > especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ > > talks about dropping packets. Dominik, did anything else change here? > > > > No, nothing I am aware of. > > Is there already a pattern in the failed runs detected, or does it fail > randomly? > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < amarchuk@redhat.com> > > wrote: > > > > > Which team is it? Is it Virt? Just checking who should open a bug in > > > libvirt as suggested. > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer <nsoffer@redhat.com> wrote: > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote: > > > > Hi, > > > > > > > > We are failing ovirt-engine master on test > 004_basic_sanity.hotplug_cpu > > > > looking at the logs, we can see that the for some reason,
> > > reports a vm as none responsive which fails the test. > > > > > > > > CQ first failure was for patch: > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for > mdevs, > > > use nodisplay to override > > > > But I do not think this is the cause of failure. > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > > > related. > > > > > > > > You can see the libvirt log here: > > > > > > > > https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac... > > > > > > > > you can see the full logs here: > > > > > > > > > > > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact... > > > > > > > > Evgheni and I confirmed this is not an infra issue and the
> > > ssh connection to the internal vm > > > > > > > > Thanks, > > > > Dafna > > > > > > > > > > > > error: > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > qemuDomainObjTaint:7521 : > > > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > tainted: hook-script > > > > > > > > Why our vm is tainted? > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > virProcessRunInMountNamespace:1159 : internal error: child reported: > unable > > > to set security context 'system_u:object_r:virt_content_t:s0' on > > > > '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': > > > No such file or directory > > > > > > > > This should not happen, libvirt is not adding labels to files in > > > /rhev/data-center. It is using using its own mount > > > > namespace and adding there the devices used by the VM. Since
wrote: libvirt problem is libvirt
> > > create the devices in its namespace > > > > it should not complain about missing paths in /rhev/data-center. > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > guest > > > agent is not connected > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > guest > > > agent is not connected > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > guest > > > agent is not connected > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > Nir > > > > > > -- > > > Anton Marchukov > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Infra mailing list -- infra@ovirt.org > > > To unsubscribe send an email to infra-leave@ovirt.org > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > oVirt Code of Conduct: > > > https://www.ovirt.org/community/about/community-guidelines/ > > > List Archives: > > > > https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW... > > > > > > > > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5... >
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
-- Ryan Barry Associate Manager - RHV Virt/SLA rbarry@redhat.com M: +16518159306 IM: rbarry
On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry@redhat.com> wrote:
On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails.
On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry@redhat.com>
ha
scritto:
+1 from me
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
still failing on jenkins, but at least one succeeds locally for me
Succeeds locally for me also.
Dafna, are we sure there's not an infra issue?
I think since its a race ( and we've seen failures on this test in the past, also a race I think ), its probably hard to reproduce locally. Also, we probably need to make sure the same Libvirt version is used. The upstream servers are quite old, it can also be local run ends up being faster and not hitting the same issues ( as we've seen in the past ) Could it be a bug in the ssh client ( paramiko? ) Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can cause this? ( other than slow servers )
On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 10:58:22 +0000 Dafna Ron <dron@redhat.com> wrote:
> This is still failing randomly >
I created https://gerrit.ovirt.org/#/c/98906/ to help to
which action is crashing the guest.
I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure.
> > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < dholler@redhat.com> wrote: > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > Ryan Barry <rbarry@redhat.com> wrote: > > > > > It may be virt, but I'm looking... > > > > > > I'm very suspicious of this happening immediately after hotplugging a > > NIC, > > > especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ > > > talks about dropping packets. Dominik, did anything else change here? > > > > > > > No, nothing I am aware of. > > > > Is there already a pattern in the failed runs detected, or does it fail > > randomly? > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < amarchuk@redhat.com> > > > wrote: > > > > > > > Which team is it? Is it Virt? Just checking who should open a bug in > > > > libvirt as suggested. > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < nsoffer@redhat.com> wrote: > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < dron@redhat.com> wrote: > > > > > Hi, > > > > > > > > > > We are failing ovirt-engine master on test > > 004_basic_sanity.hotplug_cpu > > > > > looking at the logs, we can see that the for some reason, libvirt > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > CQ first failure was for patch: > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for > > mdevs, > > > > use nodisplay to override > > > > > But I do not think this is the cause of failure. > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > > > > related. > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > > > > > > > Evgheni and I confirmed this is not an infra issue and
understand the
problem is
> > > > ssh connection to the internal vm > > > > > > > > > > Thanks, > > > > > Dafna > > > > > > > > > > > > > > > error: > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > qemuDomainObjTaint:7521 : > > > > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > tainted: hook-script > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > virProcessRunInMountNamespace:1159 : internal error: child reported: > > unable > > > > to set security context 'system_u:object_r:virt_content_t:s0' on > > > > > >
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
> > > > No such file or directory > > > > > > > > > > This should not happen, libvirt is not adding labels to files in > > > > /rhev/data-center. It is using using its own mount > > > > > namespace and adding there the devices used by the VM. Since libvirt > > > > create the devices in its namespace > > > > > it should not complain about missing paths in /rhev/data-center. > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > guest > > > > agent is not connected > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > guest > > > > agent is not connected > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > guest > > > > agent is not connected > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > Nir > > > > > > > > -- > > > > Anton Marchukov > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Infra mailing list -- infra@ovirt.org > > > > To unsubscribe send an email to infra-leave@ovirt.org > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > oVirt Code of Conduct: > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > List Archives: > > > > > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
> > > > > > > > > > > > _______________________________________________ > > Devel mailing list -- devel@ovirt.org > > To unsubscribe send an email to devel-leave@ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > >
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
> >
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7OSQ...
-- Eyal edri MANAGER RHV/CNV DevOps EMEA VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)
On Wed, 27 Mar 2019 10:07:16 +0200 Eyal Edri <eedri@redhat.com> wrote:
On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry@redhat.com> wrote:
On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails.
On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <rbarry@redhat.com>
ha
scritto:
+1 from me
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
still failing on jenkins, but at least one succeeds locally for me
Succeeds locally for me also.
Dafna, are we sure there's not an infra issue?
I think since its a race ( and we've seen failures on this test in the past, also a race I think ), its probably hard to reproduce locally. Also, we probably need to make sure the same Libvirt version is used. The upstream servers are quite old, it can also be local run ends up being faster and not hitting the same issues ( as we've seen in the past )
Could it be a bug in the ssh client ( paramiko? )
Probably wrong idea, but worth to ask: Any ideas which ssh_timeout is used or how to modify? If 100 tries including a time.sleep(1) takes 100 seconds, either the timeout is not the expected 10 seconds, or the guest refuses the connection.
Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can cause this? ( other than slow servers )
On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 26 Mar 2019 12:31:36 +0100 Dominik Holler <dholler@redhat.com> wrote:
> On Tue, 26 Mar 2019 10:58:22 +0000 > Dafna Ron <dron@redhat.com> wrote: > > > This is still failing randomly > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to
> which action is crashing the guest. >
I was not able to reproduce the failure with the change above. We could merge the change to have better information on the next failure.
> > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < dholler@redhat.com> wrote: > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > Ryan Barry <rbarry@redhat.com> wrote: > > > > > > > It may be virt, but I'm looking... > > > > > > > > I'm very suspicious of this happening immediately after hotplugging a > > > NIC, > > > > especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ > > > > talks about dropping packets. Dominik, did anything else change here? > > > > > > > > > > No, nothing I am aware of. > > > > > > Is there already a pattern in the failed runs detected, or does it fail > > > randomly? > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < amarchuk@redhat.com> > > > > wrote: > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a bug in > > > > > libvirt as suggested. > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < nsoffer@redhat.com> wrote: > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < dron@redhat.com> wrote: > > > > > > Hi, > > > > > > > > > > > > We are failing ovirt-engine master on test > > > 004_basic_sanity.hotplug_cpu > > > > > > looking at the logs, we can see that the for some reason, libvirt > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > CQ first failure was for patch: > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for > > > mdevs, > > > > > use nodisplay to override > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > > > > > related. > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and
understand the
problem is
> > > > > ssh connection to the internal vm > > > > > > > > > > > > Thanks, > > > > > > Dafna > > > > > > > > > > > > > > > > > > error: > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > qemuDomainObjTaint:7521 : > > > > > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > tainted: hook-script > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > virProcessRunInMountNamespace:1159 : internal error: child reported: > > > unable > > > > > to set security context 'system_u:object_r:virt_content_t:s0' on > > > > > > > >
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
> > > > > No such file or directory > > > > > > > > > > > > This should not happen, libvirt is not adding labels to files in > > > > > /rhev/data-center. It is using using its own mount > > > > > > namespace and adding there the devices used by the VM. Since libvirt > > > > > create the devices in its namespace > > > > > > it should not complain about missing paths in /rhev/data-center. > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > guest > > > > > agent is not connected > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > guest > > > > > agent is not connected > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > guest > > > > > agent is not connected > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > Nir > > > > > > > > > > -- > > > > > Anton Marchukov > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Infra mailing list -- infra@ovirt.org > > > > > To unsubscribe send an email to infra-leave@ovirt.org > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > oVirt Code of Conduct: > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > List Archives: > > > > > > > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
> > > > > > > > > > > > > > > > _______________________________________________ > > > Devel mailing list -- devel@ovirt.org > > > To unsubscribe send an email to devel-leave@ovirt.org > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > oVirt Code of Conduct: > > > https://www.ovirt.org/community/about/community-guidelines/ > > > List Archives: > > >
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
> > > >
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7OSQ...
Il giorno mer 27 mar 2019 alle ore 09:54 Dominik Holler <dholler@redhat.com> ha scritto:
On Wed, 27 Mar 2019 10:07:16 +0200 Eyal Edri <eedri@redhat.com> wrote:
On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry@redhat.com> wrote:
On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails.
On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <
rbarry@redhat.com> ha
scritto:
+1 from me
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
still failing on jenkins, but at least one succeeds locally for me
Succeeds locally for me also.
Dafna, are we sure there's not an infra issue?
I think since its a race ( and we've seen failures on this test in the past, also a race I think ), its probably hard to reproduce locally. Also, we probably need to make sure the same Libvirt version is used. The upstream servers are quite old, it can also be local run ends up being faster and not hitting the same issues ( as we've seen in the past )
Could it be a bug in the ssh client ( paramiko? )
Probably wrong idea, but worth to ask: Any ideas which ssh_timeout is used or how to modify?
If 100 tries including a time.sleep(1) takes 100 seconds, either the timeout is not the expected 10 seconds, or the guest refuses the connection.
I'm looking into a similar failure and found this on host1 logs at the time of the ssh failure: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a... Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered disabled state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: device vnet1 entered promiscuous mode Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered forwarding state Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9133] manager: (vnet1): new Tun device (/org/freedesktop/NetworkManager/Devices/44) Mar 27 03:35:20 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9277] device (vnet1): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9297] device (vnet1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-J-vnet1' failed: Illegal target name 'libvirt-J-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-P-vnet1' failed: Illegal target name 'libvirt-P-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: iptables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: iptables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet1 FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet1 FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet1 HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: ip6tables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: ip6tables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet1 FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet1 FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet1 HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-I-vnet1' failed: Illegal target name 'libvirt-I-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-O-vnet1' failed: Illegal target name 'libvirt-O-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet1 libvirt-O-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:28 lago-basic-suite-master-host-1 systemd: Created slice User Slice of root. On the lago log at corresponding time frame: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a... 2019-03-27 07:35:23,801::log_utils.py::end_log_task::670::nose::INFO:: # hotplug_nic: [32mSuccess [0m (in 0:00:05) 2019-03-27 07:35:23,801::log_utils.py::start_log_task::655::nose::INFO:: # hotplug_cpu: [0m [0m Around the same time on host 0: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a... Mar 27 03:33:06 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5243] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5251] device (vnet0): released from master device ovirtmgmt Mar 27 03:33:06 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-2-vm2 terminated. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Started Session c453 of user root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: devmap not registered, can't remove Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c454 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c455 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: devmap not registered, can't remove Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c456 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: device vnet0 entered promiscuous mode Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Mar 27 03:33:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4807] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/49) Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4970] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4997] device (vnet0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0' failed: Illegal target name 'libvirt-I-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet0 libvirt-O-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd-machined: New machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd: Started Virtual Machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 kvm: 1 guest now active Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Started Session c457 of user root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Started Session c458 of user root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Started Session c459 of user root. Mar 27 03:33:36 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: device vnet0 left promiscuous mode Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8430] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8437] device (vnet0): released from master device ovirtmgmt Mar 27 03:34:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:34:16 lago-basic-suite-master-host-0 kvm: 0 guests now active Mar 27 03:34:16 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-3-vm2 terminated. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=5 workers=set([<Worker name=periodic/0 waiting task#=207 at 0x7f8fac0b5310>, <Worker name=periodic/3 waiting task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/1 waiting task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>]) Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:34:20 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=6 workers=set([<Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=3.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>, <Worker name=periodic/3 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c35ba90> discarded task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/0 waiting task#=208 at 0x7f8fac0b5310>, <Worker name=periodic/1 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7f8fac0c61d0> at 0x7f8fac0c6210> timeout=7.5, duration=0.44 at 0x7f8f5c2dd250> task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/5 waiting task#=0 at 0x7f8f8c13dfd0>])
Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can cause this? ( other than slow servers )
On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <
dholler@redhat.com>
wrote: > > On Tue, 26 Mar 2019 12:31:36 +0100 > Dominik Holler <dholler@redhat.com> wrote: > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > Dafna Ron <dron@redhat.com> wrote: > > > > > This is still failing randomly > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to understand > > which action is crashing the guest. > > > > I was not able to reproduce the failure with the change above. > We could merge the change to have better information on the next > failure. > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < dholler@redhat.com> wrote: > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > Ryan Barry <rbarry@redhat.com> wrote: > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > I'm very suspicious of this happening immediately after hotplugging a > > > > NIC, > > > > > especially since the bug attached to https://gerrit.ovirt.org/#/c/98765/ > > > > > talks about dropping packets. Dominik, did anything else change here? > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > Is there already a pattern in the failed runs detected, or does it fail > > > > randomly? > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < amarchuk@redhat.com> > > > > > wrote: > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a bug in > > > > > > libvirt as suggested. > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < nsoffer@redhat.com> wrote: > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < dron@redhat.com> wrote: > > > > > > > Hi, > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > 004_basic_sanity.hotplug_cpu > > > > > > > looking at the logs, we can see that the for some reason, libvirt > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for > > > > mdevs, > > > > > > use nodisplay to override > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be netwrok > > > > > > related. > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and the problem is > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > Thanks, > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > qemuDomainObjTaint:7521 : > > > > > > Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > tainted: hook-script > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > virProcessRunInMountNamespace:1159 : internal error: child reported: > > > > unable > > > > > > to set security context 'system_u:object_r:virt_content_t:s0' on > > > > > > > > > >
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510':
> > > > > > No such file or directory > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to files in > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > namespace and adding there the devices used by the VM. Since libvirt > > > > > > create the devices in its namespace > > > > > > > it should not complain about missing paths in /rhev/data-center. > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > > guest > > > > > > agent is not connected > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > > guest > > > > > > agent is not connected > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU > > > > guest > > > > > > agent is not connected > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > Nir > > > > > > > > > > > > -- > > > > > > Anton Marchukov > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Infra mailing list -- infra@ovirt.org > > > > > > To unsubscribe send an email to infra-leave@ovirt.org > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > oVirt Code of Conduct: > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > List Archives: > > > > > > > > > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
> > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > Devel mailing list -- devel@ovirt.org > > > > To unsubscribe send an email to devel-leave@ovirt.org > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > oVirt Code of Conduct: > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > List Archives: > > > >
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
> > > > > > >
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Infra mailing list -- infra@ovirt.org To unsubscribe send an email to infra-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7OSQ...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCD6OCOWXSHZYM...
-- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig>
Hi I had a failure on my laptop when running in mock on hotplug cpu test. (when testing a change, got the same error on the jenkins) Dominik requested me to make a video of the vm0 dmesg command. https://drive.google.com/file/d/1Kr6r4SMhnVsWBvWD6E6JIZI4ddivo2pv/view?usp=s... Regards, Galit On Wed, Mar 27, 2019 at 2:39 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mer 27 mar 2019 alle ore 09:54 Dominik Holler < dholler@redhat.com> ha scritto:
On Wed, 27 Mar 2019 10:07:16 +0200 Eyal Edri <eedri@redhat.com> wrote:
On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry@redhat.com> wrote:
On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails.
On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <
rbarry@redhat.com> ha
scritto:
> +1 from me >
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
still failing on jenkins, but at least one succeeds locally for me
Succeeds locally for me also.
Dafna, are we sure there's not an infra issue?
I think since its a race ( and we've seen failures on this test in the past, also a race I think ), its probably hard to reproduce locally. Also, we probably need to make sure the same Libvirt version is used. The upstream servers are quite old, it can also be local run ends up being faster and not hitting the same issues ( as we've seen in the past )
Could it be a bug in the ssh client ( paramiko? )
Probably wrong idea, but worth to ask: Any ideas which ssh_timeout is used or how to modify?
If 100 tries including a time.sleep(1) takes 100 seconds, either the timeout is not the expected 10 seconds, or the guest refuses the connection.
I'm looking into a similar failure and found this on host1 logs at the time of the ssh failure: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered disabled state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: device vnet1 entered promiscuous mode Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered forwarding state Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9133] manager: (vnet1): new Tun device (/org/freedesktop/NetworkManager/Devices/44) Mar 27 03:35:20 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9277] device (vnet1): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9297] device (vnet1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-J-vnet1' failed: Illegal target name 'libvirt-J-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-P-vnet1' failed: Illegal target name 'libvirt-P-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: iptables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: iptables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet1 FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet1 FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet1 HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: ip6tables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: ip6tables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet1 FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet1 FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet1 HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-I-vnet1' failed: Illegal target name 'libvirt-I-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-O-vnet1' failed: Illegal target name 'libvirt-O-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet1 libvirt-O-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:28 lago-basic-suite-master-host-1 systemd: Created slice User Slice of root.
On the lago log at corresponding time frame: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
2019-03-27 07:35:23,801::log_utils.py::end_log_task::670::nose::INFO:: # hotplug_nic: [32mSuccess [0m (in 0:00:05) 2019-03-27 07:35:23,801::log_utils.py::start_log_task::655::nose::INFO:: # hotplug_cpu: [0m [0m
Around the same time on host 0: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
Mar 27 03:33:06 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5243] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5251] device (vnet0): released from master device ovirtmgmt Mar 27 03:33:06 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-2-vm2 terminated. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Started Session c453 of user root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: devmap not registered, can't remove Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c454 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c455 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: devmap not registered, can't remove Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c456 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: device vnet0 entered promiscuous mode Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Mar 27 03:33:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4807] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/49) Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4970] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4997] device (vnet0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0' failed: Illegal target name 'libvirt-I-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet0 libvirt-O-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd-machined: New machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd: Started Virtual Machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 kvm: 1 guest now active Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Started Session c457 of user root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Started Session c458 of user root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Started Session c459 of user root. Mar 27 03:33:36 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: device vnet0 left promiscuous mode Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8430] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8437] device (vnet0): released from master device ovirtmgmt Mar 27 03:34:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:34:16 lago-basic-suite-master-host-0 kvm: 0 guests now active Mar 27 03:34:16 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-3-vm2 terminated. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=5 workers=set([<Worker name=periodic/0 waiting task#=207 at 0x7f8fac0b5310>, <Worker name=periodic/3 waiting task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/1 waiting task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>]) Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:34:20 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=6 workers=set([<Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=3.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>, <Worker name=periodic/3 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c35ba90> discarded task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/0 waiting task#=208 at 0x7f8fac0b5310>, <Worker name=periodic/1 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7f8fac0c61d0> at 0x7f8fac0c6210> timeout=7.5, duration=0.44 at 0x7f8f5c2dd250> task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/5 waiting task#=0 at 0x7f8f8c13dfd0>])
Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can cause this? ( other than slow servers )
> > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <
dholler@redhat.com>
> wrote: > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > Dominik Holler <dholler@redhat.com> wrote: > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > Dafna Ron <dron@redhat.com> wrote: > > > > > > > This is still failing randomly > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to understand > > > which action is crashing the guest. > > > > > > > I was not able to reproduce the failure with the change above. > > We could merge the change to have better information on the next > > failure. > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < dholler@redhat.com> > wrote: > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > Ryan Barry <rbarry@redhat.com> wrote: > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > I'm very suspicious of this happening immediately after > hotplugging a > > > > > NIC, > > > > > > especially since the bug attached to > https://gerrit.ovirt.org/#/c/98765/ > > > > > > talks about dropping packets. Dominik, did anything else change > here? > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > Is there already a pattern in the failed runs detected, or does it > fail > > > > > randomly? > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > amarchuk@redhat.com> > > > > > > wrote: > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a > bug in > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < nsoffer@redhat.com> > wrote: > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < dron@redhat.com> > wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > looking at the logs, we can see that the for some reason, > libvirt > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > display="on" for > > > > > mdevs, > > > > > > > use nodisplay to override > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be > netwrok > > > > > > > related. > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and the > problem is > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > qemuDomainObjTaint:7521 : > > > > > > > Domain id=3 name='vm0' > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > virProcessRunInMountNamespace:1159 : internal error: child > reported: > > > > > unable > > > > > > > to set security context 'system_u:object_r:virt_content_t:s0' > on > > > > > > > > > > > > >
> > > > > > > No such file or directory > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > files in > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > namespace and adding there the devices used by
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': the VM.
Since
> libvirt > > > > > > > create the devices in its namespace > > > > > > > > it should not complain about missing paths in > /rhev/data-center. > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > -- > > > > > > > Anton Marchukov > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Infra mailing list -- infra@ovirt.org > > > > > > > To unsubscribe send an email to infra-leave@ovirt.org > > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > > oVirt Code of Conduct: > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > List Archives: > > > > > > > > > > > > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
> > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Devel mailing list -- devel@ovirt.org > > > > > To unsubscribe send an email to devel-leave@ovirt.org > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > oVirt Code of Conduct: > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > List Archives: > > > > > >
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
> > > > > > > > > > > > > -- > > Ryan Barry > > Associate Manager - RHV Virt/SLA > > rbarry@redhat.com M: +16518159306 IM: rbarry > _______________________________________________ > Infra mailing list -- infra@ovirt.org > To unsubscribe send an email to infra-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
>
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7OSQ...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCD6OCOWXSHZYM...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig>
-- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>
On Tue, 2 Apr 2019 12:24:45 +0300 Galit Rosenthal <grosenth@redhat.com> wrote:
Hi
I had a failure on my laptop when running in mock on hotplug cpu test. (when testing a change, got the same error on the jenkins) Dominik requested me to make a video of the vm0 dmesg command.
https://drive.google.com/file/d/1Kr6r4SMhnVsWBvWD6E6JIZI4ddivo2pv/view?usp=s...
Galit, thank you very much for the video! The video shows that the guest is in serious trouble and a dropbear process is killed by the oom killer. Gallit currently checks if it is possible to give the guest VM more memory.
Regards, Galit
On Wed, Mar 27, 2019 at 2:39 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mer 27 mar 2019 alle ore 09:54 Dominik Holler < dholler@redhat.com> ha scritto:
On Wed, 27 Mar 2019 10:07:16 +0200 Eyal Edri <eedri@redhat.com> wrote:
On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry@redhat.com> wrote:
On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails.
On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
> Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <
rbarry@redhat.com> ha
> scritto: > > > +1 from me > > > > Merged. I have 2 patches constantly failing on it, rebased them, you can > follow on: > https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862 >
still failing on jenkins, but at least one succeeds locally for me
Succeeds locally for me also.
Dafna, are we sure there's not an infra issue?
I think since its a race ( and we've seen failures on this test in the past, also a race I think ), its probably hard to reproduce locally. Also, we probably need to make sure the same Libvirt version is used. The upstream servers are quite old, it can also be local run ends up being faster and not hitting the same issues ( as we've seen in the past )
Could it be a bug in the ssh client ( paramiko? )
Probably wrong idea, but worth to ask: Any ideas which ssh_timeout is used or how to modify?
If 100 tries including a time.sleep(1) takes 100 seconds, either the timeout is not the expected 10 seconds, or the guest refuses the connection.
I'm looking into a similar failure and found this on host1 logs at the time of the ssh failure: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered disabled state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: device vnet1 entered promiscuous mode Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered forwarding state Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9133] manager: (vnet1): new Tun device (/org/freedesktop/NetworkManager/Devices/44) Mar 27 03:35:20 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9277] device (vnet1): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9297] device (vnet1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-J-vnet1' failed: Illegal target name 'libvirt-J-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-P-vnet1' failed: Illegal target name 'libvirt-P-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: iptables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: iptables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet1 FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet1 FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet1 HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: ip6tables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: ip6tables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet1 FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet1 FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet1 HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-I-vnet1' failed: Illegal target name 'libvirt-I-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-O-vnet1' failed: Illegal target name 'libvirt-O-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet1 libvirt-O-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:28 lago-basic-suite-master-host-1 systemd: Created slice User Slice of root.
On the lago log at corresponding time frame: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
2019-03-27 07:35:23,801::log_utils.py::end_log_task::670::nose::INFO:: # hotplug_nic: [32mSuccess [0m (in 0:00:05) 2019-03-27 07:35:23,801::log_utils.py::start_log_task::655::nose::INFO:: # hotplug_cpu: [0m [0m
Around the same time on host 0: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
Mar 27 03:33:06 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5243] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5251] device (vnet0): released from master device ovirtmgmt Mar 27 03:33:06 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-2-vm2 terminated. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Started Session c453 of user root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: devmap not registered, can't remove Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c454 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c455 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: devmap not registered, can't remove Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c456 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: device vnet0 entered promiscuous mode Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Mar 27 03:33:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4807] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/49) Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4970] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4997] device (vnet0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0' failed: Illegal target name 'libvirt-I-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet0 libvirt-O-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd-machined: New machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd: Started Virtual Machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 kvm: 1 guest now active Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Started Session c457 of user root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Started Session c458 of user root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Started Session c459 of user root. Mar 27 03:33:36 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: device vnet0 left promiscuous mode Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8430] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8437] device (vnet0): released from master device ovirtmgmt Mar 27 03:34:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:34:16 lago-basic-suite-master-host-0 kvm: 0 guests now active Mar 27 03:34:16 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-3-vm2 terminated. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=5 workers=set([<Worker name=periodic/0 waiting task#=207 at 0x7f8fac0b5310>, <Worker name=periodic/3 waiting task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/1 waiting task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>]) Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:34:20 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=6 workers=set([<Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=3.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>, <Worker name=periodic/3 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c35ba90> discarded task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/0 waiting task#=208 at 0x7f8fac0b5310>, <Worker name=periodic/1 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7f8fac0c61d0> at 0x7f8fac0c6210> timeout=7.5, duration=0.44 at 0x7f8f5c2dd250> task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/5 waiting task#=0 at 0x7f8f8c13dfd0>])
Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can cause this? ( other than slow servers )
> > > > > > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <
dholler@redhat.com>
> > wrote: > > > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > > Dominik Holler <dholler@redhat.com> wrote: > > > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > > Dafna Ron <dron@redhat.com> wrote: > > > > > > > > > This is still failing randomly > > > > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to understand > > > > which action is crashing the guest. > > > > > > > > > > I was not able to reproduce the failure with the change above. > > > We could merge the change to have better information on the next > > > failure. > > > > > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < dholler@redhat.com> > > wrote: > > > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > > Ryan Barry <rbarry@redhat.com> wrote: > > > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > > > I'm very suspicious of this happening immediately after > > hotplugging a > > > > > > NIC, > > > > > > > especially since the bug attached to > > https://gerrit.ovirt.org/#/c/98765/ > > > > > > > talks about dropping packets. Dominik, did anything else change > > here? > > > > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > > > Is there already a pattern in the failed runs detected, or does it > > fail > > > > > > randomly? > > > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > > amarchuk@redhat.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a > > bug in > > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < nsoffer@redhat.com> > > wrote: > > > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < dron@redhat.com> > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > > looking at the logs, we can see that the for some reason, > > libvirt > > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > > display="on" for > > > > > > mdevs, > > > > > > > > use nodisplay to override > > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be > > netwrok > > > > > > > > related. > > > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > > > > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and the > > problem is > > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > > qemuDomainObjTaint:7521 : > > > > > > > > Domain id=3 name='vm0' > > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > > virProcessRunInMountNamespace:1159 : internal error: child > > reported: > > > > > > unable > > > > > > > > to set security context 'system_u:object_r:virt_content_t:s0' > > on > > > > > > > > > > > > > > > >
> > > > > > > > No such file or directory > > > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > > files in > > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > > namespace and adding there the devices used by
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': the VM.
Since
> > libvirt > > > > > > > > create the devices in its namespace > > > > > > > > > it should not complain about missing paths in > > /rhev/data-center. > > > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > > QEMU > > > > > > guest > > > > > > > > agent is not connected > > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > > QEMU > > > > > > guest > > > > > > > > agent is not connected > > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > > QEMU > > > > > > guest > > > > > > > > agent is not connected > > > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > > > -- > > > > > > > > Anton Marchukov > > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Infra mailing list -- infra@ovirt.org > > > > > > > > To unsubscribe send an email to infra-leave@ovirt.org > > > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > > > oVirt Code of Conduct: > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > > List Archives: > > > > > > > > > > > > > > > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
> > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Devel mailing list -- devel@ovirt.org > > > > > > To unsubscribe send an email to devel-leave@ovirt.org > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > oVirt Code of Conduct: > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > List Archives: > > > > > > > >
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
> > > > > > > > > > > > > > > > > > > -- > > > > Ryan Barry > > > > Associate Manager - RHV Virt/SLA > > > > rbarry@redhat.com M: +16518159306 IM: rbarry > > _______________________________________________ > > Infra mailing list -- infra@ovirt.org > > To unsubscribe send an email to infra-leave@ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
> > > >
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7OSQ...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCD6OCOWXSHZYM...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig>
participants (8)
-
Anton Marchukov -
Dafna Ron -
Dominik Holler -
Eyal Edri -
Galit Rosenthal -
Nir Soffer -
Ryan Barry -
Sandro Bonazzola