Hi I had a failure on my laptop when running in mock on hotplug cpu test. (when testing a change, got the same error on the jenkins) Dominik requested me to make a video of the vm0 dmesg command. https://drive.google.com/file/d/1Kr6r4SMhnVsWBvWD6E6JIZI4ddivo2pv/view?usp=s... Regards, Galit On Wed, Mar 27, 2019 at 2:39 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mer 27 mar 2019 alle ore 09:54 Dominik Holler < dholler@redhat.com> ha scritto:
On Wed, 27 Mar 2019 10:07:16 +0200 Eyal Edri <eedri@redhat.com> wrote:
On Wed, Mar 27, 2019 at 3:06 AM Ryan Barry <rbarry@redhat.com> wrote:
On Tue, Mar 26, 2019 at 4:07 PM Dominik Holler <dholler@redhat.com> wrote:
I added in https://gerrit.ovirt.org/#/c/98925/ a ping directly before the ssh. The ping succeeds, but the ssh fails.
On Tue, 26 Mar 2019 17:07:45 +0100 Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 26 mar 2019 alle ore 16:48 Ryan Barry <
rbarry@redhat.com> ha
scritto:
> +1 from me >
Merged. I have 2 patches constantly failing on it, rebased them, you can follow on: https://gerrit.ovirt.org/#/c/98863/ and https://gerrit.ovirt.org/98862
still failing on jenkins, but at least one succeeds locally for me
Succeeds locally for me also.
Dafna, are we sure there's not an infra issue?
I think since its a race ( and we've seen failures on this test in the past, also a race I think ), its probably hard to reproduce locally. Also, we probably need to make sure the same Libvirt version is used. The upstream servers are quite old, it can also be local run ends up being faster and not hitting the same issues ( as we've seen in the past )
Could it be a bug in the ssh client ( paramiko? )
Probably wrong idea, but worth to ask: Any ideas which ssh_timeout is used or how to modify?
If 100 tries including a time.sleep(1) takes 100 seconds, either the timeout is not the expected 10 seconds, or the guest refuses the connection.
I'm looking into a similar failure and found this on host1 logs at the time of the ssh failure: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered disabled state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: device vnet1 entered promiscuous mode Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered blocking state Mar 27 03:35:20 lago-basic-suite-master-host-1 kernel: on65322a61b5f14: port 2(vnet1) entered forwarding state Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9133] manager: (vnet1): new Tun device (/org/freedesktop/NetworkManager/Devices/44) Mar 27 03:35:20 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9277] device (vnet1): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 NetworkManager[2667]: <info> [1553672120.9297] device (vnet1): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-J-vnet1' failed: Illegal target name 'libvirt-J-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-P-vnet1' failed: Illegal target name 'libvirt-P-vnet1'. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:20 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet1' failed: Chain 'libvirt-J-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-mac' failed: Chain 'J-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet1-arp-mac' failed: Chain 'J-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: iptables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: iptables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: iptables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet1 FO-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet1 FI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet1 HI-vnet1' failed: iptables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet1 -g FO-vnet1' failed: ip6tables v1.4.21: goto 'FO-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet1 -g FI-vnet1' failed: ip6tables v1.4.21: goto 'FI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet1 -g HI-vnet1' failed: ip6tables v1.4.21: goto 'HI-vnet1' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet1 FO-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet1 FI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet1 HI-vnet1' failed: ip6tables: No chain/target/match by that name. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet1 -j libvirt-I-vnet1' failed: Illegal target name 'libvirt-I-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet1 -j libvirt-O-vnet1' failed: Illegal target name 'libvirt-O-vnet1'. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet1' failed: Chain 'libvirt-I-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet1' failed: Chain 'libvirt-O-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet1 libvirt-O-vnet1' failed: Chain 'libvirt-P-vnet1' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-mac' failed: Chain 'I-vnet1-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 firewalld[2643]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet1-arp-mac' failed: Chain 'I-vnet1-arp-mac' doesn't exist. Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:21 lago-basic-suite-master-host-1 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:35:28 lago-basic-suite-master-host-1 systemd: Created slice User Slice of root.
On the lago log at corresponding time frame: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
2019-03-27 07:35:23,801::log_utils.py::end_log_task::670::nose::INFO:: # hotplug_nic: [32mSuccess [0m (in 0:00:05) 2019-03-27 07:35:23,801::log_utils.py::start_log_task::655::nose::INFO:: # hotplug_cpu: [0m [0m
Around the same time on host 0: https://jenkins.ovirt.org/job/ovirt-system-tests_standard-check-patch/3905/a...
Mar 27 03:33:06 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5243] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:33:06 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671986.5251] device (vnet0): released from master device ovirtmgmt Mar 27 03:33:06 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:06 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-2-vm2 terminated. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:06 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Started Session c453 of user root. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: devmap not registered, can't remove Mar 27 03:33:07 lago-basic-suite-master-host-0 multipathd: dm-13: remove map (uevent) Mar 27 03:33:07 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:07 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c454 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c455 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: devmap not registered, can't remove Mar 27 03:33:08 lago-basic-suite-master-host-0 multipathd: dm-12: remove map (uevent) Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Started Session c456 of user root. Mar 27 03:33:08 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:33:09 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: device vnet0 entered promiscuous mode Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered blocking state Mar 27 03:33:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered forwarding state Mar 27 03:33:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4807] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/49) Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4970] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553671996.4997] device (vnet0): state change: unavailable -> disconnected (reason 'none', sys-iface-state: 'external') Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-mac' failed: Chain 'J-vnet0-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X J-vnet0-arp-mac' failed: Chain 'J-vnet0-arp-mac' doesn't exist. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FP-vnet0 FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:16 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E FJ-vnet0 FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -E HJ-vnet0 HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FP-vnet0 FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E FJ-vnet0 FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -E HJ-vnet0 HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-I-vnet0' failed: Illegal target name 'libvirt-I-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-I-vnet0' failed: Chain 'libvirt-I-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -E libvirt-P-vnet0 libvirt-O-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-mac' failed: Chain 'I-vnet0-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X I-vnet0-arp-mac' failed: Chain 'I-vnet0-arp-mac' doesn't exist. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd-machined: New machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 systemd: Started Virtual Machine qemu-3-vm2. Mar 27 03:33:18 lago-basic-suite-master-host-0 kvm: 1 guest now active Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Started Session c457 of user root. Mar 27 03:33:28 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Started Session c458 of user root. Mar 27 03:33:29 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Created slice User Slice of root. Mar 27 03:33:35 lago-basic-suite-master-host-0 systemd: Started Session c459 of user root. Mar 27 03:33:36 lago-basic-suite-master-host-0 systemd: Removed slice User Slice of root. Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: device vnet0 left promiscuous mode Mar 27 03:34:16 lago-basic-suite-master-host-0 kernel: ovirtmgmt: port 2(vnet0) entered disabled state Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8430] device (vnet0): state change: disconnected -> unmanaged (reason 'unmanaged', sys-iface-state: 'removed') Mar 27 03:34:16 lago-basic-suite-master-host-0 NetworkManager[2671]: <info> [1553672056.8437] device (vnet0): released from master device ovirtmgmt Mar 27 03:34:16 lago-basic-suite-master-host-0 lldpad: recvfrom(Event interface): No buffer space available Mar 27 03:34:16 lago-basic-suite-master-host-0 kvm: 0 guests now active Mar 27 03:34:16 lago-basic-suite-master-host-0 systemd-machined: Machine qemu-3-vm2 terminated. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: iptables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: iptables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: iptables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FP-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HJ-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FP-vnet0' failed: ip6tables v1.4.21: goto 'FP-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FJ-vnet0' failed: ip6tables v1.4.21: goto 'FJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HJ-vnet0' failed: ip6tables v1.4.21: goto 'HJ-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FP-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HJ-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D PREROUTING -i vnet0 -j libvirt-J-vnet0' failed: Illegal target name 'libvirt-J-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-P-vnet0' failed: Illegal target name 'libvirt-P-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-J-vnet0' failed: Chain 'libvirt-J-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-P-vnet0' failed: Chain 'libvirt-P-vnet0' doesn't exist. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: iptables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: iptables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: iptables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `iptables -h' or 'iptables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FO-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X FI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -F HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -w -X HI-vnet0' failed: iptables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-is-bridged --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-out -m physdev --physdev-out vnet0 -g FO-vnet0' failed: ip6tables v1.4.21: goto 'FO-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in -m physdev --physdev-in vnet0 -g FI-vnet0' failed: ip6tables v1.4.21: goto 'FI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-host-in -m physdev --physdev-in vnet0 -g HI-vnet0' failed: ip6tables v1.4.21: goto 'HI-vnet0' is not a chain#012#012Try `ip6tables -h' or 'ip6tables --help' for more information. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -D libvirt-in-post -m physdev --physdev-in vnet0 -j ACCEPT' failed: ip6tables: Bad rule (does a matching rule exist in that chain?). Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FO-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X FI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -F HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ip6tables -w2 -w -X HI-vnet0' failed: ip6tables: No chain/target/match by that name. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -D POSTROUTING -o vnet0 -j libvirt-O-vnet0' failed: Illegal target name 'libvirt-O-vnet0'. Mar 27 03:34:17 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -L libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -F libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 firewalld[2651]: WARNING: COMMAND_FAILED: '/usr/sbin/ebtables --concurrent -t nat -X libvirt-O-vnet0' failed: Chain 'libvirt-O-vnet0' doesn't exist. Mar 27 03:34:18 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=5 workers=set([<Worker name=periodic/0 waiting task#=207 at 0x7f8fac0b5310>, <Worker name=periodic/3 waiting task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/1 waiting task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>]) Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.ovirt-guest-agent.0 already removed Mar 27 03:34:19 lago-basic-suite-master-host-0 vdsm[22587]: WARN File: /var/lib/libvirt/qemu/channels/fc3c11ae-4b16-4bab-8b5e-aa839b2a42f6.org.qemu.guest_agent.0 already removed Mar 27 03:34:20 lago-basic-suite-master-host-0 vdsm[22587]: WARN executor state: count=6 workers=set([<Worker name=periodic/2 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=3.01 at 0x7f8f5c46f650> discarded task#=214 at 0x7f8fac0b5810>, <Worker name=periodic/3 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7f8fac0c6250> at 0x7f8fac0c6290> timeout=1.0, duration=1.01 at 0x7f8f5c35ba90> discarded task#=194 at 0x7f8fac0b5b10>, <Worker name=periodic/0 waiting task#=208 at 0x7f8fac0b5310>, <Worker name=periodic/1 running <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7f8fac0c61d0> at 0x7f8fac0c6210> timeout=7.5, duration=0.44 at 0x7f8f5c2dd250> task#=200 at 0x7f8fac0b5550>, <Worker name=periodic/4 waiting task#=0 at 0x7f8f5c412190>, <Worker name=periodic/5 waiting task#=0 at 0x7f8f8c13dfd0>])
Barak,Gal,Galit, Evgheni - any thoughts on something on infra that can cause this? ( other than slow servers )
> > On Tue, Mar 26, 2019 at 11:13 AM Dominik Holler <
dholler@redhat.com>
> wrote: > > > > On Tue, 26 Mar 2019 12:31:36 +0100 > > Dominik Holler <dholler@redhat.com> wrote: > > > > > On Tue, 26 Mar 2019 10:58:22 +0000 > > > Dafna Ron <dron@redhat.com> wrote: > > > > > > > This is still failing randomly > > > > > > > > > > I created https://gerrit.ovirt.org/#/c/98906/ to help to understand > > > which action is crashing the guest. > > > > > > > I was not able to reproduce the failure with the change above. > > We could merge the change to have better information on the next > > failure. > > > > > > > > > > > > On Tue, Mar 26, 2019 at 8:15 AM Dominik Holler < dholler@redhat.com> > wrote: > > > > > > > > > On Mon, 25 Mar 2019 17:30:53 -0400 > > > > > Ryan Barry <rbarry@redhat.com> wrote: > > > > > > > > > > > It may be virt, but I'm looking... > > > > > > > > > > > > I'm very suspicious of this happening immediately after > hotplugging a > > > > > NIC, > > > > > > especially since the bug attached to > https://gerrit.ovirt.org/#/c/98765/ > > > > > > talks about dropping packets. Dominik, did anything else change > here? > > > > > > > > > > > > > > > > No, nothing I am aware of. > > > > > > > > > > Is there already a pattern in the failed runs detected, or does it > fail > > > > > randomly? > > > > > > > > > > > On Mon, Mar 25, 2019 at 12:42 PM Anton Marchukov < > amarchuk@redhat.com> > > > > > > wrote: > > > > > > > > > > > > > Which team is it? Is it Virt? Just checking who should open a > bug in > > > > > > > libvirt as suggested. > > > > > > > > > > > > > > > On 22 Mar 2019, at 20:52, Nir Soffer < nsoffer@redhat.com> > wrote: > > > > > > > > > > > > > > > > On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron < dron@redhat.com> > wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > We are failing ovirt-engine master on test > > > > > 004_basic_sanity.hotplug_cpu > > > > > > > > looking at the logs, we can see that the for some reason, > libvirt > > > > > > > reports a vm as none responsive which fails the test. > > > > > > > > > > > > > > > > CQ first failure was for patch: > > > > > > > > https://gerrit.ovirt.org/#/c/98553/ - core: Add > display="on" for > > > > > mdevs, > > > > > > > use nodisplay to override > > > > > > > > But I do not think this is the cause of failure. > > > > > > > > > > > > > > > > Adding Marcin, Milan and Dan as well as I think it may be > netwrok > > > > > > > related. > > > > > > > > > > > > > > > > You can see the libvirt log here: > > > > > > > > > > > > > > > > > > > > >
https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifac...
> > > > > > > > > > > > > > > > you can see the full logs here: > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13516/artifact...
> > > > > > > > > > > > > > > > Evgheni and I confirmed this is not an infra issue and the > problem is > > > > > > > ssh connection to the internal vm > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dafna > > > > > > > > > > > > > > > > > > > > > > > > error: > > > > > > > > 2019-03-22 15:08:22.658+0000: 22068: warning : > > > > > qemuDomainObjTaint:7521 : > > > > > > > Domain id=3 name='vm0' > uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is > > > > > > > tainted: hook-script > > > > > > > > > > > > > > > > Why our vm is tainted? > > > > > > > > > > > > > > > > 2019-03-22 15:08:22.693+0000: 22068: error : > > > > > > > virProcessRunInMountNamespace:1159 : internal error: child > reported: > > > > > unable > > > > > > > to set security context 'system_u:object_r:virt_content_t:s0' > on > > > > > > > > > > > > >
> > > > > > > No such file or directory > > > > > > > > > > > > > > > > This should not happen, libvirt is not adding labels to > files in > > > > > > > /rhev/data-center. It is using using its own mount > > > > > > > > namespace and adding there the devices used by
'/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': the VM.
Since
> libvirt > > > > > > > create the devices in its namespace > > > > > > > > it should not complain about missing paths in > /rhev/data-center. > > > > > > > > > > > > > > > > I think we should file a libvirt bug for this. > > > > > > > > > > > > > > > > 2019-03-22 15:08:28.168+0000: 22070: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > 2019-03-22 15:08:58.193+0000: 22070: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > 2019-03-22 15:13:58.179+0000: 22071: error : > > > > > > > qemuDomainAgentAvailable:9133 : Guest agent is not responding: > QEMU > > > > > guest > > > > > > > agent is not connected > > > > > > > > > > > > > > > > Do we have guest agent in the test VMs? > > > > > > > > > > > > > > > > Nir > > > > > > > > > > > > > > -- > > > > > > > Anton Marchukov > > > > > > > Associate Manager - RHV DevOps - Red Hat > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > Infra mailing list -- infra@ovirt.org > > > > > > > To unsubscribe send an email to infra-leave@ovirt.org > > > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > > > oVirt Code of Conduct: > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > List Archives: > > > > > > > > > > > > >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/B44Q3AZA7JUPMW...
> > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Devel mailing list -- devel@ovirt.org > > > > > To unsubscribe send an email to devel-leave@ovirt.org > > > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > > > oVirt Code of Conduct: > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > List Archives: > > > > > >
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7XYIPXZLPHRRI5...
> > > > > > > > > > > > > -- > > Ryan Barry > > Associate Manager - RHV Virt/SLA > > rbarry@redhat.com M: +16518159306 IM: rbarry > _______________________________________________ > Infra mailing list -- infra@ovirt.org > To unsubscribe send an email to infra-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: >
https://lists.ovirt.org/archives/list/infra@ovirt.org/message/K7WPLV4WFLNURG...
>
--
Ryan Barry
Associate Manager - RHV Virt/SLA
rbarry@redhat.com M: +16518159306 IM: rbarry _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J5LSZJVOJV7OSQ...
_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NCD6OCOWXSHZYM...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig>
-- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>