Re: [ovirt-users] Host loses all network configuration on update to oVirt 3.5.4

Hi all, I just updated my existing oVirt 3.5.3 installation (iSCSI hosted-engine on CentOS 7.1). The engine update went fine. Updating the hosts succeeds until the first reboot. After a reboot the host does not come up again. It is missing all network configuration. All network cfgs in /etc/sysconfig/network-scripts are missing except ifcfg-lo. The host boots up without working networking. Using IPMI and config backups, I was able to restore the lost network configs. Once these are restored and the host is rebooted again all seems to be back to good. This has now happend to 2 updated hosts (this installation has a total of 4 hosts, so 2 more to debug/try). I'm happy to assist in furter debugging. Before updating the second host, I gathered some information. All these hosts have 3 physical nics. One is used for the ovirtmgmt bridge and the other 2 are used for iSCSI storage vlans. ifcfgs before update: /etc/sysconfig/network-scripts/ifcfg-em1 # Generated by VDSM version 4.16.20-0.el7.centos DEVICE=em1 HWADDR=d0:67:e5:f0:e5:c6 BRIDGE=ovirtmgmt ONBOOT=yes NM_CONTROLLED=no

DEVICE=lo IPADDR=127.0.0.1 NETMASK=255.0.0.0 NETWORK=127.0.0.0 # If you're having problems with gated making 127.0.0.0/8 a martian, # you can change this to something else (255.255.255.255, for example) BROADCAST=127.255.255.255 ONBOOT=yes NAME=loopback /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt # Generated by VDSM version 4.16.20-0.el7.centos DEVICE=ovirtmgmt TYPE=Bridge DELAY=0 STP=off ONBOOT=yes IPADDR=1.2.3.16 NETMASK=255.255.255.0 GATEWAY=1.2.3.11 BOOTPROTO=none DEFROUTE=yes NM_CONTROLLED=no HOTPLUG=no /etc/sysconfig/network-scripts/ifcfg-p4p1 # Generated by VDSM version 4.16.20-0.el7.centos DEVICE=p4p1 HWADDR=68:05:ca:01:bc:0c ONBOOT=no IPADDR=4.5.7.102 NETMASK=255.255.255.0 BOOTPROTO=none MTU=9000 DEFROUTE=no NM_CONTROLLED=no /etc/sysconfig/network-scripts/ifcfg-p3p1 # Generated by VDSM version 4.16.20-0.el7.centos DEVICE=p3p1 HWADDR=68:05:ca:18:86:45 ONBOOT=no IPADDR=4.5.6.102 NETMASK=255.255.255.0 BOOTPROTO=none MTU=9000 DEFROUTE=no NM_CONTROLLED=no /etc/sysconfig/network-scripts/ifcfg-lo ip link before update: 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN mode DEFAULT link/ether 46:50:22:7a:f3:9d brd ff:ff:ff:ff:ff:ff 3: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovirtmgmt state UP mode DEFAULT qlen 1000 link/ether d0:67:e5:f0:e5:c6 brd ff:ff:ff:ff:ff:ff 4: p3p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 68:05:ca:18:86:45 brd ff:ff:ff:ff:ff:ff 5: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP mode DEFAULT qlen 1000 link/ether 68:05:ca:01:bc:0c brd ff:ff:ff:ff:ff:ff 7: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT link/ether d0:67:e5:f0:e5:c6 brd ff:ff:ff:ff:ff:ff 8: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT link/ether ce:0f:16:49:a7:da brd ff:ff:ff:ff:ff:ff vdsm files before update: /var/lib/vdsm /var/lib/vdsm/bonding-defaults.json /var/lib/vdsm/netconfback /var/lib/vdsm/netconfback/ifcfg-ovirtmgmt /var/lib/vdsm/netconfback/ifcfg-em1 /var/lib/vdsm/netconfback/route-ovirtmgmt /var/lib/vdsm/netconfback/rule-ovirtmgmt /var/lib/vdsm/netconfback/ifcfg-p4p1 /var/lib/vdsm/netconfback/ifcfg-p3p1 /var/lib/vdsm/persistence /var/lib/vdsm/persistence/netconf /var/lib/vdsm/persistence/netconf.1416666697752319079 /var/lib/vdsm/persistence/netconf.1416666697752319079/nets /var/lib/vdsm/persistence/netconf.1416666697752319079/nets/san1 /var/lib/vdsm/persistence/netconf.1416666697752319079/nets/san2 /var/lib/vdsm/persistence/netconf.1416666697752319079/nets/ovirtmgmt /var/lib/vdsm/upgrade /var/lib/vdsm/upgrade/upgrade-unified-persistence /var/lib/vdsm/transient File in /var/lib/vdsm/netconfback each only contained a comment: # original file did not exist

/var/lib/vdsm/persistence/netconf.1416666697752319079/nets/ovirtmgmt {"nic": "em1", "netmask": "255.255.255.0", "bootproto": "none", "ipaddr": "1.2.3.16", "gateway": "1.2.3.11"} /var/lib/vdsm/persistence/netconf.1416666697752319079/nets/san1 {"nic": "p3p1", "netmask": "255.255.255.0", "ipaddr": "4.5.6.102", "bridged": "false", "mtu": "9000"} /var/lib/vdsm/persistence/netconf.1416666697752319079/nets/san2 {"nic": "p4p1", "netmask": "255.255.255.0", "ipaddr": "4.5.7.102", "bridged": "false", "mtu": "9000"} After update and reboot, no ifcfg scripts are left. Only interface lo is up. Syslog doess not seem to contain anything suspicious before refore reboot.

Log excerpts from bootup: Sep 3 17:27:23 vhm-prd-02 network: Bringing up loopback interface: [ OK ] Sep 3 17:27:23 vhm-prd-02 systemd-ovirt-ha-agent: Starting ovirt-ha-agent: [ OK ] Sep 3 17:27:23 vhm-prd-02 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Sep 3 17:27:23 vhm-prd-02 kernel: IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready Sep 3 17:27:23 vhm-prd-02 kernel: device em1 entered promiscuous mode Sep 3 17:27:23 vhm-prd-02 network: Bringing up interface em1: [ OK ] Sep 3 17:27:23 vhm-prd-02 kernel: IPv6: ADDRCONF(NETDEV_UP): ovirtmgmt: link is not ready Sep 3 17:27:25 vhm-prd-02 avahi-daemon[778]: Joining mDNS multicast group on interface ovirtmgmt.IPv4 with address 1.2.3.16. Sep 3 17:27:25 vhm-prd-02 avahi-daemon[778]: New relevant interface ovirtmgmt.IPv4 for mDNS. Sep 3 17:27:25 vhm-prd-02 avahi-daemon[778]: Registering new address record for 1.2.3.16 on ovirtmgmt.IPv4. Sep 3 17:27:26 vhm-prd-02 kernel: tg3 0000:03:00.0 em1: Link is up at 1000 Mbps, full duplex Sep 3 17:27:26 vhm-prd-02 kernel: tg3 0000:03:00.0 em1: Flow control is off for TX and off for RX Sep 3 17:27:26 vhm-prd-02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready Sep 3 17:27:26 vhm-prd-02 kernel: ovirtmgmt: port 1(em1) entered forwarding state Sep 3 17:27:26 vhm-prd-02 kernel: ovirtmgmt: port 1(em1) entered forwarding state Sep 3 17:27:26 vhm-prd-02 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ovirtmgmt: link becomes ready Sep 3 17:27:26 vhm-prd-02 network: Bringing up interface ovirtmgmt: [ OK ] Sep 3 17:27:26 vhm-prd-02 systemd: Started LSB: Bring up/down networking. Sep 3 17:27:26 vhm-prd-02 systemd: Starting Network. Sep 3 17:27:26 vhm-prd-02 systemd: Reached target Network. So ovirtmgmt and em1 were restore and initialized just fine (p3p1 and p4p1 should have been started, too, but engine configured them as ONBOOT=no). Further in messages (full log is attached):

Sep 3 17:27:26 vhm-prd-02 systemd: Starting Virtual Desktop Server Manager network restoration... Sep 3 17:27:26 vhm-prd-02 systemd: Started OSAD daemon. Sep 3 17:27:27 vhm-prd-02 systemd: Started Terminate Plymouth Boot Screen. Sep 3 17:27:27 vhm-prd-02 systemd: Started Wait for Plymouth Boot Screen to Quit. Sep 3 17:27:27 vhm-prd-02 systemd: Starting Serial Getty on ttyS1... Sep 3 17:27:27 vhm-prd-02 systemd: Started Serial Getty on ttyS1. Sep 3 17:27:27 vhm-prd-02 systemd: Starting Getty on tty1... Sep 3 17:27:27 vhm-prd-02 systemd: Started Getty on tty1. Sep 3 17:27:27 vhm-prd-02 systemd: Starting Login Prompts. Sep 3 17:27:27 vhm-prd-02 systemd: Reached target Login Prompts. Sep 3 17:27:27 vhm-prd-02 iscsid: iSCSI daemon with pid=1300 started! Sep 3 17:27:27 vhm-prd-02 avahi-daemon[778]: Registering new address record for fe80::d267:e5ff:fef0:e5c6 on ovirtmgmt.*. Sep 3 17:27:27 vhm-prd-02 kdumpctl: kexec: loaded kdump kernel Sep 3 17:27:27 vhm-prd-02 kdumpctl: Starting kdump: [OK] Sep 3 17:27:27 vhm-prd-02 systemd: Started Crash recovery kernel arming. Sep 3 17:27:27 vhm-prd-02 avahi-daemon[778]: Registering new address record for fe80::d267:e5ff:fef0:e5c6 on em1.*. Sep 3 17:27:27 vhm-prd-02 avahi-daemon[778]: Withdrawing address record for 1.2.3.16 on ovirtmgmt. Sep 3 17:27:27 vhm-prd-02 avahi-daemon[778]: Leaving mDNS multicast group on interface ovirtmgmt.IPv4 with address 1.2.3.16. Sep 3 17:27:27 vhm-prd-02 avahi-daemon[778]: Interface ovirtmgmt.IPv4 no longer relevant for mDNS. Sep 3 17:27:27 vhm-prd-02 kernel: ovirtmgmt: port 1(em1) entered disabled state Sep 3 17:27:27 vhm-prd-02 avahi-daemon[778]: Withdrawing address record for fe80::d267:e5ff:fef0:e5c6 on ovirtmgmt. Sep 3 17:27:28 vhm-prd-02 avahi-daemon[778]: Withdrawing address record for fe80::d267:e5ff:fef0:e5c6 on em1. Sep 3 17:27:28 vhm-prd-02 kernel: device em1 left promiscuous mode Sep 3 17:27:28 vhm-prd-02 kernel: ovirtmgmt: port 1(em1) entered disabled state Sep 3 17:27:28 vhm-prd-02 avahi-daemon[778]: Withdrawing workstation service for ovirtmgmt. Sep 3 17:27:28 vhm-prd-02 vdsm-tool: Traceback (most recent call last): Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/vdsm-restore-net-config", line 345, in <module> Sep 3 17:27:28 vhm-prd-02 vdsm-tool: restore(args) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/vdsm-restore-net-config", line 314, in restore Sep 3 17:27:28 vhm-prd-02 vdsm-tool: unified_restoration() Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/vdsm-restore-net-config", line 93, in unified_restoration Sep 3 17:27:28 vhm-prd-02 vdsm-tool: setupNetworks(nets, bonds, connectivityCheck=False, _inRollback=True) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/network/api.py", line 642, in setupNetworks Sep 3 17:27:28 vhm-prd-02 vdsm-tool: implicitBonding=False, _netinfo=_netinfo) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/network/api.py", line 213, in wrapped Sep 3 17:27:28 vhm-prd-02 vdsm-tool: ret = func(**attrs) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/network/api.py", line 429, in delNetwork Sep 3 17:27:28 vhm-prd-02 vdsm-tool: netEnt.remove() Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/network/models.py", line 100, in remove Sep 3 17:27:28 vhm-prd-02 vdsm-tool: self.configurator.removeNic(self) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/network/configurators/ifcfg.py", line 215, in removeNic Sep 3 17:27:28 vhm-prd-02 vdsm-tool: self.configApplier.removeNic(nic.name) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/share/vdsm/network/configurators/ifcfg.py", line 657, in removeNic Sep 3 17:27:28 vhm-prd-02 vdsm-tool: with open(cf) as nicFile: Sep 3 17:27:28 vhm-prd-02 vdsm-tool: IOError: [Errno 2] No such file or directory: u'/etc/sysconfig/network-scripts/ifcfg-p4p1' Sep 3 17:27:28 vhm-prd-02 vdsm-tool: Traceback (most recent call last): Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/bin/vdsm-tool", line 219, in main Sep 3 17:27:28 vhm-prd-02 vdsm-tool: return tool_command[cmd]["command"](*args) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 40, in restore_command Sep 3 17:27:28 vhm-prd-02 vdsm-tool: exec_restore(cmd) Sep 3 17:27:28 vhm-prd-02 vdsm-tool: File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 53, in exec_restore Sep 3 17:27:28 vhm-prd-02 vdsm-tool: raise EnvironmentError('Failed to restore the persisted networks') Sep 3 17:27:28 vhm-prd-02 vdsm-tool: EnvironmentError: Failed to restore the persisted networks Sep 3 17:27:28 vhm-prd-02 systemd: vdsm-network.service: main process exited, code=exited, status=1/FAILURE Sep 3 17:27:28 vhm-prd-02 systemd: Failed to start Virtual Desktop Server Manager network restoration. Sep 3 17:27:28 vhm-prd-02 systemd: Dependency failed for Virtual Desktop Server Manager. Sep 3 17:27:28 vhm-prd-02 systemd: Sep 3 17:27:28 vhm-prd-02 systemd: Unit vdsm-network.service entered failed state. Sep 3 17:27:33 vhm-prd-02 systemd: Started Postfix Mail Transport Agent. Sep 3 17:27:33 vhm-prd-02 systemd: Starting Multi-User System. Sep 3 17:27:33 vhm-prd-02 systemd: Reached target Multi-User System. Sep 3 17:27:33 vhm-prd-02 systemd: Starting Update UTMP about System Runlevel Changes... Sep 3 17:27:33 vhm-prd-02 systemd: Starting Stop Read-Ahead Data Collection 10s After Completed Startup. Sep 3 17:27:33 vhm-prd-02 systemd: Started Stop Read-Ahead Data Collection 10s After Completed Startup. Sep 3 17:27:33 vhm-prd-02 systemd: Started Update UTMP about System Runlevel Changes. Sep 3 17:27:33 vhm-prd-02 systemd: Startup finished in 2.964s (kernel) + 2.507s (initrd) + 15.996s (userspace) = 21.468s. So, as I have two more hosts, that need updating, I'm happy to assist in bisecting and debugging this update issue. Suggestions and help are very welcome.