Hosted engine install failure: ipv6.gateway: gateway cannot be set if there are no addresses configured
Hi, I'm attempting to install hosted engine but getting this failure: [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."} Looking in vdsm.log I find this which I believe is the root cause: MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug? /Sverker _______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0: { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } } This is my interfaces before vdsm attemtpts to change the config: enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000 enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible. /Sverker Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC...
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all. Best Regards, Strahil Nikolov В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа: Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0: { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } } This is my interfaces before vdsm attemtpts to change the config: enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000 enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible. /Sverker Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither. It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x. /Sverker Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{ 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
What is you switch brand & model ? Maybe someone more experienced in networking can help. Best Regards, Strahil Nikolov В сряда, 2 септември 2020 г., 23:39:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа: Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither. It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x. /Sverker Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{ 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
Hi Strahil, there is nothing wrong with the switch other than that I don't have any control over it so the network setup is as it is. The issue is that vdsm breaks the network setup because it's not flexible enough. /Sverker Den 2020-09-03 kl. 00:47, skrev Strahil Nikolov:
What is you switch brand & model ? Maybe someone more experienced in networking can help.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 23:39:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{ 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{
'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed:
dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now. If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected. Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change. /Sverker Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > Switchports can either be tagged or untagged. > I'm not sure that your setup is supported at all. > > Best Regards, > Strahil Nikolov > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> написа: > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to remove > the ip of my underlying interface, that is enp4s0: >
> { > 'interfaces': [{ > 'name': 'enp4s0', > 'state': 'up', > 'mtu': 1500 > }, { > 'vlan': { > 'id': 4000, > 'base-iface': 'enp4s0' > }, > 'name': 'enp4s0.4000', > 'type': 'vlan', > 'state': 'up', > 'mtu': 1500, > 'ipv4': { > 'enabled': False > }, > 'ipv6': { > 'enabled': False > } > }, { > 'name': 'ovirtmgmt', > 'type': 'linux-bridge', > 'state': 'up', > 'mtu': 1500, > 'bridge': { > 'port': [{ > 'name': 'enp4s0.4000' > } > ], > 'options': { > 'stp': { > 'enabled': False > } > } > }, > 'ipv4': { > 'enabled': True, > 'address': [{ > 'ip': '172.27.1.1', > 'prefix-length': 24 > } > ], > 'dhcp': False > }, > 'ipv6': { > 'enabled': False > } > } > ], > 'dns-resolver': { > 'config' > : { > 'server': ['213.133.98.98'] > } > } > } >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > This is my interfaces before vdsm attemtpts to change the config: > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 293442 bytes 385541799 (367.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 91095 bytes 31160348 (29.7 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 memory 0xf7d00000-f7d20000 > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 13 bytes 938 (938.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I.e. enp4s0 is the external interface that must not be changed, bridge > must be created on the vlan interface. I would prefer to create the > bridge manually and not through vdsm if that is possible. > > /Sverker > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> Hi, >> I'm attempting to install hosted engine but getting this failure: >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "The host has been set in non_operational status, deployment errors: >> code 505: Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> installation failed. Failed to >> configure management network on the host., code 1120: Failed to >> configure management network on host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> due to >> setup networks failure., code 9000: Failed to verify Power Management >> configuration for Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com>., code 10802: VDSM >> h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> command HostSetupNetworksVDS failed: Internal >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when running >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> Looking in vdsm.log I find this which I believe is the root cause: >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': >> '255.255.255.0', 'ipv6autoconf': False, >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> 'commitOnSuccess': True, 'connectivityCh >> eck': 'true'}) {} >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': >> 120, 'commitOnSuccess': >> True, 'connectivityCheck': 'true'} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd >> None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge >> -- list Port -- list Interface >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- >> list Interface (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', >> 'nameservers': ['213.133.98.98']}) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset >> --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> (cwd None) >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing >> setup through nmstate >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> 'bridge': {'port': [ >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], >> 'dns-resolver': {'config' >> : {'server': ['213.133.98.98']}}} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> interfaces: ['ovirtmgmt'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> interfaces: ['enp4s0.4000', 'enp4s0'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM >> action: func=add_connection_async >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> Connection adding succeeded: dev=ovirtmgmt >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM >> action: func=commit_changes_async >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> Connection update failed: error=nm-connection-error-quark: >> ipv6.gateway: gateway cannot be set if there are no addresses >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type >> NM.DeviceState> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> Error in setupNetworks >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> line 95, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 241, in setupNetworks >> _setup_networks(networks, bondings, options, net_info) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 266, in _setup_networks >> networks, bondings, options, net_info, in_rollback >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 154, in setup >> _setup_nmstate(networks, bondings, options, in_rollback) >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 199, in _setup_nmstate >> nmstate.setup(desired_state, verify_change=not in_rollback) >> File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> line 63, in setup >> state_apply(desired_state, verify_change=verify_change) >> File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> line 40, in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 96, in wrapped >> ret = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 73, in apply >> state.State(desired_state), verify_change, commit, rollback_timeout >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 163, in _apply_ifaces_state >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ >> next(self.gen) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 232, in _setup_providers >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 177, in run >> f"Unexpected failure of libnm when running the mainloop: {err}" >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when >> running the mainloop: run execution >> >> My host have a main network interface and a vlan interface on the same >> physical interface which is an internal management network which >> should be used as management network. The vlan interface doesn't have >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. This >> seems like a bug? >> >> /Sverker >> >> _______________________________________________ >> discuss mailing list >> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker, can you please share output from "nmcli con show" and "nmstatectl show"? Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{
'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices:
60
MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed:
dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
-- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil@redhat.com IM: amusil <https://red.ht/sig>
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine): [root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500 Reverting the configuration back to not have the static bridge and a reboot I get this: [root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500 /Sverker Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > Switchports can either be tagged or untagged. > I'm not sure that your setup is supported at all. > > Best Regards, > Strahil Nikolov > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> написа: > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to remove > the ip of my underlying interface, that is enp4s0: >
> { > 'interfaces': [{ > 'name': 'enp4s0', > 'state': 'up', > 'mtu': 1500 > }, { > 'vlan': { > 'id': 4000, > 'base-iface': 'enp4s0' > }, > 'name': 'enp4s0.4000', > 'type': 'vlan', > 'state': 'up', > 'mtu': 1500, > 'ipv4': { > 'enabled': False > }, > 'ipv6': { > 'enabled': False > } > }, { > 'name': 'ovirtmgmt', > 'type': 'linux-bridge', > 'state': 'up', > 'mtu': 1500, > 'bridge': { > 'port': [{ > 'name': 'enp4s0.4000' > } > ], > 'options': { > 'stp': { > 'enabled': False > } > } > }, > 'ipv4': { > 'enabled': True, > 'address': [{ > 'ip': '172.27.1.1', > 'prefix-length': 24 > } > ], > 'dhcp': False > }, > 'ipv6': { > 'enabled': False > } > } > ], > 'dns-resolver': { > 'config' > : { > 'server': ['213.133.98.98'] > } > } > } >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > This is my interfaces before vdsm attemtpts to change the config: > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 293442 bytes 385541799 (367.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 91095 bytes 31160348 (29.7 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 memory 0xf7d00000-f7d20000 > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 13 bytes 938 (938.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I.e. enp4s0 is the external interface that must not be changed, bridge > must be created on the vlan interface. I would prefer to create the > bridge manually and not through vdsm if that is possible. > > /Sverker > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> Hi, >> I'm attempting to install hosted engine but getting this failure: >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "The host has been set in non_operational status, deployment errors: >> code 505: Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> installation failed. Failed to >> configure management network on the host., code 1120: Failed to >> configure management network on host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> due to >> setup networks failure., code 9000: Failed to verify Power Management >> configuration for Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com>., code 10802: VDSM >> h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> command HostSetupNetworksVDS failed: Internal >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when running >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> Looking in vdsm.log I find this which I believe is the root cause: >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': >> '255.255.255.0', 'ipv6autoconf': False, >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> 'commitOnSuccess': True, 'connectivityCh >> eck': 'true'}) {} >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': >> 120, 'commitOnSuccess': >> True, 'connectivityCheck': 'true'} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd >> None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge >> -- list Port -- list Interface >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- >> list Interface (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', >> 'nameservers': ['213.133.98.98']}) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset >> --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> (cwd None) >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing >> setup through nmstate >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> 'bridge': {'port': [ >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], >> 'dns-resolver': {'config' >> : {'server': ['213.133.98.98']}}} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> interfaces: ['ovirtmgmt'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> interfaces: ['enp4s0.4000', 'enp4s0'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM >> action: func=add_connection_async >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> Connection adding succeeded: dev=ovirtmgmt >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM >> action: func=commit_changes_async >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> Connection update failed: error=nm-connection-error-quark: >> ipv6.gateway: gateway cannot be set if there are no addresses >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type >> NM.DeviceState> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> Error in setupNetworks >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> line 95, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 241, in setupNetworks >> _setup_networks(networks, bondings, options, net_info) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 266, in _setup_networks >> networks, bondings, options, net_info, in_rollback >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 154, in setup >> _setup_nmstate(networks, bondings, options, in_rollback) >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 199, in _setup_nmstate >> nmstate.setup(desired_state, verify_change=not in_rollback) >> File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> line 63, in setup >> state_apply(desired_state, verify_change=verify_change) >> File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> line 40, in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 96, in wrapped >> ret = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 73, in apply >> state.State(desired_state), verify_change, commit, rollback_timeout >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 163, in _apply_ifaces_state >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ >> next(self.gen) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 232, in _setup_providers >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 177, in run >> f"Unexpected failure of libnm when running the mainloop: {err}" >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when >> running the mainloop: run execution >> >> My host have a main network interface and a vlan interface on the same >> physical interface which is an internal management network which >> should be used as management network. The vlan interface doesn't have >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. This >> seems like a bug? >> >> /Sverker >> >> _______________________________________________ >> discuss mailing list >> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson <sverker@abrahamsson.com> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again. Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{
'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed to configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02
16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper)
call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed:
dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02
16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper)
Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
-- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil@redhat.com IM: amusil <https://red.ht/sig>
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably. The enp4s0 config was created by the install, so it should be controlled by NetworkManager. /Sverker Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 <http://192.168.1.0/24> metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > Switchports can either be tagged or untagged. > I'm not sure that your setup is supported at all. > > Best Regards, > Strahil Nikolov > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> написа: > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to remove > the ip of my underlying interface, that is enp4s0: >
> { > 'interfaces': [{ > 'name': 'enp4s0', > 'state': 'up', > 'mtu': 1500 > }, { > 'vlan': { > 'id': 4000, > 'base-iface': 'enp4s0' > }, > 'name': 'enp4s0.4000', > 'type': 'vlan', > 'state': 'up', > 'mtu': 1500, > 'ipv4': { > 'enabled': False > }, > 'ipv6': { > 'enabled': False > } > }, { > 'name': 'ovirtmgmt', > 'type': 'linux-bridge', > 'state': 'up', > 'mtu': 1500, > 'bridge': { > 'port': [{ > 'name': 'enp4s0.4000' > } > ], > 'options': { > 'stp': { > 'enabled': False > } > } > }, > 'ipv4': { > 'enabled': True, > 'address': [{ > 'ip': '172.27.1.1', > 'prefix-length': 24 > } > ], > 'dhcp': False > }, > 'ipv6': { > 'enabled': False > } > } > ], > 'dns-resolver': { > 'config' > : { > 'server': ['213.133.98.98'] > } > } > } >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > This is my interfaces before vdsm attemtpts to change the config: > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 293442 bytes 385541799 (367.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 91095 bytes 31160348 (29.7 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 memory 0xf7d00000-f7d20000 > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 13 bytes 938 (938.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I.e. enp4s0 is the external interface that must not be changed, bridge > must be created on the vlan interface. I would prefer to create the > bridge manually and not through vdsm if that is possible. > > /Sverker > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> Hi, >> I'm attempting to install hosted engine but getting this failure: >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "The host has been set in non_operational status, deployment errors: >> code 505: Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> installation failed. Failed to >> configure management network on the host., code 1120: Failed to >> configure management network on host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> due to >> setup networks failure., code 9000: Failed to verify Power Management >> configuration for Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com>., code 10802: VDSM >> h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> command HostSetupNetworksVDS failed: Internal >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when running >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> Looking in vdsm.log I find this which I believe is the root cause: >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': >> '255.255.255.0', 'ipv6autoconf': False, >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> 'commitOnSuccess': True, 'connectivityCh >> eck': 'true'}) {} >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': >> 120, 'commitOnSuccess': >> True, 'connectivityCheck': 'true'} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd >> None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge >> -- list Port -- list Interface >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- >> list Interface (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', >> 'nameservers': ['213.133.98.98']}) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset >> --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> (cwd None) >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing >> setup through nmstate >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> 'bridge': {'port': [ >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], >> 'dns-resolver': {'config' >> : {'server': ['213.133.98.98']}}} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> interfaces: ['ovirtmgmt'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> interfaces: ['enp4s0.4000', 'enp4s0'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM >> action: func=add_connection_async >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> Connection adding succeeded: dev=ovirtmgmt >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM >> action: func=commit_changes_async >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> Connection update failed: error=nm-connection-error-quark: >> ipv6.gateway: gateway cannot be set if there are no addresses >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type >> NM.DeviceState> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> Error in setupNetworks >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> line 95, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 241, in setupNetworks >> _setup_networks(networks, bondings, options, net_info) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 266, in _setup_networks >> networks, bondings, options, net_info, in_rollback >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 154, in setup >> _setup_nmstate(networks, bondings, options, in_rollback) >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 199, in _setup_nmstate >> nmstate.setup(desired_state, verify_change=not in_rollback) >> File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> line 63, in setup >> state_apply(desired_state, verify_change=verify_change) >> File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> line 40, in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 96, in wrapped >> ret = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 73, in apply >> state.State(desired_state), verify_change, commit, rollback_timeout >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 163, in _apply_ifaces_state >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ >> next(self.gen) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 232, in _setup_providers >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 177, in run >> f"Unexpected failure of libnm when running the mainloop: {err}" >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when >> running the mainloop: run execution >> >> My host have a main network interface and a vlan interface on the same >> physical interface which is an internal management network which >> should be used as management network. The vlan interface doesn't have >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. This >> seems like a bug? >> >> /Sverker >> >> _______________________________________________ >> discuss mailing list >> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
# yum install network-scripts # readlink $(readlink $(which ifup)) /etc/sysconfig/network-scripts/ifup #touch /etc/sysconfig/disable-deprecation-warnings #systemctl disable NetworkManager Add "NM_CONTROLLED" to the ifcfg- file. #systemctl enable --now network For more details: https://access.redhat.com/solutions/4365931 Best Regards, Strahil Nikolov В четвъртък, 3 септември 2020 г., 13:42:19 Гринуич+3, Sverker Abrahamsson <sverker@abrahamsson.com> написа: Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably. The enp4s0 config was created by the install, so it should be controlled by NetworkManager. /Sverker Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson <sverker@abrahamsson.com> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0[root@h1-mgmt ~]# nmstatectl show---dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98route-rules: config: []routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255interfaces:- name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500- name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500- name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500- name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000- name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536- name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500- name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500- name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500- name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500- name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again. Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users <users@ovirt.org> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you.
Regards,
Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users <users@ovirt.org> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
> {
'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
Thanks, this is helpful information.
Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request,
and the parameters of the setupNetworks request from Engine to vdsm?
Both are in the vdsm.log during adding the host.
>
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: > Hi, > I'm attempting to install hosted engine but getting this failure: > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": > "The host has been set in non_operational status, deployment errors: > code 505: Host h1-mgmt.limetransit.com installation failed. Failed to > configure management network on the host., code 1120: Failed to > configure management network on host h1-mgmt.limetransit.com due to > setup networks failure., code 9000: Failed to verify Power Management > configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM > h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal > JSON-RPC error: {'reason': 'Unexpected failure of libnm when running > the mainloop: run execution'}, fix accordingly and re-deploy."} > > Looking in vdsm.log I find this which I believe is the root cause: > > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) > call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': > '255.255.255.0', 'ipv6autoconf': False, > 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', > 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, > 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, > 'commitOnSuccess': True, 'connectivityCh > eck': 'true'}) {} > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:25,897::api::220::root::(setupNetworks) Setting up network > according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', > 'netmask': '255.255.255.0', 'ipv6autoconf': Fal > se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', > 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, > 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': > 120, 'commitOnSuccess': > True, 'connectivityCheck': 'true'} > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd > None) > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; > <rc> = 0 > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,945::vsctl::74::root::(commit) Executing commands: > /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge > -- list Port -- list Interface > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl > --timeout=5 --oneline --format=json -- list Bridge -- list Port -- > list Interface (cwd None) > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; > <rc> = 0 > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding > network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', > 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True > , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, > 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', > 'nameservers': ['213.133.98.98']}) > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset > --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe > (cwd None) > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:26,154::hooks::122::root::(_runHooksDir) > /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing > setup through nmstate > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: > {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, > {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} > , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, > 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': > 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, > 'bridge': {'port': [ > {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, > 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', > 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], > 'dns-resolver': {'config' > : {'server': ['213.133.98.98']}}} > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,217::checkpoint::121::root::(create) Checkpoint > /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new > interfaces: ['ovirtmgmt'] > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing > interfaces: ['enp4s0.4000', 'enp4s0'] > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM > action: func=add_connection_async > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,231::connection::329::root::(_add_connection_callback) > Connection adding succeeded: dev=ovirtmgmt > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM > action: func=commit_changes_async > MainProcess|jsonrpc/0::ERROR::2020-09-02 > 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: > Connection update failed: error=nm-connection-error-quark: > ipv6.gateway: gateway cannot be set if there are no addresses > configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type > NM.DeviceState> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint > /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: > dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/7'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/8'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/9'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/3'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/2'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/4'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/1'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/5'): > dbus.UInt32(0)}, signature=dbus.Signature('su')) > MainProcess|jsonrpc/0::ERROR::2020-09-02 > 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) > Error in setupNetworks > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", > line 95, in wrapper > res = func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line > 241, in setupNetworks > _setup_networks(networks, bondings, options, net_info) > File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line > 266, in _setup_networks > networks, bondings, options, net_info, in_rollback > File > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > line 154, in setup > _setup_nmstate(networks, bondings, options, in_rollback) > File > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > line 199, in _setup_nmstate > nmstate.setup(desired_state, verify_change=not in_rollback) > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 63, in setup > state_apply(desired_state, verify_change=verify_change) > File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", > line 40, in wrapper > return func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > line 96, in wrapped > ret = func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > line 73, in apply > state.State(desired_state), verify_change, commit, rollback_timeout > File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > line 163, in _apply_ifaces_state > con_profiles=ifaces_add_configs + ifaces_edit_configs, > File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ > next(self.gen) > File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > line 232, in _setup_providers > mainloop.run(timeout=MAINLOOP_TIMEOUT) > File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > line 177, in run > f"Unexpected failure of libnm when running the mainloop: {err}" > libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when > running the mainloop: run execution > > My host have a main network interface and a vlan interface on the same > physical interface which is an internal management network which > should be used as management network. The vlan interface doesn't have > any ipv6 interface, but nmclient fails on setting ipv6 gateway. This > seems like a bug? > > /Sverker > > _______________________________________________ > discuss mailing list > discuss@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA
amusil@redhat.com IM: amusil
-- Ales Musil Software Engineer - RHV Network Red Hat EMEA amusil@redhat.com IM: amusil
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson <sverker@abrahamsson.com> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug. [1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{
'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast
172.27.1.255
inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed, bridge must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users:
Hi, I'm attempting to install hosted engine but getting this failure:
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The host has been set in non_operational status, deployment errors: code 505: Host h1-mgmt.limetransit.com installation failed. Failed
to
configure management network on the host., code 1120: Failed to configure management network on host h1-mgmt.limetransit.com due to setup networks failure., code 9000: Failed to verify Power Management configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': 'Unexpected failure of libnm when running the mainloop: run execution'}, fix accordingly and re-deploy."}
Looking in vdsm.log I find this which I believe is the root cause:
MainProcess|jsonrpc/0::DEBUG::2020-09-02
16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper)
call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCh eck': 'true'}) {} MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,897::api::220::root::(setupNetworks) Setting up network according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', 'netmask': '255.255.255.0', 'ipv6autoconf': Fal se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::vsctl::74::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; <rc> = 0 MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', 'nameservers': ['213.133.98.98']}) MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None) MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,154::hooks::122::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing setup through nmstate MainProcess|jsonrpc/0::INFO::2020-09-02 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [ {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], 'dns-resolver': {'config' : {'server': ['213.133.98.98']}}} MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,217::checkpoint::121::root::(create) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new interfaces: ['ovirtmgmt'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing interfaces: ['enp4s0.4000', 'enp4s0'] MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM action: func=add_connection_async MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,231::connection::329::root::(_add_connection_callback) Connection adding succeeded: dev=ovirtmgmt MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM action: func=commit_changes_async MainProcess|jsonrpc/0::ERROR::2020-09-02 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: Connection update failed: error=nm-connection-error-quark: ipv6.gateway: gateway cannot be set if there are no addresses configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type NM.DeviceState> MainProcess|jsonrpc/0::DEBUG::2020-09-02 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed:
dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'): dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'): dbus.UInt32(0)}, signature=dbus.Signature('su')) MainProcess|jsonrpc/0::ERROR::2020-09-02
16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper)
Error in setupNetworks Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks _setup_networks(networks, bondings, options, net_info) File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks networks, bondings, options, net_info, in_rollback File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 154, in setup _setup_nmstate(networks, bondings, options, in_rollback) File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
line 199, in _setup_nmstate nmstate.setup(desired_state, verify_change=not in_rollback) File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in setup state_apply(desired_state, verify_change=verify_change) File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped ret = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply state.State(desired_state), verify_change, commit, rollback_timeout File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in _apply_ifaces_state con_profiles=ifaces_add_configs + ifaces_edit_configs, File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in _setup_providers mainloop.run(timeout=MAINLOOP_TIMEOUT) File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177, in run f"Unexpected failure of libnm when running the mainloop: {err}" libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop: run execution
My host have a main network interface and a vlan interface on the same physical interface which is an internal management network which should be used as management network. The vlan interface doesn't have any ipv6 interface, but nmclient fails on setting ipv6 gateway. This seems like a bug?
/Sverker
_______________________________________________ discuss mailing list discuss@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker
Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 <http://192.168.1.0/24> metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > Switchports can either be tagged or untagged. > I'm not sure that your setup is supported at all. > > Best Regards, > Strahil Nikolov > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> написа: > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to remove > the ip of my underlying interface, that is enp4s0: >
> { > 'interfaces': [{ > 'name': 'enp4s0', > 'state': 'up', > 'mtu': 1500 > }, { > 'vlan': { > 'id': 4000, > 'base-iface': 'enp4s0' > }, > 'name': 'enp4s0.4000', > 'type': 'vlan', > 'state': 'up', > 'mtu': 1500, > 'ipv4': { > 'enabled': False > }, > 'ipv6': { > 'enabled': False > } > }, { > 'name': 'ovirtmgmt', > 'type': 'linux-bridge', > 'state': 'up', > 'mtu': 1500, > 'bridge': { > 'port': [{ > 'name': 'enp4s0.4000' > } > ], > 'options': { > 'stp': { > 'enabled': False > } > } > }, > 'ipv4': { > 'enabled': True, > 'address': [{ > 'ip': '172.27.1.1', > 'prefix-length': 24 > } > ], > 'dhcp': False > }, > 'ipv6': { > 'enabled': False > } > } > ], > 'dns-resolver': { > 'config' > : { > 'server': ['213.133.98.98'] > } > } > } >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > This is my interfaces before vdsm attemtpts to change the config: > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 293442 bytes 385541799 (367.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 91095 bytes 31160348 (29.7 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 memory 0xf7d00000-f7d20000 > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 13 bytes 938 (938.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I.e. enp4s0 is the external interface that must not be changed, bridge > must be created on the vlan interface. I would prefer to create the > bridge manually and not through vdsm if that is possible. > > /Sverker > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> Hi, >> I'm attempting to install hosted engine but getting this failure: >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "The host has been set in non_operational status, deployment errors: >> code 505: Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> installation failed. Failed to >> configure management network on the host., code 1120: Failed to >> configure management network on host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> due to >> setup networks failure., code 9000: Failed to verify Power Management >> configuration for Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com>., code 10802: VDSM >> h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> command HostSetupNetworksVDS failed: Internal >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when running >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> Looking in vdsm.log I find this which I believe is the root cause: >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': >> '255.255.255.0', 'ipv6autoconf': False, >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> 'commitOnSuccess': True, 'connectivityCh >> eck': 'true'}) {} >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': >> 120, 'commitOnSuccess': >> True, 'connectivityCheck': 'true'} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd >> None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge >> -- list Port -- list Interface >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- >> list Interface (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', >> 'nameservers': ['213.133.98.98']}) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset >> --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> (cwd None) >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing >> setup through nmstate >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> 'bridge': {'port': [ >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], >> 'dns-resolver': {'config' >> : {'server': ['213.133.98.98']}}} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> interfaces: ['ovirtmgmt'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> interfaces: ['enp4s0.4000', 'enp4s0'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM >> action: func=add_connection_async >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> Connection adding succeeded: dev=ovirtmgmt >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM >> action: func=commit_changes_async >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> Connection update failed: error=nm-connection-error-quark: >> ipv6.gateway: gateway cannot be set if there are no addresses >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type >> NM.DeviceState> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> Error in setupNetworks >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> line 95, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 241, in setupNetworks >> _setup_networks(networks, bondings, options, net_info) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 266, in _setup_networks >> networks, bondings, options, net_info, in_rollback >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 154, in setup >> _setup_nmstate(networks, bondings, options, in_rollback) >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 199, in _setup_nmstate >> nmstate.setup(desired_state, verify_change=not in_rollback) >> File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> line 63, in setup >> state_apply(desired_state, verify_change=verify_change) >> File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> line 40, in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 96, in wrapped >> ret = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 73, in apply >> state.State(desired_state), verify_change, commit, rollback_timeout >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 163, in _apply_ifaces_state >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ >> next(self.gen) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 232, in _setup_providers >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 177, in run >> f"Unexpected failure of libnm when running the mainloop: {err}" >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when >> running the mainloop: run execution >> >> My host have a main network interface and a vlan interface on the same >> physical interface which is an internal management network which >> should be used as management network. The vlan interface doesn't have >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. This >> seems like a bug? >> >> /Sverker >> >> _______________________________________________ >> discuss mailing list >> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
Sverker, thanks! On Thu, Sep 3, 2020 at 6:50 PM Sverker Abrahamsson <sverker@abrahamsson.com> wrote:
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov:
Switchports can either be tagged or untagged. I'm not sure that your setup is supported at all.
Best Regards, Strahil Nikolov
В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа:
Pretty formatting the "desired state" it seems that vdsm tries to remove the ip of my underlying interface, that is enp4s0:
{
'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config' : { 'server': ['213.133.98.98'] } } }
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
This is my interfaces before vdsm attemtpts to change the config:
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 144.76.84.73 netmask 255.255.255.255 broadcast
0.0.0.0
inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid
0x0<global>
ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 293442 bytes 385541799 (367.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 91095 bytes 31160348 (29.7 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 17 memory 0xf7d00000-f7d20000
enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.27.1.1 netmask 255.255.255.0 broadcast
172.27.1.255
inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid
0x20<link>
ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 13 bytes 938 (938.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I.e. enp4s0 is the external interface that must not be changed,
bridge
must be created on the vlan interface. I would prefer to create the bridge manually and not through vdsm if that is possible.
/Sverker
Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: > Hi, > I'm attempting to install hosted engine but getting this failure: > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": > "The host has been set in non_operational status, deployment errors: > code 505: Host h1-mgmt.limetransit.com installation failed. Failed to > configure management network on the host., code 1120: Failed to > configure management network on host h1-mgmt.limetransit.com due to > setup networks failure., code 9000: Failed to verify Power Management > configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM > h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal > JSON-RPC error: {'reason': 'Unexpected failure of libnm when running > the mainloop: run execution'}, fix accordingly and re-deploy."} > > Looking in vdsm.log I find this which I believe is the root cause: > > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) > call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': > '255.255.255.0', 'ipv6autoconf': False, > 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', > 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, > 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, > 'commitOnSuccess': True, 'connectivityCh > eck': 'true'}) {} > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:25,897::api::220::root::(setupNetworks) Setting up network > according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', > 'netmask': '255.255.255.0', 'ipv6autoconf': Fal > se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', > 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, > 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': > 120, 'commitOnSuccess': > True, 'connectivityCheck': 'true'} > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd > None) > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; > <rc> = 0 > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,945::vsctl::74::root::(commit) Executing commands: > /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge > -- list Port -- list Interface > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl > --timeout=5 --oneline --format=json -- list Bridge -- list Port -- > list Interface (cwd None) > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; > <rc> = 0 > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding > network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', > 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True > , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, > 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', > 'nameservers': ['213.133.98.98']}) > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset > --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe > (cwd None) > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:26,154::hooks::122::root::(_runHooksDir) > /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing > setup through nmstate > MainProcess|jsonrpc/0::INFO::2020-09-02 > 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: > {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, > {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} > , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, > 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': > 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, > 'bridge': {'port': [ > {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, > 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', > 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], > 'dns-resolver': {'config' > : {'server': ['213.133.98.98']}}} > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,217::checkpoint::121::root::(create) Checkpoint > /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new > interfaces: ['ovirtmgmt'] > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing > interfaces: ['enp4s0.4000', 'enp4s0'] > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM > action: func=add_connection_async > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,231::connection::329::root::(_add_connection_callback) > Connection adding succeeded: dev=ovirtmgmt > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM > action: func=commit_changes_async > MainProcess|jsonrpc/0::ERROR::2020-09-02 > 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: > Connection update failed: error=nm-connection-error-quark: > ipv6.gateway: gateway cannot be set if there are no addresses > configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type > NM.DeviceState> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint > /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: > dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/7'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/8'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/9'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/3'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/2'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/4'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/1'): > dbus.UInt32(0), > dbus.String('/org/freedesktop/NetworkManager/Devices/5'): > dbus.UInt32(0)}, signature=dbus.Signature('su')) > MainProcess|jsonrpc/0::ERROR::2020-09-02 > 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) > Error in setupNetworks > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", > line 95, in wrapper > res = func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line > 241, in setupNetworks > _setup_networks(networks, bondings, options, net_info) > File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line > 266, in _setup_networks > networks, bondings, options, net_info, in_rollback > File > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > line 154, in setup > _setup_nmstate(networks, bondings, options, in_rollback) > File > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > line 199, in _setup_nmstate > nmstate.setup(desired_state, verify_change=not in_rollback) > File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > line 63, in setup > state_apply(desired_state, verify_change=verify_change) > File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", > line 40, in wrapper > return func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > line 96, in wrapped > ret = func(*args, **kwargs) > File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > line 73, in apply > state.State(desired_state), verify_change, commit, rollback_timeout > File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > line 163, in _apply_ifaces_state > con_profiles=ifaces_add_configs + ifaces_edit_configs, > File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ > next(self.gen) > File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > line 232, in _setup_providers > mainloop.run(timeout=MAINLOOP_TIMEOUT) > File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > line 177, in run > f"Unexpected failure of libnm when running the mainloop: {err}" > libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when > running the mainloop: run execution > > My host have a main network interface and a vlan interface on the same > physical interface which is an internal management network which > should be used as management network. The vlan interface doesn't have > any ipv6 interface, but nmclient fails on setting ipv6 gateway. This > seems like a bug? > > /Sverker > > _______________________________________________ > discuss mailing list > discuss@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
Sverker, is this bug blocking you, or can you work around it? On Thu, Sep 3, 2020 at 8:52 PM Dominik Holler <dholler@redhat.com> wrote:
Sverker, thanks!
On Thu, Sep 3, 2020 at 6:50 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > Switchports can either be tagged or untagged. > I'm not sure that your setup is supported at all. > > Best Regards, > Strahil Nikolov > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org> написа: > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to remove > the ip of my underlying interface, that is enp4s0: >
{ > 'interfaces': [{ > 'name': 'enp4s0', > 'state': 'up', > 'mtu': 1500 > }, { > 'vlan': { > 'id': 4000, > 'base-iface': 'enp4s0' > }, > 'name': 'enp4s0.4000', > 'type': 'vlan', > 'state': 'up', > 'mtu': 1500, > 'ipv4': { > 'enabled': False > }, > 'ipv6': { > 'enabled': False > } > }, { > 'name': 'ovirtmgmt', > 'type': 'linux-bridge', > 'state': 'up', > 'mtu': 1500, > 'bridge': { > 'port': [{ > 'name': 'enp4s0.4000' > } > ], > 'options': { > 'stp': { > 'enabled': False > } > } > }, > 'ipv4': { > 'enabled': True, > 'address': [{ > 'ip': '172.27.1.1', > 'prefix-length': 24 > } > ], > 'dhcp': False > }, > 'ipv6': { > 'enabled': False > } > } > ], > 'dns-resolver': { > 'config' > : { > 'server': ['213.133.98.98'] > } > } > } >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > This is my interfaces before vdsm attemtpts to change the config: > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 293442 bytes 385541799 (367.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 91095 bytes 31160348 (29.7 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 memory 0xf7d00000-f7d20000 > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 13 bytes 938 (938.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I.e. enp4s0 is the external interface that must not be changed, bridge > must be created on the vlan interface. I would prefer to create the > bridge manually and not through vdsm if that is possible. > > /Sverker > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> Hi, >> I'm attempting to install hosted engine but getting this failure: >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "The host has been set in non_operational status, deployment errors: >> code 505: Host h1-mgmt.limetransit.com installation failed. Failed to >> configure management network on the host., code 1120: Failed to >> configure management network on host h1-mgmt.limetransit.com due to >> setup networks failure., code 9000: Failed to verify Power Management >> configuration for Host h1-mgmt.limetransit.com., code 10802: VDSM >> h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: Internal >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when running >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> Looking in vdsm.log I find this which I believe is the root cause: >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': >> '255.255.255.0', 'ipv6autoconf': False, >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> 'commitOnSuccess': True, 'connectivityCh >> eck': 'true'}) {} >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': >> 120, 'commitOnSuccess': >> True, 'connectivityCheck': 'true'} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd >> None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge >> -- list Port -- list Interface >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- >> list Interface (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', >> 'nameservers': ['213.133.98.98']}) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset >> --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> (cwd None) >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing >> setup through nmstate >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> 'bridge': {'port': [ >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], >> 'dns-resolver': {'config' >> : {'server': ['213.133.98.98']}}} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> interfaces: ['ovirtmgmt'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> interfaces: ['enp4s0.4000', 'enp4s0'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM >> action: func=add_connection_async >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> Connection adding succeeded: dev=ovirtmgmt >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM >> action: func=commit_changes_async >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> Connection update failed: error=nm-connection-error-quark: >> ipv6.gateway: gateway cannot be set if there are no addresses >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type >> NM.DeviceState> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> Error in setupNetworks >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> line 95, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 241, in setupNetworks >> _setup_networks(networks, bondings, options, net_info) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 266, in _setup_networks >> networks, bondings, options, net_info, in_rollback >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 154, in setup >> _setup_nmstate(networks, bondings, options, in_rollback) >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 199, in _setup_nmstate >> nmstate.setup(desired_state, verify_change=not in_rollback) >> File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> line 63, in setup >> state_apply(desired_state, verify_change=verify_change) >> File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> line 40, in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 96, in wrapped >> ret = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 73, in apply >> state.State(desired_state), verify_change, commit, rollback_timeout >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 163, in _apply_ifaces_state >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ >> next(self.gen) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 232, in _setup_providers >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 177, in run >> f"Unexpected failure of libnm when running the mainloop: {err}" >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when >> running the mainloop: run execution >> >> My host have a main network interface and a vlan interface on the same >> physical interface which is an internal management network which >> should be used as management network. The vlan interface doesn't have >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. This >> seems like a bug? >> >> /Sverker >> >> _______________________________________________ >> discuss mailing list >> discuss@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> _______________________________________________ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
Unforturnatly it is a blocker, I have not found a way to work around it. Den 2020-09-04 kl. 09:47, skrev Dominik Holler:
Sverker, is this bug blocking you, or can you work around it?
On Thu, Sep 3, 2020 at 8:52 PM Dominik Holler <dholler@redhat.com <mailto:dholler@redhat.com>> wrote:
Sverker, thanks!
On Thu, Sep 3, 2020 at 6:50 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker
Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker
Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 <http://192.168.1.0/24> metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > Switchports can either be tagged or untagged. > I'm not sure that your setup is supported at all. > > Best Regards, > Strahil Nikolov > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> написа: > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to remove > the ip of my underlying interface, that is enp4s0: >
> { > 'interfaces': [{ > 'name': 'enp4s0', > 'state': 'up', > 'mtu': 1500 > }, { > 'vlan': { > 'id': 4000, > 'base-iface': 'enp4s0' > }, > 'name': 'enp4s0.4000', > 'type': 'vlan', > 'state': 'up', > 'mtu': 1500, > 'ipv4': { > 'enabled': False > }, > 'ipv6': { > 'enabled': False > } > }, { > 'name': 'ovirtmgmt', > 'type': 'linux-bridge', > 'state': 'up', > 'mtu': 1500, > 'bridge': { > 'port': [{ > 'name': 'enp4s0.4000' > } > ], > 'options': { > 'stp': { > 'enabled': False > } > } > }, > 'ipv4': { > 'enabled': True, > 'address': [{ > 'ip': '172.27.1.1', > 'prefix-length': 24 > } > ], > 'dhcp': False > }, > 'ipv6': { > 'enabled': False > } > } > ], > 'dns-resolver': { > 'config' > : { > 'server': ['213.133.98.98'] > } > } > } >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > This is my interfaces before vdsm attemtpts to change the config: > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 293442 bytes 385541799 (367.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 91095 bytes 31160348 (29.7 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 memory 0xf7d00000-f7d20000 > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 13 bytes 938 (938.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I.e. enp4s0 is the external interface that must not be changed, bridge > must be created on the vlan interface. I would prefer to create the > bridge manually and not through vdsm if that is possible. > > /Sverker > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> Hi, >> I'm attempting to install hosted engine but getting this failure: >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "The host has been set in non_operational status, deployment errors: >> code 505: Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> installation failed. Failed to >> configure management network on the host., code 1120: Failed to >> configure management network on host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> due to >> setup networks failure., code 9000: Failed to verify Power Management >> configuration for Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com>., code 10802: VDSM >> h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> command HostSetupNetworksVDS failed: Internal >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when running >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> Looking in vdsm.log I find this which I believe is the root cause: >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': >> '255.255.255.0', 'ipv6autoconf': False, >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> 'commitOnSuccess': True, 'connectivityCh >> eck': 'true'}) {} >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': >> 120, 'commitOnSuccess': >> True, 'connectivityCheck': 'true'} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd >> None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge >> -- list Port -- list Interface >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- >> list Interface (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', >> 'nameservers': ['213.133.98.98']}) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset >> --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> (cwd None) >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing >> setup through nmstate >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> 'bridge': {'port': [ >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], >> 'dns-resolver': {'config' >> : {'server': ['213.133.98.98']}}} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> interfaces: ['ovirtmgmt'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> interfaces: ['enp4s0.4000', 'enp4s0'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM >> action: func=add_connection_async >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> Connection adding succeeded: dev=ovirtmgmt >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM >> action: func=commit_changes_async >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> Connection update failed: error=nm-connection-error-quark: >> ipv6.gateway: gateway cannot be set if there are no addresses >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type >> NM.DeviceState> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> Error in setupNetworks >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> line 95, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 241, in setupNetworks >> _setup_networks(networks, bondings, options, net_info) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 266, in _setup_networks >> networks, bondings, options, net_info, in_rollback >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 154, in setup >> _setup_nmstate(networks, bondings, options, in_rollback) >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 199, in _setup_nmstate >> nmstate.setup(desired_state, verify_change=not in_rollback) >> File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> line 63, in setup >> state_apply(desired_state, verify_change=verify_change) >> File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> line 40, in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 96, in wrapped >> ret = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 73, in apply >> state.State(desired_state), verify_change, commit, rollback_timeout >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 163, in _apply_ifaces_state >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ >> next(self.gen) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 232, in _setup_providers >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 177, in run >> f"Unexpected failure of libnm when running the mainloop: {err}" >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when >> running the mainloop: run execution >> >> My host have a main network interface and a vlan interface on the same >> physical interface which is an internal management network which >> should be used as management network. The vlan interface doesn't have >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. This >> seems like a bug? >> >> /Sverker >> >> _______________________________________________ >> discuss mailing list >> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
On Mon, Sep 7, 2020 at 12:22 PM Sverker Abrahamsson <sverker@abrahamsson.com> wrote:
Unforturnatly it is a blocker, I have not found a way to work around it.
Is there a change in behavior, if IPv6 on the non-vlanned interface (enp4s0) is configured in a static way?
Den 2020-09-04 kl. 09:47, skrev Dominik Holler:
Sverker, is this bug blocking you, or can you work around it?
On Thu, Sep 3, 2020 at 8:52 PM Dominik Holler <dholler@redhat.com> wrote:
Sverker, thanks!
On Thu, Sep 3, 2020 at 6:50 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
> Well, unforturnatly I don't have a choise since it is out of my > control. > I only have one physical network port where the external traffic is > untagged and the internal vlan is tagged. If I could run with OVN >
OVN is for VM traffic only, not usable for the management network.
> instead I wouldn't need that tagged vlan, but I haven't been able to > get > that to work neither. > > Please let us know if OVN does not work for VM traffic for you.
> It's perfectly possible to have both tagged and untagged traffic on > the > same switch port, issue is that vdsm tries to take control over the > network without being able to be flexible enough.. I'm attempting > now to > have ovirtmgmt bridge created before, that used to be possible but > according to previous mails on the list it went broken somewhere at > 4.x. > > /Sverker > > Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > > Switchports can either be tagged or untagged. > > I'm not sure that your setup is supported at all. > > > > Best Regards, > > Strahil Nikolov > > > > > > > > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker > Abrahamsson via Users <users@ovirt.org> написа: > > > > > > > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to > remove > > the ip of my underlying interface, that is enp4s0: > >
> { > > 'interfaces': [{ > > 'name': 'enp4s0', > > 'state': 'up', > > 'mtu': 1500 > > }, { > > 'vlan': { > > 'id': 4000, > > 'base-iface': 'enp4s0' > > }, > > 'name': 'enp4s0.4000', > > 'type': 'vlan', > > 'state': 'up', > > 'mtu': 1500, > > 'ipv4': { > > 'enabled': False > > }, > > 'ipv6': { > > 'enabled': False > > } > > }, { > > 'name': 'ovirtmgmt', > > 'type': 'linux-bridge', > > 'state': 'up', > > 'mtu': 1500, > > 'bridge': { > > 'port': [{ > > 'name': 'enp4s0.4000' > > } > > ], > > 'options': { > > 'stp': { > > 'enabled': False > > } > > } > > }, > > 'ipv4': { > > 'enabled': True, > > 'address': [{ > > 'ip': '172.27.1.1', > > 'prefix-length': 24 > > } > > ], > > 'dhcp': False > > }, > > 'ipv6': { > > 'enabled': False > > } > > } > > ], > > 'dns-resolver': { > > 'config' > > : { > > 'server': ['213.133.98.98'] > > } > > } > > } > > >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > > > This is my interfaces before vdsm attemtpts to change the config: > > > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > > inet 144.76.84.73 netmask 255.255.255.255 broadcast > 0.0.0.0 > > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid > 0x20<link> > > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid > 0x0<global> > > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > > RX packets 293442 bytes 385541799 (367.6 MiB) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 91095 bytes 31160348 (29.7 MiB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > device interrupt 17 memory 0xf7d00000-f7d20000 > > > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > > inet 172.27.1.1 netmask 255.255.255.0 broadcast > 172.27.1.255 > > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid > 0x20<link> > > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > > RX packets 0 bytes 0 (0.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 13 bytes 938 (938.0 B) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > I.e. enp4s0 is the external interface that must not be changed, > bridge > > must be created on the vlan interface. I would prefer to create the > > bridge manually and not through vdsm if that is possible. > > > > /Sverker > > > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: > >> Hi, > >> I'm attempting to install hosted engine but getting this failure: > >> > >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": > >> "The host has been set in non_operational status, deployment > errors: > >> code 505: Host h1-mgmt.limetransit.com installation failed. > Failed to > >> configure management network on the host., code 1120: Failed to > >> configure management network on host h1-mgmt.limetransit.com due > to > >> setup networks failure., code 9000: Failed to verify Power > Management > >> configuration for Host h1-mgmt.limetransit.com., code 10802: > VDSM > >> h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: > Internal > >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when > running > >> the mainloop: run execution'}, fix accordingly and re-deploy."} > >> > >> Looking in vdsm.log I find this which I believe is the root cause: > >> > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) > >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': > >> '255.255.255.0', 'ipv6autoconf': False, > >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', > >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, > >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, > >> 'commitOnSuccess': True, 'connectivityCh > >> eck': 'true'}) {} > >> MainProcess|jsonrpc/0::INFO::2020-09-02 > >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network > >> according to configuration: networks:{'ovirtmgmt': {'vlan': > '4000', > >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal > >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', > >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, > >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': > >> 120, 'commitOnSuccess': > >> True, 'connectivityCheck': 'true'} > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show > (cwd > >> None) > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = > b''; > >> <rc> = 0 > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: > >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list > Bridge > >> -- list Port -- list Interface > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl > >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- > >> list Interface (cwd None) > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = > b''; > >> <rc> = 0 > >> MainProcess|jsonrpc/0::INFO::2020-09-02 > >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding > >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', > >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True > >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, > >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': > 'none', > >> 'nameservers': ['213.133.98.98']}) > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:25,958::commands::153::common.commands::(start) > /usr/bin/taskset > >> --cpu-list 0-7 > /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe > >> (cwd None) > >> MainProcess|jsonrpc/0::INFO::2020-09-02 > >> 16:38:26,154::hooks::122::root::(_runHooksDir) > >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' > >> MainProcess|jsonrpc/0::INFO::2020-09-02 > >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing > >> setup through nmstate > >> MainProcess|jsonrpc/0::INFO::2020-09-02 > >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired > state: > >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, > >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} > >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': > 1500, > >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': > >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, > >> 'bridge': {'port': [ > >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, > >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', > >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': > False}}], > >> 'dns-resolver': {'config' > >> : {'server': ['213.133.98.98']}}} > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint > >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all > devices: 60 > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new > >> interfaces: ['ovirtmgmt'] > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing > >> interfaces: ['enp4s0.4000', 'enp4s0'] > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:26,224::nmclient::136::root::(execute_next_action) > Executing NM > >> action: func=add_connection_async > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:26,231::connection::329::root::(_add_connection_callback) > >> Connection adding succeeded: dev=ovirtmgmt > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:26,232::nmclient::136::root::(execute_next_action) > Executing NM > >> action: func=commit_changes_async > >> MainProcess|jsonrpc/0::ERROR::2020-09-02 > >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: > >> Connection update failed: error=nm-connection-error-quark: > >> ipv6.gateway: gateway cannot be set if there are no addresses > >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type > >> NM.DeviceState> > >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint > >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: > >> > dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): > >> dbus.UInt32(0), > >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): > >> dbus.UInt32(0)}, signature=dbus.Signature('su')) > >> MainProcess|jsonrpc/0::ERROR::2020-09-02 > >> > 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) > >> Error in setupNetworks > >> Traceback (most recent call last): > >> File > "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", > >> line 95, in wrapper > >> res = func(*args, **kwargs) > >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", > line > >> 241, in setupNetworks > >> _setup_networks(networks, bondings, options, net_info) > >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", > line > >> 266, in _setup_networks > >> networks, bondings, options, net_info, in_rollback > >> File > >> > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > >> line 154, in setup > >> _setup_nmstate(networks, bondings, options, in_rollback) > >> File > >> > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > >> line 199, in _setup_nmstate > >> nmstate.setup(desired_state, verify_change=not in_rollback) > >> File > "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > >> line 63, in setup > >> state_apply(desired_state, verify_change=verify_change) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", > >> line 40, in wrapper > >> return func(*args, **kwargs) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > >> line 96, in wrapped > >> ret = func(*args, **kwargs) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > >> line 73, in apply > >> state.State(desired_state), verify_change, commit, > rollback_timeout > >> File > "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > >> line 163, in _apply_ifaces_state > >> con_profiles=ifaces_add_configs + ifaces_edit_configs, > >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ > >> next(self.gen) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > >> line 232, in _setup_providers > >> mainloop.run(timeout=MAINLOOP_TIMEOUT) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > >> line 177, in run > >> f"Unexpected failure of libnm when running the mainloop: > {err}" > >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm > when > >> running the mainloop: run execution > >> > >> My host have a main network interface and a vlan interface on the > same > >> physical interface which is an internal management network which > >> should be used as management network. The vlan interface doesn't > have > >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. > This > >> seems like a bug? > >> > >> /Sverker > >> > >> _______________________________________________ > >> discuss mailing list > >> discuss@openvswitch.org > >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > >> _______________________________________________ > >> Users mailing list -- users@ovirt.org > >> To unsubscribe send an email to users-leave@ovirt.org > >> Privacy Statement: https://www.ovirt.org/privacy-policy.html > >> oVirt Code of Conduct: > >> https://www.ovirt.org/community/about/community-guidelines/ > >> List Archives: > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > > _______________________________________________ > > Users mailing list -- users@ovirt.org > > To unsubscribe send an email to users-leave@ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > > > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D... > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
It is static configured: [root@h1-mgmt network-scripts]# cat ifcfg-enp4s0 NAME=enp4s0 DEVICE=enp4s0 ONBOOT=yes UUID=af7ccb53-011b-4c36-998a-1878b4ae7100 TYPE=Ethernet BOOTPROTO=none IPADDR=144.76.84.73 PREFIX=32 GATEWAY=144.76.84.65 DEFROUTE=yes DNS1=213.133.98.98 IPV6INIT=yes IPV6ADDR=2a01:4f8:192:1148::2/64 IPV6_DEFAULTGW=fe80::1 IPV6_DEFROUTE=yes ZONE=public PROXY_METHOD=none BROWSER_ONLY=no IPV4_FAILURE_FATAL=no IPV6_AUTOCONF=yes IPV6_FAILURE_FATAL=no [root@h1-mgmt network-scripts]# cat ifcfg-enp4s0.4000 NAME=enp4s0.4000 DEVICE=enp4s0.4000 ONBOOT=yes VLAN=yes BOOTPROTO=none IPADDR=172.27.1.1 PREFIX=24 ZONE=internal IPV6INIT=no TYPE=Vlan PHYSDEV=enp4s0 VLAN_ID=4000 REORDER_HDR=yes GVRP=no MVRP=no HWADDR= PROXY_METHOD=none BROWSER_ONLY=no DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6_DISABLED=yes UUID=ecc8064d-18c1-99b7-3fe4-9c5a593ece6f I think that what would happen if it didn't fail due to ipv6 route is that enp4s0 would be without any ip's, based on what is logged as the desired state by vdsm: { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config': { 'server': ['213.133.98.98'] } } } The desired state should be something like this (although I don't know the correct names for the missing keys and there may be ): { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '144.76.84.73', 'prefix-length': 32, 'gateway': '144.76.84.65' } ], 'dhcp': False }, 'ipv6': { 'enabled': True, 'address': [{ 'ip': '2a01:4f8:192:1148::2', 'prefix-length': 64, 'gateway': 'fe80::1' } } }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config': { 'server': ['213.133.98.98'] } } } I.e. issue is that vdsm assumes that the underlying enp4s0 interface should not have any addresses when it should maintain all settings it has before. It's only the addresses of enp4s0.4000 that should move to the ovirtmgmt bridge. Alternatively enp4s0 should not get involved at all in the desired state, only enp4s0.4000 /Sverker Den 2020-09-07 kl. 12:54, skrev Dominik Holler:
On Mon, Sep 7, 2020 at 12:22 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Unforturnatly it is a blocker, I have not found a way to work around it.
Is there a change in behavior, if IPv6 on the non-vlanned interface (enp4s0) is configured in a static way?
Den 2020-09-04 kl. 09:47, skrev Dominik Holler:
Sverker, is this bug blocking you, or can you work around it?
On Thu, Sep 3, 2020 at 8:52 PM Dominik Holler <dholler@redhat.com <mailto:dholler@redhat.com>> wrote:
Sverker, thanks!
On Thu, Sep 3, 2020 at 6:50 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker
Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker
Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 <http://192.168.1.0/24> metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler:
On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Well, unforturnatly I don't have a choise since it is out of my control. I only have one physical network port where the external traffic is untagged and the internal vlan is tagged. If I could run with OVN
OVN is for VM traffic only, not usable for the management network.
instead I wouldn't need that tagged vlan, but I haven't been able to get that to work neither.
Please let us know if OVN does not work for VM traffic for you.
It's perfectly possible to have both tagged and untagged traffic on the same switch port, issue is that vdsm tries to take control over the network without being able to be flexible enough.. I'm attempting now to have ovirtmgmt bridge created before, that used to be possible but according to previous mails on the list it went broken somewhere at 4.x.
/Sverker
Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: > Switchports can either be tagged or untagged. > I'm not sure that your setup is supported at all. > > Best Regards, > Strahil Nikolov > > > > > > > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> написа: > > > > > > Pretty formatting the "desired state" it seems that vdsm tries to remove > the ip of my underlying interface, that is enp4s0: >
> { > 'interfaces': [{ > 'name': 'enp4s0', > 'state': 'up', > 'mtu': 1500 > }, { > 'vlan': { > 'id': 4000, > 'base-iface': 'enp4s0' > }, > 'name': 'enp4s0.4000', > 'type': 'vlan', > 'state': 'up', > 'mtu': 1500, > 'ipv4': { > 'enabled': False > }, > 'ipv6': { > 'enabled': False > } > }, { > 'name': 'ovirtmgmt', > 'type': 'linux-bridge', > 'state': 'up', > 'mtu': 1500, > 'bridge': { > 'port': [{ > 'name': 'enp4s0.4000' > } > ], > 'options': { > 'stp': { > 'enabled': False > } > } > }, > 'ipv4': { > 'enabled': True, > 'address': [{ > 'ip': '172.27.1.1', > 'prefix-length': 24 > } > ], > 'dhcp': False > }, > 'ipv6': { > 'enabled': False > } > } > ], > 'dns-resolver': { > 'config' > : { > 'server': ['213.133.98.98'] > } > } > } >
Thanks, this is helpful information. Can you please share the getCapabilities result sent from vdsm to Engine directly before the setupNetworks request, and the parameters of the setupNetworks request from Engine to vdsm? Both are in the vdsm.log during adding the host.
> > This is my interfaces before vdsm attemtpts to change the config: > > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 144.76.84.73 netmask 255.255.255.255 broadcast 0.0.0.0 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid 0x0<global> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 293442 bytes 385541799 (367.6 MiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 91095 bytes 31160348 (29.7 MiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 17 memory 0xf7d00000-f7d20000 > > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet 172.27.1.1 netmask 255.255.255.0 broadcast 172.27.1.255 > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid 0x20<link> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 13 bytes 938 (938.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > I.e. enp4s0 is the external interface that must not be changed, bridge > must be created on the vlan interface. I would prefer to create the > bridge manually and not through vdsm if that is possible. > > /Sverker > > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> Hi, >> I'm attempting to install hosted engine but getting this failure: >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "The host has been set in non_operational status, deployment errors: >> code 505: Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> installation failed. Failed to >> configure management network on the host., code 1120: Failed to >> configure management network on host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> due to >> setup networks failure., code 9000: Failed to verify Power Management >> configuration for Host h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com>., code 10802: VDSM >> h1-mgmt.limetransit.com <http://h1-mgmt.limetransit.com> command HostSetupNetworksVDS failed: Internal >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when running >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> Looking in vdsm.log I find this which I believe is the root cause: >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', 'netmask': >> '255.255.255.0', 'ipv6autoconf': False, >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> 'commitOnSuccess': True, 'connectivityCh >> eck': 'true'}) {} >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> according to configuration: networks:{'ovirtmgmt': {'vlan': '4000', >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> 'switch': 'legacy'}}, bondings:{}, options:{'connectivityTimeout': >> 120, 'commitOnSuccess': >> True, 'connectivityCheck': 'true'} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc show (cwd >> None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge >> -- list Port -- list Interface >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> --timeout=5 --oneline --format=json -- list Bridge -- list Port -- >> list Interface (cwd None) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = b''; >> <rc> = 0 >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': 'none', >> 'nameservers': ['213.133.98.98']}) >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:25,958::commands::153::common.commands::(start) /usr/bin/taskset >> --cpu-list 0-7 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> (cwd None) >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b'' >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,155::configurator::195::root::(_setup_nmstate) Processing >> setup through nmstate >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired state: >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> 'bridge': {'port': [ >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': False}}}, >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': False}}], >> 'dns-resolver': {'config' >> : {'server': ['213.133.98.98']}}} >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60 >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> interfaces: ['ovirtmgmt'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> interfaces: ['enp4s0.4000', 'enp4s0'] >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,224::nmclient::136::root::(execute_next_action) Executing NM >> action: func=add_connection_async >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> Connection adding succeeded: dev=ovirtmgmt >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,232::nmclient::136::root::(execute_next_action) Executing NM >> action: func=commit_changes_async >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> Connection update failed: error=nm-connection-error-quark: >> ipv6.gateway: gateway cannot be set if there are no addresses >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of type >> NM.DeviceState> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> dbus.UInt32(0), >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> Error in setupNetworks >> Traceback (most recent call last): >> File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> line 95, in wrapper >> res = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 241, in setupNetworks >> _setup_networks(networks, bondings, options, net_info) >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line >> 266, in _setup_networks >> networks, bondings, options, net_info, in_rollback >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 154, in setup >> _setup_nmstate(networks, bondings, options, in_rollback) >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> line 199, in _setup_nmstate >> nmstate.setup(desired_state, verify_change=not in_rollback) >> File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> line 63, in setup >> state_apply(desired_state, verify_change=verify_change) >> File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> line 40, in wrapper >> return func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 96, in wrapped >> ret = func(*args, **kwargs) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 73, in apply >> state.State(desired_state), verify_change, commit, rollback_timeout >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 163, in _apply_ifaces_state >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__ >> next(self.gen) >> File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> line 232, in _setup_providers >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> line 177, in run >> f"Unexpected failure of libnm when running the mainloop: {err}" >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when >> running the mainloop: run execution >> >> My host have a main network interface and a vlan interface on the same >> physical interface which is an internal management network which >> should be used as management network. The vlan interface doesn't have >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. This >> seems like a bug? >> >> /Sverker >> >> _______________________________________________ >> discuss mailing list >> discuss@openvswitch.org <mailto:discuss@openvswitch.org> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> _______________________________________________ >> Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > _______________________________________________ > Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D...
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
On Mon, Sep 7, 2020 at 1:16 PM Sverker Abrahamsson <sverker@abrahamsson.com> wrote:
It is static configured:
[root@h1-mgmt network-scripts]# cat ifcfg-enp4s0 NAME=enp4s0 DEVICE=enp4s0 ONBOOT=yes UUID=af7ccb53-011b-4c36-998a-1878b4ae7100 TYPE=Ethernet BOOTPROTO=none IPADDR=144.76.84.73 PREFIX=32 GATEWAY=144.76.84.65 DEFROUTE=yes DNS1=213.133.98.98
IPV6INIT=yes IPV6ADDR=2a01:4f8:192:1148::2/64 IPV6_DEFAULTGW=fe80::1 IPV6_DEFROUTE=yes
ZONE=public PROXY_METHOD=none BROWSER_ONLY=no IPV4_FAILURE_FATAL=no IPV6_AUTOCONF=yes IPV6_FAILURE_FATAL=no
Any reason to have dynamic ipv6 enabled along with static address?
[root@h1-mgmt network-scripts]# cat ifcfg-enp4s0.4000 NAME=enp4s0.4000 DEVICE=enp4s0.4000 ONBOOT=yes VLAN=yes BOOTPROTO=none IPADDR=172.27.1.1 PREFIX=24 ZONE=internal IPV6INIT=no TYPE=Vlan PHYSDEV=enp4s0 VLAN_ID=4000 REORDER_HDR=yes GVRP=no MVRP=no HWADDR= PROXY_METHOD=none BROWSER_ONLY=no DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6_DISABLED=yes UUID=ecc8064d-18c1-99b7-3fe4-9c5a593ece6f
I think that what would happen if it didn't fail due to ipv6 route is that
enp4s0 would be without any ip's, based on what is logged as the desired state by vdsm: { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config': { 'server': ['213.133.98.98'] } }
The desired state should be something like this (although I don't know the
correct names for the missing keys and there may be ): { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '144.76.84.73', 'prefix-length': 32, 'gateway': '144.76.84.65' } ], 'dhcp': False }, 'ipv6': { 'enabled': True, 'address': [{ 'ip': '2a01:4f8:192:1148::2', 'prefix-length': 64, 'gateway': 'fe80::1' } } }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config': { 'server': ['213.133.98.98'] } } }
I.e. issue is that vdsm assumes that the underlying enp4s0 interface should not have any addresses when it should maintain all settings it has before. It's only the addresses of enp4s0.4000 that should move to the ovirtmgmt bridge. Alternatively enp4s0 should not get involved at all in the desired state, only enp4s0.4000
That's not the case. In the desired state there is no removal of address on enp4s0 at all. We are requesting 2 things on enp4s0: 1) Set the interface up 2) Set the MTU to 1500 Everything else should be kept as is. That means nmstate will take what is requested and what is configured and merge those states which should result only in change of those two things mentioned. There are two things that come to my mind currently that might be wrong. First is the definition of the static and dynamic. This might actually result in flush of ip address on down/up interface cycle. The second is that enp4s0 has a default route but ovirtmgmt tries to be the default route as well.
/Sverker Den 2020-09-07 kl. 12:54, skrev Dominik Holler:
On Mon, Sep 7, 2020 at 12:22 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Unforturnatly it is a blocker, I have not found a way to work around it.
Is there a change in behavior, if IPv6 on the non-vlanned interface (enp4s0) is configured in a static way?
Den 2020-09-04 kl. 09:47, skrev Dominik Holler:
Sverker, is this bug blocking you, or can you work around it?
On Thu, Sep 3, 2020 at 8:52 PM Dominik Holler <dholler@redhat.com> wrote:
Sverker, thanks!
On Thu, Sep 3, 2020 at 6:50 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson < sverker@abrahamsson.com> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users < users@ovirt.org> wrote:
> Hi Dominik > That is my issue, I don't get to where I can get the ovirtmgmt > bridge established because vdsm insists on creating it. It used to be > possible to create that bridge statically and vdsm would just skip it but > seems to be broken now. > > If it would be possible to use OVN for the management network that > would solve my issue and would be the preferable solution, but as you write > that isn't possible which was what I suspected. > > Do you have any other suggestion on how to solve this issue? That I > get the external interface untagged and the internal network tagged is not > possible to change. > > /Sverker >
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
> Den 2020-09-03 kl. 10:52, skrev Dominik Holler: > > > > On Wed, Sep 2, 2020 at 10:38 PM Sverker Abrahamsson via Users < > users@ovirt.org> wrote: > >> Well, unforturnatly I don't have a choise since it is out of my >> control. >> I only have one physical network port where the external traffic is >> untagged and the internal vlan is tagged. If I could run with OVN >> > > OVN is for VM traffic only, not usable for the management network. > > >> instead I wouldn't need that tagged vlan, but I haven't been able >> to get >> that to work neither. >> >> > Please let us know if OVN does not work for VM traffic for you. > > >> It's perfectly possible to have both tagged and untagged traffic on >> the >> same switch port, issue is that vdsm tries to take control over the >> network without being able to be flexible enough.. I'm attempting >> now to >> have ovirtmgmt bridge created before, that used to be possible but >> according to previous mails on the list it went broken somewhere at >> 4.x. >> >> /Sverker >> >> Den 2020-09-02 kl. 21:39, skrev Strahil Nikolov: >> > Switchports can either be tagged or untagged. >> > I'm not sure that your setup is supported at all. >> > >> > Best Regards, >> > Strahil Nikolov >> > >> > >> > >> > >> > >> > >> > В сряда, 2 септември 2020 г., 20:41:57 Гринуич+3, Sverker >> Abrahamsson via Users <users@ovirt.org> написа: >> > >> > >> > >> > >> > >> > Pretty formatting the "desired state" it seems that vdsm tries to >> remove >> > the ip of my underlying interface, that is enp4s0: >> > > > > { >> > 'interfaces': [{ >> > 'name': 'enp4s0', >> > 'state': 'up', >> > 'mtu': 1500 >> > }, { >> > 'vlan': { >> > 'id': 4000, >> > 'base-iface': 'enp4s0' >> > }, >> > 'name': 'enp4s0.4000', >> > 'type': 'vlan', >> > 'state': 'up', >> > 'mtu': 1500, >> > 'ipv4': { >> > 'enabled': False >> > }, >> > 'ipv6': { >> > 'enabled': False >> > } >> > }, { >> > 'name': 'ovirtmgmt', >> > 'type': 'linux-bridge', >> > 'state': 'up', >> > 'mtu': 1500, >> > 'bridge': { >> > 'port': [{ >> > 'name': 'enp4s0.4000' >> > } >> > ], >> > 'options': { >> > 'stp': { >> > 'enabled': False >> > } >> > } >> > }, >> > 'ipv4': { >> > 'enabled': True, >> > 'address': [{ >> > 'ip': '172.27.1.1', >> > 'prefix-length': 24 >> > } >> > ], >> > 'dhcp': False >> > }, >> > 'ipv6': { >> > 'enabled': False >> > } >> > } >> > ], >> > 'dns-resolver': { >> > 'config' >> > : { >> > 'server': ['213.133.98.98'] >> > } >> > } >> > } >> > >> > > Thanks, this is helpful information. > Can you please share the getCapabilities result sent from vdsm to > Engine directly before the setupNetworks request, > and the parameters of the setupNetworks request from Engine to vdsm? > Both are in the vdsm.log during adding the host. > > >> > >> > This is my interfaces before vdsm attemtpts to change the config: >> > >> > enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> > inet 144.76.84.73 netmask 255.255.255.255 broadcast >> 0.0.0.0 >> > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid >> 0x20<link> >> > inet6 2a01:4f8:192:1148::2 prefixlen 64 scopeid >> 0x0<global> >> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) >> > RX packets 293442 bytes 385541799 (367.6 MiB) >> > RX errors 0 dropped 0 overruns 0 frame 0 >> > TX packets 91095 bytes 31160348 (29.7 MiB) >> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions >> 0 >> > device interrupt 17 memory 0xf7d00000-f7d20000 >> > >> > enp4s0.4000: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> > inet 172.27.1.1 netmask 255.255.255.0 broadcast >> 172.27.1.255 >> > inet6 fe80::62a4:4cff:fee9:4ac prefixlen 64 scopeid >> 0x20<link> >> > ether 60:a4:4c:e9:04:ac txqueuelen 1000 (Ethernet) >> > RX packets 0 bytes 0 (0.0 B) >> > RX errors 0 dropped 0 overruns 0 frame 0 >> > TX packets 13 bytes 938 (938.0 B) >> > TX errors 0 dropped 0 overruns 0 carrier 0 collisions >> 0 >> > >> > I.e. enp4s0 is the external interface that must not be changed, >> bridge >> > must be created on the vlan interface. I would prefer to create >> the >> > bridge manually and not through vdsm if that is possible. >> > >> > /Sverker >> > >> > Den 2020-09-02 kl. 19:14, skrev Sverker Abrahamsson via Users: >> >> Hi, >> >> I'm attempting to install hosted engine but getting this failure: >> >> >> >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, >> "msg": >> >> "The host has been set in non_operational status, deployment >> errors: >> >> code 505: Host h1-mgmt.limetransit.com installation failed. >> Failed to >> >> configure management network on the host., code 1120: Failed >> to >> >> configure management network on host h1-mgmt.limetransit.com >> due to >> >> setup networks failure., code 9000: Failed to verify Power >> Management >> >> configuration for Host h1-mgmt.limetransit.com., code 10802: >> VDSM >> >> h1-mgmt.limetransit.com command HostSetupNetworksVDS failed: >> Internal >> >> JSON-RPC error: {'reason': 'Unexpected failure of libnm when >> running >> >> the mainloop: run execution'}, fix accordingly and re-deploy."} >> >> >> >> Looking in vdsm.log I find this which I believe is the root >> cause: >> >> >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> >> 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) >> >> call setupNetworks with ({'ovirtmgmt': {'vlan': '4000', >> 'netmask': >> >> '255.255.255.0', 'ipv6autoconf': False, >> >> 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> >> 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, >> >> 'commitOnSuccess': True, 'connectivityCh >> >> eck': 'true'}) {} >> >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> >> 16:38:25,897::api::220::root::(setupNetworks) Setting up network >> >> according to configuration: networks:{'ovirtmgmt': {'vlan': >> '4000', >> >> 'netmask': '255.255.255.0', 'ipv6autoconf': Fal >> >> se, 'nic': 'enp4s0', 'bridged': 'true', 'ipaddr': '172.27.1.1', >> >> 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, >> >> 'switch': 'legacy'}}, bondings:{}, >> options:{'connectivityTimeout': >> >> 120, 'commitOnSuccess': >> >> True, 'connectivityCheck': 'true'} >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:25,902::cmdutils::130::root::(exec_cmd) /sbin/tc qdisc >> show (cwd >> >> None) >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:25,906::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = >> b''; >> >> <rc> = 0 >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:25,945::vsctl::74::root::(commit) Executing commands: >> >> /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list >> Bridge >> >> -- list Port -- list Interface >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:25,945::cmdutils::130::root::(exec_cmd) /usr/bin/ovs-vsctl >> >> --timeout=5 --oneline --format=json -- list Bridge -- list Port >> -- >> >> list Interface (cwd None) >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:25,952::cmdutils::138::root::(exec_cmd) SUCCESS: <err> = >> b''; >> >> <rc> = 0 >> >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> >> 16:38:25,957::netconfpersistence::58::root::(setNetwork) Adding >> >> network ovirtmgmt({'vlan': 4000, 'netmask': '255.255.255.0', >> >> 'ipv6autoconf': False, 'nic': 'enp4s0', 'bridged': True >> >> , 'ipaddr': '172.27.1.1', 'defaultRoute': True, 'dhcpv6': False, >> >> 'mtu': 1500, 'switch': 'legacy', 'stp': False, 'bootproto': >> 'none', >> >> 'nameservers': ['213.133.98.98']}) >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:25,958::commands::153::common.commands::(start) >> /usr/bin/taskset >> >> --cpu-list 0-7 >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe >> >> (cwd None) >> >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> >> 16:38:26,154::hooks::122::root::(_runHooksDir) >> >> /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 >> err=b'' >> >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> >> 16:38:26,155::configurator::195::root::(_setup_nmstate) >> Processing >> >> setup through nmstate >> >> MainProcess|jsonrpc/0::INFO::2020-09-02 >> >> 16:38:26,175::configurator::197::root::(_setup_nmstate) Desired >> state: >> >> {'interfaces': [{'name': 'enp4s0', 'state': 'up', 'mtu': 1500}, >> >> {'vlan': {'id': 4000, 'base-iface': 'enp4s0'} >> >> , 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': >> 1500, >> >> 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, {'name': >> >> 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, >> >> 'bridge': {'port': [ >> >> {'name': 'enp4s0.4000'}], 'options': {'stp': {'enabled': >> False}}}, >> >> 'ipv4': {'enabled': True, 'address': [{'ip': '172.27.1.1', >> >> 'prefix-length': 24}], 'dhcp': False}, 'ipv6': {'enabled': >> False}}], >> >> 'dns-resolver': {'config' >> >> : {'server': ['213.133.98.98']}}} >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:26,217::checkpoint::121::root::(create) Checkpoint >> >> /org/freedesktop/NetworkManager/Checkpoint/1 created for all >> devices: 60 >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:26,218::netapplier::239::root::(_add_interfaces) Adding new >> >> interfaces: ['ovirtmgmt'] >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:26,221::netapplier::251::root::(_edit_interfaces) Editing >> >> interfaces: ['enp4s0.4000', 'enp4s0'] >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:26,224::nmclient::136::root::(execute_next_action) >> Executing NM >> >> action: func=add_connection_async >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:26,231::connection::329::root::(_add_connection_callback) >> >> Connection adding succeeded: dev=ovirtmgmt >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:26,232::nmclient::136::root::(execute_next_action) >> Executing NM >> >> action: func=commit_changes_async >> >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> >> 16:38:26,233::nmclient::200::root::(quit) NM main-loop aborted: >> >> Connection update failed: error=nm-connection-error-quark: >> >> ipv6.gateway: gateway cannot be set if there are no addresses >> >> configured (7), dev=enp4s0/<enum NM_DEVICE_STATE_ACTIVATED of >> type >> >> NM.DeviceState> >> >> MainProcess|jsonrpc/0::DEBUG::2020-09-02 >> >> 16:38:26,246::checkpoint::164::root::(rollback) Checkpoint >> >> /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed: >> >> >> dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/7'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/8'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/9'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/3'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/2'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/4'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/1'): >> >> dbus.UInt32(0), >> >> dbus.String('/org/freedesktop/NetworkManager/Devices/5'): >> >> dbus.UInt32(0)}, signature=dbus.Signature('su')) >> >> MainProcess|jsonrpc/0::ERROR::2020-09-02 >> >> >> 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) >> >> Error in setupNetworks >> >> Traceback (most recent call last): >> >> File >> "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", >> >> line 95, in wrapper >> >> res = func(*args, **kwargs) >> >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", >> line >> >> 241, in setupNetworks >> >> _setup_networks(networks, bondings, options, net_info) >> >> File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", >> line >> >> 266, in _setup_networks >> >> networks, bondings, options, net_info, in_rollback >> >> File >> >> >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> >> line 154, in setup >> >> _setup_nmstate(networks, bondings, options, in_rollback) >> >> File >> >> >> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", >> >> line 199, in _setup_nmstate >> >> nmstate.setup(desired_state, verify_change=not in_rollback) >> >> File >> "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", >> >> line 63, in setup >> >> state_apply(desired_state, verify_change=verify_change) >> >> File >> "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", >> >> line 40, in wrapper >> >> return func(*args, **kwargs) >> >> File >> "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> >> line 96, in wrapped >> >> ret = func(*args, **kwargs) >> >> File >> "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> >> line 73, in apply >> >> state.State(desired_state), verify_change, commit, >> rollback_timeout >> >> File >> "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> >> line 163, in _apply_ifaces_state >> >> con_profiles=ifaces_add_configs + ifaces_edit_configs, >> >> File "/usr/lib64/python3.6/contextlib.py", line 88, in >> __exit__ >> >> next(self.gen) >> >> File >> "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", >> >> line 232, in _setup_providers >> >> mainloop.run(timeout=MAINLOOP_TIMEOUT) >> >> File >> "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", >> >> line 177, in run >> >> f"Unexpected failure of libnm when running the mainloop: >> {err}" >> >> libnmstate.error.NmstateLibnmError: Unexpected failure of libnm >> when >> >> running the mainloop: run execution >> >> >> >> My host have a main network interface and a vlan interface on >> the same >> >> physical interface which is an internal management network which >> >> should be used as management network. The vlan interface doesn't >> have >> >> any ipv6 interface, but nmclient fails on setting ipv6 gateway. >> This >> >> seems like a bug? >> >> >> >> /Sverker >> >> >> >> _______________________________________________ >> >> discuss mailing list >> >> discuss@openvswitch.org >> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >> >> _______________________________________________ >> >> Users mailing list -- users@ovirt.org >> >> To unsubscribe send an email to users-leave@ovirt.org >> >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> >> oVirt Code of Conduct: >> >> https://www.ovirt.org/community/about/community-guidelines/ >> >> List Archives: >> >> >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... >> > _______________________________________________ >> > Users mailing list -- users@ovirt.org >> > To unsubscribe send an email to users-leave@ovirt.org >> > Privacy Statement: https://www.ovirt.org/privacy-policy.html >> > oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> > List Archives: >> > >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... >> > >> _______________________________________________ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D... >> > _______________________________________________ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO... >
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com IM: amusil <https://red.ht/sig>
-- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil@redhat.com IM: amusil <https://red.ht/sig>
Ales, that was the issue. The config below was as installed by the hosting center, and rewritten by nmcli. I had not touched the ipv6 part. After your input below I looked in Cockpit, and removed the dynamic ipv6. I was able to run the wizard to prepare VM which successfully created the bridge but then failed on something else during the last step of the deploy although the engine is running.. Den 2020-09-08 kl. 07:42, skrev Ales Musil:
On Mon, Sep 7, 2020 at 1:16 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
It is static configured:
[root@h1-mgmt network-scripts]# cat ifcfg-enp4s0 NAME=enp4s0 DEVICE=enp4s0 ONBOOT=yes UUID=af7ccb53-011b-4c36-998a-1878b4ae7100 TYPE=Ethernet BOOTPROTO=none IPADDR=144.76.84.73 PREFIX=32 GATEWAY=144.76.84.65 DEFROUTE=yes DNS1=213.133.98.98
IPV6INIT=yes IPV6ADDR=2a01:4f8:192:1148::2/64 IPV6_DEFAULTGW=fe80::1 IPV6_DEFROUTE=yes
ZONE=public PROXY_METHOD=none BROWSER_ONLY=no IPV4_FAILURE_FATAL=no IPV6_AUTOCONF=yes IPV6_FAILURE_FATAL=no
Any reason to have dynamic ipv6 enabled along with static address?
[root@h1-mgmt network-scripts]# cat ifcfg-enp4s0.4000 NAME=enp4s0.4000 DEVICE=enp4s0.4000 ONBOOT=yes VLAN=yes BOOTPROTO=none IPADDR=172.27.1.1 PREFIX=24 ZONE=internal IPV6INIT=no TYPE=Vlan PHYSDEV=enp4s0 VLAN_ID=4000 REORDER_HDR=yes GVRP=no MVRP=no HWADDR= PROXY_METHOD=none BROWSER_ONLY=no DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6_DISABLED=yes UUID=ecc8064d-18c1-99b7-3fe4-9c5a593ece6f
I think that what would happen if it didn't fail due to ipv6 route is that enp4s0 would be without any ip's, based on what is logged as the desired state by vdsm: { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500 }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config': { 'server': ['213.133.98.98'] } }
The desired state should be something like this (although I don't know the correct names for the missing keys and there may be ): { 'interfaces': [{ 'name': 'enp4s0', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '144.76.84.73', 'prefix-length': 32, 'gateway': '144.76.84.65' } ], 'dhcp': False }, 'ipv6': { 'enabled': True, 'address': [{ 'ip': '2a01:4f8:192:1148::2', 'prefix-length': 64, 'gateway': 'fe80::1' } } }, { 'vlan': { 'id': 4000, 'base-iface': 'enp4s0' }, 'name': 'enp4s0.4000', 'type': 'vlan', 'state': 'up', 'mtu': 1500, 'ipv4': { 'enabled': False }, 'ipv6': { 'enabled': False } }, { 'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': { 'port': [{ 'name': 'enp4s0.4000' } ], 'options': { 'stp': { 'enabled': False } } }, 'ipv4': { 'enabled': True, 'address': [{ 'ip': '172.27.1.1', 'prefix-length': 24 } ], 'dhcp': False }, 'ipv6': { 'enabled': False } } ], 'dns-resolver': { 'config': { 'server': ['213.133.98.98'] } } }
I.e. issue is that vdsm assumes that the underlying enp4s0 interface should not have any addresses when it should maintain all settings it has before. It's only the addresses of enp4s0.4000 that should move to the ovirtmgmt bridge. Alternatively enp4s0 should not get involved at all in the desired state, only enp4s0.4000
That's not the case. In the desired state there is no removal of address on enp4s0 at all. We are requesting 2 things on enp4s0: 1) Set the interface up 2) Set the MTU to 1500
Everything else should be kept as is. That means nmstate will take what is requested and what is configured and merge those states which should result only in change of those two things mentioned.
There are two things that come to my mind currently that might be wrong. First is the definition of the static and dynamic. This might actually result in flush of ip address on down/up interface cycle. The second is that enp4s0 has a default route but ovirtmgmt tries to be the default route as well.
/Sverker
Den 2020-09-07 kl. 12:54, skrev Dominik Holler:
On Mon, Sep 7, 2020 at 12:22 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Unforturnatly it is a blocker, I have not found a way to work around it.
Is there a change in behavior, if IPv6 on the non-vlanned interface (enp4s0) is configured in a static way?
Den 2020-09-04 kl. 09:47, skrev Dominik Holler:
Sverker, is this bug blocking you, or can you work around it?
On Thu, Sep 3, 2020 at 8:52 PM Dominik Holler <dholler@redhat.com <mailto:dholler@redhat.com>> wrote:
Sverker, thanks!
On Thu, Sep 3, 2020 at 6:50 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Dominik, bug filed at https://bugzilla.redhat.com/show_bug.cgi?id=1875520. I'm doing a new install to get fresh vdsm and supervdsm logs which will be attached as soon as they've failed. /Sverker
Den 2020-09-03 kl. 18:03, skrev Dominik Holler:
On Thu, Sep 3, 2020 at 12:42 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, this is a CentOS 8 so my impression was that you always have NetworkManager then? At least my attempt to remove it failed miserably.
Yes, on CentOS 8 hosts oVirt requires the interfaces managed by NetworkManager.
The enp4s0 config was created by the install, so it should be controlled by NetworkManager.
This should work. Can you please report a bug on vdsm [1]? Would be helpful if the vdsm.log and supervdsm.log would be attached to this bug.
[1] https://bugzilla.redhat.com/enter_bug.cgi?product=vdsm
/Sverker
Den 2020-09-03 kl. 12:29, skrev Ales Musil:
On Thu, Sep 3, 2020 at 12:21 PM Sverker Abrahamsson <sverker@abrahamsson.com <mailto:sverker@abrahamsson.com>> wrote:
Hi Ales, right now I have a manually created ovirtmgmt bridge (virbr0 and vnet0 seems to be created during the failed attempt to deploy hosted engine):
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 Bridge ovirtmgmt 9a0b07c0-2983-fe97-ec7f-ad2b51c3a3f0 bridge ovirtmgmt virbr0 aa593151-2c12-4cf7-985b-f105b3575d09 bridge virbr0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 vnet0 a6db45bd-93c8-4c37-85fc-0c58ba3e9d00 tun vnet0 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 425 next-hop-address: '' next-hop-interface: ovirtmgmt table-id: 254 - destination: 192.168.1.0/24 <http://192.168.1.0/24> metric: 0 next-hop-address: '' next-hop-interface: virbr0 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: DE:D3:A8:24:27:F6 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: dhcp: false enabled: false ipv6: enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovirtmgmt type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: false forward-delay: 15 hello-time: 2 max-age: 20 priority: 32768 port: - name: enp4s0.4000 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: D2:D3:5B:BA:58:38 mtu: 1500 - name: virbr0 type: linux-bridge state: up bridge: options: group-forward-mask: 0 mac-ageing-time: 300 multicast-snooping: true stp: enabled: true forward-delay: 2 hello-time: 2 max-age: 20 priority: 32768 port: - name: virbr0-nic stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 - name: vnet0 stp-hairpin-mode: false stp-path-cost: 100 stp-priority: 32 ipv4: address: - ip: 192.168.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: virbr0-nic type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 52:54:00:C0:A5:20 mtu: 1500 - name: vnet0 type: unknown state: up ipv4: enabled: false ipv6: enabled: false mac-address: FE:16:3E:1B:21:0B mtu: 1500
Reverting the configuration back to not have the static bridge and a reboot I get this:
[root@h1-mgmt ~]# nmcli con show NAME UUID TYPE DEVICE enp4s0 af7ccb53-011b-4c36-998a-1878b4ae7100 ethernet enp4s0 enp4s0.4000 ecc8064d-18c1-99b7-3fe4-9c5a593ece6f vlan enp4s0.4000 [root@h1-mgmt ~]# nmstatectl show --- dns-resolver: config: search: [] server: - 213.133.98.98 running: search: [] server: - 213.133.98.98 route-rules: config: [] routes: config: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: -1 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 0 - destination: ::/0 metric: -1 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 0 running: - destination: 0.0.0.0/0 <http://0.0.0.0/0> metric: 100 next-hop-address: 144.76.84.65 next-hop-interface: enp4s0 table-id: 254 - destination: 144.76.84.65/32 <http://144.76.84.65/32> metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: 172.27.1.0/24 <http://172.27.1.0/24> metric: 400 next-hop-address: '' next-hop-interface: enp4s0.4000 table-id: 254 - destination: 2a01:4f8:192:1148::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ::/0 metric: 100 next-hop-address: fe80::1 next-hop-interface: enp4s0 table-id: 254 - destination: fe80::/64 metric: 100 next-hop-address: '' next-hop-interface: enp4s0 table-id: 254 - destination: ff00::/8 metric: 256 next-hop-address: '' next-hop-interface: enp4s0 table-id: 255 interfaces: - name: ;vdsmdummy; type: linux-bridge state: down ipv4: enabled: false ipv6: enabled: false mac-address: B2:9E:E0:61:71:88 mtu: 1500 - name: br-int type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: 6E:37:94:63:E0:4B mtu: 1500 - name: enp4s0 type: ethernet state: up ethernet: auto-negotiation: true duplex: full speed: 1000 ipv4: address: - ip: 144.76.84.73 prefix-length: 32 dhcp: false enabled: true ipv6: address: - ip: 2a01:4f8:192:1148::2 prefix-length: 64 - ip: fe80::62a4:4cff:fee9:4ac prefix-length: 64 auto-dns: true auto-gateway: true auto-routes: true autoconf: true dhcp: true enabled: true mac-address: 60:A4:4C:E9:04:AC mtu: 1500 - name: enp4s0.4000 type: vlan state: up ipv4: address: - ip: 172.27.1.1 prefix-length: 24 dhcp: false enabled: true ipv6: autoconf: false dhcp: false enabled: false mac-address: 60:A4:4C:E9:04:AC mtu: 1500 vlan: base-iface: enp4s0 id: 4000 - name: lo type: unknown state: down ipv4: enabled: false ipv6: enabled: false mtu: 65536 - name: ovs-system type: unknown state: down ipv4: enabled: false ipv6: enabled: false mac-address: A2:35:7A:6C:B7:EF mtu: 1500
/Sverker
Was the enp4s0 always managed by NetworkManager or only after the attempt to make the ovirtmgmt? If not that would explain the failure. Also the workaround would be to configure the interface via NetworkManager and then run the host deploy again.
Thanks, Ales
Den 2020-09-03 kl. 11:54, skrev Ales Musil:
On Thu, Sep 3, 2020 at 11:51 AM Sverker Abrahamsson via Users <users@ovirt.org <mailto:users@ovirt.org>> wrote:
Hi Dominik That is my issue, I don't get to where I can get the ovirtmgmt bridge established because vdsm insists on creating it. It used to be possible to create that bridge statically and vdsm would just skip it but seems to be broken now.
If it would be possible to use OVN for the management network that would solve my issue and would be the preferable solution, but as you write that isn't possible which was what I suspected.
Do you have any other suggestion on how to solve this issue? That I get the external interface untagged and the internal network tagged is not possible to change.
/Sverker
Hello Sverker,
can you please share output from "nmcli con show" and "nmstatectl show"?
Thank you. Regards, Ales
Den 2020-09-03 kl. 10:52, skrev Dominik Holler: > > > On Wed, Sep 2, 2020 at 10:38 PM > Sverker Abrahamsson via Users > <users@ovirt.org > <mailto:users@ovirt.org>> wrote: > > Well, unforturnatly I don't have > a choise since it is out of my > control. > I only have one physical network > port where the external traffic is > untagged and the internal vlan > is tagged. If I could run with OVN > > > OVN is for VM traffic only, not > usable for the management network. > > instead I wouldn't need that > tagged vlan, but I haven't been > able to get > that to work neither. > > > Please let us know if OVN does not > work for VM traffic for you. > > It's perfectly possible to have > both tagged and untagged traffic > on the > same switch port, issue is that > vdsm tries to take control over the > network without being able to be > flexible enough.. I'm attempting > now to > have ovirtmgmt bridge created > before, that used to be possible > but > according to previous mails on > the list it went broken > somewhere at 4.x. > > /Sverker > > Den 2020-09-02 kl. 21:39, skrev > Strahil Nikolov: > > Switchports can either be > tagged or untagged. > > I'm not sure that your setup > is supported at all. > > > > Best Regards, > > Strahil Nikolov > > > > > > > > > > > > > > В сряда, 2 септември 2020 г., > 20:41:57 Гринуич+3, Sverker > Abrahamsson via Users > <users@ovirt.org > <mailto:users@ovirt.org>> написа: > > > > > > > > > > > > Pretty formatting the "desired > state" it seems that vdsm tries > to remove > > the ip of my underlying > interface, that is enp4s0: > > > > > { > > 'interfaces': [{ > > 'name': 'enp4s0', > > 'state': 'up', > > 'mtu': 1500 > > }, { > > 'vlan': { > > 'id': 4000, > > 'base-iface': 'enp4s0' > > }, > > 'name': 'enp4s0.4000', > > 'type': 'vlan', > > 'state': 'up', > > 'mtu': 1500, > > 'ipv4': { > > 'enabled': False > > }, > > 'ipv6': { > > 'enabled': False > > } > > }, { > > 'name': 'ovirtmgmt', > > 'type': 'linux-bridge', > > 'state': 'up', > > 'mtu': 1500, > > 'bridge': { > > 'port': [{ > > 'name': 'enp4s0.4000' > > } > > ], > > 'options': { > > 'stp': { > > 'enabled': False > > } > > } > > }, > > 'ipv4': { > > 'enabled': True, > > 'address': [{ > > 'ip': '172.27.1.1', > > 'prefix-length': 24 > > } > > ], > > 'dhcp': False > > }, > > 'ipv6': { > > 'enabled': False > > } > > } > > ], > > 'dns-resolver': { > > 'config' > > : { > > 'server': ['213.133.98.98'] > > } > > } > > } > > > > > Thanks, this is helpful information. > Can you please share the > getCapabilities result sent from > vdsm to Engine directly before the > setupNetworks request, > and the parameters of the > setupNetworks request from Engine to > vdsm? > Both are in the vdsm.log during > adding the host. > > > > > This is my interfaces before > vdsm attemtpts to change the config: > > > > enp4s0: > flags=4163<UP,BROADCAST,RUNNING,MULTICAST> > mtu 1500 > > inet 144.76.84.73 netmask > 255.255.255.255 broadcast 0.0.0.0 > > inet6 fe80::62a4:4cff:fee9:4ac > prefixlen 64 scopeid 0x20<link> > > inet6 2a01:4f8:192:1148::2 > prefixlen 64 scopeid 0x0<global> > > ether 60:a4:4c:e9:04:ac > txqueuelen 1000 (Ethernet) > > RX packets 293442 bytes > 385541799 (367.6 MiB) > > RX errors 0 dropped 0 overruns > 0 frame 0 > > TX packets 91095 bytes > 31160348 (29.7 MiB) > > TX errors 0 dropped 0 overruns > 0 carrier 0 collisions 0 > > device interrupt 17 memory > 0xf7d00000-f7d20000 > > > > enp4s0.4000: > flags=4163<UP,BROADCAST,RUNNING,MULTICAST> > mtu 1500 > > inet 172.27.1.1 netmask > 255.255.255.0 broadcast 172.27.1.255 > > inet6 fe80::62a4:4cff:fee9:4ac > prefixlen 64 scopeid 0x20<link> > > ether 60:a4:4c:e9:04:ac > txqueuelen 1000 (Ethernet) > > RX packets 0 bytes 0 (0.0 B) > > RX errors 0 dropped 0 overruns > 0 frame 0 > > TX packets 13 bytes 938 (938.0 B) > > TX errors 0 dropped 0 overruns > 0 carrier 0 collisions 0 > > > > I.e. enp4s0 is the external > interface that must not be > changed, bridge > > must be created on the vlan > interface. I would prefer to > create the > > bridge manually and not > through vdsm if that is possible. > > > > /Sverker > > > > Den 2020-09-02 kl. 19:14, > skrev Sverker Abrahamsson via Users: > >> Hi, > >> I'm attempting to install > hosted engine but getting this > failure: > >> > >> [ ERROR ] fatal: [localhost]: > FAILED! => {"changed": false, "msg": > >> "The host has been set in > non_operational status, > deployment errors: > >> code 505: Host > h1-mgmt.limetransit.com > <http://h1-mgmt.limetransit.com> > installation failed. Failed to > >> configure management network > on the host., code 1120: > Failed to > >> configure management network > on host h1-mgmt.limetransit.com > <http://h1-mgmt.limetransit.com> > due to > >> setup networks failure., code > 9000: Failed to verify Power > Management > >> configuration for Host > h1-mgmt.limetransit.com > <http://h1-mgmt.limetransit.com>., > code 10802: VDSM > >> h1-mgmt.limetransit.com > <http://h1-mgmt.limetransit.com> > command HostSetupNetworksVDS > failed: Internal > >> JSON-RPC error: {'reason': > 'Unexpected failure of libnm > when running > >> the mainloop: run > execution'}, fix accordingly and > re-deploy."} > >> > >> Looking in vdsm.log I find > this which I believe is the root > cause: > >> > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,897::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) > >> call setupNetworks with > ({'ovirtmgmt': {'vlan': '4000', > 'netmask': > >> '255.255.255.0', > 'ipv6autoconf': False, > >> 'nic': 'enp4s0', 'bridged': > 'true', 'ipaddr': '172.27.1.1', > >> 'defaultRoute': True, > 'dhcpv6': False, 'STP': 'no', > 'mtu': 1500, > >> 'switch': 'legacy'}}, {}, > {'connectivityTimeout': 120, > >> 'commitOnSuccess': True, > 'connectivityCh > >> eck': 'true'}) {} > >> > MainProcess|jsonrpc/0::INFO::2020-09-02 > >> > 16:38:25,897::api::220::root::(setupNetworks) > Setting up network > >> according to configuration: > networks:{'ovirtmgmt': {'vlan': > '4000', > >> 'netmask': '255.255.255.0', > 'ipv6autoconf': Fal > >> se, 'nic': 'enp4s0', > 'bridged': 'true', 'ipaddr': > '172.27.1.1', > >> 'defaultRoute': True, > 'dhcpv6': False, 'STP': 'no', > 'mtu': 1500, > >> 'switch': 'legacy'}}, > bondings:{}, > options:{'connectivityTimeout': > >> 120, 'commitOnSuccess': > >> True, 'connectivityCheck': > 'true'} > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,902::cmdutils::130::root::(exec_cmd) > /sbin/tc qdisc show (cwd > >> None) > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,906::cmdutils::138::root::(exec_cmd) > SUCCESS: <err> = b''; > >> <rc> = 0 > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,945::vsctl::74::root::(commit) > Executing commands: > >> /usr/bin/ovs-vsctl > --timeout=5 --oneline > --format=json -- list Bridge > >> -- list Port -- list Interface > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,945::cmdutils::130::root::(exec_cmd) > /usr/bin/ovs-vsctl > >> --timeout=5 --oneline > --format=json -- list Bridge -- > list Port -- > >> list Interface (cwd None) > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,952::cmdutils::138::root::(exec_cmd) > SUCCESS: <err> = b''; > >> <rc> = 0 > >> > MainProcess|jsonrpc/0::INFO::2020-09-02 > >> > 16:38:25,957::netconfpersistence::58::root::(setNetwork) > Adding > >> network ovirtmgmt({'vlan': > 4000, 'netmask': '255.255.255.0', > >> 'ipv6autoconf': False, 'nic': > 'enp4s0', 'bridged': True > >> , 'ipaddr': '172.27.1.1', > 'defaultRoute': True, 'dhcpv6': > False, > >> 'mtu': 1500, 'switch': > 'legacy', 'stp': False, > 'bootproto': 'none', > >> 'nameservers': > ['213.133.98.98']}) > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:25,958::commands::153::common.commands::(start) > /usr/bin/taskset > >> --cpu-list 0-7 > /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe > >> (cwd None) > >> > MainProcess|jsonrpc/0::INFO::2020-09-02 > >> > 16:38:26,154::hooks::122::root::(_runHooksDir) > >> > /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: > rc=0 err=b'' > >> > MainProcess|jsonrpc/0::INFO::2020-09-02 > >> > 16:38:26,155::configurator::195::root::(_setup_nmstate) > Processing > >> setup through nmstate > >> > MainProcess|jsonrpc/0::INFO::2020-09-02 > >> > 16:38:26,175::configurator::197::root::(_setup_nmstate) > Desired state: > >> {'interfaces': [{'name': > 'enp4s0', 'state': 'up', 'mtu': > 1500}, > >> {'vlan': {'id': 4000, > 'base-iface': 'enp4s0'} > >> , 'name': 'enp4s0.4000', > 'type': 'vlan', 'state': 'up', > 'mtu': 1500, > >> 'ipv4': {'enabled': False}, > 'ipv6': {'enabled': False}}, > {'name': > >> 'ovirtmgmt', 'type': > 'linux-bridge', 'state': 'up', > 'mtu': 1500, > >> 'bridge': {'port': [ > >> {'name': 'enp4s0.4000'}], > 'options': {'stp': {'enabled': > False}}}, > >> 'ipv4': {'enabled': True, > 'address': [{'ip': '172.27.1.1', > >> 'prefix-length': 24}], > 'dhcp': False}, 'ipv6': > {'enabled': False}}], > >> 'dns-resolver': {'config' > >> : {'server': ['213.133.98.98']}}} > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:26,217::checkpoint::121::root::(create) > Checkpoint > >> > /org/freedesktop/NetworkManager/Checkpoint/1 > created for all devices: 60 > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:26,218::netapplier::239::root::(_add_interfaces) > Adding new > >> interfaces: ['ovirtmgmt'] > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:26,221::netapplier::251::root::(_edit_interfaces) > Editing > >> interfaces: ['enp4s0.4000', > 'enp4s0'] > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:26,224::nmclient::136::root::(execute_next_action) > Executing NM > >> action: func=add_connection_async > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:26,231::connection::329::root::(_add_connection_callback) > >> Connection adding succeeded: > dev=ovirtmgmt > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:26,232::nmclient::136::root::(execute_next_action) > Executing NM > >> action: func=commit_changes_async > >> > MainProcess|jsonrpc/0::ERROR::2020-09-02 > >> > 16:38:26,233::nmclient::200::root::(quit) > NM main-loop aborted: > >> Connection update failed: > error=nm-connection-error-quark: > >> ipv6.gateway: gateway cannot > be set if there are no addresses > >> configured (7), > dev=enp4s0/<enum > NM_DEVICE_STATE_ACTIVATED of type > >> NM.DeviceState> > >> > MainProcess|jsonrpc/0::DEBUG::2020-09-02 > >> > 16:38:26,246::checkpoint::164::root::(rollback) > Checkpoint > >> > /org/freedesktop/NetworkManager/Checkpoint/1 > rollback executed: > >> > dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/6'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/7'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/8'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/9'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/3'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/2'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/4'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/1'): > >> dbus.UInt32(0), > >> > dbus.String('/org/freedesktop/NetworkManager/Devices/5'): > >> dbus.UInt32(0)}, > signature=dbus.Signature('su')) > >> > MainProcess|jsonrpc/0::ERROR::2020-09-02 > >> > 16:38:31,251::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) > >> Error in setupNetworks > >> Traceback (most recent call > last): > >> File > "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", > >> line 95, in wrapper > >> res = func(*args, **kwargs) > >> File > "/usr/lib/python3.6/site-packages/vdsm/network/api.py", > line > >> 241, in setupNetworks > >> _setup_networks(networks, > bondings, options, net_info) > >> File > "/usr/lib/python3.6/site-packages/vdsm/network/api.py", > line > >> 266, in _setup_networks > >> networks, bondings, options, > net_info, in_rollback > >> File > >> > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > >> line 154, in setup > >> _setup_nmstate(networks, > bondings, options, in_rollback) > >> File > >> > "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", > >> line 199, in _setup_nmstate > >> nmstate.setup(desired_state, > verify_change=not in_rollback) > >> File > "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", > >> line 63, in setup > >> state_apply(desired_state, > verify_change=verify_change) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", > >> line 40, in wrapper > >> return func(*args, **kwargs) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > >> line 96, in wrapped > >> ret = func(*args, **kwargs) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > >> line 73, in apply > >> state.State(desired_state), > verify_change, commit, > rollback_timeout > >> File > "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > >> line 163, in _apply_ifaces_state > >> > con_profiles=ifaces_add_configs > + ifaces_edit_configs, > >> File > "/usr/lib64/python3.6/contextlib.py", > line 88, in __exit__ > >> next(self.gen) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", > >> line 232, in _setup_providers > >> > mainloop.run(timeout=MAINLOOP_TIMEOUT) > >> File > "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", > >> line 177, in run > >> f"Unexpected failure of libnm > when running the mainloop: {err}" > >> > libnmstate.error.NmstateLibnmError: > Unexpected failure of libnm when > >> running the mainloop: run > execution > >> > >> My host have a main network > interface and a vlan interface > on the same > >> physical interface which is > an internal management network which > >> should be used as management > network. The vlan interface > doesn't have > >> any ipv6 interface, but > nmclient fails on setting ipv6 > gateway. This > >> seems like a bug? > >> > >> /Sverker > >> > >> > _______________________________________________ > >> discuss mailing list > >> discuss@openvswitch.org > <mailto:discuss@openvswitch.org> > >> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > >> > _______________________________________________ > >> Users mailing list -- > users@ovirt.org > <mailto:users@ovirt.org> > >> To unsubscribe send an email > to users-leave@ovirt.org > <mailto:users-leave@ovirt.org> > >> Privacy Statement: > https://www.ovirt.org/privacy-policy.html > >> oVirt Code of Conduct: > >> > https://www.ovirt.org/community/about/community-guidelines/ > >> List Archives: > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/R4YSTOQM7I4XLC... > > > _______________________________________________ > > Users mailing list -- > users@ovirt.org > <mailto:users@ovirt.org> > > To unsubscribe send an email > to users-leave@ovirt.org > <mailto:users-leave@ovirt.org> > > Privacy Statement: > https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/UMXFPVLJHYOO3U... > > > _______________________________________________ > Users mailing list -- > users@ovirt.org > <mailto:users@ovirt.org> > To unsubscribe send an email to > users-leave@ovirt.org > <mailto:users-leave@ovirt.org> > Privacy Statement: > https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/RQIPLZGVOHXU5D... > _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ED2YKQAKVG66JO...
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil@redhat.com <mailto:amusil@redhat.com> IM: amusil
participants (4)
-
Ales Musil -
Dominik Holler -
Strahil Nikolov -
Sverker Abrahamsson