1:16 p.m.
Hello,
what is the current status both if using plain CentOS based nodes and
ovirt-node-ng?
Do the release of CentOS 8.2 impact new installation for 4.4.0 and/or
4.4.1rc?
Thanks,
Gianluca
4:01 p.m.
Hi,
newer builds (the 4.4.1 build sandro sent just now) use 8.2 and require 8.2.
If you’d upgrade your existing 4.4.0/4.4.1 host to 8.2 it may not necessarily work,
openvswitch issues might break. We were not testing it, CentOS releases are always without
any heads up.
I would suggest to do it together with upgrading to 4.4.1 rc5, but certainly not on a
production setupu just yet
Thanks,
michal
6:09 p.m.
Hi Michal,
Might the openvswitch issues you mention be tied to the issue I'm having standing up a
new 4.4 installation on 8.2, namely that you can create a network in the UI, but when you
go to apply/attach it to a host (using the drag-and-drop wizard to try to add a new
network onto the same bond ovirtmgmt uses), it fails? So I can now install a new
hosted-engine setup on 8.2 with EPYC CPUs, definitely forward progress, but I can't
configure any networks beyond ovirtmgmt. Dragging a new network onto the bond and clicking
"OK" gets "Error while executing action HostSetupNetworks: Unexpected
exception".
Thanks,
Mark
6:11 p.m.
The error with a bit more info from the events page, after adding network to an interface
fails:
VDSM rack4slot11.domain.com command HostSetupNetworksVDS failed: Internal JSON-RPC error:
{'reason': 'Unexpected failure of libnm when running the mainloop: run
execution'}
Sorry, should have included that in the other message.
Mark
9:49 p.m.
Hello Mark,
can you please share the relevant lines of supervdsm.log from the host?
Helpful to understand the intended change are the lines starting with the
relevant "call setupNetworks",
including the line containing "Desired state"
until some lines below "Unexpected failure of libnm when running the
mainloop"
The most recent line containing "return network_caps with" is helpful to
understand the initial state before the error occurred.
Thanks
Dominik
On Fri, Jun 19, 2020 at 6:16 PM Mark R <ovirtlist(a)beanz.33mail.com> wrote:
12:02 a.m.
Hi Dominik,
My apologies for the delay, have been pushed to other projects this week. I just verified
that our CentOS 8.2 and oVirt 4.4.0 installs are fully up to date, and tried to apply a
network configuration adding a tagged VLAN to the same bridge as ovirtmgmt. Here are the
logs:
```
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:53:59,832::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call setupNetworks
with ({'DMZ0': {'vlan': '22', 'bonding': 'bond0',
'ipv6autoconf': False, 'bridged': 'true', 'dhcpv6': False,
'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}},
{}, {'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck': 'true'}) {}
MainProcess|jsonrpc/3::INFO::2020-06-23 21:53:59,833::api::220::root::(setupNetworks)
Setting up network according to configuration: networks:{'DMZ0': {'vlan':
'22', 'bonding': 'bond0', 'ipv6autoconf': False,
'bridged': 'true', 'dhcpv6': False, 'STP': 'no',
'mtu': 1500, 'switch': 'legacy'}}, bondings:{},
options:{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck': 'true'}
MainProcess|jsonrpc/3::DEBUG::2020-06-23 21:53:59,844::cmdutils::130::root::(exec_cmd)
/sbin/tc qdisc show (cwd None)
MainProcess|jsonrpc/3::DEBUG::2020-06-23 21:53:59,852::cmdutils::138::root::(exec_cmd)
SUCCESS: <err> = b''; <rc> = 0
MainProcess|jsonrpc/3::DEBUG::2020-06-23 21:53:59,899::vsctl::74::root::(commit) Executing
commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list
Port -- list Interface
MainProcess|jsonrpc/3::DEBUG::2020-06-23 21:53:59,899::cmdutils::130::root::(exec_cmd)
/usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list
Interface (cwd None)
MainProcess|jsonrpc/3::DEBUG::2020-06-23 21:53:59,908::cmdutils::138::root::(exec_cmd)
SUCCESS: <err> = b''; <rc> = 0
MainProcess|jsonrpc/3::INFO::2020-06-23
21:53:59,913::netconfpersistence::58::root::(setNetwork) Adding network
ovirtmgmt({'bridged': True, 'stp': False, 'mtu': 1500,
'bonding': 'bond0', 'defaultRoute': True, 'bootproto':
'none', 'dhcpv6': True, 'ipv6autoconf': True, 'ipaddr':
'172.17.96.51', 'netmask': '255.255.255.0', 'gateway':
'172.17.96.1', 'switch': 'legacy', 'nameservers':
['x.x.x.x <scrubbed>', 'x.x.x.x <scrubbed>']})
MainProcess|jsonrpc/3::INFO::2020-06-23
21:53:59,914::netconfpersistence::69::root::(setBonding) Adding bond0({'nics':
['eno33np0', 'ens2f0np0'], 'options': 'mode=4',
'switch': 'legacy', 'hwaddr': 'bc:97:e1:24:c5:40'})
MainProcess|jsonrpc/3::INFO::2020-06-23
21:53:59,914::netconfpersistence::58::root::(setNetwork) Adding network
DMZ0({'vlan': 22, 'bonding': 'bond0', 'ipv6autoconf':
False, 'bridged': True, 'dhcpv6': False, 'mtu': 1500,
'switch': 'legacy', 'defaultRoute': False, 'stp': False,
'bootproto': 'none', 'nameservers': []})
MainProcess|jsonrpc/3::INFO::2020-06-23
21:53:59,914::netconfpersistence::69::root::(setBonding) Adding bond0({'nics':
['eno33np0', 'ens2f0np0'], 'options': 'mode=4',
'switch': 'legacy', 'hwaddr': 'bc:97:e1:24:c5:40'})
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:53:59,916::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-63
/usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None)
MainProcess|jsonrpc/3::INFO::2020-06-23 21:54:00,271::hooks::122::root::(_runHooksDir)
/usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=b''
MainProcess|jsonrpc/3::INFO::2020-06-23
21:54:00,271::configurator::190::root::(_setup_nmstate) Processing setup through nmstate
MainProcess|jsonrpc/3::INFO::2020-06-23
21:54:00,297::configurator::192::root::(_setup_nmstate) Desired state:
{'interfaces': [{'name': 'DMZ0', 'type':
'linux-bridge', 'state': 'up', 'mtu': 1500,
'bridge': {'port': [{'name': 'bond0.22'}],
'options': {'stp': {'enabled': False}}}, 'ipv4':
{'enabled': False}, 'ipv6': {'enabled': False}}, {'name':
'bond0', 'type': 'bond', 'state': 'up',
'mac-address': 'bc:97:e1:24:c5:40', 'link-aggregation':
{'slaves': ['eno33np0', 'ens2f0np0'], 'options': {},
'mode': '802.3ad'}, 'mtu': 1500}, {'vlan': {'id':
22, 'base-iface': 'bond0'}, 'name': 'bond0.22',
'type': 'vlan', 'state': 'up', 'mtu': 1500,
'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}},
{'name': 'ovirtmgmt'}]}
MainProcess|jsonrpc/3::DEBUG::2020-06-23 21:54:00,351::checkpoint::121::root::(create)
Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 created for all devices: 60
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,351::netapplier::239::root::(_add_interfaces) Adding new interfaces:
['DMZ0', 'bond0.22']
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,354::netapplier::251::root::(_edit_interfaces) Editing interfaces:
['eno33np0', 'ovirtmgmt', 'bond0', 'ens2f0np0']
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,357::nmclient::136::root::(execute_next_action) Executing NM action:
func=add_connection_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,364::connection::329::root::(_add_connection_callback) Connection adding
succeeded: dev=DMZ0
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,364::nmclient::136::root::(execute_next_action) Executing NM action:
func=add_connection_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,370::connection::329::root::(_add_connection_callback) Connection adding
succeeded: dev=bond0.22
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,370::nmclient::136::root::(execute_next_action) Executing NM action:
func=commit_changes_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,374::connection::386::root::(_commit_changes_callback) Connection update
succeeded: dev=bond0
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,374::nmclient::136::root::(execute_next_action) Executing NM action:
func=commit_changes_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,377::connection::386::root::(_commit_changes_callback) Connection update
succeeded: dev=eno33np0
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,377::nmclient::136::root::(execute_next_action) Executing NM action:
func=commit_changes_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,379::connection::386::root::(_commit_changes_callback) Connection update
succeeded: dev=ens2f0np0
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,379::nmclient::136::root::(execute_next_action) Executing NM action:
func=commit_changes_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,382::connection::386::root::(_commit_changes_callback) Connection update
succeeded: dev=ovirtmgmt
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,382::nmclient::136::root::(execute_next_action) Executing NM action:
func=safe_activate_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,385::connection::215::root::(_active_connection_callback) Connection activation
initiated: dev=DMZ0, con-state=<enum NM_ACTIVE_CONNECTION_STATE_ACTIVATING of type
NM.ActiveConnectionState>
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,423::connection::301::root::(_waitfor_active_connection_callback) Connection
activation succeeded: dev=DMZ0, con-state=<enum NM_ACTIVE_CONNECTION_STATE_ACTIVATING
of type NM.ActiveConnectionState>, dev-state=<enum NM_DEVICE_STATE_IP_CONFIG of type
NM.DeviceState>, state-flags=<flags NM_ACTIVATION_STATE_FLAG_IS_MASTER |
NM_ACTIVATION_STATE_FLAG_LAYER2_READY | NM_ACTIVATION_STATE_FLAG_MASTER_HAS_SLAVES of type
NM.ActivationStateFlags>
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,424::nmclient::136::root::(execute_next_action) Executing NM action:
func=_safe_modify_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,426::device::149::root::(_modify_callback) Device reapply succeeded:
dev=ovirtmgmt
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,426::nmclient::136::root::(execute_next_action) Executing NM action:
func=_safe_modify_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,429::device::149::root::(_modify_callback) Device reapply succeeded: dev=bond0
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,429::nmclient::136::root::(execute_next_action) Executing NM action:
func=_safe_modify_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,430::device::149::root::(_modify_callback) Device reapply succeeded:
dev=ens2f0np0
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,430::nmclient::136::root::(execute_next_action) Executing NM action:
func=_safe_modify_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,431::device::149::root::(_modify_callback) Device reapply succeeded:
dev=eno33np0
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,431::nmclient::136::root::(execute_next_action) Executing NM action:
func=safe_activate_async
MainProcess|jsonrpc/3::DEBUG::2020-06-23
21:54:00,432::connection::215::root::(_active_connection_callback) Connection activation
initiated: dev=bond0.22, con-state=<enum NM_ACTIVE_CONNECTION_STATE_ACTIVATING of type
NM.ActiveConnectionState>
MainProcess|jsonrpc/3::ERROR::2020-06-23 21:54:00,466::nmclient::200::root::(quit) NM
main-loop aborted: Connection activation failed on bond0.22: reason=<enum
NM_ACTIVE_CONNECTION_STATE_REASON_DEVICE_DISCONNECTED of type
NM.ActiveConnectionStateReason>
MainProcess|jsonrpc/3::DEBUG::2020-06-23 21:54:00,490::checkpoint::164::root::(rollback)
Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed:
dbus.Dictionary({dbus.String('/org/freedesktop/NetworkManager/Devices/19'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/3'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/10'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/11'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/1'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/9'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/5'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/7'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/6'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/18'):
dbus.UInt32(0), dbus.String('/org/freedesktop/NetworkManager/Devices/8'):
dbus.UInt32(0),
dbus.String('/org/freedesktop/NetworkManager/Devices/2'): dbus.UInt32(0),
dbus.String('/org/freedesktop/NetworkManager/Devices/4'): dbus.UInt32(0)},
signature=dbus.Signature('su'))
MainProcess|jsonrpc/3::ERROR::2020-06-23
21:54:05,497::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in
setupNetworks
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in
wrapper
res = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in
setupNetworks
_setup_networks(networks, bondings, options, net_info)
File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in
_setup_networks
networks, bondings, options, net_info, in_rollback
File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line
154, in setup
_setup_nmstate(networks, bondings, options, in_rollback)
File
"/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line
194, in _setup_nmstate
nmstate.setup(desired_state, verify_change=not in_rollback)
File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate.py", line 63, in
setup
state_apply(desired_state, verify_change=verify_change)
File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in
wrapper
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in
wrapped
ret = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in
apply
state.State(desired_state), verify_change, commit, rollback_timeout
File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 163, in
_apply_ifaces_state
con_profiles=ifaces_add_configs + ifaces_edit_configs,
File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
next(self.gen)
File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 232, in
_setup_providers
mainloop.run(timeout=MAINLOOP_TIMEOUT)
File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 177,
in run
f"Unexpected failure of libnm when running the mainloop: {err}"
libnmstate.error.NmstateLibnmError: Unexpected failure of libnm when running the mainloop:
run execution
```
Regards,
Mark
12:09 a.m.
Follow up, as well as the logged error from supervdsm.log, I see this in dmesg so it
appears that it very briefly does create DMZ0 and the bond0.22 interface, but rolls
everything back.
```
[ 2205.551920] IPv6: ADDRCONF(NETDEV_UP): DMZ0: link is not ready
[ 2205.558366] IPv6: ADDRCONF(NETDEV_UP): bond0.22: link is not ready
[ 2205.655177] DMZ0: port 1(bond0.22) entered blocking state
[ 2205.655179] DMZ0: port 1(bond0.22) entered disabled state
```
Both ports for the bond are up and active (of couse, as this is currently the first host
of the HE deployment so I couldn't manage it if they weren't).
```
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: bc:97:e1:24:c5:40
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 21
Partner Key: 3
Partner Mac Address: 0a:33:5e:69:1f:1e
Slave Interface: eno33np0
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: bc:97:e1:24:c5:40
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: bc:97:e1:24:c5:40
port key: 21
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 4096
system mac address: 0a:33:5e:69:1f:1e
oper key: 3
port priority: 8192
port number: 65
port state: 61
Slave Interface: ens2f0np0
MII Status: up
Speed: 25000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b0:26:28:cd:ec:d0
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: bc:97:e1:24:c5:40
port key: 21
port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 4096
system mac address: 0a:33:5e:69:1f:1e
oper key: 3
port priority: 32768
port number: 33
port state: 61
```
Thanks,
Mark
9:01 a.m.
On Wed, Jun 24, 2020 at 12:17 AM Mark R <ovirtlist(a)beanz.33mail.com> wrote:
Hello,
sorry for the late response. We are failing to reproduce this issue. Can
you please provide us with more info?
I need to know an exact package version of nmstate and NetworkManager.
Also if possible please enable trace logs for NetworkManager:
Please create `/etc/NetworkManager/conf.d/debug.conf` with following
content:
[logging]
level=TRACE
domains=ALL
Restart NetworkManager service and trigger the issue again.
The NM log is available via journalctl -u NetworkManager.
Thank you.
Best regards,
Ales Musil
--
Ales Musil
Software Engineer - RHV Network
Red Hat EMEA <https://www.redhat.com>
amusil(a)redhat.com IM: amusil
<https://red.ht/sig>
12:15 a.m.
Hello Ales,
The nmstate and NetworkManager versions are as follows, installed from repos that are
added while installing oVirt/HE:
Name : nmstate
Version : 0.2.10
Release : 1.el8
Architecture : noarch
Size : 32 k
Repository : @System
From repo : ovirt-4.4-copr:copr.fedorainfracloud.org:nmstate:nmstate-0.2
Name : NetworkManager
Version : 1.22.14
Release : 1.el8
Architecture : x86_64
Repository : @System
From repo :
ovirt-4.4-copr:copr.fedorainfracloud.org:networkmanager:NetworkManager-1.22
The NM debug output is 2K lines (I only included logs from the point I clicked
"OK" to apply the new network until it failed). I pastebin'ed it at
https://controlc.com/f3a08a1b , use pass ovirt44 to view it. I didn't want to throw
that much content into a forum post and can't add an attachment when using the
list's web interface to post.
Regards,
Mark
1743
days inactive
1754
days old
8 comments
5 participants
participants (5)
-
Ales Musil -
Dominik Holler -
Gianluca Cecchi -
Mark R -
Michal Skrivanek