ovirt 4.2.2-rc3 switching from legacy to OVS..
I'm getting further along with 4.2.2rc3 than the 4.2.1 when it comes to hosted engine and vlans.. it actually does install under 4.2.2rc3. But it's a complete failure when I switch the cluster from Linux Bridge/Legacy to OVS. The first time I try, vdsm does not properly configure the node, it's all messed up. I'm getting this in vdsmd logs: 2018-03-08 23:12:46,610-0800 INFO (jsonrpc/7) [api.network] START setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True, u'nic': u'eno1', u'vlan': u'50', u'ipaddr': u'192.168.85.49', u'switch': u'ovs', u'mtu': 1500, u'netmask': u'255.255.252.0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'gateway': u'192.168.85.254', u'defaultRoute': True}}, bondings={}, options={u'connectivityCheck': u'true', u'connectivityTimeout': 120}) from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:46) 2018-03-08 23:12:52,449-0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-03-08 23:12:52,511-0800 INFO (jsonrpc/7) [api.network] FINISH setupNetworks error=[Errno 19] ovirtmgmt is not present in the system from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:50) 2018-03-08 23:12:52,512-0800 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in setupNetworks File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1527, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system 2018-03-08 23:12:52,512-0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error -32603) in 5.90 seconds (__init__:573) 2018-03-08 23:12:54,769-0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-03-08 23:12:54,772-0800 INFO (jsonrpc/5) [api.host] START getCapabilities() from=::1,45562 (api:46) 2018-03-08 23:12:54,906-0800 INFO (jsonrpc/5) [api.host] FINISH getCapabilities error=[Errno 19] ovirtmgmt is not present in the system from=::1,45562 (api:50) 2018-03-08 23:12:54,906-0800 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 168, in get net_caps = supervdsm.getProxy().network_caps() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in network_caps File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system So something is dreadfully wrong with the bridge to ovs conversion in 4.2.2rc3. thomas
OVS switch support is experimental at this stage and in some cases when trying to change from one switch to the other, it fails. It was also not checked against a hosted engine setup, which handles networking a bit differently for the management network (ovirtmgmt). Nevertheless, we are interested in understanding all the problems that exists today, so if you can, please share the supervdsm log, it has the interesting networking traces. We plan to block cluster switch editing until these problems are resolved. It will be only allowed to define a new cluster as OVS, not convert an existing one from Linux Bridge to OVS. On Fri, Mar 9, 2018 at 9:54 AM, Thomas Davis <tadavis@lbl.gov> wrote:
I'm getting further along with 4.2.2rc3 than the 4.2.1 when it comes to hosted engine and vlans.. it actually does install under 4.2.2rc3.
But it's a complete failure when I switch the cluster from Linux Bridge/Legacy to OVS. The first time I try, vdsm does not properly configure the node, it's all messed up.
I'm getting this in vdsmd logs:
2018-03-08 23:12:46,610-0800 INFO (jsonrpc/7) [api.network] START setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True, u'nic': u'eno1', u'vlan': u'50', u'ipaddr': u'192.168.85.49', u'switch': u'ovs', u'mtu': 1500, u'netmask': u'255.255.252.0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'gateway': u'192.168.85.254', u'defaultRoute': True}}, bondings={}, options={u'connectivityCheck': u'true', u'connectivityTimeout': 120}) from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:46)
2018-03-08 23:12:52,449-0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)
2018-03-08 23:12:52,511-0800 INFO (jsonrpc/7) [api.network] FINISH setupNetworks error=[Errno 19] ovirtmgmt is not present in the system from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:50) 2018-03-08 23:12:52,512-0800 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in setupNetworks File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1527, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system 2018-03-08 23:12:52,512-0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error -32603) in 5.90 seconds (__init__:573) 2018-03-08 23:12:54,769-0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-03-08 23:12:54,772-0800 INFO (jsonrpc/5) [api.host] START getCapabilities() from=::1,45562 (api:46) 2018-03-08 23:12:54,906-0800 INFO (jsonrpc/5) [api.host] FINISH getCapabilities error=[Errno 19] ovirtmgmt is not present in the system from=::1,45562 (api:50) 2018-03-08 23:12:54,906-0800 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 168, in get net_caps = supervdsm.getProxy().network_caps() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in network_caps File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system
So something is dreadfully wrong with the bridge to ovs conversion in 4.2.2rc3.
thomas _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I'll work on it some more. I have 2 different clusters in the data center (1 is the Hosted Engine systems, another is not..) I had trouble with both. I'll try again on the non-hosted engine cluster to see what it is doing. I have it working in 4.1, but we are trying to do a clean wipe since the 4.1 engine has been upgraded so many times from v3.5 plus we want to move to hosted-engine-ha from a single engine node and the ansible modules/roles (which also have problems..) thomas On Tue, Mar 13, 2018 at 6:27 AM, Edward Haas <ehaas@redhat.com> wrote:
OVS switch support is experimental at this stage and in some cases when trying to change from one switch to the other, it fails. It was also not checked against a hosted engine setup, which handles networking a bit differently for the management network (ovirtmgmt). Nevertheless, we are interested in understanding all the problems that exists today, so if you can, please share the supervdsm log, it has the interesting networking traces.
We plan to block cluster switch editing until these problems are resolved. It will be only allowed to define a new cluster as OVS, not convert an existing one from Linux Bridge to OVS.
On Fri, Mar 9, 2018 at 9:54 AM, Thomas Davis <tadavis@lbl.gov> wrote:
I'm getting further along with 4.2.2rc3 than the 4.2.1 when it comes to hosted engine and vlans.. it actually does install under 4.2.2rc3.
But it's a complete failure when I switch the cluster from Linux Bridge/Legacy to OVS. The first time I try, vdsm does not properly configure the node, it's all messed up.
I'm getting this in vdsmd logs:
2018-03-08 23:12:46,610-0800 INFO (jsonrpc/7) [api.network] START setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True, u'nic': u'eno1', u'vlan': u'50', u'ipaddr': u'192.168.85.49', u'switch': u'ovs', u'mtu': 1500, u'netmask': u'255.255.252.0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'gateway': u'192.168.85.254', u'defaultRoute': True}}, bondings={}, options={u'connectivityCheck': u'true', u'connectivityTimeout': 120}) from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:46)
2018-03-08 23:12:52,449-0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)
2018-03-08 23:12:52,511-0800 INFO (jsonrpc/7) [api.network] FINISH setupNetworks error=[Errno 19] ovirtmgmt is not present in the system from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:50) 2018-03-08 23:12:52,512-0800 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in setupNetworks File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1527, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system 2018-03-08 23:12:52,512-0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error -32603) in 5.90 seconds (__init__:573) 2018-03-08 23:12:54,769-0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-03-08 23:12:54,772-0800 INFO (jsonrpc/5) [api.host] START getCapabilities() from=::1,45562 (api:46) 2018-03-08 23:12:54,906-0800 INFO (jsonrpc/5) [api.host] FINISH getCapabilities error=[Errno 19] ovirtmgmt is not present in the system from=::1,45562 (api:50) 2018-03-08 23:12:54,906-0800 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 168, in get net_caps = supervdsm.getProxy().network_caps() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in network_caps File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system
So something is dreadfully wrong with the bridge to ovs conversion in 4.2.2rc3.
thomas _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well, I just hit https://bugzilla.redhat.com/show_bug.cgi?id=1513991 And it's been closed, which means with vdsm-4.20.17-1.el7.centos.x86_64 OVS networking is totally borked.. I know OVS is Experimental, but it worked in 4.1.x, and now we have to do a step back to legacy bridge just to use 4.2.x, which in a vlan environment just wreaks havoc (every VLAN need's a unique mac assigned to the bridge, which vdsm does not do, so suddenly you get the kernel complaining about seeing it's mac address several times.) There is zero documentation on how to use OVN instead of OVS. thomas On 03/13/2018 09:22 AM, Thomas Davis wrote:
I'll work on it some more. I have 2 different clusters in the data center (1 is the Hosted Engine systems, another is not..) I had trouble with both. I'll try again on the non-hosted engine cluster to see what it is doing. I have it working in 4.1, but we are trying to do a clean wipe since the 4.1 engine has been upgraded so many times from v3.5 plus we want to move to hosted-engine-ha from a single engine node and the ansible modules/roles (which also have problems..)
thomas
On Tue, Mar 13, 2018 at 6:27 AM, Edward Haas <ehaas@redhat.com <mailto:ehaas@redhat.com>> wrote:
OVS switch support is experimental at this stage and in some cases when trying to change from one switch to the other, it fails. It was also not checked against a hosted engine setup, which handles networking a bit differently for the management network (ovirtmgmt). Nevertheless, we are interested in understanding all the problems that exists today, so if you can, please share the supervdsm log, it has the interesting networking traces.
We plan to block cluster switch editing until these problems are resolved. It will be only allowed to define a new cluster as OVS, not convert an existing one from Linux Bridge to OVS.
On Fri, Mar 9, 2018 at 9:54 AM, Thomas Davis <tadavis@lbl.gov <mailto:tadavis@lbl.gov>> wrote:
I'm getting further along with 4.2.2rc3 than the 4.2.1 when it comes to hosted engine and vlans.. it actually does install under 4.2.2rc3.
But it's a complete failure when I switch the cluster from Linux Bridge/Legacy to OVS. The first time I try, vdsm does not properly configure the node, it's all messed up.
I'm getting this in vdsmd logs:
2018-03-08 23:12:46,610-0800 INFO (jsonrpc/7) [api.network] START setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True, u'nic': u'eno1', u'vlan': u'50', u'ipaddr': u'192.168.85.49', u'switch': u'ovs', u'mtu': 1500, u'netmask': u'255.255.252.0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'gateway': u'192.168.85.254', u'defaultRoute': True}}, bondings={}, options={u'connectivityCheck': u'true', u'connectivityTimeout': 120}) from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:46)
2018-03-08 23:12:52,449-0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)
2018-03-08 23:12:52,511-0800 INFO (jsonrpc/7) [api.network] FINISH setupNetworks error=[Errno 19] ovirtmgmt is not present in the system from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:50) 2018-03-08 23:12:52,512-0800 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in setupNetworks File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1527, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system 2018-03-08 23:12:52,512-0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error -32603) in 5.90 seconds (__init__:573) 2018-03-08 23:12:54,769-0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-03-08 23:12:54,772-0800 INFO (jsonrpc/5) [api.host] START getCapabilities() from=::1,45562 (api:46) 2018-03-08 23:12:54,906-0800 INFO (jsonrpc/5) [api.host] FINISH getCapabilities error=[Errno 19] ovirtmgmt is not present in the system from=::1,45562 (api:50) 2018-03-08 23:12:54,906-0800 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 168, in get net_caps = supervdsm.getProxy().network_caps() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in network_caps File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system
So something is dreadfully wrong with the bridge to ovs conversion in 4.2.2rc3.
thomas _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
On Thu, Mar 15, 2018 at 1:50 AM, Thomas Davis <tadavis@lbl.gov> wrote:
Well, I just hit
https://bugzilla.redhat.com/show_bug.cgi?id=1513991
And it's been closed, which means with vdsm-4.20.17-1.el7.centos.x86_64 OVS networking is totally borked..
You are welcome to reopen that bug, specifying your use case for OvS. I cannot promise fixing this bug, as our resources are limited, and that bug, which was introduced in 4.2, was not deemed as urgently needed. https://gerrit.ovirt.org/#/c/86932/ attempts to fix the bug, but it still needs a lot of work.
I know OVS is Experimental, but it worked in 4.1.x, and now we have to do a step back to legacy bridge just to use 4.2.x, which in a vlan environment just wreaks havoc (every VLAN need's a unique mac assigned to the bridge, which vdsm does not do, so suddenly you get the kernel complaining about seeing it's mac address several times.)
Could you elaborate on this issue? What is wrong with a bridge that learns its mac from its underlying device? What wold like Vdsm to do, in your opinion? You can file a bug (or even send a patch) if there is a functionality that you'd like to fix.
There is zero documentation on how to use OVN instead of OVS.
I hope that https://ovirt.org/develop/release-management/features/network/provider-physi... can help.
thomas
On 03/13/2018 09:22 AM, Thomas Davis wrote:
I'll work on it some more. I have 2 different clusters in the data center (1 is the Hosted Engine systems, another is not..) I had trouble with both. I'll try again on the non-hosted engine cluster to see what it is doing. I have it working in 4.1, but we are trying to do a clean wipe since the 4.1 engine has been upgraded so many times from v3.5 plus we want to move to hosted-engine-ha from a single engine node and the ansible modules/roles (which also have problems..)
thomas
On Tue, Mar 13, 2018 at 6:27 AM, Edward Haas <ehaas@redhat.com <mailto:ehaas@redhat.com>> wrote:
OVS switch support is experimental at this stage and in some cases when trying to change from one switch to the other, it fails. It was also not checked against a hosted engine setup, which handles networking a bit differently for the management network (ovirtmgmt). Nevertheless, we are interested in understanding all the problems that exists today, so if you can, please share the supervdsm log, it has the interesting networking traces.
We plan to block cluster switch editing until these problems are resolved. It will be only allowed to define a new cluster as OVS, not convert an existing one from Linux Bridge to OVS.
On Fri, Mar 9, 2018 at 9:54 AM, Thomas Davis <tadavis@lbl.gov <mailto:tadavis@lbl.gov>> wrote:
I'm getting further along with 4.2.2rc3 than the 4.2.1 when it comes to hosted engine and vlans.. it actually does install under 4.2.2rc3.
But it's a complete failure when I switch the cluster from Linux Bridge/Legacy to OVS. The first time I try, vdsm does not properly configure the node, it's all messed up.
I'm getting this in vdsmd logs:
2018-03-08 23:12:46,610-0800 INFO (jsonrpc/7) [api.network] START setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True, u'nic': u'eno1', u'vlan': u'50', u'ipaddr': u'192.168.85.49', u'switch': u'ovs', u'mtu': 1500, u'netmask': u'255.255.252.0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'gateway': u'192.168.85.254', u'defaultRoute': True}}, bondings={}, options={u'connectivityCheck': u'true', u'connectivityTimeout': 120}) from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:46)
2018-03-08 23:12:52,449-0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)
2018-03-08 23:12:52,511-0800 INFO (jsonrpc/7) [api.network] FINISH setupNetworks error=[Errno 19] ovirtmgmt is not present in the system from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:50) 2018-03-08 23:12:52,512-0800 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in setupNetworks File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1527, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system 2018-03-08 23:12:52,512-0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error -32603) in 5.90 seconds (__init__:573) 2018-03-08 23:12:54,769-0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-03-08 23:12:54,772-0800 INFO (jsonrpc/5) [api.host] START getCapabilities() from=::1,45562 (api:46) 2018-03-08 23:12:54,906-0800 INFO (jsonrpc/5) [api.host] FINISH getCapabilities error=[Errno 19] ovirtmgmt is not present in the system from=::1,45562 (api:50) 2018-03-08 23:12:54,906-0800 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 168, in get net_caps = supervdsm.getProxy().network_caps() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in network_caps File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system
So something is dreadfully wrong with the bridge to ovs conversion in 4.2.2rc3.
thomas _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Make sure STP is OFF for each bridge. Then the warnings go away. Cheers, Robert O'Kane On 03/15/2018 11:21 AM, Dan Kenigsberg wrote:
I know OVS is Experimental, but it worked in 4.1.x, and now we have to do a step back to legacy bridge just to use 4.2.x, which in a vlan environment just wreaks havoc (every VLAN need's a unique mac assigned to the bridge, which vdsm does not do, so suddenly you get the kernel complaining about seeing it's mac address several times.) Could you elaborate on this issue? What is wrong with a bridge that learns its mac from its underlying device? What wold like Vdsm to do, in your opinion? You can file a bug (or even send a patch) if there is a functionality that you'd like to fix.
-- Robert O'Kane Systems Administrator Kunsthochschule für Medien Köln Peter-Welter-Platz 2 50676 Köln fon: +49(221)20189-223 fax: +49(221)20189-49223
Alrighty, I figured it out. 0) To setup a node in a cluster, make sure the cluster is in OVS, not legacy. 1) Make sure you have an OVN controller setup somewhere. Default appears to be the ovirt-hosted-engine. a) you should also have the external network provider for OVN configured also; see the web interface. 2) when you install the node, make sure it has openvswitch installed and running - ie: a) 'systemctl status openvswitch' says it's up and running. (be sure it's enable also) b) 'ovs-vsctl show' has vdsm bridges listed, and possibly a br-int bridge. 3) if there is no br-int bridge, do 'vdsm-tool ovn-config ovn-controller-ip host-ip' 4) when you have configured several nodes in the OVN, you should see them listed as geneve devices in 'ovs-vsctl show', ie: This is a 4 node cluster, so the other 3 nodes are expected: [root@d8-r12-c1-n3 ~]# ovs-vsctl show 42df28ba-ffd6-4e61-b7b2-219576da51ab Bridge br-int fail_mode: secure Port "ovn-27461b-0" Interface "ovn-27461b-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.85.91"} Port "vnet1" Interface "vnet1" Port "ovn-a1c08f-0" Interface "ovn-a1c08f-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.85.87"} Port "patch-br-int-to-f7a19c7d-021a-455d-bf3a-c15e212d8831" Interface "patch-br-int-to-f7a19c7d-021a-455d-bf3a-c15e212d8831" type: patch options: {peer="patch-f7a19c7d-021a-455d-bf3a-c15e212d8831-to-br-int"} Port "vnet0" Interface "vnet0" Port "patch-br-int-to-7874ba85-8f6f-4e43-9535-5a1b1353a9ec" Interface "patch-br-int-to-7874ba85-8f6f-4e43-9535-5a1b1353a9ec" type: patch options: {peer="patch-7874ba85-8f6f-4e43-9535-5a1b1353a9ec-to-br-int"} Port "ovn-8da92c-0" Interface "ovn-8da92c-0" type: geneve options: {csum="true", key=flow, remote_ip="192.168.85.95"} Port br-int Interface br-int type: internal Bridge "vdsmbr_LZmj3uJ1" Port "vdsmbr_LZmj3uJ1" Interface "vdsmbr_LZmj3uJ1" type: internal Port "net211" tag: 211 Interface "net211" type: internal Port "eno2" Interface "eno2" Bridge "vdsmbr_e7rcnufp" Port "vdsmbr_e7rcnufp" Interface "vdsmbr_e7rcnufp" type: internal Port ipmi tag: 20 Interface ipmi type: internal Port ovirtmgmt tag: 50 Interface ovirtmgmt type: internal Port "patch-f7a19c7d-021a-455d-bf3a-c15e212d8831-to-br-int" Interface "patch-f7a19c7d-021a-455d-bf3a-c15e212d8831-to-br-int" type: patch options: {peer="patch-br-int-to-f7a19c7d-021a-455d-bf3a-c15e212d8831"} Port "eno1" Interface "eno1" Port "patch-7874ba85-8f6f-4e43-9535-5a1b1353a9ec-to-br-int" Interface "patch-7874ba85-8f6f-4e43-9535-5a1b1353a9ec-to-br-int" type: patch options: {peer="patch-br-int-to-7874ba85-8f6f-4e43-9535-5a1b1353a9ec"} ovs_version: "2.7.3" 5) Create in the cluster the legacy style bridge networks - ie, ovirtmgmt, etc. Do this just like you where creating them for the legacy network. Define the VLAN #, the MTU, etc. 6) Now, create in the network config, the OVN networks - ie, ovn-ovirtmgmt is on an external provider (select OVN), and make sure 'connect to physical network' is checked, and the correct network from step 5 is picked. Save this off. This will connect the two networks together in a bridge, and all services are visible to both ie dhcp, dns.. 7) when you create the VM, select the OVN network interface, not the legacy bridge interface (this is why I decided to prefix with 'ovn-'). 8) Create the vm, start it, migrate, stop, re-start, etc, it all should work now. Lots of reading.. lots of interesting stuff found.. finally figured this out after reading a bunch of bug fixes for the latest RC (released today) thomas On 03/15/2018 03:21 AM, Dan Kenigsberg wrote:
On Thu, Mar 15, 2018 at 1:50 AM, Thomas Davis <tadavis@lbl.gov> wrote:
Well, I just hit
https://bugzilla.redhat.com/show_bug.cgi?id=1513991
And it's been closed, which means with vdsm-4.20.17-1.el7.centos.x86_64 OVS networking is totally borked..
You are welcome to reopen that bug, specifying your use case for OvS. I cannot promise fixing this bug, as our resources are limited, and that bug, which was introduced in 4.2, was not deemed as urgently needed. https://gerrit.ovirt.org/#/c/86932/ attempts to fix the bug, but it still needs a lot of work.
I know OVS is Experimental, but it worked in 4.1.x, and now we have to do a step back to legacy bridge just to use 4.2.x, which in a vlan environment just wreaks havoc (every VLAN need's a unique mac assigned to the bridge, which vdsm does not do, so suddenly you get the kernel complaining about seeing it's mac address several times.)
Could you elaborate on this issue? What is wrong with a bridge that learns its mac from its underlying device? What wold like Vdsm to do, in your opinion? You can file a bug (or even send a patch) if there is a functionality that you'd like to fix.
There is zero documentation on how to use OVN instead of OVS.
I hope that https://ovirt.org/develop/release-management/features/network/provider-physi... can help.
thomas
On 03/13/2018 09:22 AM, Thomas Davis wrote:
I'll work on it some more. I have 2 different clusters in the data center (1 is the Hosted Engine systems, another is not..) I had trouble with both. I'll try again on the non-hosted engine cluster to see what it is doing. I have it working in 4.1, but we are trying to do a clean wipe since the 4.1 engine has been upgraded so many times from v3.5 plus we want to move to hosted-engine-ha from a single engine node and the ansible modules/roles (which also have problems..)
thomas
On Tue, Mar 13, 2018 at 6:27 AM, Edward Haas <ehaas@redhat.com <mailto:ehaas@redhat.com>> wrote:
OVS switch support is experimental at this stage and in some cases when trying to change from one switch to the other, it fails. It was also not checked against a hosted engine setup, which handles networking a bit differently for the management network (ovirtmgmt). Nevertheless, we are interested in understanding all the problems that exists today, so if you can, please share the supervdsm log, it has the interesting networking traces.
We plan to block cluster switch editing until these problems are resolved. It will be only allowed to define a new cluster as OVS, not convert an existing one from Linux Bridge to OVS.
On Fri, Mar 9, 2018 at 9:54 AM, Thomas Davis <tadavis@lbl.gov <mailto:tadavis@lbl.gov>> wrote:
I'm getting further along with 4.2.2rc3 than the 4.2.1 when it comes to hosted engine and vlans.. it actually does install under 4.2.2rc3.
But it's a complete failure when I switch the cluster from Linux Bridge/Legacy to OVS. The first time I try, vdsm does not properly configure the node, it's all messed up.
I'm getting this in vdsmd logs:
2018-03-08 23:12:46,610-0800 INFO (jsonrpc/7) [api.network] START setupNetworks(networks={u'ovirtmgmt': {u'ipv6autoconf': True, u'nic': u'eno1', u'vlan': u'50', u'ipaddr': u'192.168.85.49', u'switch': u'ovs', u'mtu': 1500, u'netmask': u'255.255.252.0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'gateway': u'192.168.85.254', u'defaultRoute': True}}, bondings={}, options={u'connectivityCheck': u'true', u'connectivityTimeout': 120}) from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:46)
2018-03-08 23:12:52,449-0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573)
2018-03-08 23:12:52,511-0800 INFO (jsonrpc/7) [api.network] FINISH setupNetworks error=[Errno 19] ovirtmgmt is not present in the system from=::ffff:192.168.85.24,56806, flow_id=4147e25f-0a23-4f47-a0a4-d424a3437d11 (api:50) 2018-03-08 23:12:52,512-0800 ERROR (jsonrpc/7) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in setupNetworks File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1527, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system 2018-03-08 23:12:52,512-0800 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.setupNetworks failed (error -32603) in 5.90 seconds (__init__:573) 2018-03-08 23:12:54,769-0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:573) 2018-03-08 23:12:54,772-0800 INFO (jsonrpc/5) [api.host] START getCapabilities() from=::1,45562 (api:46) 2018-03-08 23:12:54,906-0800 INFO (jsonrpc/5) [api.host] FINISH getCapabilities error=[Errno 19] ovirtmgmt is not present in the system from=::1,45562 (api:50) 2018-03-08 23:12:54,906-0800 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:611) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 606, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 201, in _dynamicMethod result = fn(*methodArgs) File "<string>", line 2, in getCapabilities File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1339, in getCapabilities c = caps.get() File "/usr/lib/python2.7/site-packages/vdsm/host/caps.py", line 168, in get net_caps = supervdsm.getProxy().network_caps() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in __call__ return callMethod() File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in <lambda> **kwargs) File "<string>", line 2, in network_caps File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod raise convert_to_error(kind, result) IOError: [Errno 19] ovirtmgmt is not present in the system
So something is dreadfully wrong with the bridge to ovs conversion in 4.2.2rc3.
thomas _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (4)
-
Dan Kenigsberg -
Edward Haas -
Robert O'Kane -
Thomas Davis