Hi, we are using static ip, and the iptable-save is pretty empty - all rules are ACCEPT as default. We are doing this in our lab environment, not yet i production - so it's not affecting anything yet. /niklas On 2021-12-10 17:37, Nathanaël Blanchet wrote:

Hi,

Is your migration network provisionned by DHCP?

If so, there is a known bug that prevents NetworkManager to set ip rule table for your migration network, and it won't be fixed before 4.5 release.

As a workaround, you can set the protocol to static or manually assign rule table to 0 as folowing on all on your hosts:

nmcli connection mod migration ipv4.route-table 0 && nmcli con up migration

Le 10/12/2021 à 17:07, Niklas Larsson via Users a écrit :

...

Hi,

we are trying to upgrade 4.3.7 -> 4.4.9 (Ovirt-Node, hosted engine). And things looks good until we try to migrate VM from 4.3.7 host to the 4.4.9 host, it fails with an generic error - and in below is from the vsdm.log from the 4.3.7 host.

Migrating VM from 4.4.9 host -> 4.3.7 host works.

Upgraded the 4.3.7 to 4.3.10 - did not solve it

Log from the 4.3 host: 2021-12-10 16:18:22,829+0100 INFO  (jsonrpc/1) [api.virt] START migrate(params={u'incomingLimit': 2, u'src': u'kvm22.shg.mgn.weblink.se', u'dstqemu': u'10.1.2.111', u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': T rue, u'dst': u'kvm21.shg.mgn.weblink.se:54321', u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, u'limit': -1}]}, u'vmId': u'11f3a2aa-7951-4064-92f9-bb8ab515373b', u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': u'false', u'maxBandwidth': 62, u'method': u'online'}) from=::ffff:10.1.2.51,46546, flow_id=03aca28c-00c9-4b58-a1e8-3aa355e162c2, vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:48) 2021-12-10 16:18:22,831+0100 INFO  (jsonrpc/1) [api.virt] FINISH migrate return={'status': {'message': 'Migration in progress', 'code': 0}, 'progress': 0} from=::ffff:10.1.2.51,46546, flow_id=03aca28c-00c9-4b58-a1e8-3aa355e162c2, vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:54) 2021-12-10 16:18:22,831+0100 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.01 seconds (__init__:312) 2021-12-10 16:18:22,873+0100 ERROR (migsrc/11f3a2aa) [virt.vm] (vmId='11f3a2aa-7951-4064-92f9-bb8ab515373b') [Errno -2] Name or service not known (migration:282) 2021-12-10 16:18:22,875+0100 ERROR (migsrc/11f3a2aa) [virt.vm] (vmId='11f3a2aa-7951-4064-92f9-bb8ab515373b') Failed to migrate (migration:450) Traceback (most recent call last):   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 402, in _regular_run     self._setupVdsConnection()   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 239, in _setupVdsConnection     client = self._createClient(port)   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", line 227, in _createClient     client_socket = utils.create_connected_socket(host, int(port), sslctx)   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 435, in create_connected_socket     socket.AF_UNSPEC, socket.SOCK_STREAM) gaierror: [Errno -2] Name or service not known 2021-12-10 16:18:22,888+0100 INFO  (jsonrpc/7) [api.virt] START getMigrationStatus() from=::ffff:10.1.2.51,46546, vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:48) 2021-12-10 16:18:22,888+0100 INFO  (jsonrpc/7) [api.virt] FINISH getMigrationStatus return={'status': {'message': 'Done', 'code': 0}, 'migrationStats': {'status': {'message': 'Fatal error during migration', 'code': 12}, 'progress': 0}} from=::ffff:10.1.2.51,46546, vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:54) 2021-12-10 16:18:22,889+0100 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getMigrationStatus succeeded in 0.00 seconds (__init__:312) 2021-12-10 16:18:23,431+0100 INFO  (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds (__init__:312)

Any ideas?

/niklas _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BWSVMZGNRNK5MB...

Hi, Can't see any errors for networking - we only have one logical network, that has all the features: /niklas On 2021-12-13 09:26, Ales Musil wrote:

Hi,

is your engine reporting out-of-sync on that migration network?

Best regards, Ales

On Mon, Dec 13, 2021 at 9:23 AM Niklas Larsson via Users <users@ovirt.org> wrote:

Hi,

we are using static ip, and the iptable-save is pretty empty - all rules are ACCEPT as default.

We are doing this in our lab environment, not yet i production - so it's not affecting anything yet.

/niklas

On 2021-12-10 17:37, Nathanaël Blanchet wrote: > Hi, > > Is your migration network provisionned by DHCP? > > If so, there is a known bug that prevents NetworkManager to set ip > rule table for your migration network, and it won't be fixed before > 4.5 release. > > As a workaround, you can set the protocol to static or manually assign > rule table to 0 as folowing on all on your hosts: > > nmcli connection mod migration ipv4.route-table 0 && nmcli con up > migration > > Le 10/12/2021 à 17:07, Niklas Larsson via Users a écrit : >> Hi, >> >> we are trying to upgrade 4.3.7 -> 4.4.9 (Ovirt-Node, hosted engine). >> And things looks good until we try to migrate VM from 4.3.7 host to >> the 4.4.9 host, it fails with an generic error - and in below is from >> the vsdm.log from the 4.3.7 host. >> >> Migrating VM from 4.4.9 host -> 4.3.7 host works. >> >> Upgraded the 4.3.7 to 4.3.10 - did not solve it >> >> Log from the 4.3 host: >> 2021-12-10 16:18:22,829+0100 INFO  (jsonrpc/1) [api.virt] START >> migrate(params={u'incomingLimit': 2, u'src': >> u'kvm22.shg.mgn.weblink.se <http://kvm22.shg.mgn.weblink.se>', u'dstqemu': u'10.1.2.111', >> u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': T >> rue, u'dst': u'kvm21.shg.mgn.weblink.se:54321 <http://kvm21.shg.mgn.weblink.se:54321>', >> u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': >> u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], >> u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': >> [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': >> {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, >> {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': >> 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, >> u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, >> u'limit': -1}]}, u'vmId': u'11f3a2aa-7951-4064-92f9-bb8ab515373b', >> u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': >> u'false', u'maxBandwidth': 62, u'method': u'online'}) >> from=::ffff:10.1.2.51,46546, >> flow_id=03aca28c-00c9-4b58-a1e8-3aa355e162c2, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:48) >> 2021-12-10 16:18:22,831+0100 INFO  (jsonrpc/1) [api.virt] FINISH >> migrate return={'status': {'message': 'Migration in progress', >> 'code': 0}, 'progress': 0} from=::ffff:10.1.2.51,46546, >> flow_id=03aca28c-00c9-4b58-a1e8-3aa355e162c2, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:54) >> 2021-12-10 16:18:22,831+0100 INFO  (jsonrpc/1) >> [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.01 seconds >> (__init__:312) >> 2021-12-10 16:18:22,873+0100 ERROR (migsrc/11f3a2aa) [virt.vm] >> (vmId='11f3a2aa-7951-4064-92f9-bb8ab515373b') [Errno -2] Name or >> service not known (migration:282) >> 2021-12-10 16:18:22,875+0100 ERROR (migsrc/11f3a2aa) [virt.vm] >> (vmId='11f3a2aa-7951-4064-92f9-bb8ab515373b') Failed to migrate >> (migration:450) >> Traceback (most recent call last): >>   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", >> line 402, in _regular_run >>     self._setupVdsConnection() >>   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", >> line 239, in _setupVdsConnection >>     client = self._createClient(port) >>   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", >> line 227, in _createClient >>     client_socket = utils.create_connected_socket(host, int(port), >> sslctx) >>   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 435, in >> create_connected_socket >>     socket.AF_UNSPEC, socket.SOCK_STREAM) >> gaierror: [Errno -2] Name or service not known >> 2021-12-10 16:18:22,888+0100 INFO  (jsonrpc/7) [api.virt] START >> getMigrationStatus() from=::ffff:10.1.2.51,46546, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:48) >> 2021-12-10 16:18:22,888+0100 INFO  (jsonrpc/7) [api.virt] FINISH >> getMigrationStatus return={'status': {'message': 'Done', 'code': 0}, >> 'migrationStats': {'status': {'message': 'Fatal error during >> migration', 'code': 12}, 'progress': 0}} from=::ffff:10.1.2.51,46546, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:54) >> 2021-12-10 16:18:22,889+0100 INFO  (jsonrpc/7) >> [jsonrpc.JsonRpcServer] RPC call VM.getMigrationStatus succeeded in >> 0.00 seconds (__init__:312) >> 2021-12-10 16:18:23,431+0100 INFO  (jsonrpc/3) >> [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds >> (__init__:312) >> >> Any ideas? >> >> /niklas >> _______________________________________________ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BWSVMZGNRNK5MB... > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TVYCGJXVHLGQJ6...

--

Ales Musil

Software Engineer - RHV Network

Red Hat EMEA <https://www.redhat.com>

amusil@redhat.com IM: amusil

<https://red.ht/sig>

_______________________________________________ Users mailing list --users@ovirt.org To unsubscribe send an email tousers-leave@ovirt.org Privacy Statement:https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/VHY2BJZTM7SOJG...

Hi, I an sorry for the noise - and thanks for obvious question - and the answer is that resolv.conf had the wrong dns entries on the old host. Fixed it and now we can migrate. Unknown how it ended there - but we are now happy and rocking. /nikas On 2021-12-13 09:45, Ales Musil wrote:

And are you able to reach from one host to another via IP and via FQDN?

Best regards, Ales

On Mon, Dec 13, 2021 at 9:42 AM Niklas Larsson via Users <users@ovirt.org> wrote:

Hi,

Can't see any errors for networking - we only have one logical network, that has all the features:

/niklas

On 2021-12-13 09:26, Ales Musil wrote:

...

Hi,

is your engine reporting out-of-sync on that migration network?

Best regards, Ales

On Mon, Dec 13, 2021 at 9:23 AM Niklas Larsson via Users <users@ovirt.org> wrote:

Hi,

we are using static ip, and the iptable-save is pretty empty - all rules are ACCEPT as default.

We are doing this in our lab environment, not yet i production - so it's not affecting anything yet.

/niklas

On 2021-12-10 17:37, Nathanaël Blanchet wrote: > Hi, > > Is your migration network provisionned by DHCP? > > If so, there is a known bug that prevents NetworkManager to set ip > rule table for your migration network, and it won't be fixed before > 4.5 release. > > As a workaround, you can set the protocol to static or manually assign > rule table to 0 as folowing on all on your hosts: > > nmcli connection mod migration ipv4.route-table 0 && nmcli con up > migration > > Le 10/12/2021 à 17:07, Niklas Larsson via Users a écrit : >> Hi, >> >> we are trying to upgrade 4.3.7 -> 4.4.9 (Ovirt-Node, hosted engine). >> And things looks good until we try to migrate VM from 4.3.7 host to >> the 4.4.9 host, it fails with an generic error - and in below is from >> the vsdm.log from the 4.3.7 host. >> >> Migrating VM from 4.4.9 host -> 4.3.7 host works. >> >> Upgraded the 4.3.7 to 4.3.10 - did not solve it >> >> Log from the 4.3 host: >> 2021-12-10 16:18:22,829+0100 INFO (jsonrpc/1) [api.virt] START >> migrate(params={u'incomingLimit': 2, u'src': >> u'kvm22.shg.mgn.weblink.se <http://kvm22.shg.mgn.weblink.se>', u'dstqemu': u'10.1.2.111', >> u'autoConverge': u'true', u'tunneled': u'false', u'enableGuestEvents': T >> rue, u'dst': u'kvm21.shg.mgn.weblink.se:54321 <http://kvm21.shg.mgn.weblink.se:54321>', >> u'convergenceSchedule': {u'init': [{u'params': [u'100'], u'name': >> u'setDowntime'}], u'stalling': [{u'action': {u'params': [u'150'], >> u'name': u'setDowntime'}, u'limit': 1}, {u'action': {u'params': >> [u'200'], u'name': u'setDowntime'}, u'limit': 2}, {u'action': >> {u'params': [u'300'], u'name': u'setDowntime'}, u'limit': 3}, >> {u'action': {u'params': [u'400'], u'name': u'setDowntime'}, u'limit': >> 4}, {u'action': {u'params': [u'500'], u'name': u'setDowntime'}, >> u'limit': 6}, {u'action': {u'params': [], u'name': u'abort'}, >> u'limit': -1}]}, u'vmId': u'11f3a2aa-7951-4064-92f9-bb8ab515373b', >> u'abortOnError': u'true', u'outgoingLimit': 2, u'compressed': >> u'false', u'maxBandwidth': 62, u'method': u'online'}) >> from=::ffff:10.1.2.51,46546, >> flow_id=03aca28c-00c9-4b58-a1e8-3aa355e162c2, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:48) >> 2021-12-10 16:18:22,831+0100 INFO (jsonrpc/1) [api.virt] FINISH >> migrate return={'status': {'message': 'Migration in progress', >> 'code': 0}, 'progress': 0} from=::ffff:10.1.2.51,46546, >> flow_id=03aca28c-00c9-4b58-a1e8-3aa355e162c2, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:54) >> 2021-12-10 16:18:22,831+0100 INFO (jsonrpc/1) >> [jsonrpc.JsonRpcServer] RPC call VM.migrate succeeded in 0.01 seconds >> (__init__:312) >> 2021-12-10 16:18:22,873+0100 ERROR (migsrc/11f3a2aa) [virt.vm] >> (vmId='11f3a2aa-7951-4064-92f9-bb8ab515373b') [Errno -2] Name or >> service not known (migration:282) >> 2021-12-10 16:18:22,875+0100 ERROR (migsrc/11f3a2aa) [virt.vm] >> (vmId='11f3a2aa-7951-4064-92f9-bb8ab515373b') Failed to migrate >> (migration:450) >> Traceback (most recent call last): >>   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", >> line 402, in _regular_run >>     self._setupVdsConnection() >>   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", >> line 239, in _setupVdsConnection >>     client = self._createClient(port) >>   File "/usr/lib/python2.7/site-packages/vdsm/virt/migration.py", >> line 227, in _createClient >>     client_socket = utils.create_connected_socket(host, int(port), >> sslctx) >>   File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 435, in >> create_connected_socket >>     socket.AF_UNSPEC, socket.SOCK_STREAM) >> gaierror: [Errno -2] Name or service not known >> 2021-12-10 16:18:22,888+0100 INFO (jsonrpc/7) [api.virt] START >> getMigrationStatus() from=::ffff:10.1.2.51,46546, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:48) >> 2021-12-10 16:18:22,888+0100 INFO (jsonrpc/7) [api.virt] FINISH >> getMigrationStatus return={'status': {'message': 'Done', 'code': 0}, >> 'migrationStats': {'status': {'message': 'Fatal error during >> migration', 'code': 12}, 'progress': 0}} from=::ffff:10.1.2.51,46546, >> vmId=11f3a2aa-7951-4064-92f9-bb8ab515373b (api:54) >> 2021-12-10 16:18:22,889+0100 INFO (jsonrpc/7) >> [jsonrpc.JsonRpcServer] RPC call VM.getMigrationStatus succeeded in >> 0.00 seconds (__init__:312) >> 2021-12-10 16:18:23,431+0100 INFO (jsonrpc/3) >> [jsonrpc.JsonRpcServer] RPC call Host.ping2 succeeded in 0.00 seconds >> (__init__:312) >> >> Any ideas? >> >> /niklas >> _______________________________________________ >> Users mailing list -- users@ovirt.org >> To unsubscribe send an email to users-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BWSVMZGNRNK5MB... > _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TVYCGJXVHLGQJ6...

--

Ales Musil

Software Engineer - RHV Network

Red Hat EMEA <https://www.redhat.com>

amusil@redhat.com IM: amusil

<https://red.ht/sig>

_______________________________________________ Users mailing list --users@ovirt.org To unsubscribe send an email tousers-leave@ovirt.org Privacy Statement:https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/VHY2BJZTM7SOJG...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RSSDRB5PNNJ3X3...

--

Ales Musil

Software Engineer - RHV Network

Red Hat EMEA <https://www.redhat.com>

amusil@redhat.com IM: amusil

<https://red.ht/sig>

_______________________________________________ Users mailing list --users@ovirt.org To unsubscribe send an email tousers-leave@ovirt.org Privacy Statement:https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/MBZH7MREORXGIJ...