Failing to migrate hosted engine from 4.4.6 host to 4.4.7 host
Hi, I update the hosted engine to 4.4.7 and one of the 2 nodes where the engine is running. Current status is: - Hosted engine at 4.4.7 running on Node 0 - Node 0 at 4.4.6 - Node 1 at 4.4.7 Now, moving Node 0 to maintenance successfully moved the SPM from Node 0 to Node 1 but while trying to migrate hosted engine I get on Node 0 vdsm.log: 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:10.46.8.133,35048, task_id=f12d7694-d2b5-4658-9e0d-3f0dc54aca93 (api:48) 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'1996dc3b-d33f-49cb-b32a-8f7b1d50af5e': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00114065', 'valid': True, 'version': 5, 'acq uired': True, 'actual': True}} from=::ffff:10.46.8.133,35048, task_id=f12d7694-d2b5-4658-9e0d-3f0dc54aca93 (api:54) 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:10.46.8.133,35048, task_id=6515fac9-830a-4b6a-904e-cc1262e87f01 (api:48) 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.46.8.133,35048, task_id=6515fac9-830a-4b6a-904e-cc1262e87f01 (api:54) 2021-07-06 12:25:07,883+0000 ERROR (migsrc/b2072331) [virt.vm] (vmId='b2072331-1558-4186-86b4-fa83af8eba95') can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission de nied (migration:294) 2021-07-06 12:25:07,888+0000 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.46.8.133,35048 (api:54) 2021-07-06 12:25:08,166+0000 ERROR (migsrc/b2072331) [virt.vm] (vmId='b2072331-1558-4186-86b4-fa83af8eba95') Failed to migrate (migration:467) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 441, in _regular_run time.time(), machineParams File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 537, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 626, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 555, in _perform_migration self._migration_flags) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 159, in call return getattr(self._vm._dom, name)(*a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f ret = attr(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2119, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission denied 2021-07-06 12:25:08,197+0000 INFO (jsonrpc/6) [api.virt] START getMigrationStatus() from=::ffff:10.46.8.133,35048, flow_id=4e86b85d, vmId=b2072331-1558-4186-86b4-fa83af8eba95 (api:48) 2021-07-06 12:25:08,197+0000 INFO (jsonrpc/6) [api.virt] FINISH getMigrationStatus return={'status': {'code': 0, 'message': 'Done'}, 'migrationStats': {'status': {'code': 12, 'message': 'Fatal error during migr ation'}, 'progress': 0}} from=::ffff:10.46.8.133,35048, flow_id=4e86b85d, vmId=b2072331-1558-4186-86b4-fa83af8eba95 (api:54) On node 0: # ls -lZ /run/libvirt/common/system.token ls: cannot access '/run/libvirt/common/system.token': No such file or directory On node 1: # ls -lZ /run/libvirt/common/system.token -rw-------. 1 root root system_u:object_r:virt_var_run_t:s0 32 Jul 6 09:29 /run/libvirt/common/system.token any clue? -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
On Tue, Jul 6, 2021 at 3:36 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Hi, I update the hosted engine to 4.4.7 and one of the 2 nodes where the engine is running. Current status is: - Hosted engine at 4.4.7 running on Node 0 - Node 0 at 4.4.6 - Node 1 at 4.4.7
Now, moving Node 0 to maintenance successfully moved the SPM from Node 0 to Node 1 but while trying to migrate hosted engine I get on Node 0 vdsm.log:
...
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2119, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission denied
This looks like the selinux issue we had in libvirt 7.4. Do we have the latest selinux-policy-target package on the host?
On Tue, Jul 6, 2021 at 3:56 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Jul 6, 2021 at 3:36 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Hi, I update the hosted engine to 4.4.7 and one of the 2 nodes where the engine is running. Current status is: - Hosted engine at 4.4.7 running on Node 0 - Node 0 at 4.4.6 - Node 1 at 4.4.7
Now, moving Node 0 to maintenance successfully moved the SPM from Node 0 to Node 1 but while trying to migrate hosted engine I get on Node 0 vdsm.log:
...
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2119, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission denied
This looks like the selinux issue we had in libvirt 7.4. Do we have the latest selinux-policy-target package on the host?
Yeah, seems like https://bugzilla.redhat.com/show_bug.cgi?id=1964317
Il giorno mar 6 lug 2021 alle ore 15:18 Arik Hadas <ahadas@redhat.com> ha scritto:
On Tue, Jul 6, 2021 at 3:56 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Tue, Jul 6, 2021 at 3:36 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Hi, I update the hosted engine to 4.4.7 and one of the 2 nodes where the engine is running. Current status is: - Hosted engine at 4.4.7 running on Node 0 - Node 0 at 4.4.6 - Node 1 at 4.4.7
Now, moving Node 0 to maintenance successfully moved the SPM from Node 0 to Node 1 but while trying to migrate hosted engine I get on Node 0 vdsm.log:
...
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2119, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission denied
This looks like the selinux issue we had in libvirt 7.4. Do we have the latest selinux-policy-target package on the host?
Yeah, seems like https://bugzilla.redhat.com/show_bug.cgi?id=1964317
In order to get the migration working this solved: restorecon /var/run/libvirt/common/system.token ls -lZ /var/run/libvirt/common/system.token -rw-------. 1 root root system_u:object_r:virt_common_var_run_t:s0 32 Jul 6 09:29 /var/run/libvirt/common/system.token service libvirtd restart service virtlogd restart +Lev Veyde <lveyde@redhat.com> sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1955415 is not 100% fixed then -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
On Tue, Jul 6, 2021 at 4:27 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
This looks like the selinux issue we had in libvirt 7.4. Do we have the latest selinux-policy-target package on the host?
Yeah, seems like https://bugzilla.redhat.com/show_bug.cgi?id=1964317
In order to get the migration working this solved: restorecon /var/run/libvirt/common/system.token ls -lZ /var/run/libvirt/common/system.token -rw-------. 1 root root system_u:object_r:virt_common_var_run_t:s0 32 Jul 6 09:29 /var/run/libvirt/common/system.token service libvirtd restart service virtlogd restart
On the source or destination?
Il giorno mar 6 lug 2021 alle ore 15:36 Nir Soffer <nsoffer@redhat.com> ha scritto:
On Tue, Jul 6, 2021 at 4:27 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
This looks like the selinux issue we had in libvirt 7.4. Do we have the latest selinux-policy-target package on the host?
Yeah, seems like https://bugzilla.redhat.com/show_bug.cgi?id=1964317
In order to get the migration working this solved: restorecon /var/run/libvirt/common/system.token ls -lZ /var/run/libvirt/common/system.token -rw-------. 1 root root system_u:object_r:virt_common_var_run_t:s0 32 Jul 6 09:29 /var/run/libvirt/common/system.token service libvirtd restart service virtlogd restart
On the source or destination?
Destination, 4.4.7 -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*
Il giorno mar 6 lug 2021 alle ore 14:56 Nir Soffer <nsoffer@redhat.com> ha scritto:
On Tue, Jul 6, 2021 at 3:36 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Hi, I update the hosted engine to 4.4.7 and one of the 2 nodes where the engine is running. Current status is: - Hosted engine at 4.4.7 running on Node 0 - Node 0 at 4.4.6 - Node 1 at 4.4.7
Now, moving Node 0 to maintenance successfully moved the SPM from Node 0 to Node 1 but while trying to migrate hosted engine I get on Node 0 vdsm.log:
...
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2119, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission denied
This looks like the selinux issue we had in libvirt 7.4. Do we have the latest selinux-policy-target package on the host?
Nir, this is a 4.4.6 host with # rpm -qv libvirt libvirt-7.0.0-14.el8s.x86_64 # rpm -qv vdsm vdsm-4.40.60.7-1.el8.x86_64 # rpm -qv selinux-policy selinux-policy-3.14.3-67.el8.noarch which is migrating hosted engine VM to a just upgaded 4.4.7 node with: # rpm -qv libvirt libvirt-7.4.0-1.el8s.x86_64 # rpm -qv vdsm vdsm-4.40.70.6-1.el8.x86_64 # rpm -qv selinux-policy selinux-policy-3.14.3-71.el8.noarch -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*
I’m having similar problem like this. 15 host, 7 of them already upgraded to 4.4.7 and I can’t migrate any VM or HE from 4.4.6 host to 4.4.7. Regards, Nur Imam Febrianto From: Sandro Bonazzola<mailto:sbonazzo@redhat.com> Sent: 06 July 2021 19:37 To: oVirt Users<mailto:users@ovirt.org>; Arik Hadas<mailto:ahadas@redhat.com> Subject: [ovirt-users] Failing to migrate hosted engine from 4.4.6 host to 4.4.7 host Hi, I update the hosted engine to 4.4.7 and one of the 2 nodes where the engine is running. Current status is: - Hosted engine at 4.4.7 running on Node 0 - Node 0 at 4.4.6 - Node 1 at 4.4.7 Now, moving Node 0 to maintenance successfully moved the SPM from Node 0 to Node 1 but while trying to migrate hosted engine I get on Node 0 vdsm.log: 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:10.46.8.133,35048, task_id=f12d7694-d2b5-4658-9e0d-3f0dc54aca93 (api:48) 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'1996dc3b-d33f-49cb-b32a-8f7b1d50af5e': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00114065', 'valid': True, 'version': 5, 'acq uired': True, 'actual': True}} from=::ffff:10.46.8.133,35048, task_id=f12d7694-d2b5-4658-9e0d-3f0dc54aca93 (api:54) 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:10.46.8.133,35048, task_id=6515fac9-830a-4b6a-904e-cc1262e87f01 (api:48) 2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.46.8.133,35048, task_id=6515fac9-830a-4b6a-904e-cc1262e87f01 (api:54) 2021-07-06 12:25:07,883+0000 ERROR (migsrc/b2072331) [virt.vm] (vmId='b2072331-1558-4186-86b4-fa83af8eba95') can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission de nied (migration:294) 2021-07-06 12:25:07,888+0000 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.46.8.133,35048 (api:54) 2021-07-06 12:25:08,166+0000 ERROR (migsrc/b2072331) [virt.vm] (vmId='b2072331-1558-4186-86b4-fa83af8eba95') Failed to migrate (migration:467) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 441, in _regular_run time.time(), machineParams File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 537, in _startUnderlyingMigration self._perform_with_conv_schedule(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 626, in _perform_with_conv_schedule self._perform_migration(duri, muri) File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 555, in _perform_migration self._migration_flags) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 159, in call return getattr(self._vm._dom, name)(*a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f ret = attr(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2119, in migrateToURI3 raise libvirtError('virDomainMigrateToURI3() failed') libvirt.libvirtError: can't connect to virtlogd: Unable to open system token /run/libvirt/common/system.token: Permission denied 2021-07-06 12:25:08,197+0000 INFO (jsonrpc/6) [api.virt] START getMigrationStatus() from=::ffff:10.46.8.133,35048, flow_id=4e86b85d, vmId=b2072331-1558-4186-86b4-fa83af8eba95 (api:48) 2021-07-06 12:25:08,197+0000 INFO (jsonrpc/6) [api.virt] FINISH getMigrationStatus return={'status': {'code': 0, 'message': 'Done'}, 'migrationStats': {'status': {'code': 12, 'message': 'Fatal error during migr ation'}, 'progress': 0}} from=::ffff:10.46.8.133,35048, flow_id=4e86b85d, vmId=b2072331-1558-4186-86b4-fa83af8eba95 (api:54) On node 0: # ls -lZ /run/libvirt/common/system.token ls: cannot access '/run/libvirt/common/system.token': No such file or directory On node 1: # ls -lZ /run/libvirt/common/system.token -rw-------. 1 root root system_u:object_r:virt_var_run_t:s0 32 Jul 6 09:29 /run/libvirt/common/system.token any clue? -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2F&data=04%7C01%7C%7C1f2786ed85ef4ff51b2408d9407ac80b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637611718387924037%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XABzBOFycDV6ST2Muc2obGJ2g3wOzUB1Z7iyNDkXBd4%3D&reserved=0> sbonazzo@redhat.com<mailto:sbonazzo@redhat.com> [https://static.redhat.com/libs/redhat/brand-assets/2/corp/logo--200.png]<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.redhat.com%2F&data=04%7C01%7C%7C1f2786ed85ef4ff51b2408d9407ac80b%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637611718387933997%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aFK%2BVMOcG6VmmDNbzFGNYf1fm36x6y%2BMB%2BzjAUlXqjA%3D&reserved=0> Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.
participants (4)
-
Arik Hadas -
Nir Soffer -
Nur Imam Febrianto -
Sandro Bonazzola