8:32 a.m.
Hi,
I update the hosted engine to 4.4.7 and one of the 2 nodes where the engine
is running.
Current status is:
- Hosted engine at 4.4.7 running on Node 0
- Node 0 at 4.4.6
- Node 1 at 4.4.7
Now, moving Node 0 to maintenance successfully moved the SPM from Node 0 to
Node 1 but while trying to migrate hosted engine I get on Node 0 vdsm.log:
2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] START
repoStats(domains=()) from=::ffff:10.46.8.133,35048,
task_id=f12d7694-d2b5-4658-9e0d-3f0dc54aca93 (api:48)
2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] FINISH
repoStats return={'1996dc3b-d33f-49cb-b32a-8f7b1d50af5e': {'code': 0,
'lastCheck': '3.0', 'delay': '0.00114065',
'valid': True, 'version':
5, 'acq
uired': True, 'actual': True}} from=::ffff:10.46.8.133,35048,
task_id=f12d7694-d2b5-4658-9e0d-3f0dc54aca93 (api:54)
2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] START
multipath_health() from=::ffff:10.46.8.133,35048,
task_id=6515fac9-830a-4b6a-904e-cc1262e87f01 (api:48)
2021-07-06 12:25:07,882+0000 INFO (jsonrpc/5) [vdsm.api] FINISH
multipath_health return={} from=::ffff:10.46.8.133,35048,
task_id=6515fac9-830a-4b6a-904e-cc1262e87f01 (api:54)
2021-07-06 12:25:07,883+0000 ERROR (migsrc/b2072331) [virt.vm]
(vmId='b2072331-1558-4186-86b4-fa83af8eba95') can't connect to
virtlogd: Unable to open system token
/run/libvirt/common/system.token: Permission de
nied (migration:294)
2021-07-06 12:25:07,888+0000 INFO (jsonrpc/5) [api.host] FINISH
getStats return={'status': {'code': 0, 'message': 'Done'},
'info':
(suppressed)} from=::ffff:10.46.8.133,35048 (api:54)
2021-07-06 12:25:08,166+0000 ERROR (migsrc/b2072331) [virt.vm]
(vmId='b2072331-1558-4186-86b4-fa83af8eba95') Failed to migrate
(migration:467)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line
441, in _regular_run
time.time(), machineParams
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line
537, in _startUnderlyingMigration
self._perform_with_conv_schedule(duri, muri)
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line
626, in _perform_with_conv_schedule
self._perform_migration(duri, muri)
File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line
555, in _perform_migration
self._migration_flags)
File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line
159, in call
return getattr(self._vm._dom, name)(*a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in
f
ret = attr(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py",
line 131, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/common/function.py",
line 94, in wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 2119, in
migrateToURI3
raise libvirtError('virDomainMigrateToURI3() failed')
libvirt.libvirtError: can't connect to virtlogd: Unable to open system
token /run/libvirt/common/system.token: Permission denied
2021-07-06 12:25:08,197+0000 INFO (jsonrpc/6) [api.virt] START
getMigrationStatus() from=::ffff:10.46.8.133,35048, flow_id=4e86b85d,
vmId=b2072331-1558-4186-86b4-fa83af8eba95 (api:48)
2021-07-06 12:25:08,197+0000 INFO (jsonrpc/6) [api.virt] FINISH
getMigrationStatus return={'status': {'code': 0, 'message':
'Done'},
'migrationStats': {'status': {'code': 12, 'message':
'Fatal error
during migr
ation'}, 'progress': 0}} from=::ffff:10.46.8.133,35048,
flow_id=4e86b85d, vmId=b2072331-1558-4186-86b4-fa83af8eba95 (api:54)
On node 0:
# ls -lZ /run/libvirt/common/system.token
ls: cannot access '/run/libvirt/common/system.token': No such file or
directory
On node 1:
# ls -lZ /run/libvirt/common/system.token
-rw-------. 1 root root system_u:object_r:virt_var_run_t:s0 32 Jul 6 09:29
/run/libvirt/common/system.token
any clue?
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*