- Users - oVirt List Archives

VM's migrations cause timeouts on various unrelated hosts which results in host restart
by 73616c616e＠gmail.com 25 Mar '24

25 Mar '24

During the VM's migrations from one host to another, various unrelated hosts have timeouts and some of them are being restarted by Engine due to their "unresponsive" status. However they're working fine at that time... It only happens when migration process is being run on other unrelated hosts. Of course, one solution might be increasing the timeout, but it's only a workaround and in my opinion, not a proper solution. This whole problem occurs after few minutes of when the migrations were started. oVirt-engine version: 4.4.10.7 (at the moment we do not have the available window for an update) vdsm version on hosts: 4.40.100.2 ** Below logs are example of one of many hosts as all of them had similar messages ** Engine logs when first timeout appeared until it's restarted: 09:31:07,856+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] No interaction with host 'os-host06' for 20000 ms. 09:31:29,242+01 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,243+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'GetAllVmStatsVDSCommand(HostName = os-host06, VdsIdVDSCommandParametersBase:{hostId='aa11wa4d-6e20-44hf-a76e-b0878bba1f42'})' execution failed: VDSGenericException: VDSNetworkException: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,243+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Failed to fetch vms info for host 'os-host06'(aa11wa4d-6e20-44hf-a76e-b0878bba1f42) - skipping VMs monitoring. 09:31:29,246+01 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-3) [] Clearing domains data for host os-host06 09:31:29,251+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-95) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM os-host06 command Get Host Statistics failed: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,251+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-95) [] Unable to GetStats: VDSNetworkException: VDSGenericException: VDSNetworkException: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,287+01 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-3) [] Server failed to respond, vds_id='aa11wa4d-6e20-44hf-a76e-b0878bba1f42', vds_name='os-host06', vm_count=8, spm_status='None', non-responsive_timeout (seconds)=64, error: VDSGenericException: VDSNetworkException: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,289+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3) [] EVENT_ID: VDS_FAILURE(12), Host os-host06 is non responsive. 09:31:29,551+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [2ae04e58] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Executing power management status on Host os-host06 using Proxy Host os-host03 and Fence Agent ipmilan:os-host06-iDrac. 09:31:29,659+01 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (EE-ManagedThreadFactory-engine-Thread-38) [2ae04e58] Opening SSH Soft Fencing session on host 'os-host06' 09:31:29,843+01 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (EE-ManagedThreadFactory-engine-Thread-38) [2ae04e58] Executing SSH Soft Fencing command on host 'os-host06' 09:31:32,239+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to os-host06/10.10.5.17 09:31:44,358+01 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (EE-ManagedThreadFactory-engine-Thread-38) [2e93a767] Power-Management: STOP of host 'os-host06' initiated. Logs from host at the same time just before the restart: 09:30:36,926+0100 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'apparentsize': '1207959552', 'truesize': '1207959552'} from=internal, task_id=f3f073de-2c08-47b2-aae2-017a7480db61 (api:54) 09:30:39,117+0100 INFO (jsonrpc/3) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:30:39,119+0100 INFO (jsonrpc/3) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:30:39,123+0100 INFO (jsonrpc/2) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:30:39,125+0100 INFO (jsonrpc/2) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:30:39,130+0100 INFO (jsonrpc/1) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:30:39,131+0100 INFO (jsonrpc/1) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:30:39,136+0100 INFO (jsonrpc/4) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:30:39,138+0100 INFO (jsonrpc/4) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:30:39,143+0100 INFO (jsonrpc/3) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:30:39,144+0100 INFO (jsonrpc/3) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:30:39,149+0100 INFO (jsonrpc/2) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:30:39,151+0100 INFO (jsonrpc/2) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:30:39,156+0100 INFO (jsonrpc/1) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:30:39,158+0100 INFO (jsonrpc/1) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:30:39,162+0100 INFO (jsonrpc/4) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:30:39,164+0100 INFO (jsonrpc/4) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:30:39,858+0100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:30:39,863+0100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:30:39,873+0100 INFO (jsonrpc/3) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:30:39,874+0100 INFO (jsonrpc/3) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},... 09:30:41,228+0100 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:10.10.3.150,59772 (api:48) 09:30:41,249+0100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:10.10.3.150,59772, task_id=978cd75b-fb99-47ee-af62-0bc0338fc66a (api:48) 09:30:41,250+0100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000389989', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000117308', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000123364', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000135331', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '2.1', 'delay': '0.000147366', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::ffff:10.10.3.150,59772, task_id=978cd75b-fb99-47ee-af62-0bc0338fc66a (api:54) 09:30:41,250+0100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:10.10.3.150,59772, task_id=691b59a6-4a16-4e8d-9676-f1d46a3981b9 (api:48) 09:30:41,250+0100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.10.3.150,59772, task_id=691b59a6-4a16-4e8d-9676-f1d46a3981b9 (api:54) 09:30:41,254+0100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:30:48,833+0100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5bdca038-eb0c-4e50-aeca-be2349387a36 (api:48) 09:30:48,833+0100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000389989', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000117308', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000123364', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000135331', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '9.7', 'delay': '0.000147366', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5bdca038-eb0c-4e50-aeca-be2349387a36 (api:54) 09:30:54,237+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:30:54,239+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:30:54,243+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:30:54,245+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:30:54,250+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:30:54,252+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:30:54,256+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:30:54,258+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:30:54,263+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:30:54,264+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:30:54,269+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:30:54,271+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:30:54,276+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:30:54,278+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:30:54,282+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:30:54,284+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:30:54,888+0100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:30:54,892+0100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:30:54,896+0100 INFO (jsonrpc/7) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:30:54,896+0100 INFO (jsonrpc/7) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},.... 09:30:55,220+0100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:10.10.3.150,59772 (api:48) 09:30:55,223+0100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {'26157351-db69-4e59-a460-1de23cf47517': 'Up', 'f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6': 'Up', 'f335c1e8-4699-4e81-be22-dcb92a6a3ca6': 'Up', '50221e76-d8ee-414b-a3b7-f8a860b21831': 'Up', '25aee386-e8ba-4ac0-85c5-cde2e0d22348': 'Up', '1348d3e0-c02c-4c9a-b669-601e6dcc9b80': 'Up', 'bdeb4e6f-9546-438a-95c8-70c426c3490c': 'Up', '01ff01e5-5360-4549-b538-38529537a587': 'Up'} (throttledlog:104) 09:30:55,223+0100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:02,775+0100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:10.10.3.150,59772 (api:48) 09:31:02,795+0100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '2.2', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '2.2', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '3.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:54) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:54) 09:31:02,801+0100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:48) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '3.3', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '3.4', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '4.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:54) 09:31:09,351+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:31:09,353+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:31:09,358+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:31:09,360+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:31:09,365+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:31:09,366+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:31:09,371+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:31:09,373+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:31:09,378+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:31:09,379+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:31:09,384+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:31:09,386+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:31:09,391+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:31:09,392+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:31:09,397+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:31:09,398+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:31:09,911+0100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:31:09,915+0100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},... 09:30:55,220+0100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:10.10.3.150,59772 (api:48) 09:30:55,223+0100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {'26157351-db69-4e59-a460-1de23cf47517': 'Up', 'f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6': 'Up', 'f335c1e8-4699-4e81-be22-dcb92a6a3ca6': 'Up', '50221e76-d8ee-414b-a3b7-f8a860b21831': 'Up', '25aee386-e8ba-4ac0-85c5-cde2e0d22348': 'Up', '1348d3e0-c02c-4c9a-b669-601e6dcc9b80': 'Up', 'bdeb4e6f-9546-438a-95c8-70c426c3490c': 'Up', '01ff01e5-5360-4549-b538-38529537a587': 'Up'} (throttledlog:104) 09:30:55,223+0100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:02,775+0100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:10.10.3.150,59772 (api:48) 09:31:02,795+0100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '2.2', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '2.2', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '3.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:54) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:54) 09:31:02,801+0100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:48) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '3.3', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '3.4', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '4.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:54) 09:31:09,351+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:31:09,353+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:31:09,358+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:31:09,360+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:31:09,365+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:31:09,366+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:31:09,371+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:31:09,373+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:31:09,378+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:31:09,379+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:31:09,384+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:31:09,386+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:31:09,391+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:31:09,392+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:31:09,397+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:31:09,398+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:31:09,911+0100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:31:09,915+0100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},... 09:31:18,887+0100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=cd897cf9-6f3e-4514-92cf-9a9939f7ce52 (api:48) 09:31:18,887+0100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000106409', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '8.4', 'delay': '0.000115832', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '8.4', 'delay': '0.00011779', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '8.4', 'delay': '0.000125551', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '9.7', 'delay': '0.000130676', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=cd897cf9-6f3e-4514-92cf-9a9939f7ce52 (api:54) 09:31:24,466+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:31:24,468+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:31:24,473+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:31:24,474+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:31:24,479+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:31:24,481+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:31:24,486+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:31:24,487+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:31:24,492+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:31:24,494+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:31:24,499+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:31:24,501+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:31:24,505+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:31:24,507+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:31:24,512+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:31:24,513+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:31:24,925+0100 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:31:24,929+0100 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:31:24,933+0100 INFO (jsonrpc/6) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:31:24,933+0100 INFO (jsonrpc/6) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},...

1 0

virt-v2v cannot authenticate with oVirt engine API with OAuth2
by malcolm.strydom＠pacxa.com 22 Mar '24

22 Mar '24

I've been reading through archives but not able to find what i need. Essentially what I'm trying to do is migrate a larger number of VMs from our OVM environment to a new OLVM setup. In an effort to reduce lots of replication and copying of the disk image (export, convert, copy over, import etc.) I found this article which shows a pretty slick way to do it in one shot https://blogs.oracle.com/scoter/post/how-to-migrate-oracle-vm-to-oracle-lin… The main command behind it all is the virt-v2v that makes it possible. It looks something like this: virt-v2v -i libvirtxml vm-test1.xml -o rhv-upload -oc https://<OLVM-server>/ovirt-engine/api -os <my storage> -op /tmp/ovirt-admin-password -of raw -oo rhv-cluster=Default -oo rhv-cafile=/root/ca.pem The problem I'm having is I cannot authenticate with my new OLVM server at the ovirt-engine/api URL. Since user/password is depricated and you must use OAuth 2.0 with a token I'm stuck. I have OLVM 4.5.4-1.0.27.el8 and from what I've read in oVirt 4.5 (not sure what version it started) they use keycloak oAuth 2.0 and the older ovirt-aaa-jdbc-tool is now deprecated. In doing some testing I found I can use curl and authenticate against the ovirt-engine/api and get a token like this: OVIRT_ENGINE_URL="https://<myolvm1>/ovirt-engine" USERNAME="admin@ovirt@internalsso" PASSWORD="<mypassword>" CLUSTER_NAME="Default" TOKEN=$(curl -k -X POST -H "Accept: application/json" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=password &username=$USERNAME&password=$PASSWORD&scope=ovirt-app-api" $OVIRT_ENGINE_URL/sso/oauth/token | jq -r '.access_token') I was then able to query the API to validate my token works curl -k -H "Accept: application/json" -H "Authorization: Bearer $TOKEN" "$OVIRT_ENGINE_URL/api/clusters?search=name=$CLUSTER_NAME" The problem is virt-v2v does not support posting any form information or the token to authenticate. Best I can tell the -oc option is strictly the URL and if you want a username in there it's in the form of https://<name>@<server>. So even if I wrote a script and used curl to authenticate and get a token I still can't find a way to make virt-v2v use it. So I'm stuck how do I get virt-v2v working? Is there a way to re-enable the deprecated user/pass method of accessing the ovirt-engine/api ? or as a last resort a way to get virt-v2v supporting the token? Thanks for any insight Malcolm

3 5

error live migration ovirt-node 4.5.5
by Nathanaël Blanchet 21 Mar '24

21 Mar '24

Hi all, I manage three 4.5 engines with 4.5 ovirt-nodes. When I update nodes (el8 based) from 4.5.4 to 4.5.5, vms can't migrate anymore. They only migrate from 4.5.4 el8 to 4.5.4 el8 hosts, 4.5.5 el8 to 4.5.5 el8 hosts. But: They can migrate from 4.5.4(el8) to 4.5.5(el9) It may come from qemu-kvm incompatibility versions: 4.5.5 el9: qemu-kvm-8.1.0-4 4.5.4 el8: qemu-kvm-6.2.0-41 4.5.5 el8: qemu-kvm-6.2.0-20 This makes the migration impossible without a downtime. The solution could be to resintall each host in 4.5.5 el9 based, but the most of my production vms run with a qxl/spice graphic based which are not supported anymore on el9. Does anybody know how to do this migration without any downtime? Thank you. -- Nathanaël Blanchet Administrateur Systèmes et Réseaux Service Informatique et REseau (SIRE) Département des systèmes d'information 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

1 0

virt-v2v paused by system after one hour or a bit more
by Claus Serbe 21 Mar '24

21 Mar '24

Hi, I am migrating some vmware VM's from an NFS Storage via rhv-upload in virt-v2v, what is working good. But now I try to move some bigger VM's with several disks and sadly after a while (I would guess around an hour) the Ovirt-engine shows me "Paused by system" instead of transfering, so when the next disk should be imported, it will fail In the ovirt-engine.log I see the following lines for the remaining 4 disks. 2024-03-21 06:14:06,815-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-35) [f61b3906-804d-470f-8524-6507081fbdec] EVENT_ID: UPLOAD_IMAGE_PAUSED_BY_SYSTEM_TIMEOUT(1,071), Upload was paused by system. Reason: timeout due to transfer inactivity. 2024-03-21 06:14:17,915-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-14) [aef8e312-d811-4a39-b5fb-342157209bce] EVENT_ID: UPLOAD_IMAGE_PAUSED_BY_SYSTEM_TIMEOUT(1,071), Upload was paused by system. Reason: timeout due to transfer inactivity. 2024-03-21 06:14:24,959-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) [860b012d-78a4-49f8-a875-52f4299c8298] EVENT_ID: UPLOAD_IMAGE_PAUSED_BY_SYSTEM_TIMEOUT(1,071), Upload was paused by system. Reason: timeout due to transfer inactivity. 2024-03-21 06:14:46,099-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-65) [f93869ee-2ecb-4f54-b3e9-b12259637b0b] EVENT_ID: UPLOAD_IMAGE_PAUSED_BY_SYSTEM_TIMEOUT(1,071), Upload was paused by system. Reason: timeout due to transfer inactivity. There are 2 strange things. 1. When I start virt-v2v it will create all 6 disks and set them to transferring, but virt-v2v will import one after the other, what leads to kind of unused/timing out transferring tickets. 2. When I copy the disk images to a local disk before, it works. Maybe just because of faster transfer speeds. Is there a possibility to transfer parallel or maybe extend the timeout? Cause basically everything is working, but it's just timing out Thanks in advance for any suggestions

3 3

Deploy oVirt Engine fail behind proxy
by Matteo Bonardi 21 Mar '24

21 Mar '24

Hi, I am trying to deploy the ovirt engine following self-hosted engine installation procedure on documentation. Deployment servers are behind a proxy and I have set it in environment and in yum.conf before run deploy. Deploy fails because ovirt engine vm cannot resolve AppStream repository url: [ INFO ] TASK [ovirt.engine-setup : Install oVirt Engine package] [ ERROR ] fatal: [localhost -> ovirt-manager.mydomain]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'AppStream': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=st… [Could not resolve host: mirrorlist.centos.org]", "rc": 1, "results": []} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ INFO ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Force facts gathering] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Clean local storage pools] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20201109165237.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20201109164244-b3e8sd.log How I can set proxy for the engine vm? Ovirt version: [root@myhost ~]# rpm -qa | grep ovirt-engine-appliance ovirt-engine-appliance-4.4-20200916125954.1.el8.x86_64 [root@myhost ~]# rpm -qa | grep ovirt-hosted-engine-setup ovirt-hosted-engine-setup-2.4.6-1.el8.noarch OS version: [root@myhost ~]# cat /etc/centos-release CentOS Linux release 8.2.2004 (Core) [root@myhost ~]# uname -a Linux myhost.mydomain 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Thanks for the help. Regards, Matteo

7 20

Creating disks works for XML (sort of) but not JSON
by itsavant＠gmail.com 20 Mar '24

20 Mar '24

https://ovirt.github.io/ovirt-engine-api-model/4.5/#services/disks POST /ovirt-engine/api/disks <disk> <storage_domains> <storage_domain id="123"/> </storage_domains> <name>mydisk</name> <provisioned_size>1048576</provisioned_size> <format>cow</format> </disk> First off <storage_domain id="123"/> as a field is ridiculous because there is no JSON equivalent I can determine. Next Why is this a good XML structure? <storage_domains><storage_domain><name>MyStorageDomainName</name></storage_domain></storage_domains> But this is not <storage_domains><storage_domain><id>7aaa2d33-c7ba-46f0-b4fd-4cf48a724e8d</id></storage_domain></storage_domains> Next when trying to create a new disk directly on a VM POST /ovirt-engine/api/vms/396bc7c9-ed4a-4209-b9fd-f93e9a47cd31/diskattachments Name works for XML: <disk_attachment> <bootable>true</bootable> <interface>virtio</interface> <active>true</active> <disk> <description>Boot Disk</description> <format>cow</format> <name>mybootdisk</name> <provisioned_size>1048576</provisioned_size> <storage_domains> <storage_domain> <name>MyStorageDomainName</name> </storage_domain> </storage_domains> </disk> </disk_attachment> But Not for JSON { "bootable": "true", "interface": "virtio", "active": "true", "disk": { "description": "Boot Disk", "format": "cow", "name": "mybootdisk", "provisioned_size": "1048576", "storage_domains": { "storage_domain": { "name": "MyStorageDomainName" } } } } And ID fails for both I think the REST API for creating disks needs some real help. They are deprecating the only one that actually works right /ovirt-engine/api/storagedomains/123/disks Does anyone have the correct syntax in JSON, because the available documentation is almost entirely devoid of any, AND the server response..... wait for it...... "The remote server returned an error: (400) Bad Request." .....that is all

1 0

oVirt manager page fails with "500 error" after fresh install of oVirt 4.4.10 manager on Rocky Linux 8.9
by mr_tim_walsh＠hotmail.com 20 Mar '24

20 Mar '24

I did a clean install of oVirt manager 4.4.10 on a clean Rocky Linux 8.9. I have found I get through installing and running "engine-setup" successfully without errors. The manager web site errors on "500 internal server error" but the Grafana site works without a problem. I seem to be missing one step or one configuration setting. I'm working with Rocky Linux 8.9 and oVirt 4.4.10 partially to plan for migration, but also because I have older hardware at home I am using to sandbox with. I also plan to test upgrading from CentOS 7.9 with oVirt 4.3.10, so I want to make sure I have a solid installation method Rocky Linux 8.9 and oVirt 4.4.10. (my current method for installing CentOS 7.9 with oVirt 4.3.10 works flawless on hardware, and VMs). When following the documentation for installing oVirt standalone manager on Enterprise Linux on the oVirt web site: https://www.ovirt.org/documentation/installing_ovirt_as_a_standalone_manage… There’s a section specific to "RedHat derivatives" – specifically Rocky Linux https://www.ovirt.org/download/install_on_rhel.html I followed the steps to the best of my ability. Here are the specific steps I took: installed Rocky Linux - minimal install, added "system tools" when that boots up I run these commands in this order: dnf distro-sync --nobest dnf install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm dnf module -y enable javapackages-tools dnf module -y enable pki-deps dnf module -y enable postgresql:12 dnf module -y enable mod_auth_openidc:2.3 dnf module -y enable nodejs:14 dnf install -y https://resources.ovirt.org/pub/ovirt-4.4/rpm/el8/x86_64/ovirt-engine-appli… (this package gets installed when you run the command: dnf install ovirt-engine-appliance ) dnf install ovirt-engine -y engine-setup I found the documentation says to run adding repos first then install release 4.4 RPM but the Java repo errors out, so I install the RPM first. I could do that last and enable Java afterwards like this: ... everything else... dnf install https://resources.ovirt.org/pub/yum-repo/ovirt-release44.rpm dnf module -y enable nodejs:14 Again, to summarize: I have followed the documentation to the best of my ability and the process succeeds, but it always fails on 500s on trying to launch the manager. The Grafana web site works fine. I should note on the same system I can install CentOS 7.9 and oVirt 4.3.10. It succeeds and works flawlessly. Like I said earlier, I must be missing one step or one configuration setting. Thanks in advance for any help you can provide.

1 0

Create Vm without Storage Domain
by Shafi Mohammed 20 Mar '24

20 Mar '24

Hi Team , I am new to Ovirt and have been using KVM off late . I use libvirt (virt or virsh commands) to create and manage Vms . I have a pretty basic doubt in OVirt . I was trying to migrate a VM from my KVM host .I copied my .qcow2 virtual disk to the ovirt host. I am able to create a Vm after uploading the disk to a new storage domain and create a VM with the storage domain. Can I create a VM without using a storage domain ( By directly attaching the copied .qcow2 disk ) to the Vm Regards Shafi

4 4

oVirt hosts mounting old storage domain even though it's not listed in the engine
by stanleym＠missouri.edu 19 Mar '24

19 Mar '24

We have an oVirt cluster with multiple hosts that continues to mount an old storage domain we used for ISO images. The engine does not show this old storage domain and I've even looked at the engine sql database and not found this old storage domain. Manually unmounting the ISO storage domain on the hosts doesn't fix it as it will remount within a few minutes. Is there some way to manually make vdsm stop remounting this old path? I didn't know if there is a way to regenerate those files from scratch. We've went through all our VMs and removed the cdroms even trying to get it to stop. We're going to try rebuilding a host from scratch to see if it's an artifact left on the host versus an artifact that's coming from the engine. Any help would be appreciated!

1 0

install node on centos stream 9 fails with message NetworkNotFoundError
by Jirka Simon 19 Mar '24

19 Mar '24

Hello oVirt community. i just tried to add a new host(wh refresh) to our cluster with installed Centos Stream 9, bud ovirt installation fails with message network not found. there are 4 NICs (2SPF one of them is configured) and 2 ethenrent which are not configured and are down here is end instalation log. any hint what can I check ? 2024-03-11 11:21:44 CET - { "uuid" : "c43fec42-2838-4319-8f00-adaf24351341", "counter" : 412, "stdout" : "fatal: [ovirt1.domain.tld]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"10.36.191.11\", \"ovirt1.domain.tld\"], \"delta\": \"0:00:02.236199\", \"end\": \"2024-03-11 11:21:43.909643\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-03-11 11:21:41.673444\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\n KeyError: 'ovirt1.domain.tld'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*arg s)\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 11 9, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_conf ig.py\\\", line 117, in get_network\", \" return networks[net_name]\", \"KeyError: 'ovirt1.domain.tld'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \\\"/us r/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_c aps(), net_name))\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\"], \"stdout\": \ "\", \"stdout_lines\": []}", "start_line" : 414, "end_line" : 415, "runner_ident" : "ca3d44fb-7482-481b-97d3-66f2ba7c9d0b", "event" : "runner_on_failed", "pid" : 1520750, "created" : "2024-03-11T10:21:43.926054", "parent_uuid" : "00163e21-72be-4886-1543-000000000042", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "4aa34ee9-1be3-4377-a573-ef33d837d406", "play" : "all", "play_uuid" : "00163e21-72be-4886-1543-000000000002", "play_pattern" : "all", "task" : "Configure OVN for oVirt", "task_uuid" : "00163e21-72be-4886-1543-000000000042", "task_action" : "ansible.builtin.command", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml:43", "role" : "ovirt-provider-ovn-driver", "host" : "ovirt1.domain.tld", "remote_addr" : "ovirt1.domain.tld", "res" : { "changed" : true, "stdout" : "", "stderr" : "Traceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network\n return networks[net_name]\nKeyError: 'ovirt1.domain.tld'\n\nDuring handling of the above exceptio n, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/bin/vdsm-tool\", line 195, in main\n return tool_command[cmd][\"command\"](*args)\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 63, in ov n_config\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network\n raise NetworkNotFoundError(net_name)\nvdsm.tool.ovn_config.NetworkNotFoundE rror: ovirt1.domain.tld", "rc" : 1, "cmd" : [ "vdsm-tool", "ovn-config", "10.36.191.11", "ovirt1.domain.tld" ], "start" : "2024-03-11 11:21:41.673444", "end" : "2024-03-11 11:21:43.909643", "delta" : "0:00:02.236199", "msg" : "non-zero return code", "invocation" : { "module_args" : { "_raw_params" : "vdsm-tool ovn-config 10.36.191.11 ovirt1.domain.tld\n", "_uses_shell" : false, "expand_argument_vars" : true, "stdin_add_newline" : true, "strip_empty_ends" : true, "argv" : null, "chdir" : null, "executable" : null, "creates" : null, "removes" : null, "stdin" : null } }, "stdout_lines" : [ ], "stderr_lines" : [ "Traceback (most recent call last):", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network", " return networks[net_name]", "KeyError: 'ovirt1.domain.tld'", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \"/usr/bin/vdsm-tool\", line 195, in main", " return tool_command[cmd][\"command\"](*args)", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn _config.py\", line 63, in ovn_config", " ip_address = get_ip_addr(get_network(network_caps(), net_name))", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network", " raise NetworkNotFoundError(net_name)", "v dsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld" ], "_ansible_no_log" : false }, "start" : "2024-03-11T10:21:41.493826", "end" : "2024-03-11T10:21:43.925759", "duration" : 2.431933, "ignore_errors" : null, "event_loop" : null, "uuid" : "c43fec42-2838-4319-8f00-adaf24351341" } } 2024-03-11 11:21:44 CET - { "uuid" : "c43fec42-2838-4319-8f00-adaf24351341", "counter" : 412, "stdout" : "fatal: [ovirt1.domain.tld]: FAILED! => {\"changed\": true, \"cmd\": [\"vdsm-tool\", \"ovn-config\", \"10.36.191.11\", \"ovirt1.domain.tld\"], \"delta\": \"0:00:02.236199\", \"end\": \"2024-03-11 11:21:43.909643\", \"msg\": \"non-zero return code\", \"rc\": 1, \"start\": \"2024-03-11 11:21:41.673444\", \"stderr\": \"Traceback (most recent call last):\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 117, in get_network\\n return networks[net_name]\\n KeyError: 'ovirt1.domain.tld'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \\\"/usr/bin/vdsm-tool\\\", line 195, in main\\n return tool_command[cmd][\\\"command\\\"](*arg s)\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\\n File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 11 9, in get_network\\n raise NetworkNotFoundError(net_name)\\nvdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\", \"stderr_lines\": [\"Traceback (most recent call last):\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_conf ig.py\\\", line 117, in get_network\", \" return networks[net_name]\", \"KeyError: 'ovirt1.domain.tld'\", \"\", \"During handling of the above exception, another exception occurred:\", \"\", \"Traceback (most recent call last):\", \" File \\\"/us r/bin/vdsm-tool\\\", line 195, in main\", \" return tool_command[cmd][\\\"command\\\"](*args)\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 63, in ovn_config\", \" ip_address = get_ip_addr(get_network(network_c aps(), net_name))\", \" File \\\"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\\\", line 119, in get_network\", \" raise NetworkNotFoundError(net_name)\", \"vdsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld\"], \"stdout\": \ "\", \"stdout_lines\": []}", "start_line" : 414, "end_line" : 415, "runner_ident" : "ca3d44fb-7482-481b-97d3-66f2ba7c9d0b", "event" : "runner_on_failed", "pid" : 1520750, "created" : "2024-03-11T10:21:43.926054", "parent_uuid" : "00163e21-72be-4886-1543-000000000042", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "4aa34ee9-1be3-4377-a573-ef33d837d406", "play" : "all", "play_uuid" : "00163e21-72be-4886-1543-000000000002", "play_pattern" : "all", "task" : "Configure OVN for oVirt", "task_uuid" : "00163e21-72be-4886-1543-000000000042", "task_action" : "ansible.builtin.command", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-provider-ovn-driver/tasks/configure.yml:43", "role" : "ovirt-provider-ovn-driver", "host" : "ovirt1.domain.tld", "remote_addr" : "ovirt1.domain.tld", "res" : { "changed" : true, "stdout" : "", "stderr" : "Traceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network\n return networks[net_name]\nKeyError: 'ovirt1.domain.tld'\n\nDuring handling of the above exceptio n, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/bin/vdsm-tool\", line 195, in main\n return tool_command[cmd][\"command\"](*args)\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 63, in ov n_config\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network\n raise NetworkNotFoundError(net_name)\nvdsm.tool.ovn_config.NetworkNotFoundE rror: ovirt1.domain.tld", "rc" : 1, "cmd" : [ "vdsm-tool", "ovn-config", "10.36.191.11", "ovirt1.domain.tld" ], "start" : "2024-03-11 11:21:41.673444", "end" : "2024-03-11 11:21:43.909643", "delta" : "0:00:02.236199", "msg" : "non-zero return code", "invocation" : { "module_args" : { "_raw_params" : "vdsm-tool ovn-config 10.36.191.11 ovirt1.domain.tld\n", "_uses_shell" : false, "expand_argument_vars" : true, "stdin_add_newline" : true, "strip_empty_ends" : true, "argv" : null, "chdir" : null, "executable" : null, "creates" : null, "removes" : null, "stdin" : null } }, "stdout_lines" : [ ], "stderr_lines" : [ "Traceback (most recent call last):", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network", " return networks[net_name]", "KeyError: 'ovirt1.domain.tld'", "", "During handling of the above exception, another exception occurred:", "", "Traceback (most recent call last):", " File \"/usr/bin/vdsm-tool\", line 195, in main", " return tool_command[cmd][\"command\"](*args)", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn _config.py\", line 63, in ovn_config", " ip_address = get_ip_addr(get_network(network_caps(), net_name))", " File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network", " raise NetworkNotFoundError(net_name)", "v dsm.tool.ovn_config.NetworkNotFoundError: ovirt1.domain.tld" ], "_ansible_no_log" : false }, "start" : "2024-03-11T10:21:41.493826", "end" : "2024-03-11T10:21:43.925759", "duration" : 2.431933, "ignore_errors" : null, "event_loop" : null, "uuid" : "c43fec42-2838-4319-8f00-adaf24351341" } } Thank you Jirka

2 4