October 2023 - Users - oVirt List Archives

Installing oVirt on RHEL vs CentOS Stream, Alma or Rocky Linux
by Richard W.M. Jones 07 Oct '25

07 Oct '25

I'm still struggling to get oVirt installed. It absolutely will not install on RHEL 8, the documentation here is flat out wrong: https://www.ovirt.org/documentation/installing_ovirt_as_a_standalone_manage… javapackages-tools is not a module that exists in RHEL 8. Apparently it's a CentOS PowerTools module. Since original CentOS no longer really exists either, does anyone have any opinions or experience with installing oVirt on any of: - CentOS Stream 8 - Alma Linux 8.5 - Rocky Linux Alma in particular looks like a straight replacement for old CentOS. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

8 10

Error when deploy Ovirt4.4 Hosted Engine
by staybox＠gmail.com 17 Feb '25

17 Feb '25

Hello, I get error, need help. [ ERROR ] fatal: [localhost]: FAILED! => {"msg": "The conditional check 'not ipv6_deployment|bool and route_rules_ipv4.stdout | from_json | selectattr('priority', 'equalto', 100) | selectattr('dst', 'equalto', virbr_cidr_ipv4 | ipaddr('address') ) | list | length == 0' failed. The error was: error while evaluating conditional (not ipv6_deployment|bool and route_rules_ipv4.stdout | from_json | selectattr('priority', 'equalto', 100) | selectattr('dst', 'equalto', virbr_cidr_ipv4 | ipaddr('address') ) | list | length == 0): 'dict object' has no attribute 'dst'\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/bootstrap_local_vm/01_prepare_routing_rules.yml': line 81, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n changed_when: true\n - name: Add IPv4 inbound route rules\n ^ here\n"}

4 4

How to list all snapshots?
by jorgevisentini＠gmail.com 14 Feb '25

14 Feb '25

Hello everyone! First, I would like to thank everyone involved in this wonderful project. I leave here my sincere thanks! Does anyone know if it is possible to list all snapshots automatically? It can be by ansible, python, shell... any way that helps to list them all without having to enter Domain by Domain. Thank you all!

4 3

snapshot solution: Existing snapshots that were taken after this one will be erased.
by dhanaraj.ramesh＠yahoo.com 29 Jan '25

29 Jan '25

Hi Team, when I want to commit the older snapshots I'm getting warning stating " Existing snapshots that were taken after this one will be erased.". is there any way we can retain the latest snapshots as is in the chain? I knew cloning and template export options are there to secure that latest snapshot data but these are will consume additional space in storage and take time.

4 3

Restart oVirt-Engine
by Jeremey Wise 09 Jan '25

09 Jan '25

How ,without reboot of hosting system, do I restart the oVirt engine? # I tried below but do not seem to effect the virtual machine [root@thor iso]# systemctl restart ov ovirt-ha-agent.service ovirt-imageio.service ovn-controller.service ovs-delete-transient-ports.service ovirt-ha-broker.service ovirt-vmconsole-host-sshd.service ovsdb-server.service ovs-vswitchd.service [root@thor iso]# # You cannot restart the VM " HostedEngine " as it responses: Error while executing action: HostedEngine: - Cannot restart VM. This VM is not managed by the engine. Reason is I had to do some work on a node. Reboot it.. it is back up.. network is all fine.. Cockpit working fine... and gluster fine.. But oVirt-Engine refuses to accept the node is up. -- p <jeremey.wise(a)gmail.com>enguinpages

4 4

Unable to access ovirt Admin Screen from ovirt Host
by louisb＠ameritech.net 08 Dec '24

08 Dec '24

I've reinstalled ovirt 4.4 on my server remotely via cockpit terminal. I'm able to access the ovirt admin screen remotely from the laptop that I used for the install. However, using the same URL I'm unable to gain access to the admin screen. Following the instruction in the documentation I've modified the file: /etc/ovirt-engine/engine.conf.d/99-custom-sso-setup.conf, to reflect the DNS name and I enter in the IP address. But I'm still unable to access the screen from the server console. What else needs to change in order to gain access from the server console? Thanks

5 8

SPM and Task error ...
by Enrico 22 Nov '24

22 Nov '24

Hi all, my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. In the ovirt engine (engine.log) the error is this: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected : ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. while in the hypervisor (SPM) vdsm.log: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected : ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 *ERROR* [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,751+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 34ae2b2f 2019-07-25 12:39:18,752+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 34ae2b2f 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 42de0c2b 2019-07-25 12:39:18,759+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 42de0c2b 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Cleaning zombie tasks: Clearing async task 'Unknown' that started at 'Fri May 03 14:48:50 CEST 2019' 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,765+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: da77af2 2019-07-25 12:39:18,766+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: da77af2 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned there is some relation between this error and a task that has remained hanging, from SPM server: # vdsm-client Task getInfo taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e { "verb": "prepareMerge", "id": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e" } # vdsm-client Task getStatus taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e { "message": "running job 1 of 1", "code": 0, "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e", "taskResult": "", "taskState": "running" } How can I solve this problem ? Thanks a lot for your help !! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________

5 15

oVirt networks
by Enrico Becchetti 28 Aug '24

28 Aug '24

Dear all, Ineed your help to understand how to configure the network of a new oVirt cluster. Mynew system will have a 4.3 engine thatruns in a virtual machine, andsome Dell R7525 AMD EPYC hypervisors, eachholding two 4-port PCI network cards. These servers will have node-ovirt image again in version 4.3. As for the network, there are two HPE Aruba 2540G, non-stackable, with 24 1Gbs ports and 2 10Gbs uplinks to the star center. This is a simplified scheme: My goal is to make the most of the server's 8 ethernet interfaces to have both reliability and maximum possible throughput. This cluster will have two virtual networks, one forovirt management and one for the traffic of individual virtual machines. With that said here's what my idea is. I would like to have two links aggregated by 4Gbs, one for ovrtmgt and the other for vmnet. With the ovirt web interface I can createan active-passive "Mode 1" bond, but this won'tallow me to go beyond 1Gbs. Alternatively I could create a "Mode 4" bond 802.3ad but unfortunately the switches are not stacked and therefore not even this solution applies. This is an example with active passive configuration: Can you tell me if ovirt can generate//nested bonds? Or do you have other solutions ? Thanks a lot ! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Skype:enrico_becchetti Mail: Enrico.Becchetti<at>pg.infn.it Pagina web personale: https://www.pg.infn.it/home/enrico-becchetti/ ______________________________________________________________________

5 6

boot from cdrom & error code 0005
by edp＠maddalena.it 21 Aug '24

21 Aug '24

Hi. I have created a new storage domain (data domain, storage type nfs) to use it to upload iso images. I have so uploaded a new iso and then attach the iso to a new vm. But when I try to boot the vm I obtain this error: booting from dvd/cd... boot failed: could not read from cdrom (code 0005) no bootable device The iso file has been uploaded with success in the data storage domain and so the vm lets my attach the iso to the vm in the boot settings. Can you help me? Thank you

4 6

VM Migration Failed
by KSNull Zero 10 Jul '24

10 Jul '24

Running oVirt 4.4.5 VM cannot migrate between hosts. vdsm.log contains the following error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://ovhost01.local/system: authentication failed: Failed to verify peer's certificate Certificates on hosts was renewed some time ago. How this issue can be fixed ? Thank you.

4 6

How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Derek Atkins 28 Jun '24

28 Jun '24

Hi, I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs: (remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>) I am 99.99% sure this is because the old certs use SHA1. I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate. All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM). So.... Is there a command-line way to re-enroll manually and update the host certs? Or some other way to get all the leftover certs renewed? Thanks, -derek [1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself. -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

8 16

4.5.4 with Ceph only storage
by Maurice Burrows 25 Jun '24

25 Jun '24

Hey ... A long story short ... I have an existing Red Hat Virt / Gluster hyperconverged solution that I am moving away from. I have an existing Ceph cluster that I primarily use for OpenStack and a small requirement for S3 via RGW. I'm planning to build a new oVirt 4.5.4 cluster on RHEL9 using Ceph for all storage requirements. I've read many online articles on oVirt and Ceph, and they all seem to use the Ceph iSCSI gateway, which is now in maintenance, so I'm not real keen to commit to iSCSI. So my question is, IS there any reason I cannot use CephFS for both hosted-engine and as a data storage domain? I'm currently running Ceph Pacific FWIW. Cheers

5 10

i can't access console with noVNC or VNC client(console.vv)
by z84614242＠163.com 14 May '24

14 May '24

i installed the ovirt 4.5 engine on centos stream 9 and add a ovirt node(ovirt node 4.5 iso) to this engine. i am going to run my vm on this node. i follow the instruction to create the data center, the cluster, the storage domain, upload the image. everything is fine. and after i create a vm with ubuntu image attach, i found that i can't visit the console. when i using the noVNC, it says "Something went wrong, connection is closed", when i visit vnc with virt-viewver, is says "Failed to complete handshake Error in the pull function". i try to change the console type to Bochs one and it appear the same. i change to QXL mode and the vm can't start any more. i check the log, it says "unsupported configuration: domain configuration does not support video model 'qxl'". so now i can't visit my vm by anyway. i deploy the engine follow the official instruction and keep mostly option default but why still have this issue. why the noVNC says "Something went wrong" instead of telling me what is actually wrong

2 1

Changing disk QoS causes segfault with IO-Threads enabled (oVirt 4.3.0.4-1.el7)
by jloh＠squiz.net 23 Apr '24

23 Apr '24

We recently upgraded to 4.3.0 and have found that when changing disk QoS settings on VMs whilst IO-Threads is enabled causes them to segfault and the VM to reboot. We've been able to replicate this across several VMs. VMs with IO-Threads disabled/turned off do not segfault when changing the QoS. Mar 1 11:49:06 srvXX kernel: IO iothread1[30468]: segfault at fffffffffffffff8 ip 0000557649f2bd24 sp 00007f80de832f60 error 5 in qemu-kvm[5576498dd000+a03000] Mar 1 11:49:06 srvXX abrt-hook-ccpp: invalid number 'iothread1' Mar 1 11:49:11 srvXX libvirtd: 2019-03-01 00:49:11.116+0000: 13365: error : qemuMonitorIORead:609 : Unable to read from monitor: Connection reset by peer Happy to supply some more logs to someone if they'll help but just wondering whether anyone else has experienced this or knows of a current fix other than turning io-threads off. Cheers.

4 3

Deploy oVirt Engine fail behind proxy
by Matteo Bonardi 21 Mar '24

21 Mar '24

Hi, I am trying to deploy the ovirt engine following self-hosted engine installation procedure on documentation. Deployment servers are behind a proxy and I have set it in environment and in yum.conf before run deploy. Deploy fails because ovirt engine vm cannot resolve AppStream repository url: [ INFO ] TASK [ovirt.engine-setup : Install oVirt Engine package] [ ERROR ] fatal: [localhost -> ovirt-manager.mydomain]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'AppStream': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=st… [Could not resolve host: mirrorlist.centos.org]", "rc": 1, "results": []} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ INFO ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Force facts gathering] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Clean local storage pools] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20201109165237.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20201109164244-b3e8sd.log How I can set proxy for the engine vm? Ovirt version: [root@myhost ~]# rpm -qa | grep ovirt-engine-appliance ovirt-engine-appliance-4.4-20200916125954.1.el8.x86_64 [root@myhost ~]# rpm -qa | grep ovirt-hosted-engine-setup ovirt-hosted-engine-setup-2.4.6-1.el8.noarch OS version: [root@myhost ~]# cat /etc/centos-release CentOS Linux release 8.2.2004 (Core) [root@myhost ~]# uname -a Linux myhost.mydomain 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Thanks for the help. Regards, Matteo

7 20

The oVirt Counter
by Sandro Bonazzola 21 Feb '24

21 Feb '24

Hi, for those who remember the Linux Counter project, if you'd like other to know you're using oVirt and know some details about your deployment, here's a way to count you in: https://ovirt.org/community/ovirt-counter.html Enjoy! -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D PERFORMANCE & SCALE Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

1 1

Cannot restart ovirt after massive failure.
by Gilboa Davara 18 Jan '24

18 Jan '24

Hello all, During the night, one of my (smaller) setups, a single node self hosted engine (localhost NFS) crashed due to what-looks-like a massive disk failure (Software RAID6, with 10 drives + spare). After a reboot, I let the RAID resync with a fresh drive) and went on to start oVirt. However, no such luck. Two issues: 1. ovirt-ha-broker fails due to broken hosted engine state (log attached). 2. ovirt-ha-agent fails due to network test (tcp) even though both remote-host and DNS servers are active. (log attached). Two questions: 1. Can I somehow force the agent to disable the network liveliness test? 2. Can I somehow force the broker to rebuild / fix the hosted engine state? - Gilboa

4 17

Please, Please Help - New oVirt Install/Deployment Failing - "Host is not up..."
by Matthew J Black 18 Jan '24

18 Jan '24

Hi Everyone, Could someone please help me - I've been trying to do an install of oVirt for *weeks* (including false starts and self-inflicted wounds/errors) and it is still not working. My setup: - oVirt v4.5.3 - A brand new fresh vanilla install of RockyLinux 8.6 - all working AOK - 2*NICs in a bond (802.3ad) with a couple of sub-Interfaces/VLANs - all working AOK - All relevant IPv4 Address in DNS with Reverse Lookups - all working AOK - All relevant IPv4 Address in "/etc/hosts" file - all working AOK - IPv6 (using "method=auto" in the interface config file) enabled on the relevant sub-Interface/VLAN - I'm not using IPv6 on the network, only IPv4, but I'm trying to cover all the bases. - All relevant Ports (as per the oVirt documentation) set up on the firewall - ie firewall-cmd --add-service={{ libvirt-tls | ovirt-imageio | ovirt-vmconsole | vdsm }} - All the relevant Repositories installed (ie RockyLinux BaseOS, AppStream, & PowerTools, and the EPEL, plus the ones from the oVirt documentation) I have followed the oVirt documentation (including the special RHEL-instructions and RockyLinux-instructions) to the letter - no deviations, no special settings, exactly as they are written. All the dnf installs, etc, went off without a hitch, including the "dnf install centos-release-ovirt45", "dnf install ovirt-engine-appliance", and "dnf install ovirt-hosted-engine-setup" - no errors anywhere. Here is the results of a "dnf repolist": - appstream Rocky Linux 8 - AppStream - baseos Rocky Linux 8 - BaseOS - centos-ceph-pacific CentOS-8-stream - Ceph Pacific - centos-gluster10 CentOS-8-stream - Gluster 10 - centos-nfv-openvswitch CentOS-8 - NFV OpenvSwitch - centos-opstools CentOS-OpsTools - collectd - centos-ovirt45 CentOS Stream 8 - oVirt 4.5 - cs8-extras CentOS Stream 8 - Extras - cs8-extras-common CentOS Stream 8 - Extras common packages - epel Extra Packages for Enterprise Linux 8 - x86_64 - epel-modular Extra Packages for Enterprise Linux Modular 8 - x86_64 - ovirt-45-centos-stream-openstack-yoga CentOS Stream 8 - oVirt 4.5 - OpenStack Yoga Repository - ovirt-45-upstream oVirt upstream for CentOS Stream 8 - oVirt 4.5 - powertools Rocky Linux 8 - PowerTools So I kicked-off the oVirt deployment with: "hosted-engine --deploy --4 --ansible-extra-vars=he_offline_deployment=true". I used "--ansible-extra-vars=he_offline_deployment=true" because without that flag I was getting "DNF timout" issues (see my previous post `Local (Deployment) VM Can't Reach "centos-ceph-pacific" Repo`). I answer the defaults to all of questions the script asked, or entered the deployment-relevant answers where appropriate. In doing this I double-checked every answer before hitting <Enter>. Everything progressed smoothly until the deployment reached the "Wait for the host to be up" task... which then hung for more than 30 minutes before failing. From the ovirt-hosted-engine-setup... log file: - 2022-10-20 17:54:26,285+1100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"} I checked the following log files and found all of the relevant ERROR lines, then checked several 10s of proceeding and succeeding lines trying to determine what was going wrong, but I could not determine anything. - ovirt-hosted-engine-setup... - ovirt-hosted-engine-setup-ansible-bootstrap_local_vm... - ovirt-hosted-engine-setup-ansible-final_clean... - not really relevant, I believe I can include the log files (or the relevant parts of the log files) if people want - but that are very large: several 100 kilobytes each. I also googled "oVirt Host is not up" and found several entries, but after reading them all the most relevant seems to be a thread from these mailing list: `Install of RHV 4.4 failing - "Host is not up, please check logs, perhaps also on the engine machine"` - but this seems to be talking about an upgrade and I didn't gleam anything useful from it - I could, of course, be wrong about that. So my questions are: - Where else should I be looking (ie other log files, etc, and possible where to find them)? - Does anyone have any idea why this isn't working? - Does anyone have a work-around (including a completely manual process to get things working - I don't mind working in the CLI with virsh, etc)? - What am I doing wrong? Please, I'm really stumped with this, and I really do need help. Cheers Dulux-Oz

8 22

Configure OVN for oVirt failing - vdsm.tool.ovn_config.NetworkNotFoundError: hostname
by huw.m＠twinstream.com 13 Jan '24

13 Jan '24

Hello, When installing the self-hosted engine using rocky 9 as a host (using nightly builds), the install gets as far as running the below ansible task from ovirt-engine - name: Configure OVN for oVirt ansible.builtin.command: > vdsm-tool ovn-config {{ ovn_central }} {{ ovn_tunneling_interface }} {{ ovn_host_fqdn }} This command gets executed as vdsm-tool ovn-config 192.168.57.4 hostname.my.project.com and fails with error "stderr" : "Traceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 117, in get_network\n return networks[net_name]\nKeyError: 'virt-1.local.hyp.twinstream.com'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/bin/vdsm-tool\", line 195, in main\n return tool_command[cmd][\"command\"](*args)\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 63, in ovn_config\n ip_address = get_ip_addr(get_network(network_caps(), net_name))\n File \"/usr/lib/python3.9/site-packages/vdsm/tool/ovn_config.py\", line 119, in get_network\n raise NetworkNotFoundError(net_name)\nvdsm.tool.ovn_config.NetworkNotFoundError: hostname.my.project.com" Running `vdsm-tool list-nets` on the host gives an empty list. `ip a` gives 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:6d:16:65 brd ff:ff:ff:ff:ff:ff altname enp0s6 altname ens6 inet 192.168.121.29/24 brd 192.168.121.255 scope global dynamic noprefixroute eth0 valid_lft 2482sec preferred_lft 2482sec inet6 fe80::5054:ff:fe6d:1665/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:6b:f4:7b brd ff:ff:ff:ff:ff:ff altname enp0s7 altname ens7 inet 192.168.56.151/24 brd 192.168.56.255 scope global noprefixroute eth1 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe6b:f47b/64 scope link valid_lft forever preferred_lft forever 4: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP group default qlen 1000 link/ether 52:54:00:8f:40:45 brd ff:ff:ff:ff:ff:ff altname enp0s8 altname ens8 5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:2f:27:9d brd ff:ff:ff:ff:ff:ff altname enp0s9 altname ens9 6: eth4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bondstorage state UP group default qlen 1000 link/ether 52:54:00:b8:9b:d7 brd ff:ff:ff:ff:ff:ff altname enp0s10 altname ens10 7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:c2:9a:bd brd ff:ff:ff:ff:ff:ff altname enp0s11 altname ens11 8: eth6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bondvm state UP group default qlen 1000 link/ether 52:54:00:ed:f7:cc brd ff:ff:ff:ff:ff:ff altname enp0s12 altname ens12 9: eth7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:de:8a:48 brd ff:ff:ff:ff:ff:ff altname enp0s13 altname ens13 10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:8f:40:45 brd ff:ff:ff:ff:ff:ff inet 192.168.57.4/24 brd 192.168.57.255 scope global noprefixroute bond0 valid_lft forever preferred_lft forever inet6 fe80::5054:ff:fe8f:4045/64 scope link valid_lft forever preferred_lft forever 11: bondvm: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:ed:f7:cc brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:feed:f7cc/64 scope link valid_lft forever preferred_lft forever 12: bondstorage: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:b8:9b:d7 brd ff:ff:ff:ff:ff:ff inet 192.168.59.4/24 brd 192.168.59.255 scope global noprefixroute bondstorage valid_lft forever preferred_lft forever inet6 fe80::5054:ff:feb8:9bd7/64 scope link valid_lft forever preferred_lft forever 13: bondvm.20@bondvm: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:ed:f7:cc brd ff:ff:ff:ff:ff:ff inet6 fe80::5054:ff:feed:f7cc/64 scope link valid_lft forever preferred_lft forever 15: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:b2:5f:e2 brd ff:ff:ff:ff:ff:ff inet 192.168.222.1/24 brd 192.168.222.255 scope global virbr0 valid_lft forever preferred_lft forever 16: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr0 state UNKNOWN group default qlen 1000 link/ether fe:16:3e:34:3d:ea brd ff:ff:ff:ff:ff:ff inet6 fe80::fc16:3eff:fe34:3dea/64 scope link valid_lft forever preferred_lft forever 47: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 6e:27:5f:fa:e3:3a brd ff:ff:ff:ff:ff:ff 48: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 12:7c:d9:2e:cf:26 brd ff:ff:ff:ff:ff:ff 49: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether a2:35:6e:5e:4c:60 brd ff:ff:ff:ff:ff:ff bond0 was selected as the ovirtmgmt bridge NIC. It currently only has one member interface eth2 using balance-xor. In the ovirt management console I can the see host in a down state and given the rest of the playbook ran which requires ssh connectivity between hosted-engine and host, I believe the network is generally setup correctly. No other immediate errors I can. As vdsm-tool ovn-config expects a network to exist with value of the hostname, what is meant to be creating this on the host? Thanks, Huw

2 3

how to renew expired ovirt node vdsm cert manually ?
by dhanaraj.ramesh＠yahoo.com 11 Jan '24

11 Jan '24

below are the steps to renew the expired vdsm cert of ovirt node # To check CERT expired # openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates 1. Backup vdsm folder # cd /etc/pki # mv vdsm vdsm.orig # mkdir vdsm ; chown vdsm:kvm vdsm # cd vdsm # mkdir libvirt-vnc certs keys libvirt-spice libvirt-migrate # chown vdsm:kvm libvirt-vnc certs keys libvirt-spice libvirt-migrate 2. Regenerate cert & keys # vdsm-tool configure --module certificates 3. Copy the cert to destination location chmod 440 /etc/pki/vdsm/keys/vdsmkey.pem chown root /etc/pki/vdsmcerts/*pem chmod 644 /etc/pki/vdsmcerts/*pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-spice/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-vnc/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem cp -p /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-migrate/ca-cert.pem cp -p /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-migrate/server-key.pem cp -p /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem chown root:qemu /etc/pki/vdsm/libvirt-migrate/server-key.pem cp -p /etc/pki/vdsm.orig/keys/libvirt_password /etc/pki/vdsm/keys/ mv /etc/pki/libvirt/clientcert.pem /etc/pki/libvirt/clientcert.pem.orig mv /etc/pki/libvirt/private/clientkey.pem /etc/pki/libvirt/private/clientkey.pem.orig mv /etc/pki/CA/cacert.pem /etc/pki/CA/cacert.pem.orig cp -p /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/libvirt/clientcert.pem cp -p /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/libvirt/private/clientkey.pem cp -p /etc/pki/vdsm/certs/cacert.pem /etc/pki/CA/cacert.pem 3. cross check the backup folder /etc/pki/vdsm.orig vs /etc/pki/vdsm # refer to /etc/pki/vdsm.orig/*/ and set the correct owner & group permission in /etc/pki/vdsm/*/ 4. restart services # Make sure both services are up systemctl restart vdsmd libvirtd

5 4

Unable to install oVirt on RHEL7.5
by SS00514758＠techmahindra.com 05 Jan '24

05 Jan '24

Hi All, I am unable to install oVirt on RHEL7.5, to install it I am taking reference of below link, https://www.ovirt.org/documentation/install-guide/chap-Installing_oVirt.html But though it is not working for me, couple of dependencies is not getting installed, and because of this I am not able to run the ovirt-engine, below are the depencies packages that unable to install, Error: Package: collectd-write_http-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) Requires: collectd(x86-64) = 5.8.0-6.1.el7 Removing: collectd-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-6.1.el7 Updated By: collectd-5.8.1-1.el7.x86_64 (epel) collectd(x86-64) = 5.8.1-1.el7 Available: collectd-5.7.2-1.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-1.el7 Available: collectd-5.7.2-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-3.el7 Available: collectd-5.8.0-2.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-2.el7 Available: collectd-5.8.0-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-3.el7 Available: collectd-5.8.0-5.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-5.el7 Help me to install this. Looking forward to resolve this issue. Regards Sumit Sahay

6 8

Grafana - Origin Not Allowed
by Maton, Brett 14 Dec '23

14 Dec '23

oVirt 4.5.0.8-1.el8 I tried to connect to grafana via the monitoring portal link from the dash and all panels are failing to display any data with varying error messages, but all include 'Origin Not Allowed' I navigated to Data Sources and ran a test on the PostgreSQL connection (localhost) which threw the same Origin Not Allowed error message. Any suggestions?

6 8

Cannot successfully import Windows vm to new ovirt deployment
by netracerx＠mac.com 29 Nov '23

29 Nov '23

I am running into an issue that, from what little Google-fu I've been able to use, should be solved in oVirt 4.5. I'm trying to import WS2019 VMs from ESXi 7 as OVFs or even OVAs. But when I do the import and set the OS to Windows Server 2019 x64 in the admin portal, I get the error "Invalid time zone for the given OS type ". If I leave it at Other OS, the import fails with event ID 1153. I've been banging my head against this for over a week (previously couldn't get the management VM to complete setup), so any guidance is appreciated. Let me know what to pull to help me pin this down? BTW, yes this is a self hosted install as a nested VM on ESXi. Kinda have to right now to test everything (easiest for me over doing a bare metal on my old PowerEdge 2900 at the moment). Running CentOS 9 Stream.

9 37

Hosted-engine restore failing
by Devin A. Bougie 21 Nov '23

21 Nov '23

Hi, All. We are attempting to migrate to a new storage domain for our oVirt 4.5.4 self-hosted engine setup, and are failing with "cannot import name 'Callable' from 'collections'" Please see below for the errors on the console. Many thanks, Devin ------ hosted-engine --deploy --restore-from-file=backup.bck --4 ... [ INFO ] Checking available network interfaces: [ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n' [ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n' [ ERROR ] b"import name 'Callable' from 'collections'\n" [ ERROR ] b'(/usr/lib64/python3.11/collections/__init__.py)\n' [ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/__init__.py)\n" [ ERROR ] Failed to execute stage 'Environment customization': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n' [ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n' [ ERROR ] b"import name 'Callable' from 'collections'\n" [ ERROR ] b'(/usr/lib64/python3.11/collections/__init__.py)\n' [ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/__init__.py)\n" [ ERROR ] Failed to execute stage 'Clean up': Failed executing ansible-playbook [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20231011110358.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20231011110352-raupj9.log

3 4

Gluster: Ideas for migration
by jonas＠rabe.ch 10 Nov '23

10 Nov '23

Hello I have to migrate the Gluster volumes from an old oVirt cluster to a newly built one. I looked into migration strategies, but everything that Red Hat recommends is related to replacing old bricks. In a testing environment I created two clusters and wanted to migrate one volume after the other. Unfortunately that fails because a node cannot be part of two clusters at the same time. The next thing I see, is to recreate the volumes on the new cluster, then constantly rsync the files from the old cluster to the new one and at a specified point in time make the cut over where I stop the applicaiton, do a final rsync and remount the new volume under the old path. Is there any other, nicer way I could accomplish migrating a volume from one Gluster cluster to another?

2 1

Engine on EL 9
by David Carvalho 09 Nov '23

09 Nov '23

Hello, good morning. I’m using Oracle Linux and I intended to install a virtualization platffom with KVM and oracle VM. The Oracle documention only mentions Oracle Linux 8 and there are no oVirt repositories available for OL 9. I visited ovirt.org site and at the download page it only mentions: Engine: * Red Hat Enterprise Linux 8.7 (or similar) * CentOS Stream 8 I still had no reply at Oracle foruns. Will there be a possibility to use this with Oracle Linux 9 soon? I have 3 servers to install and I also intend to use Gluster FS. Thanks and regards. Os melhores cumprimentos David Alexandre M. de Carvalho ═══════════════════ Especialista de Informática Departamento de Informática Universidade da Beira Interior

2 1

VMs randomly pause due to unknown storage error, unable to resume
by Jon Sattelberger 09 Nov '23

09 Nov '23

Hi, > VM xxx has been paused due to unknown storage error. > Migration failed due to a failed validation: [Migrating a VM in paused status due to I/O error is not supported.] (VM: xxx, Source: yyy). Up until recently oVirt 4.5.4 has been running fine on our RHEL 8 hypervisors with primarily (and a few appliances) Linux guests. We started to add Windows 2019 VMs to the cluster with the guest agent installed. They seem to run fine at first, but some of the Windows VMs may randomly pause due to an unknown storage error. The VM cannot be resumed through the UI or virsh. The paused VM cannot be migrated to another Hypervisor. The GlusterFS storage volumes seem fine. Resetting the VM seems to work, but eventually it'll become paused again. The only thing that came to my mind is the virtual hard disks are thin provisioned. Is a preallocated disk necessary for Windows VMs? Any helpful hints on where to look next is greatly appreciated. Thank you, Jon

2 3

Hosted-engine restore failing when migrating to new storage domain
by Devin A. Bougie 09 Nov '23

09 Nov '23

Hello, We have a functioning oVirt 4.5.4 cluster running on fully-updated EL9.2 hosts. We are trying to migrate the self-hosted engine to a new iSCSI storage domain using the existing hosts, following the documented procedure: - set the cluster into global maintenance mode - backup the engine using "engine-backup --scope=all --mode=backup --file=backup.bck --log=backuplog.log" - shutdown the engine - restore the engine using "hosted-engine --deploy --4 --restore-from-file=backup.bck" This almost works, but fails with the attached log file. Any help or suggestions would be greatly appreciated, including alternate procedures for migrating a self-hosted engine from one domain to another. Many thanks, Devin

2 9

How to start QA and testing work in the oVirt project?
by song_chao＠massclouds.com 06 Nov '23

06 Nov '23

Hello everyone, I am a testing and development engineer who has been working for 10 years. I want to learn and participate in Ovirt's testing work, and make my own contribution. Through learning and understanding, I believe that in the near future, I can also make some contributions to the ovirt community. Currently, I would like some information about QA and testing processes and methods. Can anyone provide me with it? Thank you.

3 4

Multiple hosts stuck in Connecting state waiting for storage pool to go up.
by ivan.lezhnjov.iv＠gmail.com 06 Nov '23

06 Nov '23

Hi! We have a problem with multiple hosts stuck in Connecting state, which I hoped somebody here could help us wrap our heads around. All hosts, except one, seem to have very similar symptoms but I'll focus on one host that represents the rest. So, the host is stuck in Connecting state and this what we see in oVirt log files. /var/log/ovirt-engine/engine.log: 2023-04-20 09:51:53,021+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-37) [] Command 'GetCapabilitiesAsyncVDSCommand(HostName = ABC010-176-XYZ, VdsIdAndVdsVDSCommandParametersBase:{hostId='2c458562-3d4d-4408-afc9-9a9484984a91', vds='Host[ABC010-176-XYZ,2c458562-3d4d-4408-afc9-9a9484984a91]'})' execution failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: SSL session is invalid 2023-04-20 09:55:16,556+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-67) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ABC010-176-XYZ command Get Host Capabilities failed: Message timeout which can be caused by communication issues /var/log/vdsm/vdsm.log: 2023-04-20 17:48:51,977+0300 INFO (vmrecovery) [vdsm.api] START getConnectedStoragePoolsList() from=internal, task_id=ebce7c8c-6ded-454e-9aee-86edf72764ef (api:31) 2023-04-20 17:48:51,977+0300 INFO (vmrecovery) [vdsm.api] FINISH getConnectedStoragePoolsList return={'poollist': []} from=internal, task_id=ebce7c8c-6ded-454e-9aee-86edf72764ef (api:37) 2023-04-20 17:48:51,978+0300 INFO (vmrecovery) [vds] recovery: waiting for storage pool to go up (clientIF:723) Both engine.log and vdsm.log are flooded with these messages. They are repeated at regular intervals ad infinitum. This is one common symptom shared by multiple hosts in our deployment. They all have these message loops in engine.log and vdsm.log files. On all Running vdsm-client Host getConnectedStoragePools also returns an empty list represented by [] on all hosts (but interestingly there is one that showed Storage Pool UUID and yet it was still stuck in Connecting state). This particular host (ABC010-176-XYZ) is connected to 3 CEPH iSCSI Storage Domains and lsblk shows 3 block devices with matching UUIDs in their device components. So, the storage seems to be connected but the Storage Pool is not? How is that even possible? Now, what's even more weird is that we tried rebooting the host (via Administrator Portal) and it didn't help. We even tried removing and re-adding the host in Administrator Portal but to no avail. Additionally, the host refused to go into Maintenance mode so we had to enforce it by manually updating Engine DB. We also tried reinstalling the host via Administrator Portal and ran into another weird problem, which I'm not sure if it's a related one or a problem that deserves a dedicated discussion thread but, basically, the underlying Ansible playbook exited with the following error message: "stdout" : "fatal: [10.10.10.176]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Data could not be sent to remote host \\\"10.10.10.176\\\". Make sure this host can be reached over ssh: \", \"unreachable\": true}", Counterintuitively, just before running Reinstall via Administrator Portal we had been able to reboot the same host (which as you know oVirt does via Ansible as well). So, no changes on the host in between just different Ansible playbooks. To confirm that we actually had access to the host over ssh we successfully ran ssh -p $PORT root(a)10.10.10.176 -i /etc/pki/ovirt-engine/keys/engine_id_rsa and it worked. That made us scratch our heads for a while but what seems to had fixed Ansible's ssh access problems was manual full stop of all VDSM-related systemd services on the host. It was just a wild guess but as soon as we stopped all VDSM services Ansible stopped complaining about not being able to reach the target host and successfully did its job. I'm sure you'd like to see more logs but I'm not certain what exactly is relevant. There are a ton of logs as this deployment is comprised of nearly 80 hosts. So, I guess it's best if you just request to see specific logs, messages or configuration details and I'll cherry-pick what's relevant. We don't really understand what's going on and would appreciate any help. We tried just about anything we could think of to resolve this issue and are running out of ideas what to do next. If you have any questions just ask and I'll do my best to answer them.

2 4

Direct LUN I/O errors with SCSI Pass-through enabled
by mgs＠ordix.de 03 Nov '23

03 Nov '23

Hi, in our environment (Version 4.4.10.7) we use fibre channel LUNs, which we attach directly to the VMs (as Direct LUN) with VirtIO-SCSI and SCSI pass-through enabled. The virtual machines run an application that requires 4096 as physical_block_size and 512 as logical_block_size. For this reason, we had to enable SCSI pass-through. Only with SCSI pass-through the correct physical_block_size is passed through to the VM. Now we have the following problem on just about every VM: Error messages of the following form occur in the VMs (in /var/log/messages): kernel: blk_update_request: I/O error, dev sdd, sector 352194592 op 0x1:(WRITE) flags 0xc800 phys_seg 16 prio class 0 This error message coincides with a crash of the application. The error message seems to belong to SCSI. We are currently trying to find an alternative to SCSI pass-through. We want to use VirtIO and somehow pass the physical_block_size. Since the XML files of the VMs are transient, we cannot make any changes there. Does anyone have an idea what the error could be or how to pass the correct physical_block_size? Could VDSM hooks help with this? Thank you and regards Miguel

2 1

Failed to login: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
by p.olivera＠telfy.com 02 Nov '23

02 Nov '23

Hi community, We're encountering the following error when attempting to log in: Warning alert: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed The certificate is valid until 2027: [root@engine9 certs]# openssl x509 -subject -noout -dates -in engine.cer subject=C = US, O = telfy.com, CN = engine9.telfy.com notBefore=Sep 25 10:18:03 2022 GMT notAfter=Sep 27 10:18:03 2027 GMT It's worth noting that our time zone recently switched to GMT+1. Could this change be related to the issue? Has anyone else experienced this problem, and if so, how was it resolved? Thank you.

2 3

Migrated VM has snapshots with same id
by mihailmataras＠gmail.com 31 Oct '23

31 Oct '23

Hi community, We've encountered problem with virtual machines migrated from RHV(4.4.9.5-0.1.el8ev) to oVirt (4.5.4-1.el8) using data domain. When we create snapshot on migrated VM (in oVirt), we see that snapshot has same id as "Active VM" snapshot and disk in illegal status. (we see that in web ui). Meanwhile in database "engine" id's are different. And vdsm-tool shows that there's only one snapshot in status "LEGAL". engine=# select image_guid,parentid,imagestatus,vm_snapshot_id,volume_type,volume_format,active from images where image_group_id='12a8999f-3f6e-4377-8378-40370aa68d3f'; image_guid | parentid | imagestatus | vm_snapshot_id | volume_type | volume_format | active --------------------------------------+--------------------------------------+-------------+--------------------------------------+-------------+---------------+-------- 5cf5a41d-7236-4c41-ae9c-7c6359210db6 | 6d9cc470-d66f-4e68-a0aa-6e8985f392b0 | 1 | 88d91f15-26ad-48c1-91ec-422c1c04e79b | 2 | 4 | t 6d9cc470-d66f-4e68-a0aa-6e8985f392b0 | 00000000-0000-0000-0000-000000000000 | 4 | a47261da-4ebb-42e3-a74a-0882f8e27c73 | 2 | 4 | f (2 rows) vdsm-tool dump-volume-chains 5e88e116-fb90-41ce-b9c3-01ccb9302b16 | grep 12a8999f-3f6e-4377-8378-40370aa68d3f -A 10 image: 12a8999f-3f6e-4377-8378-40370aa68d3f - 6d9cc470-d66f-4e68-a0aa-6e8985f392b0 status: OK, voltype: LEAF, format: COW, legality: LEGAL, type: SPARSE, capacity: 214748364800, truesize: 121198608384 We know how to fix this using database commands. But when we create snapshot , it happens again. Question is how to prevent that? Thank you.

1 0

Re: Failed to synchronize networks of Provider ovirt-provider-ovn
by Mail SET Inc. Group 27 Oct '23

27 Oct '23

Yes, i use same manual to change WebUI SSL. ovirt-ca-file= is a same SSL file which use WebUI. Yes, i restart ovirt-provider-ovn, i restart engine, i restart all what i can restart. Nothing... > 12 сент. 2018 г., в 16:11, Dominik Holler <dholler(a)redhat.com> написал(а): > > On Wed, 12 Sep 2018 14:23:54 +0300 > "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: > >> Ok! > > Not exactly, please use users(a)ovirt.org for such questions. > Other should benefit from this questions, too. > Please write the next mail to users(a)ovirt.org and keep me in CC. > >> What i did: >> >> 1) install oVirt «from box» (4.2.5.2-1.el7); >> 2) generate own ssl for my engine using my FreeIPA CA, Install it and > > What means "Install it"? You can use the doc from the following link > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ht… > > Ensure that ovirt-ca-file= in > /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf > points to the correct file and ovirt-provider-ovn is restarted. > >> get tis issue; >> >> >> [root@engine ~]# tail -n 50 /var/log/ovirt-provider-ovn.log >> 2018-09-12 14:10:23,828 root [SSL: CERTIFICATE_VERIFY_FAILED] >> certificate verify failed (_ssl.c:579) Traceback (most recent call >> last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", >> line 133, in _handle_request method, path_parts, content >> File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", >> line 175, in handle_request return >> self.call_response_handler(handler, content, parameters) File >> "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in >> call_response_handler return response_handler(content, parameters) >> File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", >> line 62, in post_tokens user_password=user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in >> create_token return auth.core.plugin.create_token(user_at_domain, >> user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line >> 48, in create_token timeout=self._timeout()) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, >> in create_token username, password, engine_url, ca_file, timeout) >> File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line >> 91, in _get_sso_token timeout=timeout File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, >> in wrapper response = func(*args, **kwargs) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, >> in wrapper raise BadGateway(e) BadGateway: [SSL: >> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) >> >> >> [root@engine ~]# tail -n 20 /var/log/ovirt-engine/engine.log >> 2018-09-12 14:10:23,773+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:10:23,778+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:10:23,836+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:10:23,837+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:14:12,477+03 INFO >> [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default >> task-6) [] User admin@internal successfully logged in with scopes: >> ovirt-app-admin ovirt-app-api ovirt-app-portal >> ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all >> ovirt-ext=token-info:authz-search >> ovirt-ext=token-info:public-authz-search >> ovirt-ext=token-info:validate ovirt-ext=token:password-access >> 2018-09-12 14:14:12,587+03 INFO >> [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default >> task-6) [1bf1b763] Running command: CreateUserSessionCommand >> internal: false. 2018-09-12 14:14:12,628+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [1bf1b763] EVENT_ID: USER_VDC_LOGIN(30), User >> admin@internal-authz connecting from '10.0.3.61' using session >> 's8jAm7BUJGlicthm6yZBA3CUM8QpRdtwFaK3M/IppfhB3fHFB9gmNf0cAlbl1xIhcJ2WX+ww7e71Ri+MxJSsIg==' >> logged in. 2018-09-12 14:14:30,972+03 INFO >> [org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] Running >> command: ImportProviderCertificateCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:30,982+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] EVENT_ID: >> PROVIDER_CERTIFICATE_IMPORTED(213), Certificate for provider >> ovirt-provider-ovn was imported. (User: admin@internal-authz) >> 2018-09-12 14:14:31,006+03 INFO >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Running >> command: TestProviderConnectivityCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:31,058+03 ERROR >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Command >> 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'default' is using 0 threads out of 1, 5 threads waiting for >> tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engine' is using 0 threads out of 500, 16 threads waiting for >> tasks and 0 tasks in queue. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineScheduled' is using 0 threads out of 100, 100 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads >> waiting for tasks. 2018-09-12 14:15:23,843+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:15:23,849+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:15:23,900+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:23,901+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' >> >> >> [root@engine ~]# >> cat /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf # >> This file is automatically generated by engine-setup. Please do not >> edit manually [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641 >> [SSL] >> https-enabled=true >> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >> [OVIRT] >> ovirt-sso-client-secret=Ms7Gw9qNT6IkXu7oA54tDmxaZDIukABV >> ovirt-host=https://engine.set.local:443 >> ovirt-sso-client-id=ovirt-provider-ovn >> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem >> [PROVIDER] >> provider-host=engine.set.local >> >> >>> 12 сент. 2018 г., в 13:59, Dominik Holler <dholler(a)redhat.com> >>> написал(а): >>> >>> On Wed, 12 Sep 2018 13:04:53 +0300 >>> "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: >>> >>>> Hello Dominik! >>>> I have a same issue with OVN provider and SSL >>>> https://www.mail-archive.com/users@ovirt.org/msg47020.html >>>> <https://www.mail-archive.com/users@ovirt.org/msg47020.html> But >>>> certificate changes not helps to resolve it. Maybe you can help me >>>> with this? >>> >>> Sure. Can you please share the relevant lines of >>> ovirt-provider-ovn.log and engine.log, and the information if you >>> are using the certificates generated by engine-setup with >>> users(a)ovirt.org ? Thanks, >>> Dominik >>> >> > >

5 13

Unable to enable HPET component of an specific VM in oVirt 4.7
by ricardoot＠gmail.com 24 Oct '23

24 Oct '23

Hello community members, I'm currently using oVirt 4.7 as my virtualization environment, and I'm facing an issue with enabling the HPET (High Precision Event Timer) component in the XML configuration file of virtual machine (VM). Upon inspecting the XML file, I noticed that the `<timer name='hpet' present='no'/>` line is missing, indicating that the HPET component is disabled. Here are the steps I have taken so far: 1. I verified that the VM's XML configuration file does not include the `<timer name='hpet' present='yes'/>` line. 2. While the VM was powered on, I used the following command to edit the XML configuration file: ``` virsh edit VM_NAME ``` I added the `<timer name='hpet' present='yes'/>` line to the XML file. However, the changes did not persist after restarting the VM. To provide additional information, on the host where oVirt is running, the available clock sources can be viewed by executing the following command: ``` cat /sys/devices/system/clocksource/clocksource0/available_clocksource ``` The output shows the available clock sources, such as `tsc`, `hpet`, and `acpi_pm`. To resolve the authentication issue with the `virsh` command, I created a user with appropriate privileges using the following command: ``` sudo saslpasswd2 -a libvirt USERNAME ``` After creating the user, I was able to authenticate successfully with the `virsh` command using the newly created credentials. However, I'm unable to find an option to add the HPET parameter in the web console of oVirt. It seems that the option to configure HPET is not available in the web console. Has anyone else encountered a similar issue in oVirt 4.7? Could you please provide guidance or suggest a solution to enable the HPET component in the XML configuration file of a powered-off VM in oVirt 4.7? Any insights, experiences, or suggestions would be greatly appreciated. Thank you in advance for your assistance! Best regards,

2 4

Can I get a way to commit ovirt Project?
by rak kim 24 Oct '23

24 Oct '23

Hello, I have been used to OLV/RHEV product about 3 years I often visit our community to get the error solution as well now I would try to use ovirt and develop to improve some function.. I'm using test environment using compiled oVirt 4.5 package. but I havent yet set up build develpment environment 1. Can you share and advise your ovirt development environment experience? 2. ovirt has some project, and how merge it to compile one source? 3. Is there a process for sharing what features you want to fix before making improvements? Can I just do PR? 4. your tips.. how to debug, ..etc

3 2

fencing state/sequence diagrams
by Renaud 23 Oct '23

23 Oct '23

Hello everyone, After a network outage, we experienced a situation of chaotic connectivity between the oVirt manager (located on a VM outside the cluster) and the hypervisors themselves. During the incident, while only the connectivity between the manager and the hypervisors was difficult, during the recovery we had fencing of several of our hypervisors. I'm looking for a diagram or text that describes all the rules for triggering fencing. Specifically, I have questions about the following (just in case): * Does the manager play an active role in the decision to fence a hypervisor? Regards, Renaud

2 1

HE deploy fails at "Initialize lockspace volume" step
by Giuliano David 23 Oct '23

23 Oct '23

Hi everyone. I need help understanding a failure deploying the hosted engine on a fresh-installed oVirt 4.5.4 el8 node. After the setup via official ISO, I login via ssh in the node and I issue the command: # hosted-engine --deploy --4 --ansible-extra-vars=he_offline_deployment=true -- Note -- The extra ansible variable is the only way I found to inhibit the deployed hosted engine downloading last OS updates that will break Python compatibility between the ansible playbook in the node deploying and the ansible host in the engine deployed. Without that extra variable the deployment fails with fancy reasons. -- End note -- The deployment goes, until i specify an iSCSI target and a (free) LUN. The playbook adds the storage domain, creates the HE disk and transfert the HE vm to the domain. Then an error occurs: [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Initialize lockspace volume] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Workaround for ovirt-ha-broker start failures] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Initialize lockspace volume] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"], "delta": "0:00:00.170053", "end": "2023-10-20 11:21:18.111299", "msg": "non-zero return code", "rc": 1, "start": "2023-10-20 11:21:17.941246", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n \"__main__\", mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n stats = broker.get_stats_from_storage()\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request\n verbose=self.__verbose\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request\n return self.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request\n http_conn = self.send_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request\n self.send_content(connection, request_body)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/python3.6/http/client.py\", line 1268, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib64/python3.6/http/client.py\", line 1044, in _send_output\n self.send(msg)\n File \"/usr/lib64/python3.6/http/client.py\", line 982, in send\n self.connect()\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 76, in connect\n self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main", " \"__main__\", mod_spec)", " File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code", " exec(code, run_globals)", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", " stats = broker.get_stats_from_storage()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in single_request", " http_conn = self.send_request(host, handler, request_body, verbose)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request", " self.send_content(connection, request_body)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content", " connection.endheaders(request_body)", " File \"/usr/lib64/python3.6/http/client.py\", line 1268, in endheaders", " self._send_output(message_body, encode_chunked=encode_chunked)", " File \"/usr/lib64/python3.6/http/client.py\", line 1044, in _send_output", " self.send(msg)", " File \"/usr/lib64/python3.6/http/client.py\", line 982, in send", " self.connect()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 76, in connect", " self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} Then the playbook cleans all the installation and exits. Really cant' figure what's going on ... This is the best point i reached in deploying HE after two weeks of failures and errors of any kind. Please, can someone point me in the right direction to solve this new issue? Thanks. giuliano

2 3

No default gateway defined
by gbv.kris＠mail.ru 22 Oct '23

22 Oct '23

When deploying isolated nested virtualization, we received the error "default gateway is not defined". Two virtual machines were created in the zVirt virtualization environment: the first OS AstraLinux, the second OS zvirtNode. VM are connected to each other by a virtual network. After starting the installation in Hosted Engine mode using the command line, this error was received. The setup was performed without DNS and the default gateway

2 1

moving engine VM outside one ovirt environment
by vishal.merrit＠gmail.com 21 Oct '23

21 Oct '23

What is the best way and steps to move engine VM outside the Ovirt environment. As of now the engine VM is controlling the host but running also on the same Ovirt host. VM must run on a different Ovirt environment. Want to move/migrate engine from hosted engine to stand alone VM

2 1

VDSM host_devops command HSMGetAllTasksStatusesVDS failed: Not SPM: ()
by tapz.malz＠gmail.com 21 Oct '23

21 Oct '23

Hi All, We are facing one issue in our environment. After importing a .ova image from Virtualbox to KVM, when we tried to remove the imported vm, suddenly getting below errors in event VDSM host_devops command HSMGetAllTasksStatusesVDS failed: Not SPM: () Failed to activate Storage Domain domain_devops (Data Center datacenter_devops) by admin@internal-authz It is also observed that our datacenter is non-responsive. When we try to activate it, getting below error Failed Activating Storage Domain domain_devops on Data Center datacenter_devops Can somebody help?

3 2

Strange events during upgrade process to 4.5.4
by james.kump＠missouri.edu 21 Oct '23

21 Oct '23

I recently began managing an Ovirt cluster. It has 5 cpu nodes and 3 gpu nodes in two clusters. When I started it was at 4.2.8 with a standalone engine. I have henceforth been steadily upgrading it to 4.5.4. I followed the upgrade guide and was successful in upgrading all 9 nodes from 4.2 to 4.3.10. I then upgraded the engine to centos 8 with 4.4 engine. I then rebuilt each of the nodes in the two data clusters to centos 8 ovirt 4.4 using the ovirt node installer iso. They all joined the engine correctly. I then upgraded the compatibility to 4.3 in both clusters and rebooted all the running VM's to also be at 4.3 compatibility. I then upgraded the engine to 4.5.4 and proceeded to upgrade two nodes to 4.5.4. They were successful and reattached to the engine correctly. Here is where my issue arises. Any VM that I migrate to the 4.5.4 node from a 4.4.10 node migrates correctly and then within 5 minutes will automatically migrate back. If I force pin a VM it stays running. I have searched the logs but not found anything to explain such behavior. Any assistance or idea ?

2 1

Request to get get an account
by ReFea_IM[e] Original 17 Oct '23

17 Oct '23

Hello, I would like to participate in the oVirt translation project. How can this be done? Thank you.

2 1

[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for <JsonRpcResponse id: "..."
by nicolas＠devels.es 17 Oct '23

17 Oct '23

Hi, We're running oVirt 4.5.4, recently we had a problem due to /var partition being full, and we realized the /var/log/ovirt-engine/engine.log was pretty huge, approximately 16GB in size, despite it's configured to rotate daily. Checking the log, we have a lot (about 53000 at the time of this mail) of WARN events in this form: 2023-10-17 03:34:51,347+01 WARN [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for <JsonRpcResponse id: "d6cc2209-3126-48e1-97c8-97ca84d00f3a" result: [{"statusTime":"3329996823","status":"Up","vmId":"0fad0aa4-ec91-4beb-82a3-5641f0da674f","vmName":"SYTW-Backend-aluXXXXXXXXX","vmType":"kvm","kvmEnable":"true","acpiEnable":"true","elapsedTime":"285568","monitorResponse":"0","clientIp":"","timeOffset":"2","cpuUser":"-0.04","cpuSys": "0.40","cpuUsage":"12369410000000","cpuActual":true,"network":{"vnet3":{"macAddr":"00:1a:4a:4d:cc:d5","name":"vnet3","speed":"1000","state" ... This is just a chunk of the full line, which is actually pretty big as it contains a lot of parameters. So about 53.000 times this long line makes the log huge. Why are we getting such a big amount of these WARNings and what can we do to solve them? Thanks.

1 0

Host reports about one of the Active Storage Domains as Problematic
by Matthew J Black 16 Oct '23

16 Oct '23

Hi All, I'm hoping someone(s) can point me in the correct direction to fix this. The Architecture +++++++++++++ 3 identical VM-Hosts (vmhost{1-3}.mydomain.local): - Rocky Linux 8.8 - 4 local gluster volumes - 4 iscsi volumes provided by 3-node Ceph Cluster (also running Rocky v8.8) - 3 NICS: - One Gluster Transfer - One Ceph Data Transfer - One General Data access / OVirt management Each node of the Ceph Cluster is running an iscsi gateway, and each vmhost connects to each iscsi gateway Everything is working AOK, except vmhost1 complains that it can't reach 3 of the 4 iscsi volumes/domains. I've looked in the engine.log and vdsm.log but can't figure out how to fix things, and I don't know where else to look. engine.log (extract) +++++++++++++++ ~~~ 2023-10-16 18:21:08,892+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-27) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Before acquiring lock in order to prevent monitoring for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock acquired, from now a monitoring of host will be skipped for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,181+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 732b1116 2023-10-16 18:21:09,245+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] FINISH, SetVdsStatusVDSCommand, return: , log id: 732b1116 2023-10-16 18:21:09,248+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Activate host finished. Lock released. Monitoring can run now for host 'vmhost1.mydomain.local' from data-center 'my_datacentre' 2023-10-16 18:21:09,314+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] EVENT_ID: VDS_ACTIVATE(16), Activation of host vmhost1.mydomain.local initiated by admin@internal-authz. 2023-10-16 18:21:09,373+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock freed to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}' 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] START, GetHardwareInfoAsyncVDSCommand(HostName = vmhost1.mydomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vds='Host[vmhost1.mydomain.local,8e0e99ab-c212-41c4-8318-acd34acb6289]'}), log id: 3dac178f 2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 3dac178f 2023-10-16 18:21:10,688+11 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [44be9d7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:21:10,690+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,703+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Running command: UpdateClusterCommand internal: true. Entities affected : ID: c892710d-8611-4146-aea2-05d910f1829b Type: ClusterAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] EVENT_ID: SYSTEM_UPDATE_CLUSTER(835), Host cluster my_cluster was updated by system 2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}' 2023-10-16 18:21:10,965+11 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,032+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 630966ec 2023-10-16 18:21:11,034+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 630966ec 2023-10-16 18:21:11,170+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool 2023-10-16 18:21:11,196+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='8d7b930e-a758-4010-aaf5-a230447f2761', connection='vmhosttran1.mydomain.local:/gv0', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {8d7b930e-a758-4010-aaf5-a230447f2761=0}, log id: 30a928ed 2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='044a086b-e94b-4e5d-a118-283ffceb8de8', connection='vmhosttran1.mydomain.local:gv1', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='2d328788-ade0-4e25-b947-1c392d4f133e', connection='vmhosttran1.mydomain.local:/gv2', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local :vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', connection='vmhosttran1.mydomain.local:/gv3', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 6aaeee49 2023-10-16 18:21:11,664+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [] Fetched 0 VMs from VDS '8e0e99ab-c212-41c4-8318-acd34acb6289' 2023-10-16 18:21:11,917+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {044a086b-e94b-4e5d-a118-283ffceb8de8=0, 2d328788-ade0-4e25-b947-1c392d4f133e=0, 420b9d49-eebb-4dea-99bb-bcb89fbe2bf2=0}, log id: 6aaeee49 2023-10-16 18:21:11,920+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='ISCSI', connectionList='[StorageServerConnections:{id='78f16782-9a06-45f2-92e5-d1ca84a8c67e', connection='192.168.4.24', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='868d7c6a-c933-4475-86ce-ab1359d6f4c3', connection='192.168.4.25', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='20a52c08-a4c5-4b07-8ecc-bec0e82e582a', connection='192.168.4.23', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 4e953edb 2023-10-16 18:21:19,245+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@a9992e5, log id: 694529cf 2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@1cf19aa, log id: 786d585b 2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@7ccef72, log id: 4104430d 2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 15f1c050 2023-10-16 18:21:19,251+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@6c8a990d, log id: 15f1c050 2023-10-16 18:21:34,401+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@5690567a, log id: 3f508f9e 2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@2a7c9b44, log id: 270bce0b 2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@459c7a2b, log id: 11ebddea 2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 3c1e9f05 2023-10-16 18:21:34,408+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@72f69579, log id: 3c1e9f05 2023-10-16 18:21:49,594+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 6a366a0f 2023-10-16 18:21:49,597+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@95edf66, log id: 6a366a0f 2023-10-16 18:21:49,655+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 40d156 2023-10-16 18:21:49,656+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@3f11df90, log id: 40d156 2023-10-16 18:23:28,802+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {78f16782-9a06-45f2-92e5-d1ca84a8c67e=465, 868d7c6a-c933-4475-86ce-ab1359d6f4c3=465, 20a52c08-a4c5-4b07-8ecc-bec0e82e582a=0}, log id: 4e953edb 2023-10-16 18:23:28,811+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Host 'vmhost1.mydomain.local' storage connection was succeeded 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] START, ConnectStoragePoolVDSCommand(HostName = vmhost1.mydomain.local, ConnectStoragePoolVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vdsId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', masterVersion='1'}), log id: 3b47a386 2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Executing with domain map: {e311ddf1-7f2c-49ef-a618-050d9a2b947f=active, fa8d922a-df3c-4f5b-adf6-a77c3689c397=active, 30775056-38ef-483a-889c-70453e9aa5f9=active, ee5b0268-8141-43e0-995e-8b4586217123=active, 3e33b7ce-7279-4765-a203-cca37eff8cb0=active, 4412e492-f752-4a6c-98ff-f289043618da=active, 0dde39a4-42e4-4fba-8068-e63e42dd3334=active, d44ec7f2-3629-4110-93a5-7d8d57230efc=active} 2023-10-16 18:23:32,385+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 3b47a386 2023-10-16 18:23:32,418+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '0dde39a4-42e4-4fba-8068-e63e42dd3334:iscsi-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '3e33b7ce-7279-4765-a203-cca37eff8cb0:gluster-generaldata' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'e311ddf1-7f2c-49ef-a618-050d9a2b947f:gluster-vmimages' report isn't an actual report 2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '4412e492-f752-4a6c-98ff-f289043618da:gluster-isos' report isn't an actual report 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'fa8d922a-df3c-4f5b-adf6-a77c3689c397:iscsi-databases' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '30775056-38ef-483a-889c-70453e9aa5f9:iscsi-generaldata' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'd44ec7f2-3629-4110-93a5-7d8d57230efc:iscsi-backups' was reported with error code '358' 2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-databases' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-generaldata' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-backups' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local' 2023-10-16 18:23:32,454+11 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host vmhost1.mydomain.local reports about one of the Active Storage Domains as Problematic. 2023-10-16 18:23:32,564+11 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 992c9e 2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Clearing domains data for host vmhost1.mydomain.local 2023-10-16 18:23:32,619+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] FINISH, SetVdsStatusVDSCommand, return: , log id: 992c9e 2023-10-16 18:23:32,689+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Host vmhost1.mydomain.local cannot access the Storage Domain(s) iscsi-databases, iscsi-generaldata, iscsi-backups attached to the Data Center my_datacentre. Setting Host state to Non-Operational. 2023-10-16 18:23:32,789+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for Host vmhost1.mydomain.local.There is no other host in the data center that can be used to test the power management settings. 2023-10-16 18:23:32,856+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Failed to connect Host vmhost1.mydomain.local to Storage Pool my_datacentre 2023-10-16 18:23:33,016+11 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:23:33,018+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Host 'vmhost1.mydomain.local'(8e0e99ab-c212-41c4-8318-acd34acb6289) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped. 2023-10-16 18:23:33,019+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Received first domain report for host vmhost1.mydomain.local 2023-10-16 18:24:02,559+11 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2023-10-16 18:24:02,561+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 51d95bef 2023-10-16 18:24:02,592+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetVdsStatusVDSCommand, return: , log id: 51d95bef 2023-10-16 18:24:02,799+11 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS 2023-10-16 18:24:02,827+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 63f61978 2023-10-16 18:24:02,832+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 63f61978 2023-10-16 18:24:02,914+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Host vmhost1.mydomain.local was switched to Maintenance mode by admin@internal-authz. ~~~ vdsm.log (extract - from vmhost1) ++++++++++++++++++++++++++ ~~~ 2023-10-16 18:21:10,069+1100 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,410+1100 ERROR (jsonrpc/7) [root] Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config imp ort cfg\\n File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) (caps:165) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get caps["connector_info"] = managedvolume.connector_info() File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info return run_helper("connector_info") File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper sub_cmd, cmd_input=cmd_input) File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda> **kwargs) File "<string>", line 2, in managedvolume_run_helper File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod raise convert_to_error(kind, result) vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config import cfg\\n File "/usr/lib/python3.6/site-packag es/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) 2023-10-16 18:21:10,600+1100 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err=b'' (hooks:106) 2023-10-16 18:21:10,601+1100 INFO (jsonrpc/7) [api.host] FINISH getCapabilities return={'status': {'code': 0, 'message': 'Done'}, 'info': {'kvmEnabled': 'true', 'cpuCores': '16', 'cpuThreads': '32', 'cpuSockets': '2', 'onlineCpus': '0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23,8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31', 'cpuTopology': [{'cpu_id': 0, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 1, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 2, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 3, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 4, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 5, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 6, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 7, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 16, 'numa_cell_id': 0, 'socket_id ': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 17, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 18, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 19, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 20, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 21, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 22, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 23, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 8, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 9, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 10, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 11, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 12, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 13, 'numa_cell_id' : 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 14, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 15, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}, {'cpu_id': 24, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 25, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 26, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 27, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 28, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 29, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 30, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 31, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}], 'cpuSpeed': '4000.000', 'cpuModel': 'Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz', 'cpuFlags': 'pku,clflushopt,avx2,nopl,pni,pln,dca,smep,pat,art,cx8,msr,pts,arch_capa bilities,xsave,ht,lm,smap,de,mce,pse36,cx16,skip-l1dfl-vmentry,cpuid,bts,lahf_lm,avx512vl,tsc_deadline_timer,bmi2,md-clear,tm2,cmov,sep,sse4_1,invtsc,md_clear,intel_pt,amd-ssbd,mpx,ssse3,rdt_a,cpuid_fault,x2apic,mmx,avx512dq,dtes64,avx512cd,vme,pclmulqdq,cqm_occup_llc,xsavec,ibpb,umip,pschange-mc-no,mtrr,avx512f,stibp,clwb,xsaves,f16c,invpcid_single,avx512bw,arat,pdpe1gb,ibrs,ept,tm,invpcid,cqm_mbm_total,ibrs-all,3dnowprefetch,popcnt,cqm,rdctl-no,xsaveopt,pdcm,aperfmperf,cat_l3,xgetbv1,xtpr,syscall,smx,pbe,sse2,amd-stibp,pebs,xtopology,tsc_adjust,mca,pae,ida,nonstop_tsc,tpr_shadow,ssbd,nx,tsc,dtherm,aes,fxsr,fma,cqm_mbm_local,movbe,erms,intel_ppin,mds-no,ds_cpl,rdseed,dts,ibrs_enhanced,constant_tsc,ept_ad,flexpriority,avx512_vnni,adx,clflush,acpi,cdp_l3,sse,rdtscp,fsgsbase,ss,bmi1,arch_perfmon,monitor,pse,ospke,pcid,rdrand,hypervisor,flush_l1d,tsx-ctrl,arch-capabilities,vmx,avx,pge,rep_good,vpid,sdbg,sse4_2,mba,cqm_llc,est,epb,fpu,vnmi,apic,abm,spec_ctrl,model_pentium3,model_Penryn, model_Cascadelake-Server-noTSX,model_Haswell-noTSX-IBRS,model_Haswell-noTSX,model_kvm32,model_Nehalem-IBRS,model_pentium,model_Opteron_G1,model_IvyBridge-IBRS,model_Westmere,model_n270,model_qemu32,model_coreduo,model_486,model_IvyBridge,model_SandyBridge,model_Nehalem,model_Skylake-Client-noTSX-IBRS,model_pentium2,model_Westmere-IBRS,model_SandyBridge-IBRS,model_Conroe,model_qemu64,model_Opteron_G2,model_kvm64,model_Broadwell-noTSX,model_Skylake-Server-noTSX-IBRS,model_Broadwell-noTSX-IBRS,model_core2duo', 'vdsmToCpusAffinity': [1], 'version_name': 'Snow Man', 'software_version': '4.50.3.4', 'software_revision': '1', 'supportedENGINEs': ['4.2', '4.3', '4.4', '4.5'], 'clusterLevels': ['4.2', '4.3', '4.4', '4.5', '4.6', '4.7'], 'networks': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'iface': 'ovirtmgmt', 'bridged': True, 'addr': '192.168.2.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6 gateway': '::', 'ipv4defaultroute': True, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.20', 'vlanid': 20, 'dhcpv4': False, 'dhcpv6': False}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'iface': 'DMZ', 'bridged': True, 'addr': '192.168.1.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.10', 'vlanid': 10, 'dhcpv4': False, 'dhcpv6': False}, 'Migration': {'interface': 'ens1', 'iface': 'ens1', 'bridged': False, 'addr': '192.168.5.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'ens1', 'dhcpv4': False, 'dhcpv6': False}, 'SAN': {'interface': 'bond0.40', 'iface': 'bond0.40', 'bridged': False, 'addr': '192.168.4.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'bond0.40', 'vlanid': 40, 'dhcpv4': False, 'dhcpv6': False}}, 'bondings': {'bond0': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'slaves': ['eno1', 'eno2'], 'active_slave': '', 'opts': {'updelay': '1000', 'mode': '4', 'miimon': '1000', 'downdelay': '1000'}, 'ad_aggregator_id': '1', 'ad_partner_mac': '10:b3:c6:ae:a8:2d', 'switch': 'legacy', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 2000}}, 'bridges': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'addr': '192.168.2.20', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '17569', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '1 00', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'addr': '192.168.1.20', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '8995', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multi cast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'virbr0': {'ports': [], 'stp': 'on', 'addr': '192.168.122.1', 'ipv4addrs': ['192.168.122.1/24'], 'ipv6addrs': [], 'ipv6autoconf' : False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '22103', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.525400a8ef1c', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '103', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '1', 'multicast_start up_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.525400a8ef1c', 'no_linklocal_learn': '0', 'forward_delay': '200', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}}, 'nics': {'ens1': {'hwaddr': '00:1b:21:e4:a6:f4', 'addr': '192.168.5.20', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}, 'eno1': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e8', 'dhcpv4': False, 'dhcpv 6': False, 'speed': 1000}, 'eno2': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e9', 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}}, 'vlans': {'bond0.20': {'iface': 'bond0', 'vlanid': 20, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.10': {'iface': 'bond0', 'vlanid': 10, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.40': {'iface': 'bond0', 'vlanid': 40, 'addr': '192.168.4.20', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6 gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}}, 'nameservers': ['192.168.2.16', '192.168.2.17'], 'supportsIPv6': True, 'ovnConfigured': False, 'hooks': {'after_get_caps': {'ovirt_provider_ovn_hook': {'checksum': 'a2bdefca38b96c8ddab39822cc8282bf3f67d875c4879003ffc9661826c92421'}}, 'before_device_create': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_nic_hotplug': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_vm_start': {'50_hostedengine': {'checksum': 'e5f5262cf22e06cd34e227afb27647e479351266876019a64210dbcbd2a43830'}}}, 'operatingSystem': {'release': '1.8.el8', 'version': '8.8', 'name': 'RHEL', 'pretty_name': 'Rocky Linux 8.8 (Green Obsidian)'}, 'uuid': '18e93400-808a-11ea-8000-ac1f6b3ccce8', 'packages2': {'kernel': {'version': '4.18.0', 'release': '477.27.1.e l8_8.x86_64'}, 'glusterfs-cli': {'version': '10.4', 'release': '1.el8s'}, 'librbd1': {'version': '16.2.13', 'release': '1.el8s'}, 'libvirt': {'version': '8.0.0', 'release': '19.2.module+el8.8.0+1321+70d049b2'}, 'mom': {'version': '0.6.4', 'release': '1.el8'}, 'ovirt-hosted-engine-ha': {'version': '2.5.0', 'release': '1.el8'}, 'openvswitch': {'version': '2.15', 'release': '4.el8'}, 'nmstate': {'version': '1.4.4', 'release': '2.el8_8'}, 'qemu-img': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'qemu-kvm': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'spice-server': {'version': '0.14.3', 'release': '4.el8'}, 'vdsm': {'version': '4.50.3.4', 'release': '1.el8'}, 'glusterfs': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-fuse': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-geo-replication': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-server': {'version': '10.4', 'release': '1.el8s'}}, 'realtimeKernel': False, 'kernelArgs' : 'BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-477.27.1.el8_8.x86_64 root=UUID=9acf17b0-558a-46d3-b21c-74c208680fcb ro crashkernel=auto resume=UUID=9c5f897c-8e34-48a7-9f06-786af3c079e9', 'nestedVirtualization': True, 'emulatedMachines': ['pc-q35-rhel8.0.0', 'pc-q35-rhel7.4.0', 'pc-i440fx-rhel7.3.0', 'pc-q35-rhel8.2.0', 'pc-q35-rhel7.3.0', 'pc-i440fx-rhel7.4.0', 'pc-q35-rhel8.4.0', 'pc-q35-rhel7.6.0', 'pc-i440fx-rhel7.5.0', 'pc-q35-rhel8.1.0', 'pc-q35-rhel8.6.0', 'pc-q35-rhel8.3.0', 'pc-q35-rhel7.5.0', 'pc-q35-rhel8.5.0', 'pc-i440fx-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'pc', 'pc-i440fx-rhel7.6.0', 'pc-i440fx-rhel7.0.0'], 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d', 'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d'}], 'FC': []}, 'vmTypes': ['kvm'], 'memSize': '63805', 'reservedMem': '321', 'guestOverhead': '65', 'rngSources': ['random', 'hwrng'], 'numaNodes': {'0': {'totalMemory': '31553', 'hugepages': {'4': {'totalPages': '8077814'}, ' 2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23]}, '1': {'totalMemory': '32251', 'hugepages': {'4': {'totalPages': '8256270'}, '2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31]}}, 'numaNodeDistance': {'0': [10, 21], '1': [21, 10]}, 'autoNumaBalancing': 2, 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'liveMerge': 'true', 'kdumpStatus': 1, 'deferred_preallocation': True, 'hostdevPassthrough': 'false', 'additionalFeatures': ['libgfapi_supported', 'GLUSTER_SNAPSHOT', 'GLUSTER_GEO_REPLICATION', 'GLUSTER_BRICK_MANAGEMENT'], 'hostedEngineDeployed': True, 'hugepages': [2048, 1048576], 'kernelFeatures': {'SPECTRE_V2': '(Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: SW sequence)', 'ITLB_MULTIHIT': '(KVM: Mitigation: VMX disabled)', 'MMIO_STALE_DATA': '(Mitigation: Clear CPU buffers; SMT vulnerable)', 'MDS ': '(Not affected)', 'L1TF': '(Not affected)', 'SPEC_STORE_BYPASS': '(Mitigation: Speculative Store Bypass disabled via prctl)', 'TSX_ASYNC_ABORT': '(Mitigation: TSX disabled)', 'SPECTRE_V1': '(Mitigation: usercopy/swapgs barriers and __user pointer sanitization)', 'RETBLEED': '(Mitigation: Enhanced IBRS)', 'SRBDS': '(Not affected)', 'MELTDOWN': '(Not affected)'}, 'vncEncrypted': True, 'backupEnabled': True, 'coldBackupEnabled': True, 'clearBitmapsEnabled': True, 'fipsEnabled': False, 'boot_uuid': '68e5a5ca-3dd5-4103-9613-6065a5fc74da', 'tscFrequency': '3199999000', 'tscScaling': True, 'domain_versions': [0, 2, 3, 4, 5], 'supported_block_size': {'FCP': [512], 'GLUSTERFS': [0, 512, 4096], 'ISCSI': [512], 'LOCALFS': [0, 512, 4096], 'NFS': [512], 'POSIXFS': [512]}, 'cd_change_pdiv': True, 'refresh_disk_supported': True, 'replicate_extend': True, 'measure_subchain': True, 'measure_active': True, 'mailbox_events': True, 'zerocopy_migrations': True, 'netConfigDirty': 'False', 'openstack_b inding_host_ids': {'OVIRT_PROVIDER_OVN': 'vmhost1.mydoamin.local'}, 'lastClientIface': 'ovirtmgmt'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,620+1100 INFO (jsonrpc/1) [api.host] START getHardwareInfo() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:10,621+1100 INFO (jsonrpc/1) [api.host] FINISH getHardwareInfo return={'status': {'code': 0, 'message': 'Done'}, 'info': {'systemManufacturer': 'Supermicro', 'systemProductName': 'X11DPi-N(T)', 'systemVersion': '123456789', 'systemSerialNumber': '', 'systemUUID': '0034e918-8a80-ea11-8000-ac1f6b3ccce8', 'systemFamily': 'SMC X11'}} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:10,767+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:31) 2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:37) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:31) 2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:37) 2023-10-16 18:21:10,786+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:37) 2023-10-16 18:21:11,032+1100 INFO (jsonrpc/6) [vds] Setting Hosted Engine HA local maintenance to False (API:1697) 2023-10-16 18:21:11,198+1100 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv0', 'ipv6_enabled': 'false', 'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:31) 2023-10-16 18:21:11,199+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Shutdown requested (__init__:126) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Shutdown requested (__init__:126) 2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,202+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Closing client (__init__:641) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:37) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,216+1100 INFO (jsonrpc/2) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:gv1', 'ipv6_enabled': 'false', 'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv2', 'ipv6_enabled': 'false', 'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv3', 'i pv6_enabled': 'false', 'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:31) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1' (storageServer:217) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 mode: None (fileUtils:214) 2023-10-16 18:21:11,293+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:gv1 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 (mount:190) 2023-10-16 18:21:11,415+1100 INFO (jsonrpc/2) [IOProcessClient] (Global) Starting client (__init__:340) 2023-10-16 18:21:11,424+1100 INFO (ioprocess/86760) [IOProcess] (Global) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2' (storageServer:217) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 mode: None (fileUtils:214) 2023-10-16 18:21:11,518+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv2 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 (mount:190) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3' (storageServer:217) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 mode: None (fileUtils:214) 2023-10-16 18:21:11,730+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326) 2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv3 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 (mount:190) 2023-10-16 18:21:11,875+1100 INFO (tmap-753/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:21:11,883+1100 INFO (tmap-753/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:21:11,885+1100 INFO (ioprocess/87016) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,889+1100 INFO (tmap-753/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:21:11,892+1100 INFO (ioprocess/87022) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,898+1100 INFO (ioprocess/87027) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Closing client (__init__:641) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Closing client (__init__:641) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,904+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Killing ioprocess (__init__:191) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,905+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,906+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194) 2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:21:11,915+1100 INFO (jsonrpc/2) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'status': 0}, {'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'status': 0}, {'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:37) 2023-10-16 18:21:11,924+1100 INFO (jsonrpc/3) [vdsm.api] START connectStorageServer(domType=3, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsichapinitiator', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:31) 2023-10-16 18:21:11,925+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,053+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,174+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192) 2023-10-16 18:21:12,293+1100 INFO (jsonrpc/3) [storage.storageServer] Log in to 3 targets using 3 workers (storageServer:600) 2023-10-16 18:21:12,294+1100 INFO (iscsi-login/0) [storage.iscsi] Logging in to iscsi target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,297+1100 INFO (iscsi-login/1) [storage.iscsi] Logging in to iscsi target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:12,303+1100 INFO (iscsi-login/2) [storage.iscsi] Logging in to iscsi target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:31) 2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:37) 2023-10-16 18:21:13,676+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:37) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:31) 2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:37) 2023-10-16 18:21:13,693+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:31) 2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:37) 2023-10-16 18:21:15,485+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:37) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:31) 2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:37) 2023-10-16 18:21:15,503+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:16,692+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:16,693+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:31) 2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:37) 2023-10-16 18:21:17,366+1100 INFO (jsonrpc/1) [vdsm.api] START startMonitoringDomain(sdUUID='ee5b0268-8141-43e0-995e-8b4586217123', hostID=1) from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:31) 2023-10-16 18:21:17,367+1100 INFO (jsonrpc/1) [storage.monitor] Start monitoring ee5b0268-8141-43e0-995e-8b4586217123 (monitor:176) 2023-10-16 18:21:17,368+1100 INFO (jsonrpc/1) [vdsm.api] FINISH startMonitoringDomain return=None from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:37) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:21:17,377+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:31) 2023-10-16 18:21:17,378+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.0', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}} from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:37) 2023-10-16 18:21:17,489+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices: 0.12 seconds (utils:373) 2023-10-16 18:21:17,490+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache: 2.22 seconds (utils:373) 2023-10-16 18:21:19,594+1100 INFO (monitor/ee5b026) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Starting client (__init__:340) 2023-10-16 18:21:19,603+1100 INFO (ioprocess/87289) [IOProcess] (ee5b0268-8141-43e0-995e-8b4586217123) Starting ioprocess (__init__:465) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123/dom_md/metadata' (check:104) 2023-10-16 18:21:20,777+1100 INFO (check/loop) [storage.monitor] Domain ee5b0268-8141-43e0-995e-8b4586217123 became VALID (monitor:465) 2023-10-16 18:21:20,779+1100 INFO (event/70) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:31) 2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:37) 2023-10-16 18:21:23,720+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:37) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:31) 2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:37) 2023-10-16 18:21:24,421+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:31) 2023-10-16 18:21:24,422+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:37) 2023-10-16 18:21:25,529+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:31) 2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:37) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:31) 2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:37) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:31) 2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:37) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:31) 2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:37) 2023-10-16 18:21:28,741+1100 WARN (jsonrpc/1) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:28,742+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:28,744+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,548+1100 WARN (jsonrpc/5) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:30,549+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:30,550+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:30,763+1100 INFO (monitor/ee5b026) [storage.sanlock] Acquiring host id for domain ee5b0268-8141-43e0-995e-8b4586217123 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:32,411+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:31) 2023-10-16 18:21:32,412+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:37) 2023-10-16 18:21:33,697+1100 WARN (periodic/0) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:33,753+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:37) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:31) 2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:37) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:31) 2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:37) 2023-10-16 18:21:35,562+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:35,573+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:37) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:31) 2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:31) 2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:37) 2023-10-16 18:21:38,773+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:38,774+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:38,775+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:40,581+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:40,582+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:40,583+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:42,430+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:31) 2023-10-16 18:21:42,431+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:37) 2023-10-16 18:21:43,788+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:37) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:31) 2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:37) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:31) 2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:37) 2023-10-16 18:21:45,595+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:37) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:31) 2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:37) 2023-10-16 18:21:45,610+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:45,610+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {} (throttledlog:87) 2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:47,440+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:31) 2023-10-16 18:21:47,441+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:37) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:31) 2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:37) 2023-10-16 18:21:48,715+1100 WARN (periodic/3) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:48,808+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155) 2023-10-16 18:21:48,809+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:48,810+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:31) 2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:37) 2023-10-16 18:21:53,821+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:31) 2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:37) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:31) 2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:37) 2023-10-16 18:21:53,837+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:53,837+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:31) 2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:37) 2023-10-16 18:21:55,639+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:37) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:31) 2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:37) 2023-10-16 18:21:55,653+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155) 2023-10-16 18:21:55,653+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:31) 2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:37) 2023-10-16 18:22:00,800+1100 INFO (monitor/ee5b026) [storage.sanlock] Host id 1 for domain ee5b0268-8141-43e0-995e-8b4586217123 acquired in 30 seconds (clusterlock:427) 2023-10-16 18:22:01,744+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:01,745+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:31) 2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:37) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:31) 2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:37) 2023-10-16 18:22:03,663+1100 INFO (health) [health] LVM cache hit ratio: 39.35% (hits: 170 misses: 262) (health:114) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:31) 2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:37) 2023-10-16 18:22:03,862+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:31) 2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:37) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:31) 2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:37) 2023-10-16 18:22:03,879+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:31) 2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:37) 2023-10-16 18:22:05,682+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:37) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:31) 2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:37) 2023-10-16 18:22:05,699+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=60.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:31) 2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:37) 2023-10-16 18:22:13,908+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:37) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:31) 2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:37) 2023-10-16 18:22:13,927+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:31) 2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:37) 2023-10-16 18:22:15,726+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:37) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:31) 2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:37) 2023-10-16 18:22:15,743+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:31) 2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:37) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (Global) Closing client (__init__:641) 2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Closing client (__init__:641) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,814+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Killing ioprocess (__init__:191) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,815+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:20,816+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) ioprocess was terminated by signal 9 (__init__:200) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:31) 2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.9', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:37) 2023-10-16 18:22:23,954+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:37) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:31) 2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:37) 2023-10-16 18:22:23,974+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:31) 2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:37) 2023-10-16 18:22:25,769+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:37) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:31) 2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:37) 2023-10-16 18:22:25,786+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:31,782+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:31,783+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:32,825+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:31) 2023-10-16 18:22:32,826+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:37) 2023-10-16 18:22:33,771+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:31) 2023-10-16 18:22:33,772+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:37) 2023-10-16 18:22:33,998+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:37) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:31) 2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:37) 2023-10-16 18:22:34,014+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:34,515+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:31) 2023-10-16 18:22:34,516+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:37) 2023-10-16 18:22:35,812+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:37) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:31) 2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:37) 2023-10-16 18:22:35,830+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:37,110+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:31) 2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:37) 2023-10-16 18:22:44,040+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:37) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:31) 2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:37) 2023-10-16 18:22:44,059+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:31) 2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:37) 2023-10-16 18:22:45,855+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:37) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:31) 2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:37) 2023-10-16 18:22:45,873+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:46,786+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:22:46,787+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:31) 2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:37) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:31) 2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:37) 2023-10-16 18:22:54,086+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:37) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:31) 2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:37) 2023-10-16 18:22:54,106+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:31) 2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:37) 2023-10-16 18:22:55,899+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:37) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:31) 2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:37) 2023-10-16 18:22:55,916+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:22:57,142+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:31) 2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.2', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:37) 2023-10-16 18:23:03,813+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:31) 2023-10-16 18:23:03,814+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:37) 2023-10-16 18:23:04,130+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:31) 2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:37) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:31) 2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:37) 2023-10-16 18:23:04,147+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:31) 2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:37) 2023-10-16 18:23:05,942+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:31) 2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:37) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:31) 2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:37) 2023-10-16 18:23:05,960+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=120.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback: File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner self.run() File: "/usr/lib64/python3.6/threading.py", line 885, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod result = fn(*methodArgs) File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer connectionParams) File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper result = ctask.prepare(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper return m(self, *a, **kw) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare result = self._run(func, *args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run return fn(*args, **kargs) File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer results = storageServer.connect(domType, conList) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect return con_class.connect_all(connections) File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all for res in login_results: File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap yield results.get() File: "/usr/lib64/python3.6/queue.py", line 164, in get self.not_empty.wait() File: "/usr/lib64/python3.6/threading.py", line 295, in wait waiter.acquire() (executor:346) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:31) 2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:37) 2023-10-16 18:23:14,173+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:31) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:37) 2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:31) 2023-10-16 18:23:14,186+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:37) 2023-10-16 18:23:14,192+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:31) 2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:37) 2023-10-16 18:23:15,986+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:37) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:31) 2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:37) 2023-10-16 18:23:16,003+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:31) 2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:37) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:31) 2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:37) 2023-10-16 18:23:24,217+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:37) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:31) 2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:37) 2023-10-16 18:23:24,236+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:31) 2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:37) 2023-10-16 18:23:26,030+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:37) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:31) 2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:37) 2023-10-16 18:23:26,047+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:26,373+1100 INFO (iscsi-login/0) [storage.iscsi] Removing iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,388+1100 INFO (iscsi-login/1) [storage.iscsi] Removing iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227) 2023-10-16 18:23:26,436+1100 ERROR (iscsi-login/0) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.24:3260,3', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,497+1100 ERROR (iscsi-login/1) [storage.storageServer] Could not login to target (storageServer:607) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login "-p", portal, "-l"]) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd out = commands.run(cmd, sudo=True) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.25:3260,2', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login iscsi.loginToIscsiNode(con.iface, con.target) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode iscsiadm.node_login(iface.name, target.address, target.iqn) File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login raise IscsiNodeError(e.rc, e.out, e.err) vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n') 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices: 0.09 seconds (utils:373) 2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Removing domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 from storage domain cache (sdc:194) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'status': 0}, {'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'status': 465}, {'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'status': 465}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:37) 2023-10-16 18:23:28,797+1100 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer took more than 1.00 seconds to succeed: 136.87 (__init__:305) 2023-10-16 18:23:28,847+1100 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', hostID=1, msdUUID='ee5b0268-8141-43e0-995e-8b4586217123', masterVersion=1, domainsMap={'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'active', '30775056-38ef-483a-889c-70453e9aa5f9': 'active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'active', '4412e492-f752-4a6c-98ff-f289043618da': 'active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'active'}) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:31) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepoolmemorybackend] new storage pool master version 1 and domains map {'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'Active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'Active', '30775056-38ef-483a-889c-70453e9aa5f9': 'Active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'Active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'Active', '4412e492-f752-4a6c-98ff-f289043618da': 'Active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'Active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'Active'} (spbackends:417) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] updating pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 backend from type NoneType instance 0x7f87cc236b70 to type StoragePoolMemoryBackend instance 0x7f877a7cdcc8 (sp:149) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] Connect host #1 to the storage pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 with master domain: ee5b0268-8141-43e0-995e-8b4586217123 (ver = 1) (sp:699) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Clearing storage domain cache (sdc:182) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63) 2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices (iscsi:445) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices: 0.10 seconds (utils:373) 2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices (hba:42) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373) 2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.multipath] Waiting until multipathd is ready (multipath:95) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122) 2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices (multipath:223) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373) 2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373) 2023-10-16 18:23:31,035+1100 INFO (tmap-757/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting client (__init__:340) 2023-10-16 18:23:31,043+1100 INFO (tmap-757/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340) 2023-10-16 18:23:31,045+1100 INFO (ioprocess/87928) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,052+1100 INFO (tmap-757/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340) 2023-10-16 18:23:31,054+1100 INFO (ioprocess/87936) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,059+1100 INFO (tmap-757/3) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340) 2023-10-16 18:23:31,062+1100 INFO (ioprocess/87940) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,068+1100 INFO (ioprocess/87948) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465) 2023-10-16 18:23:31,842+1100 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:31,843+1100 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:32,194+1100 INFO (jsonrpc/5) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 0dde39a4-42e4-4fba-8068-e63e42dd3334 (monitor:176) 2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 3e33b7ce-7279-4765-a203-cca37eff8cb0 (monitor:176) 2023-10-16 18:23:32,201+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring e311ddf1-7f2c-49ef-a618-050d9a2b947f (monitor:176) 2023-10-16 18:23:32,206+1100 INFO (jsonrpc/5) [storage.monitor] Attaching monitor for ee5b0268-8141-43e0-995e-8b4586217123 to the pool (monitor:172) 2023-10-16 18:23:32,207+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 4412e492-f752-4a6c-98ff-f289043618da (monitor:176) 2023-10-16 18:23:32,208+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring fa8d922a-df3c-4f5b-adf6-a77c3689c397 (monitor:176) 2023-10-16 18:23:32,210+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 30775056-38ef-483a-889c-70453e9aa5f9 (monitor:176) 2023-10-16 18:23:32,211+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:32,218+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring d44ec7f2-3629-4110-93a5-7d8d57230efc (monitor:176) 2023-10-16 18:23:32,219+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:32,230+1100 INFO (monitor/3e33b7c) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting client (__init__:340) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.storagepool] Creating pool directory '/rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646' (sp:1323) 2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646 mode: None (fileUtils:214) 2023-10-16 18:23:32,239+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:32,260+1100 INFO (monitor/e311ddf) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting client (__init__:340) 2023-10-16 18:23:32,263+1100 INFO (ioprocess/87989) [IOProcess] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,268+1100 INFO (monitor/4412e49) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Starting client (__init__:340) 2023-10-16 18:23:32,271+1100 INFO (ioprocess/88003) [IOProcess] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,278+1100 INFO (ioprocess/88012) [IOProcess] (4412e492-f752-4a6c-98ff-f289043618da) Starting ioprocess (__init__:465) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.11 seconds (utils:373) 2023-10-16 18:23:32,323+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.monitor] Domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 became INVALID (monitor:465) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.11 seconds (utils:373) 2023-10-16 18:23:32,334+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.monitor] Domain 30775056-38ef-483a-889c-70453e9aa5f9 became INVALID (monitor:465) 2023-10-16 18:23:32,339+1100 WARN (jsonrpc/5) [storage.lvm] Marked vgs=[0dde39a4-42e4-4fba-8068-e63e42dd3334] as Unreadable due to reload failure (lvm:679) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain metadata directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (sp:1354) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (sp:1357) 2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.10 seconds (utils:373) 2023-10-16 18:23:32,340+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.monitor] Domain d44ec7f2-3629-4110-93a5-7d8d57230efc became INVALID (monitor:465) 2023-10-16 18:23:32,345+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1295) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1278) 2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1295) 2023-10-16 18:23:32,383+1100 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:37) 2023-10-16 18:23:32,384+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect took more than 1.00 seconds to succeed: 3.54 (__init__:305) 2023-10-16 18:23:32,385+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['ids'] (lvm:1839) 2023-10-16 18:23:32,390+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actua l': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:37) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:31) 2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:37) 2023-10-16 18:23:32,413+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:37) 2023-10-16 18:23:32,572+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['leases'] (lvm:1839) 2023-10-16 18:23:32,771+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['metadata', 'inbox', 'outbox', 'xleases', 'master'] (lvm:1839) 2023-10-16 18:23:32,975+1100 INFO (monitor/0dde39a) [storage.blocksd] Setting up domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (blockSD:973) 2023-10-16 18:23:33,056+1100 INFO (monitor/0dde39a) [storage.check] Start checking '/dev/0dde39a4-42e4-4fba-8068-e63e42dd3334/metadata' (check:104) 2023-10-16 18:23:33,143+1100 INFO (monitor/0dde39a) [storage.monitor] Domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 became VALID (monitor:465) 2023-10-16 18:23:33,144+1100 INFO (monitor/0dde39a) [storage.sanlock] Acquiring host id for domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:33,145+1100 INFO (event/81) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,145+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (blockSD:766) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain special volumes directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (blockSD:771) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214) 2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:31) 2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:37) 2023-10-16 18:23:33,470+1100 INFO (monitor/3e33b7c) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,471+1100 INFO (monitor/3e33b7c) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0/dom_md/metadata' (check:104) 2023-10-16 18:23:33,474+1100 INFO (monitor/4412e49) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,475+1100 INFO (monitor/4412e49) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da/dom_md/metadata' (check:104) 2023-10-16 18:23:33,519+1100 INFO (check/loop) [storage.monitor] Domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 became VALID (monitor:465) 2023-10-16 18:23:33,521+1100 INFO (event/84) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,522+1100 INFO (check/loop) [storage.monitor] Domain 4412e492-f752-4a6c-98ff-f289043618da became VALID (monitor:465) 2023-10-16 18:23:33,523+1100 INFO (event/85) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717) 2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f/dom_md/metadata' (check:104) 2023-10-16 18:23:33,528+1100 INFO (event/87) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,534+1100 INFO (event/88) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278) 2023-10-16 18:23:33,542+1100 INFO (check/loop) [storage.monitor] Domain e311ddf1-7f2c-49ef-a618-050d9a2b947f became VALID (monitor:465) 2023-10-16 18:23:33,543+1100 INFO (event/90) [vds] vm_start_stop_lock acquired (clientIF:230) 2023-10-16 18:23:33,543+1100 INFO (event/91) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:31) 2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:37) 2023-10-16 18:23:34,263+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '2.0', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:37) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:31) 2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:37) 2023-10-16 18:23:34,280+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:31) 2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:37) 2023-10-16 18:23:36,073+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:36,084+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': - 1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:37) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:31) 2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:37) 2023-10-16 18:23:36,090+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:41,693+1100 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:41,694+1100 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:42,324+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:42,335+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:42,341+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:42,423+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.09 seconds (utils:373) 2023-10-16 18:23:42,423+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:42,426+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.10 seconds (utils:373) 2023-10-16 18:23:42,426+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:31) 2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:37) 2023-10-16 18:23:42,432+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.09 seconds (utils:373) 2023-10-16 18:23:42,432+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:43,488+1100 INFO (monitor/4412e49) [storage.sanlock] Acquiring host id for domain 4412e492-f752-4a6c-98ff-f289043618da (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,488+1100 INFO (monitor/3e33b7c) [storage.sanlock] Acquiring host id for domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:43,541+1100 INFO (monitor/e311ddf) [storage.sanlock] Acquiring host id for domain e311ddf1-7f2c-49ef-a618-050d9a2b947f (id=1, wait=False) (clusterlock:359) 2023-10-16 18:23:44,308+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.2', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:37) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:31) 2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:37) 2023-10-16 18:23:44,328+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:31) 2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:37) 2023-10-16 18:23:46,096+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:31) 2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:37) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:31) 2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:37) 2023-10-16 18:23:46,113+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:46,121+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:37) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:31) 2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:37) 2023-10-16 18:23:46,137+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:23:47,219+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:31) 2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '5.7', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '5.4', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '6.4', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:37) 2023-10-16 18:23:52,424+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:23:52,428+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:23:52,433+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:31) 2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:37) 2023-10-16 18:23:52,542+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.12 seconds (utils:373) 2023-10-16 18:23:52,543+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:23:52,546+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.11 seconds (utils:373) 2023-10-16 18:23:52,546+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:23:52,551+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.12 seconds (utils:373) 2023-10-16 18:23:52,551+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:23:53,353+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '9.9', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:37) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:31) 2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:37) 2023-10-16 18:23:53,373+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:31) 2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:37) 2023-10-16 18:23:56,164+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.7', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.7', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:37) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:31) 2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:37) 2023-10-16 18:23:56,182+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37) 2023-10-16 18:23:56,696+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:23:56,697+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37) 2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301) 2023-10-16 18:24:01,111+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:31) 2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.4', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '7.6', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:37) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:31) 2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:37) 2023-10-16 18:24:01,130+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31) 2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37) 2023-10-16 18:24:02,543+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154) 2023-10-16 18:24:02,547+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154) 2023-10-16 18:24:02,552+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:31) 2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:37) 2023-10-16 18:24:02,744+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.20 seconds (utils:373) 2023-10-16 18:24:02,744+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',) 2023-10-16 18:24:02,755+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.21 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',) 2023-10-16 18:24:02,755+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.20 seconds (utils:373) 2023-10-16 18:24:02,755+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop self._setupMonitor() File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor self._setupDomain() File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper value = meth(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain domain = sdCache.produce(self.sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce domain.getRealDomain() File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce domain = self._findDomain(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain return findMethod(sdUUID) File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',) 2023-10-16 18:24:02,828+1100 INFO (jsonrpc/3) [vds] Setting Hosted Engine HA local maintenance to True (API:1697) ~~~ Further info can be provided if required (I don't know what else to include) Thanks in advance for any and all help.

1 0

Ansible - export_ova
by duparchy＠esrf.fr 16 Oct '23

16 Oct '23

We're using a script to export some VMs. This involves export_ova task - Is there a way to parallelize OVA exports (running on different hosts) ? When exporting to OVA from the GUI, we have to select a host, but ansible export_ova has no "hosts" parameter.

3 2

What is the status of the whole Ovirt Project?
by Arman Khalatyan 15 Oct '23

15 Oct '23

Hello everybody, What is the status of the ovirt project, would be continued it in the Rocky Linunx9? the last message is so sad: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/HEKKBM6MZEKBEA… Any good news/progress there? As a happy ovirt user since 2016 should we move to an another system? Thank you beforehand, Arman.

9 14

Ansible - export_ova
by duparchy＠esrf.fr 13 Oct '23

13 Oct '23

We're using a script to export some VMs. This involves export_ova task - Is there a way to parallelize OVA exports (running on different hosts) ? When exporting to OVA from the GUI, we have to select a host, but ansible export_ova has no "hosts" parameter.

1 0

How do i to drive Intel 82599ES 10-Gigabit SFP+ Adapter
by ejzhang＠hotmail.com 13 Oct '23

13 Oct '23

I can find the Adapter. [root@localhost ~]# lspci | grep 82599 06:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) But cannot to install the driver, I try to upgrade kernel to 6.1.57, but the problem is remains. How can i to solve the problem? Thanks very much! [root@localhost tmp]# dnf localinstall kmod-ixgbe-5.19.6-1.rhel9u1.x86_64.rpm Last metadata expiration check: 0:24:26 ago on Thu 12 Oct 2023 05:25:27 PM CST. Error: Problem: conflicting requests - nothing provides kernel(__SCK__tp_func_xdp_exception) = 0xc15ae5f5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(___pskb_trim) = 0x80ed79e7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__alloc_pages) = 0x26458776 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__alloc_skb) = 0xbc60587f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__dev_kfree_skb_any) = 0x8d62db77 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__free_pages) = 0x7ddd045c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__hw_addr_sync_dev) = 0x84c17451 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__hw_addr_unsync_dev) = 0xb05b216e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__napi_alloc_skb) = 0x5f802418 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__napi_schedule_irqoff) = 0xd022b30c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__netif_napi_del) = 0x310d1afd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__page_frag_cache_drain) = 0xb6ccef70 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__pci_register_driver) = 0x1cb89f6b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__skb_pad) = 0x16cd8a38 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(__tracepoint_xdp_exception) = 0xad42e583 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_err) = 0x3a3ec6c6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_info) = 0xff4f021a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(_dev_warn) = 0x1e5fac62 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(alloc_etherdev_mqs) = 0x665d1b75 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(bpf_prog_put) = 0xa1b79421 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(bpf_warn_invalid_xdp_action) = 0xb05e54b3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(build_skb) = 0x5c034c8f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(consume_skb) = 0x014a9f1b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca3_get_tag) = 0x258e3aac needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca_add_requester) = 0x14542454 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dca_remove_requester) = 0x7e094b21 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_getapp) = 0x535a6190 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_ieee_delapp) = 0xc6496dd5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_ieee_setapp) = 0x68aabfb5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dcb_setapp) = 0x0bf4f0bb needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_create_dir) = 0x11b4f7a8 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_create_file) = 0xa6202222 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(debugfs_remove) = 0x20033468 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_addr_add) = 0x96486dae needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_addr_del) = 0xe222c9e8 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_driver_string) = 0xcdafdb2a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dev_get_stats) = 0x9a9222bd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(device_set_wakeup_enable) = 0xac4a0980 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(devm_hwmon_device_register_with_groups) = 0x70a8831d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(devm_kmalloc) = 0xe9ad0580 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_alloc_attrs) = 0xed08591d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_free_attrs) = 0x5c4b9170 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_map_page_attrs) = 0xe2dc579b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_set_coherent_mask) = 0x97c42987 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_set_mask) = 0xf4110cd1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_sync_single_for_cpu) = 0x51b1550f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_sync_single_for_device) = 0x9980b0a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(dma_unmap_page_attrs) = 0xa41d1497 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(driver_for_each_device) = 0x9465cb85 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_get_headlen) = 0x98601bb3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_type_trans) = 0x25c1aba0 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(eth_validate_addr) = 0xfc0d5980 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ethtool_op_get_link) = 0xc3aa9108 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ethtool_op_get_ts_info) = 0x7eb31781 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(flow_block_cb_setup_simple) = 0x4dd5fafe needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(free_netdev) = 0x52c68cbc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ipv6_find_hdr) = 0x9c90a5b6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kfree_skb_reason) = 0xc87ace65 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kmalloc_caches) = 0x5f73fb3c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(kmem_cache_alloc_trace) = 0xa3cddcae needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(module_layout) = 0xa7f1f8eb needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_complete_done) = 0x40402bca needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_consume_skb) = 0x10e9a3dc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_disable) = 0x284fd3bf needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_enable) = 0xaf328093 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_gro_receive) = 0x4bd5d8dd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(napi_schedule_prep) = 0x76bae8ab needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ndo_dflt_bridge_getlink) = 0x089345b5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ndo_dflt_fdb_add) = 0x77dcc759 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_crit) = 0x6e6d1264 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_err) = 0xc66472cf needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_info) = 0xa71c2c40 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_notice) = 0x07efd02f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_printk) = 0x2c04b9df needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_reset_tc) = 0x8849ff8a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_set_num_tc) = 0xd82a15a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_set_tc_queue) = 0x9d6a1782 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netdev_warn) = 0xfe96cfed needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_carrier_off) = 0x14111c62 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_carrier_on) = 0x96781ba1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_device_attach) = 0x6d7af90c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_device_detach) = 0xf3103b00 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_napi_add) = 0xec77a489 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_schedule_queue) = 0xe191e05f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_real_num_rx_queues) = 0x8379688b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_real_num_tx_queues) = 0xa1e2fa4b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_set_xps_queue) = 0x2813f1cd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_tx_stop_all_queues) = 0x13392855 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(netif_tx_wake_queue) = 0x24e11830 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(param_array_ops) = 0x7a0b1ed0 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(param_ops_int) = 0xaad18ecd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_aer_clear_nonfatal_status) = 0x254f8d5e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_driver) = 0x45689a09 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_get) = 0xdd00d2c6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_dev_put) = 0x516c634d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_device) = 0x0921c2a1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_link_state) = 0xc5408d7d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_msi) = 0xe3ee7e3c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_msix) = 0x5d88de56 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_pcie_error_reporting) = 0xf4167337 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_disable_sriov) = 0xae1adbfa needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_device_mem) = 0x74ebcbd1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_msi) = 0x01bdea04 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_msix_range) = 0x98ff98d5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_pcie_error_reporting) = 0x717d4b61 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_enable_sriov) = 0x68c478f7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_find_capability) = 0x3b06b1de needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_find_ext_capability) = 0xce3f4b59 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_get_device) = 0x7e3c03c5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_num_vf) = 0xbf36cc3e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_prepare_to_sleep) = 0x4d75b46f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_byte) = 0x595813a6 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_dword) = 0x5d999172 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_read_config_word) = 0x3911c65c needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_release_selected_regions) = 0x7a3abda2 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_request_selected_regions) = 0xfab14d9f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_restore_state) = 0x24bbcb8b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_save_state) = 0xfd2da512 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_select_bars) = 0x1191f7a3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_set_master) = 0x2f6d2bb5 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_set_power_state) = 0x54621869 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_sriov_set_totalvfs) = 0xc0efcf8f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_unregister_driver) = 0xa972d633 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_vfs_assigned) = 0x6bdb5a77 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_wake_from_d3) = 0x2464b789 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pci_write_config_word) = 0x767cdb47 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pcie_capability_read_word) = 0xd51ae9d7 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pcie_print_link_status) = 0x26df835b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(printk) = 0xc5850110 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(pskb_expand_head) = 0x0d526b2b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_event) = 0xe7d77369 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_index) = 0x3876beca needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_register) = 0x0ed9b290 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(ptp_clock_unregister) = 0xb77a6063 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(register_netdev) = 0xba3ff097 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(simple_open) = 0xf9fc90f3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_add_rx_frag) = 0x21ea980a needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_checksum_help) = 0x08d53501 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_clone_tx_timestamp) = 0x5b57e627 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_copy_bits) = 0x632061f3 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_put) = 0xe6c6ba90 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(skb_tstamp_tx) = 0xc6a9499d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(softnet_data) = 0x3bf76a4e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(udp_tunnel_nic_ops) = 0x9b1d2e6b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(unregister_netdev) = 0x01d13501 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_convert_zc_to_xdp_frame) = 0xcdb7b8cd needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_do_redirect) = 0x635f6fef needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_master_redirect) = 0xd895cc1b needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_return_frame) = 0x5b1c3ac9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_return_frame_rx_napi) = 0x4742580d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_reg) = 0xc5f644e9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_reg_mem_model) = 0xcabfedb2 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_unreg) = 0x0d8f96fc needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xdp_rxq_info_unreg_mem_model) = 0x610cb9e1 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_alloc) = 0xa0815c99 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_map) = 0x0198092e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_sync_for_cpu_slow) = 0x0cac6528 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_sync_for_device_slow) = 0xe91374d9 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_dma_unmap) = 0xb4f7ca78 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_free) = 0xa353e08d needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_raw_get_dma) = 0x0eb88c7f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xp_set_rxq_info) = 0x86b2db4e needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_completed) = 0x5205b281 needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_peek_desc) = 0xe67282ce needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 - nothing provides kernel(xsk_tx_release) = 0xb016016f needed by kmod-ixgbe-5.19.6-1.rhel9u1.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

1 0

Missing data for some VM in Grafana Monitoring Portal
by Guillaume Pavese 12 Oct '23

12 Oct '23

Hi all. We are using oVirt 4.5.4 with the provided grafana monitoring tool. Everything seemed to be working as expected, however we have discovered that for some vm (not all), there is no available Disk or Ethernet data (while CPU and Memory data is available) : [image: image.png] While for some other VMs running on the same host, all data is available: [image: image.png] Does anyone have an idea what might be going wrong? -- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>

1 0