Users
Threads by month
- ----- 2026 -----
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- 6 participants
- 19176 discussions
Hello,
I have been used to OLV/RHEV product about 3 years
I often visit our community to get the error solution as well
now I would try to use ovirt and develop to improve some function..
I'm using test environment using compiled oVirt 4.5 package.
but I havent yet set up build develpment environment
1. Can you share and advise your ovirt development environment experience?
2. ovirt has some project, and how merge it to compile one source?
3. Is there a process for sharing what features you want to fix before making improvements? Can I just do PR?
4. your tips.. how to debug, ..etc
3
2
Hello everyone,
After a network outage, we experienced a situation of chaotic
connectivity between the oVirt manager (located on a VM outside the
cluster) and the hypervisors themselves.
During the incident, while only the connectivity between the manager and
the hypervisors was difficult, during the recovery we had fencing of
several of our hypervisors.
I'm looking for a diagram or text that describes all the rules for
triggering fencing.
Specifically, I have questions about the following (just in case):
* Does the manager play an active role in the decision to fence a
hypervisor?
Regards,
Renaud
2
1
Hi everyone.
I need help understanding a failure deploying the hosted engine on a
fresh-installed oVirt 4.5.4 el8 node.
After the setup via official ISO, I login via ssh in the node and I
issue the command:
# hosted-engine --deploy --4 --ansible-extra-vars=he_offline_deployment=true
-- Note --
The extra ansible variable is the only way I found to inhibit the
deployed hosted engine downloading last OS updates that will break
Python compatibility between the ansible playbook in the node deploying
and the ansible host in the engine deployed.
Without that extra variable the deployment fails with fancy reasons.
-- End note --
The deployment goes, until i specify an iSCSI target and a (free) LUN.
The playbook adds the storage domain, creates the HE disk and transfert
the HE vm to the domain. Then an error occurs:
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Initialize lockspace
volume]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Workaround for
ovirt-ha-broker start failures]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Initialize lockspace
volume]
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed":
true, "cmd": ["hosted-engine", "--reinitialize-lockspace", "--force"],
"delta": "0:00:00.170053", "end": "2023-10-20 11:21:18.111299", "msg":
"non-zero return code", "rc": 1, "start": "2023-10-20 11:21:17.941246",
"stderr": "Traceback (most recent call last):\n File
\"/usr/lib64/python3.6/runpy.py\", line 193, in _run_module_as_main\n
\"__main__\", mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line
85, in _run_code\n exec(code, run_globals)\n File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\",
line 30, in <module>\n ha_cli.reset_lockspace(force)\n File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\",
line 286, in reset_lockspace\n stats =
broker.get_stats_from_storage()\n File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\",
line 148, in get_stats_from_storage\n result =
self._proxy.get_stats()\n File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__\n
return self.__send(self.__name, args)\n File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request\n
verbose=self.__verbose\n File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request\n
return self.single_request(host, handler, request_body, verbose)\n File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in
single_request\n http_conn = self.send_request(host, handler,
request_body, verbose)\n File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in
send_request\n self.send_content(connection, request_body)\n File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in
send_content\n connection.endheaders(request_body)\n File
\"/usr/lib64/python3.6/http/client.py\", line 1268, in endheaders\n
self._send_output(message_body, encode_chunked=encode_chunked)\n File
\"/usr/lib64/python3.6/http/client.py\", line 1044, in _send_output\n
self.send(msg)\n File \"/usr/lib64/python3.6/http/client.py\", line
982, in send\n self.connect()\n File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\",
line 76, in connect\n
self.sock.connect(base64.b16decode(self.host))\nFileNotFoundError:
[Errno 2] No such file or directory", "stderr_lines": ["Traceback (most
recent call last):", " File \"/usr/lib64/python3.6/runpy.py\", line
193, in _run_module_as_main", " \"__main__\", mod_spec)", " File
\"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code", " exec(code,
run_globals)", " File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/reinitialize_lockspace.py\",
line 30, in <module>", " ha_cli.reset_lockspace(force)", " File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\",
line 286, in reset_lockspace", " stats =
broker.get_stats_from_storage()", " File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\",
line 148, in get_stats_from_storage", " result =
self._proxy.get_stats()", " File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __call__", "
return self.__send(self.__name, args)", " File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __request",
" verbose=self.__verbose", " File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request", "
return self.single_request(host, handler, request_body, verbose)", "
File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, in
single_request", " http_conn = self.send_request(host, handler,
request_body, verbose)", " File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request",
" self.send_content(connection, request_body)", " File
\"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content",
" connection.endheaders(request_body)", " File
\"/usr/lib64/python3.6/http/client.py\", line 1268, in endheaders", "
self._send_output(message_body, encode_chunked=encode_chunked)", " File
\"/usr/lib64/python3.6/http/client.py\", line 1044, in _send_output",
" self.send(msg)", " File \"/usr/lib64/python3.6/http/client.py\",
line 982, in send", " self.connect()", " File
\"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\",
line 76, in connect", " self.sock.connect(base64.b16decode(self.host))",
"FileNotFoundError: [Errno 2] No such file or directory"], "stdout": "",
"stdout_lines": []}
Then the playbook cleans all the installation and exits.
Really cant' figure what's going on ...
This is the best point i reached in deploying HE after two weeks of
failures and errors of any kind.
Please, can someone point me in the right direction to solve this new issue?
Thanks.
giuliano
2
3
When deploying isolated nested virtualization, we received the error "default gateway is not defined".
Two virtual machines were created in the zVirt virtualization environment: the first OS AstraLinux, the second OS zvirtNode. VM are connected to each other by a virtual network. After starting the installation in Hosted Engine mode using the command line, this error was received. The setup was performed without DNS and the default gateway
2
1
What is the best way and steps to move engine VM outside the Ovirt environment. As of now the engine VM is controlling the host but running also on the same Ovirt host. VM must run on a different Ovirt environment. Want to move/migrate engine from hosted engine to stand alone VM
2
1
VDSM host_devops command HSMGetAllTasksStatusesVDS failed: Not SPM: ()
by tapz.malz＠gmail.com 21 Oct '23
by tapz.malz＠gmail.com 21 Oct '23
21 Oct '23
Hi All,
We are facing one issue in our environment. After importing a .ova image from Virtualbox to KVM, when we tried to remove the imported vm, suddenly getting below errors in event
VDSM host_devops command HSMGetAllTasksStatusesVDS failed: Not SPM: ()
Failed to activate Storage Domain domain_devops (Data Center datacenter_devops) by admin@internal-authz
It is also observed that our datacenter is non-responsive.
When we try to activate it, getting below error
Failed Activating Storage Domain domain_devops on Data Center datacenter_devops
Can somebody help?
3
2
I recently began managing an Ovirt cluster. It has 5 cpu nodes and 3 gpu nodes in two clusters. When I started it was at 4.2.8 with a standalone engine. I have henceforth been steadily upgrading it to 4.5.4. I followed the upgrade guide and was successful in upgrading all 9 nodes from 4.2 to 4.3.10. I then upgraded the engine to centos 8 with 4.4 engine. I then rebuilt each of the nodes in the two data clusters to centos 8 ovirt 4.4 using the ovirt node installer iso. They all joined the engine correctly. I then upgraded the compatibility to 4.3 in both clusters and rebooted all the running VM's to also be at 4.3 compatibility. I then upgraded the engine to 4.5.4 and proceeded to upgrade two nodes to 4.5.4. They were successful and reattached to the engine correctly. Here is where my issue arises. Any VM that I migrate to the 4.5.4 node from a 4.4.10 node migrates correctly and then within 5 minutes will automatically migrate back. If I force pin a VM it stays running.
I have searched the logs but not found anything to explain such behavior.
Any assistance or idea ?
2
1
Hello,
I would like to participate in the oVirt translation project.
How can this be done?
Thank you.
2
1
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for <JsonRpcResponse id: "..."
by nicolas＠devels.es 17 Oct '23
by nicolas＠devels.es 17 Oct '23
17 Oct '23
Hi,
We're running oVirt 4.5.4, recently we had a problem due to /var
partition being full, and we realized the
/var/log/ovirt-engine/engine.log was pretty huge, approximately 16GB in
size, despite it's configured to rotate daily.
Checking the log, we have a lot (about 53000 at the time of this mail)
of WARN events in this form:
2023-10-17 03:34:51,347+01 WARN
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not
able to update response for <JsonRpcResponse id:
"d6cc2209-3126-48e1-97c8-97ca84d00f3a" result:
[{"statusTime":"3329996823","status":"Up","vmId":"0fad0aa4-ec91-4beb-82a3-5641f0da674f","vmName":"SYTW-Backend-aluXXXXXXXXX","vmType":"kvm","kvmEnable":"true","acpiEnable":"true","elapsedTime":"285568","monitorResponse":"0","clientIp":"","timeOffset":"2","cpuUser":"-0.04","cpuSys":
"0.40","cpuUsage":"12369410000000","cpuActual":true,"network":{"vnet3":{"macAddr":"00:1a:4a:4d:cc:d5","name":"vnet3","speed":"1000","state"
...
This is just a chunk of the full line, which is actually pretty big as
it contains a lot of parameters. So about 53.000 times this long line
makes the log huge.
Why are we getting such a big amount of these WARNings and what can we
do to solve them?
Thanks.
1
0
16 Oct '23
Hi All,
I'm hoping someone(s) can point me in the correct direction to fix this.
The Architecture
+++++++++++++
3 identical VM-Hosts (vmhost{1-3}.mydomain.local):
- Rocky Linux 8.8
- 4 local gluster volumes
- 4 iscsi volumes provided by 3-node Ceph Cluster (also running Rocky v8.8)
- 3 NICS:
- One Gluster Transfer
- One Ceph Data Transfer
- One General Data access / OVirt management
Each node of the Ceph Cluster is running an iscsi gateway, and each vmhost connects to each iscsi gateway
Everything is working AOK, except vmhost1 complains that it can't reach 3 of the 4 iscsi volumes/domains.
I've looked in the engine.log and vdsm.log but can't figure out how to fix things, and I don't know where else to look.
engine.log (extract)
+++++++++++++++
~~~
2023-10-16 18:21:08,892+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (default task-27) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock Acquired to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}'
2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Before acquiring lock in order to prevent monitoring for host 'vmhost1.mydomain.local' from data-center 'my_datacentre'
2023-10-16 18:21:09,179+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock acquired, from now a monitoring of host will be skipped for host 'vmhost1.mydomain.local' from data-center 'my_datacentre'
2023-10-16 18:21:09,181+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='Unassigned', nonOperationalReason='NONE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 732b1116
2023-10-16 18:21:09,245+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] FINISH, SetVdsStatusVDSCommand, return: , log id: 732b1116
2023-10-16 18:21:09,248+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Activate host finished. Lock released. Monitoring can run now for host 'vmhost1.mydomain.local' from data-center 'my_datacentre'
2023-10-16 18:21:09,314+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] EVENT_ID: VDS_ACTIVATE(16), Activation of host vmhost1.mydomain.local initiated by admin@internal-authz.
2023-10-16 18:21:09,373+11 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (EE-ManagedThreadFactory-engine-Thread-25323) [ba6cfae6-5c90-4433-8000-7e5e02b9ac24] Lock freed to object 'EngineLock:{exclusiveLocks='[8e0e99ab-c212-41c4-8318-acd34acb6289=VDS]', sharedLocks=''}'
2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] START, GetHardwareInfoAsyncVDSCommand(HostName = vmhost1.mydomain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vds='Host[vmhost1.mydomain.local,8e0e99ab-c212-41c4-8318-acd34acb6289]'}), log id: 3dac178f
2023-10-16 18:21:10,620+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 3dac178f
2023-10-16 18:21:10,688+11 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [44be9d7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS
2023-10-16 18:21:10,690+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-10-16 18:21:10,703+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Running command: UpdateClusterCommand internal: true. Entities affected : ID: c892710d-8611-4146-aea2-05d910f1829b Type: ClusterAction group EDIT_CLUSTER_CONFIGURATION with role type ADMIN
2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] EVENT_ID: SYSTEM_UPDATE_CLUSTER(835), Host cluster my_cluster was updated by system
2023-10-16 18:21:10,761+11 INFO [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [c4e6178] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[]'}'
2023-10-16 18:21:10,965+11 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool
2023-10-16 18:21:11,032+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 630966ec
2023-10-16 18:21:11,034+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [5f0caae0] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 630966ec
2023-10-16 18:21:11,170+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 7bfaa890-f892-4b3b-bf18-33dabc13e646 Type: StoragePool
2023-10-16 18:21:11,196+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='8d7b930e-a758-4010-aaf5-a230447f2761', connection='vmhosttran1.mydomain.local:/gv0', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 30a928ed
2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {8d7b930e-a758-4010-aaf5-a230447f2761=0}, log id: 30a928ed
2023-10-16 18:21:11,211+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='GLUSTERFS', connectionList='[StorageServerConnections:{id='044a086b-e94b-4e5d-a118-283ffceb8de8', connection='vmhosttran1.mydomain.local:gv1', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='2d328788-ade0-4e25-b947-1c392d4f133e', connection='vmhosttran1.mydomain.local:/gv2', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local
:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', connection='vmhosttran1.mydomain.local:/gv3', iqn='null', vfsType='glusterfs', mountOptions='backup-volfile-servers=vmhosttran2.mydomain.local:vmhosttran3.mydomain.local', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 6aaeee49
2023-10-16 18:21:11,664+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [] Fetched 0 VMs from VDS '8e0e99ab-c212-41c4-8318-acd34acb6289'
2023-10-16 18:21:11,917+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {044a086b-e94b-4e5d-a118-283ffceb8de8=0, 2d328788-ade0-4e25-b947-1c392d4f133e=0, 420b9d49-eebb-4dea-99bb-bcb89fbe2bf2=0}, log id: 6aaeee49
2023-10-16 18:21:11,920+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] START, ConnectStorageServerVDSCommand(HostName = vmhost1.mydomain.local, StorageServerConnectionManagementVDSParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', storageType='ISCSI', connectionList='[StorageServerConnections:{id='78f16782-9a06-45f2-92e5-d1ca84a8c67e', connection='192.168.4.24', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}, StorageServerConnections:{id='868d7c6a-c933-4475-86ce-ab1359d6f4c3', connection='192.168.4.25', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'},
StorageServerConnections:{id='20a52c08-a4c5-4b07-8ecc-bec0e82e582a', connection='192.168.4.23', iqn='iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', vfsType='null', mountOptions='null', nfsVersion='null', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 4e953edb
2023-10-16 18:21:19,245+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 694529cf
2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@a9992e5, log id: 694529cf
2023-10-16 18:21:19,247+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 786d585b
2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@1cf19aa, log id: 786d585b
2023-10-16 18:21:19,248+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 4104430d
2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@7ccef72, log id: 4104430d
2023-10-16 18:21:19,250+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 15f1c050
2023-10-16 18:21:19,251+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-43) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@6c8a990d, log id: 15f1c050
2023-10-16 18:21:34,401+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 3f508f9e
2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@5690567a, log id: 3f508f9e
2023-10-16 18:21:34,403+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='NVRAM'}), log id: 270bce0b
2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@2a7c9b44, log id: 270bce0b
2023-10-16 18:21:34,404+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 11ebddea
2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@459c7a2b, log id: 11ebddea
2023-10-16 18:21:34,406+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='NVRAM'}), log id: 3c1e9f05
2023-10-16 18:21:34,408+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-5) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@72f69579, log id: 3c1e9f05
2023-10-16 18:21:49,594+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='f106984f-4e0d-492e-a490-f7930ded9169', kind='TPM'}), log id: 6a366a0f
2023-10-16 18:21:49,597+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@95edf66, log id: 6a366a0f
2023-10-16 18:21:49,655+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] START, GetVmExternalDataVDSCommand(HostName = vmhost2.mydoamin.local, Parameters:{hostId='967301de-be9f-472a-8e66-03c24f01fa71', vmId='6a8236f8-22fa-4e6a-bc3d-522f3a119635', kind='TPM'}), log id: 40d156
2023-10-16 18:21:49,656+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVmExternalDataVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] FINISH, GetVmExternalDataVDSCommand, return: org.ovirt.engine.core.vdsbroker.vdsbroker.VmExternalDataReturn@3f11df90, log id: 40d156
2023-10-16 18:23:28,802+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] FINISH, ConnectStorageServerVDSCommand, return: {78f16782-9a06-45f2-92e5-d1ca84a8c67e=465, 868d7c6a-c933-4475-86ce-ab1359d6f4c3=465, 20a52c08-a4c5-4b07-8ecc-bec0e82e582a=0}, log id: 4e953edb
2023-10-16 18:23:28,811+11 INFO [org.ovirt.engine.core.bll.storage.pool.ConnectHostToStoragePoolServersCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [706708f9] Host 'vmhost1.mydomain.local' storage connection was succeeded
2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] START, ConnectStoragePoolVDSCommand(HostName = vmhost1.mydomain.local, ConnectStoragePoolVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', vdsId='8e0e99ab-c212-41c4-8318-acd34acb6289', storagePoolId='7bfaa890-f892-4b3b-bf18-33dabc13e646', masterVersion='1'}), log id: 3b47a386
2023-10-16 18:23:28,846+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Executing with domain map: {e311ddf1-7f2c-49ef-a618-050d9a2b947f=active, fa8d922a-df3c-4f5b-adf6-a77c3689c397=active, 30775056-38ef-483a-889c-70453e9aa5f9=active, ee5b0268-8141-43e0-995e-8b4586217123=active, 3e33b7ce-7279-4765-a203-cca37eff8cb0=active, 4412e492-f752-4a6c-98ff-f289043618da=active, 0dde39a4-42e4-4fba-8068-e63e42dd3334=active, d44ec7f2-3629-4110-93a5-7d8d57230efc=active}
2023-10-16 18:23:32,385+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 3b47a386
2023-10-16 18:23:32,418+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '0dde39a4-42e4-4fba-8068-e63e42dd3334:iscsi-vmimages' report isn't an actual report
2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '3e33b7ce-7279-4765-a203-cca37eff8cb0:gluster-generaldata' report isn't an actual report
2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'e311ddf1-7f2c-49ef-a618-050d9a2b947f:gluster-vmimages' report isn't an actual report
2023-10-16 18:23:32,419+11 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '4412e492-f752-4a6c-98ff-f289043618da:gluster-isos' report isn't an actual report
2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'fa8d922a-df3c-4f5b-adf6-a77c3689c397:iscsi-databases' was reported with error code '358'
2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain '30775056-38ef-483a-889c-70453e9aa5f9:iscsi-generaldata' was reported with error code '358'
2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyImpl] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Domain 'd44ec7f2-3629-4110-93a5-7d8d57230efc:iscsi-backups' was reported with error code '358'
2023-10-16 18:23:32,419+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-databases' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local'
2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-generaldata' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local'
2023-10-16 18:23:32,420+11 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] Storage Domain 'iscsi-backups' of pool 'my_datacentre' is in problem in host 'vmhost1.mydomain.local'
2023-10-16 18:23:32,454+11 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-25361) [706708f9] EVENT_ID: VDS_STORAGE_VDS_STATS_FAILED(189), Host vmhost1.mydomain.local reports about one of the Active Storage Domains as Problematic.
2023-10-16 18:23:32,564+11 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS
2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='NonOperational', nonOperationalReason='STORAGE_DOMAIN_UNREACHABLE', stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 992c9e
2023-10-16 18:23:32,566+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] Clearing domains data for host vmhost1.mydomain.local
2023-10-16 18:23:32,619+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] FINISH, SetVdsStatusVDSCommand, return: , log id: 992c9e
2023-10-16 18:23:32,689+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_SET_NONOPERATIONAL_DOMAIN(522), Host vmhost1.mydomain.local cannot access the Storage Domain(s) iscsi-databases, iscsi-generaldata, iscsi-backups attached to the Data Center my_datacentre. Setting Host state to Non-Operational.
2023-10-16 18:23:32,789+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: VDS_ALERT_FENCE_TEST_FAILED(9,001), Power Management test failed for Host vmhost1.mydomain.local.There is no other host in the data center that can be used to test the power management settings.
2023-10-16 18:23:32,856+11 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [7d00dee] EVENT_ID: CONNECT_STORAGE_POOL_FAILED(995), Failed to connect Host vmhost1.mydomain.local to Storage Pool my_datacentre
2023-10-16 18:23:33,016+11 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS
2023-10-16 18:23:33,018+11 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Host 'vmhost1.mydomain.local'(8e0e99ab-c212-41c4-8318-acd34acb6289) is already in NonOperational status for reason 'STORAGE_DOMAIN_UNREACHABLE'. SetNonOperationalVds command is skipped.
2023-10-16 18:23:33,019+11 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-28) [f7474e7] Received first domain report for host vmhost1.mydomain.local
2023-10-16 18:24:02,559+11 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDSAction group MANIPULATE_HOST with role type ADMIN
2023-10-16 18:24:02,561+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetVdsStatusVDSCommand(HostName = vmhost1.mydomain.local, SetVdsStatusVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289', status='PreparingForMaintenance', nonOperationalReason='NONE', stopSpmFailureLogged='true', maintenanceReason='null'}), log id: 51d95bef
2023-10-16 18:24:02,592+11 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetVdsStatusVDSCommand, return: , log id: 51d95bef
2023-10-16 18:24:02,799+11 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 8e0e99ab-c212-41c4-8318-acd34acb6289 Type: VDS
2023-10-16 18:24:02,827+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] START, SetHaMaintenanceModeVDSCommand(HostName = vmhost1.mydomain.local, SetHaMaintenanceModeVDSCommandParameters:{hostId='8e0e99ab-c212-41c4-8318-acd34acb6289'}), log id: 63f61978
2023-10-16 18:24:02,832+11 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetHaMaintenanceModeVDSCommand] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] FINISH, SetHaMaintenanceModeVDSCommand, return: , log id: 63f61978
2023-10-16 18:24:02,914+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-27) [e1af0e2b-fe1a-48c2-9a86-585db015fa98] EVENT_ID: USER_VDS_MAINTENANCE_WITHOUT_REASON(620), Host vmhost1.mydomain.local was switched to Maintenance mode by admin@internal-authz.
~~~
vdsm.log (extract - from vmhost1)
++++++++++++++++++++++++++
~~~
2023-10-16 18:21:10,069+1100 INFO (jsonrpc/7) [api.host] START getCapabilities() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:21:10,410+1100 ERROR (jsonrpc/7) [root] Error getting managedvolume connector info: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config imp
ort cfg\\n File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',) (caps:165)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/host/caps.py", line 161, in get
caps["connector_info"] = managedvolume.connector_info()
File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 63, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 78, in connector_info
return run_helper("connector_info")
File "/usr/lib/python3.6/site-packages/vdsm/storage/managedvolume.py", line 184, in run_helper
sub_cmd, cmd_input=cmd_input)
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 38, in __call__
return callMethod()
File "/usr/lib/python3.6/site-packages/vdsm/common/supervdsm.py", line 36, in <lambda>
**kwargs)
File "<string>", line 2, in managedvolume_run_helper
File "/usr/lib64/python3.6/multiprocessing/managers.py", line 772, in _callmethod
raise convert_to_error(kind, result)
vdsm.storage.exception.ManagedVolumeHelperFailed: Managed Volume Helper failed.: ('Error executing helper: Command [\'/usr/bin/python3\', \'/usr/libexec/vdsm/managedvolume-helper\', \'connector_info\'] failed with rc=1 out=b\'\' err=b\'Traceback (most recent call last):\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 91, in <module>\\nfrom importlib import resources as _importlib_resources\\nImportError: cannot import name \\\'resources\\\'\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File "/usr/libexec/vdsm/managedvolume-helper", line 26, in <module>\\nfrom os_brick.initiator import connector\\n File "/usr/lib/python3.6/site-packages/os_brick/initiator/connector.py", line 27, in <module>\\nfrom oslo_concurrency import lockutils\\n File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 29, in <module>\\nfrom oslo_config import cfg\\n File "/usr/lib/python3.6/site-packag
es/oslo_config/cfg.py", line 45, in <module>\\nfrom oslo_config import types\\n File "/usr/lib/python3.6/site-packages/oslo_config/types.py", line 29, in <module>\\nimport netaddr\\n File "/usr/lib/python3.6/site-packages/netaddr/__init__.py", line 18, in <module>\\nfrom netaddr.core import (AddrConversionError, AddrFormatError,\\n File "/usr/lib/python3.6/site-packages/netaddr/core.py", line 11, in <module>\\nfrom netaddr.compat import _callable, _iter_dict_keys\\n File "/usr/lib/python3.6/site-packages/netaddr/compat.py", line 93, in <module>\\nimport importlib_resources as _importlib_resources\\nModuleNotFoundError: No module named \\\'importlib_resources\\\'\\n\'',)
2023-10-16 18:21:10,600+1100 INFO (jsonrpc/7) [root] /usr/libexec/vdsm/hooks/after_get_caps/ovirt_provider_ovn_hook: rc=0 err=b'' (hooks:106)
2023-10-16 18:21:10,601+1100 INFO (jsonrpc/7) [api.host] FINISH getCapabilities return={'status': {'code': 0, 'message': 'Done'}, 'info': {'kvmEnabled': 'true', 'cpuCores': '16', 'cpuThreads': '32', 'cpuSockets': '2', 'onlineCpus': '0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23,8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31', 'cpuTopology': [{'cpu_id': 0, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 1, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 2, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 3, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 4, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 5, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 6, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 7, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 16, 'numa_cell_id': 0, 'socket_id
': 0, 'die_id': 0, 'core_id': 0}, {'cpu_id': 17, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 1}, {'cpu_id': 18, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 2}, {'cpu_id': 19, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 3}, {'cpu_id': 20, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 4}, {'cpu_id': 21, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 5}, {'cpu_id': 22, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 6}, {'cpu_id': 23, 'numa_cell_id': 0, 'socket_id': 0, 'die_id': 0, 'core_id': 7}, {'cpu_id': 8, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 9, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 10, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 11, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 12, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 13, 'numa_cell_id'
: 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 14, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 15, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}, {'cpu_id': 24, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 0}, {'cpu_id': 25, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 1}, {'cpu_id': 26, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 2}, {'cpu_id': 27, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 3}, {'cpu_id': 28, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 4}, {'cpu_id': 29, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 5}, {'cpu_id': 30, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 6}, {'cpu_id': 31, 'numa_cell_id': 1, 'socket_id': 1, 'die_id': 0, 'core_id': 7}], 'cpuSpeed': '4000.000', 'cpuModel': 'Intel(R) Xeon(R) Silver 4215R CPU @ 3.20GHz', 'cpuFlags': 'pku,clflushopt,avx2,nopl,pni,pln,dca,smep,pat,art,cx8,msr,pts,arch_capa
bilities,xsave,ht,lm,smap,de,mce,pse36,cx16,skip-l1dfl-vmentry,cpuid,bts,lahf_lm,avx512vl,tsc_deadline_timer,bmi2,md-clear,tm2,cmov,sep,sse4_1,invtsc,md_clear,intel_pt,amd-ssbd,mpx,ssse3,rdt_a,cpuid_fault,x2apic,mmx,avx512dq,dtes64,avx512cd,vme,pclmulqdq,cqm_occup_llc,xsavec,ibpb,umip,pschange-mc-no,mtrr,avx512f,stibp,clwb,xsaves,f16c,invpcid_single,avx512bw,arat,pdpe1gb,ibrs,ept,tm,invpcid,cqm_mbm_total,ibrs-all,3dnowprefetch,popcnt,cqm,rdctl-no,xsaveopt,pdcm,aperfmperf,cat_l3,xgetbv1,xtpr,syscall,smx,pbe,sse2,amd-stibp,pebs,xtopology,tsc_adjust,mca,pae,ida,nonstop_tsc,tpr_shadow,ssbd,nx,tsc,dtherm,aes,fxsr,fma,cqm_mbm_local,movbe,erms,intel_ppin,mds-no,ds_cpl,rdseed,dts,ibrs_enhanced,constant_tsc,ept_ad,flexpriority,avx512_vnni,adx,clflush,acpi,cdp_l3,sse,rdtscp,fsgsbase,ss,bmi1,arch_perfmon,monitor,pse,ospke,pcid,rdrand,hypervisor,flush_l1d,tsx-ctrl,arch-capabilities,vmx,avx,pge,rep_good,vpid,sdbg,sse4_2,mba,cqm_llc,est,epb,fpu,vnmi,apic,abm,spec_ctrl,model_pentium3,model_Penryn,
model_Cascadelake-Server-noTSX,model_Haswell-noTSX-IBRS,model_Haswell-noTSX,model_kvm32,model_Nehalem-IBRS,model_pentium,model_Opteron_G1,model_IvyBridge-IBRS,model_Westmere,model_n270,model_qemu32,model_coreduo,model_486,model_IvyBridge,model_SandyBridge,model_Nehalem,model_Skylake-Client-noTSX-IBRS,model_pentium2,model_Westmere-IBRS,model_SandyBridge-IBRS,model_Conroe,model_qemu64,model_Opteron_G2,model_kvm64,model_Broadwell-noTSX,model_Skylake-Server-noTSX-IBRS,model_Broadwell-noTSX-IBRS,model_core2duo', 'vdsmToCpusAffinity': [1], 'version_name': 'Snow Man', 'software_version': '4.50.3.4', 'software_revision': '1', 'supportedENGINEs': ['4.2', '4.3', '4.4', '4.5'], 'clusterLevels': ['4.2', '4.3', '4.4', '4.5', '4.6', '4.7'], 'networks': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'iface': 'ovirtmgmt', 'bridged': True, 'addr': '192.168.2.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6
gateway': '::', 'ipv4defaultroute': True, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.20', 'vlanid': 20, 'dhcpv4': False, 'dhcpv6': False}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'iface': 'DMZ', 'bridged': True, 'addr': '192.168.1.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '1500', 'switch': 'legacy', 'southbound': 'bond0.10', 'vlanid': 10, 'dhcpv4': False, 'dhcpv6': False}, 'Migration': {'interface': 'ens1', 'iface': 'ens1', 'bridged': False, 'addr': '192.168.5.20', 'netmask': '255.255.255.0', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'ens1', 'dhcpv4': False, 'dhcpv6': False}, 'SAN': {'interface': 'bond0.40', 'iface': 'bond0.40', 'bridged': False, 'addr': '192.168.4.20', 'netmask':
'255.255.255.0', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'ipv4defaultroute': False, 'mtu': '9000', 'switch': 'legacy', 'southbound': 'bond0.40', 'vlanid': 40, 'dhcpv4': False, 'dhcpv6': False}}, 'bondings': {'bond0': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'slaves': ['eno1', 'eno2'], 'active_slave': '', 'opts': {'updelay': '1000', 'mode': '4', 'miimon': '1000', 'downdelay': '1000'}, 'ad_aggregator_id': '1', 'ad_partner_mac': '10:b3:c6:ae:a8:2d', 'switch': 'legacy', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 2000}}, 'bridges': {'ovirtmgmt': {'ports': ['bond0.20'], 'stp': 'off', 'addr': '192.168.2.20', 'ipv4addrs': ['192.168.2.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.2.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0',
'ipv4defaultroute': True, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '17569', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '1
00', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'DMZ': {'ports': ['bond0.10'], 'stp': 'off', 'addr': '192.168.1.20', 'ipv4addrs': ['192.168.1.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '192.168.1.1', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '8995', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multi
cast_query_interval': '12500', 'bridge_id': '8000.ac1f6b3ccce8', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '0', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '0', 'multicast_startup_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.ac1f6b3ccce8', 'no_linklocal_learn': '0', 'forward_delay': '1500', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}, 'virbr0': {'ports': [], 'stp': 'on', 'addr': '192.168.122.1', 'ipv4addrs': ['192.168.122.1/24'], 'ipv6addrs': [], 'ipv6autoconf'
: False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'opts': {'ageing_time': '30000', 'multicast_query_use_ifaddr': '0', 'gc_timer': '22103', 'hello_time': '200', 'multicast_router': '1', 'nf_call_iptables': '0', 'group_addr': '01:80:c2:00:00:00', 'group_fwd_mask': '0x0', 'multicast_querier': '0', 'hash_max': '4096', 'tcn_timer': '0', 'vlan_protocol': '0x8100', 'vlan_stats_per_port': '0', 'root_port': '0', 'vlan_filtering': '0', 'multicast_query_interval': '12500', 'bridge_id': '8000.525400a8ef1c', 'max_age': '2000', 'nf_call_arptables': '0', 'multicast_startup_query_interval': '3125', 'multicast_stats_enabled': '0', 'multicast_query_response_interval': '1000', 'topology_change': '0', 'priority': '32768', 'multicast_mld_version': '1', 'hash_elasticity': '16', 'hello_timer': '103', 'default_pvid': '1', 'root_path_cost': '0', 'multicast_igmp_version': '2', 'stp_state': '1', 'multicast_start
up_query_count': '2', 'topology_change_detected': '0', 'multicast_last_member_interval': '100', 'topology_change_timer': '0', 'root_id': '8000.525400a8ef1c', 'no_linklocal_learn': '0', 'forward_delay': '200', 'multicast_membership_interval': '26000', 'multicast_querier_interval': '25500', 'vlan_stats_enabled': '0', 'multicast_snooping': '1', 'nf_call_ip6tables': '0', 'multicast_last_member_count': '2'}}}, 'nics': {'ens1': {'hwaddr': '00:1b:21:e4:a6:f4', 'addr': '192.168.5.20', 'ipv4addrs': ['192.168.5.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}, 'eno1': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e8', 'dhcpv4': False, 'dhcpv
6': False, 'speed': 1000}, 'eno2': {'hwaddr': 'ac:1f:6b:3c:cc:e8', 'ad_aggregator_id': '1', 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '9000', 'netmask': '', 'ipv4defaultroute': False, 'permhwaddr': 'ac:1f:6b:3c:cc:e9', 'dhcpv4': False, 'dhcpv6': False, 'speed': 1000}}, 'vlans': {'bond0.20': {'iface': 'bond0', 'vlanid': 20, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.10': {'iface': 'bond0', 'vlanid': 10, 'addr': '', 'ipv4addrs': [], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6gateway': '::', 'mtu': '1500', 'netmask': '', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}, 'bond0.40': {'iface': 'bond0', 'vlanid': 40, 'addr': '192.168.4.20', 'ipv4addrs': ['192.168.4.20/24'], 'ipv6addrs': [], 'ipv6autoconf': False, 'gateway': '', 'ipv6
gateway': '::', 'mtu': '9000', 'netmask': '255.255.255.0', 'ipv4defaultroute': False, 'dhcpv4': False, 'dhcpv6': False}}, 'nameservers': ['192.168.2.16', '192.168.2.17'], 'supportsIPv6': True, 'ovnConfigured': False, 'hooks': {'after_get_caps': {'ovirt_provider_ovn_hook': {'checksum': 'a2bdefca38b96c8ddab39822cc8282bf3f67d875c4879003ffc9661826c92421'}}, 'before_device_create': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_nic_hotplug': {'10_ovirt_provider_ovn_hook': {'checksum': 'b9d67afb41bd86a521ada2121e3505297b3b6dfd478275ce0bd9501fcda6dafc'}}, 'before_vm_start': {'50_hostedengine': {'checksum': 'e5f5262cf22e06cd34e227afb27647e479351266876019a64210dbcbd2a43830'}}}, 'operatingSystem': {'release': '1.8.el8', 'version': '8.8', 'name': 'RHEL', 'pretty_name': 'Rocky Linux 8.8 (Green Obsidian)'}, 'uuid': '18e93400-808a-11ea-8000-ac1f6b3ccce8', 'packages2': {'kernel': {'version': '4.18.0', 'release': '477.27.1.e
l8_8.x86_64'}, 'glusterfs-cli': {'version': '10.4', 'release': '1.el8s'}, 'librbd1': {'version': '16.2.13', 'release': '1.el8s'}, 'libvirt': {'version': '8.0.0', 'release': '19.2.module+el8.8.0+1321+70d049b2'}, 'mom': {'version': '0.6.4', 'release': '1.el8'}, 'ovirt-hosted-engine-ha': {'version': '2.5.0', 'release': '1.el8'}, 'openvswitch': {'version': '2.15', 'release': '4.el8'}, 'nmstate': {'version': '1.4.4', 'release': '2.el8_8'}, 'qemu-img': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'qemu-kvm': {'version': '6.2.0', 'release': '33.module+el8.8.0+1454+0b2cbfb8'}, 'spice-server': {'version': '0.14.3', 'release': '4.el8'}, 'vdsm': {'version': '4.50.3.4', 'release': '1.el8'}, 'glusterfs': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-fuse': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-geo-replication': {'version': '10.4', 'release': '1.el8s'}, 'glusterfs-server': {'version': '10.4', 'release': '1.el8s'}}, 'realtimeKernel': False, 'kernelArgs'
: 'BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-477.27.1.el8_8.x86_64 root=UUID=9acf17b0-558a-46d3-b21c-74c208680fcb ro crashkernel=auto resume=UUID=9c5f897c-8e34-48a7-9f06-786af3c079e9', 'nestedVirtualization': True, 'emulatedMachines': ['pc-q35-rhel8.0.0', 'pc-q35-rhel7.4.0', 'pc-i440fx-rhel7.3.0', 'pc-q35-rhel8.2.0', 'pc-q35-rhel7.3.0', 'pc-i440fx-rhel7.4.0', 'pc-q35-rhel8.4.0', 'pc-q35-rhel7.6.0', 'pc-i440fx-rhel7.5.0', 'pc-q35-rhel8.1.0', 'pc-q35-rhel8.6.0', 'pc-q35-rhel8.3.0', 'pc-q35-rhel7.5.0', 'pc-q35-rhel8.5.0', 'pc-i440fx-rhel7.1.0', 'q35', 'pc-i440fx-rhel7.2.0', 'pc', 'pc-i440fx-rhel7.6.0', 'pc-i440fx-rhel7.0.0'], 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d', 'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:dc85f9619f1d'}], 'FC': []}, 'vmTypes': ['kvm'], 'memSize': '63805', 'reservedMem': '321', 'guestOverhead': '65', 'rngSources': ['random', 'hwrng'], 'numaNodes': {'0': {'totalMemory': '31553', 'hugepages': {'4': {'totalPages': '8077814'}, '
2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 19, 20, 21, 22, 23]}, '1': {'totalMemory': '32251', 'hugepages': {'4': {'totalPages': '8256270'}, '2048': {'totalPages': '0'}, '1048576': {'totalPages': '0'}}, 'cpus': [8, 9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 30, 31]}}, 'numaNodeDistance': {'0': [10, 21], '1': [21, 10]}, 'autoNumaBalancing': 2, 'selinux': {'mode': '1'}, 'liveSnapshot': 'true', 'liveMerge': 'true', 'kdumpStatus': 1, 'deferred_preallocation': True, 'hostdevPassthrough': 'false', 'additionalFeatures': ['libgfapi_supported', 'GLUSTER_SNAPSHOT', 'GLUSTER_GEO_REPLICATION', 'GLUSTER_BRICK_MANAGEMENT'], 'hostedEngineDeployed': True, 'hugepages': [2048, 1048576], 'kernelFeatures': {'SPECTRE_V2': '(Mitigation: Enhanced IBRS, IBPB: conditional, RSB filling, PBRSB-eIBRS: SW sequence)', 'ITLB_MULTIHIT': '(KVM: Mitigation: VMX disabled)', 'MMIO_STALE_DATA': '(Mitigation: Clear CPU buffers; SMT vulnerable)', 'MDS
': '(Not affected)', 'L1TF': '(Not affected)', 'SPEC_STORE_BYPASS': '(Mitigation: Speculative Store Bypass disabled via prctl)', 'TSX_ASYNC_ABORT': '(Mitigation: TSX disabled)', 'SPECTRE_V1': '(Mitigation: usercopy/swapgs barriers and __user pointer sanitization)', 'RETBLEED': '(Mitigation: Enhanced IBRS)', 'SRBDS': '(Not affected)', 'MELTDOWN': '(Not affected)'}, 'vncEncrypted': True, 'backupEnabled': True, 'coldBackupEnabled': True, 'clearBitmapsEnabled': True, 'fipsEnabled': False, 'boot_uuid': '68e5a5ca-3dd5-4103-9613-6065a5fc74da', 'tscFrequency': '3199999000', 'tscScaling': True, 'domain_versions': [0, 2, 3, 4, 5], 'supported_block_size': {'FCP': [512], 'GLUSTERFS': [0, 512, 4096], 'ISCSI': [512], 'LOCALFS': [0, 512, 4096], 'NFS': [512], 'POSIXFS': [512]}, 'cd_change_pdiv': True, 'refresh_disk_supported': True, 'replicate_extend': True, 'measure_subchain': True, 'measure_active': True, 'mailbox_events': True, 'zerocopy_migrations': True, 'netConfigDirty': 'False', 'openstack_b
inding_host_ids': {'OVIRT_PROVIDER_OVN': 'vmhost1.mydoamin.local'}, 'lastClientIface': 'ovirtmgmt'}} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:21:10,620+1100 INFO (jsonrpc/1) [api.host] START getHardwareInfo() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:21:10,621+1100 INFO (jsonrpc/1) [api.host] FINISH getHardwareInfo return={'status': {'code': 0, 'message': 'Done'}, 'info': {'systemManufacturer': 'Supermicro', 'systemProductName': 'X11DPi-N(T)', 'systemVersion': '123456789', 'systemSerialNumber': '', 'systemUUID': '0034e918-8a80-ea11-8000-ac1f6b3ccce8', 'systemFamily': 'SMC X11'}} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:21:10,767+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:31)
2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:31)
2023-10-16 18:21:10,778+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=bab905a7-0651-41a3-a9f6-49707358a0eb (api:37)
2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:31)
2023-10-16 18:21:10,779+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=c4e6178, task_id=0f382ce1-8a94-4764-94b7-394b65b3db99 (api:37)
2023-10-16 18:21:10,786+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=c4e6178 (api:37)
2023-10-16 18:21:11,032+1100 INFO (jsonrpc/6) [vds] Setting Hosted Engine HA local maintenance to False (API:1697)
2023-10-16 18:21:11,198+1100 INFO (jsonrpc/4) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv0', 'ipv6_enabled': 'false', 'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:31)
2023-10-16 18:21:11,199+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641)
2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Shutdown requested (__init__:126)
2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191)
2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641)
2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Shutdown requested (__init__:126)
2023-10-16 18:21:11,200+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191)
2023-10-16 18:21:11,200+1100 INFO (tmap-752/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641)
2023-10-16 18:21:11,200+1100 INFO (ioprocess/77159) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:21:11,201+1100 INFO (ioprocess/77153) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Shutdown requested (__init__:126)
2023-10-16 18:21:11,201+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191)
2023-10-16 18:21:11,202+1100 INFO (ioprocess/77148) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194)
2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Closing client (__init__:641)
2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57)
2023-10-16 18:21:11,209+1100 INFO (jsonrpc/4) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '8d7b930e-a758-4010-aaf5-a230447f2761', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=be7fffbe-f4ca-48f3-b1af-064268ad9eac (api:37)
2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Killing ioprocess (__init__:191)
2023-10-16 18:21:11,210+1100 INFO (ioprocess/76327) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:21:11,216+1100 INFO (jsonrpc/2) [vdsm.api] START connectStorageServer(domType=7, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:gv1', 'ipv6_enabled': 'false', 'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv2', 'ipv6_enabled': 'false', 'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'user': '', 'tpgt': '1'}, {'password': '********', 'vfs_type': 'glusterfs', 'port': '', 'mnt_options': 'backup-volfile-servers=vmhosttran2.mydoamin.local:vmhosttran3.mydoamin.local', 'iqn': '', 'connection': 'vmhosttran1.mydoamin.local:/gv3', 'i
pv6_enabled': 'false', 'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'user': '', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:31)
2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1' (storageServer:217)
2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 mode: None (fileUtils:214)
2023-10-16 18:21:11,293+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326)
2023-10-16 18:21:11,293+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:gv1 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1 (mount:190)
2023-10-16 18:21:11,415+1100 INFO (jsonrpc/2) [IOProcessClient] (Global) Starting client (__init__:340)
2023-10-16 18:21:11,424+1100 INFO (ioprocess/86760) [IOProcess] (Global) Starting ioprocess (__init__:465)
2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2' (storageServer:217)
2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 mode: None (fileUtils:214)
2023-10-16 18:21:11,518+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326)
2023-10-16 18:21:11,518+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv2 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2 (mount:190)
2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:21:11,662+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.storageServer] Creating directory '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3' (storageServer:217)
2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.fileutils] Creating directory: /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 mode: None (fileUtils:214)
2023-10-16 18:21:11,730+1100 WARN (jsonrpc/2) [storage.storageServer] Using user specified backup-volfile-servers option (storageServer:326)
2023-10-16 18:21:11,730+1100 INFO (jsonrpc/2) [storage.mount] mounting vmhosttran1.mydoamin.local:/gv3 at /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3 (mount:190)
2023-10-16 18:21:11,875+1100 INFO (tmap-753/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340)
2023-10-16 18:21:11,883+1100 INFO (tmap-753/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340)
2023-10-16 18:21:11,885+1100 INFO (ioprocess/87016) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465)
2023-10-16 18:21:11,889+1100 INFO (tmap-753/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340)
2023-10-16 18:21:11,892+1100 INFO (ioprocess/87022) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465)
2023-10-16 18:21:11,898+1100 INFO (ioprocess/87027) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465)
2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194)
2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194)
2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Closing client (__init__:641)
2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194)
2023-10-16 18:21:11,903+1100 INFO (jsonrpc/2) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Closing client (__init__:641)
2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194)
2023-10-16 18:21:11,904+1100 INFO (jsonrpc/2) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Closing client (__init__:641)
2023-10-16 18:21:11,904+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Killing ioprocess (__init__:191)
2023-10-16 18:21:11,904+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Killing ioprocess (__init__:191)
2023-10-16 18:21:11,904+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Killing ioprocess (__init__:191)
2023-10-16 18:21:11,905+1100 INFO (ioprocess/77197) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:21:11,905+1100 INFO (ioprocess/77213) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:21:11,906+1100 INFO (ioprocess/77222) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194)
2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194)
2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194)
2023-10-16 18:21:11,910+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194)
2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain ee5b0268-8141-43e0-995e-8b4586217123 from storage domain cache (sdc:194)
2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain e311ddf1-7f2c-49ef-a618-050d9a2b947f from storage domain cache (sdc:194)
2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 from storage domain cache (sdc:194)
2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Removing domain 4412e492-f752-4a6c-98ff-f289043618da from storage domain cache (sdc:194)
2023-10-16 18:21:11,914+1100 INFO (jsonrpc/2) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57)
2023-10-16 18:21:11,915+1100 INFO (jsonrpc/2) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '044a086b-e94b-4e5d-a118-283ffceb8de8', 'status': 0}, {'id': '2d328788-ade0-4e25-b947-1c392d4f133e', 'status': 0}, {'id': '420b9d49-eebb-4dea-99bb-bcb89fbe2bf2', 'status': 0}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=92bf27ed-8388-4485-abc3-cd79f0d4163f (api:37)
2023-10-16 18:21:11,924+1100 INFO (jsonrpc/3) [vdsm.api] START connectStorageServer(domType=3, spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', conList=[{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsichapinitiator', 'tpgt': '1'}]) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:31)
2023-10-16 18:21:11,925+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192)
2023-10-16 18:21:12,053+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192)
2023-10-16 18:21:12,174+1100 INFO (jsonrpc/3) [storage.iscsi] Adding iscsi node for target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:192)
2023-10-16 18:21:12,293+1100 INFO (jsonrpc/3) [storage.storageServer] Log in to 3 targets using 3 workers (storageServer:600)
2023-10-16 18:21:12,294+1100 INFO (iscsi-login/0) [storage.iscsi] Logging in to iscsi target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214)
2023-10-16 18:21:12,297+1100 INFO (iscsi-login/1) [storage.iscsi] Logging in to iscsi target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214)
2023-10-16 18:21:12,303+1100 INFO (iscsi-login/2) [storage.iscsi] Logging in to iscsi target 192.168.4.23:3260,1 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw via iface default (iscsi:214)
2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:31)
2023-10-16 18:21:13,667+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={} from=internal, task_id=a9c9c716-740c-424c-8a49-62f84192416a (api:37)
2023-10-16 18:21:13,676+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:31)
2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=093dfcd6-6f98-49bf-9723-0aca86957167 (api:37)
2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:31)
2023-10-16 18:21:13,688+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=69a16b74-67a6-4e74-9fb5-1c738e8bf029 (api:37)
2023-10-16 18:21:13,693+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:21:14,190+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:31)
2023-10-16 18:21:14,410+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=1c78fec1-ad8d-4c4a-9396-bd8bb30248e4 (api:37)
2023-10-16 18:21:15,485+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:31)
2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=0f779632-8240-4d02-b43e-ed4737312c36 (api:37)
2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:31)
2023-10-16 18:21:15,497+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2375aa06-ec3c-4658-b775-59683a0d25de (api:37)
2023-10-16 18:21:15,503+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:16,692+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:21:16,693+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:21:16,993+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:31)
2023-10-16 18:21:17,362+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={} from=::1,50158, task_id=bd2bdfd6-4d5e-4e60-8cab-3d25a2f31745 (api:37)
2023-10-16 18:21:17,366+1100 INFO (jsonrpc/1) [vdsm.api] START startMonitoringDomain(sdUUID='ee5b0268-8141-43e0-995e-8b4586217123', hostID=1) from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:31)
2023-10-16 18:21:17,367+1100 INFO (jsonrpc/1) [storage.monitor] Start monitoring ee5b0268-8141-43e0-995e-8b4586217123 (monitor:176)
2023-10-16 18:21:17,368+1100 INFO (jsonrpc/1) [vdsm.api] FINISH startMonitoringDomain return=None from=::1,50158, task_id=fca171ce-9c55-4300-affa-5caa1227495e (api:37)
2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63)
2023-10-16 18:21:17,368+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices (iscsi:445)
2023-10-16 18:21:17,377+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:31)
2023-10-16 18:21:17,378+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.0', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}} from=::1,50158, task_id=750f996b-757e-4ccf-b08f-e82b91a88678 (api:37)
2023-10-16 18:21:17,489+1100 INFO (monitor/ee5b026) [storage.iscsi] Scanning iSCSI devices: 0.12 seconds (utils:373)
2023-10-16 18:21:17,490+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices (hba:42)
2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373)
2023-10-16 18:21:17,544+1100 INFO (monitor/ee5b026) [storage.multipath] Waiting until multipathd is ready (multipath:95)
2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122)
2023-10-16 18:21:19,586+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices (multipath:223)
2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373)
2023-10-16 18:21:19,588+1100 INFO (monitor/ee5b026) [storage.storagedomaincache] Refreshing storage domain cache: 2.22 seconds (utils:373)
2023-10-16 18:21:19,594+1100 INFO (monitor/ee5b026) [IOProcessClient] (ee5b0268-8141-43e0-995e-8b4586217123) Starting client (__init__:340)
2023-10-16 18:21:19,603+1100 INFO (ioprocess/87289) [IOProcess] (ee5b0268-8141-43e0-995e-8b4586217123) Starting ioprocess (__init__:465)
2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717)
2023-10-16 18:21:20,734+1100 INFO (monitor/ee5b026) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123/dom_md/metadata' (check:104)
2023-10-16 18:21:20,777+1100 INFO (check/loop) [storage.monitor] Domain ee5b0268-8141-43e0-995e-8b4586217123 became VALID (monitor:465)
2023-10-16 18:21:20,779+1100 INFO (event/70) [vds] vm_start_stop_lock acquired (clientIF:230)
2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:31)
2023-10-16 18:21:22,387+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cb5a305c-dc34-4a94-bf20-fbec9643d784 (api:37)
2023-10-16 18:21:23,720+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:31)
2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=c4472b6f-95ab-4803-a5fd-b91b006f0e59 (api:37)
2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:31)
2023-10-16 18:21:23,733+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=74b73927-5e56-4483-9f00-d69e5798b4c8 (api:37)
2023-10-16 18:21:24,421+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:31)
2023-10-16 18:21:24,422+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=ce22cf89-56ab-4757-97fb-c8e10692223b (api:37)
2023-10-16 18:21:25,529+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:31)
2023-10-16 18:21:25,540+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=9465754e-6bf1-4328-9a45-e3b521725225 (api:37)
2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:31)
2023-10-16 18:21:25,541+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dab8366e-1533-4155-9635-911729326a1a (api:37)
2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:21:26,665+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:21:27,006+1100 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:31)
2023-10-16 18:21:27,399+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e69c93c1-639c-4640-b7de-23724deb52f2 (api:37)
2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:31)
2023-10-16 18:21:28,689+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00120356', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=fc8a5b61-e928-425a-bd60-36a14d9334e2 (api:37)
2023-10-16 18:21:28,741+1100 WARN (jsonrpc/1) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155)
2023-10-16 18:21:28,742+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:28,744+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305)
2023-10-16 18:21:30,548+1100 WARN (jsonrpc/5) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155)
2023-10-16 18:21:30,549+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:30,550+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305)
2023-10-16 18:21:30,763+1100 INFO (monitor/ee5b026) [storage.sanlock] Acquiring host id for domain ee5b0268-8141-43e0-995e-8b4586217123 (id=1, wait=False) (clusterlock:359)
2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:21:31,712+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:21:32,411+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:31)
2023-10-16 18:21:32,412+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=b0a218d5-91b0-4dc8-9f4c-45825e601a3c (api:37)
2023-10-16 18:21:33,697+1100 WARN (periodic/0) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155)
2023-10-16 18:21:33,753+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:31)
2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=39f0b3ae-475f-4951-90ff-581637710dc9 (api:37)
2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:31)
2023-10-16 18:21:33,765+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=2fd4952e-b521-4c0a-93e0-d1186a376d20 (api:37)
2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:31)
2023-10-16 18:21:34,435+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=fad247c4-b608-4df8-88e6-51bc025f21cd (api:37)
2023-10-16 18:21:35,562+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:35,573+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:31)
2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=37f91533-90ae-4c76-a3e2-c1b8362a9494 (api:37)
2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:31)
2023-10-16 18:21:35,574+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a25a55a8-10df-48fc-82c2-f603096e58cd (api:37)
2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:21:37,022+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:31)
2023-10-16 18:21:37,420+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.6', 'delay': '0.00122199', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8270e681-279e-44a0-a9b0-b7a0110803d7 (api:37)
2023-10-16 18:21:38,773+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155)
2023-10-16 18:21:38,774+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:38,775+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305)
2023-10-16 18:21:40,581+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155)
2023-10-16 18:21:40,582+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:40,583+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305)
2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:21:41,668+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:21:42,430+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:31)
2023-10-16 18:21:42,431+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=3be41433-ac1d-4d72-963d-2a69eef1fb26 (api:37)
2023-10-16 18:21:43,788+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:31)
2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=5ab069ab-377b-47a4-a19c-e8b2606c7453 (api:37)
2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:31)
2023-10-16 18:21:43,800+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=14c99af0-527b-444a-92df-d84724c0d215 (api:37)
2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:31)
2023-10-16 18:21:44,450+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=88a44308-aac3-441f-a05b-bb0809854292 (api:37)
2023-10-16 18:21:45,595+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:31)
2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.8', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=a871b766-4ae8-4c0e-bd44-2d88034cd1b7 (api:37)
2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:31)
2023-10-16 18:21:45,607+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=5db4e011-a522-4c96-8a33-6eba3170787f (api:37)
2023-10-16 18:21:45,610+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155)
2023-10-16 18:21:45,610+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {} (throttledlog:87)
2023-10-16 18:21:46,726+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:21:47,037+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:21:47,440+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:31)
2023-10-16 18:21:47,441+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=7e1eac36-3a04-4e4f-8a20-7cb84b703d1a (api:37)
2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:31)
2023-10-16 18:21:48,712+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00111092', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=internal, task_id=e6378dc2-3d21-4cd5-851d-5db5bc5c18d4 (api:37)
2023-10-16 18:21:48,715+1100 WARN (periodic/3) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155)
2023-10-16 18:21:48,808+1100 WARN (jsonrpc/4) [root] Failed to retrieve Hosted Engine HA info: timed out (api:155)
2023-10-16 18:21:48,809+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:48,810+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getStats took more than 1.00 seconds to succeed: 5.02 (__init__:305)
2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:31)
2023-10-16 18:21:52,453+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=10828206-04bc-4cd4-b5a4-38bd0cf9dc8d (api:37)
2023-10-16 18:21:53,821+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:31)
2023-10-16 18:21:53,833+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=6c48b607-2157-4404-9927-fbfca271c525 (api:37)
2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:31)
2023-10-16 18:21:53,834+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=4e007b1a-3480-4a60-91c9-ccc4a2b8a552 (api:37)
2023-10-16 18:21:53,837+1100 WARN (jsonrpc/7) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155)
2023-10-16 18:21:53,837+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:31)
2023-10-16 18:21:54,463+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=91a5858c-775d-4172-8bd7-085a03865e4b (api:37)
2023-10-16 18:21:55,639+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:31)
2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=20121b0e-9d04-4a74-a7b8-3bbf48d1fd09 (api:37)
2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:31)
2023-10-16 18:21:55,650+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=efa8dcb2-b1cf-4ab5-ae22-7953bdf379e2 (api:37)
2023-10-16 18:21:55,653+1100 WARN (jsonrpc/6) [root] Failed to retrieve Hosted Engine HA info: [Errno 11] Resource temporarily unavailable (api:155)
2023-10-16 18:21:55,653+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:21:56,671+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:21:57,053+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:31)
2023-10-16 18:21:57,462+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00118507', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0a931307-57e8-43ed-bffa-d832f421327a (api:37)
2023-10-16 18:22:00,800+1100 INFO (monitor/ee5b026) [storage.sanlock] Host id 1 for domain ee5b0268-8141-43e0-995e-8b4586217123 acquired in 30 seconds (clusterlock:427)
2023-10-16 18:22:01,744+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:22:01,745+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:31)
2023-10-16 18:22:02,471+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=4813e4e9-f151-4727-9cf7-fcf87d1ffdcd (api:37)
2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:31)
2023-10-16 18:22:02,569+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=859cacca-c5a4-4eb9-b5a8-c28ef96b464d (api:37)
2023-10-16 18:22:03,663+1100 INFO (health) [health] LVM cache hit ratio: 39.35% (hits: 170 misses: 262) (health:114)
2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:31)
2023-10-16 18:22:03,729+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=bb148f84-5eab-4cac-b563-b89d690db0bb (api:37)
2023-10-16 18:22:03,862+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:31)
2023-10-16 18:22:03,873+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=305776e1-ad6e-4530-ae48-0f58899b0f6e (api:37)
2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:31)
2023-10-16 18:22:03,874+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f1300145-b974-4f32-90a7-553a6cefe028 (api:37)
2023-10-16 18:22:03,879+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:31)
2023-10-16 18:22:04,475+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ee277021-41dc-414c-9886-c01c807e5658 (api:37)
2023-10-16 18:22:05,682+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:31)
2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00125367', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d1d6d329-10cb-4439-9218-e00b797220ab (api:37)
2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:31)
2023-10-16 18:22:05,694+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=1e8c24fa-9d0c-49ea-8f85-f06527068e73 (api:37)
2023-10-16 18:22:05,699+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:22:07,068+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:22:11,675+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:22:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich
apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=60.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback:
File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap
self._bootstrap_inner()
File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner
self.run()
File: "/usr/lib64/python3.6/threading.py", line 885, in run
self._target(*self._args, **self._kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run
ret = func(*args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run
self._execute_task()
File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task
task()
File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__
self._callable()
File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__
self._handler(self._ctx, self._req)
File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest
response = self._handle_request(req, ctx)
File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request
res = method(**params)
File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod
result = fn(*methodArgs)
File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer
connectionParams)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper
result = ctask.prepare(func, *args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper
return m(self, *a, **kw)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare
result = self._run(func, *args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run
return fn(*args, **kargs)
File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer
File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method
ret = func(*args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer
results = storageServer.connect(domType, conList)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect
return con_class.connect_all(connections)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all
for res in login_results:
File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap
yield results.get()
File: "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File: "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire() (executor:346)
2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:31)
2023-10-16 18:22:12,654+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.8', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=837a670b-015f-44ad-9f05-d3ca615f3981 (api:37)
2023-10-16 18:22:13,908+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:31)
2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=399b71ba-f296-4c9c-9ead-870d40d751fe (api:37)
2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:31)
2023-10-16 18:22:13,921+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=0b89ae1d-b45c-4274-a63d-f5dfa0f9392e (api:37)
2023-10-16 18:22:13,927+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:22:14,253+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:31)
2023-10-16 18:22:14,488+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=8f652e74-faae-4ac9-a6cb-1c114c23cf48 (api:37)
2023-10-16 18:22:15,726+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:31)
2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '4.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=f85241ec-5b32-4efa-b1f9-8592a9fd57d5 (api:37)
2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:31)
2023-10-16 18:22:15,737+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=01d80faa-e134-42d1-bce2-28f91748a9e3 (api:37)
2023-10-16 18:22:15,743+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:22:16,763+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:22:17,082+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:31)
2023-10-16 18:22:18,750+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00122013', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=468af5ce-aa49-48f1-ac7c-762cc713710a (api:37)
2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Closing client (__init__:641)
2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Closing client (__init__:641)
2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Closing client (__init__:641)
2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (Global) Closing client (__init__:641)
2023-10-16 18:22:20,813+1100 INFO (monitor/ee5b026) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Closing client (__init__:641)
2023-10-16 18:22:20,814+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Killing ioprocess (__init__:191)
2023-10-16 18:22:20,814+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Killing ioprocess (__init__:191)
2023-10-16 18:22:20,814+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Killing ioprocess (__init__:191)
2023-10-16 18:22:20,814+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) Killing ioprocess (__init__:191)
2023-10-16 18:22:20,814+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Killing ioprocess (__init__:191)
2023-10-16 18:22:20,815+1100 INFO (ioprocess/87022) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:22:20,815+1100 INFO (ioprocess/86760) [IOProcessClient] (Global) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:22:20,815+1100 INFO (ioprocess/87016) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:22:20,816+1100 INFO (ioprocess/87027) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:22:20,816+1100 INFO (ioprocess/77141) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) ioprocess was terminated by signal 9 (__init__:200)
2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:31)
2023-10-16 18:22:22,741+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.9', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=62257d75-2f2b-432d-a752-e0ee3e8fd29b (api:37)
2023-10-16 18:22:23,954+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:31)
2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.1', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=78b74721-7d6c-4ed2-88f0-4fff08c4b407 (api:37)
2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:31)
2023-10-16 18:22:23,967+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78ff047f-a712-4dbb-99e3-25909b977799 (api:37)
2023-10-16 18:22:23,974+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:31)
2023-10-16 18:22:24,503+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0a2888a2-c8ae-49e5-89fc-a6b5bebfd77f (api:37)
2023-10-16 18:22:25,769+1100 INFO (jsonrpc/7) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:31)
2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00136499', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=59f0f76f-e6d4-47cf-8ce0-6b3f9a7e850d (api:37)
2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] START multipath_health() from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:31)
2023-10-16 18:22:25,780+1100 INFO (jsonrpc/7) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c5576936-9a21-4b28-886e-b6cd73c90cd4 (api:37)
2023-10-16 18:22:25,786+1100 INFO (jsonrpc/7) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:22:26,678+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:22:27,095+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:22:31,782+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:22:31,783+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:22:32,825+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:31)
2023-10-16 18:22:32,826+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=66cc2698-9c5a-4c9d-803b-d89bd549f4f0 (api:37)
2023-10-16 18:22:33,771+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:31)
2023-10-16 18:22:33,772+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c267fd6f-633d-4c6d-94ad-b5cb5bb8d02d (api:37)
2023-10-16 18:22:33,998+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:31)
2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=2654a62d-b4f9-425c-8442-b4c418c4f542 (api:37)
2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:31)
2023-10-16 18:22:34,009+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=c02547e7-20f6-437a-84b4-4fdf98cfd003 (api:37)
2023-10-16 18:22:34,014+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:34,515+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:31)
2023-10-16 18:22:34,516+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=667f9b48-b2ab-4113-ae86-1b586bbdf5d6 (api:37)
2023-10-16 18:22:35,812+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:31)
2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00115401', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c04a2aa5-1f9d-4c87-b51f-34eaeeedb8e2 (api:37)
2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:31)
2023-10-16 18:22:35,824+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=f6ac211c-50fa-481a-aeaa-01779cc54412 (api:37)
2023-10-16 18:22:35,830+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:37,110+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:22:37,111+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:22:41,681+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:31)
2023-10-16 18:22:42,911+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c1a8bf5a-4aa0-4c85-a9fa-8c240a07daa9 (api:37)
2023-10-16 18:22:44,040+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:31)
2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=d7eb5985-0470-4aea-bd23-937c1a03da72 (api:37)
2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:31)
2023-10-16 18:22:44,053+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=a32bf2e0-f35c-46f4-b582-a28de56e1960 (api:37)
2023-10-16 18:22:44,059+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:31)
2023-10-16 18:22:44,528+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=68c2e209-d655-437d-add5-769c0a4932ba (api:37)
2023-10-16 18:22:45,855+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:31)
2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=b93990c2-b263-4c89-ba4b-ab0e962f6ce8 (api:37)
2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:31)
2023-10-16 18:22:45,867+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=444bef31-e827-478f-a2fc-16e18dc0b9a6 (api:37)
2023-10-16 18:22:45,873+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:46,786+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:22:46,787+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:22:47,126+1100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:31)
2023-10-16 18:22:48,792+1100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00120423', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=50b7c32e-202c-40ec-b484-5eb3f45f7f28 (api:37)
2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:31)
2023-10-16 18:22:52,996+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=28687ab3-2149-42d2-92aa-d28cc5ad8d96 (api:37)
2023-10-16 18:22:54,086+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:31)
2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.2', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=96f37ad8-6f60-4c9b-8986-cad8c4a5bdf3 (api:37)
2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:31)
2023-10-16 18:22:54,099+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=aa6dc50d-c1c9-4a8c-bf9d-b7ca456f73a6 (api:37)
2023-10-16 18:22:54,106+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:31)
2023-10-16 18:22:54,533+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ada1e11f-23e9-4704-a8c4-27bfecfd9974 (api:37)
2023-10-16 18:22:55,899+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:31)
2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132588', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a0dd0ff4-5a0a-43cf-8612-ac5ff8e953a8 (api:37)
2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:31)
2023-10-16 18:22:55,911+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9211755e-e9f9-4c7b-b0dc-8a7689a7884a (api:37)
2023-10-16 18:22:55,916+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:22:56,684+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:22:57,141+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:22:57,142+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:23:01,805+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:31)
2023-10-16 18:23:03,079+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.2', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=95612d72-c18c-4dfe-87f2-f8501f713aca (api:37)
2023-10-16 18:23:03,813+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:31)
2023-10-16 18:23:03,814+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=c193e1ec-73f7-4858-aa76-fb9f7d977517 (api:37)
2023-10-16 18:23:04,130+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:31)
2023-10-16 18:23:04,141+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1df7dacf-ebad-4493-ad11-a7ce10fbf851 (api:37)
2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:31)
2023-10-16 18:23:04,142+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=cbc0b187-25a2-43aa-bfa5-9cf54f675e08 (api:37)
2023-10-16 18:23:04,147+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:31)
2023-10-16 18:23:04,541+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=77176abe-b0e4-4f22-bce7-2f26f571743d (api:37)
2023-10-16 18:23:05,942+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:31)
2023-10-16 18:23:05,954+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00103904', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=99736bd8-b5b6-4e29-a345-cec8db30f976 (api:37)
2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:31)
2023-10-16 18:23:05,955+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=286f88d9-373c-42c6-b990-b70d2eb73137 (api:37)
2023-10-16 18:23:05,960+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:23:07,157+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:23:11,687+1100 INFO (jsonrpc/5) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:23:11,925+1100 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'StoragePool.connectStorageServer', 'params': {'storagepoolID': '7bfaa890-f892-4b3b-bf18-33dabc13e646', 'domainType': 3, 'connectionParams': [{'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.24', 'ipv6_enabled': 'false', 'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'user': 'cephiscsichapinitiator', 'tpgt': '3'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.25', 'ipv6_enabled': 'false', 'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'user': 'cephiscsichapinitiator', 'tpgt': '2'}, {'password': '********', 'port': '3260', 'iqn': 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', 'connection': '192.168.4.23', 'ipv6_enabled': 'false', 'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'user': 'cephiscsich
apinitiator', 'tpgt': '1'}]}, 'id': 'd19073fd-9420-4dee-ac37-509a5b321b58'} at 0x7f877a5fa128> timeout=60, duration=120.00 at 0x7f877a69d550> task#=1179 at 0x7f8778933240>, traceback:
File: "/usr/lib64/python3.6/threading.py", line 905, in _bootstrap
self._bootstrap_inner()
File: "/usr/lib64/python3.6/threading.py", line 937, in _bootstrap_inner
self.run()
File: "/usr/lib64/python3.6/threading.py", line 885, in run
self._target(*self._args, **self._kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 243, in run
ret = func(*args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 284, in _run
self._execute_task()
File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 298, in _execute_task
task()
File: "/usr/lib/python3.6/site-packages/vdsm/executor.py", line 374, in __call__
self._callable()
File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 253, in __call__
self._handler(self._ctx, self._req)
File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest
response = self._handle_request(req, ctx)
File: "/usr/lib/python3.6/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request
res = method(**params)
File: "/usr/lib/python3.6/site-packages/vdsm/rpc/Bridge.py", line 182, in _dynamicMethod
result = fn(*methodArgs)
File: "/usr/lib/python3.6/site-packages/vdsm/API.py", line 1117, in connectStorageServer
connectionParams)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/dispatcher.py", line 57, in wrapper
result = ctask.prepare(func, *args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 93, in wrapper
return m(self, *a, **kw)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1163, in prepare
result = self._run(func, *args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 867, in _run
return fn(*args, **kargs)
File: "</usr/lib/python3.6/site-packages/decorator.py:decorator-gen-117>", line 2, in connectStorageServer
File: "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 33, in method
ret = func(*args, **kwargs)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 2164, in connectStorageServer
results = storageServer.connect(domType, conList)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 914, in connect
return con_class.connect_all(connections)
File: "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 617, in connect_all
for res in login_results:
File: "/usr/lib/python3.6/site-packages/vdsm/common/concurrent.py", line 207, in tmap
yield results.get()
File: "/usr/lib64/python3.6/queue.py", line 164, in get
self.not_empty.wait()
File: "/usr/lib64/python3.6/threading.py", line 295, in wait
waiter.acquire() (executor:346)
2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:31)
2023-10-16 18:23:13,165+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=0cb1a91c-f096-4daa-bb08-ec407a86433a (api:37)
2023-10-16 18:23:14,173+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:31)
2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=fbf38d2b-bc3e-49ad-afd9-de55861129db (api:37)
2023-10-16 18:23:14,185+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:31)
2023-10-16 18:23:14,186+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=3e936d0b-7879-45c8-89d4-ae6bbae466a2 (api:37)
2023-10-16 18:23:14,192+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:23:14,314+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:31)
2023-10-16 18:23:14,554+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=316fa299-920a-4c26-a829-9e93647ea804 (api:37)
2023-10-16 18:23:15,986+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:31)
2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=c7094132-9870-44f0-8274-54527984da18 (api:37)
2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:31)
2023-10-16 18:23:15,998+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=61b7e2c2-106d-42ff-b6cd-e91e557af60d (api:37)
2023-10-16 18:23:16,003+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:23:16,823+1100 INFO (jsonrpc/6) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:23:17,172+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:31)
2023-10-16 18:23:18,834+1100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '7.9', 'delay': '0.00132095', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5086291d-9172-41f1-9db9-7011a6a34abb (api:37)
2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:31)
2023-10-16 18:23:23,249+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=bf9f76c3-bead-4087-947b-ebd063ebab6d (api:37)
2023-10-16 18:23:24,217+1100 INFO (jsonrpc/1) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:31)
2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.3', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=3296546b-846b-48f8-b2f3-9fa13e06864e (api:37)
2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] START multipath_health() from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:31)
2023-10-16 18:23:24,229+1100 INFO (jsonrpc/1) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=77d3abd5-6cf4-4a3f-874e-0a0a1adcf3d5 (api:37)
2023-10-16 18:23:24,236+1100 INFO (jsonrpc/1) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:31)
2023-10-16 18:23:24,566+1100 INFO (jsonrpc/7) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.7', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=12ad1f12-b845-4107-9bc0-326453b99721 (api:37)
2023-10-16 18:23:26,030+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:31)
2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '5.1', 'delay': '0.00160067', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=cf9b3b2a-8182-440a-82b8-0c2dd0e33cee (api:37)
2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:31)
2023-10-16 18:23:26,041+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=20a2b03c-eee3-4d1d-a47e-cc6e581df61f (api:37)
2023-10-16 18:23:26,047+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:26,373+1100 INFO (iscsi-login/0) [storage.iscsi] Removing iscsi node for target 192.168.4.24:3260,3 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227)
2023-10-16 18:23:26,388+1100 INFO (iscsi-login/1) [storage.iscsi] Removing iscsi node for target 192.168.4.25:3260,2 iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw iface default (iscsi:227)
2023-10-16 18:23:26,436+1100 ERROR (iscsi-login/0) [storage.storageServer] Could not login to target (storageServer:607)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login
"-p", portal, "-l"])
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd
out = commands.run(cmd, sudo=True)
File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run
raise cmdutils.Error(args, p.returncode, out, err)
vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.24:3260,3', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login
iscsi.loginToIscsiNode(con.iface, con.target)
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login
raise IscsiNodeError(e.rc, e.out, e.err)
vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.24,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n')
2023-10-16 18:23:26,497+1100 ERROR (iscsi-login/1) [storage.storageServer] Could not login to target (storageServer:607)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 301, in node_login
"-p", portal, "-l"])
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 104, in run_cmd
out = commands.run(cmd, sudo=True)
File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 85, in run
raise cmdutils.Error(args, p.returncode, out, err)
vdsm.common.cmdutils.Error: Command ['/sbin/iscsiadm', '-m', 'node', '-T', 'iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw', '-I', 'default', '-p', '192.168.4.25:3260,2', '-l'] failed with rc=11 out=b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n' err=b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/storageServer.py", line 604, in iscsi_login
iscsi.loginToIscsiNode(con.iface, con.target)
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsi.py", line 216, in loginToIscsiNode
iscsiadm.node_login(iface.name, target.address, target.iqn)
File "/usr/lib/python3.6/site-packages/vdsm/storage/iscsiadm.py", line 321, in node_login
raise IscsiNodeError(e.rc, e.out, e.err)
vdsm.storage.iscsiadm.IscsiNodeError: (11, b'Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260]\n', b'iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw, portal: 192.168.4.25,3260].\niscsiadm: initiator reported error (11 - iSCSI PDU timed out)\niscsiadm: Could not log into all portals\n')
2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63)
2023-10-16 18:23:26,514+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices (iscsi:445)
2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.iscsi] Scanning iSCSI devices: 0.09 seconds (utils:373)
2023-10-16 18:23:26,609+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices (hba:42)
2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373)
2023-10-16 18:23:26,663+1100 INFO (jsonrpc/3) [storage.multipath] Waiting until multipathd is ready (multipath:95)
2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:23:26,690+1100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:23:27,186+1100 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122)
2023-10-16 18:23:28,705+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices (multipath:223)
2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373)
2023-10-16 18:23:28,706+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373)
2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Removing domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 from storage domain cache (sdc:194)
2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57)
2023-10-16 18:23:28,796+1100 INFO (jsonrpc/3) [vdsm.api] FINISH connectStorageServer return={'statuslist': [{'id': '20a52c08-a4c5-4b07-8ecc-bec0e82e582a', 'status': 0}, {'id': '78f16782-9a06-45f2-92e5-d1ca84a8c67e', 'status': 465}, {'id': '868d7c6a-c933-4475-86ce-ab1359d6f4c3', 'status': 465}]} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=32063392-d746-458c-b143-919fbf9df35b (api:37)
2023-10-16 18:23:28,797+1100 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call StoragePool.connectStorageServer took more than 1.00 seconds to succeed: 136.87 (__init__:305)
2023-10-16 18:23:28,847+1100 INFO (jsonrpc/5) [vdsm.api] START connectStoragePool(spUUID='7bfaa890-f892-4b3b-bf18-33dabc13e646', hostID=1, msdUUID='ee5b0268-8141-43e0-995e-8b4586217123', masterVersion=1, domainsMap={'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'active', '30775056-38ef-483a-889c-70453e9aa5f9': 'active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'active', '4412e492-f752-4a6c-98ff-f289043618da': 'active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'active'}) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:31)
2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepoolmemorybackend] new storage pool master version 1 and domains map {'e311ddf1-7f2c-49ef-a618-050d9a2b947f': 'Active', 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': 'Active', '30775056-38ef-483a-889c-70453e9aa5f9': 'Active', 'ee5b0268-8141-43e0-995e-8b4586217123': 'Active', '3e33b7ce-7279-4765-a203-cca37eff8cb0': 'Active', '4412e492-f752-4a6c-98ff-f289043618da': 'Active', '0dde39a4-42e4-4fba-8068-e63e42dd3334': 'Active', 'd44ec7f2-3629-4110-93a5-7d8d57230efc': 'Active'} (spbackends:417)
2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] updating pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 backend from type NoneType instance 0x7f87cc236b70 to type StoragePoolMemoryBackend instance 0x7f877a7cdcc8 (sp:149)
2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagepool] Connect host #1 to the storage pool 7bfaa890-f892-4b3b-bf18-33dabc13e646 with master domain: ee5b0268-8141-43e0-995e-8b4586217123 (ver = 1) (sp:699)
2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Invalidating storage domain cache (sdc:57)
2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Clearing storage domain cache (sdc:182)
2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache (resize=True) (sdc:63)
2023-10-16 18:23:28,848+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices (iscsi:445)
2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.iscsi] Scanning iSCSI devices: 0.10 seconds (utils:373)
2023-10-16 18:23:28,946+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices (hba:42)
2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.hba] Scanning FC devices: 0.05 seconds (utils:373)
2023-10-16 18:23:28,996+1100 INFO (jsonrpc/5) [storage.multipath] Waiting until multipathd is ready (multipath:95)
2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Waited 2.04 seconds for multipathd (tries=2, ready=2) (multipath:122)
2023-10-16 18:23:31,033+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices (multipath:223)
2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.multipath] Resizing multipath devices: 0.00 seconds (utils:373)
2023-10-16 18:23:31,034+1100 INFO (jsonrpc/5) [storage.storagedomaincache] Refreshing storage domain cache: 2.19 seconds (utils:373)
2023-10-16 18:23:31,035+1100 INFO (tmap-757/0) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting client (__init__:340)
2023-10-16 18:23:31,043+1100 INFO (tmap-757/1) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting client (__init__:340)
2023-10-16 18:23:31,045+1100 INFO (ioprocess/87928) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv0) Starting ioprocess (__init__:465)
2023-10-16 18:23:31,052+1100 INFO (tmap-757/2) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting client (__init__:340)
2023-10-16 18:23:31,054+1100 INFO (ioprocess/87936) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:gv1) Starting ioprocess (__init__:465)
2023-10-16 18:23:31,059+1100 INFO (tmap-757/3) [IOProcessClient] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting client (__init__:340)
2023-10-16 18:23:31,062+1100 INFO (ioprocess/87940) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv2) Starting ioprocess (__init__:465)
2023-10-16 18:23:31,068+1100 INFO (ioprocess/87948) [IOProcess] (glusterSD/vmhosttran1.mydoamin.local:_gv3) Starting ioprocess (__init__:465)
2023-10-16 18:23:31,842+1100 INFO (jsonrpc/7) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:23:31,843+1100 INFO (jsonrpc/7) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:23:32,194+1100 INFO (jsonrpc/5) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717)
2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 0dde39a4-42e4-4fba-8068-e63e42dd3334 (monitor:176)
2023-10-16 18:23:32,195+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 3e33b7ce-7279-4765-a203-cca37eff8cb0 (monitor:176)
2023-10-16 18:23:32,201+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring e311ddf1-7f2c-49ef-a618-050d9a2b947f (monitor:176)
2023-10-16 18:23:32,206+1100 INFO (jsonrpc/5) [storage.monitor] Attaching monitor for ee5b0268-8141-43e0-995e-8b4586217123 to the pool (monitor:172)
2023-10-16 18:23:32,207+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 4412e492-f752-4a6c-98ff-f289043618da (monitor:176)
2023-10-16 18:23:32,208+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring fa8d922a-df3c-4f5b-adf6-a77c3689c397 (monitor:176)
2023-10-16 18:23:32,210+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring 30775056-38ef-483a-889c-70453e9aa5f9 (monitor:176)
2023-10-16 18:23:32,211+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154)
2023-10-16 18:23:32,218+1100 INFO (jsonrpc/5) [storage.monitor] Start monitoring d44ec7f2-3629-4110-93a5-7d8d57230efc (monitor:176)
2023-10-16 18:23:32,219+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154)
2023-10-16 18:23:32,230+1100 INFO (monitor/3e33b7c) [IOProcessClient] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting client (__init__:340)
2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.storagepool] Creating pool directory '/rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646' (sp:1323)
2023-10-16 18:23:32,234+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646 mode: None (fileUtils:214)
2023-10-16 18:23:32,239+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154)
2023-10-16 18:23:32,260+1100 INFO (monitor/e311ddf) [IOProcessClient] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting client (__init__:340)
2023-10-16 18:23:32,263+1100 INFO (ioprocess/87989) [IOProcess] (3e33b7ce-7279-4765-a203-cca37eff8cb0) Starting ioprocess (__init__:465)
2023-10-16 18:23:32,268+1100 INFO (monitor/4412e49) [IOProcessClient] (4412e492-f752-4a6c-98ff-f289043618da) Starting client (__init__:340)
2023-10-16 18:23:32,271+1100 INFO (ioprocess/88003) [IOProcess] (e311ddf1-7f2c-49ef-a618-050d9a2b947f) Starting ioprocess (__init__:465)
2023-10-16 18:23:32,278+1100 INFO (ioprocess/88012) [IOProcess] (4412e492-f752-4a6c-98ff-f289043618da) Starting ioprocess (__init__:465)
2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.11 seconds (utils:373)
2023-10-16 18:23:32,323+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',)
2023-10-16 18:23:32,323+1100 INFO (monitor/fa8d922) [storage.monitor] Domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 became INVALID (monitor:465)
2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.11 seconds (utils:373)
2023-10-16 18:23:32,334+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',)
2023-10-16 18:23:32,334+1100 INFO (monitor/3077505) [storage.monitor] Domain 30775056-38ef-483a-889c-70453e9aa5f9 became INVALID (monitor:465)
2023-10-16 18:23:32,339+1100 WARN (jsonrpc/5) [storage.lvm] Marked vgs=[0dde39a4-42e4-4fba-8068-e63e42dd3334] as Unreadable due to reload failure (lvm:679)
2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain metadata directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (sp:1354)
2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214)
2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.storagepool] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (sp:1357)
2023-10-16 18:23:32,339+1100 INFO (jsonrpc/5) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214)
2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.10 seconds (utils:373)
2023-10-16 18:23:32,340+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',)
2023-10-16 18:23:32,340+1100 INFO (monitor/d44ec7f) [storage.monitor] Domain d44ec7f2-3629-4110-93a5-7d8d57230efc became INVALID (monitor:465)
2023-10-16 18:23:32,345+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1295)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1295)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1295)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1295)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1278)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/ee5b0268-8141-43e0-995e-8b4586217123 (sp:1295)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1278)
2023-10-16 18:23:32,346+1100 INFO (jsonrpc/5) [storage.storagepool] Creating symlink from /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv0/ee5b0268-8141-43e0-995e-8b4586217123 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/mastersd (sp:1295)
2023-10-16 18:23:32,383+1100 INFO (jsonrpc/5) [vdsm.api] FINISH connectStoragePool return=True from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=22550292-d106-45d1-b5b7-17ff11f6f8a7 (api:37)
2023-10-16 18:23:32,384+1100 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call StoragePool.connect took more than 1.00 seconds to succeed: 3.54 (__init__:305)
2023-10-16 18:23:32,385+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['ids'] (lvm:1839)
2023-10-16 18:23:32,390+1100 INFO (jsonrpc/2) [api.host] START getStats() from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:31)
2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:31)
2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.2', 'delay': '0', 'valid': True, 'version': -1, 'acquired': False, 'actual': False}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actua
l': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.1', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=ac4e0a4f-b2d6-4d28-b765-6ec18d5ecb94 (api:37)
2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:31)
2023-10-16 18:23:32,404+1100 INFO (jsonrpc/2) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, flow_id=706708f9, task_id=c006784d-0604-489e-a8a8-8be4c44e3bec (api:37)
2023-10-16 18:23:32,413+1100 INFO (jsonrpc/2) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698, flow_id=706708f9 (api:37)
2023-10-16 18:23:32,572+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['leases'] (lvm:1839)
2023-10-16 18:23:32,771+1100 INFO (monitor/0dde39a) [storage.lvm] Activating lvs: vg=0dde39a4-42e4-4fba-8068-e63e42dd3334 lvs=['metadata', 'inbox', 'outbox', 'xleases', 'master'] (lvm:1839)
2023-10-16 18:23:32,975+1100 INFO (monitor/0dde39a) [storage.blocksd] Setting up domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (blockSD:973)
2023-10-16 18:23:33,056+1100 INFO (monitor/0dde39a) [storage.check] Start checking '/dev/0dde39a4-42e4-4fba-8068-e63e42dd3334/metadata' (check:104)
2023-10-16 18:23:33,143+1100 INFO (monitor/0dde39a) [storage.monitor] Domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 became VALID (monitor:465)
2023-10-16 18:23:33,144+1100 INFO (monitor/0dde39a) [storage.sanlock] Acquiring host id for domain 0dde39a4-42e4-4fba-8068-e63e42dd3334 (id=1, wait=False) (clusterlock:359)
2023-10-16 18:23:33,145+1100 INFO (event/81) [vds] vm_start_stop_lock acquired (clientIF:230)
2023-10-16 18:23:33,145+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain images directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images' (blockSD:766)
2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/images mode: None (fileUtils:214)
2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagedomainmanifest] Creating domain special volumes directory '/rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md' (blockSD:771)
2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.fileutils] Creating directory: /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334/dom_md mode: None (fileUtils:214)
2023-10-16 18:23:33,146+1100 INFO (event/82) [storage.storagepool] Linking /rhev/data-center/mnt/blockSD/0dde39a4-42e4-4fba-8068-e63e42dd3334 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/0dde39a4-42e4-4fba-8068-e63e42dd3334 (sp:1278)
2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:31)
2023-10-16 18:23:33,338+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=50cf6835-08a1-4b31-8a61-6e18b85fb86e (api:37)
2023-10-16 18:23:33,470+1100 INFO (monitor/3e33b7c) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717)
2023-10-16 18:23:33,471+1100 INFO (monitor/3e33b7c) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0/dom_md/metadata' (check:104)
2023-10-16 18:23:33,474+1100 INFO (monitor/4412e49) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717)
2023-10-16 18:23:33,475+1100 INFO (monitor/4412e49) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da/dom_md/metadata' (check:104)
2023-10-16 18:23:33,519+1100 INFO (check/loop) [storage.monitor] Domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 became VALID (monitor:465)
2023-10-16 18:23:33,521+1100 INFO (event/84) [vds] vm_start_stop_lock acquired (clientIF:230)
2023-10-16 18:23:33,522+1100 INFO (check/loop) [storage.monitor] Domain 4412e492-f752-4a6c-98ff-f289043618da became VALID (monitor:465)
2023-10-16 18:23:33,523+1100 INFO (event/85) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv2/3e33b7ce-7279-4765-a203-cca37eff8cb0 to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/3e33b7ce-7279-4765-a203-cca37eff8cb0 (sp:1278)
2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.storagedomain] Removing remnants of deleted images [] (fileSD:717)
2023-10-16 18:23:33,524+1100 INFO (monitor/e311ddf) [storage.check] Start checking '/rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f/dom_md/metadata' (check:104)
2023-10-16 18:23:33,528+1100 INFO (event/87) [vds] vm_start_stop_lock acquired (clientIF:230)
2023-10-16 18:23:33,534+1100 INFO (event/88) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:_gv3/4412e492-f752-4a6c-98ff-f289043618da to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/4412e492-f752-4a6c-98ff-f289043618da (sp:1278)
2023-10-16 18:23:33,542+1100 INFO (check/loop) [storage.monitor] Domain e311ddf1-7f2c-49ef-a618-050d9a2b947f became VALID (monitor:465)
2023-10-16 18:23:33,543+1100 INFO (event/90) [vds] vm_start_stop_lock acquired (clientIF:230)
2023-10-16 18:23:33,543+1100 INFO (event/91) [storage.storagepool] Linking /rhev/data-center/mnt/glusterSD/vmhosttran1.mydoamin.local:gv1/e311ddf1-7f2c-49ef-a618-050d9a2b947f to /rhev/data-center/7bfaa890-f892-4b3b-bf18-33dabc13e646/e311ddf1-7f2c-49ef-a618-050d9a2b947f (sp:1278)
2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:31)
2023-10-16 18:23:33,856+1100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.6', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version':
-1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=67318041-086e-4992-a2c3-dc0b7e5ebd4e (api:37)
2023-10-16 18:23:34,263+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:31)
2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '2.0', 'delay': '0', 'valid': False, 'version': -
1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=8cf2bf3c-b4ac-4f24-9179-29054ce22a46 (api:37)
2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:31)
2023-10-16 18:23:34,275+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dfd9aafe-1cbf-445f-afa9-6b19287c9955 (api:37)
2023-10-16 18:23:34,280+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:31)
2023-10-16 18:23:34,572+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=efc38225-ad10-4a67-97d8-1f7f2854456d (api:37)
2023-10-16 18:23:36,073+1100 INFO (jsonrpc/0) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:36,084+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:31)
2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00160059', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '2.9', 'delay': '0.00174884', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00116006', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.00604746', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00091358', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': -
1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=e1664ede-6677-4756-97d9-d1b033ebb9b6 (api:37)
2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] START multipath_health() from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:31)
2023-10-16 18:23:36,085+1100 INFO (jsonrpc/0) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=592b6419-5402-4f74-83ae-b088d9b4b978 (api:37)
2023-10-16 18:23:36,090+1100 INFO (jsonrpc/0) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:23:37,204+1100 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:23:41,693+1100 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:23:41,694+1100 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:23:42,324+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154)
2023-10-16 18:23:42,335+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154)
2023-10-16 18:23:42,341+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154)
2023-10-16 18:23:42,423+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.09 seconds (utils:373)
2023-10-16 18:23:42,423+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',)
2023-10-16 18:23:42,426+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.10 seconds (utils:373)
2023-10-16 18:23:42,426+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',)
2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:31)
2023-10-16 18:23:42,428+1100 INFO (jsonrpc/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=139f761e-b9f7-4451-8864-f96a498176ed (api:37)
2023-10-16 18:23:42,432+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.09 seconds (utils:373)
2023-10-16 18:23:42,432+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',)
2023-10-16 18:23:43,488+1100 INFO (monitor/4412e49) [storage.sanlock] Acquiring host id for domain 4412e492-f752-4a6c-98ff-f289043618da (id=1, wait=False) (clusterlock:359)
2023-10-16 18:23:43,488+1100 INFO (monitor/3e33b7c) [storage.sanlock] Acquiring host id for domain 3e33b7ce-7279-4765-a203-cca37eff8cb0 (id=1, wait=False) (clusterlock:359)
2023-10-16 18:23:43,541+1100 INFO (monitor/e311ddf) [storage.sanlock] Acquiring host id for domain e311ddf1-7f2c-49ef-a618-050d9a2b947f (id=1, wait=False) (clusterlock:359)
2023-10-16 18:23:44,308+1100 INFO (jsonrpc/4) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:31)
2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.1', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '1.2', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '0.7', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '0.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1,
'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '1.9', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=0d42d65f-0534-47c0-aaf9-7dbac3067a91 (api:37)
2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] START multipath_health() from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:31)
2023-10-16 18:23:44,321+1100 INFO (jsonrpc/4) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=dad656d9-6a05-443b-82ca-c4b7632334dd (api:37)
2023-10-16 18:23:44,328+1100 INFO (jsonrpc/4) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:31)
2023-10-16 18:23:44,587+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=ae63dc53-56a4-4d9a-8bc1-63c6f3c2f160 (api:37)
2023-10-16 18:23:46,096+1100 INFO (jsonrpc/3) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:31)
2023-10-16 18:23:46,107+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.5', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1,
'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=9d13808b-d90d-449b-9f9c-85e91b946eeb (api:37)
2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:31)
2023-10-16 18:23:46,108+1100 INFO (jsonrpc/3) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=53f94ff6-5532-41b5-ae7a-3271108c20d2 (api:37)
2023-10-16 18:23:46,113+1100 INFO (jsonrpc/3) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:23:46,121+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:31)
2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.6', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1,
'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.7', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=19f3fa69-f932-4edd-96f6-76ba56449cf2 (api:37)
2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:31)
2023-10-16 18:23:46,132+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=78db57fe-4ab9-4256-b239-2fb8fa2c6a61 (api:37)
2023-10-16 18:23:46,137+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:23:46,860+1100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:23:47,219+1100 INFO (jsonrpc/4) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:23:47,220+1100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] START repoStats(domains=()) from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:31)
2023-10-16 18:23:48,878+1100 INFO (periodic/2) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '6.7', 'delay': '0.00151051', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '5.7', 'delay': '0.00285536', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '5.4', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '5.3', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1,
'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '6.5', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '6.4', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=internal, task_id=56a338bf-2d23-4dfe-8065-e8e5acc5736a (api:37)
2023-10-16 18:23:52,424+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154)
2023-10-16 18:23:52,428+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154)
2023-10-16 18:23:52,433+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154)
2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:31)
2023-10-16 18:23:52,516+1100 INFO (jsonrpc/3) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=a77c47c6-03ba-40fd-9bfa-29375ea32ae4 (api:37)
2023-10-16 18:23:52,542+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.12 seconds (utils:373)
2023-10-16 18:23:52,543+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',)
2023-10-16 18:23:52,546+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.11 seconds (utils:373)
2023-10-16 18:23:52,546+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',)
2023-10-16 18:23:52,551+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.12 seconds (utils:373)
2023-10-16 18:23:52,551+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',)
2023-10-16 18:23:53,353+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:31)
2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '1.1', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '0.2', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '9.9', 'delay': '0.0011416', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0128674', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '9.8', 'delay': '0.0150353', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1,
'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '0.8', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=72f4cb81-c54d-4153-9425-0a5766308ed4 (api:37)
2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:31)
2023-10-16 18:23:53,366+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=65572c4a-c042-49c8-834d-dee8d879896a (api:37)
2023-10-16 18:23:53,373+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:31)
2023-10-16 18:23:54,605+1100 INFO (jsonrpc/0) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '2.4', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=1ba074d9-5e35-4527-93ca-0423ed3187f8 (api:37)
2023-10-16 18:23:56,164+1100 INFO (jsonrpc/6) [api.host] START getStats() from=::1,50158 (api:31)
2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:31)
2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '3.9', 'delay': '0.00151323', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '3.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '2.7', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '2.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '2.7', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version':
-1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '3.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::1,50158, task_id=cdb6b35e-ab67-48e1-a210-2f4ab737a00f (api:37)
2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:31)
2023-10-16 18:23:56,176+1100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::1,50158, task_id=9099f061-8efd-4c2e-9669-ff848b5aa791 (api:37)
2023-10-16 18:23:56,182+1100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::1,50158 (api:37)
2023-10-16 18:23:56,696+1100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:23:56,697+1100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] START getStats() from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:31)
2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'} (api:112)
2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'b861933b-556a-440a-b21b-6d4b9ef39ed5'}"}} from=::1,50158, vmId=b861933b-556a-440a-b21b-6d4b9ef39ed5 (api:37)
2023-10-16 18:23:57,235+1100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:301)
2023-10-16 18:24:01,111+1100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:192.168.2.28,43698 (api:31)
2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:31)
2023-10-16 18:24:01,123+1100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.4', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '0dde39a4-42e4-4fba-8068-e63e42dd3334': {'code': 0, 'lastCheck': '8.0', 'delay': '0.00305946', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '3e33b7ce-7279-4765-a203-cca37eff8cb0': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00109704', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'e311ddf1-7f2c-49ef-a618-050d9a2b947f': {'code': 0, 'lastCheck': '7.6', 'delay': '0.00924556', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, '4412e492-f752-4a6c-98ff-f289043618da': {'code': 0, 'lastCheck': '7.6', 'delay': '0.000824828', 'valid': True, 'version': 5, 'acquired': False, 'actual': True}, 'fa8d922a-df3c-4f5b-adf6-a77c3689c397': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version':
-1, 'acquired': False, 'actual': True}, '30775056-38ef-483a-889c-70453e9aa5f9': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}, 'd44ec7f2-3629-4110-93a5-7d8d57230efc': {'code': 358, 'lastCheck': '8.6', 'delay': '0', 'valid': False, 'version': -1, 'acquired': False, 'actual': True}} from=::ffff:192.168.2.28,43698, task_id=49598180-2b1c-493c-a4ff-f49a3e988b19 (api:37)
2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:31)
2023-10-16 18:24:01,124+1100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:192.168.2.28,43698, task_id=ed16e0f7-5985-4608-94dd-28a587cc92d2 (api:37)
2023-10-16 18:24:01,130+1100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:192.168.2.28,43698 (api:37)
2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,50162 (api:31)
2023-10-16 18:24:01,879+1100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,50162 (api:37)
2023-10-16 18:24:02,543+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9 (sdc:154)
2023-10-16 18:24:02,547+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc (sdc:154)
2023-10-16 18:24:02,552+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397 (sdc:154)
2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] START repoStats(domains=['ee5b0268-8141-43e0-995e-8b4586217123']) from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:31)
2023-10-16 18:24:02,610+1100 INFO (jsonrpc/1) [vdsm.api] FINISH repoStats return={'ee5b0268-8141-43e0-995e-8b4586217123': {'code': 0, 'lastCheck': '0.3', 'delay': '0.00137321', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::1,50158, task_id=6b15ba3d-d2fe-4568-b235-25ded159d77c (api:37)
2023-10-16 18:24:02,744+1100 INFO (monitor/3077505) [storage.storagedomaincache] Looking up domain 30775056-38ef-483a-889c-70453e9aa5f9: 0.20 seconds (utils:373)
2023-10-16 18:24:02,744+1100 ERROR (monitor/3077505) [storage.monitor] Setting up monitor for 30775056-38ef-483a-889c-70453e9aa5f9 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('30775056-38ef-483a-889c-70453e9aa5f9',)
2023-10-16 18:24:02,755+1100 INFO (monitor/d44ec7f) [storage.storagedomaincache] Looking up domain d44ec7f2-3629-4110-93a5-7d8d57230efc: 0.21 seconds (utils:373)
2023-10-16 18:24:02,755+1100 ERROR (monitor/d44ec7f) [storage.monitor] Setting up monitor for d44ec7f2-3629-4110-93a5-7d8d57230efc failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('d44ec7f2-3629-4110-93a5-7d8d57230efc',)
2023-10-16 18:24:02,755+1100 INFO (monitor/fa8d922) [storage.storagedomaincache] Looking up domain fa8d922a-df3c-4f5b-adf6-a77c3689c397: 0.20 seconds (utils:373)
2023-10-16 18:24:02,755+1100 ERROR (monitor/fa8d922) [storage.monitor] Setting up monitor for fa8d922a-df3c-4f5b-adf6-a77c3689c397 failed (monitor:346)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 343, in _setupLoop
self._setupMonitor()
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 365, in _setupMonitor
self._setupDomain()
File "/usr/lib/python3.6/site-packages/vdsm/utils.py", line 136, in wrapper
value = meth(self, *a, **kw)
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line 581, in _setupDomain
domain = sdCache.produce(self.sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 98, in produce
domain.getRealDomain()
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 34, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 122, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 139, in _findDomain
return findMethod(sdUUID)
File "/usr/lib/python3.6/site-packages/vdsm/storage/sdc.py", line 169, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
vdsm.storage.exception.StorageDomainDoesNotExist: Storage domain does not exist: ('fa8d922a-df3c-4f5b-adf6-a77c3689c397',)
2023-10-16 18:24:02,828+1100 INFO (jsonrpc/3) [vds] Setting Hosted Engine HA local maintenance to True (API:1697)
~~~
Further info can be provided if required (I don't know what else to include)
Thanks in advance for any and all help.
1
0