March 2024 - Users - oVirt List Archives

Hosted engine installation oVirt 4.5 initialize lockspace volume
by lbayerlein＠lbayerlein.de 03 Apr '24

03 Apr '24

Hello, hope, you are doing well. I try to install a new hosted engine on my environment to switch the storage this engine. I use following command to deploy the instance: hosted-engine --deploy --4 --ansible-extra-vars=he_offline_deployment=true Everything works fine, except the initialization of lockspace volume. Here I get: ``` [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set the name for add_host] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Fetch the value of HOST_KEY_CHECKING] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Get the username running the deploy] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Register the engine FQDN as a host] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Start ovirt-ha-broker service on the host] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Initialize lockspace volume] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Workaround for ovirt-ha-broker start failures] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Initialize lockspace volume] [ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 5, "changed": true, "cmd": ["hosted-engine", "--reinitialize-lock space", "--force"], "delta": "0:00:00.269920", "end": "2024-03-20 14:43:00.366557", "msg": "non-zero return code", "rc" : 1, "start": "2024-03-20 14:43:00.096637", "stderr": "Traceback (most recent call last):\n File \"/usr/lib64/python3. 6/runpy.py\", line 193, in _run_module_as_main\n \"__main__\", mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup /reinitialize_lockspace.py\", line 30, in <module>\n ha_cli.reset_lockspace(force)\n File \"/usr/lib/python3.6/site -packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace\n stats = broker.get_stats_from_st orage()\n File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", line 148, in get_stats_f rom_storage\n result = self._proxy.get_stats()\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1112, in __ca ll__\n return self.__send(self.__name, args)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1452, in __requ est\n verbose=self.__verbose\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1154, in request\n return se lf.single_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1166, i n single_request\n http_conn = self.send_request(host, handler, request_body, verbose)\n File \"/usr/lib64/python3. 6/xmlrpc/client.py\", line 1279, in send_request\n self.send_content(connection, request_body)\n File \"/usr/lib64/ python3.6/xmlrpc/client.py\", line 1309, in send_content\n connection.endheaders(request_body)\n File \"/usr/lib64/ python3.6/http/client.py\", line 1268, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked )\n File \"/usr/lib64/python3.6/http/client.py\", line 1044, in _send_output\n self.send(msg)\n File \"/usr/lib64/ python3.6/http/client.py\", line 982, in send\n self.connect()\n File \"/usr/lib/python3.6/site-packages/ovirt_host ed_engine_ha/lib/unixrpc.py\", line 76, in connect\n self.sock.connect(base64.b16decode(self.host))\nFileNotFoundErr or: [Errno 2] No such file or directory", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/lib64/p ython3.6/runpy.py\", line 193, in _run_module_as_main", " \"__main__\", mod_spec)", " File \"/usr/lib64/python3.6/r unpy.py\", line 85, in _run_code", " exec(code, run_globals)", " File \"/usr/lib/python3.6/site-packages/ovirt_host ed_engine_setup/reinitialize_lockspace.py\", line 30, in <module>", " ha_cli.reset_lockspace(force)", " File \"/usr /lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py\", line 286, in reset_lockspace", " stats = bro ker.get_stats_from_storage()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py\", l ine 148, in get_stats_from_storage", " result = self._proxy.get_stats()", " File \"/usr/lib64/python3.6/xmlrpc/clie nt.py\", line 1112, in __call__", " return self.__send(self.__name, args)", " File \"/usr/lib64/python3.6/xmlrpc/cl ient.py\", line 1452, in __request", " verbose=self.__verbose", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", l ine 1154, in request", " return self.single_request(host, handler, request_body, verbose)", " File \"/usr/lib64/pyt hon3.6/xmlrpc/client.py\", line 1166, in single_request", " http_conn = self.send_request(host, handler, request_bod y, verbose)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1279, in send_request", " self.send_content(co nnection, request_body)", " File \"/usr/lib64/python3.6/xmlrpc/client.py\", line 1309, in send_content", " connecti on.endheaders(request_body)", " File \"/usr/lib64/python3.6/http/client.py\", line 1268, in endheaders", " self._se nd_output(message_body, encode_chunked=encode_chunked)", " File \"/usr/lib64/python3.6/http/client.py\", line 1044, in _send_output", " self.send(msg)", " File \"/usr/lib64/python3.6/http/client.py\", line 982, in send", " self.co nnect()", " File \"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py\", line 76, in connect", " self.sock.connect(base64.b16decode(self.host))", "FileNotFoundError: [Errno 2] No such file or directory"], "stdout": "", "stdout_lines": []} ``` And I dont know, what is the reason. I zeroed the iscsi target, rebooted the hypervisor and updated all software packages. ``` [root@hypervisor01 ~]# rpm -qa|grep ovirt-host ovirt-hosted-engine-setup-2.6.5-1.el8.noarch ovirt-host-dependencies-4.5.0-3.el8.x86_64 ovirt-hosted-engine-ha-2.5.0-1.el8.noarch ovirt-host-4.5.0-3.el8.x86_64 ``` I tried the deployment without the ansible arguments. Unfortunately, the same error comes up. What can I do to solve this? Thanks very much balu

2 1

Recreate engine without backup
by Igor Filipovic 03 Apr '24

03 Apr '24

Hi all, I have a situation in which customer has deleted standalone engine completely (it was running on vmware), and all backups were saved on the same machine - so there is no backup available. Can someone suggest how to preserve virtual machines which are still running on kvm hosts, and recreate standalone engine to be able to manage existing kvm hosts and machines? What could be scenario for this? Thanks in advance and best regards! Igor

2 1

Re: Issue with Accessing oVirt Manager Console
by Gianluca Cecchi 30 Mar '24

30 Mar '24

On Fri, Mar 29, 2024 at 11:59 AM Sachendra Shukla < sachendra.shukla(a)yagnaiq.com> wrote: > Hi Gianluca, > > I hope this email finds you well. I wanted to bring to your attention an > issue regarding the status of ovirt-engine and httpd. While httpd is > showing as active, unfortunately, the ovirt-engine status is indicating a > failure. I have attached the engine logs for your reference to provide more > insight into this matter. I do not want any impact on running VMs. Please > check and provide me the resolution if possible. > It could be related to the ovirt-ovn-provider and/or its certificates, as we get thousands of errors related to it inside engine.log $ grep ERROR /tmp/engine.log | awk '{print $4}' | sort | uniq -c 2532 [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] the last being: 2024-03-28 22:15:33,484+05 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-53) [260c120a] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error NotAfter: Fri Dec 30 19:27:18 IST 2022 and code 5050) Which version of oVirt are you using? You could try to run this and see if it recreates certificates if this is the problem (or if it asks to regenerate them, say yes). You should not get downtime for VMs. The "--offline" option makes the command only refresh its configuration without going through actual updates or modifications engine-setup --offline It restarts also the needed services (ovirt-provider-ovn too) in the correct order, that could help sometimes HIH, Gianluca

2 1

Re: Issue with Accessing oVirt Manager Console
by Alex K 29 Mar '24

29 Mar '24

On Fri, Mar 29, 2024 at 12:59 PM Sachendra Shukla < sachendra.shukla(a)yagnaiq.com> wrote: > Hi Gianluca, > > I hope this email finds you well. I wanted to bring to your attention an > issue regarding the status of ovirt-engine and httpd. While httpd is > showing as active, unfortunately, the ovirt-engine status is indicating a > failure. I have attached the engine logs for your reference to provide more > insight into this matter. I do not want any impact on running VMs. Please > check and provide me the resolution if possible. > > > [image: image.png] > > Did you try to restart it? systemctl restart ovirt-engine > On Fri, Mar 29, 2024 at 3:43 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com> > wrote: > >> On the engine host >> . check contents of /var/log/ovirt-engine/engine.log >> >> . check the output of the commands: >> systemctl status ovirt-engine >> systemctl status httpd >> >> . check contents of JBoss instance related logs >> /var/log/ovirt-engine/server.log >> In 4.5.x you should see something like this (see the deployed engine.ear >> and ovirt-web-ui.war lines for example): >> >> 2024-03-26 17:26:29,515+01 INFO [org.wildfly.extension.undertow] >> (ServerService Thread Pool -- 47) WFLYUT0021: Registered web context: >> '/ovirt-engine/docs' for server 'default-server' >> 2024-03-26 17:26:29,530+01 INFO [org.wildfly.extension.undertow] >> (ServerService Thread Pool -- 46) WFLYUT0021: Registered web context: '/' >> for server 'default-server' >> 2024-03-26 17:26:29,575+01 INFO [org.wildfly.extension.undertow] >> (ServerService Thread Pool -- 46) WFLYUT0021: Registered web context: >> '/ovirt-engine/services' for server 'default-server' >> 2024-03-26 17:26:29,588+01 WARN >> [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] >> (ServerService Thread Pool -- 46) IJ000407: No lazy enlistment available >> for ENGINEDataSource >> 2024-03-26 17:26:29,633+01 INFO [org.wildfly.extension.undertow] >> (ServerService Thread Pool -- 51) WFLYUT0021: Registered web context: >> '/ovirt-engine' for server 'default-server' >> 2024-03-26 17:26:29,648+01 WARN >> [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] >> (ServerService Thread Pool -- 46) IJ000407: No lazy enlistment available >> for DWHDataSource >> 2024-03-26 17:26:29,666+01 INFO [org.wildfly.extension.undertow] >> (ServerService Thread Pool -- 46) WFLYUT0021: Registered web context: >> '/ovirt-engine/webadmin' for server 'default-server' >> 2024-03-26 17:26:29,843+01 INFO [org.wildfly.extension.undertow] >> (ServerService Thread Pool -- 49) WFLYUT0021: Registered web context: >> '/ovirt-engine/sso' for server 'default-server' >> 2024-03-26 17:26:30,097+01 INFO [org.jboss.as.clustering.infinispan] >> (ServerService Thread Pool -- 50) WFLYCLINF0002: Started timeout-base cache >> from ovirt-engine container >> 2024-03-26 17:26:31,880+01 INFO [org.jboss.as.server] (ServerService >> Thread Pool -- 27) WFLYSRV0010: Deployed "restapi.war" (runtime-name : >> "restapi.war") >> 2024-03-26 17:26:31,880+01 INFO [org.jboss.as.server] (ServerService >> Thread Pool -- 27) WFLYSRV0010: Deployed "ovirt-web-ui.war" (runtime-name : >> "ovirt-web-ui.war") >> 2024-03-26 17:26:31,880+01 INFO [org.jboss.as.server] (ServerService >> Thread Pool -- 27) WFLYSRV0010: Deployed "apidoc.war" (runtime-name : >> "apidoc.war") >> 2024-03-26 17:26:31,880+01 INFO [org.jboss.as.server] (ServerService >> Thread Pool -- 27) WFLYSRV0010: Deployed "engine.ear" (runtime-name : >> "engine.ear") >> 2024-03-26 17:26:31,930+01 INFO [org.jboss.as.server] (Controller Boot >> Thread) WFLYSRV0212: Resuming server >> 2024-03-26 17:26:31,933+01 INFO [org.jboss.as] (Controller Boot Thread) >> WFLYSRV0025: WildFly Full 24.0.1.Final (WildFly Core 16.0.1.Final) started >> in 16651ms - Started 1676 of 1890 services (393 services are lazy, passive >> or on-demand) >> 2024-03-26 17:26:31,940+01 INFO [org.jboss.as] (Controller Boot Thread) >> WFLYSRV0060: Http management interface listening on >> http://127.0.0.1:8706/management >> 2024-03-26 17:26:31,940+01 INFO [org.jboss.as] (Controller Boot Thread) >> WFLYSRV0051: Admin console listening on http://127.0.0.1:8706 >> 2024-03-26 17:41:31,872+01 INFO >> [org.apache.sshd.common.util.security.eddsa.EdDSASecurityProviderRegistrar] >> (EE-ManagedExecutorService-hostUpdatesChecker-Thread-1) >> getOrCreateProvider(EdDSA) created instance of >> net.i2p.crypto.eddsa.EdDSASecurityProvider >> >> eventually if in error try to restart ovirt-engine service >> systemctl restart ovirt-engine >> >> HIH, >> Gianluca >> >> >> >> On Fri, Mar 29, 2024 at 9:59 AM Sachendra Shukla < >> sachendra.shukla(a)yagnaiq.com> wrote: >> >>> Hi Alex, >>> >>> After conducting a thorough check, we have confirmed that all virtual >>> machines are operating smoothly, and the network is functioning without any >>> issues (we can successfully SSH into both the VMs and the manager). >>> Additionally, there is available storage on both the node and the manager. >>> However, we are encountering difficulty accessing the oVirt manager >>> console, as we are consistently receiving a "503 Service Unavailable" error. >>> >>> Thanks, >>> Sachendra >>> >>> > > -- > > Regards, > > > > > > > > > *Sachendra Shukla* > > IT Administrator > > Yagna iQ, Inc. and subsidiaries > > Email: Sachendra.shukla(a)yagnaiq.com <dnyanesh.tisge(a)yagnaiq.com> > > Website: https://yagnaiq.com > > Privacy Policy: https://www.yagnaiq.com/privacy-policy/ > > <https://www.linkedin.com/company/yagnaiq/mycompany/> > <https://www.youtube.com/channel/UCeHXOpcUxWvOJO0aegD99Jg> > > > This communication and any attachments may contain confidential > information and/or Yagna iQ, Inc. copyright material. > > All unauthorized use, disclosure, or distribution is prohibited. If you > are not the intended recipient, please notify Yagna iQ immediately by > replying to the email and destroy all copies of this communication. > > This email has been scanned for all known viruses. The sender does not > accept liability for any damage inflicted by viewing the content of this > email. > >

1 0

Re: Issue with Accessing oVirt Manager Console
by Sachendra Shukla 29 Mar '24

29 Mar '24

Hi Alex, After conducting a thorough check, we have confirmed that all virtual machines are operating smoothly, and the network is functioning without any issues (we can successfully SSH into both the VMs and the manager). Additionally, there is available storage on both the node and the manager. However, we are encountering difficulty accessing the oVirt manager console, as we are consistently receiving a "503 Service Unavailable" error. Thanks, Sachendra On Fri, Mar 29, 2024 at 2:11 PM Alex K <rightkicktech(a)gmail.com> wrote: > > > On Fri, Mar 29, 2024 at 8:26 AM Sachendra Shukla < > sachendra.shukla(a)yagnaiq.com> wrote: > >> Hi Team, >> >> We are currently experiencing difficulties accessing the oVirt Manager >> console. Please find attached a screenshot of the error for your reference. >> >> [image: image.png] >> > You need to check what is causing it. There could be many causes. I would > start to check the engine VM status, storage, network, logs, etc. > > >> Please help me if you have any resolution. >> -- >> >> Regards, >> >> >> >> >> >> >> >> >> *Sachendra Shukla* >> >> IT Administrator >> >> Yagna iQ, Inc. and subsidiaries >> >> Email: Sachendra.shukla(a)yagnaiq.com <dnyanesh.tisge(a)yagnaiq.com> >> >> Website: https://yagnaiq.com >> >> Privacy Policy: https://www.yagnaiq.com/privacy-policy/ >> >> <https://www.linkedin.com/company/yagnaiq/mycompany/> >> <https://www.youtube.com/channel/UCeHXOpcUxWvOJO0aegD99Jg> >> >> >> This communication and any attachments may contain confidential >> information and/or Yagna iQ, Inc. copyright material. >> >> All unauthorized use, disclosure, or distribution is prohibited. If you >> are not the intended recipient, please notify Yagna iQ immediately by >> replying to the email and destroy all copies of this communication. >> >> This email has been scanned for all known viruses. The sender does not >> accept liability for any damage inflicted by viewing the content of this >> email. >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/2W3MRMOOSPB5X… >> > -- Regards, *Sachendra Shukla* IT Administrator Yagna iQ, Inc. and subsidiaries Email: Sachendra.shukla(a)yagnaiq.com <dnyanesh.tisge(a)yagnaiq.com> Website: https://yagnaiq.com Privacy Policy: https://www.yagnaiq.com/privacy-policy/ <https://www.linkedin.com/company/yagnaiq/mycompany/> <https://www.youtube.com/channel/UCeHXOpcUxWvOJO0aegD99Jg> This communication and any attachments may contain confidential information and/or Yagna iQ, Inc. copyright material. All unauthorized use, disclosure, or distribution is prohibited. If you are not the intended recipient, please notify Yagna iQ immediately by replying to the email and destroy all copies of this communication. This email has been scanned for all known viruses. The sender does not accept liability for any damage inflicted by viewing the content of this email.

2 1

Issue with Accessing oVirt Manager Console
by Sachendra Shukla 29 Mar '24

29 Mar '24

Hi Team, We are currently experiencing difficulties accessing the oVirt Manager console. Please find attached a screenshot of the error for your reference. [image: image.png] Please help me if you have any resolution. -- Regards, *Sachendra Shukla* IT Administrator Yagna iQ, Inc. and subsidiaries Email: Sachendra.shukla(a)yagnaiq.com <dnyanesh.tisge(a)yagnaiq.com> Website: https://yagnaiq.com Privacy Policy: https://www.yagnaiq.com/privacy-policy/ <https://www.linkedin.com/company/yagnaiq/mycompany/> <https://www.youtube.com/channel/UCeHXOpcUxWvOJO0aegD99Jg> This communication and any attachments may contain confidential information and/or Yagna iQ, Inc. copyright material. All unauthorized use, disclosure, or distribution is prohibited. If you are not the intended recipient, please notify Yagna iQ immediately by replying to the email and destroy all copies of this communication. This email has been scanned for all known viruses. The sender does not accept liability for any damage inflicted by viewing the content of this email.

1 0

How to Add Centos9 KVM Host to oVirt engine?
by it＠tlivesupport.com 27 Mar '24

27 Mar '24

Hello Team I have installed KVM and oVirt in Same VM (Centos9) i am trying add Host with IP. After adding i am getting message like Status Non-operational. When i create VM I am getting message like " Not available when no Data Center is up." Please advise what is missing? Please provide me steps to add KVM into oVirt Enginee

2 1

Network interface configuration
by Jorge Visentini 27 Mar '24

27 Mar '24

Hi! There is any configuration on interface, kernel or sysctl.conf to tuning of network performance? We have Connect-X 40Gbps and 100Gbps but we guess we can have more performance, you know? Our VMs have a +- 1 or 2gigas (1500 MTU) of throughput on iperf test. ovirtmgmt has around 27 ~ 30Gbps (9000 MTU) of throughput on iperf test. We already use 9000 MTU in all network devices and ovirt's bonds. Tests were performed without concorrence workload. Cheers!

1 0

Understanding Preallocation (Actual) Size of a thin DIsk
by Shafi Mohammed 27 Mar '24

27 Mar '24

HI , I need to know about the Pre allocation size of a thin disk while creating a disk in a storage domain . Can we change the actual size (allocated Size ) after disk creation .Can anyone can enlighten me on how it will increase when the size of the disk exceeds the allocated limit ? If I try to write to the disk from the hypervisor , the write gets failed after the allocated size whereas if i create a thick disk i am able to write till the disk capacity Regards Shafi

2 1

VM's migrations cause timeouts on various unrelated hosts which results in host restart
by 73616c616e＠gmail.com 25 Mar '24

25 Mar '24

During the VM's migrations from one host to another, various unrelated hosts have timeouts and some of them are being restarted by Engine due to their "unresponsive" status. However they're working fine at that time... It only happens when migration process is being run on other unrelated hosts. Of course, one solution might be increasing the timeout, but it's only a workaround and in my opinion, not a proper solution. This whole problem occurs after few minutes of when the migrations were started. oVirt-engine version: 4.4.10.7 (at the moment we do not have the available window for an update) vdsm version on hosts: 4.40.100.2 ** Below logs are example of one of many hosts as all of them had similar messages ** Engine logs when first timeout appeared until it's restarted: 09:31:07,856+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] No interaction with host 'os-host06' for 20000 ms. 09:31:29,242+01 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,243+01 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Command 'GetAllVmStatsVDSCommand(HostName = os-host06, VdsIdVDSCommandParametersBase:{hostId='aa11wa4d-6e20-44hf-a76e-b0878bba1f42'})' execution failed: VDSGenericException: VDSNetworkException: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,243+01 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-25) [] Failed to fetch vms info for host 'os-host06'(aa11wa4d-6e20-44hf-a76e-b0878bba1f42) - skipping VMs monitoring. 09:31:29,246+01 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-3) [] Clearing domains data for host os-host06 09:31:29,251+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-95) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM os-host06 command Get Host Statistics failed: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,251+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-95) [] Unable to GetStats: VDSNetworkException: VDSGenericException: VDSNetworkException: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,287+01 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-3) [] Server failed to respond, vds_id='aa11wa4d-6e20-44hf-a76e-b0878bba1f42', vds_name='os-host06', vm_count=8, spm_status='None', non-responsive_timeout (seconds)=64, error: VDSGenericException: VDSNetworkException: Connection timeout for host 'os-host06', last response arrived 41571 ms ago. 09:31:29,289+01 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-3) [] EVENT_ID: VDS_FAILURE(12), Host os-host06 is non responsive. 09:31:29,551+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-38) [2ae04e58] EVENT_ID: FENCE_OPERATION_USING_AGENT_AND_PROXY_STARTED(9,020), Executing power management status on Host os-host06 using Proxy Host os-host03 and Fence Agent ipmilan:os-host06-iDrac. 09:31:29,659+01 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (EE-ManagedThreadFactory-engine-Thread-38) [2ae04e58] Opening SSH Soft Fencing session on host 'os-host06' 09:31:29,843+01 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (EE-ManagedThreadFactory-engine-Thread-38) [2ae04e58] Executing SSH Soft Fencing command on host 'os-host06' 09:31:32,239+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to os-host06/10.10.5.17 09:31:44,358+01 INFO [org.ovirt.engine.core.bll.pm.StopVdsCommand] (EE-ManagedThreadFactory-engine-Thread-38) [2e93a767] Power-Management: STOP of host 'os-host06' initiated. Logs from host at the same time just before the restart: 09:30:36,926+0100 INFO (periodic/3) [vdsm.api] FINISH getVolumeSize return={'apparentsize': '1207959552', 'truesize': '1207959552'} from=internal, task_id=f3f073de-2c08-47b2-aae2-017a7480db61 (api:54) 09:30:39,117+0100 INFO (jsonrpc/3) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:30:39,119+0100 INFO (jsonrpc/3) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:30:39,123+0100 INFO (jsonrpc/2) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:30:39,125+0100 INFO (jsonrpc/2) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:30:39,130+0100 INFO (jsonrpc/1) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:30:39,131+0100 INFO (jsonrpc/1) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:30:39,136+0100 INFO (jsonrpc/4) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:30:39,138+0100 INFO (jsonrpc/4) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:30:39,143+0100 INFO (jsonrpc/3) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:30:39,144+0100 INFO (jsonrpc/3) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:30:39,149+0100 INFO (jsonrpc/2) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:30:39,151+0100 INFO (jsonrpc/2) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:30:39,156+0100 INFO (jsonrpc/1) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:30:39,158+0100 INFO (jsonrpc/1) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:30:39,162+0100 INFO (jsonrpc/4) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:30:39,164+0100 INFO (jsonrpc/4) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:30:39,858+0100 INFO (jsonrpc/0) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:30:39,863+0100 INFO (jsonrpc/0) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:30:39,873+0100 INFO (jsonrpc/3) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:30:39,874+0100 INFO (jsonrpc/3) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},... 09:30:41,228+0100 INFO (jsonrpc/6) [api.host] START getStats() from=::ffff:10.10.3.150,59772 (api:48) 09:30:41,249+0100 INFO (jsonrpc/6) [vdsm.api] START repoStats(domains=()) from=::ffff:10.10.3.150,59772, task_id=978cd75b-fb99-47ee-af62-0bc0338fc66a (api:48) 09:30:41,250+0100 INFO (jsonrpc/6) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000389989', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000117308', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000123364', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '0.7', 'delay': '0.000135331', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '2.1', 'delay': '0.000147366', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::ffff:10.10.3.150,59772, task_id=978cd75b-fb99-47ee-af62-0bc0338fc66a (api:54) 09:30:41,250+0100 INFO (jsonrpc/6) [vdsm.api] START multipath_health() from=::ffff:10.10.3.150,59772, task_id=691b59a6-4a16-4e8d-9676-f1d46a3981b9 (api:48) 09:30:41,250+0100 INFO (jsonrpc/6) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.10.3.150,59772, task_id=691b59a6-4a16-4e8d-9676-f1d46a3981b9 (api:54) 09:30:41,254+0100 INFO (jsonrpc/6) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:30:48,833+0100 INFO (periodic/0) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5bdca038-eb0c-4e50-aeca-be2349387a36 (api:48) 09:30:48,833+0100 INFO (periodic/0) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000389989', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000117308', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000123364', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000135331', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '9.7', 'delay': '0.000147366', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5bdca038-eb0c-4e50-aeca-be2349387a36 (api:54) 09:30:54,237+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:30:54,239+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:30:54,243+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:30:54,245+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:30:54,250+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:30:54,252+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:30:54,256+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:30:54,258+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:30:54,263+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:30:54,264+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:30:54,269+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:30:54,271+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:30:54,276+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:30:54,278+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:30:54,282+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:30:54,284+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:30:54,888+0100 INFO (jsonrpc/2) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:30:54,892+0100 INFO (jsonrpc/2) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:30:54,896+0100 INFO (jsonrpc/7) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:30:54,896+0100 INFO (jsonrpc/7) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},.... 09:30:55,220+0100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:10.10.3.150,59772 (api:48) 09:30:55,223+0100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {'26157351-db69-4e59-a460-1de23cf47517': 'Up', 'f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6': 'Up', 'f335c1e8-4699-4e81-be22-dcb92a6a3ca6': 'Up', '50221e76-d8ee-414b-a3b7-f8a860b21831': 'Up', '25aee386-e8ba-4ac0-85c5-cde2e0d22348': 'Up', '1348d3e0-c02c-4c9a-b669-601e6dcc9b80': 'Up', 'bdeb4e6f-9546-438a-95c8-70c426c3490c': 'Up', '01ff01e5-5360-4549-b538-38529537a587': 'Up'} (throttledlog:104) 09:30:55,223+0100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:02,775+0100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:10.10.3.150,59772 (api:48) 09:31:02,795+0100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '2.2', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '2.2', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '3.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:54) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:54) 09:31:02,801+0100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:48) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '3.3', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '3.4', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '4.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:54) 09:31:09,351+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:31:09,353+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:31:09,358+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:31:09,360+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:31:09,365+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:31:09,366+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:31:09,371+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:31:09,373+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:31:09,378+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:31:09,379+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:31:09,384+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:31:09,386+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:31:09,391+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:31:09,392+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:31:09,397+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:31:09,398+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:31:09,911+0100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:31:09,915+0100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},... 09:30:55,220+0100 INFO (jsonrpc/1) [api.host] START getAllVmStats() from=::ffff:10.10.3.150,59772 (api:48) 09:30:55,223+0100 INFO (jsonrpc/1) [throttled] Current getAllVmStats: {'26157351-db69-4e59-a460-1de23cf47517': 'Up', 'f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6': 'Up', 'f335c1e8-4699-4e81-be22-dcb92a6a3ca6': 'Up', '50221e76-d8ee-414b-a3b7-f8a860b21831': 'Up', '25aee386-e8ba-4ac0-85c5-cde2e0d22348': 'Up', '1348d3e0-c02c-4c9a-b669-601e6dcc9b80': 'Up', 'bdeb4e6f-9546-438a-95c8-70c426c3490c': 'Up', '01ff01e5-5360-4549-b538-38529537a587': 'Up'} (throttledlog:104) 09:30:55,223+0100 INFO (jsonrpc/1) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:02,775+0100 INFO (jsonrpc/5) [api.host] START getStats() from=::ffff:10.10.3.150,59772 (api:48) 09:31:02,795+0100 INFO (jsonrpc/5) [vdsm.api] START repoStats(domains=()) from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '2.2', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '2.2', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '2.3', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '3.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=::ffff:10.10.3.150,59772, task_id=fb00b06c-bd84-450f-91eb-e0e21ce44db9 (api:54) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] START multipath_health() from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:48) 09:31:02,796+0100 INFO (jsonrpc/5) [vdsm.api] FINISH multipath_health return={} from=::ffff:10.10.3.150,59772, task_id=0edc02f9-41e9-430a-97f9-135ba36e8941 (api:54) 09:31:02,801+0100 INFO (jsonrpc/5) [api.host] FINISH getStats return={'status': {'code': 0, 'message': 'Done'}, 'info': (suppressed)} from=::ffff:10.10.3.150,59772 (api:54) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] START repoStats(domains=()) from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:48) 09:31:03,859+0100 INFO (periodic/3) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '3.3', 'delay': '9.6718e-05', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106966', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '3.3', 'delay': '0.000106646', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '3.4', 'delay': '0.000113708', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '4.6', 'delay': '0.000144113', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=5f40ee63-140d-416d-bc5f-dd3b696c1d10 (api:54) 09:31:09,351+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:31:09,353+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:31:09,358+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:31:09,360+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:31:09,365+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:31:09,366+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:31:09,371+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:31:09,373+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:31:09,378+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:31:09,379+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:31:09,384+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:31:09,386+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:31:09,391+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:31:09,392+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:31:09,397+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:31:09,398+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:31:09,911+0100 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:31:09,915+0100 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:31:09,919+0100 INFO (jsonrpc/0) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},... 09:31:18,887+0100 INFO (periodic/1) [vdsm.api] START repoStats(domains=()) from=internal, task_id=cd897cf9-6f3e-4514-92cf-9a9939f7ce52 (api:48) 09:31:18,887+0100 INFO (periodic/1) [vdsm.api] FINISH repoStats return={'d178ad4b-37f9-40a3-81b3-826e30900d42': {'code': 0, 'lastCheck': '8.3', 'delay': '0.000106409', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '8f2febd6-b864-4753-9d5a-4f050a0506a7': {'code': 0, 'lastCheck': '8.4', 'delay': '0.000115832', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, 'e54b09c8-e9e3-4c31-93ce-ca84c65045b1': {'code': 0, 'lastCheck': '8.4', 'delay': '0.00011779', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '578d6e98-a849-4c86-a094-459152af0997': {'code': 0, 'lastCheck': '8.4', 'delay': '0.000125551', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}, '23a31eca-4a47-4f4d-a0df-2edf8fe89f85': {'code': 0, 'lastCheck': '9.7', 'delay': '0.000130676', 'valid': True, 'version': 5, 'acquired': True, 'actual': True}} from=internal, task_id=cd897cf9-6f3e-4514-92cf-9a9939f7ce52 (api:54) 09:31:24,466+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:48) 09:31:24,468+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=26157351-db69-4e59-a460-1de23cf47517 (api:54) 09:31:24,473+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:48) 09:31:24,474+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f68f945c-8441-4ba7-ad7c-3ed3cc4efbe6 (api:54) 09:31:24,479+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:48) 09:31:24,481+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=f335c1e8-4699-4e81-be22-dcb92a6a3ca6 (api:54) 09:31:24,486+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:48) 09:31:24,487+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=25aee386-e8ba-4ac0-85c5-cde2e0d22348 (api:54) 09:31:24,492+0100 INFO (jsonrpc/6) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:48) 09:31:24,494+0100 INFO (jsonrpc/6) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=1348d3e0-c02c-4c9a-b669-601e6dcc9b80 (api:54) 09:31:24,499+0100 INFO (jsonrpc/7) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:48) 09:31:24,501+0100 INFO (jsonrpc/7) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=bdeb4e6f-9546-438a-95c8-70c426c3490c (api:54) 09:31:24,505+0100 INFO (jsonrpc/5) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:48) 09:31:24,507+0100 INFO (jsonrpc/5) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=50221e76-d8ee-414b-a3b7-f8a860b21831 (api:54) 09:31:24,512+0100 INFO (jsonrpc/0) [api.virt] START setCpuTunePeriod(period=100000) from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:48) 09:31:24,513+0100 INFO (jsonrpc/0) [api.virt] FINISH setCpuTunePeriod return={'status': {'code': 0, 'message': 'Done'}} from=::1,36652, vmId=01ff01e5-5360-4549-b538-38529537a587 (api:54) 09:31:24,925+0100 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::1,36652 (api:48) 09:31:24,929+0100 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'code': 0, 'message': 'Done'}, 'statsList': (suppressed)} from=::1,36652 (api:54) 09:31:24,933+0100 INFO (jsonrpc/6) [api.host] START getAllVmIoTunePolicies() from=::1,36652 (api:48) 09:31:24,933+0100 INFO (jsonrpc/6) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'code': 0, 'message': 'Done'},...

1 0