Maybe the libvirtd < - > libgfapi communication is somehow broken. I'm using FUSE and I have no issues, but my lab is not I/O intensive .

Have you noticed if the issue happens during I/O (for example - full backup) intensive load? You can even try to stop using libgfapi and check if it happens again. Best Regards, Strahil NikolovOn Aug 3, 2019 00:39, Darrell Budic <budic(a)onholyground.com&gt; wrote: > > I’ve been seeing similar issues lately (since upgrading to 4.3.4 and later) with gluster storage (and libgfapi), but haven’t pinned to anything particular yet. But libvirt seems to have some issues and vdsmd stops being able to poll disk usage. My VMs don’t crash or stop working, but they do report as ? in the management interface and can’t be migrated. Current work around is to restart libvirtd then vdsmd, occasionally restarting libvirtd restarts a VM (presumably the one that had broken some disk something…). > > Sorry I don’t have more info to go on, but I’ll update here if I get more on anything similar. > > -Darrell > > > On Aug 2, 2019, at 9:23 AM, kevin.doyle(a)manchester.ac.uk wrote: > > > > HI > > I think it is a bug with qemu, and isci disks. Has anyone else seen this issue VM crashes and lost communication with qemu Can I update qemu ? if so what is the latest support version to use on Ovirt 4.3.2.1-1.el7 Current versions of qemu installed > > > > Installed Packages > > qemu-img-ev.x86_64 10:2.12.0-18.el7_6.3.1 > > qemu-kvm-common-ev.x86_64 10:2.12.0-18.el7_6.3.1 > > qemu-kvm-ev.x86_64 10:2.12.0-18.el7_6.3.1 > > > > Many thanks > > Kevin > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDMHYEJXRG6... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZAOILBQRM7X... _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WEC76OXKJ54...

I caught one that I could track down a bit more, looks like it’s not directly libgfapi related, at least for me: Aug 5 07:04:35 ov8 libvirtd: 2019-08-05 07:04:35.893+0000: 4558: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:04:35 ov8 libvirtd: 2019-08-05 07:04:35.894+0000: 4560: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:09:35 ov8 libvirtd: 2019-08-05 07:09:35.888+0000: 4557: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:09:35 ov8 libvirtd: 2019-08-05 07:09:35.889+0000: 4561: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected It looks like once this happens, a few (6?) hours later libvirtd starts getting something backed up or queued, and eventually vdsmd gets too many things hung in a queue. It may be that libvirt isn’t responding at all, I can’t currently use virsh on that host, so I’m trying to figure out how to track down what VM that is and see if it’s logs will give me any more info. At this point, that host is responding, but I have no VM stats for it and can’t migrate or go to maintenance, understandable if libvirt isn’t talking I suppose. If I restart libvirt, then vdsmd, I will get control of the host back and can then migrate, etc. This has reoccured recently, so even if I do so, I expect it will happen again. Wondering if I’ve got a broken guest agent somewhere?

This is from /var/log/messages, couldn’t find anything useful in any of the /var/log/libvirt/qemu/* files, and vdsm.log has more normal operations, with some individual timeout lines, I throw in a few below. Aug 6 15:15:41 ov8 vdsm[5437]: WARN executor state: <Executor periodic workers=4 max_workers=30 <TaskQueue periodic max_tasks=400 tasks(400)=deque([<Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675c4890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394ad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643940d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576390>, <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7fc0643e2090> at 0x7fc0643e20d0> timeout=15, duration=0.00 at 0x7fc0675c40d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674ce150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c72f5d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067680110>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0643f8810>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a490>, <Task <Operation action=<function <lambda> at 0x7fc0643630c8> at 0x7fc064358090> timeout=20.0, duration=0.00 at 0x7fc0642ccf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06762ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680750>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642b1950>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067525f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642ccf10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06401afd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c722a90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b14d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642cc650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0644a6b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389b10>, <Task <Operation action=<function <lambda> at 0x7fc064363050> at 0x7fc064358050> timeout=30.0, duration=0.00 at 0x7fc0643bf490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064389ed0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064179050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179f50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643942d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674d4410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640c6390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0641790d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0677d7890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067684a90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064529250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642b1bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d2d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d8d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06402ea90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064373050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abc10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06413edd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a23d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0675246d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067576590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ad90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642cc890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640bae10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675245d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0670e8d90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a2d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06413e5d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06449f9d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642ccad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a0d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a66d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06419ee50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0674d4110>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06762ffd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bc490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517310>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643944d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2a50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702c790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06449fdd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bcb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bc610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6d50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0675171d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06401a210>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc064394190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc09c252f90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06702ce90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067757c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764afd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680f10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675767d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643f84d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06768a950>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06768ad50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fcd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676800d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402eb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c221410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115ad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0677aba50>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067115690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba9d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647a50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768aa50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676474d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc09c02a250>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067115390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fc50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f8d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fc90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643e2690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064373c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc06702c490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c252590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06702cb90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bce10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fe10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676809d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06413e4d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064394350>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644b1e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702cb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675768d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ad50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642bc450>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067576410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647a10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcc90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677d7d90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067680d90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bc310>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fa90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc06413e990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402ef90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc04c76abd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064264110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640f55d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6cd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06440ebd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640a2f90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bc290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402ee10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064080c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc04c76aad0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c76a590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676c17d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676479d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674d4e10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0644b1210>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644a6b90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064529050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8810>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ae90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc04c722350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640d0350>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06439f490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677abc10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067757e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707fdd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677aba90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06439fed0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640ab1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676471d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680b90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643bf050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643f6710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abf10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642b1c10>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640a26d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab5d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647210>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067684690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0643890d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064264e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264910>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764ab90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c7e7890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707fb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264e90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0677abe90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524c50>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c7e7e90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642642d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064142150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644a6090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640d0dd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644d7690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b10d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677abed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a69d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643730d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675766d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640c65d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067647290>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0643f8510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643bf250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6c50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640de050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064389cd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067647390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677ab990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675177d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0640ab990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775de10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064264190>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0670e80d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768aa10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768abd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064273c50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768afd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06417cbd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576550>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524990>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640d0550>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702c3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644a6750>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abc50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524690>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0675c48d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518050>, <Tas /var/log/vdsm/vdsm.log 2019-08-06 15:19:23,850+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:25,543+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<function <lambda> at 0x7fc09c09b578> at 0x7fc09c073590>, executor queue full (periodic:220) 2019-08-06 15:19:25,664+0000 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:10.203.1.5,54780 (api:48) 2019-08-06 15:19:25,665+0000 WARN (jsonrpc/3) [virt.vm] (vmId='d118c766-0c2c-47de-a834-0cd15c2c134a') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,666+0000 WARN (jsonrpc/3) [virt.vm] (vmId='bfa558f1-62c0-4c86-b09f-72ba01d5cdda') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,667+0000 WARN (jsonrpc/3) [virt.vm] (vmId='8fd8a5c4-8bbf-428d-8bd0-632541ba2baf') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,668+0000 WARN (jsonrpc/3) [virt.vm] (vmId='0f1316ea-1687-4dec-86cf-f38778be8e35') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,669+0000 WARN (jsonrpc/3) [virt.vm] (vmId='3850812c-142c-40c9-9c01-539bfbdc44c6') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,669+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e4320abc-3051-42d3-9c9b-e0d523c9103b') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,670+0000 WARN (jsonrpc/3) [virt.vm] (vmId='ff7d7589-082b-4e5d-b47d-846d59de5da0') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,671+0000 WARN (jsonrpc/3) [virt.vm] (vmId='0bec8d12-9085-47ef-adba-27326ad8632e') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,672+0000 WARN (jsonrpc/3) [virt.vm] (vmId='f472f1ad-1f23-4390-ab87-9201967fb93a') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,673+0000 WARN (jsonrpc/3) [virt.vm] (vmId='c23308e7-cba1-421d-8779-742c06e6a7a1') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,674+0000 WARN (jsonrpc/3) [virt.vm] (vmId='ff84a9af-2728-4e0e-9366-38004053e0af') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,675+0000 WARN (jsonrpc/3) [virt.vm] (vmId='f18619df-f060-43ef-960f-0840f7765045') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,676+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e6a1756c-5a69-47a6-ac35-799f54e10350') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,677+0000 WARN (jsonrpc/3) [virt.vm] (vmId='8e9d85fb-86f1-4c57-93f0-2bf39c0cae24') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,680+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e81a3935-0779-4ee3-9b64-b0fc2b3e577a') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,680+0000 WARN (jsonrpc/3) [virt.vm] (vmId='c33743fd-3228-4418-a7d0-8edce9a02cdb') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,681+0000 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:10.203.1.5,54780 (api:54) 2019-08-06 15:19:25,685+0000 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.02 seconds (__init__:312) 2019-08-06 15:19:25,856+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:27,859+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:29,683+0000 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,47764 (api:48) 2019-08-06 15:19:29,684+0000 WARN (jsonrpc/4) [virt.vm] (vmId='d118c766-0c2c-47de-a834-0cd15c2c134a') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,685+0000 WARN (jsonrpc/4) [virt.vm] (vmId='bfa558f1-62c0-4c86-b09f-72ba01d5cdda') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,686+0000 WARN (jsonrpc/4) [virt.vm] (vmId='8fd8a5c4-8bbf-428d-8bd0-632541ba2baf') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,687+0000 WARN (jsonrpc/4) [virt.vm] (vmId='0f1316ea-1687-4dec-86cf-f38778be8e35') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,688+0000 WARN (jsonrpc/4) [virt.vm] (vmId='3850812c-142c-40c9-9c01-539bfbdc44c6') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,689+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e4320abc-3051-42d3-9c9b-e0d523c9103b') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,690+0000 WARN (jsonrpc/4) [virt.vm] (vmId='ff7d7589-082b-4e5d-b47d-846d59de5da0') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,691+0000 WARN (jsonrpc/4) [virt.vm] (vmId='0bec8d12-9085-47ef-adba-27326ad8632e') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,692+0000 WARN (jsonrpc/4) [virt.vm] (vmId='f472f1ad-1f23-4390-ab87-9201967fb93a') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,693+0000 WARN (jsonrpc/4) [virt.vm] (vmId='c23308e7-cba1-421d-8779-742c06e6a7a1') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,694+0000 WARN (jsonrpc/4) [virt.vm] (vmId='ff84a9af-2728-4e0e-9366-38004053e0af') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,695+0000 WARN (jsonrpc/4) [virt.vm] (vmId='f18619df-f060-43ef-960f-0840f7765045') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,698+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e6a1756c-5a69-47a6-ac35-799f54e10350') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,699+0000 WARN (jsonrpc/4) [virt.vm] (vmId='8e9d85fb-86f1-4c57-93f0-2bf39c0cae24') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,699+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e81a3935-0779-4ee3-9b64-b0fc2b3e577a') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,701+0000 WARN (jsonrpc/4) [virt.vm] (vmId='c33743fd-3228-4418-a7d0-8edce9a02cdb') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,701+0000 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,47764 (api:54) 2019-08-06 15:19:29,706+0000 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.02 seconds (__init__:312) On Aug 6, 2019, at 6:12 AM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: Il giorno sab 3 ago 2019 alle ore 12:38 Strahil <hunter86_bg(a)yahoo.com&gt; ha scritto: > Maybe the libvirtd < - > libgfapi communication is somehow broken. > I'm using FUSE and I have no issues, but my lab is not I/O intensive . > I would recommend to use fuse instead of libgfapi on glusterfs 6 storage. Adding +Sahina Bose <sabose(a)redhat.com&gt; to the discussion > > Have you noticed if the issue happens during I/O (for example - full > backup) intensive load? > > You can even try to stop using libgfapi and check if it happens again. > > Best Regards, > Strahil NikolovOn Aug 3, 2019 00:39, Darrell Budic < > budic(a)onholyground.com&gt; wrote: > > > > I’ve been seeing similar issues lately (since upgrading to 4.3.4 and > later) with gluster storage (and libgfapi), but haven’t pinned to anything > particular yet. But libvirt seems to have some issues and vdsmd stops being > able to poll disk usage. My VMs don’t crash or stop working, but they do > report as ? in the management interface and can’t be migrated. Current work > around is to restart libvirtd then vdsmd, occasionally restarting libvirtd > restarts a VM (presumably the one that had broken some disk something…). > > > > Sorry I don’t have more info to go on, but I’ll update here if I get > more on anything similar. > > > > -Darrell > > > > > On Aug 2, 2019, at 9:23 AM, kevin.doyle(a)manchester.ac.uk wrote: > > > > > > HI > > > I think it is a bug with qemu, and isci disks. Has anyone else seen > this issue VM crashes and lost communication with qemu Can I update qemu ? > if so what is the latest support version to use on Ovirt 4.3.2.1-1.el7 > Current versions of qemu installed > > > > > > Installed Packages > > > qemu-img-ev.x86_64 10:2.12.0-18.el7_6.3.1 > > > qemu-kvm-common-ev.x86_64 10:2.12.0-18.el7_6.3.1 > > > qemu-kvm-ev.x86_64 10:2.12.0-18.el7_6.3.1 > > > > > > Many thanks > > > Kevin > > > _______________________________________________ > > > Users mailing list -- users(a)ovirt.org > > > To unsubscribe send an email to users-leave(a)ovirt.org > > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > > > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDMHYEJXRG6... > > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZAOILBQRM7X... > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/WEC76OXKJ54... > -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*