[ovirt-users] Re: Linux VM keeps crashing on copying files "with lost communication with qemu"

I caught one that I could track down a bit more, looks like it’s not directly libgfapi related, at least for me: Aug 5 07:04:35 ov8 libvirtd: 2019-08-05 07:04:35.893+0000: 4558: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:04:35 ov8 libvirtd: 2019-08-05 07:04:35.894+0000: 4560: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:09:35 ov8 libvirtd: 2019-08-05 07:09:35.888+0000: 4557: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:09:35 ov8 libvirtd: 2019-08-05 07:09:35.889+0000: 4561: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected It looks like once this happens, a few (6?) hours later libvirtd starts getting something backed up or queued, and eventually vdsmd gets too many things hung in a queue. It may be that libvirt isn’t responding at all, I can’t currently use virsh on that host, so I’m trying to figure out how to track down what VM that is and see if it’s logs will give me any more info. At this point, that host is responding, but I have no VM stats for it and can’t migrate or go to maintenance, understandable if libvirt isn’t talking I suppose. If I restart libvirt, then vdsmd, I will get control of the host back and can then migrate, etc. This has reoccured recently, so even if I do so, I expect it will happen again. Wondering if I’ve got a broken guest agent somewhere? This is from /var/log/messages, couldn’t find anything useful in any of the /var/log/libvirt/qemu/* files, and vdsm.log has more normal operations, with some individual timeout lines, I throw in a few below. Aug 6 15:15:41 ov8 vdsm[5437]: WARN executor state: <Executor periodic workers=4 max_workers=30 <TaskQueue periodic max_tasks=400 tasks(400)=deque([<Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675c4890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394ad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643940d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576390>, <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7fc0643e2090> at 0x7fc0643e20d0> timeout=15, duration=0.00 at 0x7fc0675c40d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674ce150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c72f5d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067680110>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0643f8810>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a490>, <Task <Operation action=<function <lambda> at 0x7fc0643630c8> at 0x7fc064358090> timeout=20.0, duration=0.00 at 0x7fc0642ccf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06762ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680750>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642b1950>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067525f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642ccf10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06401afd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c722a90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b14d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642cc650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0644a6b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389b10>, <Task <Operation action=<function <lambda> at 0x7fc064363050> at 0x7fc064358050> timeout=30.0, duration=0.00 at 0x7fc0643bf490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064389ed0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064179050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179f50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643942d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674d4410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640c6390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0641790d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0677d7890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067684a90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064529250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642b1bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d2d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d8d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06402ea90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064373050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abc10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06413edd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a23d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0675246d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067576590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ad90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642cc890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640bae10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675245d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0670e8d90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a2d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06413e5d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06449f9d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642ccad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a0d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a66d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06419ee50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0674d4110>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06762ffd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bc490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517310>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643944d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2a50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702c790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06449fdd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bcb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bc610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6d50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0675171d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06401a210>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc064394190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc09c252f90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06702ce90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067757c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764afd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680f10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675767d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643f84d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06768a950>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06768ad50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fcd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676800d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402eb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c221410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115ad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0677aba50>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067115690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba9d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647a50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768aa50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676474d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc09c02a250>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067115390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fc50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f8d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fc90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643e2690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064373c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc06702c490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c252590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06702cb90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bce10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fe10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676809d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06413e4d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064394350>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644b1e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702cb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675768d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ad50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642bc450>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067576410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647a10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcc90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677d7d90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067680d90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bc310>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fa90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc06413e990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402ef90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc04c76abd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064264110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640f55d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6cd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06440ebd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640a2f90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bc290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402ee10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064080c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc04c76aad0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c76a590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676c17d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676479d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674d4e10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0644b1210>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644a6b90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064529050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8810>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ae90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc04c722350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640d0350>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06439f490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677abc10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067757e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707fdd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677aba90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06439fed0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640ab1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676471d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680b90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643bf050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643f6710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abf10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642b1c10>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640a26d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab5d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647210>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067684690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0643890d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064264e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264910>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764ab90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c7e7890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707fb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264e90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0677abe90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524c50>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c7e7e90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642642d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064142150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644a6090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640d0dd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644d7690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b10d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677abed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a69d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643730d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675766d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640c65d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067647290>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0643f8510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643bf250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6c50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640de050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064389cd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067647390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677ab990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675177d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0640ab990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775de10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064264190>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0670e80d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768aa10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768abd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064273c50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768afd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06417cbd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576550>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524990>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640d0550>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702c3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644a6750>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abc50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524690>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0675c48d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518050>, <Tas /var/log/vdsm/vdsm.log 2019-08-06 15:19:23,850+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:25,543+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<function <lambda> at 0x7fc09c09b578> at 0x7fc09c073590>, executor queue full (periodic:220) 2019-08-06 15:19:25,664+0000 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:10.203.1.5,54780 (api:48) 2019-08-06 15:19:25,665+0000 WARN (jsonrpc/3) [virt.vm] (vmId='d118c766-0c2c-47de-a834-0cd15c2c134a') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,666+0000 WARN (jsonrpc/3) [virt.vm] (vmId='bfa558f1-62c0-4c86-b09f-72ba01d5cdda') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,667+0000 WARN (jsonrpc/3) [virt.vm] (vmId='8fd8a5c4-8bbf-428d-8bd0-632541ba2baf') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,668+0000 WARN (jsonrpc/3) [virt.vm] (vmId='0f1316ea-1687-4dec-86cf-f38778be8e35') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,669+0000 WARN (jsonrpc/3) [virt.vm] (vmId='3850812c-142c-40c9-9c01-539bfbdc44c6') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,669+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e4320abc-3051-42d3-9c9b-e0d523c9103b') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,670+0000 WARN (jsonrpc/3) [virt.vm] (vmId='ff7d7589-082b-4e5d-b47d-846d59de5da0') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,671+0000 WARN (jsonrpc/3) [virt.vm] (vmId='0bec8d12-9085-47ef-adba-27326ad8632e') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,672+0000 WARN (jsonrpc/3) [virt.vm] (vmId='f472f1ad-1f23-4390-ab87-9201967fb93a') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,673+0000 WARN (jsonrpc/3) [virt.vm] (vmId='c23308e7-cba1-421d-8779-742c06e6a7a1') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,674+0000 WARN (jsonrpc/3) [virt.vm] (vmId='ff84a9af-2728-4e0e-9366-38004053e0af') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,675+0000 WARN (jsonrpc/3) [virt.vm] (vmId='f18619df-f060-43ef-960f-0840f7765045') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,676+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e6a1756c-5a69-47a6-ac35-799f54e10350') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,677+0000 WARN (jsonrpc/3) [virt.vm] (vmId='8e9d85fb-86f1-4c57-93f0-2bf39c0cae24') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,680+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e81a3935-0779-4ee3-9b64-b0fc2b3e577a') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,680+0000 WARN (jsonrpc/3) [virt.vm] (vmId='c33743fd-3228-4418-a7d0-8edce9a02cdb') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,681+0000 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:10.203.1.5,54780 (api:54) 2019-08-06 15:19:25,685+0000 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.02 seconds (__init__:312) 2019-08-06 15:19:25,856+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:27,859+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:29,683+0000 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,47764 (api:48) 2019-08-06 15:19:29,684+0000 WARN (jsonrpc/4) [virt.vm] (vmId='d118c766-0c2c-47de-a834-0cd15c2c134a') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,685+0000 WARN (jsonrpc/4) [virt.vm] (vmId='bfa558f1-62c0-4c86-b09f-72ba01d5cdda') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,686+0000 WARN (jsonrpc/4) [virt.vm] (vmId='8fd8a5c4-8bbf-428d-8bd0-632541ba2baf') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,687+0000 WARN (jsonrpc/4) [virt.vm] (vmId='0f1316ea-1687-4dec-86cf-f38778be8e35') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,688+0000 WARN (jsonrpc/4) [virt.vm] (vmId='3850812c-142c-40c9-9c01-539bfbdc44c6') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,689+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e4320abc-3051-42d3-9c9b-e0d523c9103b') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,690+0000 WARN (jsonrpc/4) [virt.vm] (vmId='ff7d7589-082b-4e5d-b47d-846d59de5da0') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,691+0000 WARN (jsonrpc/4) [virt.vm] (vmId='0bec8d12-9085-47ef-adba-27326ad8632e') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,692+0000 WARN (jsonrpc/4) [virt.vm] (vmId='f472f1ad-1f23-4390-ab87-9201967fb93a') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,693+0000 WARN (jsonrpc/4) [virt.vm] (vmId='c23308e7-cba1-421d-8779-742c06e6a7a1') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,694+0000 WARN (jsonrpc/4) [virt.vm] (vmId='ff84a9af-2728-4e0e-9366-38004053e0af') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,695+0000 WARN (jsonrpc/4) [virt.vm] (vmId='f18619df-f060-43ef-960f-0840f7765045') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,698+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e6a1756c-5a69-47a6-ac35-799f54e10350') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,699+0000 WARN (jsonrpc/4) [virt.vm] (vmId='8e9d85fb-86f1-4c57-93f0-2bf39c0cae24') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,699+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e81a3935-0779-4ee3-9b64-b0fc2b3e577a') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,701+0000 WARN (jsonrpc/4) [virt.vm] (vmId='c33743fd-3228-4418-a7d0-8edce9a02cdb') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,701+0000 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,47764 (api:54) 2019-08-06 15:19:29,706+0000 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.02 seconds (__init__:312)