Il giorno mar 6 ago 2019 alle ore 17:25 Darrell Budic < budic@onholyground.com> ha scritto:

I caught one that I could track down a bit more, looks like it’s not directly libgfapi related, at least for me:

Aug 5 07:04:35 ov8 libvirtd: 2019-08-05 07:04:35.893+0000: 4558: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:04:35 ov8 libvirtd: 2019-08-05 07:04:35.894+0000: 4560: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:09:35 ov8 libvirtd: 2019-08-05 07:09:35.888+0000: 4557: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected Aug 5 07:09:35 ov8 libvirtd: 2019-08-05 07:09:35.889+0000: 4561: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected

It looks like once this happens, a few (6?) hours later libvirtd starts getting something backed up or queued, and eventually vdsmd gets too many things hung in a queue. It may be that libvirt isn’t responding at all, I can’t currently use virsh on that host, so I’m trying to figure out how to track down what VM that is and see if it’s logs will give me any more info.

At this point, that host is responding, but I have no VM stats for it and can’t migrate or go to maintenance, understandable if libvirt isn’t talking I suppose. If I restart libvirt, then vdsmd, I will get control of the host back and can then migrate, etc. This has reoccured recently, so even if I do so, I expect it will happen again. Wondering if I’ve got a broken guest agent somewhere?

+Ryan Barry <rbarry@redhat.com> , +Michal Skrivanek <mskrivan@redhat.com> can you have a look at this?

This is from /var/log/messages, couldn’t find anything useful in any of the /var/log/libvirt/qemu/* files, and vdsm.log has more normal operations, with some individual timeout lines, I throw in a few below.

Aug 6 15:15:41 ov8 vdsm[5437]: WARN executor state: <Executor periodic workers=4 max_workers=30 <TaskQueue periodic max_tasks=400 tasks(400)=deque([<Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675c4890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394ad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643940d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576390>, <Task <Operation action=<vdsm.virt.sampling.HostMonitor object at 0x7fc0643e2090> at 0x7fc0643e20d0> timeout=15, duration=0.00 at 0x7fc0675c40d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674ce150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c72f5d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067680110>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0643f8810>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a490>, <Task <Operation action=<function <lambda> at 0x7fc0643630c8> at 0x7fc064358090> timeout=20.0, duration=0.00 at 0x7fc0642ccf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06762ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680750>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642b1950>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067525f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642ccf10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06401afd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c722a90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b14d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642cc650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0644a6b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389b10>, <Task <Operation action=<function <lambda> at 0x7fc064363050> at 0x7fc064358050> timeout=30.0, duration=0.00 at 0x7fc0643bf490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064389ed0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064179050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179f50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643942d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674d4410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640c6390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0641790d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0677d7890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067684a90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064529250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642b1bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d2d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d8d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06402ea90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064373050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abc10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06413edd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a23d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0675246d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067576590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ad90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642cc890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640bae10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675245d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0670e8d90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a2d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06413e5d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06449f9d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642ccad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a0d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a66d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06419ee50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0674d4110>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06762ffd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bc490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517310>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643944d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2a50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702c790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06449fdd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bcb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bc610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064394290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6d50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0675171d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06401a210>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064179f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc064394190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc09c252f90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06702ce90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067757c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764afd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680f10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675767d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643f84d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06768a950>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06768ad50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768a190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fcd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676800d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402eb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c221410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115ad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0677aba50>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067115690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ba9d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647a50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768aa50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676474d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc09c02a250>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067115390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fc50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f8d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fc90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775d410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643e2690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064373c10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc06702c490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c252590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06702cb90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bce10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fe10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676809d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06413e4d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064394350>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644b1e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702cb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675768d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ad50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642bc450>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067576410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647a10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcc90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06754d250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1190>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449fad0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677d7d90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067680d90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bc310>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264950>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fa90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc06413e990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402ef90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc04c76abd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064264110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640f55d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6cd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06440ebd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067115c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640a2f90>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0642bc290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640a2c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402ee10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064080c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc04c76aad0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c76a590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676c17d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676479d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647bd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0674d4e10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a6d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0644b1210>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644a6b90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064529050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8810>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6fd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455ae90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc04c722350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f610>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640d0350>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc06439f490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677abc10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067757e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc09c02a450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707fdd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06455a450>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677aba90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c722590>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc06439fed0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640ab1d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0676471d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067680b90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643bf050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f410>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643f6710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abf10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0642b1c10>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640a26d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abf50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640ab5d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06449f490>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067647210>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067684690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0643890d0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc064264e50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264910>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764ab90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c7e7890>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707fb10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06402e110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264e90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0677abe90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264710>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524c50>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc04c7e7e90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06764a650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642642d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064142150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064264390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1dd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644a6090>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642c7690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc0640d0dd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0644d7690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b10d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677abed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a69d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643730d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576690>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675766d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640c65d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067647290>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0643f8510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6150>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0643bf250>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1ed0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6c50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064389d10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640de050>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524510>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642bcb50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064389cd0>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc067647390>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644b1f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8f90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0677ab990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0670e8350>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0675177d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06707ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.UpdateVolumes'> at 0x7fc0643e7f90> at 0x7fc064389e50> timeout=30.0, duration=0.00 at 0x7fc0640ab990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06775de10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc064264190>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0670e80d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768aa10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067517110>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768abd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc064273c50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06768afd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06417cbd0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067524790>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067576550>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524990>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0640d0550>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0642a6650>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06702c3d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439ff90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439fb90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0644a6750>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc0640abc50>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.BlockjobMonitor'> at 0x7fc064389e10> at 0x7fc064389f50> timeout=7.5, duration=0.00 at 0x7fc067524690>, <Task discardable <Operation action=<vdsm.virt.sampling.VMBulkstatsMonitor object at 0x7fc064358150> at 0x7fc0643e2050> timeout=7.5, duration=0.00 at 0x7fc0675c48d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc04c76a7d0>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc06439f990>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518850>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518290>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518c90>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518b10>, <Task discardable <Operation action=<VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90> at 0x7fc064389fd0> timeout=1.0, duration=0.00 at 0x7fc067518050>, <Tas

/var/log/vdsm/vdsm.log

2019-08-06 15:19:23,850+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:25,543+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<function <lambda> at 0x7fc09c09b578> at 0x7fc09c073590>, executor queue full (periodic:220) 2019-08-06 15:19:25,664+0000 INFO (jsonrpc/3) [api.host] START getAllVmStats() from=::ffff:10.203.1.5,54780 (api:48) 2019-08-06 15:19:25,665+0000 WARN (jsonrpc/3) [virt.vm] (vmId='d118c766-0c2c-47de-a834-0cd15c2c134a') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,666+0000 WARN (jsonrpc/3) [virt.vm] (vmId='bfa558f1-62c0-4c86-b09f-72ba01d5cdda') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,667+0000 WARN (jsonrpc/3) [virt.vm] (vmId='8fd8a5c4-8bbf-428d-8bd0-632541ba2baf') monitor became unresponsive (command timeout, age=115786.55) (vm:6060) 2019-08-06 15:19:25,668+0000 WARN (jsonrpc/3) [virt.vm] (vmId='0f1316ea-1687-4dec-86cf-f38778be8e35') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,669+0000 WARN (jsonrpc/3) [virt.vm] (vmId='3850812c-142c-40c9-9c01-539bfbdc44c6') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,669+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e4320abc-3051-42d3-9c9b-e0d523c9103b') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,670+0000 WARN (jsonrpc/3) [virt.vm] (vmId='ff7d7589-082b-4e5d-b47d-846d59de5da0') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,671+0000 WARN (jsonrpc/3) [virt.vm] (vmId='0bec8d12-9085-47ef-adba-27326ad8632e') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,672+0000 WARN (jsonrpc/3) [virt.vm] (vmId='f472f1ad-1f23-4390-ab87-9201967fb93a') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,673+0000 WARN (jsonrpc/3) [virt.vm] (vmId='c23308e7-cba1-421d-8779-742c06e6a7a1') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,674+0000 WARN (jsonrpc/3) [virt.vm] (vmId='ff84a9af-2728-4e0e-9366-38004053e0af') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,675+0000 WARN (jsonrpc/3) [virt.vm] (vmId='f18619df-f060-43ef-960f-0840f7765045') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,676+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e6a1756c-5a69-47a6-ac35-799f54e10350') monitor became unresponsive (command timeout, age=115786.56) (vm:6060) 2019-08-06 15:19:25,677+0000 WARN (jsonrpc/3) [virt.vm] (vmId='8e9d85fb-86f1-4c57-93f0-2bf39c0cae24') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,680+0000 WARN (jsonrpc/3) [virt.vm] (vmId='e81a3935-0779-4ee3-9b64-b0fc2b3e577a') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,680+0000 WARN (jsonrpc/3) [virt.vm] (vmId='c33743fd-3228-4418-a7d0-8edce9a02cdb') monitor became unresponsive (command timeout, age=115786.57) (vm:6060) 2019-08-06 15:19:25,681+0000 INFO (jsonrpc/3) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::ffff:10.203.1.5,54780 (api:54) 2019-08-06 15:19:25,685+0000 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.02 seconds (__init__:312) 2019-08-06 15:19:25,856+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:27,859+0000 WARN (vdsm.Scheduler) [virt.periodic.Operation] could not run <VmDispatcher operation=<class 'vdsm.virt.periodic.DriveWatermarkMonitor'> at 0x7fc064389f90>, executor queue full (periodic:220) 2019-08-06 15:19:29,683+0000 INFO (jsonrpc/4) [api.host] START getAllVmStats() from=::1,47764 (api:48) 2019-08-06 15:19:29,684+0000 WARN (jsonrpc/4) [virt.vm] (vmId='d118c766-0c2c-47de-a834-0cd15c2c134a') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,685+0000 WARN (jsonrpc/4) [virt.vm] (vmId='bfa558f1-62c0-4c86-b09f-72ba01d5cdda') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,686+0000 WARN (jsonrpc/4) [virt.vm] (vmId='8fd8a5c4-8bbf-428d-8bd0-632541ba2baf') monitor became unresponsive (command timeout, age=115790.57) (vm:6060) 2019-08-06 15:19:29,687+0000 WARN (jsonrpc/4) [virt.vm] (vmId='0f1316ea-1687-4dec-86cf-f38778be8e35') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,688+0000 WARN (jsonrpc/4) [virt.vm] (vmId='3850812c-142c-40c9-9c01-539bfbdc44c6') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,689+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e4320abc-3051-42d3-9c9b-e0d523c9103b') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,690+0000 WARN (jsonrpc/4) [virt.vm] (vmId='ff7d7589-082b-4e5d-b47d-846d59de5da0') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,691+0000 WARN (jsonrpc/4) [virt.vm] (vmId='0bec8d12-9085-47ef-adba-27326ad8632e') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,692+0000 WARN (jsonrpc/4) [virt.vm] (vmId='f472f1ad-1f23-4390-ab87-9201967fb93a') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,693+0000 WARN (jsonrpc/4) [virt.vm] (vmId='c23308e7-cba1-421d-8779-742c06e6a7a1') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,694+0000 WARN (jsonrpc/4) [virt.vm] (vmId='ff84a9af-2728-4e0e-9366-38004053e0af') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,695+0000 WARN (jsonrpc/4) [virt.vm] (vmId='f18619df-f060-43ef-960f-0840f7765045') monitor became unresponsive (command timeout, age=115790.58) (vm:6060) 2019-08-06 15:19:29,698+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e6a1756c-5a69-47a6-ac35-799f54e10350') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,699+0000 WARN (jsonrpc/4) [virt.vm] (vmId='8e9d85fb-86f1-4c57-93f0-2bf39c0cae24') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,699+0000 WARN (jsonrpc/4) [virt.vm] (vmId='e81a3935-0779-4ee3-9b64-b0fc2b3e577a') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,701+0000 WARN (jsonrpc/4) [virt.vm] (vmId='c33743fd-3228-4418-a7d0-8edce9a02cdb') monitor became unresponsive (command timeout, age=115790.59) (vm:6060) 2019-08-06 15:19:29,701+0000 INFO (jsonrpc/4) [api.host] FINISH getAllVmStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': (suppressed)} from=::1,47764 (api:54) 2019-08-06 15:19:29,706+0000 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded in 0.02 seconds (__init__:312)

On Aug 6, 2019, at 6:12 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:

Il giorno sab 3 ago 2019 alle ore 12:38 Strahil <hunter86_bg@yahoo.com> ha scritto:

...

Maybe the libvirtd < - > libgfapi communication is somehow broken. I'm using FUSE and I have no issues, but my lab is not I/O intensive .

I would recommend to use fuse instead of libgfapi on glusterfs 6 storage. Adding +Sahina Bose <sabose@redhat.com> to the discussion

...

Have you noticed if the issue happens during I/O (for example - full backup) intensive load?

You can even try to stop using libgfapi and check if it happens again.

Best Regards, Strahil NikolovOn Aug 3, 2019 00:39, Darrell Budic < budic@onholyground.com> wrote:

...

I’ve been seeing similar issues lately (since upgrading to 4.3.4 and

later) with gluster storage (and libgfapi), but haven’t pinned to anything particular yet. But libvirt seems to have some issues and vdsmd stops being able to poll disk usage. My VMs don’t crash or stop working, but they do report as ? in the management interface and can’t be migrated. Current work around is to restart libvirtd then vdsmd, occasionally restarting libvirtd restarts a VM (presumably the one that had broken some disk something…).

...

Sorry I don’t have more info to go on, but I’ll update here if I get

more on anything similar.

...

-Darrell

...

On Aug 2, 2019, at 9:23 AM, kevin.doyle@manchester.ac.uk wrote:

HI I think it is a bug with qemu, and isci disks. Has anyone else seen

this issue VM crashes and lost communication with qemu Can I update qemu ? if so what is the latest support version to use on Ovirt 4.3.2.1-1.el7 Current versions of qemu installed

...

...

Installed Packages qemu-img-ev.x86_64 10:2.12.0-18.el7_6.3.1 qemu-kvm-common-ev.x86_64 10:2.12.0-18.el7_6.3.1 qemu-kvm-ev.x86_64 10:2.12.0-18.el7_6.3.1

Many thanks Kevin _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct:

https://www.ovirt.org/community/about/community-guidelines/

...

List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FDMHYEJXRG6IGB...

...

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZAOILBQRM7XPYU...

---

Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WEC76OXKJ54WCV...

-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*

-- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://www.redhat.com/>*Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*