[ovirt-users] Lots of "domain in problem" warnings

[Nir Soffer]

On Thu, Apr 7, 2016 at 5:05 PM,  <nicolas at devels.es> wrote:
> Hi,
>
> Lately we're having a lot of events like these:
>
> 2016-04-07 14:54:25,247 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-2) [] domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
> 'host7.domain.com'
> 2016-04-07 14:54:40,501 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-17) [] Domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' recovered from problem. vds:
> 'host7.domain.com'
> 2016-04-07 14:54:40,501 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-17) [] Domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' has recovered from problem.
> No active host in the DC is reporting it as problematic, so clearing the
> domain recovery timer.
> 2016-04-07 14:54:46,314 WARN
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-30) [] domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:
> 'host5.domain.com'
> 2016-04-07 14:55:01,589 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-32) [] Domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' recovered from problem. vds:
> 'host5.domain.com'
> 2016-04-07 14:55:01,589 INFO
> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]
> (org.ovirt.thread.pool-8-thread-32) [] Domain
> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' has recovered from problem.
> No active host in the DC is reporting it as problematic, so clearing the
> domain recovery timer.
>
> Up until now it's been only one domain that I see with this warning, this
> doesn't look good nevertheless. Not sure if related, but I can't find a disk
> with this UUID. How can I start debugging?
>
> This is oVirt 3.6.4.1-1, and using an iSCSI-based storage backend.

This may be related to this bug:
https://bugzilla.redhat.com/1081962

Running this tool on vdsm log will give better picture of what is
happening in the vdsm side:
https://github.com/oVirt/vdsm/blob/master/contrib/repoplot

You can see examples of the output in the bug:
https://bugzilla.redhat.com/attachment.cgi?id=1130967

We are working on improve monitoring that will eliminate this issue.


Nir