[ovirt-users] Quick question about oVirt 3.6 and vdsm log in DEBUG mode (apprently by default??)

[Christopher Cox]

We're running oVirt 3.6 on 9 Dell Blades but with just two fairly fat 
fabrics, one for LAN stuff, ovirtmgmt and one for iSCSI to the storage 
domains.

15 VM Storage Domains
iSCSI has 4 paths going through a 40Gbit i/o blade to switch

115 VMs or thereabouts
9 VLANS, sharing an i/o blade with ovirtmgmt 40Gbit to switch
500+ virtual disks

What we are seeing more and more is that if we do an operation like 
expose a new LUN and configure a new storage domain, that all of the 
hyervisors go "red triangle" and "Connecting..." and it takes a very 
long time (all day) to straighten out.

My guess is that there's too much to look at vdsm wise and so it's 
waiting a short(er) period of time for a completed response than what 
vdsm is going to us, and it just cycles over and over until it just 
happens to work.

I'm thinking that vdsm having DEBUG enabled isn't helping the latency, 
but as far as I know it came this way be default.  Can we safely disable 
DEBUG on the hypervisor hosts for vdsm?  Can we do this while things are 
roughly in a steady state?  Remember, just doing the moves could throw 
everything into vdsm la-la-land (actually, that might not be true, might 
take a new storage thing to do that).

Just thinking out loud... can we safely turn off DEBUG logging on the 
vdsms? Can we do this "live" through bouncing of the vdsm if everything 
is "steady state"?  Do you think this might help the problems we're 
having with storage operations? (I can see all the blades logging in 
iSCSI wise, but ovirt engine does the whole red triangle connecting 
thing, for many, many, many hours).

Thanks,
Christopher