4:47 p.m.
We're running oVirt 3.6 on 9 Dell Blades but with just two fairly fat
fabrics, one for LAN stuff, ovirtmgmt and one for iSCSI to the storage
domains.
15 VM Storage Domains
iSCSI has 4 paths going through a 40Gbit i/o blade to switch
115 VMs or thereabouts
9 VLANS, sharing an i/o blade with ovirtmgmt 40Gbit to switch
500+ virtual disks
What we are seeing more and more is that if we do an operation like
expose a new LUN and configure a new storage domain, that all of the
hyervisors go "red triangle" and "Connecting..." and it takes a very
long time (all day) to straighten out.
My guess is that there's too much to look at vdsm wise and so it's
waiting a short(er) period of time for a completed response than what
vdsm is going to us, and it just cycles over and over until it just
happens to work.
I'm thinking that vdsm having DEBUG enabled isn't helping the latency,
but as far as I know it came this way be default. Can we safely disable
DEBUG on the hypervisor hosts for vdsm? Can we do this while things are
roughly in a steady state? Remember, just doing the moves could throw
everything into vdsm la-la-land (actually, that might not be true, might
take a new storage thing to do that).
Just thinking out loud... can we safely turn off DEBUG logging on the
vdsms? Can we do this "live" through bouncing of the vdsm if everything
is "steady state"? Do you think this might help the problems we're
having with storage operations? (I can see all the blades logging in
iSCSI wise, but ovirt engine does the whole red triangle connecting
thing, for many, many, many hours).
Thanks,
Christopher
1:34 a.m.
On Mar 13, 2018 11:48 PM, "Christopher Cox" <ccox(a)endlessnow.com> wrote:
We're running oVirt 3.6 on 9 Dell Blades but with just two fairly fat
fabrics, one for LAN stuff, ovirtmgmt and one for iSCSI to the storage
domains.
15 VM Storage Domains
iSCSI has 4 paths going through a 40Gbit i/o blade to switch
115 VMs or thereabouts
9 VLANS, sharing an i/o blade with ovirtmgmt 40Gbit to switch
500+ virtual disks
What we are seeing more and more is that if we do an operation like expose
a new LUN and configure a new storage domain, that all of the hyervisors go
"red triangle" and "Connecting..." and it takes a very long time (all
day)
to straighten out.
My guess is that there's too much to look at vdsm wise and so it's waiting
a short(er) period of time for a completed response than what vdsm is going
to us, and it just cycles over and over until it just happens to work.
Please upgrade. We have solved issues and improved performance and scale
substantially since 3.6.
You may also wish to apply lvm filters.
Y.
I'm thinking that vdsm having DEBUG enabled isn't helping the latency, but
as far as I know it came this way be default. Can we safely disable DEBUG
on the hypervisor hosts for vdsm? Can we do this while things are roughly
in a steady state? Remember, just doing the moves could throw everything
into vdsm la-la-land (actually, that might not be true, might take a new
storage thing to do that).
Just thinking out loud... can we safely turn off DEBUG logging on the
vdsms? Can we do this "live" through bouncing of the vdsm if everything is
"steady state"? Do you think this might help the problems we're having
with storage operations? (I can see all the blades logging in iSCSI wise,
but ovirt engine does the whole red triangle connecting thing, for many,
many, many hours).
Thanks,
Christopher
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
9:10 a.m.
On 03/14/2018 01:34 AM, Yaniv Kaul wrote:
On Mar 13, 2018 11:48 PM, "Christopher Cox" <ccox(a)endlessnow.com
<mailto:ccox@endlessnow.com>> wrote:
...snip...
What we are seeing more and more is that if we do an operation like expose a
new LUN and configure a new storage domain, that all of the hyervisors go
"red triangle" and "Connecting..." and it takes a very long time
(all day)
to straighten out.
My guess is that there's too much to look at vdsm wise and so it's waiting a
short(er) period of time for a completed response than what vdsm is going to
us, and it just cycles over and over until it just happens to work.
Please upgrade. We have solved issues and improved performance and scale
substantially since 3.6.
You may also wish to apply lvm filters.
Y.
Oh, we know and are looking at what we'll have to do to upgrade. With that
said, is there more information on what you mentioned as "lvm filters" posted
somewhere?
Also, would VM reduction, and IMHO, virtual disk reduction help this problem?
Is there and engine config parameters that might help as well?
Thanks for any help on this.
10:08 a.m.
On 03/14/2018 09:10 AM, Christopher Cox wrote:
On 03/14/2018 01:34 AM, Yaniv Kaul wrote:
>
>
> On Mar 13, 2018 11:48 PM, "Christopher Cox" <ccox(a)endlessnow.com
> <mailto:ccox@endlessnow.com>> wrote:
>
...snip...
>
> What we are seeing more and more is that if we do an operation
> like expose a
> new LUN and configure a new storage domain, that all of the
> hyervisors go
> "red triangle" and "Connecting..." and it takes a very long
time
> (all day)
> to straighten out.
>
> My guess is that there's too much to look at vdsm wise and so it's
> waiting a
> short(er) period of time for a completed response than what vdsm
> is going to
> us, and it just cycles over and over until it just happens to work.
>
>
> Please upgrade. We have solved issues and improved performance and scale
> substantially since 3.6.
> You may also wish to apply lvm filters.
> Y.
Oh, we know and are looking at what we'll have to do to upgrade. With
that said, is there more information on what you mentioned as "lvm
filters" posted somewhere?
Also, would VM reduction, and IMHO, virtual disk reduction help this
problem?
Is there and engine config parameters that might help as well?
Thanks for any help on this.
Based on a different older thread about having lots of virtual networks
which sounded somewhat similar, I have increased our vdsTimeout value.
Any opinions on whether or not that might help? Right now I'm forced to
tell my management that we'll have to "roll the dice" to find out. But
kind of hoping to hear someone "say" it should help. Anyone?
Just looking for something more substantial...
2389
days inactive
2392
days old
3 comments
2 participants
participants (2)
-
Christopher Cox -
Yaniv Kaul