[ovirt-users] Ovirt causing strange network issues?

[Derek Atkins]

Hi,

Yes, I have (well, wireshark, but effectively the same thing).
Nothing is standing out.

I'm trying to visually coordinate the wireshark traces with my mtr run to
try to see "what's going on when my RTTs skyrocket".  Honestly the only
correlation I'm seeing is that it's when the ovirt host is checking the
ovirt engine health (and I get a bunch of TCP out of order messages).

I've already ruled out overflow of my Arris modem NAT/forwarding table.
I've already ruled out Ethernet Pause Frames.

I don't understand how something inside my network can affect the Arris in
such a profound way across both the switch and router.

-derek

On Tue, October 3, 2017 7:38 am, Jason Keltz wrote:
> Derek,
> Have you used tcpdump to check what network traffic is coming out of your
> box? Is it possible that it is some kind of DoS attack from outside in or
> that your VM was compromised and is attacking other external hosts?
>
> Hope you get to the bottom of it!
> Jason.
>
> Sent with AquaMail for Android
> http://www.aqua-mail.com
>
>
> On October 2, 2017 4:56:54 PM Derek Atkins <derek at ihtfp.com> wrote:
>
>> Hi,
>>
>> I'm at my wits end so I'm tossing this here in the hopes that SOMEONE
>> will be able to help me.
>>
>> tl;dr: Ovirt is doing something on my network that is causing my fiber
>> modem to go from 3-5ms to 300-1000+ms round trip times.  I know it's
>> ovirt because when I unplug ovirt from my network the issue goes away;
>> when I plug it back in, the issue recurs.
>>
>> Long version:
>>
>> I've been running Ovirt 4.0.6 happily on CentOS 7.3 for several months
>> on a single host machine. Indeed, the host had an uptime of 200+ days
>> and was working great until approximately midnight, September 21/22
>> (just over a week ago).  I was on an airplane halfway across the
>> Atlantic at that time, so it wasn't anything I did.
>>
>> My network is configured as:
>>
>>   fiber modem <-> edgerouter <-> switch <-> everything else
>>
>> ovirt is living in the "everything else" area.
>>
>> When I sit with a laptop connected to either the everything else range
>> or even directly connected to the fiber modem, I run 'mtr' and see
>> network times (starting at the fiber modem) that bounce all over the
>> place.  When I unplug ovirt I see consistent 3-5ms times.  Plug it back
>> in, voom, back up to badness.
>>
>> I've spent several hours plugging and unplugging different devices
>> trying to isolate the issue.  The only "device" that has any effect is
>> my ovirt box.
>>
>> I have tried to debug this in several ways, but really the only thing
>> that seems to have helped at all is shutting down all the VMs and the
>> hosted engine.  Once nothing else is running (but the host itself), only
>> then does the network seem to return to normal.
>>
>> I'm really at my wits end on this; I have no idea what is causing this
>> or what might have changed to cause the issue right at that time.  I
>> also can't imagine what ovirt is doing over the network that could cause
>> the modem, two physical hops away, to lose its mind in this way.  But my
>> experiementation is definitely showing a direct correlation.
>>
>> Help!!
>>
>> -derek
>>
>> --
>>        Derek Atkins                 617-623-3745
>>        derek at ihtfp.com             www.ihtfp.com
>>        Computer and Internet Security Consultant
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
>


-- 
       Derek Atkins                 617-623-3745
       derek at ihtfp.com             www.ihtfp.com
       Computer and Internet Security Consultant