Issue with kernel 2.6.32-573.3.1.el6.x86_64?
So I patched my vhosts and updated the kernel to 2.6.32-573.3.1.el6.x86_64. Afterwards the networking became unstable for my vyatta firewall vm. Lots of packet loss and out of order packets (based on my tshark at the time). Has anybody else experienced this? -- *Michael Kleinpaste* Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
Nobody's seen this? On Wed, Sep 16, 2015 at 9:08 AM Michael Kleinpaste < michael.kleinpaste@sharperlending.com> wrote:
So I patched my vhosts and updated the kernel to 2.6.32-573.3.1.el6.x86_64. Afterwards the networking became unstable for my vyatta firewall vm. Lots of packet loss and out of order packets (based on my tshark at the time).
Has anybody else experienced this? -- *Michael Kleinpaste* Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
-- *Michael Kleinpaste* Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
Hi Michael, Can you describe your network architecture for this vm (inside the host). Do you know where are the packets get dropped? Ido On Tue, Sep 22, 2015 at 1:19 AM, Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
Nobody's seen this?
On Wed, Sep 16, 2015 at 9:08 AM Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
So I patched my vhosts and updated the kernel to 2.6.32-573.3.1.el6.x86_64. Afterwards the networking became unstable for my vyatta firewall vm. Lots of packet loss and out of order packets (based on my tshark at the time).
Has anybody else experienced this? -- Michael Kleinpaste Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
-- Michael Kleinpaste Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Thanks, Ido Barkan
VMs are on different VLANs and use a central Vyos VM as the firewall and default gateway. The only indication I had that the packets were getting dropped or being sent out of order was by tshark'ing the traffic. Tons and tons of resends. The problem was definitely resolved after I dropped back to the prior kernel (2.6.32-504.12.2.el6.x86_64). On Tue, Oct 20, 2015 at 11:51 PM Ido Barkan <ibarkan@redhat.com> wrote:
Hi Michael, Can you describe your network architecture for this vm (inside the host). Do you know where are the packets get dropped? Ido
On Tue, Sep 22, 2015 at 1:19 AM, Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
Nobody's seen this?
On Wed, Sep 16, 2015 at 9:08 AM Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
So I patched my vhosts and updated the kernel to 2.6.32-573.3.1.el6.x86_64. Afterwards the networking became unstable
for my
vyatta firewall vm. Lots of packet loss and out of order packets (based on my tshark at the time).
Has anybody else experienced this? -- Michael Kleinpaste Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
-- Michael Kleinpaste Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Thanks, Ido Barkan
-- *Michael Kleinpaste* Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
It might help to tcpdump at different points (devices: bridge, vlan device, physical nic or bond) in order to isolate the evil dropper. Ido Thanks, Ido On Oct 22, 2015 12:06 AM, "Michael Kleinpaste" < michael.kleinpaste@sharperlending.com> wrote:
VMs are on different VLANs and use a central Vyos VM as the firewall and default gateway. The only indication I had that the packets were getting dropped or being sent out of order was by tshark'ing the traffic. Tons and tons of resends.
The problem was definitely resolved after I dropped back to the prior kernel (2.6.32-504.12.2.el6.x86_64).
On Tue, Oct 20, 2015 at 11:51 PM Ido Barkan <ibarkan@redhat.com> wrote:
Hi Michael, Can you describe your network architecture for this vm (inside the host). Do you know where are the packets get dropped? Ido
On Tue, Sep 22, 2015 at 1:19 AM, Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
Nobody's seen this?
On Wed, Sep 16, 2015 at 9:08 AM Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
So I patched my vhosts and updated the kernel to 2.6.32-573.3.1.el6.x86_64. Afterwards the networking became unstable
for my
vyatta firewall vm. Lots of packet loss and out of order packets (based on my tshark at the time).
Has anybody else experienced this? -- Michael Kleinpaste Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
-- Michael Kleinpaste Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Thanks, Ido Barkan
-- *Michael Kleinpaste* Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Michael, if you have the possibility to test with kernel-2.6.32-504.16.2.el6.x86_64 and kernel-2.6.32-504.23.4.el6.x86_64 please do it. I'm expecting that 2.6.32-504.16.2.el6 is the latest correctly working kernel. Please tell us if this is your case. There are clear signs of misbehaviours due to changes in the VLAN code between the two kernels above. Also take a look at https://bugs.centos.org/view.php?id=9467 and https://bugzilla.redhat.com/show_bug.cgi?id=1263561 . Best regards, Giorgio. P.S. sorry for top posting. I tried to revert this thread but I couldn't do it in an effective way :( 2015-10-21 23:05 GMT+02:00 Michael Kleinpaste <michael.kleinpaste@sharperlending.com>:
VMs are on different VLANs and use a central Vyos VM as the firewall and default gateway. The only indication I had that the packets were getting dropped or being sent out of order was by tshark'ing the traffic. Tons and tons of resends.
The problem was definitely resolved after I dropped back to the prior kernel (2.6.32-504.12.2.el6.x86_64).
On Tue, Oct 20, 2015 at 11:51 PM Ido Barkan <ibarkan@redhat.com> wrote:
Hi Michael, Can you describe your network architecture for this vm (inside the host). Do you know where are the packets get dropped? Ido
On Tue, Sep 22, 2015 at 1:19 AM, Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
Nobody's seen this?
On Wed, Sep 16, 2015 at 9:08 AM Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
So I patched my vhosts and updated the kernel to 2.6.32-573.3.1.el6.x86_64. Afterwards the networking became unstable for my vyatta firewall vm. Lots of packet loss and out of order packets (based on my tshark at the time).
Has anybody else experienced this? -- Michael Kleinpaste
Not an option unfortunately. The system is a production one and we don't have spare equipment to test that on. On Fri, Oct 23, 2015, 12:47 AM Giorgio Bersano <giorgio.bersano@gmail.com> wrote:
Hi Michael, if you have the possibility to test with kernel-2.6.32-504.16.2.el6.x86_64 and kernel-2.6.32-504.23.4.el6.x86_64 please do it.
I'm expecting that 2.6.32-504.16.2.el6 is the latest correctly working kernel. Please tell us if this is your case. There are clear signs of misbehaviours due to changes in the VLAN code between the two kernels above.
Also take a look at https://bugs.centos.org/view.php?id=9467 and https://bugzilla.redhat.com/show_bug.cgi?id=1263561 .
Best regards, Giorgio.
P.S. sorry for top posting. I tried to revert this thread but I couldn't do it in an effective way :(
2015-10-21 23:05 GMT+02:00 Michael Kleinpaste <michael.kleinpaste@sharperlending.com>:
VMs are on different VLANs and use a central Vyos VM as the firewall and default gateway. The only indication I had that the packets were getting dropped or being sent out of order was by tshark'ing the traffic. Tons and tons of resends.
The problem was definitely resolved after I dropped back to the prior kernel (2.6.32-504.12.2.el6.x86_64).
On Tue, Oct 20, 2015 at 11:51 PM Ido Barkan <ibarkan@redhat.com> wrote:
Hi Michael, Can you describe your network architecture for this vm (inside the
host).
Do you know where are the packets get dropped? Ido
On Tue, Sep 22, 2015 at 1:19 AM, Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
Nobody's seen this?
On Wed, Sep 16, 2015 at 9:08 AM Michael Kleinpaste <michael.kleinpaste@sharperlending.com> wrote:
So I patched my vhosts and updated the kernel to 2.6.32-573.3.1.el6.x86_64. Afterwards the networking became unstable for my vyatta firewall vm. Lots of packet loss and out of order packets (based on my tshark at the time).
Has anybody else experienced this? -- Michael Kleinpaste
-- *Michael Kleinpaste* Senior Systems Administrator SharperLending, LLC. www.SharperLending.com Michael.Kleinpaste@SharperLending.com (509) 324-1230 Fax: (509) 324-1234
This is a multi-part message in MIME format. --------------000308090100020801060508 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit On 23.10.2015 09:47, Giorgio Bersano wrote:
Hi Michael, if you have the possibility to test with kernel-2.6.32-504.16.2.el6.x86_64 and kernel-2.6.32-504.23.4.el6.x86_64 please do it.
I have similar VLAN problems with kernels 573.7.1 and 573.8.1. On 2 of my hosts VLANs are not working. I use bonding and VLANs over the bond0. The problematic hosts use the "igb" network driver with "Intel Corporation I350 Gigabit Network Connection". Other hosts have working VLANs. With .504. kernels everything works. PS: https://www.centos.org/forums/viewtopic.php?f=16&t=54330 -- Ernest Beinrohr, AXON PRO Ing <http://www.beinrohr.sk/ing.php>, RHCE <http://www.beinrohr.sk/rhce.php>, RHCVA <http://www.beinrohr.sk/rhce.php>, LPIC <http://www.beinrohr.sk/lpic.php>, VCA <http://www.beinrohr.sk/vca.php>, +421-2-62410360 +421-903-482603 --------------000308090100020801060508 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <div class="moz-cite-prefix">On 23.10.2015 09:47, Giorgio Bersano wrote:<br> </div> <blockquote cite="mid:CA+ZC=Dg6jxshxnYc17A4=_YN6hxMB6nziLngADQ6GLzWcvtcsg@mail.gmail.com" type="cite"> <pre wrap="">Hi Michael, if you have the possibility to test with kernel-2.6.32-504.16.2.el6.x86_64 and kernel-2.6.32-504.23.4.el6.x86_64 please do it. </pre> </blockquote> <br> I have similar VLAN problems with kernels 573.7.1 and 573.8.1. On 2 of my hosts VLANs are not working. I use bonding and VLANs over the bond0. The problematic hosts use the "igb" network driver with "Intel Corporation I350 Gigabit Network Connection". Other hosts have working VLANs. <br> <br> With .504. kernels everything works.<br> <br> <br> PS: <a class="moz-txt-link-freetext" href="https://www.centos.org/forums/viewtopic.php?f=16&t=54330">https://www.centos.org/forums/viewtopic.php?f=16&t=54330</a><br> <div class="moz-signature">-- <br> <div id="oernii_footer" style="color: gray;"> <span style="font-family: Lucida Console, Luxi Mono, Courier, monospace; font-size: 90%;"> Ernest Beinrohr, AXON PRO<br> <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/ing.php">Ing</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCE</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCVA</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/lpic.php">LPIC</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/vca.php">VCA</a>, <br> +421-2-62410360 +421-903-482603 <br> </span> </div> <img src="http://nojsstats.appspot.com/UA-44497096-1/email.beinrohr.sk" moz-do-not-send="true" border="0" height="1" width="1"> </div> </body> </html> --------------000308090100020801060508--
participants (4)
-
Ernest Beinrohr -
Giorgio Bersano -
Ido Barkan -
Michael Kleinpaste