10:49 a.m.
Hi all,
Since we upgraded to the latest ovirt node running 7.2, we're seeing that
nodes become unavailable after a while. It's running fine, with a couple of
VM's on it, untill it becomes non responsive. At that moment it doesn't
even respond to ICMP. It'll come back by itself after a while, but oVirt
fences the machine before that time and restarts VM's elsewhere.
Engine tells me this message:
VDSM host09 command failed: Message timeout which can be caused by
communication issues
Is anyone else experiencing these issues with ixgbe drivers? I'm running on
Intel X540-AT2 cards.
--
Met vriendelijke groeten / With kind regards,
Johan Kooijman
1:52 p.m.
On Thu, Mar 17, 2016 at 10:49 AM, Johan Kooijman <mail(a)johankooijman.com> wrote:
Hi all,
Since we upgraded to the latest ovirt node running 7.2, we're seeing that
nodes become unavailable after a while. It's running fine, with a couple of
VM's on it, untill it becomes non responsive. At that moment it doesn't even
respond to ICMP. It'll come back by itself after a while, but oVirt fences
the machine before that time and restarts VM's elsewhere.
Engine tells me this message:
VDSM host09 command failed: Message timeout which can be caused by
communication issues
Is anyone else experiencing these issues with ixgbe drivers? I'm running on
Intel X540-AT2 cards.
We will need engine and vdsm logs to understand this issue.
Can you file a bug and attach ful logs?
Nir
1:57 p.m.
Johan,
It there is temporary networking issue and you still want engine not to
fence the host you can
increase heartbeat interval in the engine configuration. It would tell
engine to wait longer
before assuming that the host is not responding.
Please provide the logs so we can understand why there is communication
issue in the first
place.
Thanks,
Piotr
On Thu, Mar 17, 2016 at 12:52 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Thu, Mar 17, 2016 at 10:49 AM, Johan Kooijman
<mail(a)johankooijman.com>
wrote:
> Hi all,
>
> Since we upgraded to the latest ovirt node running 7.2, we're seeing that
> nodes become unavailable after a while. It's running fine, with a couple
of
> VM's on it, untill it becomes non responsive. At that moment it doesn't
even
> respond to ICMP. It'll come back by itself after a while, but oVirt
fences
> the machine before that time and restarts VM's elsewhere.
>
> Engine tells me this message:
>
> VDSM host09 command failed: Message timeout which can be caused by
> communication issues
>
> Is anyone else experiencing these issues with ixgbe drivers? I'm running
on
> Intel X540-AT2 cards.
We will need engine and vdsm logs to understand this issue.
Can you file a bug and attach ful logs?
Nir
5:27 p.m.
Hi,
Is this on CentOS/RHEL 7.2?
Log in as root as see if you can see any messages from ixgbe about "tx queue
hung" in dmesg. I
currently have an open support case for RHEL7.2 and the ixgbe driver, where there is a
driver
issue causing the network adapter to reset continuously when there are network traffic.
Regards,
Siggi
On Thu, March 17, 2016 12:52, Nir Soffer wrote:
On Thu, Mar 17, 2016 at 10:49 AM, Johan Kooijman
<mail(a)johankooijman.com> wrote:
> Hi all,
>
>
> Since we upgraded to the latest ovirt node running 7.2, we're seeing that
> nodes become unavailable after a while. It's running fine, with a couple of
VM's on it, untill it
> becomes non responsive. At that moment it doesn't even respond to ICMP. It'll
come back by
> itself after a while, but oVirt fences the machine before that time and restarts
VM's elsewhere.
>
>
> Engine tells me this message:
>
>
> VDSM host09 command failed: Message timeout which can be caused by
> communication issues
>
> Is anyone else experiencing these issues with ixgbe drivers? I'm running on
> Intel X540-AT2 cards.
>
We will need engine and vdsm logs to understand this issue.
Can you file a bug and attach ful logs?
Nir
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
5:34 p.m.
I had the same issue, and I also have a support case open. They referenced
https://bugzilla.redhat.com/show_bug.cgi?id=1288237 which is private. I
didn't have any success getting that bugzilla changed to public. We
couldn't keep waiting for the issue to be fixed so we replaced the NICs
with Broadcom/Qlogic that we knew had no issues in other hosts.
On Thu, Mar 17, 2016 at 11:27 AM, Sigbjorn Lie <sigbjorn(a)nixtra.com> wrote:
Hi,
Is this on CentOS/RHEL 7.2?
Log in as root as see if you can see any messages from ixgbe about "tx
queue hung" in dmesg. I
currently have an open support case for RHEL7.2 and the ixgbe driver,
where there is a driver
issue causing the network adapter to reset continuously when there are
network traffic.
Regards,
Siggi
On Thu, March 17, 2016 12:52, Nir Soffer wrote:
> On Thu, Mar 17, 2016 at 10:49 AM, Johan Kooijman <mail(a)johankooijman.com>
wrote:
>
>> Hi all,
>>
>>
>> Since we upgraded to the latest ovirt node running 7.2, we're seeing
that
>> nodes become unavailable after a while. It's running fine, with a
couple of VM's on it, untill it
>> becomes non responsive. At that moment it doesn't even respond to ICMP.
It'll come back by
>> itself after a while, but oVirt fences the machine before that time and
restarts VM's elsewhere.
>>
>>
>> Engine tells me this message:
>>
>>
>> VDSM host09 command failed: Message timeout which can be caused by
>> communication issues
>>
>> Is anyone else experiencing these issues with ixgbe drivers? I'm
running on
>> Intel X540-AT2 cards.
>>
>
> We will need engine and vdsm logs to understand this issue.
>
>
> Can you file a bug and attach ful logs?
>
>
> Nir
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:01 a.m.
--_000_56EBB5D93000101dmcamcnetworkscom_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
Hi Johan,
Could you check if you see the following in you dmesg or message log file?
[1123306.014288] ------------[ cut here ]------------
[1123306.014302] WARNING: at net/core/dev.c:2189 skb_warn_bad_offload+0xcd/=
0xda()
[1123306.014306] : caps=3D(0x0000000200004849, 0x0000000000000000) len=3D33=
0 data_len=3D276 gso_size=3D276 gso_type=3D1 ip_summed=3D1
[1123306.014308] Modules linked in: vhost_net macvtap macvlan ip6table_filt=
er ip6_tables iptable_filter ip_tables ebt_arp ebtable_nat ebtables tun scs=
i_transport_iscsi iTCO_wdt iTCO_vendor_support dm_service_time intel_powerc=
lamp coretemp intel_rapl kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_=
clmulni_intel cryptd pcspkr sb_edac edac_core i2c_i801 lpc_ich mfd_core mei=
_me mei wmi ioatdma shpchp ipmi_devintf ipmi_si ipmi_msghandler acpi_power_=
meter acpi_pad 8021q garp mrp bridge stp llc bonding dm_multipath xfs libcr=
c32c sd_mod crc_t10dif crct10dif_common ast syscopyarea sysfillrect sysimgb=
lt drm_kms_helper ttm crc32c_intel igb drm ahci ixgbe i2c_algo_bit libahci =
libata mdio i2c_core ptp megaraid_sas pps_core dca dm_mirror dm_region_hash=
dm_log dm_mod
[1123306.014360] CPU: 30 PID: 0 Comm: swapper/30 Tainted: G W ----=
---------- 3.10.0-229.1.2.el7.x86_64 #1
[1123306.014362] Hardware name: Supermicro SYS-2028TP-HC1TR/X10DRT-PT, BIOS=
1.1 08/03/2015
[1123306.014364] ffff881fffc439a8 5326fb90ad1041ea ffff881fffc43960 ffffff=
ff81604afa
[1123306.014371] ffff881fffc43998 ffffffff8106e34b ffff881fcebb0500 ffff88=
1fce88c000
[1123306.014376] 0000000000000001 0000000000000001 ffff881fcebb0500 ffff88=
1fffc43a00
[1123306.014381] Call Trace:
[1123306.014383] <IRQ> [<ffffffff81604afa>] dump_stack+0x19/0x1b
[1123306.014396] [<ffffffff8106e34b>] warn_slowpath_common+0x6b/0xb0
[1123306.014399] [<ffffffff8106e3ec>] warn_slowpath_fmt+0x5c/0x80
[1123306.014405] [<ffffffff812db093>] ? ___ratelimit+0x93/0x100
[1123306.014409] [<ffffffff816076c3>] skb_warn_bad_offload+0xcd/0xda
[1123306.014425] [<ffffffff814fdeb9>] __skb_gso_segment+0x79/0xb0
[1123306.014429] [<ffffffff814fe1c2>] dev_hard_start_xmit+0x1a2/0x580
[1123306.014438] [<ffffffffa0168790>] ? deliver_clone+0x50/0x50 [bridge]
[1123306.014443] [<ffffffff8151df1e>] sch_direct_xmit+0xee/0x1c0
[1123306.014447] [<ffffffff814fe798>] dev_queue_xmit+0x1f8/0x4a0
[1123306.014453] [<ffffffffa016880b>] br_dev_queue_push_xmit+0x7b/0xc0 [br=
idge]
[1123306.014458] [<ffffffffa0168a22>] br_forward_finish+0x22/0x60 [bridge]
[1123306.014464] [<ffffffffa0168ae0>] __br_forward+0x80/0xf0 [bridge]
[1123306.014469] [<ffffffffa0168ebb>] br_forward+0x8b/0xa0 [bridge]
[1123306.014476] [<ffffffffa0169e65>] br_handle_frame_finish+0x175/0x410 [=
bridge]
[1123306.014481] [<ffffffffa016a275>] br_handle_frame+0x175/0x260 [bridge]
[1123306.014485] [<ffffffff814fc112>] __netif_receive_skb_core+0x282/0x870
[1123306.014490] [<ffffffff8101b589>] ? read_tsc+0x9/0x10
[1123306.014493] [<ffffffff814fc718>] __netif_receive_skb+0x18/0x60
[1123306.014497] [<ffffffff814fc7a0>] netif_receive_skb+0x40/0xd0
[1123306.014500] [<ffffffff814fd2b0>] napi_gro_receive+0x80/0xb0
[1123306.014512] [<ffffffffa00cde2c>] ixgbe_clean_rx_irq+0x7ac/0xb30 [ixgb=
e]
[1123306.014519] [<ffffffffa00cf07b>] ixgbe_poll+0x4bb/0x930 [ixgbe]
[1123306.014524] [<ffffffff814fcb62>] net_rx_action+0x152/0x240
[1123306.014528] [<ffffffff81077bf7>] __do_softirq+0xf7/0x290
[1123306.014533] [<ffffffff8161635c>] call_softirq+0x1c/0x30
[1123306.014539] [<ffffffff81015de5>] do_softirq+0x55/0x90
[1123306.014543] [<ffffffff81077f95>] irq_exit+0x115/0x120
[1123306.014546] [<ffffffff81616ef8>] do_IRQ+0x58/0xf0
[1123306.014551] [<ffffffff8160c0ed>] common_interrupt+0x6d/0x6d
[1123306.014553] <EOI> [<ffffffff814aa6d2>] ? cpuidle_enter_state+0x52/0x=
c0
[1123306.014561] [<ffffffff814aa6c8>] ? cpuidle_enter_state+0x48/0xc0
[1123306.014565] [<ffffffff814aa805>] cpuidle_idle_call+0xc5/0x200
[1123306.014569] [<ffffffff8101d21e>] arch_cpu_idle+0xe/0x30
[1123306.014574] [<ffffffff810c6945>] cpu_startup_entry+0xf5/0x290
[1123306.014580] [<ffffffff810423ca>] start_secondary+0x1ba/0x230
[1123306.014582] ---[ end trace 4d5a1bc838e1fcc0 ]---
If so, then could you try the following:
ethtool -K <nic name> lro off
Do this for all the 10G intel nics and check if the problems still exists
Kind regards,
Jurri=EBn Bloemen
On 17-03-16 09:49, Johan Kooijman wrote:
Hi all,
Since we upgraded to the latest ovirt node running 7.2, we're seeing that n=
odes become unavailable after a while. It's running fine, with a couple of =
VM's on it, untill it becomes non responsive. At that moment it doesn't eve=
n respond to ICMP. It'll come back by itself after a while, but oVirt fence=
s the machine before that time and restarts VM's elsewhere.
Engine tells me this message:
VDSM host09 command failed: Message timeout which can be caused by communic=
ation issues
Is anyone else experiencing these issues with ixgbe drivers? I'm running on=
Intel X540-AT2 cards.
--
Met vriendelijke groeten / With kind regards,
Johan Kooijman
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
This message (including any attachments) may contain information that is pr=
ivileged or confidential. If you are not the intended recipient, please not=
ify the sender and delete this email immediately from your systems and dest=
roy all copies of it. You may not, directly or indirectly, use, disclose, d=
istribute, print or copy this email or any part of it if you are not the in=
tended recipient
--_000_56EBB5D93000101dmcamcnetworkscom_
Content-Type: text/html; charset="Windows-1252"
Content-ID: <69C5549CA6C47E439841A1DD44699816(a)chellomedia.com>
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html;
charset=3DWindows-1=
252">
</head>
<body bgcolor=3D"#FFFFFF" text=3D"#000000">
<tt>Hi Johan,<br>
<br>
Could you check if you see the following in you dmesg or message log file?<=
br>
<br>
[1123306.014288] ------------[ cut here ]------------<br>
[1123306.014302] WARNING: at net/core/dev.c:2189 skb_warn_bad_offload+0=
xcd/0xda()<br>
[1123306.014306] : caps=3D(0x0000000200004849, 0x0000000000000000) len=3D33=
0 data_len=3D276 gso_size=3D276 gso_type=3D1 ip_summed=3D1<br>
[1123306.014308] Modules linked in: vhost_net macvtap macvlan ip6table_filt=
er ip6_tables iptable_filter ip_tables ebt_arp ebtable_nat ebtables tun scs=
i_transport_iscsi iTCO_wdt iTCO_vendor_support dm_service_time intel_powerc=
lamp coretemp intel_rapl kvm_intel
kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cryptd pcspkr sb_eda=
c edac_core i2c_i801 lpc_ich mfd_core mei_me mei wmi ioatdma shpchp ipmi_de=
vintf ipmi_si ipmi_msghandler acpi_power_meter acpi_pad 8021q garp mrp brid=
ge stp llc bonding dm_multipath
xfs libcrc32c sd_mod crc_t10dif crct10dif_common ast syscopyarea sysfillre=
ct sysimgblt drm_kms_helper ttm crc32c_intel igb drm ahci ixgbe i2c_algo_bi=
t libahci libata mdio i2c_core ptp megaraid_sas pps_core dca dm_mirror dm_r=
egion_hash dm_log dm_mod<br>
[1123306.014360] CPU: 30 PID: 0 Comm: swapper/30 Tainted: G &nbs=
p; W
-------------- 3.10.0-=
229.1.2.el7.x86_64 #1<br>
[1123306.014362] Hardware name: Supermicro SYS-2028TP-HC1TR/X10DRT-PT, BIOS=
1.1 08/03/2015<br>
[1123306.014364] ffff881fffc439a8 5326fb90ad1041ea ffff881fffc43960 f=
fffffff81604afa<br>
[1123306.014371] ffff881fffc43998 ffffffff8106e34b ffff881fcebb0500 f=
fff881fce88c000<br>
[1123306.014376] 0000000000000001 0000000000000001 ffff881fcebb0500 f=
fff881fffc43a00<br>
[1123306.014381] Call Trace:<br>
[1123306.014383] <IRQ>
[<ffffffff81604afa>] dump_st=
ack+0x19/0x1b<br>
[1123306.014396] [<ffffffff8106e34b>]
warn_slowpath_common+=
0x6b/0xb0<br>
[1123306.014399] [<ffffffff8106e3ec>]
warn_slowpath_fmt+0x5=
c/0x80<br>
[1123306.014405] [<ffffffff812db093>] ?
___ratelimit+0x93/0=
x100<br>
[1123306.014409] [<ffffffff816076c3>]
skb_warn_bad_offload+=
0xcd/0xda<br>
[1123306.014425] [<ffffffff814fdeb9>]
__skb_gso_segment+0x7=
9/0xb0<br>
[1123306.014429] [<ffffffff814fe1c2>]
dev_hard_start_xmit+0=
x1a2/0x580<br>
[1123306.014438] [<ffffffffa0168790>] ?
deliver_clone+0x50/=
0x50 [bridge]<br>
[1123306.014443] [<ffffffff8151df1e>]
sch_direct_xmit+0xee/=
0x1c0<br>
[1123306.014447] [<ffffffff814fe798>]
dev_queue_xmit+0x1f8/=
0x4a0<br>
[1123306.014453] [<ffffffffa016880b>]
br_dev_queue_push_xmit=
3;0x7b/0xc0 [bridge]<br>
[1123306.014458] [<ffffffffa0168a22>]
br_forward_finish+0x2=
2/0x60 [bridge]<br>
[1123306.014464] [<ffffffffa0168ae0>]
__br_forward+0x80/0xf=
0 [bridge]<br>
[1123306.014469] [<ffffffffa0168ebb>] br_forward+0x8b/0xa0
=
[bridge]<br>
[1123306.014476] [<ffffffffa0169e65>]
br_handle_frame_finish=
3;0x175/0x410 [bridge]<br>
[1123306.014481] [<ffffffffa016a275>]
br_handle_frame+0x175=
/0x260 [bridge]<br>
[1123306.014485] [<ffffffff814fc112>]
__netif_receive_skb_core&=
#43;0x282/0x870<br>
[1123306.014490] [<ffffffff8101b589>] ?
read_tsc+0x9/0x10<b=
r>
[1123306.014493] [<ffffffff814fc718>]
__netif_receive_skb+0=
x18/0x60<br>
[1123306.014497] [<ffffffff814fc7a0>]
netif_receive_skb+0x4=
0/0xd0<br>
[1123306.014500] [<ffffffff814fd2b0>]
napi_gro_receive+0x80=
/0xb0<br>
[1123306.014512] [<ffffffffa00cde2c>]
ixgbe_clean_rx_irq+0x=
7ac/0xb30 [ixgbe]<br>
[1123306.014519] [<ffffffffa00cf07b>]
ixgbe_poll+0x4bb/0x93=
0 [ixgbe]<br>
[1123306.014524] [<ffffffff814fcb62>]
net_rx_action+0x152/0=
x240<br>
[1123306.014528] [<ffffffff81077bf7>]
__do_softirq+0xf7/0x2=
90<br>
[1123306.014533] [<ffffffff8161635c>]
call_softirq+0x1c/0x3=
0<br>
[1123306.014539] [<ffffffff81015de5>]
do_softirq+0x55/0x90<=
br>
[1123306.014543] [<ffffffff81077f95>]
irq_exit+0x115/0x120<=
br>
[1123306.014546] [<ffffffff81616ef8>]
do_IRQ+0x58/0xf0<br>
[1123306.014551] [<ffffffff8160c0ed>]
common_interrupt+0x6d=
/0x6d<br>
[1123306.014553] <EOI>
[<ffffffff814aa6d2>] ? cpuid=
le_enter_state+0x52/0xc0<br>
[1123306.014561] [<ffffffff814aa6c8>] ?
cpuidle_enter_state+=
;0x48/0xc0<br>
[1123306.014565] [<ffffffff814aa805>]
cpuidle_idle_call+0xc=
5/0x200<br>
[1123306.014569] [<ffffffff8101d21e>]
arch_cpu_idle+0xe/0x3=
0<br>
[1123306.014574] [<ffffffff810c6945>]
cpu_startup_entry+0xf=
5/0x290<br>
[1123306.014580] [<ffffffff810423ca>]
start_secondary+0x1ba=
/0x230<br>
[1123306.014582] ---[ end trace 4d5a1bc838e1fcc0 ]---<br>
<br>
If so, then could you try the following:<br>
<br>
ethtool -K <nic name> lro off<br>
<br>
Do this for all the 10G intel nics and check if the problems still exists <=
br>
<br>
<br>
</tt>
<div class=3D"moz-signature">
<title></title>
<div style=3D"color: rgb(0, 0, 0);">
<p class=3D"MsoNormal" style=3D"font-size: 14px; font-family:
Calibri, sans-serif; margin: 0cm 0cm 0.0001pt;">
<b><font color=3D"#2c8cb6"
face=3D"Arial,sans-serif"><span style=3D"font-si=
ze: 10pt;">K</span><span style=3D"font-size:
13px;">i</span><span style=3D"font-size:
10pt;">nd regards,=
</span></font></b></p>
<p class=3D"MsoNormal" style=3D"font-size: 11pt; font-family:
Calibri, sans-serif; margin: 0cm 0cm 0.0001pt;">
<b><span style=3D"font-size: 10pt; font-family: Arial, sans-serif;
color: rgb(44, 140,
182);"> </span></b></p>
<p class=3D"MsoNormal" style=3D"font-size: 14px; font-family:
Calibri, sans-serif; margin: 0cm 0cm 0.0001pt;">
<b style=3D"font-size: 11pt;"><span style=3D"font-size: 10pt;
font-family: Arial, sans-serif; color: rgb(44, 140, 182);">Ju=
rri=EBn Bloemen</span></b><b style=3D"font-size:
11pt;"><span style=3D"font=
-size: 10pt; font-family: Arial, sans-serif;
color: gray;"><br>
</span></b><font color=3D"#808080"
face=3D"Arial,sans-serif"><span style=3D=
"font-size: 10pt;"></span></font></p>
<br>
</div>
</div>
<div class=3D"moz-cite-prefix">On 17-03-16 09:49, Johan Kooijman
wrote:<br>
</div>
<blockquote cite=3D"mid:CAHvs-HWDFNrQ1uXuZdXqG9_PNZdUN+OYAUU9ZuWzavHe8y=
CoKw(a)mail.gmail.com" type=3D"cite">
<div dir=3D"ltr">Hi all,
<div><br>
</div>
<div>Since we upgraded to the latest ovirt node running 7.2, we're seeing t=
hat nodes become unavailable after a while. It's running fine, with a coupl=
e of VM's on it, untill it becomes non responsive. At that moment it doesn'=
t even respond to ICMP. It'll come
back by itself after a while, but oVirt fences the machine before that tim=
e and restarts VM's elsewhere.</div>
<div><br>
</div>
<div>Engine tells me this message:</div>
<div><br>
</div>
<div>VDSM host09 command failed: Message timeout which can be caused by com=
munication issues</div>
<div><br>
</div>
<div>Is anyone else experiencing these issues with ixgbe drivers? I'm runni=
ng on Intel X540-AT2 cards.<br clear=3D"all">
<div><br>
</div>
-- <br>
<div class=3D"gmail_signature">
<div dir=3D"ltr">Met vriendelijke groeten / With kind regards,<br>
Johan Kooijman<br>
</div>
</div>
</div>
</div>
<br>
<fieldset class=3D"mimeAttachmentHeader"></fieldset> <br>
<pre wrap=3D"">_______________________________________________
Users mailing list
<a class=3D"moz-txt-link-abbreviated"
href=3D"mailto:Users@ovirt.org">Users=
@ovirt.org</a>
<a class=3D"moz-txt-link-freetext"
href=3D"http://lists.ovirt.org/mailman/l=
istinfo/users">http://lists.ovirt.org/mailman/listinfo/users</...
</pre>
</blockquote>
<br>
This message (including any attachments) may contain information that is pr=
ivileged or confidential. If you are not the intended recipient, please not=
ify the sender and delete this email immediately from your systems and dest=
roy all copies of it. You may not,
directly or indirectly, use, disclose, distribute, print or copy this emai=
l or any part of it if you are not the intended recipient
</body>
</html>
--_000_56EBB5D93000101dmcamcnetworkscom_--
11:54 a.m.
Hi Jurrien,
I don't see anything in logs on the nodes itself. The only thing we see in
logs are in engine log - it looses connectivity to the host.
Definitely CentOS 7.1/7.2 related. Downgraded the hosts to ovirt-iso 3.5,
this resolves the issue.
On Fri, Mar 18, 2016 at 9:01 AM, Bloemen, Jurriën <
Jurrien.Bloemen(a)dmc.amcnetworks.com> wrote:
Hi Johan,
Could you check if you see the following in you dmesg or message log file?
[1123306.014288] ------------[ cut here ]------------
[1123306.014302] WARNING: at net/core/dev.c:2189
skb_warn_bad_offload+0xcd/0xda()
[1123306.014306] : caps=(0x0000000200004849, 0x0000000000000000) len=330
data_len=276 gso_size=276 gso_type=1 ip_summed=1
[1123306.014308] Modules linked in: vhost_net macvtap macvlan
ip6table_filter ip6_tables iptable_filter ip_tables ebt_arp ebtable_nat
ebtables tun scsi_transport_iscsi iTCO_wdt iTCO_vendor_support
dm_service_time intel_powerclamp coretemp intel_rapl kvm_intel kvm
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cryptd pcspkr sb_edac
edac_core i2c_i801 lpc_ich mfd_core mei_me mei wmi ioatdma shpchp
ipmi_devintf ipmi_si ipmi_msghandler acpi_power_meter acpi_pad 8021q garp
mrp bridge stp llc bonding dm_multipath xfs libcrc32c sd_mod crc_t10dif
crct10dif_common ast syscopyarea sysfillrect sysimgblt drm_kms_helper ttm
crc32c_intel igb drm ahci ixgbe i2c_algo_bit libahci libata mdio i2c_core
ptp megaraid_sas pps_core dca dm_mirror dm_region_hash dm_log dm_mod
[1123306.014360] CPU: 30 PID: 0 Comm: swapper/30 Tainted: G W
-------------- 3.10.0-229.1.2.el7.x86_64 #1
[1123306.014362] Hardware name: Supermicro SYS-2028TP-HC1TR/X10DRT-PT,
BIOS 1.1 08/03/2015
[1123306.014364] ffff881fffc439a8 5326fb90ad1041ea ffff881fffc43960
ffffffff81604afa
[1123306.014371] ffff881fffc43998 ffffffff8106e34b ffff881fcebb0500
ffff881fce88c000
[1123306.014376] 0000000000000001 0000000000000001 ffff881fcebb0500
ffff881fffc43a00
[1123306.014381] Call Trace:
[1123306.014383] <IRQ> [<ffffffff81604afa>] dump_stack+0x19/0x1b
[1123306.014396] [<ffffffff8106e34b>] warn_slowpath_common+0x6b/0xb0
[1123306.014399] [<ffffffff8106e3ec>] warn_slowpath_fmt+0x5c/0x80
[1123306.014405] [<ffffffff812db093>] ? ___ratelimit+0x93/0x100
[1123306.014409] [<ffffffff816076c3>] skb_warn_bad_offload+0xcd/0xda
[1123306.014425] [<ffffffff814fdeb9>] __skb_gso_segment+0x79/0xb0
[1123306.014429] [<ffffffff814fe1c2>] dev_hard_start_xmit+0x1a2/0x580
[1123306.014438] [<ffffffffa0168790>] ? deliver_clone+0x50/0x50 [bridge]
[1123306.014443] [<ffffffff8151df1e>] sch_direct_xmit+0xee/0x1c0
[1123306.014447] [<ffffffff814fe798>] dev_queue_xmit+0x1f8/0x4a0
[1123306.014453] [<ffffffffa016880b>] br_dev_queue_push_xmit+0x7b/0xc0
[bridge]
[1123306.014458] [<ffffffffa0168a22>] br_forward_finish+0x22/0x60 [bridge]
[1123306.014464] [<ffffffffa0168ae0>] __br_forward+0x80/0xf0 [bridge]
[1123306.014469] [<ffffffffa0168ebb>] br_forward+0x8b/0xa0 [bridge]
[1123306.014476] [<ffffffffa0169e65>] br_handle_frame_finish+0x175/0x410
[bridge]
[1123306.014481] [<ffffffffa016a275>] br_handle_frame+0x175/0x260 [bridge]
[1123306.014485] [<ffffffff814fc112>] __netif_receive_skb_core+0x282/0x870
[1123306.014490] [<ffffffff8101b589>] ? read_tsc+0x9/0x10
[1123306.014493] [<ffffffff814fc718>] __netif_receive_skb+0x18/0x60
[1123306.014497] [<ffffffff814fc7a0>] netif_receive_skb+0x40/0xd0
[1123306.014500] [<ffffffff814fd2b0>] napi_gro_receive+0x80/0xb0
[1123306.014512] [<ffffffffa00cde2c>] ixgbe_clean_rx_irq+0x7ac/0xb30
[ixgbe]
[1123306.014519] [<ffffffffa00cf07b>] ixgbe_poll+0x4bb/0x930 [ixgbe]
[1123306.014524] [<ffffffff814fcb62>] net_rx_action+0x152/0x240
[1123306.014528] [<ffffffff81077bf7>] __do_softirq+0xf7/0x290
[1123306.014533] [<ffffffff8161635c>] call_softirq+0x1c/0x30
[1123306.014539] [<ffffffff81015de5>] do_softirq+0x55/0x90
[1123306.014543] [<ffffffff81077f95>] irq_exit+0x115/0x120
[1123306.014546] [<ffffffff81616ef8>] do_IRQ+0x58/0xf0
[1123306.014551] [<ffffffff8160c0ed>] common_interrupt+0x6d/0x6d
[1123306.014553] <EOI> [<ffffffff814aa6d2>] ?
cpuidle_enter_state+0x52/0xc0
[1123306.014561] [<ffffffff814aa6c8>] ? cpuidle_enter_state+0x48/0xc0
[1123306.014565] [<ffffffff814aa805>] cpuidle_idle_call+0xc5/0x200
[1123306.014569] [<ffffffff8101d21e>] arch_cpu_idle+0xe/0x30
[1123306.014574] [<ffffffff810c6945>] cpu_startup_entry+0xf5/0x290
[1123306.014580] [<ffffffff810423ca>] start_secondary+0x1ba/0x230
[1123306.014582] ---[ end trace 4d5a1bc838e1fcc0 ]---
If so, then could you try the following:
ethtool -K <nic name> lro off
Do this for all the 10G intel nics and check if the problems still exists
*Kind regards,*
*Jurriën Bloemen*
On 17-03-16 09:49, Johan Kooijman wrote:
Hi all,
Since we upgraded to the latest ovirt node running 7.2, we're seeing that
nodes become unavailable after a while. It's running fine, with a couple of
VM's on it, untill it becomes non responsive. At that moment it doesn't
even respond to ICMP. It'll come back by itself after a while, but oVirt
fences the machine before that time and restarts VM's elsewhere.
Engine tells me this message:
VDSM host09 command failed: Message timeout which can be caused by
communication issues
Is anyone else experiencing these issues with ixgbe drivers? I'm running
on Intel X540-AT2 cards.
--
Met vriendelijke groeten / With kind regards,
Johan Kooijman
_______________________________________________
Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
This message (including any attachments) may contain information that is
privileged or confidential. If you are not the intended recipient, please
notify the sender and delete this email immediately from your systems and
destroy all copies of it. You may not, directly or indirectly, use,
disclose, distribute, print or copy this email or any part of it if you are
not the intended recipient
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Met vriendelijke groeten / With kind regards,
Johan Kooijman
4:53 p.m.
Hi Jeff,
was the issue ever resolved? Don't have permissions to view the bugzilla.
On Thu, Mar 17, 2016 at 4:34 PM, Jeff Spahr <spahrj(a)gmail.com> wrote:
I had the same issue, and I also have a support case open. They
referenced https://bugzilla.redhat.com/show_bug.cgi?id=1288237 which is
private. I didn't have any success getting that bugzilla changed to
public. We couldn't keep waiting for the issue to be fixed so we replaced
the NICs with Broadcom/Qlogic that we knew had no issues in other hosts.
On Thu, Mar 17, 2016 at 11:27 AM, Sigbjorn Lie <sigbjorn(a)nixtra.com>
wrote:
> Hi,
>
> Is this on CentOS/RHEL 7.2?
>
> Log in as root as see if you can see any messages from ixgbe about "tx
> queue hung" in dmesg. I
> currently have an open support case for RHEL7.2 and the ixgbe driver,
> where there is a driver
> issue causing the network adapter to reset continuously when there are
> network traffic.
>
>
> Regards,
> Siggi
>
>
>
> On Thu, March 17, 2016 12:52, Nir Soffer wrote:
> > On Thu, Mar 17, 2016 at 10:49 AM, Johan Kooijman <
> mail(a)johankooijman.com> wrote:
> >
> >> Hi all,
> >>
> >>
> >> Since we upgraded to the latest ovirt node running 7.2, we're seeing
> that
> >> nodes become unavailable after a while. It's running fine, with a
> couple of VM's on it, untill it
> >> becomes non responsive. At that moment it doesn't even respond to
> ICMP. It'll come back by
> >> itself after a while, but oVirt fences the machine before that time
> and restarts VM's elsewhere.
> >>
> >>
> >> Engine tells me this message:
> >>
> >>
> >> VDSM host09 command failed: Message timeout which can be caused by
> >> communication issues
> >>
> >> Is anyone else experiencing these issues with ixgbe drivers? I'm
> running on
> >> Intel X540-AT2 cards.
> >>
> >
> > We will need engine and vdsm logs to understand this issue.
> >
> >
> > Can you file a bug and attach ful logs?
> >
> >
> > Nir
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> >
> >
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Met vriendelijke groeten / With kind regards,
Johan Kooijman
10:41 p.m.
This is a multi-part message in MIME format.
--------------C1E6C0A57787BAEC7E6BD4CD
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Hi Johan,
On 07/18/2016 09:53 AM, Johan Kooijman wrote:
Hi Jeff,
was the issue ever resolved? Don't have permissions to view the bugzilla.
There are proposal patches in the bugzilla, I have requested more
information about upstream status.
As soon I have updates, I will reply here.
For now, if you have the hardware and want to give a test against our
latest upstream build jobs, links below:
ovirt-node 3.6:
http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_create-iso-el7_merged/
ovirt-node 4.0 (next):
http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.0-snapshot_build-artif...
Thanks!
On Thu, Mar 17, 2016 at 4:34 PM, Jeff Spahr <spahrj(a)gmail.com
<mailto:spahrj@gmail.com>> wrote:
I had the same issue, and I also have a support case open. They
referenced https://bugzilla.redhat.com/show_bug.cgi?id=1288237
which is private. I didn't have any success getting that bugzilla
changed to public. We couldn't keep waiting for the issue to be
fixed so we replaced the NICs with Broadcom/Qlogic that we knew
had no issues in other hosts.
On Thu, Mar 17, 2016 at 11:27 AM, Sigbjorn Lie
<sigbjorn(a)nixtra.com <mailto:sigbjorn@nixtra.com>> wrote:
Hi,
Is this on CentOS/RHEL 7.2?
Log in as root as see if you can see any messages from ixgbe
about "tx queue hung" in dmesg. I
currently have an open support case for RHEL7.2 and the ixgbe
driver, where there is a driver
issue causing the network adapter to reset continuously when
there are network traffic.
Regards,
Siggi
On Thu, March 17, 2016 12:52, Nir Soffer wrote:
> On Thu, Mar 17, 2016 at 10:49 AM, Johan Kooijman
<mail(a)johankooijman.com <mailto:mail@johankooijman.com>> wrote:
>
>> Hi all,
>>
>>
>> Since we upgraded to the latest ovirt node running 7.2,
we're seeing that
>> nodes become unavailable after a while. It's running fine,
with a couple of VM's on it, untill it
>> becomes non responsive. At that moment it doesn't even
respond to ICMP. It'll come back by
>> itself after a while, but oVirt fences the machine before
that time and restarts VM's elsewhere.
>>
>>
>> Engine tells me this message:
>>
>>
>> VDSM host09 command failed: Message timeout which can be
caused by
>> communication issues
>>
>> Is anyone else experiencing these issues with ixgbe
drivers? I'm running on
>> Intel X540-AT2 cards.
>>
>
> We will need engine and vdsm logs to understand this issue.
>
>
> Can you file a bug and attach ful logs?
>
>
> Nir
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
>
>
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users
--
Met vriendelijke groeten / With kind regards,
Johan Kooijman
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--------------C1E6C0A57787BAEC7E6BD4CD
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi Johan,<br>
</p>
<br>
<div class="moz-cite-prefix">On 07/18/2016 09:53 AM, Johan Kooijman
wrote:<br>
</div>
<blockquote
cite="mid:CAHvs-HX2=e9qR61g9vu9inVikaspxFJgR8aRZwGOpUWN0wC1rQ@mail.gmail.com"
type="cite">
<div dir="ltr">Hi Jeff,
<div><br>
</div>
<div>was the issue ever resolved? Don't have permissions to view
the bugzilla.</div>
</div>
</blockquote>
<br>
There are proposal patches in the bugzilla, I have requested more
information about upstream status.<br>
As soon I have updates, I will reply here. <br>
<br>
For now, if you have the hardware and want to give a test against
our latest upstream build jobs, links below:<br>
<br>
ovirt-node 3.6:<br>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-node_ovirt-3.6_create-iso-e...
<br>
ovirt-node 4.0 (next):<br>
<a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-4.0-snapshot_...
<br>
Thanks!<br>
<br>
<blockquote
cite="mid:CAHvs-HX2=e9qR61g9vu9inVikaspxFJgR8aRZwGOpUWN0wC1rQ@mail.gmail.com"
type="cite">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Mar 17, 2016 at 4:34 PM, Jeff
Spahr <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:spahrj@gmail.com"
target="_blank">spahrj(a)gmail.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I had the same issue, and I also have a
support case open. They referenced <a
moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1288237"
target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=...
which is private. I didn't have any success getting that
bugzilla changed to public. We couldn't keep waiting for
the issue to be fixed so we replaced the NICs with
Broadcom/Qlogic that we knew had no issues in other hosts.<br>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Mar 17, 2016 at 11:27
AM, Sigbjorn Lie <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:sigbjorn@nixtra.com"
target="_blank">sigbjorn(a)nixtra.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hi,<br>
<br>
Is this on CentOS/RHEL 7.2?<br>
<br>
Log in as root as see if you can see any messages
from ixgbe about "tx queue hung" in dmesg. I<br>
currently have an open support case for RHEL7.2
and the ixgbe driver, where there is a driver<br>
issue causing the network adapter to reset
continuously when there are network traffic.<br>
<br>
<br>
Regards,<br>
Siggi<br>
<br>
<br>
<br>
On Thu, March 17, 2016 12:52, Nir Soffer wrote:<br>
> On Thu, Mar 17, 2016 at 10:49 AM, Johan
Kooijman <<a moz-do-not-send="true"
href="mailto:mail@johankooijman.com"
target="_blank">mail(a)johankooijman.com</a>&gt;
wrote:<br>
><br>
>> Hi all,<br>
>><br>
>><br>
>> Since we upgraded to the latest ovirt
node running 7.2, we're seeing that<br>
>> nodes become unavailable after a while.
It's running fine, with a couple of VM's on it,
untill it<br>
>> becomes non responsive. At that moment it
doesn't even respond to ICMP. It'll come back by<br>
>> itself after a while, but oVirt fences
the machine before that time and restarts VM's
elsewhere.<br>
>><br>
>><br>
>> Engine tells me this message:<br>
>><br>
>><br>
>> VDSM host09 command failed: Message
timeout which can be caused by<br>
>> communication issues<br>
>><br>
>> Is anyone else experiencing these issues
with ixgbe drivers? I'm running on<br>
>> Intel X540-AT2 cards.<br>
>><br>
><br>
> We will need engine and vdsm logs to
understand this issue.<br>
><br>
><br>
> Can you file a bug and attach ful logs?<br>
><br>
><br>
> Nir<br>
>
_______________________________________________<br>
> Users mailing list<br>
> <a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
> <a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
><br>
><br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users"
rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman/listinfo/user...
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Met vriendelijke groeten / With kind
regards,<br>
Johan Kooijman<br>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext"
href="http://lists.ovirt.org/mailman/listinfo/users">http://...
</pre>
</blockquote>
<br>
</body>
</html>
--------------C1E6C0A57787BAEC7E6BD4CD--
2839
days inactive
2962
days old
8 comments
7 participants
participants (7)
-
Bloemen, Jurriën -
Douglas Schilling Landgraf -
Jeff Spahr -
Johan Kooijman -
Nir Soffer -
Piotr Kliczewski -
Sigbjorn Lie