11:17 p.m.
Hi all, I have a question about Windows VMs and a networking issue I'm
having.
Here's the setup -
* oVirt - 3.5.1.1-1
* Hypervisors are CentOS 6.7 box with 2 NICs in bond0 'mode=4 miimon=100
lacp_rate=1'
* On bond0 I have a few networks using vlan tagging.
* Networks are 5,10,15,20 - All on an external switch
Network 15 has a Windows 2012 R2 server and a CentOS 6.7 server on it.
The rest of the networks have a few linux.
Every linux box on every network is happy. However any and all Windows
boxes I bring online are incapable of patching or hitting the web. I
pointed the Windows box to the linux box next to it as a proxy (after
installing squid on it) When I do that the Windows box has no issues at
all; it's only when he's attempting to leave on his own.
On my firewall I put in a 'permit any any' on the M$ box IP however all I
see is tcp resets in PCAPs,
I've been playing with for some time but can't seem to find the issue. It
would be one thing if everything on the 15 was bad but the linux box on the
network is fine. Here's the rub, I'm 99.999% sure this used to work.
gggrrr...
Any assistance anyone can offer would be amazingly appreciated.
Thank you for taking the time to read this.
11:30 a.m.
On Tue, Nov 24, 2015 at 11:17 PM, Matt Wells <matt.wells(a)mosaic451.com>
wrote:
Hi all, I have a question about Windows VMs and a networking issue
I'm
having.
Here's the setup -
* oVirt - 3.5.1.1-1
* Hypervisors are CentOS 6.7 box with 2 NICs in bond0 'mode=4 miimon=100
lacp_rate=1'
* On bond0 I have a few networks using vlan tagging.
* Networks are 5,10,15,20 - All on an external switch
Network 15 has a Windows 2012 R2 server and a CentOS 6.7 server on it.
The rest of the networks have a few linux.
Every linux box on every network is happy. However any and all Windows
boxes I bring online are incapable of patching or hitting the web. I
pointed the Windows box to the linux box next to it as a proxy (after
installing squid on it) When I do that the Windows box has no issues at
all; it's only when he's attempting to leave on his own.
On my firewall I put in a 'permit any any' on the M$ box IP however all I
see is tcp resets in PCAPs,
Can you verify basic IP networking is working correctly for those VMs?
For example, we've established that they can get to the Linux VMs - how?
Are they on the same subnet? Or do they go through their default gateway?
Without knowing the IP topology, if the Linux machines were on the same
subnet as the Windows one, and the Windows machine fail to get to their
default gateway for some reason, this may perfectly explain the issue.
Y.
I've been playing with for some time but can't seem to find the issue. It
would be one thing if everything on the 15 was bad but the linux box on the
network is fine. Here's the rub, I'm 99.999% sure this used to work.
gggrrr...
Any assistance anyone can offer would be amazingly appreciated.
Thank you for taking the time to read this.
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
4:49 p.m.
All networking within oVirt seems fine. I can ping devices on other
networks, I'm able to use a squid proxy on the same network with no
issues. I'm able to use Terminal Services from another VM in oVirt. One
oddity is when I first boot, I'm able to hit Google and search for a Dog.
On my 2nd search it starts to timeout.
All and all ip networking seems to be configured well. The DG lives
outside of oVirt and I'm always able to ping it; even increasing the packet
size.
MTU size is 1500 and matches the VLAN MTU in oVirt; the bond0 is set to set
to an 9000 MTU.
EM1 and EM2 are in bond0 and show no framing errors.
On Wed, Nov 25, 2015 at 1:30 AM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Tue, Nov 24, 2015 at 11:17 PM, Matt Wells
<matt.wells(a)mosaic451.com>
wrote:
> Hi all, I have a question about Windows VMs and a networking issue I'm
> having.
>
> Here's the setup -
> * oVirt - 3.5.1.1-1
> * Hypervisors are CentOS 6.7 box with 2 NICs in bond0 'mode=4 miimon=100
> lacp_rate=1'
> * On bond0 I have a few networks using vlan tagging.
> * Networks are 5,10,15,20 - All on an external switch
>
> Network 15 has a Windows 2012 R2 server and a CentOS 6.7 server on it.
> The rest of the networks have a few linux.
>
> Every linux box on every network is happy. However any and all Windows
> boxes I bring online are incapable of patching or hitting the web. I
> pointed the Windows box to the linux box next to it as a proxy (after
> installing squid on it) When I do that the Windows box has no issues at
> all; it's only when he's attempting to leave on his own.
>
> On my firewall I put in a 'permit any any' on the M$ box IP however all I
> see is tcp resets in PCAPs,
>
Can you verify basic IP networking is working correctly for those VMs?
For example, we've established that they can get to the Linux VMs - how?
Are they on the same subnet? Or do they go through their default gateway?
Without knowing the IP topology, if the Linux machines were on the same
subnet as the Windows one, and the Windows machine fail to get to their
default gateway for some reason, this may perfectly explain the issue.
Y.
>
> I've been playing with for some time but can't seem to find the issue.
> It would be one thing if everything on the 15 was bad but the linux box on
> the network is fine. Here's the rub, I'm 99.999% sure this used to work.
> gggrrr...
>
> Any assistance anyone can offer would be amazingly appreciated.
>
> Thank you for taking the time to read this.
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
4:31 p.m.
I wanted to follow up on this after I found my resolution.
I started to see kernel errors when I migrated all but my windows host off
a hypervisor and generated traffic. I then took those errors and started
looking back at all of the hypervisors only to find this error was on each
of them; also it actively reporting on systems with Windows VMs. Tracing
back the logs lead me to the bug reports below where I learned that this
issue had been re-introduced to the 2.6.32-X kernel.
ABRT info -
Nov 30 19:28:42 server2.example.com kernel: WARNING: at net/core/dev.c:1915
skb_warn_bad_offload+0x99/0xb0() (Tainted: G W -- ------------ )
Nov 30 19:28:42 server2.example.com kernel: Hardware name: PowerEdge M620
Nov 30 19:28:42 server2.example.com kernel: : caps=(0x40c9, 0x0) len=1514
data_len=1460 ip_summed=1
Nov 30 19:28:42 server2.example.com kernel: Modules linked in: sch_prio
act_mirred cls_u32 sch_ingress ebt_arp xt_physdev ipt_REJECT
nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport iptable_filter ip_tables fuse
nfs lockd fscache auth_rpcgss nfs_acl sunrpc vfat fat bonding ebtable_nat
ebtables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i iw_cxgb4 cxgb4
cxgb3i libcxgbi iw_cxgb3 cxgb3 ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad
ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu
autofs4 bridge 8021q garp stp llc ip6t_REJECT nf_conntrack_ipv6
nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6
dm_round_robin dm_multipath vhost_net macvtap macvlan tun kvm_intel kvm sg
ipmi_devintf sr_mod cdrom joydev power_meter acpi_ipmi ipmi_si
ipmi_msghandler iTCO_wdt iTCO_vendor_support ixgbe dca ptp pps_core mdio
dcdbas sb_edac edac_core lpc_ich mfd_core shpchp usb_storage ext4 jbd2
mbcache sd_mod crc_t10dif megaraid_sas wmi ahci dm_mirror dm_region_hash
dm_log dm_mod [last unloaded: ip_tables]
Nov 30 19:28:42 server2.example.com kernel: Pid: 0, comm: swapper Tainted:
G W -- ------------ 2.6.32-573.7.1.el6.x86_64 #1
Nov 30 19:28:42 server2.example.com kernel: Call Trace:
Nov 30 19:28:42 server2.example.com kernel: <IRQ> [<ffffffff81077461>] ?
warn_slowpath_common+0x91/0xe0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81077566>] ?
warn_slowpath_fmt+0x46/0x60
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81296a55>] ?
__ratelimit+0xd5/0x120
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146b2e9>] ?
skb_warn_bad_offload+0x99/0xb0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146f9b1>] ?
__skb_gso_segment+0x71/0xc0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146fa13>] ?
skb_gso_segment+0x13/0x20
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146fabb>] ?
dev_hard_start_xmit+0x9b/0x490
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8148d17a>] ?
sch_direct_xmit+0x15a/0x1c0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81470158>] ?
dev_queue_xmit+0x228/0x320
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034d8b0>] ?
__br_forward+0x0/0xd0 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034d818>] ?
br_dev_queue_push_xmit+0x88/0xc0 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034d8a8>] ?
br_forward_finish+0x58/0x60 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034d95a>] ?
__br_forward+0xaa/0xd0 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8145f788>] ?
skb_clone+0x58/0xb0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034d4de>] ?
deliver_clone+0x3e/0x60 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034d9c1>] ?
br_forward+0x41/0x70 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034e87e>] ?
br_handle_frame_finish+0x17e/0x330 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034ebf0>] ?
br_handle_frame+0x1c0/0x270 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa034ea30>] ?
br_handle_frame+0x0/0x270 [bridge]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146aaf7>] ?
__netif_receive_skb+0x1c7/0x570
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146e3d8>] ?
netif_receive_skb+0x58/0x60
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146e4e0>] ?
napi_skb_finish+0x50/0x70
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8146eaf9>] ?
napi_gro_receive_gr+0x39/0x50
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81512aeb>] ?
vlan_gro_receive+0x1b/0x30
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa0185e25>] ?
ixgbe_clean_rx_irq+0x995/0xc70 [ixgbe]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffffa018663a>] ?
ixgbe_poll+0x40a/0x760 [ixgbe]
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81470443>] ?
net_rx_action+0x103/0x2f0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff810ad4bd>] ?
ktime_get+0x6d/0x100
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8107ffa1>] ?
__do_softirq+0xc1/0x1e0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff810ed920>] ?
handle_IRQ_event+0x60/0x170
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8100c38c>] ?
call_softirq+0x1c/0x30
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8100fbd5>] ?
do_softirq+0x65/0xa0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8107fe55>] ?
irq_exit+0x85/0x90
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff815426d5>] ?
do_IRQ+0x75/0xf0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8100ba53>] ?
ret_from_intr+0x0/0x11
Nov 30 19:28:42 server2.example.com kernel: <EOI> [<ffffffff812f109e>] ?
intel_idle+0xfe/0x1b0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff812f1081>] ?
intel_idle+0xe1/0x1b0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8143376a>] ?
cpuidle_idle_call+0x7a/0xe0
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81009fe6>] ?
cpu_idle+0xb6/0x110
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff8151f21a>] ?
rest_init+0x7a/0x80
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81c38122>] ?
start_kernel+0x424/0x431
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81c3733a>] ?
x86_64_start_reservations+0x125/0x129
Nov 30 19:28:42 server2.example.com kernel: [<ffffffff81c37453>] ?
x86_64_start_kernel+0x115/0x124
Nov 30 19:28:42 server2.example.com kernel: ---[ end trace 38b6983a05ecf4b4
]---
I was able to resolve this by re-configuring the network cards em1/em2 with
lro = off.
ethtool to disable lro -
ethtool -K em1 lro off; ethtool -K em2 lro off
Bug reports -
1270892 - https://bugzilla.redhat.com/show_bug.cgi?id=1270892
772317 - https://bugzilla.redhat.com/show_bug.cgi?id=772317
3279
days inactive
3286
days old
3 comments
2 participants
participants (2)
-
Matt Wells -
Yaniv Kaul