8:17 a.m.
Hello all!
I'm having quite some trouble with VMs that have a large amount of dropped packets on
RX.
This, plus customers complain about short dropped connections, for example one customer
has a SQL server and an other serevr connecting to it, and it is randomly dropping
connections. Before they moved their VM:s to us they did not have any of these issues.
Does anyone have an idea of what this can be due to? And how can i fix it? It is starting
to be a deal breaker for our customers on whether they will stay with us or not.
I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS,
would this perhaps fix the issue?
The enviroment is:
Huawei x6000 with 4 nodes
Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX
4600. Storage via FC to a IBM FS900.
Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2
Regards
Magnus
8:25 a.m.
Hi Magnus,
do you notice any repetition there ? Does it happen completely random ?
Usually to debug network issues you will need tcpdump from Guest, Host and the other side
if possible.Is that an option ?
Do you see in the host's tab those RX errors ?
What is the output of "ip -s link" on the Guest ?
Best Regards,Strahil Nikolov
В четвъртък, 16 май 2019 г., 9:19:57 ч. Гринуич-4, Magnus Isaksson
<magnus(a)vmar.se> написа:
Hello all!
I'm having quite some trouble with VMs that have a large amount of dropped packets on
RX.
This, plus customers complain about short dropped connections, for example one customer
has a SQL server and an other serevr connecting to it, and it is randomly dropping
connections. Before they moved their VM:s to us they did not have any of these issues.
Does anyone have an idea of what this can be due to? And how can i fix it? It is starting
to be a deal breaker for our customers on whether they will stay with us or not.
I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS,
would this perhaps fix the issue?
The enviroment is:
Huawei x6000 with 4 nodes
Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX
4600. Storage via FC to a IBM FS900.
Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2
Regards
Magnus
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCF...
12:09 p.m.
Hello
@strahil
The packet drops are frequent, every time i run "ip -s link" on the guest there
is new dropped packets, on the hosts it says "0" and in oVirt it says
"0".
I can run tcpdump on hosts and guests, but i don't know how to capture the dropped
packets with tcpdump.
There are no RX or TX errors anywhere, not on hosts, guests or switches.
The connection drops are completely random, sometimes after a few minutes and sometimes
after a couple of hours, really hard to narrow down, but this may be some errors in our
customers network, they are investigating it now, so i will come back with that issue is
it still persists.
@Oliver
I tried this, unfortunately still same result, still dropping packets.
@Darell
I tried increasing the RX and TX buffer on the hosts, but the guests still drop packets.
I am using dual 10G, setup in Active-Backup going to two switches, but the second switch
is now turned off during the testing to narrow this down.
Regards
Magnus
9:38 a.m.
Hi Magnus,
I've had a bad **virtual** network card three times in the last five
years. Yes it' possible.
I my case, NFS services didn't work as expected, but other services were ok.
Today if this would happen again, i unplug and replug the VM nic. Like:
GUI::Compute::VirtualMachines::VMname::Network Interfaces::nicN
-> Edit CardStatus -> Unplugged :: OK
-> Edit CardStatus -> Plugged :: OK
HTH
Oliver
On 16.05.19 15:17, Magnus Isaksson wrote:
> Hello all!
>
> I'm having quite some trouble with VMs that have a large amount of dropped
packets on RX.
> This, plus customers complain about short dropped connections, for example one
customer has a SQL server and an other serevr connecting to it, and it is randomly
dropping connections. Before they moved their VM:s to us they did not have any of these
issues.
>
> Does anyone have an idea of what this can be due to? And how can i fix it? It is
starting to be a deal breaker for our customers on whether they will stay with us or not.
>
> I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS,
would this perhaps fix the issue?
>
> The enviroment is:
> Huawei x6000 with 4 nodes
> Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper
EX 4600. Storage via FC to a IBM FS900.
> Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2
>
> Regards
> Magnus
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCF...
10:13 a.m.
Check your host for dropped packets as well. I had found that some of my older 10G cards
were setting smaller buffers than they could, and using ethtool to set tx and rx buffers
to their max values significantly improved things for those cards. And look at your switch
to be sure it/they are not dropping packets for some reason.
If you’re using dual 10g links, how do you have them configured on the host?
On May 16, 2019, at 9:38 AM, Oliver Riesener
<Oliver.Riesener(a)hs-bremen.de> wrote:
Hi Magnus,
I've had a bad **virtual** network card three times in the last five years. Yes
it' possible.
I my case, NFS services didn't work as expected, but other services were ok.
Today if this would happen again, i unplug and replug the VM nic. Like:
GUI::Compute::VirtualMachines::VMname::Network Interfaces::nicN
-> Edit CardStatus -> Unplugged :: OK
-> Edit CardStatus -> Plugged :: OK
HTH
Oliver
On 16.05.19 15:17, Magnus Isaksson wrote:
> Hello all!
>
> I'm having quite some trouble with VMs that have a large amount of dropped
packets on RX.
> This, plus customers complain about short dropped connections, for example one
customer has a SQL server and an other serevr connecting to it, and it is randomly
dropping connections. Before they moved their VM:s to us they did not have any of these
issues.
>
> Does anyone have an idea of what this can be due to? And how can i fix it? It is
starting to be a deal breaker for our customers on whether they will stay with us or not.
>
> I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS,
would this perhaps fix the issue?
>
> The enviroment is:
> Huawei x6000 with 4 nodes
> Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper
EX 4600. Storage via FC to a IBM FS900.
> Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2
>
> Regards
> Magnus
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCF...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/FQXYN3P2QD7...
2047
days inactive
2047
days old
4 comments
4 participants
participants (4)
-
Darrell Budic -
Magnus Isaksson -
Oliver Riesener -
Strahil Nikolov