
Hello all! I'm having quite some trouble with VMs that have a large amount of dropped packets on RX. This, plus customers complain about short dropped connections, for example one customer has a SQL server and an other serevr connecting to it, and it is randomly dropping connections. Before they moved their VM:s to us they did not have any of these issues. Does anyone have an idea of what this can be due to? And how can i fix it? It is starting to be a deal breaker for our customers on whether they will stay with us or not. I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS, would this perhaps fix the issue? The enviroment is: Huawei x6000 with 4 nodes Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX 4600. Storage via FC to a IBM FS900. Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2 Regards Magnus

Hi Magnus, do you notice any repetition there ? Does it happen completely random ? Usually to debug network issues you will need tcpdump from Guest, Host and the other side if possible.Is that an option ? Do you see in the host's tab those RX errors ? What is the output of "ip -s link" on the Guest ? Best Regards,Strahil Nikolov В четвъртък, 16 май 2019 г., 9:19:57 ч. Гринуич-4, Magnus Isaksson <magnus@vmar.se> написа: Hello all! I'm having quite some trouble with VMs that have a large amount of dropped packets on RX. This, plus customers complain about short dropped connections, for example one customer has a SQL server and an other serevr connecting to it, and it is randomly dropping connections. Before they moved their VM:s to us they did not have any of these issues. Does anyone have an idea of what this can be due to? And how can i fix it? It is starting to be a deal breaker for our customers on whether they will stay with us or not. I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS, would this perhaps fix the issue? The enviroment is: Huawei x6000 with 4 nodes Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX 4600. Storage via FC to a IBM FS900. Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2 Regards Magnus _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCFPDC...

Hello @strahil The packet drops are frequent, every time i run "ip -s link" on the guest there is new dropped packets, on the hosts it says "0" and in oVirt it says "0". I can run tcpdump on hosts and guests, but i don't know how to capture the dropped packets with tcpdump. There are no RX or TX errors anywhere, not on hosts, guests or switches. The connection drops are completely random, sometimes after a few minutes and sometimes after a couple of hours, really hard to narrow down, but this may be some errors in our customers network, they are investigating it now, so i will come back with that issue is it still persists. @Oliver I tried this, unfortunately still same result, still dropping packets. @Darell I tried increasing the RX and TX buffer on the hosts, but the guests still drop packets. I am using dual 10G, setup in Active-Backup going to two switches, but the second switch is now turned off during the testing to narrow this down. Regards Magnus

Hi Magnus, I've had a bad **virtual** network card three times in the last five years. Yes it' possible. I my case, NFS services didn't work as expected, but other services were ok. Today if this would happen again, i unplug and replug the VM nic. Like: GUI::Compute::VirtualMachines::VMname::Network Interfaces::nicN -> Edit CardStatus -> Unplugged :: OK -> Edit CardStatus -> Plugged :: OK HTH Oliver On 16.05.19 15:17, Magnus Isaksson wrote:
Hello all!
I'm having quite some trouble with VMs that have a large amount of dropped packets on RX. This, plus customers complain about short dropped connections, for example one customer has a SQL server and an other serevr connecting to it, and it is randomly dropping connections. Before they moved their VM:s to us they did not have any of these issues.
Does anyone have an idea of what this can be due to? And how can i fix it? It is starting to be a deal breaker for our customers on whether they will stay with us or not.
I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS, would this perhaps fix the issue?
The enviroment is: Huawei x6000 with 4 nodes Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX 4600. Storage via FC to a IBM FS900. Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2
Regards Magnus _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCFPDC...

Check your host for dropped packets as well. I had found that some of my older 10G cards were setting smaller buffers than they could, and using ethtool to set tx and rx buffers to their max values significantly improved things for those cards. And look at your switch to be sure it/they are not dropping packets for some reason. If you’re using dual 10g links, how do you have them configured on the host?
On May 16, 2019, at 9:38 AM, Oliver Riesener <Oliver.Riesener@hs-bremen.de> wrote:
Hi Magnus,
I've had a bad **virtual** network card three times in the last five years. Yes it' possible.
I my case, NFS services didn't work as expected, but other services were ok.
Today if this would happen again, i unplug and replug the VM nic. Like:
GUI::Compute::VirtualMachines::VMname::Network Interfaces::nicN -> Edit CardStatus -> Unplugged :: OK -> Edit CardStatus -> Plugged :: OK
HTH
Oliver
On 16.05.19 15:17, Magnus Isaksson wrote:
Hello all!
I'm having quite some trouble with VMs that have a large amount of dropped packets on RX. This, plus customers complain about short dropped connections, for example one customer has a SQL server and an other serevr connecting to it, and it is randomly dropping connections. Before they moved their VM:s to us they did not have any of these issues.
Does anyone have an idea of what this can be due to? And how can i fix it? It is starting to be a deal breaker for our customers on whether they will stay with us or not.
I was thinking of reinstalling the nodes with oVirt Node, instead of the full CentOS, would this perhaps fix the issue?
The enviroment is: Huawei x6000 with 4 nodes Each node having Intel X722 network card and connecting with 10G (fiber) to a Juniper EX 4600. Storage via FC to a IBM FS900. Each node is running a full CentOS 7.6 connecting to a Engine 4.2.8.2
Regards Magnus _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QXGQSKYBUCFPDC...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FQXYN3P2QD727Z...
participants (4)
-
Darrell Budic
-
Magnus Isaksson
-
Oliver Riesener
-
Strahil Nikolov