I've been seeing these warnings myself, on 1Gb ovirtmanagement (glusterFS is 10Gbe backend). I haven't correlated to network graphs yet but I don't know what would be happening on my management network that would be exhausting 1Gb network. On Fri, Aug 31, 2018 at 3:27 AM Florian Schmid <fschmid(a)ubimet.com&gt; wrote: > Good morning, > > since we have upgraded to version 4.2.5, we get a lot of warnings about > network interface exceeded defined threshold limits. > > For example: > Aug 31, 2018, 7:54:05 AM > Host xxx has network interface which exceeded the defined threshold [95%] > (enp9s0.80: transmit rate[100%], receive rate [12%]) > > This is a 10 Gbit interface and on our monitoring software, which is > getting network statistics every 10s, the bandwidth of TX was 150 Mbit > maximum at this time, so far away from being 100%. > > Could it be, that the engine detected the wrong interface speed or there > is a calculation error? > In the engine for this host, I have 10000 Mbps for all interfaces. > > I have checked now all those warnings on our different hosts and they > happen every time, we go over 100 Mbit and this is for sure quite often... > > Can I maybe disable these warnings, because we have it anyway in our > monitoring software? > > If you need any logs, please ask. > > BR Florian Schmid > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community- > guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ > message/2NFL3O66IN4Z6HUK45WQFXRUBMQDUY7P/ > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ message/3UPGCAKVDJDPNSODTO6PWXOD2ETT63N6/

Hello Edward, I have applied the patch and it looks very good! vdsm-client Host getStats -> ... "enp9s0.88": { "rxErrors": "0", "name": "enp9s0.88", "tx": "1226", "txDropped": "0", "sampleTime": 1536043097.701361, "rx": "98642", "txErrors": "0", "state": "up", "speed": "10000", "rxDropped": "0" }, ... Bridge devices still have only 1000 configured: "vm-int-dev": { "rxErrors": "0", "name": "vm-int-dev", "tx": "578", "txDropped": "0", "sampleTime": 1536043097.701361, "rx": "27843284", "txErrors": "0", "state": "up", "speed": "1000", "rxDropped": "0" }, One important question: I want to apply this patch without upgrading all hosts, because this is a huge task. When I apply that patch only to this particular file, which service do I need to restart? I have restarted now all three vdsm services, but I think, I can't do that while VMs are running on the hosts, do I? LG Florian ------------------------------ *Von: *"Florian Schmid" <fschmid(a)ubimet.com&gt; *An: *"edwardh" <edwardh(a)redhat.com&gt; *CC: *"users" <users(a)ovirt.org&gt; *Gesendet: *Dienstag, 4. September 2018 08:32:09 *Betreff: *[ovirt-users] Re: Wrong network threshold limit warnings on 4.2.5 Hello Edward, raised a bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1625098 I will try your patch. LG Florian ------------------------------ *Von: *"Edward Haas" <ehaas(a)redhat.com&gt; *An: *"p staniforth" <P.Staniforth(a)leedsbeckett.ac.uk&gt;, "Florian Schmid" < fschmid(a)ubimet.com&gt; *CC: *"users" <users(a)ovirt.org&gt; *Gesendet: *Montag, 3. September 2018 16:42:25 *Betreff: *Re: Wrong network threshold limit warnings on 4.2.5 Indeed looks like a nasty bug. Could you please open a bug on this? https://tinyurl.com/ya7crjhf If you can, could you also verify the fix? https://gerrit.ovirt.org/#/c/ 94132/ Thanks, Edy. On Mon, Sep 3, 2018 at 2:32 PM, Staniforth, Paul < P.Staniforth(a)leedsbeckett.ac.uk&gt; wrote: > Hello Edward, > > I am also seeing this problem, it's on our > ovirtmgmt. > > > cat /sys/class/net/eno49/speed > 10000 > > > > cat /sys/class/net/eno49.20/speed > 10000 > > > > cat /sys/class/net/ovirtmgmt/speed > cat: /sys/class/net/ovirtmgmt/speed: Invalid argument > > > vdsm-client Host getStats -> > ... > > "eno49": { > "rxErrors": "0", > "name": "eno49", > "tx": "3456777", > "txDropped": "0", > "sampleTime": 1535974190.687987, > "rx": "121362321", > "txErrors": "0", > "state": "up", > "speed": "10000", > "rxDropped": "2" > }, > > "eno49.20": { > "rxErrors": "0", > "name": "eno49.20", > "tx": "3384452", > "txDropped": "0", > "sampleTime": 1535974190.687987, > "rx": "115884579", > "txErrors": "0", > "state": "up", > "speed": "1000", > "rxDropped": "0" > }, > > "ovirtmgmt": { > "rxErrors": "0", > "name": "ovirtmgmt", > "tx": "3383804", > "txDropped": "0", > "sampleTime": 1535974190.687987, > "rx": "115710919", > "txErrors": "0", > "state": "up", > "speed": "1000", > "rxDropped": "0" > }, > > > Regards, > > Paul S. > > ------------------------------ > *From:* Florian Schmid <fschmid(a)ubimet.com&gt; > *Sent:* 03 September 2018 11:44 > *To:* edwardh(a)redhat.com > *Cc:* users > *Subject:* [ovirt-users] Re: Wrong network threshold limit warnings on > 4.2.5 > > Hi Edward, > > I got some alarms today from a server and I have checked your command > there. (not at the time the issue happened!!) > Hosts are on latest patch level CentOS 7.5 and oVirt 4.2.5 > > Example: > cat /sys/class/net/enp9s0/speed > 10000 > > cat /sys/class/net/enp9s0.80/speed > 10000 > > cat /sys/class/net/vm-int-nfs/speed > cat: /sys/class/net/vm-int-nfs/speed: invalid argument <- this is > the bridge for the VMs > > vdsm-client Host getStats -> > ... > "enp9s0": { > "rxErrors": "0", > "name": "enp9s0", > "tx": "3335325754762", > "txDropped": "0", > "sampleTime": 1535970960.602359, > "rx": "5916567956502", > "txErrors": "0", > "state": "up", > "speed": "10000", > "rxDropped": "0" > }, > ... > "enp9s0.80": { > "rxErrors": "0", > "name": "enp9s0.80", > "tx": "3180024039398", > "txDropped": "0", > "sampleTime": 1535970960.602359, > "rx": "5669421065686", > "txErrors": "0", > "state": "up", > "speed": "1000", > "rxDropped": "0" > }, > ... > "vm-int-nfs": { > "rxErrors": "0", > "name": "vm-int-nfs", > "tx": "508", > "txDropped": "0", > "sampleTime": 1535970960.602359, > "rx": "4428568", > "txErrors": "0", > "state": "up", > "speed": "1000", > "rxDropped": "0" > }, > ... > > As you see here, vdsm is reporting the wrong speed for the vlan devices. > > BR Florian Schmid > > > ------------------------------ > *Von: *"Edward Haas" <ehaas(a)redhat.com&gt; > *An: *"Jayme" <jaymef(a)gmail.com&gt;, "Florian Schmid" <fschmid(a)ubimet.com&gt; > *CC: *"users" <users(a)ovirt.org&gt;, "Alona Kaplan" <alkaplan(a)redhat.com&gt; > *Gesendet: *Montag, 3. September 2018 11:38:25 > *Betreff: *Re: [ovirt-users] Re: Wrong network threshold limit warnings > on 4.2.5 > > If you manage to recreate this, please collect a few samples from what > the hypervisor reports back: > Run the command: vdsm-client Host getStats > > Engine is calculating based on this information the rate. > (and the agent collects it from /sys/class/net/<device>/statistics/) > > Please also mention on what OS you are running the hosts. > > Thanks, > Edy > > > On Fri, Aug 31, 2018 at 5:35 PM, Jayme <jaymef(a)gmail.com&gt; wrote: > >> I've been seeing these warnings myself, on 1Gb ovirtmanagement >> (glusterFS is 10Gbe backend). I haven't correlated to network graphs yet >> but I don't know what would be happening on my management network that >> would be exhausting 1Gb network. >> >> On Fri, Aug 31, 2018 at 3:27 AM Florian Schmid <fschmid(a)ubimet.com&gt; >> wrote: >> >>> Good morning, >>> >>> since we have upgraded to version 4.2.5, we get a lot of warnings about >>> network interface exceeded defined threshold limits. >>> >>> For example: >>> Aug 31, 2018, 7:54:05 AM >>> Host xxx has network interface which exceeded the defined threshold >>> [95%] (enp9s0.80: transmit rate[100%], receive rate [12%]) >>> >>> This is a 10 Gbit interface and on our monitoring software, which is >>> getting network statistics every 10s, the bandwidth of TX was 150 Mbit >>> maximum at this time, so far away from being 100%. >>> >>> Could it be, that the engine detected the wrong interface speed or >>> there is a calculation error? >>> In the engine for this host, I have 10000 Mbps for all interfaces. >>> >>> I have checked now all those warnings on our different hosts and they >>> happen every time, we go over 100 Mbit and this is for sure quite often... >>> >>> Can I maybe disable these warnings, because we have it anyway in our >>> monitoring software? >>> >>> If you need any logs, please ask. >>> >>> BR Florian Schmid >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >>> guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ >>> message/2NFL3O66IN4Z6HUK45WQFXRUBMQDUY7P/ >>> >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >> guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ >> message/3UPGCAKVDJDPNSODTO6PWXOD2ETT63N6/ >> >> > To view the terms under which this email is distributed, please go to:- > http://disclaimer.leedsbeckett.ac.uk/disclaimer/disclaimer.html > > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community- guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ message/O2JKN5QYZUOV4T7RBSGSBTBSU6V6HQ62/

On Tue, Sep 4, 2018 at 9:02 AM Edward Haas <ehaas(a)redhat.com&gt; wrote: > Hello Florian, > > Thanks for checking the patch and posting the bug. > > You need to restart vdsmd and supervdsmd. > It should not affect running VM/s, but you always have a risk that > something unexpected can happen. Perhaps try it on a host and then proceed > with others. > > Thanks, > Edy. > I'm having similar problem in a 3 hosts oVirt test cluster with these notifications every day on 1Gbit adapters. I have bond0 on em1 and em2 and then bondo.65, bond0.68, bond0.167 vlans defined for the VMs I get these warnings Message:Host ov300 has network interface which exceeded the defined threshold [95%] (em1: transmit rate[98%], receive rate [0%]) when actually I think the 3 VMs running on this host generate few MB/s of traffic I applied the changes to the 3 hosts. I notice that due to dependencies it is sufficient to restart supervdsmd and then also vdsmd will be automatically restarted, correct? In my case for each of the 3 hosts, after restarting supervdsmd I got messages like these, but without impacts on runnign VMs VDSM ov300 command GetStatsAsyncVDS failed: Broken pipe 9/4/18 9:07:52 AM Host ov300 is not responding. It will stay in Connecting state for a grace period of 61 seconds and after that an attempt to fence the host will be issued. 9/4/18 9:07:52 AM No faulty multipath paths on host ov300 9/4/18 9:07:58 AM Executing power management status on Host ov300 using Proxy Host ov200 and Fence Agent ipmilan:10.10.193.103. 9/4/18 9:07:58 AM Status of host ov300 was set to Up. 9/4/18 9:07:58 AM Host ov300 power management was verified successfully. 9/4/18 9:07:58 AM Please note that when doing on SPM host you could also get these: VDSM ov301 command SpmStatusVDS failed: Broken pipe 9/4/18 9:10:00 AM Host ov301 is not responding. It will stay in Connecting state for a grace period of 81 seconds and after that an attempt to fence the host will be issued. 9/4/18 9:10:00 AM Invalid status on Data Center MYDC. Setting Data Center status to Non Responsive (On host ov301, Error: Network error during communication with the Host.). 9/4/18 9:10:00 AM with reassignment of SPM role: VDSM command GetStoragePoolInfoVDS failed: Heartbeat exceeded 9/4/18 9:10:12 AM Storage Pool Manager runs on Host ov200 (Address: ov200), Data Center MYDC. 9/4/18 9:10:14 AM Probably safer to manually move the SPM before restarting supervdsmd on that host. Let's see this evening if I will get any message about thresholds. BTW: one question. I see in the code iface.Type.NIC and now also iface.Type.BOND. Don't you think that you should manage also the network teaming option available in RH EL 7, as described here: https://access.redhat.com/documentation/en-us/red_hat_ enterprise_linux/7/html/networking_guide/ch-configure_network_teaming ? This only if it is supported to use the new network teaming implementation in oVirt, and I'm not sure about it...

Thanks, Gianluca