[ovirt-users] Re: ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) Penalizing score by 1600 due to network status

On Mon, Jul 19, 2021 at 2:20 PM Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > On Mon, Jul 19, 2021 at 1:54 PM Christoph Timm <ovirt(a)timmi.org&gt; wrote: >> >> >> Am 19.07.21 um 10:52 schrieb Yedidyah Bar David: >>> On Mon, Jul 19, 2021 at 11:39 AM Christoph Timm <ovirt(a)timmi.org&gt; wrote: >>>> Am 19.07.21 um 10:25 schrieb Yedidyah Bar David: >>>>> On Mon, Jul 19, 2021 at 11:02 AM Christoph Timm <ovirt(a)timmi.org&gt; wrote: >>>>>> Am 19.07.21 um 09:27 schrieb Yedidyah Bar David: >>>>>>> On Mon, Jul 19, 2021 at 10:04 AM Christoph Timm <ovirt(a)timmi.org&gt; wrote: >>>>>>>> Hi Didi, >>>>>>>> >>>>>>>> thank you for the quick response. >>>>>>>> >>>>>>>> >>>>>>>> Am 19.07.21 um 07:59 schrieb Yedidyah Bar David: >>>>>>>>> On Mon, Jul 19, 2021 at 8:39 AM Christoph Timm <ovirt(a)timmi.org&gt; wrote: >>>>>>>>>> Hi List, >>>>>>>>>> >>>>>>>>>> I'm trying to understand why my hosted engine is moved from one node to >>>>>>>>>> another from time to time. >>>>>>>>>> It is happening sometime multiple times a day. But there are also days >>>>>>>>>> without it. >>>>>>>>>> >>>>>>>>>> I can see the following in the ovirt-hosted-engine-ha/agent.log: >>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(score) >>>>>>>>>> Penalizing score by 1600 due to network status >>>>>>>>>> >>>>>>>>>> After that the engine will be shutdown and started on another host. >>>>>>>>>> The oVirt Admin portal is showing the following around the same time: >>>>>>>>>> Invalid status on Data Center Default. Setting status to Non Responsive. >>>>>>>>>> >>>>>>>>>> But the whole cluster is working normally during that time. >>>>>>>>>> >>>>>>>>>> I believe that I have somehow a network issue on my side but I have no >>>>>>>>>> clue what kind of check is causing the network status to penalized. >>>>>>>>>> >>>>>>>>>> Does anyone have an idea how to investigate this further? >>>>>>>>> Please check also broker.log. Do you see 'dig' failures? >>>>>>>> Yes I found them as well. >>>>>>>> >>>>>>>> Thread-1::WARNING::2021-07-19 >>>>>>>> 08:02:00,032::network::120::network.Network::(_dns) DNS query failed: >>>>>>>> ; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> +tries=1 +time=5 >>>>>>>> ;; global options: +cmd >>>>>>>> ;; connection timed out; no servers could be reached >>>>>>>> >>>>>>>>> This happened several times already on our CI infrastructure, but yours is >>>>>>>>> the first report from an actual real user. See also: >>>>>>>>> >>>>>>>>> https://lists.ovirt.org/archives/list/infra@ovirt.org/thread/LIGS5WXGEKWA... >>>>>>>> So I understand that the following command is triggered to test the >>>>>>>> network: "dig +tries=1 +time=5" >>>>>>> Indeed. >>>>>>> >>>>>>>>> I didn't open a bug for this (yet?), also because I never reproduced on my >>>>>>>>> own machines and am not sure about the exact failing flow. If this is >>>>>>>>> reproducible >>>>>>>>> reliably for you, you might want to test the patch I pushed: >>>>>>>>> >>>>>>>>> https://gerrit.ovirt.org/c/ovirt-hosted-engine-ha/+/115596 Now filed this bug and linked to it in the above patch. Thanks for your report! https://bugzilla.redhat.com/show_bug.cgi?id=1984356

Best regards, >>>>>>>> I'm happy to give it a try. >>>>>>>> Please confirm that I need to replace this file (network.py) on all my >>>>>>>> nodes (CentOS 8.4 based) which can host my engine. >>>>>>> It definitely makes sense to do so, but in principle there is no problem >>>>>>> with applying it only on some of them. That's especially useful if you try >>>>>>> this first on a test env and try to enforce a reproduction somehow (overload >>>>>>> the network, disconnect stuff, etc.). >>>>>> OK will give it a try and report back. >>>>> Thanks and good luck. >> Do I need to restart anything after that change? > Yes, the broker. This might restart some other services there, so best put the > host to maintenance during this. > >> Also please confirm that the comma after TCP is correct as there wasn't >> one before after the timeout in row 110. > It is correct, but not mandatory. We (my team, at least) often add it > in such cases > to make a theoretical future patch that adds another parameter not > require adding > it again (thus making the patch smaller and hopefully cleaner). > >>>>>>>>> Other ideas/opinions about how to enhance this part of the monitoring >>>>>>>>> are most welcome. >>>>>>>>> >>>>>>>>> If this phenomenon is new for you, and you can reliably say it's not due to >>>>>>>>> a recent "natural" higher network load, I wonder if it's due to some weird >>>>>>>>> bug/change somewhere. >>>>>>>> I'm quite sure that I see this since we moved to 4.4.(4). >>>>>>>> Just for house keeping I'm running 4.4.7 now. >>>>>>> We use 'dig' as the network monitor since 4.3.5, around one year before 4.4 >>>>>>> was released: https://bugzilla.redhat.com/1659052 >>>>>>> >>>>>>> Which version did you use before 4.4? >>>>>> The last 4.3 versions have been 4.3.7, 4.3.9 and 4.3.10 before migrating >>>>>> to 4.4.4. >>>>> I now realize that in above-linked bug we only changed the default, for new >>>>> setups. So if you deployed He before 4.3.5, upgrade to later 4.3 would not >>>>> change the default (as opposed to upgrade to 4.4, which was actually a >>>>> new deployment with engine backup/restore). Do you know which version >>>>> your cluster was originally deployed with? >>>> Hm, I'm sorry but I don't recall this. I'm quite sure that we started >>> OK, thanks for trying. >>> >>>> with 4.0 something. But we moved to a HE setup around September 2019. >>>> But I don't recall the version. But we installed also the backup from >>>> the old installation into the HE environment if I'm not wrong. >>> If indeed this change was the trigger for you, you can rather easily try to >>> change this to 'ping' and see if this helps - I think it's enough to change >>> 'network_test' to 'ping' in /etc/ovirt-hosted-engine/hosted-engine.conf >>> and restart the broker - didn't try, though. But generally speaking, I do not >>> think we want to change the default back to 'ping', but rather make 'dns' >>> work better/well. We had valid reasons to move away from ping... >> OK I will try this if the tcp change does not help me. > Ok. > > In parallel, especially if this is reproducible, you might want to do > some general > monitoring of your network - packet losses, etc. - and correlate this with the > failures you see. > > Best regards, > -- > Didi