<div dir="ltr"><div>Hi Stack,<br><br></div>I read through your ordeal and I would like to post a few comments:<br><ul><li>When I try to reproduce your scenario with the second network set to 'not required' before on-boarding the second host, it is processed and set to 'up' by the engine without any hiccups or any errors in the log. <br></li><li>On the other hand, if the network is 'required' the scenario reproduces, but on my setup it can be resolved: initially the second network is proclaimed missing and the host becomes non-operational, with its interfaces disappearing from the engine as you reported. But if the second network is rendered 'not-required' or even deleted for that matter from the engine, engine succeeds in reconnecting to the second host within a couple of minutes, and the host gains 'up' status.</li></ul><p>HTH<br></p></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 17, 2018 at 11:35 PM, ~Stack~ <span dir="ltr"><<a href="mailto:i.am.stack@gmail.com" target="_blank">i.am.stack@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Greetings,<br>
<br>
After a few days of trial, error, and madness - I *think* I found the<br>
source of my problem. Or at least I can now replicate it reliably. These<br>
are the basics of my speed-run-to-test-failures setup.<br>
<br>
Fresh minimal install of Scientific Linux 7.4 on a physical host for my<br>
engine. Add the 4.2 repo and run engine-setup - just blast through the<br>
defaults. Configure it with default DC and cluster.<br>
<br>
Fresh minimal install of Scientific Linux 7.4 on node1 - configure only<br>
the primary network card. Add the ovirt repo.<br>
<br>
Add the host into cluster. Provisions just fine. Life is good.<br>
<br>
Now here is where things split.<br>
<br>
Scenario 1: build node2 same as node 1 configuring only the primary<br>
network card and add it as a host. Provisions just fine. Life is good.<br>
<br>
Scenario 2: Configure a second network. In my case a BMC/IPMI network.<br>
Doesn't matter if it is required or not - both will cause failures<br>
however the errors are slightly more evident with required. Make sure<br>
the network is assigned to your node1 and is properly assigned an IP and<br>
configured in the up state. Now build node2 same as before with only the<br>
primary network configured and add it as a host.<br>
<br>
Failure followed by infinite loop of setting it into Non-Operational!<br>
<br>
<br>
The pop-up gives you some crap about "Host has no default route." but<br>
that is 100% a red-herring.<br>
<br>
Dig a little deeper and you get a message like this:<br>
"node2 does not comply with the cluster Default networks, the following<br>
networks are missing on host: 'ovirtmgmt'"<br>
<br>
Ah. That's a bit more relevant, but why can't it configure it? Or at<br>
least get to the point where it asks me "Hey, networking is a bit off -<br>
do you want to configure that now?" That would be nice...<br>
<br>
Fortunately the troubleshooting guide has something about that!<br>
<a href="https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting/" rel="noreferrer" target="_blank">https://www.ovirt.org/<wbr>documentation/how-to/<wbr>troubleshooting/<wbr>troubleshooting/</a><br>
<br>
Unfortunately, it doesn't do anything to help. Even after doing these<br>
steps, the loop just keeps going...nothing changes.<br>
<a href="https://www.ovirt.org/develop/developer-guide/vdsm/installing-vdsm-from-rpm/" rel="noreferrer" target="_blank">https://www.ovirt.org/develop/<wbr>developer-guide/vdsm/<wbr>installing-vdsm-from-rpm/</a><br>
<br>
Scratch it all and completely rebuild AGAIN for...<br>
Scenario 3: Configure a second network (BMC) and assign it to node1 just<br>
like before. Build out node2 same as node1 but this time add in the<br>
EXACT SAME NETWORK CONFIGURATION THAT IS WORKING ON NODE1 - ALL of the<br>
ifcfg-* files (but update the IP address to correct host, obviously).<br>
Now add it as a host.<br>
<br>
Doh! Same error. :-/<br>
<br>
OK fine. Let's really get into it. First off, the networking page for<br>
the host is blank. It never pulls back the network cards so you can't<br>
actually make changes via the web page. Nor can you assign networks. So<br>
the web interface doesn't help at all.<br>
<br>
Let's look at the engine log instead.<br>
<br>
<br>
2018-04-17 14:33:00,336-05 INFO<br>
[org.ovirt.engine.core.bll.<wbr>VdsEventListener]<br>
(EE-ManagedThreadFactory-<wbr>engine-Thread-1091) []<br>
ResourceManager::<wbr>vdsNotResponding entered for Host<br>
'f0a3d515-8ba2-490e-8d65-<wbr>54edbb52cefc', '192.168.1.4'<br>
2018-04-17 14:33:00,360-05 INFO<br>
[<a href="http://org.ovirt.engine.core.bll.pm">org.ovirt.engine.core.bll.pm</a>.<wbr>VdsNotRespondingTreatmentComma<wbr>nd]<br>
(EE-ManagedThreadFactory-<wbr>engine-Thread-1091) [5291eee5] Lock Acquired to<br>
object<br>
'EngineLock:{exclusiveLocks='[<wbr>f0a3d515-8ba2-490e-8d65-<wbr>54edbb52cefc=VDS_FENCE]',<br>
sharedLocks=''}'<br>
2018-04-17 14:33:00,388-05 ERROR<br>
[org.ovirt.engine.core.bll.<wbr>SetNonOperationalVdsCommand]<br>
(EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-44) [2b853e43] Host<br>
'node2' is set to Non-Operational, it is missing the following networks:<br>
'ovirtmgmt'<br>
2018-04-17 14:33:00,403-05 WARN<br>
[org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector]<br>
(EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-44) [2b853e43] EVENT_ID:<br>
VDS_SET_NONOPERATIONAL_<wbr>NETWORK(519), Host node2 does not comply with the<br>
cluster Default networks, the following networks are missing on host:<br>
'ovirtmgmt'<br>
2018-04-17 14:33:00,407-05 INFO<br>
[<a href="http://org.ovirt.engine.core.bll.pm">org.ovirt.engine.core.bll.pm</a>.<wbr>VdsNotRespondingTreatmentComma<wbr>nd]<br>
(EE-ManagedThreadFactory-<wbr>engine-Thread-1091) [5291eee5] Running command:<br>
VdsNotRespondingTreatmentComma<wbr>nd internal: true. Entities affected :<br>
ID: f0a3d515-8ba2-490e-8d65-<wbr>54edbb52cefc Type: VDS<br>
<br>
<br>
There's the message from before. Good. On the right track. Not sure why<br>
it thinks the host is unreachable because the host is just fine.<br>
<br>
2018-04-17 14:33:01,978-05 ERROR<br>
[org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>GetAllVmStatsVDSCommand]<br>
(EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-31) [] Command<br>
'GetAllVmStatsVDSCommand(<wbr>HostName = node2,<br>
VdsIdVDSCommandParametersBase:<wbr>{hostId='f0a3d515-8ba2-490e-<wbr>8d65-54edbb52cefc'})'<br>
execution failed: <a href="http://java.net">java.net</a>.<wbr>NoRouteToHostException: No route to host<br>
<br>
Huh. Again with the no route to host. But THERE IS! The network is<br>
functioning perfectly. IP's all work. DNS all works. Routing is fine. I<br>
have no idea what it is complaining about.<br>
<br>
2018-04-17 14:33:03,873-05 INFO<br>
[org.ovirt.engine.core.<wbr>vdsbroker.<wbr>SetVdsStatusVDSCommand]<br>
(EE-ManagedThreadFactory-<wbr>engineScheduled-Thread-39) [4f72afaa] START,<br>
SetVdsStatusVDSCommand(<wbr>HostName = node2,<br>
SetVdsStatusVDSCommandParamete<wbr>rs:{hostId='f0a3d515-8ba2-<wbr>490e-8d65-54edbb52cefc',<br>
status='NonOperational', nonOperationalReason='NETWORK_<wbr>UNREACHABLE',<br>
stopSpmFailureLogged='false', maintenanceReason='null'}), log id: 7459a748<br>
<br>
Which network is unreachable? Because every single one of them is fine! Ugh!<br>
<br>
I am completely stumped as to why it works perfectly<br>
pre-additional-networks but fails every time after a network is configured.<br>
<br>
A couple of questions.<br>
<br>
1. I assume people have added hosts _after_ they've configured multiple<br>
networks. So what am I doing wrong? Why am I unable to add a host?<br>
Again, if I don't configure that second network, it will happily add all<br>
my hosts. But what happens when I want to add a host in the future?<br>
<br>
2. How do I break that infuriating infinite non-operational loop? I<br>
can't put it into maintenance mode, I can't delete the host, or anything<br>
else. The options are greyed out. The only solution I've found is yank<br>
the power and after it freaks out for about 30 minutes because it can't<br>
find the host, it will stop trying. But I still can't seem to remove the<br>
bad host. There has to be a way via command-line to say "stop timing<br>
out, knock that off, and delete this host!" but I'm not finding it in my<br>
searching.<br>
<br>
3. I feel like I go through periods with oVirt where everything is<br>
running exactly the way I want then something happens (like me trying to<br>
add a host! Or thinking I can just change a host IP without the whole<br>
thing dying on me!) and it all just falls apart. I feel like I am just<br>
stumbling through most of it. I've previously gotten a lot out of the<br>
Red Hat classes and work has offered to send me to a training of my<br>
choice this year. I am really considering taking the 318 Virtualization<br>
class. I'm curious though, how close is that to what I would be working<br>
with oVirt? I'm guessing that since 4.2 recently came out, there is<br>
probably minimal chance the class will be over 4.2 but maybe it is close<br>
enough? I would love to hear feedback.<br>
<br>
Thanks!<br>
<span class="HOEnZb"><font color="#888888">~Stack~<br>
<br>
</font></span><br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Eitan Raviv<br>IRC: erav (#ovirt #vdsm #devel #rhev-dev)<br></div></div></div></div>
</div>