I must admit I’m getting a bit weary of fighting oVirt problems at this point… Before I move on to deploying any VMs onto my new infra, I’d like to get the base infra working… I’m still experiencing a “Non Operational” problem on my “ovirt-node-02” host: http://s1096.photobucket.com/user/willdennis/media/ovirt-node-02_problem.... I have pored thru the logs (all the engine logs, plus the syslogs from the engine VM + and my three hypervisor/storage hosts) and I can’t pin down why the one node is having a problem… Of course with how voluminous all these logs are, it’s kind of like looking for a needle in a haystack, and I’m not even sure what the needle looks like, or if it’s even a needle :-/ I have also rebooted this host in past days, this also did not fix the problem. Note that on the screenshot I posted above, that the webadmin hosts screen says that -node-01 has one VM running, and the others 0… You’d think that would be the HE VM running on there, but it’s actually on -node-02: $ ansible istgroup-ovirt -f 1 -i prod -u root -m shell -a "hosted-engine --vm-status | grep -e '^Hostname' -e '^Engine'" ovirt-node-01 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} ovirt-node-02 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} ovirt-node-03 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown”} So it looks like the webadmin UI is wrong as well… It would be awesome if the UI would give a reason for the “Non Operational” status somehow… Or if there was a troubleshooter that could be used to analyze the problem… As it is, being so new to all of this, I am completely at the list’s mercy to figure this out. This software has such promise, so I’ll keep working thru these issues, but it sure hasn’t been a smooth ride so far… On Jan 4, 2016, at 7:54 AM, Will Dennis <wdennis@nec-labs.com<mailto:wdennis@nec-labs.com>> wrote: I put all of the engine logs up there now… Try engine.log-20160103.gzhttp://i1096.photobucket.com/albums/g330/willdennis...

Feel like I’m in a bit of an echo chamber here… Is anyone out there? ;) Or have I worn out the oVirt crew? Anyhow, not sure if this is a cause, or an effect, but I noticed tonight that the data storage domain (which I’m using Gluster for in a hyperconverged way) is not mounted on the problem hypervisor host… $ ansible istgroup-ovirt -f 1 -i prod -u root -m shell -a "df -h | grep ':’" ovirt-node-01 | success | rc=0 >> localhost:/engine 1.9T 3.0G 1.9T 1% /rhev/data-center/mnt/glusterSD/localhost:_engine ovirt-node-01.nec-labs.com<http://ovirt-node-01.nec-labs.com>:/vmdata 3.7T 70M 3.7T 1% /rhev/data-center/mnt/glusterSD/ovirt-node-01.nec-labs.com<http://ovirt-node-01.nec-labs.com>:_vmdata ovirt-node-02 | success | rc=0 >> localhost:/engine 1.9T 3.0G 1.9T 1% /rhev/data-center/mnt/glusterSD/localhost:_engine ovirt-node-03 | success | rc=0 >> localhost:/engine 1.9T 3.0G 1.9T 1% /rhev/data-center/mnt/glusterSD/localhost:_engine ovirt-node-01.nec-labs.com<http://ovirt-node-01.nec-labs.com>:/vmdata 3.7T 70M 3.7T 1% /rhev/data-center/mnt/glusterSD/ovirt-node-01.nec-labs.com<http://ovirt-node-01.nec-labs.com>:_vmdata What causes this mount to occur, and is there a way to trigger the mount manually? On Jan 4, 2016, at 10:47 PM, Will Dennis <wdennis@nec-labs.com<mailto:wdennis@nec-labs.com>> wrote: I must admit I’m getting a bit weary of fighting oVirt problems at this point… Before I move on to deploying any VMs onto my new infra, I’d like to get the base infra working… I’m still experiencing a “Non Operational” problem on my “ovirt-node-02” host: http://s1096.photobucket.com/user/willdennis/media/ovirt-node-02_problem....

On Jan 6, 2016, at 1:39 AM, Sahina Bose <sabose@redhat.com<mailto:sabose@redhat.com>> wrote: The reason why the host is Non-operational is usually in the General sub-tab for the host. Ah, did not know that… It does say at the bottom of that pane: “Host failed to attach one of the Storage Domains attached to it.” As previously reported to the list last evening, this is true - it has not mounted the data SD (which is a Gluster SD.) Any way to troubleshoot why?

I actually had the opposite problem - I (not knowing) elected to say “Yes” (default answer) to “iptables was detected on your computer, do you wish setup to configure it?” which then put in the oVirt iptables rules, which assume the standard Gluster TCP ports… Since I am running hyperconverged and had followed the instructions found at: http://www.ovirt.org/Features/Self_Hosted_Engine_Hyper_Converged_Gluster_... which ends up changing the Gluster ports, then I experienced a fault with Gluster where it lost quorum and went read-only since the firewall on the hosts were blocking Gluster communications... On Jan 6, 2016, at 11:10 AM, Sahina Bose <sabose(a)redhat.com&gt; wrote: Also, worth checking that glusterd ports are open on the gluster hosts (we had an issue where HE install overrides glusterd ports and gluster volume was inaccessible)

Define “reinstall the host” - do you just mean 'yum remove ovirt* vdsm*’ then ‘yum install ovirt* vdsm*’, or completely reinstall the OS, reset-up Gluster, etc.? On Jan 6, 2016, at 4:15 AM, Eliraz Levi <elevi@redhat.com<mailto:elevi@redhat.com>> wrote: Hi Will how are you? The log is first pointing about certifications issues: 2016-01-04 00:02:11,259 ERROR [org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer] (DefaultQuartzScheduler_Worker-81) [] Failed to get peer certification for host 'ovirt-node-02': SSL session is invalid 2016-01-04 00:02:11,259 ERROR [org.ovirt.engine.core.bll.CertificationValidityChecker] (DefaultQuartzScheduler_Worker-81) [] Failed to retrieve peer certifications for host 'ovirt-node-02' So first thing we should do is to try and solve this problem. Please try to re install the host. Thanks. Eliraz :) ----- Original Message ----- From: "Will Dennis" <wdennis@nec-labs.com<mailto:wdennis@nec-labs.com>> To: "Eliraz Levi" <elevi@redhat.com<mailto:elevi@redhat.com>>, "users" <users@ovirt.org<mailto:users@ovirt.org>> Sent: Tuesday, 5 January, 2016 5:46:23 AM Subject: Re: [ovirt-users] host status "Non Operational" - how to diagnose & fix? I must admit I’m getting a bit weary of fighting oVirt problems at this point… Before I move on to deploying any VMs onto my new infra, I’d like to get the base infra working… I’m still experiencing a “Non Operational” problem on my “ovirt-node-02” host: http://s1096.photobucket.com/user/willdennis/media/ovirt-node-02_problem.... I have pored thru the logs (all the engine logs, plus the syslogs from the engine VM + and my three hypervisor/storage hosts) and I can’t pin down why the one node is having a problem… Of course with how voluminous all these logs are, it’s kind of like looking for a needle in a haystack, and I’m not even sure what the needle looks like, or if it’s even a needle :-/ I have also rebooted this host in past days, this also did not fix the problem. Note that on the screenshot I posted above, that the webadmin hosts screen says that -node-01 has one VM running, and the others 0… You’d think that would be the HE VM running on there, but it’s actually on -node-02: $ ansible istgroup-ovirt -f 1 -i prod -u root -m shell -a "hosted-engine --vm-status | grep -e '^Hostname' -e '^Engine'" ovirt-node-01 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} ovirt-node-02 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} ovirt-node-03 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown”} So it looks like the webadmin UI is wrong as well… It would be awesome if the UI would give a reason for the “Non Operational” status somehow… Or if there was a troubleshooter that could be used to analyze the problem… As it is, being so new to all of this, I am completely at the list’s mercy to figure this out. This software has such promise, so I’ll keep working thru these issues, but it sure hasn’t been a smooth ride so far… On Jan 4, 2016, at 7:54 AM, Will Dennis <wdennis@nec-labs.com<mailto:wdennis@nec-labs.com><mailto:wdennis@nec-labs.com>> wrote: I put all of the engine logs up there now… Try engine.log-20160103.gzhttp://i1096.photobucket.com/albums/g330/willdennis...

On Jan 6, 2016, at 7:59 AM, Moti Asayag <masayag@redhat.com<mailto:masayag@redhat.com>> wrote: In order to see the configuration of 'ovirtmgmt' network please paste the output of the following command to be executed on the host: vdsClient -s 0 getVdsCaps http://fpaste.org/307742/20853451/ In addition, in order to see the reported status of the networks run and paste on the host: vdsClient -s 0 getVdsStats http://fpaste.org/307744/45208555/

To follow up on this, after the migrations as a result of the troubleshooting, the webadmin UI of the hosts in my datacenter now has each host with “1” VM running… https://drive.google.com/file/d/0B88nnCy4LpFMYklDVDhFUV96Y00/view?usp=sha... However, The only VM that is running currently is the hosted engine, which is currently running on host “ovirt-node-03” — $ ansible istgroup-ovirt -f 1 -i prod -u root -m shell -a "hosted-engine --vm-status | grep -e '^Hostname' -e '^Engine'" ovirt-node-01 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-03 Engine status : {"health": "good", "vm": "up", "detail": "up"} ovirt-node-02 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-03 Engine status : {"health": "good", "vm": "up", "detail": "up"} ovirt-node-03 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-03 Engine status : {"health": "good", "vm": "up", "detail": "up”} Is this a UI bug of some sort? On Jan 4, 2016, at 10:47 PM, Will Dennis <wdennis@nec-labs.com<mailto:wdennis@nec-labs.com>> wrote: Note that on the screenshot I posted above, that the webadmin hosts screen says that -node-01 has one VM running, and the others 0… You’d think that would be the HE VM running on there, but it’s actually on -node-02: $ ansible istgroup-ovirt -f 1 -i prod -u root -m shell -a "hosted-engine --vm-status | grep -e '^Hostname' -e '^Engine'" ovirt-node-01 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} ovirt-node-02 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"} ovirt-node-03 | success | rc=0 >> Hostname : ovirt-node-01 Engine status : {"reason": "bad vm status", "health": "bad", "vm": "down", "detail": "down"} Hostname : ovirt-node-02 Engine status : {"health": "good", "vm": "up", "detail": "up"} Hostname : ovirt-node-03 Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown”} So it looks like the webadmin UI is wrong as well…

No, it was definitely wrong at the time - there were no other VMs other than the hosted engine, and no migrations… However, when I was able to finally re-import/activate the hosted_storage SD, and the HE VM correctly showed up in the UI, then it corrected itself, and has remained correct since… As shown in my post, it also did not agree at the time with the output of “hosted-engine —vm-status”, but since the re-import of the SD, it now is correctly in sync with that as well.