hosted-engine migration problems on 3.6, 3.6.1, 3.6.2
I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs. Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem. The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance. I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor. I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help.
Can you please attach logs? Yaniv Dary Technical Product Manager Red Hat Israel Ltd. 34 Jerusalem Road Building A, 4th floor Ra'anana, Israel 4350109 Tel : +972 (9) 7692306 8272306 Email: ydary@redhat.com IRC : ydary On Jan 28, 2016 06:09, "Peter" <laniman1@laniman.net> wrote:
I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs.
Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem.
The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance.
I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see
MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor.
I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Adding Martin On Thu, Jan 28, 2016 at 5:54 AM, Peter <laniman1@laniman.net> wrote:
I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs.
Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem.
The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance.
I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see
MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor.
I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi, we really need more logs. Preferably the full agent, vdsm and engine log around the time the migration is attempted. The warn/error messages are all related to the fact that hosted engine runs in the "3.5 mode" with the storage domain not being imported to the engine yet. -- Martin Sivak oVirt / SLA On Thu, Jan 28, 2016 at 9:28 AM, Nir Soffer <nsoffer@redhat.com> wrote:
Adding Martin
On Thu, Jan 28, 2016 at 5:54 AM, Peter <laniman1@laniman.net> wrote:
I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs.
Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem.
The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance.
I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see
MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor.
I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Martin, The current state is HE on node2 and all other VMs on node1. Node2 is in local maintenance as of last night and HE should have migrated but can't. The requested logs for this situation are at ftp://aftp.fsl.noaa.gov/divisions/its/peter/ovirt-logs-201601028a/. I will try shutting down HE on node2 and see if it will run on node1. Last night it would not. Thanks. Peter On 01/28/2016 03:14 AM, Martin Sivak wrote:
Hi,
we really need more logs. Preferably the full agent, vdsm and engine log around the time the migration is attempted.
The warn/error messages are all related to the fact that hosted engine runs in the "3.5 mode" with the storage domain not being imported to the engine yet.
-- Martin Sivak oVirt / SLA
On Thu, Jan 28, 2016 at 9:28 AM, Nir Soffer <nsoffer@redhat.com> wrote:
Adding Martin
On Thu, Jan 28, 2016 at 5:54 AM, Peter <laniman1@laniman.net> wrote:
I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs.
Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem.
The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance.
I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see
MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor.
I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I cannot get the HE to run on node1. I did 'hosted-engine -vm-shutdown' on node2. After it was down I did 'hosted-engine --vm-start' on node1 but did not even get a qemu process. I did 'hosted-engine --vm-poweroff' on node1 and 'hosted-engine --vm-start' on node2 and got it up and running again. How do I get the HE storage domain imported to the engine to fix all of this? Is it a database thing? Thanks, Peter On 01/28/2016 11:02 AM, Peter wrote:
Martin,
The current state is HE on node2 and all other VMs on node1. Node2 is in local maintenance as of last night and HE should have migrated but can't. The requested logs for this situation are at ftp://aftp.fsl.noaa.gov/divisions/its/peter/ovirt-logs-201601028a/.
I will try shutting down HE on node2 and see if it will run on node1. Last night it would not.
Thanks. Peter
On 01/28/2016 03:14 AM, Martin Sivak wrote:
Hi,
we really need more logs. Preferably the full agent, vdsm and engine log around the time the migration is attempted.
The warn/error messages are all related to the fact that hosted engine runs in the "3.5 mode" with the storage domain not being imported to the engine yet.
-- Martin Sivak oVirt / SLA
On Thu, Jan 28, 2016 at 9:28 AM, Nir Soffer <nsoffer@redhat.com> wrote:
Adding Martin
On Thu, Jan 28, 2016 at 5:54 AM, Peter <laniman1@laniman.net> wrote:
I am running oVirt 3.6.2 (original install was 3.6) hosted-engine on a couple of Centos 7.2 servers with SAS attached storage using the new FC support to connect to the LUNs.
Neither the hosted-engine storage nor the hosted-engine VM show up in the GUI. I know there have been a lot of bugs that have been fixed for this but I still can't import anything. This may be the basis of my problem.
The problem is that I can't reliably get the hosted-engine to migrate to the other head. If all VMs and HE are running on node1 and I try to put it in maintenance the VMs will migrate but the HE will not and node1 will show 1 VM remaining on the node (even though the HE VM doesn't show up in the VM list) and will not go into maintenance.
I try setting 'hosted-engine --set-maintenance --mode=local' but that doesn't always help. The output of 'hosted-engine --vm-status' eventually changes to show the local node in maintenance with score 0 but often it doesn't move. In /var/log/ovirt-hosted-engine-ha/agent.log I see
MainThread::INFO::2016-01-28 03:15:42,951::config::205::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::WARNING::2016-01-28 03:15:43,296::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE MainThread::ERROR::2016-01-28 03:15:43,297::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf MainThread::WARNING::2016-01-28 03:15:43,316::hosted_engine::767::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor_if_possible) The VM is running locally or we have no data, keeping the domain monitor.
I've restarted servers and/or processes but I'm not getting anywhere. I appreciate any help.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (4)
-
Martin Sivak -
Nir Soffer -
Peter -
Yaniv Dary