Re: [ovirt-users] hosted engine setup on second host fails
On Sep 23, 2014 7:03 PM, Stefan Wendler <stefan.wendler@tngtech.com> wrote:
On 09/23/2014 17:01, Itamar Heim wrote:
On 09/23/2014 05:17 PM, Stefan Wendler wrote:
On 09/22/2014 10:52, Stefan Wendler wrote:
On 09/19/2014 15:58, Itamar Heim wrote:
On 09/19/2014 03:32 PM, Stefan Wendler wrote:
Hi there.
I'm trying to install a hosted-engine on our second node (fist engine runs on node1).
But I always get the message:
[ ERROR ] Cannot automatically add the host to the Default cluster: Cannot add Host. Host with the same address already exists.
I'm not entirely sure what I have to do when this message comes, so I just press ENTER:
####### To continue make a selection from the options below: (1) Continue setup - engine installation is complete (2) Power off and restart the VM (3) Abort setup
(1, 2, 3)[1]: ########
Is there any other interaction required prior to selecting "1"?
In the Web Gui I get the following message:
X Adding new Host hosted_engine_2 to Cluster Default
Here is the console output:
# hosted-engine --deploy [ INFO ] Stage: Initializing Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards. Are you sure you want to continue? (Yes, No)[Yes]: [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20140919141012-k2lag6.log
Version: otopi-1.2.3 (otopi-1.2.3-1.el6) [ INFO ] Hardware supports virtualization [ INFO ] Bridge ovirtmgmt already created [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization
--== STORAGE CONFIGURATION ==--
During customization use CTRL-D to abort. Please specify the storage you would like to use (nfs3, nfs4)[nfs3]: Please specify the full shared storage connection path to use (example: host:/path): <some address>:/volume1 The specified storage location already contains a data domain. Is this an additional host setup (Yes, No)[Yes]? [ INFO ] Installing on additional host Please specify the Host ID [Must be integer, default: 2]: The Host ID is already known. Is this a re-deployment on an additional host that was previously set up (Yes, No)[Yes]?
--== SYSTEM CONFIGURATION ==--
[WARNING] A configuration file must be supplied to deploy Hosted Engine on an additional host. The answer file may be fetched from the first host using scp. If you do not want to download it automatically you can abort the setup answering no to the following question. Do you want to scp the answer file from the first host? (Yes, No)[Yes]: Please provide the FQDN or IP of the first host: <node1.domain> Enter 'root' user password for host <node1.domain>: [ INFO ] Answer file successfully downloaded
--== NETWORK CONFIGURATION ==--
The following CPU types are supported by this host: - model_Westmere: Intel Westmere Family - model_Nehalem: Intel Nehalem Family - model_Penryn: Intel Penryn Family - model_Conroe: Intel Conroe Family
--== HOSTED ENGINE CONFIGURATION ==--
Enter the name which will be used to identify this host inside the Administrator Portal [hosted_engine_2]: Enter 'admin@internal' user password that will be used for accessing the Administrator Portal: Confirm 'admin@internal' user password: [ INFO ] Stage: Setup validation
--== CONFIGURATION PREVIEW ==--
Engine FQDN : engine.domain Bridge name : ovirtmgmt SSH daemon port : 22 Gateway address : <some address> Host name for web application : hosted_engine_2 Host ID : 2 Image size GB : 25 Storage connection : <some address>:/volume1 Console type : vnc Memory size MB : 8192 MAC address : 00:16:3e:3b:8d:66 Boot type : disk Number of CPUs : 2 CPU Type : model_Westmere
Please confirm installation settings (Yes, No)[No]: yes [ ERROR ] Invalid value
Please confirm installation settings (Yes, No)[No]: Yes [ INFO ] Stage: Transaction setup [ INFO ] Stage: Misc configuration [ INFO ] Stage: Package installation [ INFO ] Stage: Misc configuration [ INFO ] Configuring libvirt [ INFO ] Configuring VDSM [ INFO ] Starting vdsmd [ INFO ] Waiting for VDSM hardware info [ INFO ] Waiting for VDSM hardware info [ INFO ] Connecting Storage Domain [ INFO ] Configuring VM [ INFO ] Updating hosted-engine configuration [ INFO ] Stage: Transaction commit [ INFO ] Stage: Closing up To continue make a selection from the options below: (1) Continue setup - engine installation is complete (2) Power off and restart the VM (3) Abort setup
(1, 2, 3)[1]: [ INFO ] Engine replied: DB Up!Welcome to Health Status! [ ERROR ] Cannot automatically add the host to the Default cluster: Cannot add Host. Host with the same address already exists. [ INFO ] Waiting for the host to become operational in the engine. This may take several minutes... [ INFO ] Still waiting for VDSM host to become operational... [ INFO ] Still waiting for VDSM host to become operational...
And the stacktrace from the logs:
2014-09-19 14:14:07 DEBUG otopi.plugins.ovirt_hosted_engine_setup.engine.add_host add_host._closeup:438 Cannot add the host to the Default cluster Traceback (most recent call last): File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/engine/add_host.py",
line 432, in _closeup override_iptables=True, File "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/brokers.py", line 9779, in add headers={"Expect":expect, "Correlation-Id":correlation_id} File "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", line 88, in add return self.request('POST', url, body, headers) File "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", line 118, in request persistent_auth=self._persistent_auth) File "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", line 140, in __doRequest persistent_auth=persistent_auth File "/usr/lib/python2.6/site-packages/ovirtsdk/web/connection.py", line 134, in doRequest raise RequestError, response RequestError: status: 409 reason: Conflict detail: Cannot add Host. Host with the same address already exists. 2014-09-19 14:14:07 ERROR otopi.plugins.ovirt_hosted_engine_setup.engine.add_host add_host._closeup:445 Cannot automatically add the host to the Default cluster: Cannot add Host. Host with the same address already exists.
Cheers, Stefan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
what type of hardware is this? check the host uuid is not the same on both please. http://lists.ovirt.org/pipermail/users/2014-September/027278.html
Hi,
this is Supermicro Hardware. But the uuids differ. Anyway, I regenerated the uuid on node2. But I first have to fix another problem. After a reboot the first engine on node1 doesn't start because it says the storage path cannot be found. arrgh
I will investigate this and report my findings.
Cheers, Stefan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well everything is up and running again but I still have the same problem. I did a uuidgen > /etc/vdsm/vdsm.id on ovirt node 2 and 3 even though the uuids were different already.
What does ovirt mean by "Host with the same address already exists"? What HOST is it referring to?
Greetings, Stefan
oh, not same uuid, rather same host name (like ip address or dns name used to access it) it - is the engine or the ovirt-node? The nodes have definitely different ip addresses and dns entries. This is verified.
If it is the engine then I might have a problem understanding some things I Basically want one engine that runs on each node, if needed.
Cheers
The hosts. Didi/Sandro?
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Stefan Wendler" <stefan.wendler@tngtech.com> Cc: "Yedidyah Bar David" <ybardavi@redhat.com>, users@ovirt.org Sent: Tuesday, September 23, 2014 7:07:12 PM Subject: Re: [ovirt-users] hosted engine setup on second host fails
On Sep 23, 2014 7:03 PM, Stefan Wendler <stefan.wendler@tngtech.com> wrote:
On 09/23/2014 17:01, Itamar Heim wrote:
On 09/23/2014 05:17 PM, Stefan Wendler wrote:
On 09/22/2014 10:52, Stefan Wendler wrote:
On 09/19/2014 15:58, Itamar Heim wrote:
On 09/19/2014 03:32 PM, Stefan Wendler wrote: > Hi there. > > I'm trying to install a hosted-engine on our second node (fist engine > runs on node1). > > But I always get the message: > > [ ERROR ] Cannot automatically add the host to the Default cluster: > Cannot add Host. Host with the same address already exists. > > I'm not entirely sure what I have to do when this message comes, so I > just press ENTER: > > ####### > To continue make a selection from the options below: > (1) Continue setup - engine installation is complete > (2) Power off and restart the VM > (3) Abort setup > > (1, 2, 3)[1]: > ######## > > Is there any other interaction required prior to selecting "1"? > > In the Web Gui I get the following message: > > X Adding new Host hosted_engine_2 to Cluster Default > > Here is the console output: > > # hosted-engine --deploy > [ INFO ] Stage: Initializing > Continuing will configure this host for serving as > hypervisor > and create a VM where you have to install oVirt Engine afterwards. > Are you sure you want to continue? (Yes, No)[Yes]: > [ INFO ] Generating a temporary VNC password. > [ INFO ] Stage: Environment setup > Configuration files: [] > Log file: > /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20140919141012-k2lag6.log > > > Version: otopi-1.2.3 (otopi-1.2.3-1.el6) > [ INFO ] Hardware supports virtualization > [ INFO ] Bridge ovirtmgmt already created > [ INFO ] Stage: Environment packages setup > [ INFO ] Stage: Programs detection > [ INFO ] Stage: Environment setup > [ INFO ] Stage: Environment customization > > --== STORAGE CONFIGURATION ==-- > > During customization use CTRL-D to abort. > Please specify the storage you would like to use (nfs3, > nfs4)[nfs3]: > Please specify the full shared storage connection path > to use > (example: host:/path): <some address>:/volume1 > The specified storage location already contains a data > domain. > Is this an additional host setup (Yes, No)[Yes]? > [ INFO ] Installing on additional host > Please specify the Host ID [Must be integer, default: 2]: > The Host ID is already known. Is this a re-deployment > on an > additional host that was previously set up (Yes, No)[Yes]?
I admit I never tried that. Not sure how exactly it's supposed to work.
> > --== SYSTEM CONFIGURATION ==-- > > [WARNING] A configuration file must be supplied to deploy Hosted > Engine > on an additional host. > The answer file may be fetched from the first host > using scp. > If you do not want to download it automatically you can > abort > the setup answering no to the following question. > Do you want to scp the answer file from the first host? > (Yes, > No)[Yes]: > Please provide the FQDN or IP of the first host: > <node1.domain> > Enter 'root' user password for host <node1.domain>: > [ INFO ] Answer file successfully downloaded > > --== NETWORK CONFIGURATION ==-- > > The following CPU types are supported by this host: > - model_Westmere: Intel Westmere Family > - model_Nehalem: Intel Nehalem Family > - model_Penryn: Intel Penryn Family > - model_Conroe: Intel Conroe Family > > --== HOSTED ENGINE CONFIGURATION ==-- > > Enter the name which will be used to identify this host > inside > the Administrator Portal [hosted_engine_2]: > Enter 'admin@internal' user password that will be used > for > accessing the Administrator Portal: > Confirm 'admin@internal' user password: > [ INFO ] Stage: Setup validation > > --== CONFIGURATION PREVIEW ==-- > > Engine FQDN : engine.domain > Bridge name : ovirtmgmt > SSH daemon port : 22 > Gateway address : <some address> > Host name for web application : hosted_engine_2 > Host ID : 2 > Image size GB : 25 > Storage connection : <some > address>:/volume1 > Console type : vnc > Memory size MB : 8192 > MAC address : 00:16:3e:3b:8d:66 > Boot type : disk > Number of CPUs : 2 > CPU Type : model_Westmere > > Please confirm installation settings (Yes, No)[No]: yes > [ ERROR ] Invalid value > > Please confirm installation settings (Yes, No)[No]: Yes > [ INFO ] Stage: Transaction setup > [ INFO ] Stage: Misc configuration > [ INFO ] Stage: Package installation > [ INFO ] Stage: Misc configuration > [ INFO ] Configuring libvirt > [ INFO ] Configuring VDSM > [ INFO ] Starting vdsmd > [ INFO ] Waiting for VDSM hardware info > [ INFO ] Waiting for VDSM hardware info > [ INFO ] Connecting Storage Domain > [ INFO ] Configuring VM > [ INFO ] Updating hosted-engine configuration > [ INFO ] Stage: Transaction commit > [ INFO ] Stage: Closing up > To continue make a selection from the options below: > (1) Continue setup - engine installation is complete > (2) Power off and restart the VM > (3) Abort setup > > (1, 2, 3)[1]: > [ INFO ] Engine replied: DB Up!Welcome to Health Status! > [ ERROR ] Cannot automatically add the host to the Default cluster: > Cannot add Host. Host with the same address already exists. > [ INFO ] Waiting for the host to become operational in the engine. > This > may take several minutes... > [ INFO ] Still waiting for VDSM host to become operational... > [ INFO ] Still waiting for VDSM host to become operational... > > And the stacktrace from the logs: > > 2014-09-19 14:14:07 DEBUG > otopi.plugins.ovirt_hosted_engine_setup.engine.add_host > add_host._closeup:438 Cannot add the host to the Default cluster > Traceback (most recent call last): > File > "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/engine/add_host.py", > > > line 432, in _closeup > override_iptables=True, > File > "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/brokers.py", > line 9779, in add > headers={"Expect":expect, "Correlation-Id":correlation_id} > File > "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", > line 88, in add > return self.request('POST', url, body, headers) > File > "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", > line 118, in request > persistent_auth=self._persistent_auth) > File > "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", > line 140, in __doRequest > persistent_auth=persistent_auth > File > "/usr/lib/python2.6/site-packages/ovirtsdk/web/connection.py", > line 134, in doRequest > raise RequestError, response > RequestError: > status: 409 > reason: Conflict > detail: Cannot add Host. Host with the same address already exists. > 2014-09-19 14:14:07 ERROR > otopi.plugins.ovirt_hosted_engine_setup.engine.add_host > add_host._closeup:445 Cannot automatically add the host to the > Default > cluster: > Cannot add Host. Host with the same address already exists. > > > Cheers, > Stefan > > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
what type of hardware is this? check the host uuid is not the same on both please. http://lists.ovirt.org/pipermail/users/2014-September/027278.html
Hi,
this is Supermicro Hardware. But the uuids differ. Anyway, I regenerated the uuid on node2. But I first have to fix another problem. After a reboot the first engine on node1 doesn't start because it says the storage path cannot be found. arrgh
I will investigate this and report my findings.
Cheers, Stefan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well everything is up and running again but I still have the same problem. I did a uuidgen > /etc/vdsm/vdsm.id on ovirt node 2 and 3 even though the uuids were different already.
What does ovirt mean by "Host with the same address already exists"? What HOST is it referring to?
Greetings, Stefan
oh, not same uuid, rather same host name (like ip address or dns name used to access it) it - is the engine or the ovirt-node? The nodes have definitely different ip addresses and dns entries. This is verified.
If it is the engine then I might have a problem understanding some things I Basically want one engine that runs on each node, if needed.
Cheers
The hosts. Didi/Sandro?
It will probably be simplest to remove the second (and third?) host from then engine, then try again. -- Didi
----- Original Message -----
From: "Yedidyah Bar David" <didi@redhat.com> To: "Itamar Heim" <iheim@redhat.com> Cc: "Stefan Wendler" <stefan.wendler@tngtech.com>, users@ovirt.org Sent: Wednesday, September 24, 2014 8:40:58 AM Subject: Re: [ovirt-users] hosted engine setup on second host fails
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Stefan Wendler" <stefan.wendler@tngtech.com> Cc: "Yedidyah Bar David" <ybardavi@redhat.com>, users@ovirt.org Sent: Tuesday, September 23, 2014 7:07:12 PM Subject: Re: [ovirt-users] hosted engine setup on second host fails
On Sep 23, 2014 7:03 PM, Stefan Wendler <stefan.wendler@tngtech.com> wrote:
On 09/23/2014 17:01, Itamar Heim wrote:
On 09/23/2014 05:17 PM, Stefan Wendler wrote:
On 09/22/2014 10:52, Stefan Wendler wrote:
On 09/19/2014 15:58, Itamar Heim wrote: > On 09/19/2014 03:32 PM, Stefan Wendler wrote: >> Hi there. >> >> I'm trying to install a hosted-engine on our second node (fist >> engine >> runs on node1). >> >> But I always get the message: >> >> [ ERROR ] Cannot automatically add the host to the Default cluster: >> Cannot add Host. Host with the same address already exists. >> >> I'm not entirely sure what I have to do when this message comes, so >> I >> just press ENTER: >> >> ####### >> To continue make a selection from the options below: >> (1) Continue setup - engine installation is complete >> (2) Power off and restart the VM >> (3) Abort setup >> >> (1, 2, 3)[1]: >> ######## >> >> Is there any other interaction required prior to selecting "1"? >> >> In the Web Gui I get the following message: >> >> X Adding new Host hosted_engine_2 to Cluster Default >> >> Here is the console output: >> >> # hosted-engine --deploy >> [ INFO ] Stage: Initializing >> Continuing will configure this host for serving as >> hypervisor >> and create a VM where you have to install oVirt Engine afterwards. >> Are you sure you want to continue? (Yes, No)[Yes]: >> [ INFO ] Generating a temporary VNC password. >> [ INFO ] Stage: Environment setup >> Configuration files: [] >> Log file: >> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20140919141012-k2lag6.log >> >> >> Version: otopi-1.2.3 (otopi-1.2.3-1.el6) >> [ INFO ] Hardware supports virtualization >> [ INFO ] Bridge ovirtmgmt already created >> [ INFO ] Stage: Environment packages setup >> [ INFO ] Stage: Programs detection >> [ INFO ] Stage: Environment setup >> [ INFO ] Stage: Environment customization >> >> --== STORAGE CONFIGURATION ==-- >> >> During customization use CTRL-D to abort. >> Please specify the storage you would like to use (nfs3, >> nfs4)[nfs3]: >> Please specify the full shared storage connection path >> to use >> (example: host:/path): <some address>:/volume1 >> The specified storage location already contains a data >> domain. >> Is this an additional host setup (Yes, No)[Yes]? >> [ INFO ] Installing on additional host >> Please specify the Host ID [Must be integer, default: >> 2]: >> The Host ID is already known. Is this a re-deployment >> on an >> additional host that was previously set up (Yes, No)[Yes]?
I admit I never tried that. Not sure how exactly it's supposed to work.
A bit more details: Normally, a host is registered only in the engine's database. A hosted engine is additionally registered in a special hosted-engine metadata file managed by the ha daemon [1]. The question above appears if the host id is found in this metadata file. It seems we never check if it's already in the engine database - the assumption is that if an existing host is re-purposed as a hosted-engine, it should first be "uninstalled" - at least not be in use (no VMs) and removed from its cluster/dc/the engine. [1] http://www.ovirt.org/images/d/d5/Fosdem-hosted-engine.pdf pages 17-18
>> >> --== SYSTEM CONFIGURATION ==-- >> >> [WARNING] A configuration file must be supplied to deploy Hosted >> Engine >> on an additional host. >> The answer file may be fetched from the first host >> using scp. >> If you do not want to download it automatically you can >> abort >> the setup answering no to the following question. >> Do you want to scp the answer file from the first host? >> (Yes, >> No)[Yes]: >> Please provide the FQDN or IP of the first host: >> <node1.domain> >> Enter 'root' user password for host <node1.domain>: >> [ INFO ] Answer file successfully downloaded >> >> --== NETWORK CONFIGURATION ==-- >> >> The following CPU types are supported by this host: >> - model_Westmere: Intel Westmere Family >> - model_Nehalem: Intel Nehalem Family >> - model_Penryn: Intel Penryn Family >> - model_Conroe: Intel Conroe Family >> >> --== HOSTED ENGINE CONFIGURATION ==-- >> >> Enter the name which will be used to identify this host >> inside >> the Administrator Portal [hosted_engine_2]: >> Enter 'admin@internal' user password that will be used >> for >> accessing the Administrator Portal: >> Confirm 'admin@internal' user password: >> [ INFO ] Stage: Setup validation >> >> --== CONFIGURATION PREVIEW ==-- >> >> Engine FQDN : engine.domain >> Bridge name : ovirtmgmt >> SSH daemon port : 22 >> Gateway address : <some address> >> Host name for web application : hosted_engine_2 >> Host ID : 2 >> Image size GB : 25 >> Storage connection : <some >> address>:/volume1 >> Console type : vnc >> Memory size MB : 8192 >> MAC address : 00:16:3e:3b:8d:66 >> Boot type : disk >> Number of CPUs : 2 >> CPU Type : model_Westmere >> >> Please confirm installation settings (Yes, No)[No]: yes >> [ ERROR ] Invalid value >> >> Please confirm installation settings (Yes, No)[No]: Yes >> [ INFO ] Stage: Transaction setup >> [ INFO ] Stage: Misc configuration >> [ INFO ] Stage: Package installation >> [ INFO ] Stage: Misc configuration >> [ INFO ] Configuring libvirt >> [ INFO ] Configuring VDSM >> [ INFO ] Starting vdsmd >> [ INFO ] Waiting for VDSM hardware info >> [ INFO ] Waiting for VDSM hardware info >> [ INFO ] Connecting Storage Domain >> [ INFO ] Configuring VM >> [ INFO ] Updating hosted-engine configuration >> [ INFO ] Stage: Transaction commit >> [ INFO ] Stage: Closing up >> To continue make a selection from the options below: >> (1) Continue setup - engine installation is complete >> (2) Power off and restart the VM >> (3) Abort setup >> >> (1, 2, 3)[1]: >> [ INFO ] Engine replied: DB Up!Welcome to Health Status! >> [ ERROR ] Cannot automatically add the host to the Default cluster: >> Cannot add Host. Host with the same address already exists. >> [ INFO ] Waiting for the host to become operational in the engine. >> This >> may take several minutes... >> [ INFO ] Still waiting for VDSM host to become operational... >> [ INFO ] Still waiting for VDSM host to become operational... >> >> And the stacktrace from the logs: >> >> 2014-09-19 14:14:07 DEBUG >> otopi.plugins.ovirt_hosted_engine_setup.engine.add_host >> add_host._closeup:438 Cannot add the host to the Default cluster >> Traceback (most recent call last): >> File >> "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/engine/add_host.py", >> >> >> line 432, in _closeup >> override_iptables=True, >> File >> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/brokers.py", >> line 9779, in add >> headers={"Expect":expect, "Correlation-Id":correlation_id} >> File >> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", >> line 88, in add >> return self.request('POST', url, body, headers) >> File >> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", >> line 118, in request >> persistent_auth=self._persistent_auth) >> File >> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", >> line 140, in __doRequest >> persistent_auth=persistent_auth >> File >> "/usr/lib/python2.6/site-packages/ovirtsdk/web/connection.py", >> line 134, in doRequest >> raise RequestError, response >> RequestError: >> status: 409 >> reason: Conflict >> detail: Cannot add Host. Host with the same address already exists. >> 2014-09-19 14:14:07 ERROR >> otopi.plugins.ovirt_hosted_engine_setup.engine.add_host >> add_host._closeup:445 Cannot automatically add the host to the >> Default >> cluster: >> Cannot add Host. Host with the same address already exists. >> >> >> Cheers, >> Stefan >> >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > > > what type of hardware is this? check the host uuid is not the same > on > both please. > http://lists.ovirt.org/pipermail/users/2014-September/027278.html
Hi,
this is Supermicro Hardware. But the uuids differ. Anyway, I regenerated the uuid on node2. But I first have to fix another problem. After a reboot the first engine on node1 doesn't start because it says the storage path cannot be found. arrgh
I will investigate this and report my findings.
Cheers, Stefan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Well everything is up and running again but I still have the same problem. I did a uuidgen > /etc/vdsm/vdsm.id on ovirt node 2 and 3 even though the uuids were different already.
What does ovirt mean by "Host with the same address already exists"? What HOST is it referring to?
Greetings, Stefan
oh, not same uuid, rather same host name (like ip address or dns name used to access it) it - is the engine or the ovirt-node? The nodes have definitely different ip addresses and dns entries. This is verified.
If it is the engine then I might have a problem understanding some things I Basically want one engine that runs on each node, if needed.
Cheers
The hosts. Didi/Sandro?
It will probably be simplest to remove the second (and third?) host from then engine, then try again. -- Didi
-- Didi
Hi, it's getting a little too long, so please forgive the top post. The engine emits the message "Host with the same address already exists." only if you trying to add host with the same hostname it doesn't have any connection to it's ID, so please check if your hosts have unique hostnames (e.g. I ran into this when I didn't get hostname from dhcp and both of my hosts were localhost.localdomain). Regards, Jirka On 09/24/2014 07:59 AM, Yedidyah Bar David wrote:
----- Original Message -----
From: "Yedidyah Bar David" <didi@redhat.com> To: "Itamar Heim" <iheim@redhat.com> Cc: "Stefan Wendler" <stefan.wendler@tngtech.com>, users@ovirt.org Sent: Wednesday, September 24, 2014 8:40:58 AM Subject: Re: [ovirt-users] hosted engine setup on second host fails
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Stefan Wendler" <stefan.wendler@tngtech.com> Cc: "Yedidyah Bar David" <ybardavi@redhat.com>, users@ovirt.org Sent: Tuesday, September 23, 2014 7:07:12 PM Subject: Re: [ovirt-users] hosted engine setup on second host fails
On Sep 23, 2014 7:03 PM, Stefan Wendler <stefan.wendler@tngtech.com> wrote:
On 09/23/2014 17:01, Itamar Heim wrote:
On 09/23/2014 05:17 PM, Stefan Wendler wrote:
On 09/22/2014 10:52, Stefan Wendler wrote: > On 09/19/2014 15:58, Itamar Heim wrote: >> On 09/19/2014 03:32 PM, Stefan Wendler wrote: >>> Hi there. >>> >>> I'm trying to install a hosted-engine on our second node (fist >>> engine >>> runs on node1). >>> >>> But I always get the message: >>> >>> [ ERROR ] Cannot automatically add the host to the Default cluster: >>> Cannot add Host. Host with the same address already exists. >>> >>> I'm not entirely sure what I have to do when this message comes, so >>> I >>> just press ENTER: >>> >>> ####### >>> To continue make a selection from the options below: >>> (1) Continue setup - engine installation is complete >>> (2) Power off and restart the VM >>> (3) Abort setup >>> >>> (1, 2, 3)[1]: >>> ######## >>> >>> Is there any other interaction required prior to selecting "1"? >>> >>> In the Web Gui I get the following message: >>> >>> X Adding new Host hosted_engine_2 to Cluster Default >>> >>> Here is the console output: >>> >>> # hosted-engine --deploy >>> [ INFO ] Stage: Initializing >>> Continuing will configure this host for serving as >>> hypervisor >>> and create a VM where you have to install oVirt Engine afterwards. >>> Are you sure you want to continue? (Yes, No)[Yes]: >>> [ INFO ] Generating a temporary VNC password. >>> [ INFO ] Stage: Environment setup >>> Configuration files: [] >>> Log file: >>> /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20140919141012-k2lag6.log >>> >>> >>> Version: otopi-1.2.3 (otopi-1.2.3-1.el6) >>> [ INFO ] Hardware supports virtualization >>> [ INFO ] Bridge ovirtmgmt already created >>> [ INFO ] Stage: Environment packages setup >>> [ INFO ] Stage: Programs detection >>> [ INFO ] Stage: Environment setup >>> [ INFO ] Stage: Environment customization >>> >>> --== STORAGE CONFIGURATION ==-- >>> >>> During customization use CTRL-D to abort. >>> Please specify the storage you would like to use (nfs3, >>> nfs4)[nfs3]: >>> Please specify the full shared storage connection path >>> to use >>> (example: host:/path): <some address>:/volume1 >>> The specified storage location already contains a data >>> domain. >>> Is this an additional host setup (Yes, No)[Yes]? >>> [ INFO ] Installing on additional host >>> Please specify the Host ID [Must be integer, default: >>> 2]: >>> The Host ID is already known. Is this a re-deployment >>> on an >>> additional host that was previously set up (Yes, No)[Yes]?
I admit I never tried that. Not sure how exactly it's supposed to work.
A bit more details:
Normally, a host is registered only in the engine's database. A hosted engine is additionally registered in a special hosted-engine metadata file managed by the ha daemon [1]. The question above appears if the host id is found in this metadata file. It seems we never check if it's already in the engine database - the assumption is that if an existing host is re-purposed as a hosted-engine, it should first be "uninstalled" - at least not be in use (no VMs) and removed from its cluster/dc/the engine.
[1] http://www.ovirt.org/images/d/d5/Fosdem-hosted-engine.pdf pages 17-18
>>> >>> --== SYSTEM CONFIGURATION ==-- >>> >>> [WARNING] A configuration file must be supplied to deploy Hosted >>> Engine >>> on an additional host. >>> The answer file may be fetched from the first host >>> using scp. >>> If you do not want to download it automatically you can >>> abort >>> the setup answering no to the following question. >>> Do you want to scp the answer file from the first host? >>> (Yes, >>> No)[Yes]: >>> Please provide the FQDN or IP of the first host: >>> <node1.domain> >>> Enter 'root' user password for host <node1.domain>: >>> [ INFO ] Answer file successfully downloaded >>> >>> --== NETWORK CONFIGURATION ==-- >>> >>> The following CPU types are supported by this host: >>> - model_Westmere: Intel Westmere Family >>> - model_Nehalem: Intel Nehalem Family >>> - model_Penryn: Intel Penryn Family >>> - model_Conroe: Intel Conroe Family >>> >>> --== HOSTED ENGINE CONFIGURATION ==-- >>> >>> Enter the name which will be used to identify this host >>> inside >>> the Administrator Portal [hosted_engine_2]: >>> Enter 'admin@internal' user password that will be used >>> for >>> accessing the Administrator Portal: >>> Confirm 'admin@internal' user password: >>> [ INFO ] Stage: Setup validation >>> >>> --== CONFIGURATION PREVIEW ==-- >>> >>> Engine FQDN : engine.domain >>> Bridge name : ovirtmgmt >>> SSH daemon port : 22 >>> Gateway address : <some address> >>> Host name for web application : hosted_engine_2 >>> Host ID : 2 >>> Image size GB : 25 >>> Storage connection : <some >>> address>:/volume1 >>> Console type : vnc >>> Memory size MB : 8192 >>> MAC address : 00:16:3e:3b:8d:66 >>> Boot type : disk >>> Number of CPUs : 2 >>> CPU Type : model_Westmere >>> >>> Please confirm installation settings (Yes, No)[No]: yes >>> [ ERROR ] Invalid value >>> >>> Please confirm installation settings (Yes, No)[No]: Yes >>> [ INFO ] Stage: Transaction setup >>> [ INFO ] Stage: Misc configuration >>> [ INFO ] Stage: Package installation >>> [ INFO ] Stage: Misc configuration >>> [ INFO ] Configuring libvirt >>> [ INFO ] Configuring VDSM >>> [ INFO ] Starting vdsmd >>> [ INFO ] Waiting for VDSM hardware info >>> [ INFO ] Waiting for VDSM hardware info >>> [ INFO ] Connecting Storage Domain >>> [ INFO ] Configuring VM >>> [ INFO ] Updating hosted-engine configuration >>> [ INFO ] Stage: Transaction commit >>> [ INFO ] Stage: Closing up >>> To continue make a selection from the options below: >>> (1) Continue setup - engine installation is complete >>> (2) Power off and restart the VM >>> (3) Abort setup >>> >>> (1, 2, 3)[1]: >>> [ INFO ] Engine replied: DB Up!Welcome to Health Status! >>> [ ERROR ] Cannot automatically add the host to the Default cluster: >>> Cannot add Host. Host with the same address already exists. >>> [ INFO ] Waiting for the host to become operational in the engine. >>> This >>> may take several minutes... >>> [ INFO ] Still waiting for VDSM host to become operational... >>> [ INFO ] Still waiting for VDSM host to become operational... >>> >>> And the stacktrace from the logs: >>> >>> 2014-09-19 14:14:07 DEBUG >>> otopi.plugins.ovirt_hosted_engine_setup.engine.add_host >>> add_host._closeup:438 Cannot add the host to the Default cluster >>> Traceback (most recent call last): >>> File >>> "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/engine/add_host.py", >>> >>> >>> line 432, in _closeup >>> override_iptables=True, >>> File >>> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/brokers.py", >>> line 9779, in add >>> headers={"Expect":expect, "Correlation-Id":correlation_id} >>> File >>> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", >>> line 88, in add >>> return self.request('POST', url, body, headers) >>> File >>> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", >>> line 118, in request >>> persistent_auth=self._persistent_auth) >>> File >>> "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", >>> line 140, in __doRequest >>> persistent_auth=persistent_auth >>> File >>> "/usr/lib/python2.6/site-packages/ovirtsdk/web/connection.py", >>> line 134, in doRequest >>> raise RequestError, response >>> RequestError: >>> status: 409 >>> reason: Conflict >>> detail: Cannot add Host. Host with the same address already exists. >>> 2014-09-19 14:14:07 ERROR >>> otopi.plugins.ovirt_hosted_engine_setup.engine.add_host >>> add_host._closeup:445 Cannot automatically add the host to the >>> Default >>> cluster: >>> Cannot add Host. Host with the same address already exists. >>> >>> >>> Cheers, >>> Stefan >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >> >> >> what type of hardware is this? check the host uuid is not the same >> on >> both please. >> http://lists.ovirt.org/pipermail/users/2014-September/027278.html > > Hi, > > this is Supermicro Hardware. But the uuids differ. Anyway, I > regenerated > the uuid on node2. But I first have to fix another problem. After a > reboot the first engine on node1 doesn't start because it says the > storage path cannot be found. arrgh > > I will investigate this and report my findings. > > Cheers, > Stefan > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
Well everything is up and running again but I still have the same problem. I did a uuidgen > /etc/vdsm/vdsm.id on ovirt node 2 and 3 even though the uuids were different already.
What does ovirt mean by "Host with the same address already exists"? What HOST is it referring to?
Greetings, Stefan
oh, not same uuid, rather same host name (like ip address or dns name used to access it) it - is the engine or the ovirt-node? The nodes have definitely different ip addresses and dns entries. This is verified.
If it is the engine then I might have a problem understanding some things I Basically want one engine that runs on each node, if needed.
Cheers
The hosts. Didi/Sandro?
It will probably be simplest to remove the second (and third?) host from then engine, then try again. -- Didi
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --E5GO1OSL2NCqTbJOjhHLaKRI5fU1eCCcJ Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Okay, I'm truncating the previous mails here.... Davids hint was the solution. I had the ovirt hosts already added to the cluster and tried to do the hosted-engine-ha setup on them. After removing the hosts from the cluster and putting the data domain to maintenance mode I was able to deploy an all other nodes. I now have a HA'd hosted engine. Which can also be migrated \o/ Maybe that is something that could be stated in the documentation more clearly? Unfortunately now I have a new problem. The agents crash rapidly after startup. The error is the following: (/var/log/ovirt-hosted-engine-ha/agent.log) AttributeError: 'NoneType' object has no attribute 'iteritems' And the whole output here - The agents have been started and I tried a migration of the hosted engine from ovirt host 1 to host 2 which succeeded. But the agents crashed afterwards: MainThread::INFO::2014-09-24 15:09:24,839::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 1.1.5 started MainThread::INFO::2014-09-24 15:09:24,871::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(_get_hostname) Found certificate common name: 10.8.2.101 MainThread::INFO::2014-09-24 15:09:25,081::hosted_engine::367::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2014-09-24 15:09:25,082::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Starting monitor ping, options {'addr': '10.8.2.1'} MainThread::INFO::2014-09-24 15:09:25,083::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Success, id 25293072 MainThread::INFO::2014-09-24 15:09:25,083::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,086::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Success, id 25294160 MainThread::INFO::2014-09-24 15:09:25,086::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,088::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Success, id 25293968 MainThread::INFO::2014-09-24 15:09:25,088::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': 'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,089::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Success, id 25360400 MainThread::INFO::2014-09-24 15:09:25,089::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': 'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,091::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(start_monitor) Success, id 25509776 MainThread::INFO::2014-09-24 15:09:25,091::hosted_engine::391::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started MainThread::INFO::2014-09-24 15:09:25,125::hosted_engine::476::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 2 is acquired (file: /rhev/data-center/mnt/10.8.2.12:_volume1_engine-store/e313da39-594c-46b5-= 95c9-c445889c745c/ha_agent/hosted-engine.lockspace) MainThread::INFO::2014-09-24 15:09:25,134::state_machine::153::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(refresh) Host 10.8.2.100 (id 1): {'live-data': True, 'extra': 'metadata_parse_version=3D1\nmetadata_feature_version=3D1\ntimestamp=3D14= 11564164 (Wed Sep 24 15:09:24 2014)\nhost-id=3D1\nscore=3D2400\nmaintenance=3DFalse\nstate=3DEngineUp\n= ', 'hostname': '10.8.2.100', 'host-id': 1, 'engine-status': {'health': 'good', 'vm': 'up', 'detail': 'up'}, 'score': 2400, 'maintenance': False, 'host-ts': 1411564164} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(refresh) Host 10.8.2.102 (id 3): {'live-data': False, 'extra': 'metadata_parse_version=3D1\nmetadata_feature_version=3D1\ntimestamp=3D14= 11562496 (Wed Sep 24 14:41:36 2014)\nhost-id=3D3\nscore=3D0\nmaintenance=3DFalse\nstate=3DEngineUnexpec= tedlyDown\ntimeout=3DWed Sep 24 14:50:24 2014\n', 'hostname': '10.8.2.102', 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 0, 'maintenance': False, 'host-ts': 1411562496} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::161::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(refresh) Local (id 2): {'engine-health': None, 'bridge': True, 'mem-free': None, 'maintenance': False, 'cpu-load': None, 'gateway': True} MainThread::INFO::2014-09-24 15:09:25,135::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(notify) Trying: notify time=3D1411564165.14 type=3Dstate_transition detail=3DStartState-ReinitializeFSM hostname=3D'ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:25,170::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(notify) Success, was notification of state_transition (StartState-ReinitializeFSM) sent? sent MainThread::INFO::2014-09-24 15:09:25,383::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(start_monitoring) Current state ReinitializeFSM (score: 0) MainThread::INFO::2014-09-24 15:09:35,409::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(notify) Trying: notify time=3D1411564175.41 type=3Dstate_transition detail=3DReinitializeFSM-EngineDown hostname=3D'ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:35,410::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(notify) Success, was notification of state_transition (ReinitializeFSM-EngineDown) sent? ignored MainThread::INFO::2014-09-24 15:09:35,627::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-09-24 15:09:45,652::states::441::ovirt_hosted_engine_ha.agent.hosted_engine.Hos= tedEngine::(consume) The engine is not running, but we do not have enough data to decide which hosts are alive MainThread::INFO::2014-09-24 15:09:45,653::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(notify) Trying: notify time=3D1411564185.65 type=3Dstate_transition detail=3DEngineDown-EngineDown hostname=3D'ovirt-node-mapconv2.int.tngtec= h.com' MainThread::INFO::2014-09-24 15:09:45,653::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.Brok= erLink::(notify) Success, was notification of state_transition (EngineDown-EngineDown) sent? ignored MainThread::INFO::2014-09-24 15:09:45,875::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_eng= ine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::CRITICAL::2014-09-24 15:09:55,899::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run)= Could not start ha-agent Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py",= line 97, in run self._run_agent() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py",= line 154, in _run_agent hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring(= ) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_eng= ine.py", line 307, in start_monitoring for old_state, state, delay in self.fsm: File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.= py", line 125, in next new_data =3D self.refresh(self._state.data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_mach= ine.py", line 77, in refresh stats.update(self.hosted_engine.collect_stats()) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_eng= ine.py", line 700, in collect_stats stats =3D self.process_remote_metadata(host_id, remote_data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_eng= ine.py", line 747, in process_remote_metadata md['engine-status'] =3D engine_status(md["engine-status"]) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_eng= ine.py", line 79, in engine_status in json.loads(status).iteritems()]) AttributeError: 'NoneType' object has no attribute 'iteritems' --E5GO1OSL2NCqTbJOjhHLaKRI5fU1eCCcJ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iEYEARECAAYFAlQixdMACgkQ01vvrYDXSqvF6ACfVH9LxU0cxyEKTZNqey0Rh/Mz AOEAoKbQQPP71vfgUl+VXS6gsck9Rgcs =mwy+ -----END PGP SIGNATURE----- --E5GO1OSL2NCqTbJOjhHLaKRI5fU1eCCcJ--
Oh well. I think this is fixed. I upgraded to 3.4.4 and the message seems to be gone. the agents are running :) Thank you very much !!! :) On 09/24/2014 15:23, Stefan Wendler wrote:
Okay, I'm truncating the previous mails here....
Davids hint was the solution. I had the ovirt hosts already added to the cluster and tried to do the hosted-engine-ha setup on them.
After removing the hosts from the cluster and putting the data domain to maintenance mode I was able to deploy an all other nodes. I now have a HA'd hosted engine. Which can also be migrated \o/
Maybe that is something that could be stated in the documentation more clearly?
Unfortunately now I have a new problem. The agents crash rapidly after startup. The error is the following: (/var/log/ovirt-hosted-engine-ha/agent.log)
AttributeError: 'NoneType' object has no attribute 'iteritems'
And the whole output here - The agents have been started and I tried a migration of the hosted engine from ovirt host 1 to host 2 which succeeded. But the agents crashed afterwards:
MainThread::INFO::2014-09-24 15:09:24,839::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 1.1.5 started MainThread::INFO::2014-09-24 15:09:24,871::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: 10.8.2.101 MainThread::INFO::2014-09-24 15:09:25,081::hosted_engine::367::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2014-09-24 15:09:25,082::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '10.8.2.1'} MainThread::INFO::2014-09-24 15:09:25,083::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25293072 MainThread::INFO::2014-09-24 15:09:25,083::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,086::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25294160 MainThread::INFO::2014-09-24 15:09:25,086::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,088::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25293968 MainThread::INFO::2014-09-24 15:09:25,088::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': 'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,089::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25360400 MainThread::INFO::2014-09-24 15:09:25,089::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': 'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,091::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25509776 MainThread::INFO::2014-09-24 15:09:25,091::hosted_engine::391::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started MainThread::INFO::2014-09-24 15:09:25,125::hosted_engine::476::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 2 is acquired (file: /rhev/data-center/mnt/10.8.2.12:_volume1_engine-store/e313da39-594c-46b5-95c9-c445889c745c/ha_agent/hosted-engine.lockspace) MainThread::INFO::2014-09-24 15:09:25,134::state_machine::153::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.8.2.100 (id 1): {'live-data': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1411564164 (Wed Sep 24 15:09:24 2014)\nhost-id=1\nscore=2400\nmaintenance=False\nstate=EngineUp\n', 'hostname': '10.8.2.100', 'host-id': 1, 'engine-status': {'health': 'good', 'vm': 'up', 'detail': 'up'}, 'score': 2400, 'maintenance': False, 'host-ts': 1411564164} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.8.2.102 (id 3): {'live-data': False, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1411562496 (Wed Sep 24 14:41:36 2014)\nhost-id=3\nscore=0\nmaintenance=False\nstate=EngineUnexpectedlyDown\ntimeout=Wed Sep 24 14:50:24 2014\n', 'hostname': '10.8.2.102', 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 0, 'maintenance': False, 'host-ts': 1411562496} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::161::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 2): {'engine-health': None, 'bridge': True, 'mem-free': None, 'maintenance': False, 'cpu-load': None, 'gateway': True} MainThread::INFO::2014-09-24 15:09:25,135::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1411564165.14 type=state_transition detail=StartState-ReinitializeFSM hostname='ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:25,170::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (StartState-ReinitializeFSM) sent? sent MainThread::INFO::2014-09-24 15:09:25,383::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state ReinitializeFSM (score: 0) MainThread::INFO::2014-09-24 15:09:35,409::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1411564175.41 type=state_transition detail=ReinitializeFSM-EngineDown hostname='ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:35,410::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (ReinitializeFSM-EngineDown) sent? ignored MainThread::INFO::2014-09-24 15:09:35,627::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-09-24 15:09:45,652::states::441::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) The engine is not running, but we do not have enough data to decide which hosts are alive MainThread::INFO::2014-09-24 15:09:45,653::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1411564185.65 type=state_transition detail=EngineDown-EngineDown hostname='ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:45,653::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineDown) sent? ignored MainThread::INFO::2014-09-24 15:09:45,875::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::CRITICAL::2014-09-24 15:09:55,899::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could not start ha-agent Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 97, in run self._run_agent() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 154, in _run_agent hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 307, in start_monitoring for old_state, state, delay in self.fsm: File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", line 125, in next new_data = self.refresh(self._state.data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", line 77, in refresh stats.update(self.hosted_engine.collect_stats()) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 700, in collect_stats stats = self.process_remote_metadata(host_id, remote_data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 747, in process_remote_metadata md['engine-status'] = engine_status(md["engine-status"]) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 79, in engine_status in json.loads(status).iteritems()]) AttributeError: 'NoneType' object has no attribute 'iteritems'
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
------=_Part_24816046_1946051580.1411575237782 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Seems we should consider not adding the host if already there. Please open a bug. Though I really hope in 3.6 to see this done from the gui On Sep 24, 2014 4:23 PM, Stefan Wendler <stefan.wendler@tngtech.com> wrote:
Okay, I'm truncating the previous mails here....
Davids hiOkay, I'm truncating the previous mails here....
Davids hint was the solution. I had the ovirt hosts already added to the cluster and tried to do the hosted-engine-ha setup on them. After removing the hosts from the cluster and putting the data domain to maintenance mode I was able to deploy an all other nodes. I now have a HA'd hosted engine. Which can also be migrated \o/ Maybe that is something that could be stated in the documentation more clearly? Unfortunately now I have a new problem. The agents crash rapidly after startup. The error is the following: (/var/log/ovirt-hosted-engine-ha/agent.log) AttributeError: 'NoneType' object has no attribute 'iteritems' And the whole output here - The agents have been started and I tried a migration of the hosted engine from ovirt host 1 to host 2 which succeeded. But the agents crashed afterwards: MainThread::INFO::2014-09-24 15:09:24,839::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 1.1.5 started MainThread::INFO::2014-09-24 15:09:24,871::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: 10.8.2.101 MainThread::INFO::2014-09-24 15:09:25,081::hosted_engine::367::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2014-09-24 15:09:25,082::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '10.8.2.1'} MainThread::INFO::2014-09-24 15:09:25,083::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25293072 MainThread::INFO::2014-09-24 15:09:25,083::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,086::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25294160 MainThread::INFO::2014-09-24 15:09:25,086::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,088::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25293968 MainThread::INFO::2014-09-24 15:09:25,088::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': 'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,089::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25360400 MainThread::INFO::2014-09-24 15:09:25,089::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': 'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} MainThread::INFO::2014-09-24 15:09:25,091::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 25509776 MainThread::INFO::2014-09-24 15:09:25,091::hosted_engine::391::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started MainThread::INFO::2014-09-24 15:09:25,125::hosted_engine::476::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 2 is acquired (file: /rhev/data-center/mnt/10.8.2.12:_volume1_engine-store/e313da39-594c-46b5-95c9-c445889c745c/ha_agent/hosted-engine.lockspace) MainThread::INFO::2014-09-24 15:09:25,134::state_machine::153::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.8.2.100 (id 1): {'live-data': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1411564164 (Wed Sep 24 15:09:24 2014)\nhost-id=1\nscore=2400\nmaintenance=False\nstate=EngineUp\n', 'hostname': '10.8.2.100', 'host-id': 1, 'engine-status': {'health': 'good', 'vm': 'up', 'detail': 'up'}, 'score': 2400, 'maintenance': False, 'host-ts': 1411564164} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host 10.8.2.102 (id 3): {'live-data': False, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1411562496 (Wed Sep 24 14:41:36 2014)\nhost-id=3\nscore=0\nmaintenance=False\nstate=EngineUnexpectedlyDown\ntimeout=Wed Sep 24 14:50:24 2014\n', 'hostname': '10.8.2.102', 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 0, 'maintenance': False, 'host-ts': 1411562496} MainThread::INFO::2014-09-24 15:09:25,134::state_machine::161::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 2): {'engine-health': None, 'bridge': True, 'mem-free': None, 'maintenance': False, 'cpu-load': None, 'gateway': True} MainThread::INFO::2014-09-24 15:09:25,135::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1411564165.14 type=state_transition detail=StartState-ReinitializeFSM hostname='ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:25,170::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (StartState-ReinitializeFSM) sent? sent MainThread::INFO::2014-09-24 15:09:25,383::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state ReinitializeFSM (score: 0) MainThread::INFO::2014-09-24 15:09:35,409::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1411564175.41 type=state_transition detail=ReinitializeFSM-EngineDown hostname='ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:35,410::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (ReinitializeFSM-EngineDown) sent? ignored MainThread::INFO::2014-09-24 15:09:35,627::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::INFO::2014-09-24 15:09:45,652::states::441::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) The engine is not running, but we do not have enough data to decide which hosts are alive MainThread::INFO::2014-09-24 15:09:45,653::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1411564185.65 type=state_transition detail=EngineDown-EngineDown hostname='ovirt-node-mapconv2.int.tngtech.com' MainThread::INFO::2014-09-24 15:09:45,653::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineDown) sent? ignored MainThread::INFO::2014-09-24 15:09:45,875::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 2400) MainThread::CRITICAL::2014-09-24 15:09:55,899::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Could not start ha-agent Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 97, in run self._run_agent() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 154, in _run_agent hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 307, in start_monitoring for old_state, state, delay in self.fsm: File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", line 125, in next new_data = self.refresh(self._state.data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", line 77, in refresh stats.update(self.hosted_engine.collect_stats()) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 700, in collect_stats stats = self.process_remote_metadata(host_id, remote_data) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 747, in process_remote_metadata md['engine-status'] = engine_status(md["engine-status"]) File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 79, in engine_status in json.loads(status).iteritems()]) AttributeError: 'NoneType' object has no attribute 'iteritems' ------=_Part_24816046_1946051580.1411575237782 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html><body><div>Seems we should consider not adding the host if already th= ere. Please open a bug.<br>Though I really hope in 3.6 to see this done fro= m the gui<br><br>On Sep 24, 2014 4:23 PM, Stefan Wendler <stefan.wendler= @tngtech.com> wrote:<br>><br>> Okay, I'm truncating the previous m= ails here.... <br>><br>> Davids hi</div><br><div>Okay, I'm truncating= the previous mails here.... <br> <br>Davids hint was the solution. I had the ovirt hosts already added to th= e <br>cluster and tried to do the hosted-engine-ha setup on them. <br> <br>After removing the hosts from the cluster and putting the data domain t= o <br>maintenance mode I was able to deploy an all other nodes. I now have a <br>HA'd hosted engine. Which can also be migrated \o/ <br> <br>Maybe that is something that could be stated in the documentation more <br>clearly? <br> <br>Unfortunately now I have a new problem. The agents crash rapidly after <br>startup. The error is the following: <br>(/var/log/ovirt-hosted-engine-ha/agent.log) <br> <br>AttributeError: 'NoneType' object has no attribute 'iteritems' <br> <br>And the whole output here - The agents have been started and I tried a <br>migration of the hosted engine from ovirt host 1 to host 2 which <br>succeeded. But the agents crashed afterwards: <br> <br>MainThread::INFO::2014-09-24 <br>15:09:24,839::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run= ) <br>ovirt-hosted-engine-ha agent 1.1.5 started <br>MainThread::INFO::2014-09-24 <br>15:09:24,871::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(_get_hostname) <br>Found certificate common name: 10.8.2.101 <br>MainThread::INFO::2014-09-24 <br>15:09:25,081::hosted_engine::367::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(_initialize_broker) <br>Initializing ha-broker connection <br>MainThread::INFO::2014-09-24 <br>15:09:25,082::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Starting monitor ping, options {'addr': '10.8.2.1'} <br>MainThread::INFO::2014-09-24 <br>15:09:25,083::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Success, id 25293072 <br>MainThread::INFO::2014-09-24 <br>15:09:25,083::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name'= : <br>'ovirtmgmt', 'address': '0'} <br>MainThread::INFO::2014-09-24 <br>15:09:25,086::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Success, id 25294160 <br>MainThread::INFO::2014-09-24 <br>15:09:25,086::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} <br>MainThread::INFO::2014-09-24 <br>15:09:25,088::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Success, id 25293968 <br>MainThread::INFO::2014-09-24 <br>15:09:25,088::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', <br>'vm_uuid': 'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} <br>MainThread::INFO::2014-09-24 <br>15:09:25,089::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Success, id 25360400 <br>MainThread::INFO::2014-09-24 <br>15:09:25,089::brokerlink::126::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': <br>'e1ca293f-09e0-4d2e-8915-221839af1489', 'address': '0'} <br>MainThread::INFO::2014-09-24 <br>15:09:25,091::brokerlink::137::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(start_monitor) <br>Success, id 25509776 <br>MainThread::INFO::2014-09-24 <br>15:09:25,091::hosted_engine::391::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(_initialize_broker) <br>Broker initialized, all submonitors started <br>MainThread::INFO::2014-09-24 <br>15:09:25,125::hosted_engine::476::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(_initialize_sanlock) <br>Ensuring lease for lockspace hosted-engine, host id 2 is acquired (file= : <br>/rhev/data-center/mnt/10.8.2.12:_volume1_engine-store/e313da39-594c-46b= 5-95c9-c445889c745c/ha_agent/hosted-engine.lockspace) <br>MainThread::INFO::2014-09-24 <br>15:09:25,134::state_machine::153::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(refresh) <br>Global metadata: {'maintenance': False} <br>MainThread::INFO::2014-09-24 <br>15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(refresh) <br>Host 10.8.2.100 (id 1): {'live-data': True, 'extra': <br>'metadata_parse_version=3D1\nmetadata_feature_version=3D1\ntimestamp=3D= 1411564164 <br>(Wed Sep 24 15:09:24 <br>2014)\nhost-id=3D1\nscore=3D2400\nmaintenance=3DFalse\nstate=3DEngineUp= \n', <br>'hostname': '10.8.2.100', 'host-id': 1, 'engine-status': {'health': <br>'good', 'vm': 'up', 'detail': 'up'}, 'score': 2400, 'maintenance': <br>False, 'host-ts': 1411564164} <br>MainThread::INFO::2014-09-24 <br>15:09:25,134::state_machine::158::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(refresh) <br>Host 10.8.2.102 (id 3): {'live-data': False, 'extra': <br>'metadata_parse_version=3D1\nmetadata_feature_version=3D1\ntimestamp=3D= 1411562496 <br>(Wed Sep 24 14:41:36 <br>2014)\nhost-id=3D3\nscore=3D0\nmaintenance=3DFalse\nstate=3DEngineUnexp= ectedlyDown\ntimeout=3DWed <br>Sep 24 14:50:24 2014\n', 'hostname': '10.8.2.102', 'host-id': 3, <br>'engine-status': {'reason': 'vm not running on this host', 'health': <br>'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 0, 'maintenance': <br>False, 'host-ts': 1411562496} <br>MainThread::INFO::2014-09-24 <br>15:09:25,134::state_machine::161::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(refresh) <br>Local (id 2): {'engine-health': None, 'bridge': True, 'mem-free': None, <br>'maintenance': False, 'cpu-load': None, 'gateway': True} <br>MainThread::INFO::2014-09-24 <br>15:09:25,135::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(notify) <br>Trying: notify time=3D1411564165.14 type=3Dstate_transition <br>detail=3DStartState-ReinitializeFSM <br>hostname=3D'ovirt-node-mapconv2.int.tngtech.com' <br>MainThread::INFO::2014-09-24 <br>15:09:25,170::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(notify) <br>Success, was notification of state_transition <br>(StartState-ReinitializeFSM) sent? sent <br>MainThread::INFO::2014-09-24 <br>15:09:25,383::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(start_monitoring) <br>Current state ReinitializeFSM (score: 0) <br>MainThread::INFO::2014-09-24 <br>15:09:35,409::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(notify) <br>Trying: notify time=3D1411564175.41 type=3Dstate_transition <br>detail=3DReinitializeFSM-EngineDown <br>hostname=3D'ovirt-node-mapconv2.int.tngtech.com' <br>MainThread::INFO::2014-09-24 <br>15:09:35,410::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(notify) <br>Success, was notification of state_transition <br>(ReinitializeFSM-EngineDown) sent? ignored <br>MainThread::INFO::2014-09-24 <br>15:09:35,627::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(start_monitoring) <br>Current state EngineDown (score: 2400) <br>MainThread::INFO::2014-09-24 <br>15:09:45,652::states::441::ovirt_hosted_engine_ha.agent.hosted_engine.H= ostedEngine::(consume) <br>The engine is not running, but we do not have enough data to decide <br>which hosts are alive <br>MainThread::INFO::2014-09-24 <br>15:09:45,653::brokerlink::108::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(notify) <br>Trying: notify time=3D1411564185.65 type=3Dstate_transition <br>detail=3DEngineDown-EngineDown hostname=3D'ovirt-node-mapconv2.int.tngt= ech.com' <br>MainThread::INFO::2014-09-24 <br>15:09:45,653::brokerlink::117::ovirt_hosted_engine_ha.lib.brokerlink.Br= okerLink::(notify) <br>Success, was notification of state_transition (EngineDown-EngineDown) <br>sent? ignored <br>MainThread::INFO::2014-09-24 <br>15:09:45,875::hosted_engine::327::ovirt_hosted_engine_ha.agent.hosted_e= ngine.HostedEngine::(start_monitoring) <br>Current state EngineDown (score: 2400) <br>MainThread::CRITICAL::2014-09-24 <br>15:09:55,899::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(ru= n) Could <br>not start ha-agent <br>Traceback (most recent call last): <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/age= nt.py", line <br>97, in run <br> self._run_agent() <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/age= nt.py", line <br>154, in _run_agent <br> hosted_engine.HostedEngine(self.shutdown_requested).start= _monitoring() <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hos= ted_engine.py", <br>line 307, in start_monitoring <br> for old_state, state, delay in self.fsm: <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/m= achine.py", <br>line 125, in next <br> new_data =3D self.refresh(self._state.data) <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/sta= te_machine.py", <br>line 77, in refresh <br> stats.update(self.hosted_engine.collect_stats()) <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hos= ted_engine.py", <br>line 700, in collect_stats <br> stats =3D self.process_remote_metadata(host_id, remote_da= ta) <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hos= ted_engine.py", <br>line 747, in process_remote_metadata <br> md['engine-status'] =3D engine_status(md["engine-sta= tus"]) <br> File <br>"/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hos= ted_engine.py", <br>line 79, in engine_status <br> in json.loads(status).iteritems()]) <br>AttributeError: 'NoneType' object has no attribute 'iteritems' <br> <br> <br></div></body></html> ------=_Part_24816046_1946051580.1411575237782--
participants (4)
-
Itamar Heim -
Jiri Moskovcak -
Stefan Wendler -
Yedidyah Bar David