[Users] Is there a way to force remove a host?
I could not remove old host even if others where up. Can I force remove I do not need it anymore. -- Dominic Kaiser Greater Boston Vineyard Director of Operations cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org
----- Original Message -----
From: "Dominic Kaiser" <dominic@bostonvineyard.org> To: users@ovirt.org Sent: Thursday, September 20, 2012 6:44:58 PM Subject: [Users] Is there a way to force remove a host?
I could not remove old host even if others where up. Can I force remove I do not need it anymore.
Dominic, please attach engine/vdsm logs so we will be able to see why the Host is not removed. Thanks
-- Dominic Kaiser Greater Boston Vineyard Director of Operations
cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Sorry I did not explain. I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force. dk On Thu, Sep 20, 2012 at 12:00 PM, Eli Mesika <emesika@redhat.com> wrote:
----- Original Message -----
From: "Dominic Kaiser" <dominic@bostonvineyard.org> To: users@ovirt.org Sent: Thursday, September 20, 2012 6:44:58 PM Subject: [Users] Is there a way to force remove a host?
I could not remove old host even if others where up. Can I force remove I do not need it anymore.
Dominic, please attach engine/vdsm logs so we will be able to see why the Host is not removed. Thanks
-- Dominic Kaiser Greater Boston Vineyard Director of Operations
cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Dominic Kaiser Greater Boston Vineyard Director of Operations cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org
On 09/20/2012 07:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
why can't you just move it to maint and delete it? (you can right click and 'confirm host shutdown manually' to release any resources supposedly held by it)
dk
On Thu, Sep 20, 2012 at 12:00 PM, Eli Mesika <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > To: users@ovirt.org <mailto:users@ovirt.org> > Sent: Thursday, September 20, 2012 6:44:58 PM > Subject: [Users] Is there a way to force remove a host? > > > I could not remove old host even if others where up. Can I force > remove I do not need it anymore.
Dominic, please attach engine/vdsm logs so we will be able to see why the Host is not removed. Thanks > > > -- > Dominic Kaiser > Greater Boston Vineyard > Director of Operations > > cell: 617-230-1412 <tel:617-230-1412> > fax: 617-252-0238 <tel:617-252-0238> > email: dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org> > > > > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/users >
-- Dominic Kaiser Greater Boston Vineyard Director of Operations
cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Dominic Kaiser" <dominic@bostonvineyard.org> Cc: users@ovirt.org Sent: Thursday, September 20, 2012 2:38:23 PM Subject: Re: [Users] Is there a way to force remove a host?
On 09/20/2012 07:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
why can't you just move it to maint and delete it? (you can right click and 'confirm host shutdown manually' to release any resources supposedly held by it)
Is it the only host in the DC, if that's the case you can't put it to maintenance mode.
dk
On Thu, Sep 20, 2012 at 12:00 PM, Eli Mesika <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > To: users@ovirt.org <mailto:users@ovirt.org> > Sent: Thursday, September 20, 2012 6:44:58 PM > Subject: [Users] Is there a way to force remove a host? > > > I could not remove old host even if others where up. Can I > force > remove I do not need it anymore.
Dominic, please attach engine/vdsm logs so we will be able to see why the Host is not removed. Thanks > > > -- > Dominic Kaiser > Greater Boston Vineyard > Director of Operations > > cell: 617-230-1412 <tel:617-230-1412> > fax: 617-252-0238 <tel:617-252-0238> > email: dominic@bostonvineyard.org > <mailto:dominic@bostonvineyard.org> > > > > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/users >
-- Dominic Kaiser Greater Boston Vineyard Director of Operations
cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
No there is an active host in the cluster. It still will not allow removal of the non-existent host. dk On Fri, Sep 21, 2012 at 11:45 AM, Andrew Cathrow <acathrow@redhat.com>wrote:
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Dominic Kaiser" <dominic@bostonvineyard.org> Cc: users@ovirt.org Sent: Thursday, September 20, 2012 2:38:23 PM Subject: Re: [Users] Is there a way to force remove a host?
On 09/20/2012 07:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
why can't you just move it to maint and delete it? (you can right click and 'confirm host shutdown manually' to release any resources supposedly held by it)
Is it the only host in the DC, if that's the case you can't put it to maintenance mode.
dk
On Thu, Sep 20, 2012 at 12:00 PM, Eli Mesika <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > To: users@ovirt.org <mailto:users@ovirt.org> > Sent: Thursday, September 20, 2012 6:44:58 PM > Subject: [Users] Is there a way to force remove a host? > > > I could not remove old host even if others where up. Can I > force > remove I do not need it anymore.
Dominic, please attach engine/vdsm logs so we will be able to see why the Host is not removed. Thanks > > > -- > Dominic Kaiser > Greater Boston Vineyard > Director of Operations > > cell: 617-230-1412 <tel:617-230-1412> > fax: 617-252-0238 <tel:617-252-0238> > email: dominic@bostonvineyard.org > <mailto:dominic@bostonvineyard.org> > > > > _______________________________________________ > Users mailing list > Users@ovirt.org <mailto:Users@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/users >
-- Dominic Kaiser Greater Boston Vineyard Director of Operations
cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Dominic Kaiser Greater Boston Vineyard Director of Operations cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org
Hi Dominic, On 09/20/2012 12:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
dk
Fell free to try the below script (not part of official project) for brute force: (from the engine side) # yum install python-psycopg2 -y # wget https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... # (edit the file and change the db password) # python ./engine_force_remove_Host.py Thanks -- Cheers Douglas
----- Original Message -----
From: "Douglas Landgraf" <dougsland@redhat.com> To: "Dominic Kaiser" <dominic@bostonvineyard.org> Cc: "Eli Mesika" <emesika@redhat.com>, users@ovirt.org, "Robert Middleswarth" <robert@middleswarth.net> Sent: Friday, September 21, 2012 8:12:27 PM Subject: Re: [Users] Is there a way to force remove a host?
Hi Dominic,
On 09/20/2012 12:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
dk
Fell free to try the below script (not part of official project) for brute force:
(from the engine side) # yum install python-psycopg2 -y # wget https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... # (edit the file and change the db password) # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
Thanks
-- Cheers Douglas
On 09/22/2012 11:19 PM, Eli Mesika wrote:
----- Original Message -----
From: "Douglas Landgraf" <dougsland@redhat.com> To: "Dominic Kaiser" <dominic@bostonvineyard.org> Cc: "Eli Mesika" <emesika@redhat.com>, users@ovirt.org, "Robert Middleswarth" <robert@middleswarth.net> Sent: Friday, September 21, 2012 8:12:27 PM Subject: Re: [Users] Is there a way to force remove a host?
Hi Dominic,
On 09/20/2012 12:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
dk
Fell free to try the below script (not part of official project) for brute force:
(from the engine side) # yum install python-psycopg2 -y # wget https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... # (edit the file and change the db password) # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database.
shouldn't foreign keys protect against this? (if not cascade deletes which i remember caused some issues with locks)
In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
Thanks
-- Cheers Douglas
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Eli Mesika" <emesika@redhat.com> Cc: dougsland@redhat.com, users@ovirt.org Sent: Sunday, September 23, 2012 3:07:03 PM Subject: Re: [Users] Is there a way to force remove a host?
On 09/22/2012 11:19 PM, Eli Mesika wrote:
----- Original Message -----
From: "Douglas Landgraf" <dougsland@redhat.com> To: "Dominic Kaiser" <dominic@bostonvineyard.org> Cc: "Eli Mesika" <emesika@redhat.com>, users@ovirt.org, "Robert Middleswarth" <robert@middleswarth.net> Sent: Friday, September 21, 2012 8:12:27 PM Subject: Re: [Users] Is there a way to force remove a host?
Hi Dominic,
On 09/20/2012 12:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
dk
Fell free to try the below script (not part of official project) for brute force:
(from the engine side) # yum install python-psycopg2 -y # wget https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... # (edit the file and change the db password) # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database.
shouldn't foreign keys protect against this? (if not cascade deletes which i remember caused some issues with locks) Yes, this is exactly the case, therefore the SP does the deletes explicitly in the right order... BTW Since those locks were from the time we had used MS SQL, maybe we should change back to FK and give it a chance in PG ...
In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
Thanks
-- Cheers Douglas
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well. Dominic On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com> wrote:
----- Original Message -----
From: "Douglas Landgraf" <dougsland@redhat.com> To: "Dominic Kaiser" <dominic@bostonvineyard.org> Cc: "Eli Mesika" <emesika@redhat.com>, users@ovirt.org, "Robert Middleswarth" <robert@middleswarth.net> Sent: Friday, September 21, 2012 8:12:27 PM Subject: Re: [Users] Is there a way to force remove a host?
Hi Dominic,
On 09/20/2012 12:11 PM, Dominic Kaiser wrote:
Sorry I did not explain.
I had tried to remove the host and had not luck troubleshooting it. I then had removed it and used it for a storage unit reinstalling fedora 17. I foolishly thought that I could just remove the host manually. It physically is not there. (My fault I know) Is there a way that you know of to remove a host brute force.
dk
Fell free to try the below script (not part of official project) for brute force:
(from the engine side) # yum install python-psycopg2 -y # wget
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
# (edit the file and change the db password) # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
Thanks
-- Cheers Douglas
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget > https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... > # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference. I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- @jasonbrooks
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well? what is the status of the host after you confirm shutdown? any error on log on this specific command? shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
On Mon 24 Sep 2012 02:55:54 PM PDT, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well?
Yes, it is.
what is the status of the host after you confirm shutdown? any error on log on this specific command?
I have two hosts I want to remove -- one, a failed add that never completed, and the other, an F17 machine that I subsequently updated to F18. So neither of those hosts are actually connected to my engine. Their status is either non-responsive or maintenance, if I activate, it's the former, if I set it to maintenance, it's the latter. I can only attempt to remove in maintenance mode, so that's where I have them. There don't appear to be any errors confirming manual reboot.
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
-- @jasonbrooks
On 09/25/2012 12:03 AM, Jason Brooks wrote:
On Mon 24 Sep 2012 02:55:54 PM PDT, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well?
Yes, it is.
that's a very important piece of information... is there a more detailed exception in the log on the error? can you uncheck gluster and try to remove it again? Dominic - are you also using a cluster with gluster service enabled?
what is the status of the host after you confirm shutdown? any error on log on this specific command?
I have two hosts I want to remove -- one, a failed add that never completed, and the other, an F17 machine that I subsequently updated to F18. So neither of those hosts are actually connected to my engine. Their status is either non-responsive or maintenance, if I activate, it's the former, if I set it to maintenance, it's the latter. I can only attempt to remove in maintenance mode, so that's where I have them.
There don't appear to be any errors confirming manual reboot.
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
--
@jasonbrooks
----- Original Message -----
From: "Itamar Heim" <iheim@redhat.com> To: "Jason Brooks" <jbrooks@redhat.com> Cc: users@ovirt.org, "Dominic Kaiser" <dominic@bostonvineyard.org>, "Shireesh Anjal" <sanjal@redhat.com> Sent: Monday, September 24, 2012 3:07:24 PM Subject: Re: [Users] Is there a way to force remove a host?
On 09/25/2012 12:03 AM, Jason Brooks wrote:
On Mon 24 Sep 2012 02:55:54 PM PDT, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well?
Yes, it is.
that's a very important piece of information... is there a more detailed exception in the log on the error? can you uncheck gluster and try to remove it again?
OK. Interestingly, I could not check or uncheck anything in my default cluster. The boxes were greyed out. But, I was able to remove the hosts by adding a new cluster w/o gluster enabled, moving the hosts there, and removing them. It does seem that gluster is unwilling to let hosts go, even if they aren't hosting any bricks...
Dominic - are you also using a cluster with gluster service enabled?
what is the status of the host after you confirm shutdown? any error on log on this specific command?
I have two hosts I want to remove -- one, a failed add that never completed, and the other, an F17 machine that I subsequently updated to F18. So neither of those hosts are actually connected to my engine. Their status is either non-responsive or maintenance, if I activate, it's the former, if I set it to maintenance, it's the latter. I can only attempt to remove in maintenance mode, so that's where I have them.
There don't appear to be any errors confirming manual reboot.
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a > host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the > > host > > manually. > > It physically is not there. (My fault I know) Is > > there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
--
@jasonbrooks
Dominic - are you also using a cluster with gluster service enabled? Yes Gluster Service is enabled. And I can not remove hosts from that cluster. Dominic On Mon, Sep 24, 2012 at 6:07 PM, Itamar Heim <iheim@redhat.com> wrote:
On 09/25/2012 12:03 AM, Jason Brooks wrote:
On Mon 24 Sep 2012 02:55:54 PM PDT, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well?
Yes, it is.
that's a very important piece of information... is there a more detailed exception in the log on the error? can you uncheck gluster and try to remove it again?
Dominic - are you also using a cluster with gluster service enabled?
what is the status of the host after you confirm shutdown?
any error on log on this specific command?
I have two hosts I want to remove -- one, a failed add that never completed, and the other, an F17 machine that I subsequently updated to F18. So neither of those hosts are actually connected to my engine. Their status is either non-responsive or maintenance, if I activate, it's the former, if I set it to maintenance, it's the latter. I can only attempt to remove in maintenance mode, so that's where I have them.
There don't appear to be any errors confirming manual reboot.
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@**bostonvineyard.org <dominic@bostonvineyard.org> >> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.**net<robert@middleswarth.net> >> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/**dougsland/misc-rhev/master/** engine_force_remove_Host.py<https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.py>
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
______________________________**_________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/**mailman/listinfo/users<http://lists.ovirt.org/mailman/listinfo/users>
______________________________**_________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/**mailman/listinfo/users<http://lists.ovirt.org/mailman/listinfo/users>
--
@jasonbrooks
--
@jasonbrooks
-- Dominic Kaiser Greater Boston Vineyard Director of Operations cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org
On 09/25/2012 03:27 AM, Dominic Kaiser wrote:
Dominic - are you also using a cluster with gluster service enabled?
Yes Gluster Service is enabled. And I can not remove hosts from that cluster.
shireesh - please take a look at this. dominic - move the host to a virt only cluster and delete the host.
Dominic
On Mon, Sep 24, 2012 at 6:07 PM, Itamar Heim <iheim@redhat.com <mailto:iheim@redhat.com>> wrote:
On 09/25/2012 12:03 AM, Jason Brooks wrote:
On Mon 24 Sep 2012 02:55:54 PM PDT, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well?
Yes, it is.
that's a very important piece of information... is there a more detailed exception in the log on the error? can you uncheck gluster and try to remove it again?
Dominic - are you also using a cluster with gluster service enabled?
what is the status of the host after you confirm shutdown? any error on log on this specific command?
I have two hosts I want to remove -- one, a failed add that never completed, and the other, an F17 machine that I subsequently updated to F18. So neither of those hosts are actually connected to my engine. Their status is either non-responsive or maintenance, if I activate, it's the former, if I set it to maintenance, it's the latter. I can only attempt to remove in maintenance mode, so that's where I have them.
There don't appear to be any errors confirming manual reboot.
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com> <mailto:emesika@redhat.com <mailto:emesika@redhat.com>>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com> <mailto:dougsland@redhat.com <mailto:dougsland@redhat.com>>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org> <mailto:dominic@__bostonvineyard.org <mailto:dominic@bostonvineyard.org>>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com> <mailto:emesika@redhat.com <mailto:emesika@redhat.com>>>, users@ovirt.org <mailto:users@ovirt.org> <mailto:users@ovirt.org <mailto:users@ovirt.org>>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net> <mailto:robert@middleswarth.__net <mailto:robert@middleswarth.net>>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/__dougsland/misc-rhev/master/__engine_force_remove_Ho... <https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.py>
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_________________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/__mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
_________________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/__mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
--
@jasonbrooks
--
@jasonbrooks
-- Dominic Kaiser Greater Boston Vineyard Director of Operations
cell: 617-230-1412 fax: 617-252-0238 email: dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>
On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well? what is the status of the host after you confirm shutdown? any error on log on this specific command?
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Yes, currently the system tries the 'gluster peer detach <hostname>' command when trying to remove a server, which fails if the server is unavailable. This can be enhanced to show the error to user and then allow 'force remove' which can use the 'gluster peer detach <hostname> *force*' command that forcefully removes the server from the cluster, even if it is not available or has bricks on it.
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
On 09/25/2012 09:44 AM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well? what is the status of the host after you confirm shutdown? any error on log on this specific command?
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Yes, currently the system tries the 'gluster peer detach <hostname>' command when trying to remove a server, which fails if the server is unavailable. This can be enhanced to show the error to user and then allow 'force remove' which can use the 'gluster peer detach <hostname> *force*' command that forcefully removes the server from the cluster, even if it is not available or has bricks on it.
what if it is the last server in the cluster? what if there is another server in the cluster but no communication to it as well?
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
On Tuesday 25 September 2012 01:42 PM, Itamar Heim wrote:
On 09/25/2012 09:44 AM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
This conversation is fine but if I want to force remove no matter what I should be able to from the GUI. The nodes are no longer available I want to get rid of them ovirt does not let me. I can delete from database but why not from the GUI? I am sure others may run into this problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well? what is the status of the host after you confirm shutdown? any error on log on this specific command?
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Yes, currently the system tries the 'gluster peer detach <hostname>' command when trying to remove a server, which fails if the server is unavailable. This can be enhanced to show the error to user and then allow 'force remove' which can use the 'gluster peer detach <hostname> *force*' command that forcefully removes the server from the cluster, even if it is not available or has bricks on it.
what if it is the last server in the cluster? what if there is another server in the cluster but no communication to it as well?
A quick look at code tells me that in case of virt, we don't allow removing a host if it has VM(s) in it (even if the host is currently not available) i.e. vdsDynamic.getvm_count() > 0. Please correct me if I'm wrong. If that's correct, and if we want to keep it consistent for gluster as well, then we should not allow removing a host if it has gluster volume(s) in it. This is how it behaves in case of 'last server in cluster' today. In case of no up server available in the cluster, we can show the error and provide a 'force' option that will just remove it from the engine DB and will not attempt gluster peer detach.
Dominic
On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>> wrote:
----- Original Message ----- > From: "Douglas Landgraf" <dougsland@redhat.com <mailto:dougsland@redhat.com>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org <mailto:dominic@bostonvineyard.org>> > Cc: "Eli Mesika" <emesika@redhat.com <mailto:emesika@redhat.com>>, users@ovirt.org <mailto:users@ovirt.org>, "Robert Middleswarth" <robert@middleswarth.net <mailto:robert@middleswarth.net>> > Sent: Friday, September 21, 2012 8:12:27 PM > Subject: Re: [Users] Is there a way to force remove a host? > > Hi Dominic, > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > Sorry I did not explain. > > > > I had tried to remove the host and had not luck troubleshooting it. > > I > > then had removed it and used it for a storage unit reinstalling > > fedora > > 17. I foolishly thought that I could just remove the host > > manually. > > It physically is not there. (My fault I know) Is there a way that > > you know of to remove a host brute force. > > > > dk > > Fell free to try the below script (not part of official project) for > brute force: > > (from the engine side) > # yum install python-psycopg2 -y > # wget >
https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p...
> # (edit the file and change the db password) > # python ./engine_force_remove_Host.py
Hi , had looked in the Python script you had provided: First, I must say that handling the database directly may leave DB in inconsistent state, therefore, if there is no other option, the database should be backed up prior to this operation. In addition, I do not like the execution of the SQL statements in the script. There is a SP called DeleteVds(v_vds_id UUID) and you should use that since it encapsulates all details. For example, your script does not handle permission clean-up as the SP does and therefore leaves garbage in the database. In addition, a failure in your script may leave database in inconsistent state while the SP is executed in one transaction and will leave DB consistent. So, in short I would prefer in this case that the relevant SP will do the clean-up since this is the one that is used by the code and that insures (at least I hope so) , that all related entities are removed as well.
> > Thanks > > -- > Cheers > Douglas > >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
On 09/25/2012 12:32 PM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 01:42 PM, Itamar Heim wrote:
On 09/25/2012 09:44 AM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
On 09/24/2012 08:49 PM, Dominic Kaiser wrote: > This conversation is fine but if I want to force remove no matter > what I > should be able to from the GUI. The nodes are no longer available I > want to get rid of them ovirt does not let me. I can delete from > database but why not from the GUI? I am sure others may run into > this > problem as well.
what happens to the status of the host when you right click on the host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well? what is the status of the host after you confirm shutdown? any error on log on this specific command?
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Yes, currently the system tries the 'gluster peer detach <hostname>' command when trying to remove a server, which fails if the server is unavailable. This can be enhanced to show the error to user and then allow 'force remove' which can use the 'gluster peer detach <hostname> *force*' command that forcefully removes the server from the cluster, even if it is not available or has bricks on it.
what if it is the last server in the cluster? what if there is another server in the cluster but no communication to it as well?
A quick look at code tells me that in case of virt, we don't allow removing a host if it has VM(s) in it (even if the host is currently not available) i.e. vdsDynamic.getvm_count() > 0. Please correct me if I'm wrong. If that's correct, and if we want to keep it consistent for gluster as well, then we should not allow removing a host if it has gluster volume(s) in it. This is how it behaves in case of 'last server in cluster' today.
true, but user can fence the host or confirm shutdown manually, which will release all resources on it, then it can be removed.
In case of no up server available in the cluster, we can show the error and provide a 'force' option that will just remove it from the engine DB and will not attempt gluster peer detach.
something like that. i assume the gluster storage will handle this somehow?
> > Dominic > > On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com > <mailto:emesika@redhat.com>> wrote: > > > > ----- Original Message ----- > > From: "Douglas Landgraf" <dougsland@redhat.com > <mailto:dougsland@redhat.com>> > > To: "Dominic Kaiser" <dominic@bostonvineyard.org > <mailto:dominic@bostonvineyard.org>> > > Cc: "Eli Mesika" <emesika@redhat.com > <mailto:emesika@redhat.com>>, users@ovirt.org > <mailto:users@ovirt.org>, "Robert Middleswarth" > <robert@middleswarth.net <mailto:robert@middleswarth.net>> > > Sent: Friday, September 21, 2012 8:12:27 PM > > Subject: Re: [Users] Is there a way to force remove a host? > > > > Hi Dominic, > > > > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: > > > Sorry I did not explain. > > > > > > I had tried to remove the host and had not luck > troubleshooting it. > > > I > > > then had removed it and used it for a storage unit > reinstalling > > > fedora > > > 17. I foolishly thought that I could just remove the host > > > manually. > > > It physically is not there. (My fault I know) Is there a > way that > > > you know of to remove a host brute force. > > > > > > dk > > > > Fell free to try the below script (not part of official > project) for > > brute force: > > > > (from the engine side) > > # yum install python-psycopg2 -y > > # wget > > > > https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... > > > > > > # (edit the file and change the db password) > > # python ./engine_force_remove_Host.py > > Hi , had looked in the Python script you had provided: > First, I must say that handling the database directly may > leave DB > in inconsistent state, therefore, if there is no other > option, the > database should be backed up prior to this operation. > In addition, I do not like the execution of the SQL > statements in > the script. > There is a SP called DeleteVds(v_vds_id UUID) and you should use > that since it encapsulates all details. > For example, your script does not handle permission clean-up as > the > SP does and therefore leaves garbage in the database. > In addition, a failure in your script may leave database in > inconsistent state while the SP is executed in one > transaction and > will leave DB consistent. > So, in short I would prefer in this case that the relevant SP > will > do the clean-up since this is the one that is used by the > code and > that insures (at least I hope so) , that all related entities > are > removed as well. > > > > > > Thanks > > > > -- > > Cheers > > Douglas > > > > > > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
On Tuesday 25 September 2012 04:04 PM, Itamar Heim wrote:
On 09/25/2012 12:32 PM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 01:42 PM, Itamar Heim wrote:
On 09/25/2012 09:44 AM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote:
On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote: > On 09/24/2012 08:49 PM, Dominic Kaiser wrote: >> This conversation is fine but if I want to force remove no matter >> what I >> should be able to from the GUI. The nodes are no longer >> available I >> want to get rid of them ovirt does not let me. I can delete from >> database but why not from the GUI? I am sure others may run into >> this >> problem as well. > > what happens to the status of the host when you right click on the > host and specify you confirm it was shutdown?
I'm having this same issue. Confirming the host is shut down doesn't make a difference.
I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = Unexpected exception" errors in my engine log that seem to correspond w/ the failed remove host attempts.
is cluster defined as gluster as well? what is the status of the host after you confirm shutdown? any error on log on this specific command?
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Yes, currently the system tries the 'gluster peer detach <hostname>' command when trying to remove a server, which fails if the server is unavailable. This can be enhanced to show the error to user and then allow 'force remove' which can use the 'gluster peer detach <hostname> *force*' command that forcefully removes the server from the cluster, even if it is not available or has bricks on it.
what if it is the last server in the cluster? what if there is another server in the cluster but no communication to it as well?
A quick look at code tells me that in case of virt, we don't allow removing a host if it has VM(s) in it (even if the host is currently not available) i.e. vdsDynamic.getvm_count() > 0. Please correct me if I'm wrong. If that's correct, and if we want to keep it consistent for gluster as well, then we should not allow removing a host if it has gluster volume(s) in it. This is how it behaves in case of 'last server in cluster' today.
true, but user can fence the host or confirm shutdown manually, which will release all resources on it, then it can be removed.
I see. In that case, we can just remove the validation and allow removing the host irrespective of whether it contains volume(s) or not. Since it's the only host in the cluster, this won't cause any harm.
In case of no up server available in the cluster, we can show the error and provide a 'force' option that will just remove it from the engine DB and will not attempt gluster peer detach.
something like that. i assume the gluster storage will handle this somehow?
What would you expect gluster storage to do in such a case? If all servers are not accessible to a gluster client, the client can't read/write from/to volumes of the cluster. Cluster management operations in gluster (like removing a server from the cluster) are always done from one of the servers of the cluster. So if no servers are available, nothing can be done. Vijay can shed more light on this if required. Assuming that some of the servers come up at a later point in time, they would continue to consider this (removed from engine) server as one of the peers. This would create an inconsistency between actual gluster configuration and the engine DB. This, however can be handled once we have a feature to sync configuration with gluster (this is WIP). This feature will automatically identify such servers, and allow the user to either import them to engine, or remove (peer detach) from the gluster cluster.
> >> >> Dominic >> >> On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com >> <mailto:emesika@redhat.com>> wrote: >> >> >> >> ----- Original Message ----- >> > From: "Douglas Landgraf" <dougsland@redhat.com >> <mailto:dougsland@redhat.com>> >> > To: "Dominic Kaiser" <dominic@bostonvineyard.org >> <mailto:dominic@bostonvineyard.org>> >> > Cc: "Eli Mesika" <emesika@redhat.com >> <mailto:emesika@redhat.com>>, users@ovirt.org >> <mailto:users@ovirt.org>, "Robert Middleswarth" >> <robert@middleswarth.net <mailto:robert@middleswarth.net>> >> > Sent: Friday, September 21, 2012 8:12:27 PM >> > Subject: Re: [Users] Is there a way to force remove a host? >> > >> > Hi Dominic, >> > >> > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: >> > > Sorry I did not explain. >> > > >> > > I had tried to remove the host and had not luck >> troubleshooting it. >> > > I >> > > then had removed it and used it for a storage unit >> reinstalling >> > > fedora >> > > 17. I foolishly thought that I could just remove the >> host >> > > manually. >> > > It physically is not there. (My fault I know) Is >> there a >> way that >> > > you know of to remove a host brute force. >> > > >> > > dk >> > >> > Fell free to try the below script (not part of official >> project) for >> > brute force: >> > >> > (from the engine side) >> > # yum install python-psycopg2 -y >> > # wget >> > >> >> https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... >> >> >> >> >> >> > # (edit the file and change the db password) >> > # python ./engine_force_remove_Host.py >> >> Hi , had looked in the Python script you had provided: >> First, I must say that handling the database directly may >> leave DB >> in inconsistent state, therefore, if there is no other >> option, the >> database should be backed up prior to this operation. >> In addition, I do not like the execution of the SQL >> statements in >> the script. >> There is a SP called DeleteVds(v_vds_id UUID) and you >> should use >> that since it encapsulates all details. >> For example, your script does not handle permission >> clean-up as >> the >> SP does and therefore leaves garbage in the database. >> In addition, a failure in your script may leave database in >> inconsistent state while the SP is executed in one >> transaction and >> will leave DB consistent. >> So, in short I would prefer in this case that the relevant SP >> will >> do the clean-up since this is the one that is used by the >> code and >> that insures (at least I hope so) , that all related entities >> are >> removed as well. >> >> >> > >> > Thanks >> > >> > -- >> > Cheers >> > Douglas >> > >> > >> >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users
--
@jasonbrooks
On 09/25/2012 01:45 PM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 04:04 PM, Itamar Heim wrote:
On 09/25/2012 12:32 PM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 01:42 PM, Itamar Heim wrote:
On 09/25/2012 09:44 AM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote:
On 09/24/2012 11:53 PM, Jason Brooks wrote: > On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote: >> On 09/24/2012 08:49 PM, Dominic Kaiser wrote: >>> This conversation is fine but if I want to force remove no matter >>> what I >>> should be able to from the GUI. The nodes are no longer >>> available I >>> want to get rid of them ovirt does not let me. I can delete from >>> database but why not from the GUI? I am sure others may run into >>> this >>> problem as well. >> >> what happens to the status of the host when you right click on the >> host and specify you confirm it was shutdown? > > I'm having this same issue. Confirming the host is shut down doesn't > make a difference. > > I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = > Unexpected > exception" errors in my engine log that seem to correspond w/ the > failed > remove host attempts.
is cluster defined as gluster as well? what is the status of the host after you confirm shutdown? any error on log on this specific command?
shireesh - not sure if relevant to this flow, but need to make sure removing a host from the engine isn't blocked on gluster needing to remove it from the gluster cluster if the host is not available any more, or last host in gluster cluster?
Yes, currently the system tries the 'gluster peer detach <hostname>' command when trying to remove a server, which fails if the server is unavailable. This can be enhanced to show the error to user and then allow 'force remove' which can use the 'gluster peer detach <hostname> *force*' command that forcefully removes the server from the cluster, even if it is not available or has bricks on it.
what if it is the last server in the cluster? what if there is another server in the cluster but no communication to it as well?
A quick look at code tells me that in case of virt, we don't allow removing a host if it has VM(s) in it (even if the host is currently not available) i.e. vdsDynamic.getvm_count() > 0. Please correct me if I'm wrong. If that's correct, and if we want to keep it consistent for gluster as well, then we should not allow removing a host if it has gluster volume(s) in it. This is how it behaves in case of 'last server in cluster' today.
true, but user can fence the host or confirm shutdown manually, which will release all resources on it, then it can be removed.
I see. In that case, we can just remove the validation and allow removing the host irrespective of whether it contains volume(s) or not. Since it's the only host in the cluster, this won't cause any harm.
In case of no up server available in the cluster, we can show the error and provide a 'force' option that will just remove it from the engine DB and will not attempt gluster peer detach.
something like that. i assume the gluster storage will handle this somehow?
What would you expect gluster storage to do in such a case? If all servers are not accessible to a gluster client, the client can't read/write from/to volumes of the cluster. Cluster management operations in gluster (like removing a server from the cluster) are always done from one of the servers of the cluster. So if no servers are available, nothing can be done. Vijay can shed more light on this if required.
Assuming that some of the servers come up at a later point in time, they would continue to consider this (removed from engine) server as one of the peers. This would create an inconsistency between actual gluster configuration and the engine DB. This, however can be handled once we have a feature to sync configuration with gluster (this is WIP). This feature will automatically identify such servers, and allow the user to either import them to engine, or remove (peer detach) from the gluster cluster.
why is that an issue though - worst case the server wouldn't appear in the admin console[1] if it is alive, and if it is dead, it is something the gluster cluster is supposed to deal with? [1] though i assume the admin will continue to alert on its presence for being out-of-sync on list of servers in cluster.
> > >> >>> >>> Dominic >>> >>> On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com >>> <mailto:emesika@redhat.com>> wrote: >>> >>> >>> >>> ----- Original Message ----- >>> > From: "Douglas Landgraf" <dougsland@redhat.com >>> <mailto:dougsland@redhat.com>> >>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org >>> <mailto:dominic@bostonvineyard.org>> >>> > Cc: "Eli Mesika" <emesika@redhat.com >>> <mailto:emesika@redhat.com>>, users@ovirt.org >>> <mailto:users@ovirt.org>, "Robert Middleswarth" >>> <robert@middleswarth.net <mailto:robert@middleswarth.net>> >>> > Sent: Friday, September 21, 2012 8:12:27 PM >>> > Subject: Re: [Users] Is there a way to force remove a host? >>> > >>> > Hi Dominic, >>> > >>> > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: >>> > > Sorry I did not explain. >>> > > >>> > > I had tried to remove the host and had not luck >>> troubleshooting it. >>> > > I >>> > > then had removed it and used it for a storage unit >>> reinstalling >>> > > fedora >>> > > 17. I foolishly thought that I could just remove the >>> host >>> > > manually. >>> > > It physically is not there. (My fault I know) Is >>> there a >>> way that >>> > > you know of to remove a host brute force. >>> > > >>> > > dk >>> > >>> > Fell free to try the below script (not part of official >>> project) for >>> > brute force: >>> > >>> > (from the engine side) >>> > # yum install python-psycopg2 -y >>> > # wget >>> > >>> >>> https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... >>> >>> >>> >>> >>> >>> > # (edit the file and change the db password) >>> > # python ./engine_force_remove_Host.py >>> >>> Hi , had looked in the Python script you had provided: >>> First, I must say that handling the database directly may >>> leave DB >>> in inconsistent state, therefore, if there is no other >>> option, the >>> database should be backed up prior to this operation. >>> In addition, I do not like the execution of the SQL >>> statements in >>> the script. >>> There is a SP called DeleteVds(v_vds_id UUID) and you >>> should use >>> that since it encapsulates all details. >>> For example, your script does not handle permission >>> clean-up as >>> the >>> SP does and therefore leaves garbage in the database. >>> In addition, a failure in your script may leave database in >>> inconsistent state while the SP is executed in one >>> transaction and >>> will leave DB consistent. >>> So, in short I would prefer in this case that the relevant SP >>> will >>> do the clean-up since this is the one that is used by the >>> code and >>> that insures (at least I hope so) , that all related entities >>> are >>> removed as well. >>> >>> >>> > >>> > Thanks >>> > >>> > -- >>> > Cheers >>> > Douglas >>> > >>> > >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users > > > > -- > > @jasonbrooks
On Friday 28 September 2012 01:00 PM, Itamar Heim wrote:
On 09/25/2012 01:45 PM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 04:04 PM, Itamar Heim wrote:
On 09/25/2012 12:32 PM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 01:42 PM, Itamar Heim wrote:
On 09/25/2012 09:44 AM, Shireesh Anjal wrote:
On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote: > On 09/24/2012 11:53 PM, Jason Brooks wrote: >> On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote: >>> On 09/24/2012 08:49 PM, Dominic Kaiser wrote: >>>> This conversation is fine but if I want to force remove no >>>> matter >>>> what I >>>> should be able to from the GUI. The nodes are no longer >>>> available I >>>> want to get rid of them ovirt does not let me. I can delete from >>>> database but why not from the GUI? I am sure others may run >>>> into >>>> this >>>> problem as well. >>> >>> what happens to the status of the host when you right click on >>> the >>> host and specify you confirm it was shutdown? >> >> I'm having this same issue. Confirming the host is shut down >> doesn't >> make a difference. >> >> I'm seeing lots of "Failed to GlusterHostRemoveVDS, error = >> Unexpected >> exception" errors in my engine log that seem to correspond w/ the >> failed >> remove host attempts. > > is cluster defined as gluster as well? > what is the status of the host after you confirm shutdown? > any error on log on this specific command? > > shireesh - not sure if relevant to this flow, but need to make sure > removing a host from the engine isn't blocked on gluster needing to > remove it from the gluster cluster if the host is not available any > more, or last host in gluster cluster?
Yes, currently the system tries the 'gluster peer detach <hostname>' command when trying to remove a server, which fails if the server is unavailable. This can be enhanced to show the error to user and then allow 'force remove' which can use the 'gluster peer detach <hostname> *force*' command that forcefully removes the server from the cluster, even if it is not available or has bricks on it.
what if it is the last server in the cluster? what if there is another server in the cluster but no communication to it as well?
A quick look at code tells me that in case of virt, we don't allow removing a host if it has VM(s) in it (even if the host is currently not available) i.e. vdsDynamic.getvm_count() > 0. Please correct me if I'm wrong. If that's correct, and if we want to keep it consistent for gluster as well, then we should not allow removing a host if it has gluster volume(s) in it. This is how it behaves in case of 'last server in cluster' today.
true, but user can fence the host or confirm shutdown manually, which will release all resources on it, then it can be removed.
I see. In that case, we can just remove the validation and allow removing the host irrespective of whether it contains volume(s) or not. Since it's the only host in the cluster, this won't cause any harm.
In case of no up server available in the cluster, we can show the error and provide a 'force' option that will just remove it from the engine DB and will not attempt gluster peer detach.
something like that. i assume the gluster storage will handle this somehow?
What would you expect gluster storage to do in such a case? If all servers are not accessible to a gluster client, the client can't read/write from/to volumes of the cluster. Cluster management operations in gluster (like removing a server from the cluster) are always done from one of the servers of the cluster. So if no servers are available, nothing can be done. Vijay can shed more light on this if required.
Assuming that some of the servers come up at a later point in time, they would continue to consider this (removed from engine) server as one of the peers. This would create an inconsistency between actual gluster configuration and the engine DB. This, however can be handled once we have a feature to sync configuration with gluster (this is WIP). This feature will automatically identify such servers, and allow the user to either import them to engine, or remove (peer detach) from the gluster cluster.
why is that an issue though - worst case the server wouldn't appear in the admin console[1] if it is alive, and if it is dead, it is something the gluster cluster is supposed to deal with?
It's just that I think it's not good to have the management console being out of sync with gluster configuration. However, as I said, we will soon have a mechanism to handle such cases. Also, we're thinking of a simpler approach by just providing a 'force remove' checkbox on the remove host confirmation dialog (only if the host belongs to a gluster enabled cluster). User can then tick this checkbox when normal remove flow doesn't work in above discussed scenarios.
[1] though i assume the admin will continue to alert on its presence for being out-of-sync on list of servers in cluster.
Yes - this feature is WIP.
> >> >> >>> >>>> >>>> Dominic >>>> >>>> On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika@redhat.com >>>> <mailto:emesika@redhat.com>> wrote: >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> > From: "Douglas Landgraf" <dougsland@redhat.com >>>> <mailto:dougsland@redhat.com>> >>>> > To: "Dominic Kaiser" <dominic@bostonvineyard.org >>>> <mailto:dominic@bostonvineyard.org>> >>>> > Cc: "Eli Mesika" <emesika@redhat.com >>>> <mailto:emesika@redhat.com>>, users@ovirt.org >>>> <mailto:users@ovirt.org>, "Robert Middleswarth" >>>> <robert@middleswarth.net <mailto:robert@middleswarth.net>> >>>> > Sent: Friday, September 21, 2012 8:12:27 PM >>>> > Subject: Re: [Users] Is there a way to force remove a >>>> host? >>>> > >>>> > Hi Dominic, >>>> > >>>> > On 09/20/2012 12:11 PM, Dominic Kaiser wrote: >>>> > > Sorry I did not explain. >>>> > > >>>> > > I had tried to remove the host and had not luck >>>> troubleshooting it. >>>> > > I >>>> > > then had removed it and used it for a storage unit >>>> reinstalling >>>> > > fedora >>>> > > 17. I foolishly thought that I could just remove the >>>> host >>>> > > manually. >>>> > > It physically is not there. (My fault I know) Is >>>> there a >>>> way that >>>> > > you know of to remove a host brute force. >>>> > > >>>> > > dk >>>> > >>>> > Fell free to try the below script (not part of official >>>> project) for >>>> > brute force: >>>> > >>>> > (from the engine side) >>>> > # yum install python-psycopg2 -y >>>> > # wget >>>> > >>>> >>>> https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.p... >>>> >>>> >>>> >>>> >>>> >>>> >>>> > # (edit the file and change the db password) >>>> > # python ./engine_force_remove_Host.py >>>> >>>> Hi , had looked in the Python script you had provided: >>>> First, I must say that handling the database directly may >>>> leave DB >>>> in inconsistent state, therefore, if there is no other >>>> option, the >>>> database should be backed up prior to this operation. >>>> In addition, I do not like the execution of the SQL >>>> statements in >>>> the script. >>>> There is a SP called DeleteVds(v_vds_id UUID) and you >>>> should use >>>> that since it encapsulates all details. >>>> For example, your script does not handle permission >>>> clean-up as >>>> the >>>> SP does and therefore leaves garbage in the database. >>>> In addition, a failure in your script may leave database in >>>> inconsistent state while the SP is executed in one >>>> transaction and >>>> will leave DB consistent. >>>> So, in short I would prefer in this case that the >>>> relevant SP >>>> will >>>> do the clean-up since this is the one that is used by the >>>> code and >>>> that insures (at least I hope so) , that all related >>>> entities >>>> are >>>> removed as well. >>>> >>>> >>>> > >>>> > Thanks >>>> > >>>> > -- >>>> > Cheers >>>> > Douglas >>>> > >>>> > >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users@ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users@ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >> >> >> >> -- >> >> @jasonbrooks > >
participants (7)
-
Andrew Cathrow -
Dominic Kaiser -
Douglas Landgraf -
Eli Mesika -
Itamar Heim -
Jason Brooks -
Shireesh Anjal