[Users] Is there a way to force remove a host?
Itamar Heim
iheim at redhat.com
Fri Sep 28 07:30:37 UTC 2012
On 09/25/2012 01:45 PM, Shireesh Anjal wrote:
> On Tuesday 25 September 2012 04:04 PM, Itamar Heim wrote:
>> On 09/25/2012 12:32 PM, Shireesh Anjal wrote:
>>> On Tuesday 25 September 2012 01:42 PM, Itamar Heim wrote:
>>>> On 09/25/2012 09:44 AM, Shireesh Anjal wrote:
>>>>> On Tuesday 25 September 2012 03:25 AM, Itamar Heim wrote:
>>>>>> On 09/24/2012 11:53 PM, Jason Brooks wrote:
>>>>>>> On Mon 24 Sep 2012 01:24:44 PM PDT, Itamar Heim wrote:
>>>>>>>> On 09/24/2012 08:49 PM, Dominic Kaiser wrote:
>>>>>>>>> This conversation is fine but if I want to force remove no matter
>>>>>>>>> what I
>>>>>>>>> should be able to from the GUI. The nodes are no longer
>>>>>>>>> available I
>>>>>>>>> want to get rid of them ovirt does not let me. I can delete from
>>>>>>>>> database but why not from the GUI? I am sure others may run into
>>>>>>>>> this
>>>>>>>>> problem as well.
>>>>>>>>
>>>>>>>> what happens to the status of the host when you right click on the
>>>>>>>> host and specify you confirm it was shutdown?
>>>>>>>
>>>>>>> I'm having this same issue. Confirming the host is shut down doesn't
>>>>>>> make a difference.
>>>>>>>
>>>>>>> I'm seeing lots of "Failed to GlusterHostRemoveVDS, error =
>>>>>>> Unexpected
>>>>>>> exception" errors in my engine log that seem to correspond w/ the
>>>>>>> failed
>>>>>>> remove host attempts.
>>>>>>
>>>>>> is cluster defined as gluster as well?
>>>>>> what is the status of the host after you confirm shutdown?
>>>>>> any error on log on this specific command?
>>>>>>
>>>>>> shireesh - not sure if relevant to this flow, but need to make sure
>>>>>> removing a host from the engine isn't blocked on gluster needing to
>>>>>> remove it from the gluster cluster if the host is not available any
>>>>>> more, or last host in gluster cluster?
>>>>>
>>>>> Yes, currently the system tries the 'gluster peer detach <hostname>'
>>>>> command when trying to remove a server, which fails if the server is
>>>>> unavailable. This can be enhanced to show the error to user and then
>>>>> allow 'force remove' which can use the 'gluster peer detach <hostname>
>>>>> *force*' command that forcefully removes the server from the cluster,
>>>>> even if it is not available or has bricks on it.
>>>>
>>>> what if it is the last server in the cluster?
>>>> what if there is another server in the cluster but no communication to
>>>> it as well?
>>>
>>> A quick look at code tells me that in case of virt, we don't allow
>>> removing a host if it has VM(s) in it (even if the host is currently
>>> not available) i.e. vdsDynamic.getvm_count() > 0. Please correct me if
>>> I'm wrong. If that's correct, and if we want to keep it consistent for
>>> gluster as well, then we should not allow removing a host if it has
>>> gluster volume(s) in it. This is how it behaves in case of 'last server
>>> in cluster' today.
>>
>> true, but user can fence the host or confirm shutdown manually, which
>> will release all resources on it, then it can be removed.
>
> I see. In that case, we can just remove the validation and allow
> removing the host irrespective of whether it contains volume(s) or not.
> Since it's the only host in the cluster, this won't cause any harm.
>
>>
>>>
>>> In case of no up server available in the cluster, we can show the error
>>> and provide a 'force' option that will just remove it from the engine DB
>>> and will not attempt gluster peer detach.
>>
>> something like that.
>> i assume the gluster storage will handle this somehow?
>
> What would you expect gluster storage to do in such a case? If all
> servers are not accessible to a gluster client, the client can't
> read/write from/to volumes of the cluster. Cluster management operations
> in gluster (like removing a server from the cluster) are always done
> from one of the servers of the cluster. So if no servers are available,
> nothing can be done. Vijay can shed more light on this if required.
>
> Assuming that some of the servers come up at a later point in time, they
> would continue to consider this (removed from engine) server as one of
> the peers. This would create an inconsistency between actual gluster
> configuration and the engine DB. This, however can be handled once we
> have a feature to sync configuration with gluster (this is WIP). This
> feature will automatically identify such servers, and allow the user to
> either import them to engine, or remove (peer detach) from the gluster
> cluster.
why is that an issue though - worst case the server wouldn't appear in
the admin console[1] if it is alive, and if it is dead, it is something
the gluster cluster is supposed to deal with?
[1] though i assume the admin will continue to alert on its presence for
being out-of-sync on list of servers in cluster.
>
>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Dominic
>>>>>>>>>
>>>>>>>>> On Sep 22, 2012 4:19 PM, "Eli Mesika" <emesika at redhat.com
>>>>>>>>> <mailto:emesika at redhat.com>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>> > From: "Douglas Landgraf" <dougsland at redhat.com
>>>>>>>>> <mailto:dougsland at redhat.com>>
>>>>>>>>> > To: "Dominic Kaiser" <dominic at bostonvineyard.org
>>>>>>>>> <mailto:dominic at bostonvineyard.org>>
>>>>>>>>> > Cc: "Eli Mesika" <emesika at redhat.com
>>>>>>>>> <mailto:emesika at redhat.com>>, users at ovirt.org
>>>>>>>>> <mailto:users at ovirt.org>, "Robert Middleswarth"
>>>>>>>>> <robert at middleswarth.net <mailto:robert at middleswarth.net>>
>>>>>>>>> > Sent: Friday, September 21, 2012 8:12:27 PM
>>>>>>>>> > Subject: Re: [Users] Is there a way to force remove a host?
>>>>>>>>> >
>>>>>>>>> > Hi Dominic,
>>>>>>>>> >
>>>>>>>>> > On 09/20/2012 12:11 PM, Dominic Kaiser wrote:
>>>>>>>>> > > Sorry I did not explain.
>>>>>>>>> > >
>>>>>>>>> > > I had tried to remove the host and had not luck
>>>>>>>>> troubleshooting it.
>>>>>>>>> > > I
>>>>>>>>> > > then had removed it and used it for a storage unit
>>>>>>>>> reinstalling
>>>>>>>>> > > fedora
>>>>>>>>> > > 17. I foolishly thought that I could just remove the
>>>>>>>>> host
>>>>>>>>> > > manually.
>>>>>>>>> > > It physically is not there. (My fault I know) Is
>>>>>>>>> there a
>>>>>>>>> way that
>>>>>>>>> > > you know of to remove a host brute force.
>>>>>>>>> > >
>>>>>>>>> > > dk
>>>>>>>>> >
>>>>>>>>> > Fell free to try the below script (not part of official
>>>>>>>>> project) for
>>>>>>>>> > brute force:
>>>>>>>>> >
>>>>>>>>> > (from the engine side)
>>>>>>>>> > # yum install python-psycopg2 -y
>>>>>>>>> > # wget
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> https://raw.github.com/dougsland/misc-rhev/master/engine_force_remove_Host.py
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> > # (edit the file and change the db password)
>>>>>>>>> > # python ./engine_force_remove_Host.py
>>>>>>>>>
>>>>>>>>> Hi , had looked in the Python script you had provided:
>>>>>>>>> First, I must say that handling the database directly may
>>>>>>>>> leave DB
>>>>>>>>> in inconsistent state, therefore, if there is no other
>>>>>>>>> option, the
>>>>>>>>> database should be backed up prior to this operation.
>>>>>>>>> In addition, I do not like the execution of the SQL
>>>>>>>>> statements in
>>>>>>>>> the script.
>>>>>>>>> There is a SP called DeleteVds(v_vds_id UUID) and you
>>>>>>>>> should use
>>>>>>>>> that since it encapsulates all details.
>>>>>>>>> For example, your script does not handle permission
>>>>>>>>> clean-up as
>>>>>>>>> the
>>>>>>>>> SP does and therefore leaves garbage in the database.
>>>>>>>>> In addition, a failure in your script may leave database in
>>>>>>>>> inconsistent state while the SP is executed in one
>>>>>>>>> transaction and
>>>>>>>>> will leave DB consistent.
>>>>>>>>> So, in short I would prefer in this case that the relevant SP
>>>>>>>>> will
>>>>>>>>> do the clean-up since this is the one that is used by the
>>>>>>>>> code and
>>>>>>>>> that insures (at least I hope so) , that all related entities
>>>>>>>>> are
>>>>>>>>> removed as well.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> >
>>>>>>>>> > Thanks
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > Cheers
>>>>>>>>> > Douglas
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list
>>>>>>>>> Users at ovirt.org
>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at ovirt.org
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> @jasonbrooks
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
More information about the Users
mailing list