[ovirt-users] Change gluster primary

[Alex Crow]

On 20/01/15 20:46, John Gardeniers wrote:
> Hi Alex,
>
> I understand what you're saying and certainly there is no primary from 
> the Gluster perspective. However, things are quite different as far as 
> Ovirt/RHEV is concerned.
>
> We had an incident last week where we had to take nix off-line. A 
> network glitch then caused a our RHEV to briefly lose connection to 
> jupiter. This resulted in all VMs crashing because the system was 
> trying to reconnect to nix. It did not try to reconnect to jupiter, 
> despite it being configured as the fail-over server. 

Hi,

As for the above, if you had quorum configured on the gluster side 
(either by applying the relevant recommended options on gluster, or by 
having created the volume on ovirt), loss of storage functionality is to 
be expected. In a two-node cluster if one goes down you lose quorum and 
the volume will become read only.  In this case Ovirt should really 
pause the VMs.

> In the end I had to bring nix back on line. RHEV still wouldn't 
> connect. Finally, I had to reboot each hypervisor. Even then, two of 
> them still failed to reconnect and could only be brought back by 
> performing a full reinstall (we're using the cut-down dedicated RH 
> hypervisors, not the RHEL+hypervisor that you use). All in all, quite 
> a disastrous situation that lost us a couple of hours. So yes, there 
> is a primary from the Ovirt/RHEV perspective and I'm really 
> disappointed in how the system completely failed to handled the 
> situation.

Looks like there are some bugs there. When we have had storage issues on 
RHEV we see all our VMS pausing, not crashing. BTW we do use the ded. 
hypervisor (like oVirt "node).

Cheers

Alex


>
> regards,
> John
>
>
> On 21/01/15 00:20, Alex Crow wrote:
>> Hi John,
>>
>> There isn't really a primary in gluster. If you're using a glusterfs 
>> storage domain, you could turn off "nix" and the VMs would continue 
>> to run (although you'd have to disable quorum if you currently have 
>> it enabled on the volume, and you'd have to repoint the domain at 
>> some later point). If you're using NFS access you would have to 
>> repoint your storage to the remaining machine immediately.
>>
>> The only snag I can see is that you can't detach the master storage 
>> domain in Ovirt if any VMs are running. I think you'd have to shut 
>> the VMs down, put the storage domain into maintenance, and then edit it.
>>
>> Cheers
>>
>> Alex
>>
>> On 19/01/15 23:44, John Gardeniers wrote:
>>> We are using Gluster as our storage backend. Gluster is configured 
>>> as 2 node replica. The two nodes are name nix and jupiter. At the 
>>> Ovirt (RHEV really) end we have the gluster path configured as 
>>> "nix:/gluster-rhev", with a mount option of 
>>> "backupvolfile-server=jupiter.om.net". We now need to replace nix 
>>> with a new server, which cannot have the same name. That new server 
>>> will be the primary, with jupiter remaining the secondary.
>>>
>>> We will have all VMs and hypervisors shut down when we make this 
>>> change.
>>>
>>> What is the best and/or easiest way to do this? Should we just 
>>> disconnect the storage and re-attach it using the new gluster 
>>> primary? If we do that will our VMs just work or do we need to take 
>>> other steps?
>>>
>>> An alternative, which I suspect will be somewhat controversial, 
>>> would be to make a direct edit of the engine database. Would that 
>>> work any better or does that add more dangers (assuming the edit is 
>>> done correctly)?
>>>
>>> regards,
>>> John
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>