[+Sas - thanks for the link to virt-store usecase article inline] On 12/18/2014 06:56 PM, Brent Hartzell wrote:
Hello,
I had actually gotten this sorted out, somewhat. If I disable server quorum on the volume, the storage domain will activate. The volume is/was optimized for virt store via oVirt. The brick in question was not the first brick added to the volume through oVirt however, it appears that it may have been the first brick in the replica being used, but I'm not certain how to find this out.
The recommended setting is to have both client and server side quorum turned on. But turning on server-side quorum with a 2-way replica volume would mean that your volume goes offline when one of the bricks goes down. "gluster volume info" command will give you information about the volume topology. So will the bricks sub-tab for Volume in oVirt. The order in which the bricks are listed, is the order of the replica sets.
Disabling quorum allowed me to get the VM's affected back online however, is this the recommended procedure? I tried to use replace-brick with another node but it failed because the failed brick was not available. Would we leave quorum disabled until that brick gets replaced? IE - rebuild the server with the same hostname/IP file structure and rebalance the cluster?
http://www.gluster.org/community/documentation/index.php/Virt-store-usecase - for recommendations on volume tunables. You could add another brick to your volume to make it a replica 3 and then turn on quorum? For help on recovering your volume, I suggest you write to gluster-users@gluster.org
////
While that happened, I read somewhere about this happening with a replica 2 - I've created a new volume with replica 3 and plan to test this again. Is there any info you can point me to for how to handle this when it happens or what the correct procedure is when a "first" brick fails?
-----Original Message----- From: Sahina Bose [mailto:sabose@redhat.com] Sent: Thursday, December 18, 2014 3:51 AM To: Vered Volansky; Brent Hartzell Cc: users@ovirt.org Subject: Re: [ovirt-users] Cannot activate storage domain
Adding Sahina.
----- Original Message -----
From: "Brent Hartzell" <brent.hartzell@outlook.com> To: users@ovirt.org Sent: Thursday, December 18, 2014 3:38:11 AM Subject: [ovirt-users] Cannot activate storage domain
Have the following:
6 hosts - virt + Gluster shared
Gluster volume is distributed-replicate - replica 2
Shutting down servers one at a time all work except for 1 brick. If we shut down one specific brick (1 brick per host) - we're unable to activate the storage domain. VM's that were actively running from other bricks continue to run. Whatever was running form that specific brick fails to run, gets paused etc.
Error log shows the entry below. I'm not certain what it's saying is read only.nothing is read only that I can find.
2014-12-17 19:57:13,362 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [4e9290a2] Command SpmStatusVDSCommand(HostName = U23.domainame.net, HostId = 0db58e46-68a3-4ba0-a8aa-094893c045a1, storagePoolId = 7ccd6ea9-7d80-4170-afa1-64c10c185aa6) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SpmStatusVDS, error = [Errno 30] Read-only file system, code = 100
2014-12-17 19:57:13,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData] (DefaultQuartzScheduler_Worker-47) [4e9290a2] hostFromVds::selectedVds - U23.domainname.net, spmStatus returned null!
According to Ovirt/Gluster, if a brick goes down, the VM should be able to be restarted from another brick without issue. This does not appear to be the case. If we take other bricks offline, it appears to work as expected. Something with this specific brick cases everything to break which then makes any VM's that were running from the brick unable to start. Do you have the recommended options for using volume as virt store turned on? Is client-side quorum turned on for the volume? Is the brick that causes
On 12/18/2014 01:35 PM, Vered Volansky wrote: the issue, the first brick in the replica set?
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users