[ovirt-users] Recovering from a multi-node failure

Sahina Bose sabose at redhat.com
Wed Aug 16 12:22:46 UTC 2017


On Sun, Aug 6, 2017 at 4:42 AM, Jim Kusznir <jim at palousetech.com> wrote:

> Well, after a very stressful weekend, I think I have things largely
> working.  Turns out that most of the above issues were caused by the linux
> permissions of the exports for all three volumes (they had been reset to
> 600; setting them to 774 or 770 fixed many of the issues).  Of course, I
> didn't find that until a much more harrowing outage, and hours and hours of
> work, including beginning to look at rebuilding my cluster....
>
> So, now my cluster is operating again, and everything looks good EXCEPT
> for one major Gluster issue/question that I haven't found any references or
> info on.
>
> my host ovirt2, one of the replica gluster servers, is the one that lost
> its storage and had to reinitialize it from the cluster.  the iso volume is
> perfectly fine and complete, but the engine and data volumes are smaller on
> disk on this node than on the other node (and this node before the crash).
> On the engine store, the entire cluster reports the smaller utilization on
> mounted gluster filesystems; on the data partition, it reports the larger
> size (rest of cluster).  Here's some df statments to help clarify:
>
> (brick1 = engine; brick2=data, brick4=iso):
> Filesystem                     Size  Used Avail Use% Mounted on
> /dev/mapper/gluster-engine      25G   12G   14G  47% /gluster/brick1
> /dev/mapper/gluster-data       136G  125G   12G  92% /gluster/brick2
> /dev/mapper/gluster-iso         25G  7.3G   18G  29% /gluster/brick4
> 192.168.8.11:/engine            15G  9.7G  5.4G  65%
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine
> 192.168.8.11:/data             136G  125G   12G  92%
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_data
> 192.168.8.11:/iso               13G  7.3G  5.8G  56%
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_iso
>
> View from ovirt2:
> Filesystem                     Size  Used Avail Use% Mounted on
> /dev/mapper/gluster-engine      15G  9.7G  5.4G  65% /gluster/brick1
> /dev/mapper/gluster-data       174G  119G   56G  69% /gluster/brick2
> /dev/mapper/gluster-iso         13G  7.3G  5.8G  56% /gluster/brick4
> 192.168.8.11:/engine            15G  9.7G  5.4G  65%
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine
> 192.168.8.11:/data             136G  125G   12G  92%
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_data
> 192.168.8.11:/iso               13G  7.3G  5.8G  56%
> /rhev/data-center/mnt/glusterSD/192.168.8.11:_iso
>
> As you can see, in the process of rebuilding the hard drive for ovirt2, I
> did resize some things to give more space to data, where I desperately need
> it.  If this goes well and the storage is given a clean bill of health at
> this time, then I will take ovirt1 down and resize to match ovirt2, and
> thus score a decent increase in storage for data.  I fully realize that
> right now the gluster mounted volumes should have the total size as the
> least common denominator.
>
> So, is this size reduction appropriate?  A big part of me thinks data is
> missing, but I even went through and shut down ovirt2's gluster daemons,
> wiped all the gluster data, and restarted gluster to allow it a fresh heal
> attempt, and it again came back to the exact same size.  This cluster was
> originally built about the time ovirt 4.0 came out, and has been upgraded
> to 'current', so perhaps some new gluster features are making more
> efficient use of space (dedupe or something)?
>

The used capacity should be consistent on all nodes - I see you have a
discrepancy with the data volume brick. What does "gluster vol heal data
info" tell you? Are there entries to be healed?

Can you provide the glustershd logs?



>
> Thank  you for your assistance!
> --JIm
>
> On Fri, Aug 4, 2017 at 7:49 PM, Jim Kusznir <jim at palousetech.com> wrote:
>
>> Hi all:
>>
>> Today has been rough.  two of my three nodes went down today, and self
>> heal has not been healing well.  4 hours later, VMs are running.  but the
>> engine is not happy.  It claims the storage domain is down (even though it
>> is up on all hosts and VMs are running).  I'm getting a ton of these
>> messages logging:
>>
>> VDSM engine3 command HSMGetAllTasksStatusesVDS failed: Not SPM
>>
>> Aug 4, 2017 7:23:00 PM
>>
>> VDSM engine3 command SpmStatusVDS failed: Error validating master storage
>> domain: ('MD read error',)
>>
>> Aug 4, 2017 7:22:49 PM
>>
>> VDSM engine3 command ConnectStoragePoolVDS failed: Cannot find master
>> domain: u'spUUID=5868392a-0148-02cf-014d-000000000121,
>> msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770'
>>
>> Aug 4, 2017 7:22:47 PM
>>
>> VDSM engine1 command ConnectStoragePoolVDS failed: Cannot find master
>> domain: u'spUUID=5868392a-0148-02cf-014d-000000000121,
>> msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770'
>>
>> Aug 4, 2017 7:22:46 PM
>>
>> VDSM engine2 command SpmStatusVDS failed: Error validating master storage
>> domain: ('MD read error',)
>>
>> Aug 4, 2017 7:22:44 PM
>>
>> VDSM engine2 command ConnectStoragePoolVDS failed: Cannot find master
>> domain: u'spUUID=5868392a-0148-02cf-014d-000000000121,
>> msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770'
>>
>> Aug 4, 2017 7:22:42 PM
>>
>> VDSM engine1 command HSMGetAllTasksStatusesVDS failed: Not SPM: ()
>>
>>
>> ------------
>> I cannot set an SPM as it claims the storage domain is down; I cannot set
>> the storage domain up.
>>
>> Also in the storage realm, one of my exports shows substantially less
>> data than is actually there.
>>
>> Here's what happened, as best as I understood them:
>> I went to do maintence on ovirt2 (needed to replace a faulty ram stick
>> and rework the disk).  I put it in maintence mode, then shut it down and
>> did my work.  In the process, much of the disk contents was lost (all the
>> gluster data).  I figure, no big deal, the gluster data is redundant on the
>> network, it will heal when it comes back up.
>>
>> While I was doing maintence, all but one of the VMs were running on
>> engine1.  When I turned on engine2, all of the sudden, all vms including
>> the main engine stop and go non-responsive.  As far as I can tell, this
>> should not have happened, as I turned ON one host, but none the less, I
>> waited for recovery to occur (while customers started calling asking why
>> everything stopped working....).  As I waited, I  was checking, and gluster
>> volume status only showed ovirt1 and ovirt2....Apparently gluster had
>> stopped/failed at some point on ovirt3.  I assume that was the cause of the
>> outage, still, if everything was working fine with ovirt1 gluster, and
>> ovirt2 powers on with a very broke gluster (the volume status was showing
>> NA for the port fileds for the gluster volumes), I would not expect to have
>> a working gluster go stupid like that.
>>
>> After starting ovirt3 glusterd and checking the status, all three showed
>> ovirt1 and ovirt3 as operational, and ovirt2 as NA.  Unfortunately,
>> recovery was still not happening, so I did some googling and found about
>> the commands to inquire about the hosted-engine status.  It appeared to be
>> stuck "paused" and I couldn't find a way to unpause it, so I poweroff'ed
>> it, then started it manually on engine 1, and the cluster came back up.  It
>> showed all VMs paused.  I was able to unpause them and they worked again.
>>
>> So now I began to work the ovirt2 gluster healing problem.  It didn't
>> appear to be self-healing, but eventually I found this document:
>> https://support.rackspace.com/how-to/recover-from-a-failed-s
>> erver-in-a-glusterfs-array/
>> and from that found the magic xattr commands.  After setting them,
>> gluster volumes on ovirt2 came online.  I told iso to heal, and it did but
>> only came up about half as much data as it should have.  I told it heal
>> full, and it did finish off the remaining data, and came up to full.  I
>> then told engine to do a full heal (gluster volume heal engine full), and
>> it transferred its data from the other gluster hosts too.  However, it said
>> it was done when it hit 9.7GB while there was 15GB on disk!  It is still
>> stuck that way; ovirt gui and gluster volume heal engine info both show the
>> volume fully healed, but it is not:
>> [root at ovirt1 ~]# df -h
>> Filesystem                     Size  Used Avail Use% Mounted on
>> /dev/mapper/centos_ovirt-root   20G  4.2G   16G  21% /
>> devtmpfs                        16G     0   16G   0% /dev
>> tmpfs                           16G   16K   16G   1% /dev/shm
>> tmpfs                           16G   26M   16G   1% /run
>> tmpfs                           16G     0   16G   0% /sys/fs/cgroup
>> /dev/mapper/gluster-engine      25G   12G   14G  47% /gluster/brick1
>> /dev/sda1                      497M  315M  183M  64% /boot
>> /dev/mapper/gluster-data       136G  124G   13G  92% /gluster/brick2
>> /dev/mapper/gluster-iso         25G  7.3G   18G  29% /gluster/brick4
>> tmpfs                          3.2G     0  3.2G   0% /run/user/0
>> 192.168.8.11:/engine            15G  9.7G  5.4G  65%
>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine
>> 192.168.8.11:/data             136G  124G   13G  92%
>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_data
>> 192.168.8.11:/iso               13G  7.3G  5.8G  56%
>> /rhev/data-center/mnt/glusterSD/192.168.8.11:_iso
>>
>> This is from ovirt1, and before the work, both ovirt1 and ovirt2's brings
>> had the same usage.  ovirt2's bricks and the gluster mountpoints agree on
>> iso and engine, but as you can see, not here.  If I do a du -sh on
>> /rhev/data-center/mnt/glusterSD/..../_engine, it comes back with the
>> 12GB number (/brick1 is engine, brick2 is data and brick4 is iso).
>> However, gluster still says its only 9.7G.  I haven't figured out how to
>> get it to finish "healing".
>>
>> data is in the process of healing currently.
>>
>> So, I think I have two main things to solve right now:
>>
>> 1) how do I get ovirt to see the data center/storage domain as online
>> again?
>> 2) How do I get engine to finish healing to ovirt2?
>>
>> Thanks all for reading this very long message!
>> --Jim
>>
>>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170816/c5b8179e/attachment.html>


More information about the Users mailing list