[ovirt-users] sanlock + gluster recovery -- RFE

[Ted Miller]

Itamar, I am addressing this to you because one of your assignments seems to 
be to coordinate other oVirt contributors when dealing with issues that are 
raised on the ovirt-users email list.

As you are aware, there is an ongoing split-brain problem with running 
sanlock on replicated gluster storage.  Personally, I believe that this is 
the 5th time that I have been bitten by this sanlock+gluster problem.

I believe that the following are true (if not, my entire request is probably 
off base).

  * ovirt uses sanlock in such a way that when the sanlock storage is on a
    replicated gluster file system, very small storage disruptions can result
    in a gluster split-brain on the sanlock space
      o gluster is aware of the problem, and is working on a different way of
        replicating data, which will reduce these problems.
  * most (maybe all) of the sanlock locks have a short duration, measured in
    seconds
  * there are only a couple of things that a user can safely do from the
    command line when a file is in split-brain
      o delete the file
      o rename (mv) the file
  * x

_How did I get into this mess?_

had 3 hosts running ovirt 3.3
     each hosted VMs
     gluster replica 3 storage
     engine was external to cluster
upgraded 3 hosts from ovirt 3.3 to 3.4
hosted-engine deploy
     used new gluster volume (accessed via nfs) for storage
         storage was accessed using localhost:engVM1 link (localhost was 
probably a poor choice)
     created new engine on VM (did not transfer any data from old engine)
added 3 hosts to new engine via web-gui
ran above setup for 3 days
shut entire system down before I left on vacation (holiday)
came back from vacation
powered on hosts
found that iptables did not have rules for gluster access
     (a continuing problem if host installation is allowed to set up firewall)
added rules for gluster
glusterfs now up and running
added storage manually
tried "hosted-engine --vm-start"
vm did not start
logs show sanlock errors
"gluster volume heal engVM1full:
"gluster volume heal engVM1 info split-brain" showed 6 files in split-brain
         all 5 prefixed by /rhev/data-center/mnt/localhost\:_engVM1
     UUID/dom_md/ids
     UUID/images/UUID/UUID (VM hard disk)
     UUID/images/UUID/UUID.lease
     UUID/ha_agent/hosted-engine.lockspace
     UUID/ha_agent/hosted-engine.metadata
I copied each of the above files off of each of the three bricks to a safe 
place (15 files copied)
I renamed the 5 files on /rhev/....
I copied the 5 files from one of the bricks to /rhev/
     files can now be read OK (e.g. cat ids)
sanlock.log shows error sets like these:

2014-05-20 03:23:39-0400 36199 [2843]: s3358 lockspace 5ebb3b40-a394-405b-bbac-4c0e21ccd659:1:/rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids:0
2014-05-20 03:23:39-0400 36199 [18873]: open error -5 /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids
2014-05-20 03:23:39-0400 36199 [18873]: s3358 open_disk /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids error -5
2014-05-20 03:23:40-0400 36200 [2843]: s3358 add_lockspace fail result -19

I am now stuck

What I would like to see in ovirt to help me (and others like me). Alternates 
listed in order from most desirable (automatic) to least desirable (set of 
commands to type, with lots of variables to figure out).

1. automagic recovery

  *   When a host is not able to access sanlock, it writes a small "problem"
    text file into the shared storage
      o the host-ID as part of the name (so only one host ever accesses that
        file)
      o a status number for the error causing problems
      o time stamp
      o time stamp when last sanlock lease will expire
      o if sanlock is able to access the file, the "problem" file is deleted
  * when time passes for its last sanlock lease to be expired, highest number
    host does a survey
      o did all other hosts create "problem" files?
      o do all "problem" files show same (or compatible) error codes related
        to file access problems?
      o are all hosts communicating by network?
      o if yes to all above
  * delete all sanlock storage space
  * initialize sanlock from scratch
  * restart whatever may have given up because of sanlock
  * restart VM if necessary

2. recovery subcommand

  * add "hosted-engine --lock-initialize" command that would delete sanlock,
    start over from scratch

3. script

  * publish a script (in ovirt packages or available on web) which, when run,
    does all (or most) of the recovery process needed.

4. commands

  * publish on the web a "recipe" for dealing with files that commonly go
    split-brain
      o ids
      o *.lease
      o *.lockspace

Any chance of any help on any of the above levels?

Ted Miller
Elkhart, IN, USA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140520/af5eab45/attachment-0001.html>