[ovirt-users] sanlock + gluster recovery -- RFE

[Itamar Heim]

On 05/21/2014 12:31 AM, Ted Miller wrote:
> Itamar, I am addressing this to you because one of your assignments
> seems to be to coordinate other oVirt contributors when dealing with
> issues that are raised on the ovirt-users email list.
>
> As you are aware, there is an ongoing split-brain problem with running
> sanlock on replicated gluster storage.  Personally, I believe that this
> is the 5th time that I have been bitten by this sanlock+gluster problem.
>
> I believe that the following are true (if not, my entire request is
> probably off base).
>
>   * ovirt uses sanlock in such a way that when the sanlock storage is on
>     a replicated gluster file system, very small storage disruptions can
>     result in a gluster split-brain on the sanlock space
>       o gluster is aware of the problem, and is working on a different
>         way of replicating data, which will reduce these problems.
>   * most (maybe all) of the sanlock locks have a short duration,
>     measured in seconds
>   * there are only a couple of things that a user can safely do from the
>     command line when a file is in split-brain
>       o delete the file
>       o rename (mv) the file
>   * x
>
> _How did I get into this mess?_
>
> had 3 hosts running ovirt 3.3
>      each hosted VMs
>      gluster replica 3 storage
>      engine was external to cluster
> upgraded 3 hosts from ovirt 3.3 to 3.4
> hosted-engine deploy
>      used new gluster volume (accessed via nfs) for storage
>          storage was accessed using localhost:engVM1 link (localhost was
> probably a poor choice)
>      created new engine on VM (did not transfer any data from old engine)
> added 3 hosts to new engine via web-gui
> ran above setup for 3 days
> shut entire system down before I left on vacation (holiday)
> came back from vacation
> powered on hosts
> found that iptables did not have rules for gluster access
>      (a continuing problem if host installation is allowed to set up
> firewall)
> added rules for gluster
> glusterfs now up and running
> added storage manually
> tried "hosted-engine --vm-start"
> vm did not start
> logs show sanlock errors
> "gluster volume heal engVM1full:
> "gluster volume heal engVM1 info split-brain" showed 6 files in split-brain
>          all 5 prefixed by /rhev/data-center/mnt/localhost\:_engVM1
>      UUID/dom_md/ids
>      UUID/images/UUID/UUID (VM hard disk)
>      UUID/images/UUID/UUID.lease
>      UUID/ha_agent/hosted-engine.lockspace
>      UUID/ha_agent/hosted-engine.metadata
> I copied each of the above files off of each of the three bricks to a
> safe place (15 files copied)
> I renamed the 5 files on /rhev/....
> I copied the 5 files from one of the bricks to /rhev/
>      files can now be read OK (e.g. cat ids)
> sanlock.log shows error sets like these:
>
> 2014-05-20 03:23:39-0400 36199 [2843]: s3358 lockspace 5ebb3b40-a394-405b-bbac-4c0e21ccd659:1:/rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids:0
> 2014-05-20 03:23:39-0400 36199 [18873]: open error -5 /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids
> 2014-05-20 03:23:39-0400 36199 [18873]: s3358 open_disk /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids error -5
> 2014-05-20 03:23:40-0400 36200 [2843]: s3358 add_lockspace fail result -19
>
> I am now stuck
>
> What I would like to see in ovirt to help me (and others like me).
> Alternates listed in order from most desirable (automatic) to least
> desirable (set of commands to type, with lots of variables to figure out).
>
> 1. automagic recovery
>
>   *   When a host is not able to access sanlock, it writes a small
>     "problem" text file into the shared storage
>       o the host-ID as part of the name (so only one host ever accesses
>         that file)
>       o a status number for the error causing problems
>       o time stamp
>       o time stamp when last sanlock lease will expire
>       o if sanlock is able to access the file, the "problem" file is deleted
>   * when time passes for its last sanlock lease to be expired, highest
>     number host does a survey
>       o did all other hosts create "problem" files?
>       o do all "problem" files show same (or compatible) error codes
>         related to file access problems?
>       o are all hosts communicating by network?
>       o if yes to all above
>   * delete all sanlock storage space
>   * initialize sanlock from scratch
>   * restart whatever may have given up because of sanlock
>   * restart VM if necessary
>
> 2. recovery subcommand
>
>   * add "hosted-engine --lock-initialize" command that would delete
>     sanlock, start over from scratch
>
> 3. script
>
>   * publish a script (in ovirt packages or available on web) which, when
>     run, does all (or most) of the recovery process needed.
>
> 4. commands
>
>   * publish on the web a "recipe" for dealing with files that commonly
>     go split-brain
>       o ids
>       o *.lease
>       o *.lockspace
>
> Any chance of any help on any of the above levels?
>
> Ted Miller
> Elkhart, IN, USA
>

vijay/allon/federico --^ ?