--------------050904040002030300050808 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit Itamar, I am addressing this to you because one of your assignments seems to be to coordinate other oVirt contributors when dealing with issues that are raised on the ovirt-users email list. As you are aware, there is an ongoing split-brain problem with running sanlock on replicated gluster storage. Personally, I believe that this is the 5th time that I have been bitten by this sanlock+gluster problem. I believe that the following are true (if not, my entire request is probably off base). * ovirt uses sanlock in such a way that when the sanlock storage is on a replicated gluster file system, very small storage disruptions can result in a gluster split-brain on the sanlock space o gluster is aware of the problem, and is working on a different way of replicating data, which will reduce these problems. * most (maybe all) of the sanlock locks have a short duration, measured in seconds * there are only a couple of things that a user can safely do from the command line when a file is in split-brain o delete the file o rename (mv) the file * x _How did I get into this mess?_ had 3 hosts running ovirt 3.3 each hosted VMs gluster replica 3 storage engine was external to cluster upgraded 3 hosts from ovirt 3.3 to 3.4 hosted-engine deploy used new gluster volume (accessed via nfs) for storage storage was accessed using localhost:engVM1 link (localhost was probably a poor choice) created new engine on VM (did not transfer any data from old engine) added 3 hosts to new engine via web-gui ran above setup for 3 days shut entire system down before I left on vacation (holiday) came back from vacation powered on hosts found that iptables did not have rules for gluster access (a continuing problem if host installation is allowed to set up firewall) added rules for gluster glusterfs now up and running added storage manually tried "hosted-engine --vm-start" vm did not start logs show sanlock errors "gluster volume heal engVM1full: "gluster volume heal engVM1 info split-brain" showed 6 files in split-brain all 5 prefixed by /rhev/data-center/mnt/localhost\:_engVM1 UUID/dom_md/ids UUID/images/UUID/UUID (VM hard disk) UUID/images/UUID/UUID.lease UUID/ha_agent/hosted-engine.lockspace UUID/ha_agent/hosted-engine.metadata I copied each of the above files off of each of the three bricks to a safe place (15 files copied) I renamed the 5 files on /rhev/.... I copied the 5 files from one of the bricks to /rhev/ files can now be read OK (e.g. cat ids) sanlock.log shows error sets like these: 2014-05-20 03:23:39-0400 36199 [2843]: s3358 lockspace 5ebb3b40-a394-405b-bbac-4c0e21ccd659:1:/rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids:0 2014-05-20 03:23:39-0400 36199 [18873]: open error -5 /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids 2014-05-20 03:23:39-0400 36199 [18873]: s3358 open_disk /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids error -5 2014-05-20 03:23:40-0400 36200 [2843]: s3358 add_lockspace fail result -19 I am now stuck What I would like to see in ovirt to help me (and others like me). Alternates listed in order from most desirable (automatic) to least desirable (set of commands to type, with lots of variables to figure out). 1. automagic recovery * When a host is not able to access sanlock, it writes a small "problem" text file into the shared storage o the host-ID as part of the name (so only one host ever accesses that file) o a status number for the error causing problems o time stamp o time stamp when last sanlock lease will expire o if sanlock is able to access the file, the "problem" file is deleted * when time passes for its last sanlock lease to be expired, highest number host does a survey o did all other hosts create "problem" files? o do all "problem" files show same (or compatible) error codes related to file access problems? o are all hosts communicating by network? o if yes to all above * delete all sanlock storage space * initialize sanlock from scratch * restart whatever may have given up because of sanlock * restart VM if necessary 2. recovery subcommand * add "hosted-engine --lock-initialize" command that would delete sanlock, start over from scratch 3. script * publish a script (in ovirt packages or available on web) which, when run, does all (or most) of the recovery process needed. 4. commands * publish on the web a "recipe" for dealing with files that commonly go split-brain o ids o *.lease o *.lockspace Any chance of any help on any of the above levels? Ted Miller Elkhart, IN, USA --------------050904040002030300050808 Content-Type: text/html; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body text="#000000" bgcolor="#FFFFFF"> Itamar, I am addressing this to you because one of your assignments seems to be to coordinate other oVirt contributors when dealing with issues that are raised on the ovirt-users email list.<br> <br> As you are aware, there is an ongoing split-brain problem with running sanlock on replicated gluster storage.&nbsp; Personally, I believe that this is the 5th time that I have been bitten by this sanlock+gluster problem.<br> <br> I believe that the following are true (if not, my entire request is probably off base).<br> <ul> <li>ovirt uses sanlock in such a way that when the sanlock storage is on a replicated gluster file system, very small storage disruptions can result in a gluster split-brain on the sanlock space</li> <ul> <li>gluster is aware of the problem, and is working on a different way of replicating data, which will reduce these problems.</li> </ul> <li>most (maybe all) of the sanlock locks have a short duration, measured in seconds</li> <li>there are only a couple of things that a user can safely do from the command line when a file is in split-brain</li> <ul> <li>delete the file</li> <li>rename (mv) the file<br> </li> </ul> <li>x</li> </ul> <u>How did I get into this mess?</u><br> <br> had 3 hosts running ovirt 3.3<br> &nbsp;&nbsp;&nbsp; each hosted VMs<br> &nbsp;&nbsp;&nbsp; gluster replica 3 storage<br> &nbsp;&nbsp;&nbsp; engine was external to cluster<br> upgraded 3 hosts from ovirt 3.3 to 3.4<br> hosted-engine deploy<br> &nbsp;&nbsp;&nbsp; used new gluster volume (accessed via nfs) for storage<br> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; storage was accessed using localhost:engVM1 link (localhost was probably a poor choice)<br> &nbsp;&nbsp;&nbsp; created new engine on VM (did not transfer any data from old engine)<br> added 3 hosts to new engine via web-gui<br> ran above setup for 3 days<br> shut entire system down before I left on vacation (holiday)<br> came back from vacation<br> powered on hosts<br> found that iptables did not have rules for gluster access<br> &nbsp; &nbsp; (a continuing problem if host installation is allowed to set up firewall)<br> added rules for gluster<br> glusterfs now up and running<br> added storage manually<br> tried "hosted-engine --vm-start"<br> vm did not start<br> logs show sanlock errors<br> "gluster volume heal engVM1full:<br> "gluster volume heal engVM1 info split-brain" showed 6 files in split-brain<br> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; all 5 prefixed by /rhev/data-center/mnt/localhost\:_engVM1<br> &nbsp;&nbsp;&nbsp; UUID/dom_md/ids<br> &nbsp;&nbsp;&nbsp; UUID/images/UUID/UUID (VM hard disk)<br> &nbsp;&nbsp;&nbsp; UUID/images/UUID/UUID.lease<br> &nbsp;&nbsp;&nbsp; UUID/ha_agent/hosted-engine.lockspace<br> &nbsp;&nbsp;&nbsp; UUID/ha_agent/hosted-engine.metadata<br> I copied each of the above files off of each of the three bricks to a safe place (15 files copied)<br> I renamed the 5 files on /rhev/....<br> I copied the 5 files from one of the bricks to /rhev/<br> &nbsp;&nbsp;&nbsp; files can now be read OK (e.g. cat ids)<br> sanlock.log shows error sets like these:<br> <pre>2014-05-20 03:23:39-0400 36199 [2843]: s3358 lockspace 5ebb3b40-a394-405b-bbac-4c0e21ccd659:1:/rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids:0 2014-05-20 03:23:39-0400 36199 [18873]: open error -5 /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids 2014-05-20 03:23:39-0400 36199 [18873]: s3358 open_disk /rhev/data-center/mnt/localhost:_engVM1/5ebb3b40-a394-405b-bbac-4c0e21ccd659/dom_md/ids error -5 2014-05-20 03:23:40-0400 36200 [2843]: s3358 add_lockspace fail result -19</pre> I am now stuck<br> <br> What I would like to see in ovirt to help me (and others like me).&nbsp; Alternates listed in order from most desirable (automatic) to least desirable (set of commands to type, with lots of variables to figure out).<br> <br> 1. automagic recovery<br> <ul> <li>&nbsp;When a host is not able to access sanlock, it writes a small "problem" text file into the shared storage</li> <ul> <li>the host-ID as part of the name (so only one host ever accesses that file)</li> <li>a status number for the error causing problems</li> <li>time stamp</li> <li>time stamp when last sanlock lease will expire</li> <li>if sanlock is able to access the file, the "problem" file is deleted</li> </ul> <li>when time passes for its last sanlock lease to be expired, highest number host does a survey</li> <ul> <li>did all other hosts create "problem" files?</li> <li>do all "problem" files show same (or compatible) error codes related to file access problems?</li> <li>are all hosts communicating by network?</li> <li>if yes to all above</li> </ul> <li>delete all sanlock storage space<br> </li> <li>initialize sanlock from scratch</li> <li>restart whatever may have given up because of sanlock</li> <li>restart VM if necessary</li> </ul> <p>2. recovery subcommand<br> </p> <ul> <li>add "hosted-engine --lock-initialize" command that would delete sanlock, start over from scratch</li> </ul> <p>3. script<br> </p> <ul> <li>publish a script (in ovirt packages or available on web) which, when run, does all (or most) of the recovery process needed.</li> </ul> <p>4. commands<br> </p> <ul> <li>publish on the web a "recipe" for dealing with files that commonly go split-brain</li> <ul> <li>ids</li> <li>*.lease</li> <li>*.lockspace</li> </ul> </ul> <p>Any chance of any help on any of the above levels?<br> </p> <p>Ted Miller<br> Elkhart, IN, USA<br> <br> </p> </body> </html> --------------050904040002030300050808--