Re: [ovirt-users] [Gluster-users] open error -13 = sanlock

Wednesday, 2 March 2016

This is a multi-part message in MIME format.
--------------020603060204030604090206
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi guys,
thx a lot for your support .......at first.

Because we had been under huge time pressure, we found "google 
workaround"  which delete both files . It helped, probabbly at first 
steps of recover .
eg: " #  find /STORAGES/g1r5p5/GFS/ -samefile 
/STORAGES/g1r5p5/GFS/3da46e07-d1ea-4f10-9250-6cbbb7b94d80/dom_md/ids 
-print -delete "

---------------------->
Well at first I'll  fix permittions from mount points  to 660 .
If "ids"  file will be writeable , can't  became gluster colaps ??

regs.Pavel

On 2.3.2016 08:16, Ravishankar N wrote:
...
 On 03/02/2016 12:02 PM, Sahina Bose wrote:
>
>
> On 03/02/2016 03:45 AM, Nir Soffer wrote:
>> On Tue, Mar 1, 2016 at 10:51 PM, paf1(a)email.cz <paf1(a)email.cz&gt; wrote:
>> >
>> > HI,
>> > requested output:
>> >
>> > # ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:
>> > total 2,1M
>> > -rw-rw---- 1 vdsm kvm 1,0M  1. bře 21.28 ids  <-- good
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.17 leases
>> > -rw-r--r-- 1 vdsm kvm  335  7. lis 22.17 metadata
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:
>> > total 1,1M
>> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.41 ids  <-- bad (sanlock 
>> cannot write, other can read)
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 03.56 leases
>> > -rw-r--r-- 1 vdsm kvm  333  7. lis 03.56 metadata
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:
>> > total 1,1M
>> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids  <-- bad (sanlock 
>> cannot write, other can read)
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.14 leases
>> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.14 metadata
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:
>> > total 1,1M
>> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids  <-- bad (sanlock 
>> cannot write, other can read)
>> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.51 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases
>> > -rw-r--r-- 1 vdsm kvm  998 25. úno 00.35 metadata
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.16 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:
>> > total 1,1M
>> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.44 ids  <-- bad (sanlock 
>> cannot write, other can read)
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 00.18 leases
>> > -rw-r--r-- 1 vdsm kvm  333  7. lis 00.18 metadata
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:
>> > total 1,1M
>> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.32 ids  <-- bad (other can read)
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.18 leases
>> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.18 metadata
>> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:
>> > total 3,0M
>> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids  <-- bad (other can read)
>> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.42 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases
>> > -rw-r--r-- 1 vdsm kvm  997 24. úno 02.46 metadata
>> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.44 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:
>> > total 2,1M
>> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.34 ids  <-- bad (sanlock 
>> cannot write, other can read)
>> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.35 inbox
>> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases
>> > -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata
>> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.27 outbox
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:
>> > total 3,0M
>> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids  <-- bad (other can read)
>> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 23.50 inbox  <-- bad (other can 
>> read)
>> > -rw-rw-r-- 1 vdsm kvm 2,0M  6. lis 23.51 leases      <-- bad 
>> (other can read)
>> > -rw-rw-r-- 1 vdsm kvm  734  7. lis 02.13 metadata    <-- bad 
>> (group can write, other can read)
>> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 16.55 outbox  <-- bad (other 
>> can read)
>> >
>> > 
>>
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:
>> > total 1,1M
>> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.35 ids  <-- bad (other can read)
>> > -rw-rw-r-- 1 vdsm kvm  16M 24. úno 01.06 inbox
>> > -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases
>> > -rw-r--r-- 1 vdsm kvm  998 24. úno 19.07 metadata
>> > -rw-rw-r-- 1 vdsm kvm  16M  7. lis 22.20 outbox
>>
>>
>> It should look like this:
>>
>> -rw-rw----. 1 vdsm kvm 1.0M Mar  1 23:36 ids
>> -rw-rw----. 1 vdsm kvm 2.0M Mar  1 23:35 leases
>> -rw-r--r--. 1 vdsm kvm  353 Mar  1 23:35 metadata
>> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 outbox
>> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 inbox
>>
>> This explains the EACCES error.
>>
>> You can start by fixing the permissions manually, you can do this 
>> online.
>>
>> >  The ids files was generated by "touch" command after deleting 
>> them due "sanlock locking hang"  gluster crash & reboot
>> > I expected that they will be filled automaticaly after gluster 
>> reboot ( the  shadow copy from   ".gluster "   directory  was 
>> deleted & created empty  too )
>>
>> I don't know about gluster shadow copy, I would not play with 
>> gluster internals.
>> Adding Sahina for advice.
>
> Did you generate the ids file on the mount point.
>
> Ravi, can you help here?
>

 Okay, so what I understand from the output above is you have different 
 gluster volumes mounted and some of them have incorrect permissions 
 for the 'ids' file. The way to fix it is to do it from the mount like 
 Nir said.
 Why did you delete the file from the .glusterfs in the brick(s)? Was 
 there a gfid split brain?

 -Ravi

>>
>> > OK, it looks that sanlock  can't work with empty file or rewrite 
>> them .
>> > Am I right ??
>>
>> Yes, the files must be initialized before sanlock can use them.
>>
>> You can initialize the file like this:
>>
>> sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0
>>
>> Taken from 
>> http://lists.ovirt.org/pipermail/users/2016-February/038046.html
>>
>> > The last point - about "ids" workaround - this is offline version

>> = VMs have to be moved out from for continual running with 
>> maintenance volume mode
>> > But this is not acceptable in current situation, so the question 
>> again,  is it safe to do it online ??  ( YES / NO )
>>
>> The ids file is accessed only by sanlock. I guess that you don't 
>> have a running
>> SPM on this DC, since sanlock fails to acquire a host id, so you are 
>> pretty safe
>> to fix the permissions and initialize the ids files.
>>
>> I would do this:
>>
>> 1. Stop engine,  so it will not try to start vdsm
>> 2. Stop vdsm on all hosts, so they do not try to acquire a host id 
>> with sanlock
>>     This does not affect running vms
>> 3. Fix the permissions on the ids file, via glusterfs mount
>> 4. Initialize the ids files from one of the hosts, via the glusterfs 
>> mount
>>     This should fix the ids files on all replicas
>> 5. Start vdsm on all hosts
>> 6. Start engine
>>
>> Engine will connect to all hosts, hosts will connect to storage and 
>> try to acquire a host id.
>> Then Engine will start the SPM on one of the hosts, and your DC 
>> should become up.
>>
>> David, Sahina, can you confirm that this procedure is safe?
>
> Yes, correcting from the mount point should fix it on all replicas
>
>
>>
>> Nir
>>
>> >
>> > regs.
>> > Pavel
>> >
>> >
>> >
>> > On 1.3.2016 18:38, Nir Soffer wrote:
>> >
>> > On Tue, Mar 1, 2016 at 5:07 PM, paf1(a)email.cz <paf1(a)email.cz&gt; wrote:
>> >>
>> >> Hello,  can anybody  explain this error no.13 ( open file ) in 
>> sanlock.log .
>> >
>> >
>> > This is EACCES
>> >
>> > Can you share the outoput of:
>> >
>> >     ls -lh
/rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md
>> >
>> >>
>> >>
>> >> The size of  "ids" file is zero (0)
>> >
>> >
>> > This is how we create the ids file when initializing it.
>> >
>> > But then we use sanlock to initialize the ids file, and it should 
>> be 1MiB after that.
>> >
>> > Is this ids files created by vdsm, or one you created yourself?
>> >
>> >>
>> >> 2016-02-28 03:25:46+0100 269626 [1951]: open error -13 
>>
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids
>> >> 2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk 
>>
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids

>> error -13
>> >> 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 
>>
7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0
>> >>
>> >> If the main problem is about zero file size, can I regenerate 
>>  this file online securely , with no VM dependence  ????
>> >
>> >
>> > Yes, I think I already referred to the instructions how to do that 
>> in a previous mail.
>> >
>> >>
>> >>
>> >> dist = RHEL - 7 - 2.1511
>> >> kernel = 3.10.0 - 327.10.1.el7.x86_64
>> >> KVM = 2.3.0 - 29.1.el7
>> >> libvirt = libvirt-1.2.17-13.el7_2.3
>> >> vdsm = vdsm-4.16.30-0.el7
>> >> GlusterFS = glusterfs-3.7.8-1.el7
>> >>
>> >>
>> >> regs.
>> >> Pavel
>> >>
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users(a)ovirt.org <mailto:Users@ovirt.org>
>> >> http://lists.ovirt.org/mailman/listinfo/users
>> >>
>> >
>> >
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users(a)gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

--------------020603060204030604090206
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>
    <meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
  </head>
  <body text="#000066" bgcolor="#FFFFFF">
    Hi guys, <br>
    thx a lot for your support .......at first.<br>
    <br>
    Because we had been under huge time pressure, we found "google
    workaround"  which delete both files . It helped, probabbly at first
    steps of recover .<br>
    eg: " #  find /STORAGES/g1r5p5/GFS/ -samefile
    /STORAGES/g1r5p5/GFS/3da46e07-d1ea-4f10-9250-6cbbb7b94d80/dom_md/ids
    -print -delete "<br>
    <br>
    ----------------------&gt;<br>
    Well at first I'll  fix permittions from mount points  to 660 .<br>
    If "ids"  file will be writeable , can't  became gluster colaps
??<br>
    <br>
    regs.Pavel<br>
    <br>
    <br>
    <div class="moz-cite-prefix">On 2.3.2016 08:16, Ravishankar N
wrote:<br>
    </div>
    <blockquote cite="mid:56D69365.4090303@redhat.com"
type="cite">
      <meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
      <div class="moz-cite-prefix">On 03/02/2016 12:02 PM, Sahina Bose
        wrote:<br>
      </div>
      <blockquote cite="mid:56D68910.8040602@redhat.com"
type="cite">
        <meta content="text/html; charset=utf-8"
          http-equiv="Content-Type">
        <br>
        <br>
        <div class="moz-cite-prefix">On 03/02/2016 03:45 AM, Nir Soffer
          wrote:<br>
        </div>
        <blockquote
cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com"
          type="cite">
          <div dir="ltr">On Tue, Mar 1, 2016 at 10:51 PM, <a
              moz-do-not-send="true" class="moz-txt-link-abbreviated"
              href="mailto:paf1@email.cz">paf1@email.cz</a>
&lt;<a
              moz-do-not-send="true" class="moz-txt-link-abbreviated"
              href="mailto:paf1@email.cz">paf1@email.cz</a>&gt;
wrote:<br>
            &gt;<br>
            &gt; HI,<br>
            &gt; requested output:<br>
            &gt;<br>
            &gt; # ls -lh
            /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md<br>
            &gt;  <br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:<br>
            &gt; total 2,1M<br>
            &gt; -rw-rw---- 1 vdsm kvm 1,0M  1. bře 21.28 ids      
             &lt;-- good<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 inbox<br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.17 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm  335  7. lis 22.17 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:<br>
            &gt; total 1,1M<br>
            &gt; -rw-r--r-- 1 vdsm kvm    0 24. úno 07.41 ids      
             &lt;-- bad (sanlock cannot write, other can read)<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 inbox<br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M  7. lis 03.56 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm  333  7. lis 03.56 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:<br>
            &gt; total 1,1M<br>
            &gt; -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids      
             &lt;-- bad (sanlock cannot write, other can read)<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 inbox<br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.14 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm  333  7. lis 22.14 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:<br>
            &gt; total 1,1M<br>
            &gt; -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids      
             &lt;-- bad (sanlock cannot write, other can read)<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M 23. úno 22.51 inbox<br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm  998 25. úno 00.35 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 00.16 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:<br>
            &gt; total 1,1M<br>
            &gt; -rw-r--r-- 1 vdsm kvm    0 24. úno 07.44 ids      
             &lt;-- bad (sanlock cannot write, other can read)<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 inbox<br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M  7. lis 00.18 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm  333  7. lis 00.18 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:<br>
            &gt; total 1,1M<br>
            &gt; -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.32 ids      
             &lt;-- bad (other can read)<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 inbox<br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.18 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm  333  7. lis 22.18 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:<br>
            &gt; total 3,0M<br>
            &gt; -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids      
             &lt;-- bad (other can read)<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M 25. úno 00.42 inbox <br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm  997 24. úno 02.46 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M 25. úno 00.44 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:<br>
            &gt; total 2,1M<br>
            &gt; -rw-r--r-- 1 vdsm kvm    0 24. úno 07.34 ids      
             &lt;-- bad (sanlock cannot write, other can read)<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M 23. úno 22.35 inbox<br>
            &gt; -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases<br>
            &gt; -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata<br>
            &gt; -rw-rw---- 1 vdsm kvm  16M 23. úno 22.27 outbox<br>
            &gt;<br>
            &gt;
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:<br>
            &gt; total 3,0M<br>
            &gt; -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids      
             &lt;-- bad (other can read)<br>
            &gt; -rw-rw-r-- 1 vdsm kvm  16M  6. lis 23.50 inbox      
             &lt;-- bad (other can read)
            <div>&gt; -rw-rw-r-- 1 vdsm kvm 2,0M  6. lis 23.51 leases  
                   &lt;-- bad (other can read)<br>
              &gt; -rw-rw-r-- 1 vdsm kvm  734  7. lis 02.13 metadata    
                 &lt;-- bad (group can write, other can read)<br>
              &gt; -rw-rw-r-- 1 vdsm kvm  16M  6. lis 16.55 outbox      
               &lt;-- bad (other can read)<br>
              &gt;<br>
              &gt;
/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:<br>
              &gt; total 1,1M<br>
              &gt; -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.35 ids      
               &lt;-- bad (other can read)<br>
              &gt; -rw-rw-r-- 1 vdsm kvm  16M 24. úno 01.06 inbox<br>
              &gt; -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases<br>
              &gt; -rw-r--r-- 1 vdsm kvm  998 24. úno 19.07 metadata<br>
              &gt; -rw-rw-r-- 1 vdsm kvm  16M  7. lis 22.20 outbox<br>
              <br>
              <br>
              It should look like this:<br>
              <br>
              -rw-rw----. 1 vdsm kvm 1.0M Mar  1 23:36 ids<br>
              -rw-rw----. 1 vdsm kvm 2.0M Mar  1 23:35 leases<br>
              -rw-r--r--. 1 vdsm kvm  353 Mar  1 23:35 metadata<br>
              -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 outbox<br>
              -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 inbox<br>
              <br>
              This explains the EACCES error.<br>
              <br>
              You can start by fixing the permissions manually, you can
              do this online.<br>
               <br>
              &gt;  The ids files was generated by "touch" command after
              deleting them due "sanlock locking hang"  gluster crash
              &amp; reboot<br>
              &gt; I expected that they will be filled automaticaly
              after gluster reboot ( the  shadow copy from   ".gluster "
                directory  was deleted &amp; created empty  too )<br>
              <br>
              I don't know about gluster shadow copy, I would not play
              with gluster internals.</div>
            <div>Adding Sahina for advice.<br>
            </div>
          </div>
        </blockquote>
        <br>
        Did you generate the ids file on the mount point.<br>
        <br>
        Ravi, can you help here?<br>
        <br>
      </blockquote>
      <br>
      Okay, so what I understand from the output above is you have
      different gluster volumes mounted and some of them have incorrect
      permissions for the 'ids' file. The way to fix it is to do it from
      the mount like Nir said.<br>
      Why did you delete the file from the .glusterfs in the brick(s)? 
      Was there a gfid split brain? <br>
      <br>
      -Ravi<br>
      <br>
      <blockquote cite="mid:56D68910.8040602@redhat.com"
type="cite">
        <blockquote
cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com"
          type="cite">
          <div dir="ltr">
            <div><br>
              &gt; OK, it looks that sanlock  can't work with empty file
              or rewrite them .<br>
              &gt; Am I right ??<br>
              <br>
              Yes, the files must be initialized before sanlock can use
              them.<br>
              <br>
              You can initialize the file like this:<br>
              <br>
              sanlock direct init -s

&lt;sd_uuid&gt;:0:repair/&lt;sd_uuid&gt;/dom_md/ids:0<br>
              <br>
              Taken from <a moz-do-not-send="true"

href="http://lists.ovirt.org/pipermail/users/2016-February/038046.ht...
              <br>
              &gt; The last point - about "ids" workaround - this is
              offline version = VMs have to be moved out from for
              continual running with maintenance volume mode<br>
              &gt; But this is not acceptable in current situation, so
              the question again,  is it safe to do it online ??  ( YES
              / NO )</div>
            <div><br>
            </div>
            <div>The ids file is accessed only by sanlock. I guess that
              you don't have a running</div>
            <div>SPM on this DC, since sanlock fails to acquire a host
              id, so you are pretty safe</div>
            <div>to fix the permissions and initialize the ids files.</div>
            <div><br>
            </div>
            <div>I would do this:</div>
            <div><br>
            </div>
            <div>1. Stop engine,  so it will not try to start vdsm</div>
            <div>2. Stop vdsm on all hosts, so they do not try to
              acquire a host id with sanlock</div>
            <div>    This does not affect running vms</div>
            <div>3. Fix the permissions on the ids file, via glusterfs
              mount</div>
            <div>4. Initialize the ids files from one of the hosts, via
              the glusterfs mount</div>
            <div>    This should fix the ids files on all replicas</div>
            <div>5. Start vdsm on all hosts</div>
            <div>6. Start engine</div>
            <div><br>
            </div>
            <div>Engine will connect to all hosts, hosts will connect to
              storage and try to acquire a host id.</div>
            <div>Then Engine will start the SPM on one of the hosts, and
              your DC should become up.</div>
            <div><br>
            </div>
            <div>David, Sahina, can you confirm that this procedure is
              safe?</div>
          </div>
        </blockquote>
        <br>
        Yes, correcting from the mount point should fix it on all
        replicas<br>
        <br>
        <br>
        <blockquote
cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com"
          type="cite">
          <div dir="ltr">
            <div><br>
            </div>
            <div>Nir</div>
            <div><br>
            </div>
            <div>&gt;<br>
              &gt; regs.<br>
              &gt; Pavel<br>
              &gt;<br>
              &gt;<br>
              &gt;<br>
              &gt; On 1.3.2016 18:38, Nir Soffer wrote:<br>
              &gt;<br>
              &gt; On Tue, Mar 1, 2016 at 5:07 PM, <a
                moz-do-not-send="true"
class="moz-txt-link-abbreviated"
                href="mailto:paf1@email.cz">paf1@email.cz</a>
&lt;<a
                moz-do-not-send="true"
class="moz-txt-link-abbreviated"
                href="mailto:paf1@email.cz">paf1@email.cz</a>&gt;
wrote:<br>
              &gt;&gt;<br>
              &gt;&gt; Hello,  can anybody  explain this error no.13 (
              open file ) in sanlock.log .<br>
              &gt;<br>
              &gt;<br>
              &gt; This is EACCES<br>
              &gt;<br>
              &gt; Can you share the outoput of:<br>
              &gt;<br>
              &gt;     ls -lh
/rhev/data-center/mnt/&lt;server&gt;:&lt;_path&gt;/&lt;sd_uuid&gt;/dom_md<br>
              &gt;  <br>
              &gt;&gt;<br>
              &gt;&gt;<br>
              &gt;&gt; The size of  "ids" file is zero (0)<br>
              &gt;<br>
              &gt;<br>
              &gt; This is how we create the ids file when initializing
              it.<br>
              &gt;<br>
              &gt; But then we use sanlock to initialize the ids file,
              and it should be 1MiB after that.<br>
              &gt;<br>
              &gt; Is this ids files created by vdsm, or one you created
              yourself?<br>
              &gt;  <br>
              &gt;&gt;<br>
              &gt;&gt; 2016-02-28 03:25:46+0100 269626 [1951]: open
              error -13
/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids<br>
              &gt;&gt; 2016-02-28 03:25:46+0100 269626 [1951]: s187985
              open_disk

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids

              error -13<br>
              &gt;&gt; 2016-02-28 03:25:56+0100 269636 [11304]: s187992
              lockspace
7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0<br>
              &gt;&gt;<br>
              &gt;&gt; If the main problem is about zero file size, can
              I regenerate  this file online securely , with no VM
              dependence  ????<br>
              &gt;<br>
              &gt;<br>
              &gt; Yes, I think I already referred to the instructions
              how to do that in a previous mail.<br>
              &gt;<br>
              &gt;&gt;<br>
              &gt;&gt;<br>
              &gt;&gt; dist = RHEL - 7 - 2.1511<br>
              &gt;&gt; kernel = 3.10.0 - 327.10.1.el7.x86_64<br>
              &gt;&gt; KVM = 2.3.0 - 29.1.el7<br>
              &gt;&gt; libvirt = libvirt-1.2.17-13.el7_2.3<br>
              &gt;&gt; vdsm = vdsm-4.16.30-0.el7<br>
              &gt;&gt; GlusterFS = glusterfs-3.7.8-1.el7<br>
              &gt;&gt;<br>
              &gt;&gt;<br>
              &gt;&gt; regs.<br>
              &gt;&gt; Pavel<br>
              &gt;&gt;<br>
              &gt;&gt; _______________________________________________<br>
              &gt;&gt; Users mailing list<br>
              &gt;&gt; <a moz-do-not-send="true"

href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
              &gt;&gt; <a moz-do-not-send="true"

href="http://lists.ovirt.org/mailman/listinfo/users">http://...
              &gt;&gt;<br>
              &gt;<br>
              &gt;<br>
            </div>
          </div>
        </blockquote>
        <br>
        <br>
        <fieldset class="mimeAttachmentHeader"></fieldset>
        <br>
        <pre wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.gluster.org/mailman/listinfo/gluster-users">...
      </blockquote>
      <br>
      <br>
    </blockquote>
    <br>
  </body>
</html>

--------------020603060204030604090206--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] [Gluster-users] open error -13 = sanlock