On Tue, Mar 1, 2016 at 10:51 PM, paf1@email.cz <paf1@email.cz> wrote:

HI, requested output:

# ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:

total 2,1M -rw-rw---- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- good -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.17 leases -rw-r--r-- 1 vdsm kvm 335 7. lis 22.17 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.41 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 03.56 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 03.56 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.14 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.14 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.51 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases -rw-r--r-- 1 vdsm kvm 998 25. úno 00.35 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.44 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 00.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 00.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.32 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 25. úno 00.42 inbox -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases -rw-r--r-- 1 vdsm kvm 997 24. úno 02.46 metadata -rw-rw---- 1 vdsm kvm 16M 25. úno 00.44 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:

total 2,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.34 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.35 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata -rw-rw---- 1 vdsm kvm 16M 23. úno 22.27 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 23.50 inbox <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 2,0M 6. lis 23.51 leases <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 734 7. lis 02.13 metadata <-- bad (group can write, other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 16.55 outbox <-- bad (other can read)

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.35 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 24. úno 01.06 inbox -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases -rw-r--r-- 1 vdsm kvm 998 24. úno 19.07 metadata -rw-rw-r-- 1 vdsm kvm 16M 7. lis 22.20 outbox

The ids files was generated by "touch" command after deleting them due "sanlock locking hang" gluster crash & reboot I expected that they will be filled automaticaly after gluster reboot (

It should look like this: -rw-rw----. 1 vdsm kvm 1.0M Mar 1 23:36 ids -rw-rw----. 1 vdsm kvm 2.0M Mar 1 23:35 leases -rw-r--r--. 1 vdsm kvm 353 Mar 1 23:35 metadata -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 outbox -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 inbox This explains the EACCES error. You can start by fixing the permissions manually, you can do this online. the shadow copy from ".gluster " directory was deleted & created empty too ) I don't know about gluster shadow copy, I would not play with gluster internals. Adding Sahina for advice.

OK, it looks that sanlock can't work with empty file or rewrite them . Am I right ??

Yes, the files must be initialized before sanlock can use them. You can initialize the file like this: sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0 Taken from http://lists.ovirt.org/pipermail/users/2016-February/038046.html

The last point - about "ids" workaround - this is offline version = VMs have to be moved out from for continual running with maintenance volume mode But this is not acceptable in current situation, so the question again, is it safe to do it online ?? ( YES / NO )

The ids file is accessed only by sanlock. I guess that you don't have a running SPM on this DC, since sanlock fails to acquire a host id, so you are pretty safe to fix the permissions and initialize the ids files. I would do this: 1. Stop engine, so it will not try to start vdsm 2. Stop vdsm on all hosts, so they do not try to acquire a host id with sanlock This does not affect running vms 3. Fix the permissions on the ids file, via glusterfs mount 4. Initialize the ids files from one of the hosts, via the glusterfs mount This should fix the ids files on all replicas 5. Start vdsm on all hosts 6. Start engine Engine will connect to all hosts, hosts will connect to storage and try to acquire a host id. Then Engine will start the SPM on one of the hosts, and your DC should become up. David, Sahina, can you confirm that this procedure is safe? Nir

regs. Pavel

On 1.3.2016 18:38, Nir Soffer wrote:

On Tue, Mar 1, 2016 at 5:07 PM, paf1@email.cz <paf1@email.cz> wrote:

...

Hello, can anybody explain this error no.13 ( open file ) in

sanlock.log .

This is EACCES

Can you share the outoput of:

ls -lh /rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md

...

The size of "ids" file is zero (0)

This is how we create the ids file when initializing it.

But then we use sanlock to initialize the ids file, and it should be 1MiB

after that.

Is this ids files created by vdsm, or one you created yourself?

...

2016-02-28 03:25:46+0100 269626 [1951]: open error -13

...

2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids error -13 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0

If the main problem is about zero file size, can I regenerate this file online securely , with no VM dependence ????

Yes, I think I already referred to the instructions how to do that in a

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids previous mail.

...

dist = RHEL - 7 - 2.1511 kernel = 3.10.0 - 327.10.1.el7.x86_64 KVM = 2.3.0 - 29.1.el7 libvirt = libvirt-1.2.17-13.el7_2.3 vdsm = vdsm-4.16.30-0.el7 GlusterFS = glusterfs-3.7.8-1.el7

regs. Pavel

_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users

This is a multi-part message in MIME format. --------------030409010206020806050907 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 03/02/2016 03:45 AM, Nir Soffer wrote:

On Tue, Mar 1, 2016 at 10:51 PM, paf1@email.cz <mailto:paf1@email.cz> <paf1@email.cz <mailto:paf1@email.cz>> wrote:

...

HI, requested output:

# ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:

...

total 2,1M -rw-rw---- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- good -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.17 leases -rw-r--r-- 1 vdsm kvm 335 7. lis 22.17 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.41 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 03.56 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 03.56 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.14 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.14 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.51 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases -rw-r--r-- 1 vdsm kvm 998 25. úno 00.35 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.44 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 00.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 00.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:

...

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.32 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:

...

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 25. úno 00.42 inbox -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases -rw-r--r-- 1 vdsm kvm 997 24. úno 02.46 metadata -rw-rw---- 1 vdsm kvm 16M 25. úno 00.44 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:

...

total 2,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.34 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.35 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata -rw-rw---- 1 vdsm kvm 16M 23. úno 22.27 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:

...

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 23.50 inbox <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 2,0M 6. lis 23.51 leases <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 734 7. lis 02.13 metadata <-- bad (group can write, other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 16.55 outbox <-- bad (other can read)

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:

...

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.35 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 24. úno 01.06 inbox -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases -rw-r--r-- 1 vdsm kvm 998 24. úno 19.07 metadata -rw-rw-r-- 1 vdsm kvm 16M 7. lis 22.20 outbox

It should look like this:

-rw-rw----. 1 vdsm kvm 1.0M Mar 1 23:36 ids -rw-rw----. 1 vdsm kvm 2.0M Mar 1 23:35 leases -rw-r--r--. 1 vdsm kvm 353 Mar 1 23:35 metadata -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 outbox -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 inbox

This explains the EACCES error.

You can start by fixing the permissions manually, you can do this online.

...

The ids files was generated by "touch" command after deleting them due "sanlock locking hang" gluster crash & reboot I expected that they will be filled automaticaly after gluster reboot ( the shadow copy from ".gluster " directory was deleted & created empty too )

I don't know about gluster shadow copy, I would not play with gluster internals. Adding Sahina for advice.

Did you generate the ids file on the mount point. Ravi, can you help here?

...

OK, it looks that sanlock can't work with empty file or rewrite them . Am I right ??

Yes, the files must be initialized before sanlock can use them.

You can initialize the file like this:

sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0

Taken from http://lists.ovirt.org/pipermail/users/2016-February/038046.html

...

The last point - about "ids" workaround - this is offline version = VMs have to be moved out from for continual running with maintenance volume mode But this is not acceptable in current situation, so the question again, is it safe to do it online ?? ( YES / NO )

The ids file is accessed only by sanlock. I guess that you don't have a running SPM on this DC, since sanlock fails to acquire a host id, so you are pretty safe to fix the permissions and initialize the ids files.

I would do this:

1. Stop engine, so it will not try to start vdsm 2. Stop vdsm on all hosts, so they do not try to acquire a host id with sanlock This does not affect running vms 3. Fix the permissions on the ids file, via glusterfs mount 4. Initialize the ids files from one of the hosts, via the glusterfs mount This should fix the ids files on all replicas 5. Start vdsm on all hosts 6. Start engine

Engine will connect to all hosts, hosts will connect to storage and try to acquire a host id. Then Engine will start the SPM on one of the hosts, and your DC should become up.

David, Sahina, can you confirm that this procedure is safe?

Yes, correcting from the mount point should fix it on all replicas

Nir

...

regs. Pavel

On 1.3.2016 18:38, Nir Soffer wrote:

On Tue, Mar 1, 2016 at 5:07 PM, paf1@email.cz <mailto:paf1@email.cz>

<paf1@email.cz <mailto:paf1@email.cz>> wrote:

...

...

Hello, can anybody explain this error no.13 ( open file ) in

sanlock.log .

This is EACCES

Can you share the outoput of:

ls -lh /rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md

...

The size of "ids" file is zero (0)

This is how we create the ids file when initializing it.

But then we use sanlock to initialize the ids file, and it should be 1MiB after that.

Is this ids files created by vdsm, or one you created yourself?

...

2016-02-28 03:25:46+0100 269626 [1951]: open error -13

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids

...

2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids error -13 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0

If the main problem is about zero file size, can I regenerate this file online securely , with no VM dependence ????

Yes, I think I already referred to the instructions how to do that in a previous mail.

...

dist = RHEL - 7 - 2.1511 kernel = 3.10.0 - 327.10.1.el7.x86_64 KVM = 2.3.0 - 29.1.el7 libvirt = libvirt-1.2.17-13.el7_2.3 vdsm = vdsm-4.16.30-0.el7 GlusterFS = glusterfs-3.7.8-1.el7

regs. Pavel

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users

--------------030409010206020806050907 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <br> <br> <div class="moz-cite-prefix">On 03/02/2016 03:45 AM, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr">On Tue, Mar 1, 2016 at 10:51 PM, <a moz-do-not-send="true" href="mailto:paf1@email.cz"><a class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a></a> <<a moz-do-not-send="true" href="mailto:paf1@email.cz">paf1@email.cz</a>> wrote:<br> ><br> > HI,<br> > requested output:<br> ><br> > # ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md<br> >  <br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:<br> > total 2,1M<br> > -rw-rw---- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- good<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.17 leases<br> > -rw-r--r-- 1 vdsm kvm  335  7. lis 22.17 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.41 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 03.56 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 03.56 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.14 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.14 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.51 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases<br> > -rw-r--r-- 1 vdsm kvm  998 25. úno 00.35 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.16 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.44 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 00.18 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 00.18 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:<br> > total 1,1M<br> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.32 ids        <-- bad (other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.18 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.18 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:<br> > total 3,0M<br> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- bad (other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.42 inbox <br> > -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases<br> > -rw-r--r-- 1 vdsm kvm  997 24. úno 02.46 metadata<br> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.44 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:<br> > total 2,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.34 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.35 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases<br> > -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.27 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:<br> > total 3,0M<br> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 23.50 inbox        <-- bad (other can read) <div>> -rw-rw-r-- 1 vdsm kvm 2,0M  6. lis 23.51 leases        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  734  7. lis 02.13 metadata        <-- bad (group can write, other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 16.55 outbox        <-- bad (other can read)<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:<br> > total 1,1M<br> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.35 ids        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M 24. úno 01.06 inbox<br> > -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases<br> > -rw-r--r-- 1 vdsm kvm  998 24. úno 19.07 metadata<br> > -rw-rw-r-- 1 vdsm kvm  16M  7. lis 22.20 outbox<br> <br> <br> It should look like this:<br> <br> -rw-rw----. 1 vdsm kvm 1.0M Mar  1 23:36 ids<br> -rw-rw----. 1 vdsm kvm 2.0M Mar  1 23:35 leases<br> -rw-r--r--. 1 vdsm kvm  353 Mar  1 23:35 metadata<br> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 outbox<br> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 inbox<br> <br> This explains the EACCES error.<br> <br> You can start by fixing the permissions manually, you can do this online.<br>  <br> >  The ids files was generated by "touch" command after deleting them due "sanlock locking hang"  gluster crash & reboot<br> > I expected that they will be filled automaticaly after gluster reboot ( the  shadow copy from   ".gluster "   directory  was deleted & created empty  too )<br> <br> I don't know about gluster shadow copy, I would not play with gluster internals.</div> <div>Adding Sahina for advice.<br> </div> </div> </blockquote> <br> Did you generate the ids file on the mount point.<br> <br> Ravi, can you help here?<br> <br> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr"> <div><br> > OK, it looks that sanlock  can't work with empty file or rewrite them .<br> > Am I right ??<br> <br> Yes, the files must be initialized before sanlock can use them.<br> <br> You can initialize the file like this:<br> <br> sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0<br> <br> Taken from <a moz-do-not-send="true" href="http://lists.ovirt.org/pipermail/users/2016-February/038046.html">http://lists.ovirt.org/pipermail/users/2016-February/038046.html</a><br> <br> > The last point - about "ids" workaround - this is offline version = VMs have to be moved out from for continual running with maintenance volume mode<br> > But this is not acceptable in current situation, so the question again,  is it safe to do it online ??  ( YES / NO )</div> <div><br> </div> <div>The ids file is accessed only by sanlock. I guess that you don't have a running</div> <div>SPM on this DC, since sanlock fails to acquire a host id, so you are pretty safe</div> <div>to fix the permissions and initialize the ids files.</div> <div><br> </div> <div>I would do this:</div> <div><br> </div> <div>1. Stop engine,  so it will not try to start vdsm</div> <div>2. Stop vdsm on all hosts, so they do not try to acquire a host id with sanlock</div> <div>    This does not affect running vms</div> <div>3. Fix the permissions on the ids file, via glusterfs mount</div> <div>4. Initialize the ids files from one of the hosts, via the glusterfs mount</div> <div>    This should fix the ids files on all replicas</div> <div>5. Start vdsm on all hosts</div> <div>6. Start engine</div> <div><br> </div> <div>Engine will connect to all hosts, hosts will connect to storage and try to acquire a host id.</div> <div>Then Engine will start the SPM on one of the hosts, and your DC should become up.</div> <div><br> </div> <div>David, Sahina, can you confirm that this procedure is safe?</div> </div> </blockquote> <br> Yes, correcting from the mount point should fix it on all replicas<br> <br> <br> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr"> <div><br> </div> <div>Nir</div> <div><br> </div> <div>><br> > regs.<br> > Pavel<br> ><br> ><br> ><br> > On 1.3.2016 18:38, Nir Soffer wrote:<br> ><br> > On Tue, Mar 1, 2016 at 5:07 PM, <a moz-do-not-send="true" href="mailto:paf1@email.cz"><a class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a></a> <<a moz-do-not-send="true" href="mailto:paf1@email.cz">paf1@email.cz</a>> wrote:<br> >><br> >> Hello,  can anybody  explain this error no.13 ( open file ) in sanlock.log .<br> ><br> ><br> > This is EACCES<br> ><br> > Can you share the outoput of:<br> ><br> >     ls -lh /rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md<br> >  <br> >><br> >><br> >> The size of  "ids" file is zero (0)<br> ><br> ><br> > This is how we create the ids file when initializing it.<br> ><br> > But then we use sanlock to initialize the ids file, and it should be 1MiB after that.<br> ><br> > Is this ids files created by vdsm, or one you created yourself?<br> >  <br> >><br> >> 2016-02-28 03:25:46+0100 269626 [1951]: open error -13 /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids<br> >> 2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids error -13<br> >> 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0<br> >><br> >> If the main problem is about zero file size, can I regenerate  this file online securely , with no VM dependence  ????<br> ><br> ><br> > Yes, I think I already referred to the instructions how to do that in a previous mail.<br> ><br> >><br> >><br> >> dist = RHEL - 7 - 2.1511<br> >> kernel = 3.10.0 - 327.10.1.el7.x86_64<br> >> KVM = 2.3.0 - 29.1.el7<br> >> libvirt = libvirt-1.2.17-13.el7_2.3<br> >> vdsm = vdsm-4.16.30-0.el7<br> >> GlusterFS = glusterfs-3.7.8-1.el7<br> >><br> >><br> >> regs.<br> >> Pavel<br> >><br> >> _______________________________________________<br> >> Users mailing list<br> >> <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> >> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> >><br> ><br> ><br> </div> </div> </blockquote> <br> </body> </html> --------------030409010206020806050907--

This is a multi-part message in MIME format. --------------020603060204030604090206 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi guys, thx a lot for your support .......at first. Because we had been under huge time pressure, we found "google workaround" which delete both files . It helped, probabbly at first steps of recover . eg: " # find /STORAGES/g1r5p5/GFS/ -samefile /STORAGES/g1r5p5/GFS/3da46e07-d1ea-4f10-9250-6cbbb7b94d80/dom_md/ids -print -delete " ----------------------> Well at first I'll fix permittions from mount points to 660 . If "ids" file will be writeable , can't became gluster colaps ?? regs.Pavel On 2.3.2016 08:16, Ravishankar N wrote:

On 03/02/2016 12:02 PM, Sahina Bose wrote:

...

On 03/02/2016 03:45 AM, Nir Soffer wrote:

...

On Tue, Mar 1, 2016 at 10:51 PM, paf1@email.cz <paf1@email.cz> wrote:

...

HI, requested output:

# ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:

...

total 2,1M -rw-rw---- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- good -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.17 leases -rw-r--r-- 1 vdsm kvm 335 7. lis 22.17 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.41 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 03.56 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 03.56 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.14 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.14 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.51 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases -rw-r--r-- 1 vdsm kvm 998 25. úno 00.35 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.44 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 00.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 00.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:

...

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.32 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:

...

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 25. úno 00.42 inbox -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases -rw-r--r-- 1 vdsm kvm 997 24. úno 02.46 metadata -rw-rw---- 1 vdsm kvm 16M 25. úno 00.44 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:

...

total 2,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.34 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.35 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata -rw-rw---- 1 vdsm kvm 16M 23. úno 22.27 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:

...

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 23.50 inbox <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 2,0M 6. lis 23.51 leases <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 734 7. lis 02.13 metadata <-- bad (group can write, other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 16.55 outbox <-- bad (other can read)

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:

...

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.35 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 24. úno 01.06 inbox -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases -rw-r--r-- 1 vdsm kvm 998 24. úno 19.07 metadata -rw-rw-r-- 1 vdsm kvm 16M 7. lis 22.20 outbox

It should look like this:

-rw-rw----. 1 vdsm kvm 1.0M Mar 1 23:36 ids -rw-rw----. 1 vdsm kvm 2.0M Mar 1 23:35 leases -rw-r--r--. 1 vdsm kvm 353 Mar 1 23:35 metadata -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 outbox -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 inbox

This explains the EACCES error.

You can start by fixing the permissions manually, you can do this online.

...

The ids files was generated by "touch" command after deleting them due "sanlock locking hang" gluster crash & reboot I expected that they will be filled automaticaly after gluster reboot ( the shadow copy from ".gluster " directory was deleted & created empty too )

I don't know about gluster shadow copy, I would not play with gluster internals. Adding Sahina for advice.

Did you generate the ids file on the mount point.

Ravi, can you help here?

Okay, so what I understand from the output above is you have different gluster volumes mounted and some of them have incorrect permissions for the 'ids' file. The way to fix it is to do it from the mount like Nir said. Why did you delete the file from the .glusterfs in the brick(s)? Was there a gfid split brain?

-Ravi

...

...

...

OK, it looks that sanlock can't work with empty file or rewrite them . Am I right ??

Yes, the files must be initialized before sanlock can use them.

You can initialize the file like this:

sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0

Taken from http://lists.ovirt.org/pipermail/users/2016-February/038046.html

...

The last point - about "ids" workaround - this is offline version = VMs have to be moved out from for continual running with maintenance volume mode But this is not acceptable in current situation, so the question again, is it safe to do it online ?? ( YES / NO )

The ids file is accessed only by sanlock. I guess that you don't have a running SPM on this DC, since sanlock fails to acquire a host id, so you are pretty safe to fix the permissions and initialize the ids files.

I would do this:

1. Stop engine, so it will not try to start vdsm 2. Stop vdsm on all hosts, so they do not try to acquire a host id with sanlock This does not affect running vms 3. Fix the permissions on the ids file, via glusterfs mount 4. Initialize the ids files from one of the hosts, via the glusterfs mount This should fix the ids files on all replicas 5. Start vdsm on all hosts 6. Start engine

Engine will connect to all hosts, hosts will connect to storage and try to acquire a host id. Then Engine will start the SPM on one of the hosts, and your DC should become up.

David, Sahina, can you confirm that this procedure is safe?

Yes, correcting from the mount point should fix it on all replicas

...

Nir

...

regs. Pavel

On 1.3.2016 18:38, Nir Soffer wrote:

On Tue, Mar 1, 2016 at 5:07 PM, paf1@email.cz <paf1@email.cz> wrote:

...

Hello, can anybody explain this error no.13 ( open file ) in

sanlock.log .

...

This is EACCES

Can you share the outoput of:

ls -lh /rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md

...

The size of "ids" file is zero (0)

This is how we create the ids file when initializing it.

But then we use sanlock to initialize the ids file, and it should

be 1MiB after that.

...

Is this ids files created by vdsm, or one you created yourself?

...

2016-02-28 03:25:46+0100 269626 [1951]: open error -13

...

...

2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids error -13 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0

If the main problem is about zero file size, can I regenerate

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids this file online securely , with no VM dependence ????

...

Yes, I think I already referred to the instructions how to do that

in a previous mail.

...

...

dist = RHEL - 7 - 2.1511 kernel = 3.10.0 - 327.10.1.el7.x86_64 KVM = 2.3.0 - 29.1.el7 libvirt = libvirt-1.2.17-13.el7_2.3 vdsm = vdsm-4.16.30-0.el7 GlusterFS = glusterfs-3.7.8-1.el7

regs. Pavel

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users

--------------020603060204030604090206 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000066" bgcolor="#FFFFFF"> Hi guys, <br> thx a lot for your support .......at first.<br> <br> Because we had been under huge time pressure, we found "google workaround"  which delete both files . It helped, probabbly at first steps of recover .<br> eg: " #  find /STORAGES/g1r5p5/GFS/ -samefile /STORAGES/g1r5p5/GFS/3da46e07-d1ea-4f10-9250-6cbbb7b94d80/dom_md/ids -print -delete "<br> <br> ----------------------><br> Well at first I'll  fix permittions from mount points  to 660 .<br> If "ids"  file will be writeable , can't  became gluster colaps ??<br> <br> regs.Pavel<br> <br> <br> <div class="moz-cite-prefix">On 2.3.2016 08:16, Ravishankar N wrote:<br> </div> <blockquote cite="mid:56D69365.4090303@redhat.com" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 03/02/2016 12:02 PM, Sahina Bose wrote:<br> </div> <blockquote cite="mid:56D68910.8040602@redhat.com" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <br> <br> <div class="moz-cite-prefix">On 03/02/2016 03:45 AM, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr">On Tue, Mar 1, 2016 at 10:51 PM, <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a> <<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a>> wrote:<br> ><br> > HI,<br> > requested output:<br> ><br> > # ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md<br> >  <br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:<br> > total 2,1M<br> > -rw-rw---- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- good<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.17 leases<br> > -rw-r--r-- 1 vdsm kvm  335  7. lis 22.17 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.41 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 03.56 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 03.56 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.14 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.14 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.51 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases<br> > -rw-r--r-- 1 vdsm kvm  998 25. úno 00.35 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.16 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.44 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 00.18 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 00.18 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:<br> > total 1,1M<br> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.32 ids        <-- bad (other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.18 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.18 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:<br> > total 3,0M<br> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- bad (other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.42 inbox <br> > -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases<br> > -rw-r--r-- 1 vdsm kvm  997 24. úno 02.46 metadata<br> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.44 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:<br> > total 2,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.34 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.35 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases<br> > -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.27 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:<br> > total 3,0M<br> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 23.50 inbox        <-- bad (other can read) <div>> -rw-rw-r-- 1 vdsm kvm 2,0M  6. lis 23.51 leases        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  734  7. lis 02.13 metadata        <-- bad (group can write, other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 16.55 outbox        <-- bad (other can read)<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:<br> > total 1,1M<br> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.35 ids        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M 24. úno 01.06 inbox<br> > -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases<br> > -rw-r--r-- 1 vdsm kvm  998 24. úno 19.07 metadata<br> > -rw-rw-r-- 1 vdsm kvm  16M  7. lis 22.20 outbox<br> <br> <br> It should look like this:<br> <br> -rw-rw----. 1 vdsm kvm 1.0M Mar  1 23:36 ids<br> -rw-rw----. 1 vdsm kvm 2.0M Mar  1 23:35 leases<br> -rw-r--r--. 1 vdsm kvm  353 Mar  1 23:35 metadata<br> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 outbox<br> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 inbox<br> <br> This explains the EACCES error.<br> <br> You can start by fixing the permissions manually, you can do this online.<br>  <br> >  The ids files was generated by "touch" command after deleting them due "sanlock locking hang"  gluster crash & reboot<br> > I expected that they will be filled automaticaly after gluster reboot ( the  shadow copy from   ".gluster "   directory  was deleted & created empty  too )<br> <br> I don't know about gluster shadow copy, I would not play with gluster internals.</div> <div>Adding Sahina for advice.<br> </div> </div> </blockquote> <br> Did you generate the ids file on the mount point.<br> <br> Ravi, can you help here?<br> <br> </blockquote> <br> Okay, so what I understand from the output above is you have different gluster volumes mounted and some of them have incorrect permissions for the 'ids' file. The way to fix it is to do it from the mount like Nir said.<br> Why did you delete the file from the .glusterfs in the brick(s)?  Was there a gfid split brain? <br> <br> -Ravi<br> <br> <blockquote cite="mid:56D68910.8040602@redhat.com" type="cite"> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr"> <div><br> > OK, it looks that sanlock  can't work with empty file or rewrite them .<br> > Am I right ??<br> <br> Yes, the files must be initialized before sanlock can use them.<br> <br> You can initialize the file like this:<br> <br> sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0<br> <br> Taken from <a moz-do-not-send="true" href="http://lists.ovirt.org/pipermail/users/2016-February/038046.html">http://lists.ovirt.org/pipermail/users/2016-February/038046.html</a><br> <br> > The last point - about "ids" workaround - this is offline version = VMs have to be moved out from for continual running with maintenance volume mode<br> > But this is not acceptable in current situation, so the question again,  is it safe to do it online ??  ( YES / NO )</div> <div><br> </div> <div>The ids file is accessed only by sanlock. I guess that you don't have a running</div> <div>SPM on this DC, since sanlock fails to acquire a host id, so you are pretty safe</div> <div>to fix the permissions and initialize the ids files.</div> <div><br> </div> <div>I would do this:</div> <div><br> </div> <div>1. Stop engine,  so it will not try to start vdsm</div> <div>2. Stop vdsm on all hosts, so they do not try to acquire a host id with sanlock</div> <div>    This does not affect running vms</div> <div>3. Fix the permissions on the ids file, via glusterfs mount</div> <div>4. Initialize the ids files from one of the hosts, via the glusterfs mount</div> <div>    This should fix the ids files on all replicas</div> <div>5. Start vdsm on all hosts</div> <div>6. Start engine</div> <div><br> </div> <div>Engine will connect to all hosts, hosts will connect to storage and try to acquire a host id.</div> <div>Then Engine will start the SPM on one of the hosts, and your DC should become up.</div> <div><br> </div> <div>David, Sahina, can you confirm that this procedure is safe?</div> </div> </blockquote> <br> Yes, correcting from the mount point should fix it on all replicas<br> <br> <br> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr"> <div><br> </div> <div>Nir</div> <div><br> </div> <div>><br> > regs.<br> > Pavel<br> ><br> ><br> ><br> > On 1.3.2016 18:38, Nir Soffer wrote:<br> ><br> > On Tue, Mar 1, 2016 at 5:07 PM, <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a> <<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a>> wrote:<br> >><br> >> Hello,  can anybody  explain this error no.13 ( open file ) in sanlock.log .<br> ><br> ><br> > This is EACCES<br> ><br> > Can you share the outoput of:<br> ><br> >     ls -lh /rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md<br> >  <br> >><br> >><br> >> The size of  "ids" file is zero (0)<br> ><br> ><br> > This is how we create the ids file when initializing it.<br> ><br> > But then we use sanlock to initialize the ids file, and it should be 1MiB after that.<br> ><br> > Is this ids files created by vdsm, or one you created yourself?<br> >  <br> >><br> >> 2016-02-28 03:25:46+0100 269626 [1951]: open error -13 /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids<br> >> 2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids error -13<br> >> 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0<br> >><br> >> If the main problem is about zero file size, can I regenerate  this file online securely , with no VM dependence  ????<br> ><br> ><br> > Yes, I think I already referred to the instructions how to do that in a previous mail.<br> ><br> >><br> >><br> >> dist = RHEL - 7 - 2.1511<br> >> kernel = 3.10.0 - 327.10.1.el7.x86_64<br> >> KVM = 2.3.0 - 29.1.el7<br> >> libvirt = libvirt-1.2.17-13.el7_2.3<br> >> vdsm = vdsm-4.16.30-0.el7<br> >> GlusterFS = glusterfs-3.7.8-1.el7<br> >><br> >><br> >> regs.<br> >> Pavel<br> >><br> >> _______________________________________________<br> >> Users mailing list<br> >> <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> >> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> >><br> ><br> ><br> </div> </div> </blockquote> <br> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Gluster-users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre> </blockquote> <br> <br> </blockquote> <br> </body> </html> --------------020603060204030604090206--

This is a multi-part message in MIME format. --------------000700000508080601030108 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit UPDATE: all "ids" file have permittion fixed to 660 now # find /STORAGES -name ids -exec ls -l {} \; -rw-rw---- 2 vdsm kvm 0 24. úno 07.41 /STORAGES/g1r5p1/GFS/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md/ids -rw-rw---- 2 vdsm kvm 0 24. úno 07.43 /STORAGES/g1r5p2/GFS/88adbd49-62d6-45b1-9992-b04464a04112/dom_md/ids -rw-rw---- 2 vdsm kvm 0 24. úno 07.43 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -rw-rw---- 2 vdsm kvm 0 24. úno 07.44 /STORAGES/g1r5p4/GFS/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids -rw-rw---- 2 vdsm kvm 1048576 24. úno 13.03 /STORAGES/g1r5p5/GFS/3b24d023-fd35-4666-af2f-f5e1d19531ad/dom_md/ids -rw-rw---- 2 vdsm kvm 1048576 2. bře 17.47 /STORAGES/g2r5p1/GFS/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids SPM is and was running continually ....... I tried to update "ids" file - ONLINE ( offline not possible yet ) # sanlock direct init -s 3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 # find /STORAGES -name ids -exec ls -l {} \; | grep p3 -rw-rw---- 1 vdsm kvm 1048576 2. bře 18.32 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids The storage ids file has correct permittions, size, owners , but is not checking by sanlock = the same access time What's wrong ?? regs. Pa. PS: # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids = missing "shadowfile" in " .gluster " dir. How can I fix it ?? - online ! On 2.3.2016 08:16, Ravishankar N wrote:

On 03/02/2016 12:02 PM, Sahina Bose wrote:

...

On 03/02/2016 03:45 AM, Nir Soffer wrote:

...

On Tue, Mar 1, 2016 at 10:51 PM, paf1@email.cz <paf1@email.cz> wrote:

...

HI, requested output:

# ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:

...

total 2,1M -rw-rw---- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- good -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.17 leases -rw-r--r-- 1 vdsm kvm 335 7. lis 22.17 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.41 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 03.56 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 03.56 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.14 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.14 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.14 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.15 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.43 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.51 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases -rw-r--r-- 1 vdsm kvm 998 25. úno 00.35 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.16 outbox

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:

...

total 1,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.44 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 00.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 00.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 00.17 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:

...

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.32 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 inbox -rw-rw---- 1 vdsm kvm 2,0M 7. lis 22.18 leases -rw-r--r-- 1 vdsm kvm 333 7. lis 22.18 metadata -rw-rw---- 1 vdsm kvm 16M 7. lis 22.18 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:

...

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw---- 1 vdsm kvm 16M 25. úno 00.42 inbox -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases -rw-r--r-- 1 vdsm kvm 997 24. úno 02.46 metadata -rw-rw---- 1 vdsm kvm 16M 25. úno 00.44 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:

...

total 2,1M -rw-r--r-- 1 vdsm kvm 0 24. úno 07.34 ids <-- bad (sanlock cannot write, other can read) -rw-rw---- 1 vdsm kvm 16M 23. úno 22.35 inbox -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata -rw-rw---- 1 vdsm kvm 16M 23. úno 22.27 outbox

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:

...

total 3,0M -rw-rw-r-- 1 vdsm kvm 1,0M 1. bře 21.28 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 23.50 inbox <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 2,0M 6. lis 23.51 leases <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 734 7. lis 02.13 metadata <-- bad (group can write, other can read) -rw-rw-r-- 1 vdsm kvm 16M 6. lis 16.55 outbox <-- bad (other can read)

/rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:

...

total 1,1M -rw-rw-r-- 1 vdsm kvm 0 24. úno 07.35 ids <-- bad (other can read) -rw-rw-r-- 1 vdsm kvm 16M 24. úno 01.06 inbox -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases -rw-r--r-- 1 vdsm kvm 998 24. úno 19.07 metadata -rw-rw-r-- 1 vdsm kvm 16M 7. lis 22.20 outbox

It should look like this:

-rw-rw----. 1 vdsm kvm 1.0M Mar 1 23:36 ids -rw-rw----. 1 vdsm kvm 2.0M Mar 1 23:35 leases -rw-r--r--. 1 vdsm kvm 353 Mar 1 23:35 metadata -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 outbox -rw-rw----. 1 vdsm kvm 16M Mar 1 23:34 inbox

This explains the EACCES error.

You can start by fixing the permissions manually, you can do this online.

...

The ids files was generated by "touch" command after deleting them due "sanlock locking hang" gluster crash & reboot I expected that they will be filled automaticaly after gluster reboot ( the shadow copy from ".gluster " directory was deleted & created empty too )

I don't know about gluster shadow copy, I would not play with gluster internals. Adding Sahina for advice.

Did you generate the ids file on the mount point.

Ravi, can you help here?

Okay, so what I understand from the output above is you have different gluster volumes mounted and some of them have incorrect permissions for the 'ids' file. The way to fix it is to do it from the mount like Nir said. Why did you delete the file from the .glusterfs in the brick(s)? Was there a gfid split brain?

-Ravi

...

...

...

OK, it looks that sanlock can't work with empty file or rewrite them . Am I right ??

Yes, the files must be initialized before sanlock can use them.

You can initialize the file like this:

sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0

Taken from http://lists.ovirt.org/pipermail/users/2016-February/038046.html

...

The last point - about "ids" workaround - this is offline version = VMs have to be moved out from for continual running with maintenance volume mode But this is not acceptable in current situation, so the question again, is it safe to do it online ?? ( YES / NO )

The ids file is accessed only by sanlock. I guess that you don't have a running SPM on this DC, since sanlock fails to acquire a host id, so you are pretty safe to fix the permissions and initialize the ids files.

I would do this:

1. Stop engine, so it will not try to start vdsm 2. Stop vdsm on all hosts, so they do not try to acquire a host id with sanlock This does not affect running vms 3. Fix the permissions on the ids file, via glusterfs mount 4. Initialize the ids files from one of the hosts, via the glusterfs mount This should fix the ids files on all replicas 5. Start vdsm on all hosts 6. Start engine

Engine will connect to all hosts, hosts will connect to storage and try to acquire a host id. Then Engine will start the SPM on one of the hosts, and your DC should become up.

David, Sahina, can you confirm that this procedure is safe?

Yes, correcting from the mount point should fix it on all replicas

...

Nir

...

regs. Pavel

On 1.3.2016 18:38, Nir Soffer wrote:

On Tue, Mar 1, 2016 at 5:07 PM, paf1@email.cz <paf1@email.cz> wrote:

...

Hello, can anybody explain this error no.13 ( open file ) in

sanlock.log .

...

This is EACCES

Can you share the outoput of:

ls -lh /rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md

...

The size of "ids" file is zero (0)

This is how we create the ids file when initializing it.

But then we use sanlock to initialize the ids file, and it should

be 1MiB after that.

...

Is this ids files created by vdsm, or one you created yourself?

...

2016-02-28 03:25:46+0100 269626 [1951]: open error -13

...

...

2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids error -13 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0

If the main problem is about zero file size, can I regenerate

/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids this file online securely , with no VM dependence ????

...

Yes, I think I already referred to the instructions how to do that

in a previous mail.

...

...

dist = RHEL - 7 - 2.1511 kernel = 3.10.0 - 327.10.1.el7.x86_64 KVM = 2.3.0 - 29.1.el7 libvirt = libvirt-1.2.17-13.el7_2.3 vdsm = vdsm-4.16.30-0.el7 GlusterFS = glusterfs-3.7.8-1.el7

regs. Pavel

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users

_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users

--------------000700000508080601030108 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000066" bgcolor="#FFFFFF"> UPDATE:<br> <br> all "ids"  file have permittion fixed to 660 now<br> <br> #  find /STORAGES -name ids -exec ls -l {} \;<br> -rw-rw---- 2 vdsm kvm 0 24. úno 07.41 /STORAGES/g1r5p1/GFS/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md/ids<br> -rw-rw---- 2 vdsm kvm 0 24. úno 07.43 /STORAGES/g1r5p2/GFS/88adbd49-62d6-45b1-9992-b04464a04112/dom_md/ids<br> -rw-rw---- 2 vdsm kvm 0 24. úno 07.43 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids<br> -rw-rw---- 2 vdsm kvm 0 24. úno 07.44 /STORAGES/g1r5p4/GFS/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids<br> -rw-rw---- 2 vdsm kvm 1048576 24. úno 13.03 /STORAGES/g1r5p5/GFS/3b24d023-fd35-4666-af2f-f5e1d19531ad/dom_md/ids<br> -rw-rw---- 2 vdsm kvm 1048576  2. bře 17.47 /STORAGES/g2r5p1/GFS/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md/ids<br> <br> SPM is and  was  running continually .......<br> <br> I tried to update "ids" file - ONLINE  ( offline not possible yet )<br> # sanlock direct init -s 3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0<br> <br> #  find /STORAGES -name ids -exec ls -l {} \; | grep p3<br> -rw-rw---- 1 vdsm kvm 1048576  2. bře 18.32 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids<br> <br> The storage ids file has correct permittions, size, owners , but is not checking by sanlock = the same access time <br> What's wrong ??<br> <br> regs.<br> Pa.<br> PS:  # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print<br> /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids<br> = missing "shadowfile" in " .gluster " dir.<br> How can I fix it ?? - online !<br> <br> <br> <br> <div class="moz-cite-prefix">On 2.3.2016 08:16, Ravishankar N wrote:<br> </div> <blockquote cite="mid:56D69365.4090303@redhat.com" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 03/02/2016 12:02 PM, Sahina Bose wrote:<br> </div> <blockquote cite="mid:56D68910.8040602@redhat.com" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <br> <br> <div class="moz-cite-prefix">On 03/02/2016 03:45 AM, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr">On Tue, Mar 1, 2016 at 10:51 PM, <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a> <<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a>> wrote:<br> ><br> > HI,<br> > requested output:<br> ><br> > # ls -lh /rhev/data-center/mnt/glusterSD/localhost:*/*/dom_md<br> >  <br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-BCK/0fcad888-d573-47be-bef3-0bc0b7a99fb7/dom_md:<br> > total 2,1M<br> > -rw-rw---- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- good<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.17 leases<br> > -rw-r--r-- 1 vdsm kvm  335  7. lis 22.17 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.16 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P1/553d9b92-e4a0-4042-a579-4cabeb55ded4/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.41 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 03.56 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 03.56 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.14 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P2/88adbd49-62d6-45b1-9992-b04464a04112/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.14 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.14 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.15 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P3/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.43 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.51 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 23.12 leases<br> > -rw-r--r-- 1 vdsm kvm  998 25. úno 00.35 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.16 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md:<br> > total 1,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.44 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 00.18 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 00.18 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 00.17 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P1/42d710a9-b844-43dc-be41-77002d1cd553/dom_md:<br> > total 1,1M<br> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.32 ids        <-- bad (other can read)<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M  7. lis 22.18 leases<br> > -rw-r--r-- 1 vdsm kvm  333  7. lis 22.18 metadata<br> > -rw-rw---- 1 vdsm kvm  16M  7. lis 22.18 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P2/ff71b47b-0f72-4528-9bfe-c3da888e47f0/dom_md:<br> > total 3,0M<br> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- bad (other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.42 inbox <br> > -rw-rw---- 1 vdsm kvm 2,0M 25. úno 00.44 leases<br> > -rw-r--r-- 1 vdsm kvm  997 24. úno 02.46 metadata<br> > -rw-rw---- 1 vdsm kvm  16M 25. úno 00.44 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P3/ef010d08-aed1-41c4-ba9a-e6d9bdecb4b4/dom_md:<br> > total 2,1M<br> > -rw-r--r-- 1 vdsm kvm    0 24. úno 07.34 ids        <-- bad (sanlock cannot write, other can read)<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.35 inbox<br> > -rw-rw---- 1 vdsm kvm 2,0M 23. úno 22.38 leases<br> > -rw-r--r-- 1 vdsm kvm 1,1K 24. úno 19.07 metadata<br> > -rw-rw---- 1 vdsm kvm  16M 23. úno 22.27 outbox<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12__P4/300e9ac8-3c2f-4703-9bb1-1df2130c7c97/dom_md:<br> > total 3,0M<br> > -rw-rw-r-- 1 vdsm kvm 1,0M  1. bře 21.28 ids        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 23.50 inbox        <-- bad (other can read) <div>> -rw-rw-r-- 1 vdsm kvm 2,0M  6. lis 23.51 leases        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  734  7. lis 02.13 metadata        <-- bad (group can write, other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M  6. lis 16.55 outbox        <-- bad (other can read)<br> ><br> > /rhev/data-center/mnt/glusterSD/localhost:_2KVM12-P5/1ca56b45-701e-4c22-9f59-3aebea4d8477/dom_md:<br> > total 1,1M<br> > -rw-rw-r-- 1 vdsm kvm    0 24. úno 07.35 ids        <-- bad (other can read)<br> > -rw-rw-r-- 1 vdsm kvm  16M 24. úno 01.06 inbox<br> > -rw-rw-r-- 1 vdsm kvm 2,0M 24. úno 02.44 leases<br> > -rw-r--r-- 1 vdsm kvm  998 24. úno 19.07 metadata<br> > -rw-rw-r-- 1 vdsm kvm  16M  7. lis 22.20 outbox<br> <br> <br> It should look like this:<br> <br> -rw-rw----. 1 vdsm kvm 1.0M Mar  1 23:36 ids<br> -rw-rw----. 1 vdsm kvm 2.0M Mar  1 23:35 leases<br> -rw-r--r--. 1 vdsm kvm  353 Mar  1 23:35 metadata<br> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 outbox<br> -rw-rw----. 1 vdsm kvm  16M Mar  1 23:34 inbox<br> <br> This explains the EACCES error.<br> <br> You can start by fixing the permissions manually, you can do this online.<br>  <br> >  The ids files was generated by "touch" command after deleting them due "sanlock locking hang"  gluster crash & reboot<br> > I expected that they will be filled automaticaly after gluster reboot ( the  shadow copy from   ".gluster "   directory  was deleted & created empty  too )<br> <br> I don't know about gluster shadow copy, I would not play with gluster internals.</div> <div>Adding Sahina for advice.<br> </div> </div> </blockquote> <br> Did you generate the ids file on the mount point.<br> <br> Ravi, can you help here?<br> <br> </blockquote> <br> Okay, so what I understand from the output above is you have different gluster volumes mounted and some of them have incorrect permissions for the 'ids' file. The way to fix it is to do it from the mount like Nir said.<br> Why did you delete the file from the .glusterfs in the brick(s)?  Was there a gfid split brain? <br> <br> -Ravi<br> <br> <blockquote cite="mid:56D68910.8040602@redhat.com" type="cite"> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr"> <div><br> > OK, it looks that sanlock  can't work with empty file or rewrite them .<br> > Am I right ??<br> <br> Yes, the files must be initialized before sanlock can use them.<br> <br> You can initialize the file like this:<br> <br> sanlock direct init -s <sd_uuid>:0:repair/<sd_uuid>/dom_md/ids:0<br> <br> Taken from <a moz-do-not-send="true" href="http://lists.ovirt.org/pipermail/users/2016-February/038046.html">http://lists.ovirt.org/pipermail/users/2016-February/038046.html</a><br> <br> > The last point - about "ids" workaround - this is offline version = VMs have to be moved out from for continual running with maintenance volume mode<br> > But this is not acceptable in current situation, so the question again,  is it safe to do it online ??  ( YES / NO )</div> <div><br> </div> <div>The ids file is accessed only by sanlock. I guess that you don't have a running</div> <div>SPM on this DC, since sanlock fails to acquire a host id, so you are pretty safe</div> <div>to fix the permissions and initialize the ids files.</div> <div><br> </div> <div>I would do this:</div> <div><br> </div> <div>1. Stop engine,  so it will not try to start vdsm</div> <div>2. Stop vdsm on all hosts, so they do not try to acquire a host id with sanlock</div> <div>    This does not affect running vms</div> <div>3. Fix the permissions on the ids file, via glusterfs mount</div> <div>4. Initialize the ids files from one of the hosts, via the glusterfs mount</div> <div>    This should fix the ids files on all replicas</div> <div>5. Start vdsm on all hosts</div> <div>6. Start engine</div> <div><br> </div> <div>Engine will connect to all hosts, hosts will connect to storage and try to acquire a host id.</div> <div>Then Engine will start the SPM on one of the hosts, and your DC should become up.</div> <div><br> </div> <div>David, Sahina, can you confirm that this procedure is safe?</div> </div> </blockquote> <br> Yes, correcting from the mount point should fix it on all replicas<br> <br> <br> <blockquote cite="mid:CAMRbyyu9gwPfVpPxpDa4_gKWyXq1PavTm2V2rG2cU0AvE=JJPA@mail.gmail.com" type="cite"> <div dir="ltr"> <div><br> </div> <div>Nir</div> <div><br> </div> <div>><br> > regs.<br> > Pavel<br> ><br> ><br> ><br> > On 1.3.2016 18:38, Nir Soffer wrote:<br> ><br> > On Tue, Mar 1, 2016 at 5:07 PM, <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a> <<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a>> wrote:<br> >><br> >> Hello,  can anybody  explain this error no.13 ( open file ) in sanlock.log .<br> ><br> ><br> > This is EACCES<br> ><br> > Can you share the outoput of:<br> ><br> >     ls -lh /rhev/data-center/mnt/<server>:<_path>/<sd_uuid>/dom_md<br> >  <br> >><br> >><br> >> The size of  "ids" file is zero (0)<br> ><br> ><br> > This is how we create the ids file when initializing it.<br> ><br> > But then we use sanlock to initialize the ids file, and it should be 1MiB after that.<br> ><br> > Is this ids files created by vdsm, or one you created yourself?<br> >  <br> >><br> >> 2016-02-28 03:25:46+0100 269626 [1951]: open error -13 /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids<br> >> 2016-02-28 03:25:46+0100 269626 [1951]: s187985 open_disk /rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids error -13<br> >> 2016-02-28 03:25:56+0100 269636 [11304]: s187992 lockspace 7f52b697-c199-4f58-89aa-102d44327124:1:/rhev/data-center/mnt/glusterSD/localhost:_1KVM12-P4/7f52b697-c199-4f58-89aa-102d44327124/dom_md/ids:0<br> >><br> >> If the main problem is about zero file size, can I regenerate  this file online securely , with no VM dependence  ????<br> ><br> ><br> > Yes, I think I already referred to the instructions how to do that in a previous mail.<br> ><br> >><br> >><br> >> dist = RHEL - 7 - 2.1511<br> >> kernel = 3.10.0 - 327.10.1.el7.x86_64<br> >> KVM = 2.3.0 - 29.1.el7<br> >> libvirt = libvirt-1.2.17-13.el7_2.3<br> >> vdsm = vdsm-4.16.30-0.el7<br> >> GlusterFS = glusterfs-3.7.8-1.el7<br> >><br> >><br> >> regs.<br> >> Pavel<br> >><br> >> _______________________________________________<br> >> Users mailing list<br> >> <a moz-do-not-send="true" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> >> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> >><br> ><br> ><br> </div> </div> </blockquote> <br> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Gluster-users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre> </blockquote> <br> <br> </blockquote> <br> </body> </html> --------------000700000508080601030108--

This is a multi-part message in MIME format. --------------050101000701070505090304 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit On 03/03/2016 12:43 AM, Nir Soffer wrote:

PS: # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids = missing "shadowfile" in " .gluster " dir. How can I fix it ?? - online !

Ravi?

Is this the case in all 3 bricks of the replica? BTW, you can just stat the file on the brick and see the link count (it must be 2) instead of running the more expensive find command. --------------050101000701070505090304 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 03/03/2016 12:43 AM, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyysFqJwke8vdmYL7NGUuzpFNJppZC3riLR8jC7pd33DVdQ@mail.gmail.com" type="cite"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000066" bgcolor="#FFFFFF">PS:  # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print<br> /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids<br> = missing "shadowfile" in " .gluster " dir.<br> How can I fix it ?? - online !</div> </blockquote> <div><br> </div> <div>Ravi?</div> </blockquote> Is this the case in all 3 bricks of the replica? <br> BTW, you can just stat the file on the brick and see the link count (it must be 2) instead of running the more expensive find command.<br> <br> </body> </html> --------------050101000701070505090304--

This is a multi-part message in MIME format. --------------010103050402020505080306 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 03/03/2016 02:53 PM, paf1@email.cz wrote:

This is replica 2, only , with following settings

Options Reconfigured: performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: fixed Not sure why you have set this option. Ideally replica 3 or arbiter volumes are recommended for gluster+ovirt use. (client) quorum does not make sense for a 2 node setup. I have a detailed write up here which explains things http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volum... which explains things.

cluster.server-quorum-type: none storage.owner-uid: 36 storage.owner-gid: 36 cluster.quorum-count: 1 cluster.self-heal-daemon: enable

If I'll create "ids" file manually ( eg. " sanlock direct init -s 3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 " ) on both bricks, vdsm is writing only to half of them ( that with 2 links = correct ) "ids" file has correct permittions, owner, size on both bricks. brick 1: -rw-rw---- 1 vdsm kvm 1048576 2. bře 18.56 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - not updated

Okay, so this one has link count =1 which means the .glusterfs hardlink is missing. Can you try deleting this file from the brick and perform a stat on the file from the mount? That should heal (i.e recreate it ) on this brick from the other brick with the appropriate .glusterfs hard link.

brick 2: -rw-rw---- 2 vdsm kvm 1048576 3. bře 10.16 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - is continually updated

What happens when I'll restart vdsm ? Will oVirt storages go to "disable " state ??? = disconnect VMs storages ?

No idea on this one... -Ravi

regs.Pa.

On 3.3.2016 02:02, Ravishankar N wrote:

...

On 03/03/2016 12:43 AM, Nir Soffer wrote:

...

PS: # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids = missing "shadowfile" in " .gluster " dir. How can I fix it ?? - online !

Ravi?

Is this the case in all 3 bricks of the replica? BTW, you can just stat the file on the brick and see the link count (it must be 2) instead of running the more expensive find command.

--------------010103050402020505080306 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">On 03/03/2016 02:53 PM, <a class="moz-txt-link-abbreviated" href="mailto:paf1@email.cz">paf1@email.cz</a> wrote:<br> </div> <blockquote cite="mid:56D80297.6080407@email.cz" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> This is replica 2, only , with following settings<br> <br> Options Reconfigured:<br> performance.quick-read: off<br> performance.read-ahead: off<br> performance.io-cache: off<br> performance.stat-prefetch: off<br> cluster.eager-lock: enable<br> network.remote-dio: enable<br> cluster.quorum-type: fixed<br> </blockquote> Not sure why you have set this option.<br> Ideally replica 3 or arbiter volumes are recommended for gluster+ovirt use.  (client) quorum does not make sense for a 2 node setup. I have a detailed write up here which explains things <a class="moz-txt-link-freetext" href="http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/">http://gluster.readthedocs.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/</a> which explains things.<br> <br> <blockquote cite="mid:56D80297.6080407@email.cz" type="cite"> cluster.server-quorum-type: none<br> storage.owner-uid: 36<br> storage.owner-gid: 36<br> cluster.quorum-count: 1<br> cluster.self-heal-daemon: enable<br> <br> If I'll create "ids" file manually (  eg. " sanlock direct init -s 3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 " ) on both bricks,<br> vdsm is writing only to half of them ( that with 2 links = correct )<br> "ids" file has correct permittions, owner, size  on both bricks.<br> brick 1:  -rw-rw---- 1 vdsm kvm 1048576  2. bře 18.56 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - not updated<br> </blockquote> <br> Okay, so this one has link count =1 which means the .glusterfs hardlink is missing.  Can you try deleting this file from the brick and perform a stat on the file from the mount? That should heal (i.e recreate it ) on this brick from the other brick with the appropriate .glusterfs hard link.<br> <br> <br> <blockquote cite="mid:56D80297.6080407@email.cz" type="cite"> brick 2:  -rw-rw---- 2 vdsm kvm 1048576  3. bře 10.16 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - is continually updated<br> <br> What happens when I'll restart vdsm ? Will oVirt storages go to "disable " state ??? = disconnect VMs storages ?<br> </blockquote> <br> No idea on this one...<br> -Ravi<br> <blockquote cite="mid:56D80297.6080407@email.cz" type="cite"> <br> regs.Pa.<br> <br> <div class="moz-cite-prefix">On 3.3.2016 02:02, Ravishankar N wrote:<br> </div> <blockquote cite="mid:56D78D3B.9060101@redhat.com" type="cite"> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> <div class="moz-cite-prefix">On 03/03/2016 12:43 AM, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyysFqJwke8vdmYL7NGUuzpFNJppZC3riLR8jC7pd33DVdQ@mail.gmail.com" type="cite"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000066" bgcolor="#FFFFFF">PS:  # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print<br> /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids<br> = missing "shadowfile" in " .gluster " dir.<br> How can I fix it ?? - online !</div> </blockquote> <div><br> </div> <div>Ravi?</div> </blockquote> Is this the case in all 3 bricks of the replica? <br> BTW, you can just stat the file on the brick and see the link count (it must be 2) instead of running the more expensive find command.<br> <br> </blockquote> <br> </blockquote> <br> <br> </body> </html> --------------010103050402020505080306--

This is a multi-part message in MIME format. --------------050603070709080002010605 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit OK, will extend replica 2 to replica 3 ( arbiter ) ASAP . If is deleted "untouching" ids file on brick , healing of this file doesn't work . regs.Pa. On 3.3.2016 12:19, Nir Soffer wrote:

On Thu, Mar 3, 2016 at 11:23 AM, paf1@email.cz <mailto:paf1@email.cz> <paf1@email.cz <mailto:paf1@email.cz>> wrote:

This is replica 2, only , with following settings

Replica 2 is not supported. Even if you "fix" this now, you will have the same issue soon.

Options Reconfigured: performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.stat-prefetch: off cluster.eager-lock: enable network.remote-dio: enable cluster.quorum-type: fixed cluster.server-quorum-type: none storage.owner-uid: 36 storage.owner-gid: 36 cluster.quorum-count: 1 cluster.self-heal-daemon: enable

If I'll create "ids" file manually ( eg. " sanlock direct init -s 3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 " ) on both bricks, vdsm is writing only to half of them ( that with 2 links = correct ) "ids" file has correct permittions, owner, size on both bricks. brick 1: -rw-rw---- 1 vdsm kvm 1048576 2. bře 18.56 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - not updated brick 2: -rw-rw---- 2 vdsm kvm 1048576 3. bře 10.16 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - is continually updated

What happens when I'll restart vdsm ? Will oVirt storages go to "disable " state ??? = disconnect VMs storages ?

Nothing will happen, the vms will continue to run normally.

On block storage, stopping vdsm will prevent automatic extending of vm disks when the disk become too full, but on file based storage (like gluster) there is no issue.

regs.Pa.

On 3.3.2016 02:02, Ravishankar N wrote:

...

On 03/03/2016 12:43 AM, Nir Soffer wrote:

...

PS: # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids = missing "shadowfile" in " .gluster " dir. How can I fix it ?? - online !

Ravi?

Is this the case in all 3 bricks of the replica? BTW, you can just stat the file on the brick and see the link count (it must be 2) instead of running the more expensive find command.

--------------050603070709080002010605 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body text="#000066" bgcolor="#FFFFFF"> OK, <br> will extend replica 2 to replica 3 ( arbiter )  ASAP .<br> <br> If is deleted "untouching" ids file on brick , healing of this file doesn't work .<br> <br> regs.Pa.<br> <br> <div class="moz-cite-prefix">On 3.3.2016 12:19, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyyth7DEGXAGnbpxtNX73RBK73e7_Yb_z-wpbnkZHX=yMSQ@mail.gmail.com" type="cite"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote">On Thu, Mar 3, 2016 at 11:23 AM, <a moz-do-not-send="true" href="mailto:paf1@email.cz">paf1@email.cz</a> <span dir="ltr"><<a moz-do-not-send="true" href="mailto:paf1@email.cz" target="_blank">paf1@email.cz</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000066" bgcolor="#FFFFFF"> This is replica 2, only , with following settings<br> </div> </blockquote> <div><br> </div> <div>Replica 2 is not supported. Even if you "fix" this now, you will have the same issue</div> <div>soon.</div> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000066" bgcolor="#FFFFFF"> <br> Options Reconfigured:<br> performance.quick-read: off<br> performance.read-ahead: off<br> performance.io-cache: off<br> performance.stat-prefetch: off<br> cluster.eager-lock: enable<br> network.remote-dio: enable<br> cluster.quorum-type: fixed<br> cluster.server-quorum-type: none<br> storage.owner-uid: 36<br> storage.owner-gid: 36<br> cluster.quorum-count: 1<br> cluster.self-heal-daemon: enable<br> <br> If I'll create "ids" file manually (  eg. " sanlock direct init -s 3c34ad63-6c66-4e23-ab46-084f3d70b147:0:/STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids:0 " ) on both bricks,<br> vdsm is writing only to half of them ( that with 2 links = correct )<br> "ids" file has correct permittions, owner, size  on both bricks.<br> brick 1:  -rw-rw---- 1 vdsm kvm 1048576  2. bře 18.56 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - not updated<br> brick 2:  -rw-rw---- 2 vdsm kvm 1048576  3. bře 10.16 /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids - is continually updated<br> <br> What happens when I'll restart vdsm ? Will oVirt storages go to "disable " state ??? = disconnect VMs storages ?<br> </div> </blockquote> <div><br> </div> <div> Nothing will happen, the vms will continue to run normally.</div> <div><br> </div> <div>On block storage, stopping vdsm will prevent automatic extending of vm disks</div> <div>when the disk become too full, but on file based storage (like gluster) there is no issue.</div> <div> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000066" bgcolor="#FFFFFF"> <br> regs.Pa. <div> <div class="h5"><br> <br> <div>On 3.3.2016 02:02, Ravishankar N wrote:<br> </div> <blockquote type="cite"> <div>On 03/03/2016 12:43 AM, Nir Soffer wrote:<br> </div> <blockquote type="cite"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div text="#000066" bgcolor="#FFFFFF">PS:  # find /STORAGES -samefile /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids -print<br> /STORAGES/g1r5p3/GFS/3c34ad63-6c66-4e23-ab46-084f3d70b147/dom_md/ids<br> = missing "shadowfile" in " .gluster " dir.<br> How can I fix it ?? - online !</div> </blockquote> <div><br> </div> <div>Ravi?</div> </blockquote> Is this the case in all 3 bricks of the replica? <br> BTW, you can just stat the file on the brick and see the link count (it must be 2) instead of running the more expensive find command.<br> <br> </blockquote> <br> </div> </div> </div> </blockquote> </div> <br> </div> </div> </blockquote> <br> </body> </html> --------------050603070709080002010605--