3:40 p.m.
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd from each disk or
even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and rollback via 'yum
history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil NikolovOn Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <sbonazzo(a)redhat.com> ha
scritto:
>
>
>
> Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <fschmid(a)ubimet.com> ha
scritto:
>>
>> Hello,
>>
>> I have a question about bugs, which are flagged as [downstream clone - 4.3.7],
but are not yet released.
>>
>> I'm talking about this bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1749202
>>
>> I can't see it in 4.3.7 release notes. Will it be included in a further
release candidate? This fix is very important I think and I can't upgrade yet because
of this bug.
>
>
>
> Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaa
> ovirt-engine-4.3.6.6
> ovirt-engine-4.3.6.7
> ovirt-engine-4.3.7.0
> ovirt-engine-4.3.7.1
>
> So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes: https://github.com/oVirt/ovirt-site/pull/2143. @Ryan
Barry can you please review?
>
>
>
>
>
>
>>
>>
>> BR Florian Schmid
>>
>> ________________________________
>> Von: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
>> An: "users" <users(a)ovirt.org>
>> Gesendet: Mittwoch, 13. November 2019 13:34:59
>> Betreff: [ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now available
for testing
>>
>> The oVirt Project is pleased to announce the availability of the oVirt 4.3.7
Third Release Candidate for testing, as of November 13th, 2019.
>>
>> This update is a release candidate of the seventh in a series of stabilization
updates to the 4.3 series.
>> This is pre-release software. This pre-release should not to be used in
production.
>>
>> This release is available now on x86_64 architecture for:
>> * Red Hat Enterprise Linux 7.7 or later (but <8)
>> * CentOS Linux (or similar) 7.7 or later (but <8)
>>
>> This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for:
>> * Red Hat Enterprise Linux 7.7 or later (but <8)
>> * CentOS Linux (or similar) 7.7 or later (but <8)
>> * oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7
Release
>>
>> See the release notes [1] for known issues, new features and bugs fixed.
>>
>> While testing this release candidate please note that oVirt node now includes:
>> - ansible 2.9.0
>> - GlusterFS 6.6
>>
>> Notes:
>> - oVirt Appliance is already available
>> - oVirt Node is already available
>>
>> Additional Resources:
>> * Read more about the oVirt 4.3.7 release highlights:
http://www.ovirt.org/release/4.3.7/
>> * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
>> * Check out the latest project news on the oVirt blog:http://www.ovirt.org/blog/
>>
>> [1] http://www.ovirt.org/release/4.3.7/
>> [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
>>
>> --
>>
>> Sandro Bonazzola
>>
>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>>
>> Red Hat EMEA
>>
>> sbonazzo(a)redhat.com
>>
>> Red Hat respects your work life balance. Therefore there is no need to answer
this email out of your office hours.
>>
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
>
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA
>
> sbonazzo(a)redhat.com
>
> Red Hat respects your work life balance. Therefore there is no need to answer this
email out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
5 p.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate the storage.It
seems that gluster itself is working, but vdsm and the oVirt stack cannot access the
storage - cannot create new VM disks, cannot start a VM and I'm on the verge of
redeploy.
Best Regards,Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil
<hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd from each disk or
even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and rollback via 'yum
history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <sbonazzo(a)redhat.com> ha
scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <fschmid(a)ubimet.com> ha
scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone - 4.3.7], but are not
yet released.
I'm talking about this bug:https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further release
candidate? This fix is very important I think and I can't upgrade yet because of this
bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaaovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes: https://github.com/oVirt/ovirt-site/pull/2143. @Ryan
Barry can you please review?
BR Florian Schmid
Von: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
An: "users" <users(a)ovirt.org>
Gesendet: Mittwoch, 13. November 2019 13:34:59
Betreff: [ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now available for
testing
The oVirt Project is pleased to announce the availability of the oVirt 4.3.7 Third Release
Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of stabilization updates to
the 4.3 series.
This is pre-release software. This pre-release should not to be used in production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
11:27 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
+Sahina Bose <sabose(a)redhat.com> +Gobinda Das <godas(a)redhat.com> +Nir Soffer
<nsoffer(a)redhat.com> +Tal Nisan <tnisan(a)redhat.com> can you please help
here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <
hunter86_bg(a)yahoo.com> ha scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate
the storage.
It seems that gluster itself is working, but vdsm and the oVirt stack
cannot access the storage - cannot create new VM disks, cannot start a VM
and I'm on the verge of redeploy.
Best Regards,
Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil <
hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd
from each disk or even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and
rollback via 'yum history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <
sbonazzo(a)redhat.com> ha scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <
fschmid(a)ubimet.com> ha scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone -
4.3.7], but are not yet released.
I'm talking about this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further
release candidate? This fix is very important I think and I can't upgrade
yet because of this bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaa
ovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes:
https://github.com/oVirt/ovirt-site/pull/2143. @Ryan Barry
<rbarry(a)redhat.com> can you please review?
BR Florian Schmid
------------------------------
*Von: *"Sandro Bonazzola" <sbonazzo(a)redhat.com>
*An: *"users" <users(a)ovirt.org>
*Gesendet: *Mittwoch, 13. November 2019 13:34:59
*Betreff: *[ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now
available for testing
The oVirt Project is pleased to announce the availability of the oVirt
4.3.7 Third Release Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of
stabilization updates to the 4.3 series.
This is pre-release software. This pre-release should not to be used in
production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming
CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now
includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights:
http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.*
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
11:37 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
+Sahina Bose <sabose(a)redhat.com> +Gobinda Das
<godas(a)redhat.com> +Nir
Soffer <nsoffer(a)redhat.com> +Tal Nisan <tnisan(a)redhat.com> can you please
help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <
hunter86_bg(a)yahoo.com> ha scritto:
> So far,
>
> I have rolled back the engine and the 3 hosts - still cannot manipulate
> the storage.
> It seems that gluster itself is working, but vdsm and the oVirt stack
> cannot access the storage - cannot create new VM disks, cannot start a VM
> and I'm on the verge of redeploy.
>
Any errors in vdsm logs? engine logs?
> Best Regards,
> Strahil Nikolov
>
> В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil <
> hunter86_bg(a)yahoo.com> написа:
>
>
> I got upgraded to RC3 and now cannot power any VM .
> Constantly getting I/O error, but checking at gluster level - I can dd
> from each disk or even create a new one.
>
> Removing the HighAvailability doesn't help.
>
> I guess I should restore the engine from the gluster snapshot and
> rollback via 'yum history undo last'.
>
> Does anyone else have my issues ?
>
> Best Regards,
> Strahil Nikolov
> On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
>
>
>
> Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <
> sbonazzo(a)redhat.com> ha scritto:
>
>
>
> Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <
> fschmid(a)ubimet.com> ha scritto:
>
> Hello,
>
> I have a question about bugs, which are flagged as [downstream clone -
> 4.3.7], but are not yet released.
>
> I'm talking about this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1749202
>
> I can't see it in 4.3.7 release notes. Will it be included in a further
> release candidate? This fix is very important I think and I can't upgrade
> yet because of this bug.
>
>
>
> Looking at the bug, the fix was done with $ git tag --contains
> 12bd5cb1fe7c95e29b4065fca968913722fe9eaa
> ovirt-engine-4.3.6.6
> ovirt-engine-4.3.6.7
> ovirt-engine-4.3.7.0
> ovirt-engine-4.3.7.1
>
> So the fix is already included in release oVirt 4.3.6.
>
>
> Sent a fix to 4.3.6 release notes:
> https://github.com/oVirt/ovirt-site/pull/2143. @Ryan Barry
> <rbarry(a)redhat.com> can you please review?
>
>
>
>
>
>
>
>
>
>
> BR Florian Schmid
>
> ------------------------------
> *Von: *"Sandro Bonazzola" <sbonazzo(a)redhat.com>
> *An: *"users" <users(a)ovirt.org>
> *Gesendet: *Mittwoch, 13. November 2019 13:34:59
> *Betreff: *[ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is
> now available for testing
>
> The oVirt Project is pleased to announce the availability of the oVirt
> 4.3.7 Third Release Candidate for testing, as of November 13th, 2019.
>
> This update is a release candidate of the seventh in a series of
> stabilization updates to the 4.3 series.
> This is pre-release software. This pre-release should not to be used in
> production.
>
> This release is available now on x86_64 architecture for:
> * Red Hat Enterprise Linux 7.7 or later (but <8)
> * CentOS Linux (or similar) 7.7 or later (but <8)
>
> This release supports Hypervisor Hosts on x86_64 and ppc64le
> architectures for:
> * Red Hat Enterprise Linux 7.7 or later (but <8)
> * CentOS Linux (or similar) 7.7 or later (but <8)
> * oVirt Node 4.3 (available for x86_64 only) has been built consuming
> CentOS 7.7 Release
>
> See the release notes [1] for known issues, new features and bugs fixed.
>
> While testing this release candidate please note that oVirt node now
> includes:
> - ansible 2.9.0
> - GlusterFS 6.6
>
> Notes:
> - oVirt Appliance is already available
> - oVirt Node is already available
>
> Additional Resources:
> * Read more about the oVirt 4.3.7 release highlights:
> http://www.ovirt.org/release/4.3.7/
> * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
> * Check out the latest project news on the oVirt blog:
> http://www.ovirt.org/blog/
>
> [1] http://www.ovirt.org/release/4.3.7/
> [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://www.redhat.com/>*Red Hat respects your work life balance.
> Therefore there is no need to answer this email out of your office hours.*
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
>
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://www.redhat.com/>*Red Hat respects your work life balance.
> Therefore there is no need to answer this email out of your office hours.
> <https://mojo.redhat.com/docs/DOC-1199578>*
>
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://www.redhat.com/>*Red Hat respects your work life balance.
> Therefore there is no need to answer this email out of your office hours.
> <https://mojo.redhat.com/docs/DOC-1199578>*
>
>
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
6:40 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi Sahina,
Sadly engine logs have no errors.I've got only an I/O error, but in the debug of the
vdsm I can clearly see that "qemu-img" is giving an "OK".During the
upgrade I got some metadata files pending heal, but I have recovered the conflict manually
and should be OK.Today I have defined one of the VMs manually (virsh define) and then
started it , but the issue is the same.It seems to be storage-related issue,as VMs that
are on specific domain can be started , but most of my VMs are on the fast storage domains
and none of them can be started.
After the gluster snapshot restore , the engine is having issues and I have to separately
investigate that (as I poweroff my HostedEngine before creating the snapshot).
The logs can be find at
: https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
Any ideas where to look at , as I can definitely read (using "dd if=disk" or
qemu-img info) the disks of the rhel7 VM ?
Best Regards,Strahil Nikolov
В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose
<sabose(a)redhat.com> написа:
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
+Sahina Bose +Gobinda Das +Nir Soffer +Tal Nisan can you please help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <hunter86_bg(a)yahoo.com> ha
scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate the storage.It
seems that gluster itself is working, but vdsm and the oVirt stack cannot access the
storage - cannot create new VM disks, cannot start a VM and I'm on the verge of
redeploy.
Any errors in vdsm logs? engine logs?
Best Regards,Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil
<hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd from each disk or
even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and rollback via 'yum
history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <sbonazzo(a)redhat.com> ha
scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <fschmid(a)ubimet.com> ha
scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone - 4.3.7], but are not
yet released.
I'm talking about this bug:https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further release
candidate? This fix is very important I think and I can't upgrade yet because of this
bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaaovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes: https://github.com/oVirt/ovirt-site/pull/2143. @Ryan
Barry can you please review?
BR Florian Schmid
Von: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
An: "users" <users(a)ovirt.org>
Gesendet: Mittwoch, 13. November 2019 13:34:59
Betreff: [ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now available for
testing
The oVirt Project is pleased to announce the availability of the oVirt 4.3.7 Third Release
Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of stabilization updates to
the 4.3 series.
This is pre-release software. This pre-release should not to be used in production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
7:01 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Tue, Nov 19, 2019 at 10:10 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi Sahina,
Sadly engine logs have no errors.
I've got only an I/O error, but in the debug of the vdsm I can clearly see
that "qemu-img" is giving an "OK".
During the upgrade I got some metadata files pending heal, but I have
recovered the conflict manually and should be OK.
Today I have defined one of the VMs manually (virsh define) and then
started it , but the issue is the same.
It seems to be storage-related issue,as VMs that are on specific domain
can be started , but most of my VMs are on the fast storage domains and
none of them can be started.
After the gluster snapshot restore , the engine is having issues and I
have to separately investigate that (as I poweroff my HostedEngine before
creating the snapshot).
The logs can be find at :
https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
Any ideas where to look at , as I can definitely read (using "dd if=disk"
or qemu-img info) the disks of the rhel7 VM ?
The vdsm logs have this:
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') abnormal vm stop device
ua-94f763e9-fd96-4bee-a6b2-31af841a918b error eother (vm:5075)
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') CPU stopped: onIOError
(vm:6062)
2019-11-17 10:21:23,893+0200 DEBUG (libvirt/events) [jsonrpc.Notification]
Sending event {"params": {"notify_time": 4356025830,
"b3c4d84a-9784-470c-b70e-7ad7cc45e913": {"status":
"WaitForLaunch",
"ioerror": {"alias":
"ua-94f763e9-fd96-4bee-a6b2-31af841a918b", "name":
"sda", "path":
"/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f"},
"pauseCode": "EOTHER"}}, "jsonrpc": "2.0",
"method":
"|virt|VM_status|b3c4d84a-9784-470c-b70e-7ad7cc45e913"} (__init__:181)
Can you check the permissions of the file
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f.
Was it reset after upgrade?
Are you able to copy this file to a different location and try running a VM
with this image?
Any errors in the mount log of gluster1:_data__fast volume?
Best Regards,
Strahil Nikolov
В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose <
sabose(a)redhat.com> написа:
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
+Sahina Bose <sabose(a)redhat.com> +Gobinda Das <godas(a)redhat.com> +Nir
Soffer <nsoffer(a)redhat.com> +Tal Nisan <tnisan(a)redhat.com> can you please
help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <
hunter86_bg(a)yahoo.com> ha scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate
the storage.
It seems that gluster itself is working, but vdsm and the oVirt stack
cannot access the storage - cannot create new VM disks, cannot start a VM
and I'm on the verge of redeploy.
Any errors in vdsm logs? engine logs?
Best Regards,
Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil <
hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd
from each disk or even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and
rollback via 'yum history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <
sbonazzo(a)redhat.com> ha scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <
fschmid(a)ubimet.com> ha scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone -
4.3.7], but are not yet released.
I'm talking about this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further
release candidate? This fix is very important I think and I can't upgrade
yet because of this bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaa
ovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes:
https://github.com/oVirt/ovirt-site/pull/2143. @Ryan Barry
<rbarry(a)redhat.com> can you please review?
BR Florian Schmid
------------------------------
*Von: *"Sandro Bonazzola" <sbonazzo(a)redhat.com>
*An: *"users" <users(a)ovirt.org>
*Gesendet: *Mittwoch, 13. November 2019 13:34:59
*Betreff: *[ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now
available for testing
The oVirt Project is pleased to announce the availability of the oVirt
4.3.7 Third Release Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of
stabilization updates to the 4.3 series.
This is pre-release software. This pre-release should not to be used in
production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming
CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now
includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights:
http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.*
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
6:17 p.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hello All,
my engine is back online , but I'm still having difficulties to make vdsm powerup the
systems.I think that the events generated today can lead me to the right direction(just an
example , many more are there):
VDSM ovirt3.localdomain command SpmStatusVDS failed: Cannot inquire
Lease(name='SDM',
path=u'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast3/ecc3bf0e-8214-45c1-98a6-0afa642e591f/dom_md/leases',
offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
directory')
I will try to collect a fresh log and see what is it complaining about this time.
Best Regards,Strahil Nikolov
Hi Sahina,
I have a strange situation:
1. When I try to access the file via 'sudo -u vdsm dd if=disk of=test bs=4M' the
command fails on aprox 60MB.
2. If I run same command as root , remove the file and then run again via vdsm user ->
this time no i/o error reported.
My guess is that I need to check what's going on the bricks
themselve ...
Best Regards,
Strahil Nikolov
В вторник, 19 ноември 2019 г., 0:02:16 ч. Гринуич-5, Sahina Bose
<sabose(a)redhat.com> написа:
On Tue, Nov 19, 2019 at 10:10 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi Sahina,
Sadly engine logs have no errors.I've got only an I/O error, but in the debug of the
vdsm I can clearly see that "qemu-img" is giving an "OK".During the
upgrade I got some metadata files pending heal, but I have recovered the conflict manually
and should be OK.Today I have defined one of the VMs manually (virsh define) and then
started it , but the issue is the same.It seems to be storage-related issue,as VMs that
are on specific domain can be started , but most of my VMs are on the fast storage domains
and none of them can be started.
After the gluster snapshot restore , the engine is having issues and I have to separately
investigate that (as I poweroff my HostedEngine before creating the snapshot).
The logs can be find at
: https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
Any ideas where to look at , as I can definitely read (using "dd if=disk" or
qemu-img info) the disks of the rhel7 VM ?
The vdsm logs have this:2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') abnormal vm stop device
ua-94f763e9-fd96-4bee-a6b2-31af841a918b error eother (vm:5075)
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') CPU stopped: onIOError (vm:6062)
2019-11-17 10:21:23,893+0200 DEBUG (libvirt/events) [jsonrpc.Notification] Sending event
{"params": {"notify_time": 4356025830,
"b3c4d84a-9784-470c-b70e-7ad7cc45e913": {"status":
"WaitForLaunch", "ioerror": {"alias":
"ua-94f763e9-fd96-4bee-a6b2-31af841a918b", "name": "sda",
"path":
"/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f"},
"pauseCode": "EOTHER"}}, "jsonrpc": "2.0",
"method": "|virt|VM_status|b3c4d84a-9784-470c-b70e-7ad7cc45e913"}
(__init__:181)
Can you check the permissions of the file
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f.
Was it reset after upgrade?
Are you able to copy this file to a different location and try running a VM with this
image?
Any errors in the mount log of gluster1:_data__fast volume?
Best Regards,Strahil Nikolov
В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose
<sabose(a)redhat.com> написа:
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
+Sahina Bose +Gobinda Das +Nir Soffer +Tal Nisan can you please help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <hunter86_bg(a)yahoo.com> ha
scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate the storage.It
seems that gluster itself is working, but vdsm and the oVirt stack cannot access the
storage - cannot create new VM disks, cannot start a VM and I'm on the verge of
redeploy.
Any errors in vdsm logs? engine logs?
Best Regards,Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil
<hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd from each disk or
even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and rollback via 'yum
history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <sbonazzo(a)redhat.com> ha
scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <fschmid(a)ubimet.com> ha
scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone - 4.3.7], but are not
yet released.
I'm talking about this bug:https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further release
candidate? This fix is very important I think and I can't upgrade yet because of this
bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaaovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes: https://github.com/oVirt/ovirt-site/pull/2143. @Ryan
Barry can you please review?
BR Florian Schmid
Von: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
An: "users" <users(a)ovirt.org>
Gesendet: Mittwoch, 13. November 2019 13:34:59
Betreff: [ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now available for
testing
The oVirt Project is pleased to announce the availability of the oVirt 4.3.7 Third Release
Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of stabilization updates to
the 4.3 series.
This is pre-release software. This pre-release should not to be used in production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
2:33 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi All,
another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission
denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c
fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission
denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4
fd=0x7fc63c01a058 (Permission denied)
Anyone got an idea why is it happening?I checked user/group and selinux permissions - all
OKI even replaced one of the disks and healed , but the result is the same for all my
VMs.
Best Regards,Strahil Nikolov
В сряда, 20 ноември 2019 г., 18:17:18 ч. Гринуич+2, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hello All,
my engine is back online , but I'm still having difficulties to make vdsm powerup the
systems.I think that the events generated today can lead me to the right direction(just an
example , many more are there):
VDSM ovirt3.localdomain command SpmStatusVDS failed: Cannot inquire
Lease(name='SDM',
path=u'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast3/ecc3bf0e-8214-45c1-98a6-0afa642e591f/dom_md/leases',
offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
directory')
I will try to collect a fresh log and see what is it complaining about this time.
Best Regards,Strahil Nikolov
Hi Sahina,
I have a strange situation:
1. When I try to access the file via 'sudo -u vdsm dd if=disk of=test bs=4M' the
command fails on aprox 60MB.
2. If I run same command as root , remove the file and then run again via vdsm user ->
this time no i/o error reported.
My guess is that I need to check what's going on the bricks
themselve ...
Best Regards,
Strahil Nikolov
В вторник, 19 ноември 2019 г., 0:02:16 ч. Гринуич-5, Sahina Bose
<sabose(a)redhat.com> написа:
On Tue, Nov 19, 2019 at 10:10 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi Sahina,
Sadly engine logs have no errors.I've got only an I/O error, but in the debug of the
vdsm I can clearly see that "qemu-img" is giving an "OK".During the
upgrade I got some metadata files pending heal, but I have recovered the conflict manually
and should be OK.Today I have defined one of the VMs manually (virsh define) and then
started it , but the issue is the same.It seems to be storage-related issue,as VMs that
are on specific domain can be started , but most of my VMs are on the fast storage domains
and none of them can be started.
After the gluster snapshot restore , the engine is having issues and I have to separately
investigate that (as I poweroff my HostedEngine before creating the snapshot).
The logs can be find at
: https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
Any ideas where to look at , as I can definitely read (using "dd if=disk" or
qemu-img info) the disks of the rhel7 VM ?
The vdsm logs have this:2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') abnormal vm stop device
ua-94f763e9-fd96-4bee-a6b2-31af841a918b error eother (vm:5075)
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') CPU stopped: onIOError (vm:6062)
2019-11-17 10:21:23,893+0200 DEBUG (libvirt/events) [jsonrpc.Notification] Sending event
{"params": {"notify_time": 4356025830,
"b3c4d84a-9784-470c-b70e-7ad7cc45e913": {"status":
"WaitForLaunch", "ioerror": {"alias":
"ua-94f763e9-fd96-4bee-a6b2-31af841a918b", "name": "sda",
"path":
"/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f"},
"pauseCode": "EOTHER"}}, "jsonrpc": "2.0",
"method": "|virt|VM_status|b3c4d84a-9784-470c-b70e-7ad7cc45e913"}
(__init__:181)
Can you check the permissions of the file
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f.
Was it reset after upgrade?
Are you able to copy this file to a different location and try running a VM with this
image?
Any errors in the mount log of gluster1:_data__fast volume?
Best Regards,Strahil Nikolov
В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose
<sabose(a)redhat.com> написа:
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
+Sahina Bose +Gobinda Das +Nir Soffer +Tal Nisan can you please help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <hunter86_bg(a)yahoo.com> ha
scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate the storage.It
seems that gluster itself is working, but vdsm and the oVirt stack cannot access the
storage - cannot create new VM disks, cannot start a VM and I'm on the verge of
redeploy.
Any errors in vdsm logs? engine logs?
Best Regards,Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil
<hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd from each disk or
even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and rollback via 'yum
history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <sbonazzo(a)redhat.com> ha
scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <fschmid(a)ubimet.com> ha
scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone - 4.3.7], but are not
yet released.
I'm talking about this bug:https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further release
candidate? This fix is very important I think and I can't upgrade yet because of this
bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaaovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes: https://github.com/oVirt/ovirt-site/pull/2143. @Ryan
Barry can you please review?
BR Florian Schmid
Von: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
An: "users" <users(a)ovirt.org>
Gesendet: Mittwoch, 13. November 2019 13:34:59
Betreff: [ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now available for
testing
The oVirt Project is pleased to announce the availability of the oVirt 4.3.7 Third Release
Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of stabilization updates to
the 4.3 series.
This is pre-release software. This pre-release should not to be used in production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
8:19 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi All,
another clue in the logs :
[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536798] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536959] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.537007] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1
gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission
denied)
[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
[2019-11-21 00:30:13.132756] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.132824] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133217] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133238] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c
[Permission denied]
[2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1
gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission
denied)
[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489520] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489669] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489717] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4
[Permission denied]
[2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1
gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission
denied)
Anyone got an idea why is it happening?
I checked user/group and selinux permissions - all OK
I even replaced one of the disks and healed , but the result is the same
for all my VMs.
Have you checked the permission for user/group are set correctly across all
the bricks in the cluster?
What does ls -la on the images directory from mount of the volume show you.
Adding Krutika and Rafi as they ran into a similar issue in the past.
Best Regards,
Strahil Nikolov
В сряда, 20 ноември 2019 г., 18:17:18 ч. Гринуич+2, Strahil Nikolov <
hunter86_bg(a)yahoo.com> написа:
Hello All,
my engine is back online , but I'm still having difficulties to make vdsm
powerup the systems.
I think that the events generated today can lead me to the right
direction(just an example , many more are there):
VDSM ovirt3.localdomain command SpmStatusVDS failed: Cannot inquire
Lease(name='SDM',
path=u'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast3/ecc3bf0e-8214-45c1-98a6-0afa642e591f/dom_md/leases',
offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
directory')
I will try to collect a fresh log and see what is it complaining about
this time.
Best Regards,
Strahil Nikolov
>Hi Sahina,
>I have a strange situation:
>1. When I try to access the file via 'sudo -u vdsm dd if=disk of=test
bs=4M' the command fails on aprox 60MB.
>2. If I run same command as root , remove the file and then run again via
vdsm user -> this time no i/o error reported.
>My guess is that I need to check what's going on the bricks themselve ...
>Best Regards,
>Strahil Nikolov
В вторник, 19 ноември 2019 г., 0:02:16 ч. Гринуич-5, Sahina Bose <
sabose(a)redhat.com> написа:
On Tue, Nov 19, 2019 at 10:10 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi Sahina,
Sadly engine logs have no errors.
I've got only an I/O error, but in the debug of the vdsm I can clearly see
that "qemu-img" is giving an "OK".
During the upgrade I got some metadata files pending heal, but I have
recovered the conflict manually and should be OK.
Today I have defined one of the VMs manually (virsh define) and then
started it , but the issue is the same.
It seems to be storage-related issue,as VMs that are on specific domain
can be started , but most of my VMs are on the fast storage domains and
none of them can be started.
After the gluster snapshot restore , the engine is having issues and I
have to separately investigate that (as I poweroff my HostedEngine before
creating the snapshot).
The logs can be find at :
https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
Any ideas where to look at , as I can definitely read (using "dd if=disk"
or qemu-img info) the disks of the rhel7 VM ?
The vdsm logs have this:
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') abnormal vm stop device
ua-94f763e9-fd96-4bee-a6b2-31af841a918b error eother (vm:5075)
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') CPU stopped: onIOError
(vm:6062)
2019-11-17 10:21:23,893+0200 DEBUG (libvirt/events) [jsonrpc.Notification]
Sending event {"params": {"notify_time": 4356025830,
"b3c4d84a-9784-470c-b70e-7ad7cc45e913": {"status":
"WaitForLaunch",
"ioerror": {"alias":
"ua-94f763e9-fd96-4bee-a6b2-31af841a918b", "name":
"sda", "path":
"/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f"},
"pauseCode": "EOTHER"}}, "jsonrpc": "2.0",
"method":
"|virt|VM_status|b3c4d84a-9784-470c-b70e-7ad7cc45e913"} (__init__:181)
Can you check the permissions of the file
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f.
Was it reset after upgrade?
Are you able to copy this file to a different location and try running a
VM with this image?
Any errors in the mount log of gluster1:_data__fast volume?
Best Regards,
Strahil Nikolov
В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose <
sabose(a)redhat.com> написа:
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
+Sahina Bose <sabose(a)redhat.com> +Gobinda Das <godas(a)redhat.com> +Nir
Soffer <nsoffer(a)redhat.com> +Tal Nisan <tnisan(a)redhat.com> can you please
help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <
hunter86_bg(a)yahoo.com> ha scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate
the storage.
It seems that gluster itself is working, but vdsm and the oVirt stack
cannot access the storage - cannot create new VM disks, cannot start a VM
and I'm on the verge of redeploy.
Any errors in vdsm logs? engine logs?
Best Regards,
Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil <
hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd
from each disk or even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and
rollback via 'yum history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <
sbonazzo(a)redhat.com> ha scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <
fschmid(a)ubimet.com> ha scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone -
4.3.7], but are not yet released.
I'm talking about this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further
release candidate? This fix is very important I think and I can't upgrade
yet because of this bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaa
ovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes:
https://github.com/oVirt/ovirt-site/pull/2143. @Ryan Barry
<rbarry(a)redhat.com> can you please review?
BR Florian Schmid
------------------------------
*Von: *"Sandro Bonazzola" <sbonazzo(a)redhat.com>
*An: *"users" <users(a)ovirt.org>
*Gesendet: *Mittwoch, 13. November 2019 13:34:59
*Betreff: *[ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now
available for testing
The oVirt Project is pleased to announce the availability of the oVirt
4.3.7 Third Release Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of
stabilization updates to the 4.3 series.
This is pre-release software. This pre-release should not to be used in
production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming
CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now
includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights:
http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.*
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
11:58 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
> Hi All,
>
> another clue in the logs :
> [2019-11-21 00:29:50.536631] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
> remote operation failed. Path:
> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:29:50.536798] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
> remote operation failed. Path:
> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:29:50.536959] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
> remote operation failed. Path:
> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:29:50.537007] E [MSGID: 133010]
> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
> shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
> [Permission denied]
> [2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
> 0-glusterfs-fuse: 12458: READ => -1
> gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission
> denied)
> [2019-11-21 00:30:01.177665] I [MSGID: 133022]
> [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
> gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
> [2019-11-21 00:30:13.132756] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
> remote operation failed. Path:
> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:13.132824] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
> remote operation failed. Path:
> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:13.133217] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
> remote operation failed. Path:
> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:13.133238] E [MSGID: 133010]
> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
> shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c
> [Permission denied]
> [2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
> 0-glusterfs-fuse: 12660: READ => -1
> gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission
> denied)
> [2019-11-21 00:30:38.489449] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
> remote operation failed. Path:
> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:38.489520] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
> remote operation failed. Path:
> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:38.489669] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
> remote operation failed. Path:
> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:38.489717] E [MSGID: 133010]
> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
> shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4
> [Permission denied]
> [2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
> 0-glusterfs-fuse: 12928: READ => -1
> gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission
> denied)
>
>
> Anyone got an idea why is it happening?
> I checked user/group and selinux permissions - all OK
>
Can you share the commands (and output) used to check this?
I would try this on the hypervisor to check what vdsm/qemu see:
$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path
Also, to make sure we don't have selinux issue on the hypervisor, you can
change
selinux to permissive mode:
setenforce 0
And then try again. If this was selinux issue the permission denied issue
will disappear.
If this is the case please provide the output of:
ausearh -m AVC -ts today
If the issue still exists, we eliminated selinux, and you can enable it
again:
setenforce 1
Nir
I even replaced one of the disks and healed , but the result is the
same
> for all my VMs.
>
Have you checked the permission for user/group are set correctly across
all the bricks in the cluster?
What does ls -la on the images directory from mount of the volume show you.
Adding Krutika and Rafi as they ran into a similar issue in the past.
> Best Regards,
> Strahil Nikolov
>
>
> В сряда, 20 ноември 2019 г., 18:17:18 ч. Гринуич+2, Strahil Nikolov <
> hunter86_bg(a)yahoo.com> написа:
>
>
> Hello All,
>
> my engine is back online , but I'm still having difficulties to make vdsm
> powerup the systems.
> I think that the events generated today can lead me to the right
> direction(just an example , many more are there):
>
> VDSM ovirt3.localdomain command SpmStatusVDS failed: Cannot inquire
> Lease(name='SDM',
>
path=u'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast3/ecc3bf0e-8214-45c1-98a6-0afa642e591f/dom_md/leases',
> offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
> directory')
>
> I will try to collect a fresh log and see what is it complaining about
> this time.
>
> Best Regards,
> Strahil Nikolov
>
> >Hi Sahina,
>
> >I have a strange situation:
> >1. When I try to access the file via 'sudo -u vdsm dd if=disk of=test
> bs=4M' the command fails on aprox 60MB.
> >2. If I run same command as root , remove the file and then run again
> via vdsm user -> this time no i/o error reported.
>
> >My guess is that I need to check what's going on the bricks themselve ...
>
> >Best Regards,
> >Strahil Nikolov
>
>
> В вторник, 19 ноември 2019 г., 0:02:16 ч. Гринуич-5, Sahina Bose <
> sabose(a)redhat.com> написа:
>
>
>
>
> On Tue, Nov 19, 2019 at 10:10 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
> wrote:
>
> Hi Sahina,
>
> Sadly engine logs have no errors.
> I've got only an I/O error, but in the debug of the vdsm I can clearly
> see that "qemu-img" is giving an "OK".
> During the upgrade I got some metadata files pending heal, but I have
> recovered the conflict manually and should be OK.
> Today I have defined one of the VMs manually (virsh define) and then
> started it , but the issue is the same.
> It seems to be storage-related issue,as VMs that are on specific domain
> can be started , but most of my VMs are on the fast storage domains and
> none of them can be started.
>
> After the gluster snapshot restore , the engine is having issues and I
> have to separately investigate that (as I poweroff my HostedEngine before
> creating the snapshot).
>
> The logs can be find at :
> https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
>
>
> Any ideas where to look at , as I can definitely read (using "dd if=disk"
> or qemu-img info) the disks of the rhel7 VM ?
>
>
> The vdsm logs have this:
> 2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
> (vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') abnormal vm stop device
> ua-94f763e9-fd96-4bee-a6b2-31af841a918b error eother (vm:5075)
> 2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
> (vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') CPU stopped: onIOError
> (vm:6062)
> 2019-11-17 10:21:23,893+0200 DEBUG (libvirt/events)
> [jsonrpc.Notification] Sending event {"params": {"notify_time":
4356025830,
> "b3c4d84a-9784-470c-b70e-7ad7cc45e913": {"status":
"WaitForLaunch",
> "ioerror": {"alias":
"ua-94f763e9-fd96-4bee-a6b2-31af841a918b", "name":
> "sda", "path":
>
"/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f"},
> "pauseCode": "EOTHER"}}, "jsonrpc": "2.0",
"method":
> "|virt|VM_status|b3c4d84a-9784-470c-b70e-7ad7cc45e913"} (__init__:181)
>
> Can you check the permissions of the file
>
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f.
> Was it reset after upgrade?
>
> Are you able to copy this file to a different location and try running a
> VM with this image?
>
> Any errors in the mount log of gluster1:_data__fast volume?
>
>
> Best Regards,
> Strahil Nikolov
>
>
>
>
> В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose <
> sabose(a)redhat.com> написа:
>
>
>
>
> On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com>
> wrote:
>
> +Sahina Bose <sabose(a)redhat.com> +Gobinda Das <godas(a)redhat.com> +Nir
> Soffer <nsoffer(a)redhat.com> +Tal Nisan <tnisan(a)redhat.com> can you
> please help here?
>
>
> Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <
> hunter86_bg(a)yahoo.com> ha scritto:
>
> So far,
>
> I have rolled back the engine and the 3 hosts - still cannot manipulate
> the storage.
> It seems that gluster itself is working, but vdsm and the oVirt stack
> cannot access the storage - cannot create new VM disks, cannot start a VM
> and I'm on the verge of redeploy.
>
>
>
> Any errors in vdsm logs? engine logs?
>
>
>
> Best Regards,
> Strahil Nikolov
>
> В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil <
> hunter86_bg(a)yahoo.com> написа:
>
>
> I got upgraded to RC3 and now cannot power any VM .
> Constantly getting I/O error, but checking at gluster level - I can dd
> from each disk or even create a new one.
>
> Removing the HighAvailability doesn't help.
>
> I guess I should restore the engine from the gluster snapshot and
> rollback via 'yum history undo last'.
>
> Does anyone else have my issues ?
>
> Best Regards,
> Strahil Nikolov
> On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
>
>
>
> Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <
> sbonazzo(a)redhat.com> ha scritto:
>
>
>
> Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <
> fschmid(a)ubimet.com> ha scritto:
>
> Hello,
>
> I have a question about bugs, which are flagged as [downstream clone -
> 4.3.7], but are not yet released.
>
> I'm talking about this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1749202
>
> I can't see it in 4.3.7 release notes. Will it be included in a further
> release candidate? This fix is very important I think and I can't upgrade
> yet because of this bug.
>
>
>
> Looking at the bug, the fix was done with $ git tag --contains
> 12bd5cb1fe7c95e29b4065fca968913722fe9eaa
> ovirt-engine-4.3.6.6
> ovirt-engine-4.3.6.7
> ovirt-engine-4.3.7.0
> ovirt-engine-4.3.7.1
>
> So the fix is already included in release oVirt 4.3.6.
>
>
> Sent a fix to 4.3.6 release notes:
> https://github.com/oVirt/ovirt-site/pull/2143. @Ryan Barry
> <rbarry(a)redhat.com> can you please review?
>
>
>
>
>
>
>
>
>
>
> BR Florian Schmid
>
> ------------------------------
> *Von: *"Sandro Bonazzola" <sbonazzo(a)redhat.com>
> *An: *"users" <users(a)ovirt.org>
> *Gesendet: *Mittwoch, 13. November 2019 13:34:59
> *Betreff: *[ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is
> now available for testing
>
> The oVirt Project is pleased to announce the availability of the oVirt
> 4.3.7 Third Release Candidate for testing, as of November 13th, 2019.
>
> This update is a release candidate of the seventh in a series of
> stabilization updates to the 4.3 series.
> This is pre-release software. This pre-release should not to be used in
> production.
>
> This release is available now on x86_64 architecture for:
> * Red Hat Enterprise Linux 7.7 or later (but <8)
> * CentOS Linux (or similar) 7.7 or later (but <8)
>
> This release supports Hypervisor Hosts on x86_64 and ppc64le
> architectures for:
> * Red Hat Enterprise Linux 7.7 or later (but <8)
> * CentOS Linux (or similar) 7.7 or later (but <8)
> * oVirt Node 4.3 (available for x86_64 only) has been built consuming
> CentOS 7.7 Release
>
> See the release notes [1] for known issues, new features and bugs fixed.
>
> While testing this release candidate please note that oVirt node now
> includes:
> - ansible 2.9.0
> - GlusterFS 6.6
>
> Notes:
> - oVirt Appliance is already available
> - oVirt Node is already available
>
> Additional Resources:
> * Read more about the oVirt 4.3.7 release highlights:
> http://www.ovirt.org/release/4.3.7/
> * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
> * Check out the latest project news on the oVirt blog:
> http://www.ovirt.org/blog/
>
> [1] http://www.ovirt.org/release/4.3.7/
> [2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://www.redhat.com/>*Red Hat respects your work life balance.
> Therefore there is no need to answer this email out of your office hours.*
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
>
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://www.redhat.com/>*Red Hat respects your work life balance.
> Therefore there is no need to answer this email out of your office hours.
> <https://mojo.redhat.com/docs/DOC-1199578>*
>
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://www.redhat.com/>*Red Hat respects your work life balance.
> Therefore there is no need to answer this email out of your office hours.
> <https://mojo.redhat.com/docs/DOC-1199578>*
>
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://www.redhat.com/>*Red Hat respects your work life balance.
> Therefore there is no need to answer this email out of your office hours.
> <https://mojo.redhat.com/docs/DOC-1199578>*
>
>
10:41 p.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi All,
another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission
denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c
fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission
denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4
fd=0x7fc63c01a058 (Permission denied)
Anyone got an idea why is it happening?I checked user/group and selinux permissions - all
OK
Can you share the commands (and output) used to check this?I first
thought that the file is cached in memory and that's why vdsm user can read the file ,
but the following shows opposite:
[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# lltotal 562145-rw-rw----. 1 vdsm kvm 5368709120 Nov
12 23:29 5b1d3113-5cca-4582-9029-634b16338a2f-rw-rw----. 1 vdsm kvm 1048576 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.lease-rw-r--r--. 1 vdsm kvm 313 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.meta[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]#
pwd/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches [root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progressdd: error
reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied16+0 records in16+0
records out67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# dd if=5b1d3113-5cca-4582-9029-634b16338a2f
of=/dev/null bs=4M status=progress5356126208 bytes (5.4 GB) copied, 12.061393 s, 444
MB/s1280+0 records in1280+0 records out5368709120 bytes (5.4 GB) copied, 12.0876 s, 444
MB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress3598712832 bytes
(3.6 GB) copied, 1.000540 s, 3.6 GB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 1.47071 s, 3.7 GB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]#
echo 3 > /proc/sys/vm/drop_caches [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]#
sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M
status=progress5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s1280+0 records
in1280+0 records out5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do that.
I would try this on the hypervisor to check what vdsm/qemu see:
$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_pathI'm attaching the
output of the find I run, but this one should be enough:[root@ovirt1 ~]# find
/rhev/data-center/mnt/glusterSD/*/[0-9]*/images/ -not -user vdsm -print[root@ovirt1 ~]#
Also, to make sure we don't have selinux issue on the hypervisor,
you can change>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive when I tried
again.I'm 99.99% sure it's not selinux.
And then try again. If this was selinux issue the permission denied
issue will disappear.>If this is the case please provide the output of:
>
ausearh -m AVC -ts today
If the issue still exists, we eliminated selinux, and you can enable
it again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today<no matches>[root@ovirt2 ~]# ausearch -m
AVC -ts today<no matches>[root@ovirt1 ~]# ausearch -m AVC -ts today<no
matches>
I have a vague feeling that the issue is related to gluster v6.5 to 6.6 upgrade which I
several days before... So if any logs are needed (or debug enabled), just mention.
Nir
I even replaced one of the disks and healed , but the result is the same for all my VMs.
Have you checked the permission for user/group are set correctly across all the bricks in
the cluster?What does ls -la on the images directory from mount of the volume show you.
Adding Krutika and Rafi as they ran into a similar issue in the past.
Best Regards,Strahil Nikolov
В сряда, 20 ноември 2019 г., 18:17:18 ч. Гринуич+2, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hello All,
my engine is back online , but I'm still having difficulties to make vdsm powerup the
systems.I think that the events generated today can lead me to the right direction(just an
example , many more are there):
VDSM ovirt3.localdomain command SpmStatusVDS failed: Cannot inquire
Lease(name='SDM',
path=u'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast3/ecc3bf0e-8214-45c1-98a6-0afa642e591f/dom_md/leases',
offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
directory')
I will try to collect a fresh log and see what is it complaining about this time.
Best Regards,Strahil Nikolov
Hi Sahina,
I have a strange situation:
1. When I try to access the file via 'sudo -u vdsm dd if=disk of=test bs=4M' the
command fails on aprox 60MB.
2. If I run same command as root , remove the file and then run again via vdsm user ->
this time no i/o error reported.
My guess is that I need to check what's going on the bricks
themselve ...
Best Regards,
Strahil Nikolov
В вторник, 19 ноември 2019 г., 0:02:16 ч. Гринуич-5, Sahina Bose
<sabose(a)redhat.com> написа:
On Tue, Nov 19, 2019 at 10:10 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi Sahina,
Sadly engine logs have no errors.I've got only an I/O error, but in the debug of the
vdsm I can clearly see that "qemu-img" is giving an "OK".During the
upgrade I got some metadata files pending heal, but I have recovered the conflict manually
and should be OK.Today I have defined one of the VMs manually (virsh define) and then
started it , but the issue is the same.It seems to be storage-related issue,as VMs that
are on specific domain can be started , but most of my VMs are on the fast storage domains
and none of them can be started.
After the gluster snapshot restore , the engine is having issues and I have to separately
investigate that (as I poweroff my HostedEngine before creating the snapshot).
The logs can be find at
: https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
Any ideas where to look at , as I can definitely read (using "dd if=disk" or
qemu-img info) the disks of the rhel7 VM ?
The vdsm logs have this:2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') abnormal vm stop device
ua-94f763e9-fd96-4bee-a6b2-31af841a918b error eother (vm:5075)
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') CPU stopped: onIOError (vm:6062)
2019-11-17 10:21:23,893+0200 DEBUG (libvirt/events) [jsonrpc.Notification] Sending event
{"params": {"notify_time": 4356025830,
"b3c4d84a-9784-470c-b70e-7ad7cc45e913": {"status":
"WaitForLaunch", "ioerror": {"alias":
"ua-94f763e9-fd96-4bee-a6b2-31af841a918b", "name": "sda",
"path":
"/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f"},
"pauseCode": "EOTHER"}}, "jsonrpc": "2.0",
"method": "|virt|VM_status|b3c4d84a-9784-470c-b70e-7ad7cc45e913"}
(__init__:181)
Can you check the permissions of the file
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f.
Was it reset after upgrade?
Are you able to copy this file to a different location and try running a VM with this
image?
Any errors in the mount log of gluster1:_data__fast volume?
Best Regards,Strahil Nikolov
В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose
<sabose(a)redhat.com> написа:
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
+Sahina Bose +Gobinda Das +Nir Soffer +Tal Nisan can you please help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <hunter86_bg(a)yahoo.com> ha
scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate the storage.It
seems that gluster itself is working, but vdsm and the oVirt stack cannot access the
storage - cannot create new VM disks, cannot start a VM and I'm on the verge of
redeploy.
Any errors in vdsm logs? engine logs?
Best Regards,Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil
<hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd from each disk or
even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and rollback via 'yum
history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <sbonazzo(a)redhat.com> ha
scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <fschmid(a)ubimet.com> ha
scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone - 4.3.7], but are not
yet released.
I'm talking about this bug:https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further release
candidate? This fix is very important I think and I can't upgrade yet because of this
bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaaovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes: https://github.com/oVirt/ovirt-site/pull/2143. @Ryan
Barry can you please review?
BR Florian Schmid
Von: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
An: "users" <users(a)ovirt.org>
Gesendet: Mittwoch, 13. November 2019 13:34:59
Betreff: [ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now available for
testing
The oVirt Project is pleased to announce the availability of the oVirt 4.3.7 Third Release
Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of stabilization updates to
the 4.3 series.
This is pre-release software. This pre-release should not to be used in production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4YLTOEMD2RE...
11:41 p.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com>
wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi All,
another clue in the logs :
[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536798] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536959] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.537007] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1
gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission
denied)
[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
[2019-11-21 00:30:13.132756] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.132824] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133217] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133238] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c
[Permission denied]
[2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1
gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission
denied)
[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489520] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489669] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489717] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4
[Permission denied]
[2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1
gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission
denied)
Anyone got an idea why is it happening?
I checked user/group and selinux permissions - all OK
>Can you share the commands (and output) used to check this?
I first thought that the file is cached in memory and that's why vdsm user
can read the file , but the following shows opposite:
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# ll
total 562145
-rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29
5b1d3113-5cca-4582-9029-634b16338a2f
-rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.lease
-rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.meta
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always
use direct I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
dd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied...
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct bs=8M
status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for this
test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do
that.
Smells like issue on gluster side.
>I would try this on the hypervisor to check what vdsm/qemu see:
>$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path
I'm attaching the output of the find I run, but this one should be enough:
[root@ovirt1 ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/
-not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and
selinux label
of the entire tree will be more useful.
>Also, to make sure we don't have selinux issue on the
hypervisor, you can
change
>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive
when I tried again.I'm 99.99% sure it's not selinux.
>And then try again. If this was selinux issue the permission denied issue
will disappear.
>If this is the case please provide the output of:
> ausearh -m AVC -ts today
>If the issue still exists, we eliminated selinux, and you can enable it
again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt2 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt1 ~]# ausearch -m AVC -ts today
<no matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on
the gluster side?
I have a vague feeling that the issue is related to gluster v6.5 to
6.6
upgrade which I several days before... So if any logs are needed (or debug
enabled), just mention.
If this is the last change, and it worked before, most likely.
Nir
1:03 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi Nir,
thanks for your feedback. My guess is that as gluster is setup to use shard - the first
one (the volume file) is OK ,but then the shard is giving an error.
As gluster1=ovirt1 and gluster2=ovirt2 & ovirt3 is the arbiter , I think it's not
SELINUX issue (all nodes are still in permissive - just to be sure that nothing comes up)
on gluster level.
As I can't powerup any VM (in the data_fastX domains), I took more radical step and
enabled Trace log on the Fuse mount and started a VM.
As per the log , I got :[2019-11-23 17:15:16.961636] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449da
e.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-23 17:15:16.961669] D [MSGID: 0] [client-rpc-fops_v2.c:2642:client4_0_lookup_cbk]
0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-client-2 returned -1 error:
Permission denied
[Permission denied]
[2019-11-23 17:15:16.961705] D [MSGID: 0] [afr-common.c:2464:afr_lookup_done]
0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-replicate-0 returned -1 error:
Permission denied [Permissi
on denied]
[2019-11-23 17:15:16.961735] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 79
failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-23 17:15:16.961764] D [MSGID: 0] [shard.c:842:shard_common_failure_unwind]
0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-shard returned -1 error:
Permission denied [Permissi
on denied]
[2019-11-23 17:15:16.961793] D [MSGID: 0] [utime-autogen-fops.c:100:gf_utime_readv_cbk]
0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-utime returned -1 error:
Permission denied [Perm
ission denied]
[2019-11-23 17:15:16.961821] D [MSGID: 0] [defaults.c:1189:default_readv_cbk]
0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-write-behind returned -1 error:
Permission denied [Permiss
ion denied]
[2019-11-23 17:15:16.961848] D [MSGID: 0] [defaults.c:1189:default_readv_cbk]
0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-open-behind returned -1 error:
Permission denied [Permissi
on denied]
[2019-11-23 17:15:16.961876] D [MSGID: 0] [md-cache.c:2010:mdc_readv_cbk] 0-stack-trace:
stack-address: 0x7fbeb4005218, data_fast-md-cache returned -1 error: Permission denied
[Permission deni
ed]
[2019-11-23 17:15:16.961902] D [MSGID: 0] [defaults.c:1189:default_readv_cbk]
0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-io-threads returned -1 error:
Permission denied [Permissio
n denied]
[2019-11-23 17:15:16.961938] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 703:
READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fbeb4017018 (Permission
denied)
This indicates that the shard 79 for gfid b0af2b81-22cf-482e-9b2f-c431b6449dae has to be
checked.
So I first check the brick logs on ovirt1 from the moment when VM was powered
up:[2019-11-23 17:15:16.962013] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client:
CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:1842
4-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid:
be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-
fop:INVALID, acl:-) [Permission denied]
[2019-11-23 17:15:16.962113] E [MSGID: 115050]
[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 541: LOOKUP
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (be318638-e8a0-4c6d
-977d-7a937aa84806/b0af2b81-22cf-482e-9b2f-c431b6449dae.79), client:
CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:18424-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_N
O:-0, error-xlator: data_fast-access-control [Permission denied]
[2019-11-23 17:15:16.962096] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client:
CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:1842
4-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid:
be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-
fop:INVALID, acl:-) [Permission denied]
All of the above doesn't give me any clue what is it going about.And the actual
shard:
[root@ovirt1 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79
-rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0
b0af2b81-22cf-482e-9b2f-c431b6449dae.79
[root@ovirt1 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79
# file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31
trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739
trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1
[root@ovirt2 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79
-rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0
b0af2b81-22cf-482e-9b2f-c431b6449dae.79
[root@ovirt2 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79
# file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31
trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739
trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1
[root@ovirt3 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79
-rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0
b0af2b81-22cf-482e-9b2f-c431b6449dae.79
[root@ovirt3 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79
# file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31
trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739
trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1
Access for user qemu (uid 107/gid 107 -> gluster brick log on ovirt1) is OK:
[root@ovirt1 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79
of=/dev/null bs=8M iflag=direct status=progress
8+0 прочетени блока
8+0 записани блока
изкопирани са 67108864 байта (67 MB), 0,0130595 s, 5,1 GB/s
[root@ovirt2 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79
of=/dev/null bs=8M iflag=direct status=progress
8+0 прочетени блока
8+0 записани блока
изкопирани са 67108864 байта (67 MB), 0,0224745 s, 3,0 GB/s
[root@ovirt3 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79
of=/dev/null bs=8M iflag=direct status=progress
0+0 прочетени блока
0+0 записани блока
изкопирани са 0 байта (0 B), 0,00103977 s, 0,0 kB/s
[root@ovirt1 .shard]# grep 107 /etc/passwd /etc/group
/etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin
/etc/group:qemu:x:107:vdsm,sanlock
[root@ovirt2 ~]# grep 107 /etc/passwd /etc/group
/etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin
/etc/group:qemu:x:107:vdsm,sanlock
[root@ovirt3 .shard]# grep 107 /etc/passwd /etc/group
/etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin
/etc/group:qemu:x:107:vdsm,sanlock
I've created 3 strace files (1 as root and 2 as vdsm) and attached them to this
e-mail:
[root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm strace -fvttTyyx -s 4096
-o /tmp/dd-strace-vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null
iflag=direct bs=8M stat
us=progress
dd: грешка при четене на „5b1d3113-5cca-4582-9029-634b16338a2f“: Отказан достъп
8+0 прочетени блока
8+0 записани блока
изкопирани са 67108864 байта (67 MB), 0,118223 s, 568 MB/s
[root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# strace -fvttTyyx -s 4096 -o
/tmp/dd-strace-root dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null iflag=direct
bs=8M status=progress
изкопирани са 5083496448 байта (5,1 GB), 10,144199 s, 501 MB/s
640+0 прочетени блока
640+0 записани блока
изкопирани са 5368709120 байта (5,4 GB), 10,7444 s, 500 MB/s
[root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm strace -fvttTyyx -s 4096
-o /tmp/dd-strace-vdsm-2nd-try dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null
iflag=direct bs
=8M status=progress
изкопирани са 4815060992 байта (4,8 GB), 8,056124 s, 598 MB/s
640+0 прочетени блока
640+0 записани блока
изкопирани са 5368709120 байта (5,4 GB), 8,98862 s, 597 MB/s
P.S.: Sorry for the cyrrilic , it's my wife's laptop :). First case gives an error
and reads 67MB, while the second one works as expected.Then vdsm can access the volume
without issue (quite strange)!
В петък, 22 ноември 2019 г., 23:44:27 ч. Гринуич+2, Nir Soffer
<nsoffer(a)redhat.com> написа:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi All,
another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission
denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c
fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission
denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4
fd=0x7fc63c01a058 (Permission denied)
Anyone got an idea why is it happening?I checked user/group and selinux permissions - all
OK
Can you share the commands (and output) used to check this?I first
thought that the file is cached in memory and that's why vdsm user can read the file ,
but the following shows opposite:
[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# lltotal 562145-rw-rw----. 1 vdsm kvm 5368709120 Nov
12 23:29 5b1d3113-5cca-4582-9029-634b16338a2f-rw-rw----. 1 vdsm kvm 1048576 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.lease-rw-r--r--. 1 vdsm kvm 313 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.meta[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]#
pwd/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches
>I would use iflag=direct instead, no need to mess with caches.
Vdsm always use direct I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]#
sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M
status=progressdd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission
denied
>You got permissions denied...
16+0 records in16+0 records
out67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>And dd continue to read data?!
>I have never seen anything like this.
>It will be helpful to run this with strace:
> strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct bs=8M
status=progress
>And share dd.strace.
>Logs in /var/log/glusterfs/exportname.log will contain useful info for this test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5356126208 bytes
(5.4 GB) copied, 12.061393 s, 444 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.0876 s, 444 MB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]#
sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M
status=progress3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s1280+0 records
in1280+0 records out5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches [root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5171576832 bytes
(5.2 GB) copied, 12.071837 s, 428 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do that.
>Smells like issue on gluster side.
I would try this on the hypervisor to check what vdsm/qemu see:
$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_pathI'm
attaching the output of the find I run, but this one should be enough:[root@ovirt1 ~]#
find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/ -not -user vdsm -print
>A full output of ls -lahRZ, showing user, group, permissions bits,
and selinux label>>of the entire tree will be more useful.
[root@ovirt2
gluster1:_data__fast]# tree
.
├── 396604d9-2a9e-49cd-9563-fdc79981f67b
│ ├── dom_md
│ │ ├── ids
│ │ ├── inbox
│ │ ├── leases
│ │ ├── metadata
│ │ ├── outbox
│ │ └── xleases
│ ├── images
│ │ ├── 225e4c45-0f70-40aa-8d51-7b705f259cd7
│ │ │ ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5
│ │ │ ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5.lease
│ │ │ ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5.meta
│ │ │ ├── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d
│ │ │ ├── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d.lease
│ │ │ └── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d.meta
│ │ ├── 25107759-f56b-4d94-872e-0f9b145886d1
│ │ │ ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2
│ │ │ ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2.lease
│ │ │ ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
│ │ │ └── 571a5eb4-eb92-479c-8305-79eaa0673bb2.meta_old
│ │ ├── 2c79e7ab-47cc-4f41-a190-40cd94053d83
│ │ │ ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d
│ │ │ ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.lease
│ │ │ ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.meta
│ │ │ └── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.meta_old
│ │ ├── 36fed07b-a08f-4fb9-a273-9dbf0ad1f27b
│ │ │ ├── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9
│ │ │ ├── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9.lease
│ │ │ └── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9.meta
│ │ ├── 7d11479e-1a02-4a74-a9be-14b4e56faaa1
│ │ │ ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a
│ │ │ ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a.lease
│ │ │ ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a.meta
│ │ │ ├── a2b94064-9e34-47c9-8eab-867673c5e48f
│ │ │ ├── a2b94064-9e34-47c9-8eab-867673c5e48f.lease
│ │ │ └── a2b94064-9e34-47c9-8eab-867673c5e48f.meta
│ │ ├── 85f18244-6d5b-4d02-ab66-140bfd2d734b
│ │ │ ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108
│ │ │ ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108.lease
│ │ │ ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108.meta
│ │ │ ├── 116191b2-99a1-437b-8b7d-ab1927b5f4aa
│ │ │ ├── 116191b2-99a1-437b-8b7d-ab1927b5f4aa.lease
│ │ │ └── 116191b2-99a1-437b-8b7d-ab1927b5f4aa.meta
│ │ ├── 94f763e9-fd96-4bee-a6b2-31af841a918b
│ │ │ ├── 5b1d3113-5cca-4582-9029-634b16338a2f
│ │ │ ├── 5b1d3113-5cca-4582-9029-634b16338a2f.lease
│ │ │ └── 5b1d3113-5cca-4582-9029-634b16338a2f.meta
│ │ ├── 99c3c20c-a7bb-4851-8ae9-0b39c0c86b66
│ │ │ ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661
│ │ │ ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661.lease
│ │ │ ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661.meta
│ │ │ ├── 95c260da-8dfb-46be-9c4a-6a55eb3967d5
│ │ │ ├── 95c260da-8dfb-46be-9c4a-6a55eb3967d5.lease
│ │ │ └── 95c260da-8dfb-46be-9c4a-6a55eb3967d5.meta
│ │ ├── e56eb458-e459-4828-a268-82a9b91c60f8
│ │ │ ├── 5a7ca769-4657-4f65-92bd-72fa13cc2871
│ │ │ ├── 5a7ca769-4657-4f65-92bd-72fa13cc2871.lease
│ │ │ └── 5a7ca769-4657-4f65-92bd-72fa13cc2871.meta
│ │ └── f5336a0f-de34-482e-b437-47f9d276ed85
│ │ ├── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4
│ │ ├── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4.lease
│ │ └── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4.meta
│ └── master
└── __DIRECT_IO_TEST__
14 directories, 51 files
Full find of "data_fastX" domains are attached.
Also, to make sure we don't have selinux issue on the hypervisor,
you can change>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive when I tried
again.I'm 99.99% sure it's not selinux.
And then try again. If this was selinux issue the permission denied
issue will disappear.>If this is the case please provide the output of:
>
ausearh -m AVC -ts today
If the issue still exists, we eliminated selinux, and you can enable
it again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today<no matches>[root@ovirt2 ~]# ausearch -m
AVC -ts today<no matches>[root@ovirt1 ~]# ausearch -m AVC -ts today<no
matches>
So this is not selinux on the hypervisor. I wonder if it can be
selinux on the gluster side?
I have a vague feeling that the issue is related to
gluster v6.5 to 6.6 upgrade which I several days before... So if any logs are needed (or
debug enabled), just mention.
If this is the last change, and it worked before, most likely.
Gluster was patched several days before RC3 , but all the storage-related issues
lead me to that. The strage thing is that the HostedEngine is not affected, nor the
"data" storage domain .
Any ideas appreciated. I would like to find out what has caused this instead of just
taking the shortcut to redeploy the whole setup.
Best Regards,Strahil Nikolov
11:03 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov
<hunter86_bg(a)yahoo.com>
wrote:
> On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
>
>
>
> On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
> wrote:
>
> Hi All,
>
> another clue in the logs :
> [2019-11-21 00:29:50.536631] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
> remote operation failed. Path:
> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:29:50.536798] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
> remote operation failed. Path:
> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:29:50.536959] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
> remote operation failed. Path:
> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:29:50.537007] E [MSGID: 133010]
> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
> shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
> [Permission denied]
> [2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
> 0-glusterfs-fuse: 12458: READ => -1
> gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission
> denied)
> [2019-11-21 00:30:01.177665] I [MSGID: 133022]
> [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
> gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
> [2019-11-21 00:30:13.132756] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
> remote operation failed. Path:
> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:13.132824] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
> remote operation failed. Path:
> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:13.133217] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
> remote operation failed. Path:
> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:13.133238] E [MSGID: 133010]
> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
> shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c
> [Permission denied]
> [2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
> 0-glusterfs-fuse: 12660: READ => -1
> gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission
> denied)
> [2019-11-21 00:30:38.489449] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
> remote operation failed. Path:
> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:38.489520] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
> remote operation failed. Path:
> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:38.489669] W [MSGID: 114031]
> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
> remote operation failed. Path:
> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
> (00000000-0000-0000-0000-000000000000) [Permission denied]
> [2019-11-21 00:30:38.489717] E [MSGID: 133010]
> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
> shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4
> [Permission denied]
> [2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
> 0-glusterfs-fuse: 12928: READ => -1
> gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission
> denied)
>
>
> Anyone got an idea why is it happening?
> I checked user/group and selinux permissions - all OK
>
>
> >Can you share the commands (and output) used to check this?
> I first thought that the file is cached in memory and that's why vdsm
> user can read the file , but the following shows opposite:
>
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# ll
> total 562145
> -rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29
> 5b1d3113-5cca-4582-9029-634b16338a2f
> -rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11
> 5b1d3113-5cca-4582-9029-634b16338a2f.lease
> -rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11
> 5b1d3113-5cca-4582-9029-634b16338a2f.meta
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd
>
>
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
> /proc/sys/vm/drop_caches
>
I would use iflag=direct instead, no need to mess with caches. Vdsm always
use direct I/O.
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
> dd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission
> denied
>
You got permissions denied...
> 16+0 records in
> 16+0 records out
> 67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>
Seems like it could read upto ~67MB successfully before it encountered
'Permission denied' errors. Assuming a shard-block-size of 64MB, looks like
all the shards under /.shard could not be accessed.
Could you share the following pieces of information:
1. brick logs of data_fast
2. ls -la of .shard relative to the bricks (NOT the mount) on all the
bricks of data_fast
3. and ls -la of all shards under .shard of data_fast (perhaps a handful of
them have root permission assigned somehow which is causing access to be
denied? Perhaps while resolving pending heals manually? )
-Krutika
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct
bs=8M status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for
this test.
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
> 5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
> 3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
> /proc/sys/vm/drop_caches
> [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
> 5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s
>
> As you can see , once root user reads the file -> vdsm user can also do
> that.
>
Smells like issue on gluster side.
> >I would try this on the hypervisor to check what vdsm/qemu see:
>
> >$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path
> I'm attaching the output of the find I run, but this one should be enough:
> [root@ovirt1 ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/
> -not -user vdsm -print
>
A full output of ls -lahRZ, showing user, group, permissions bits, and
selinux label
of the entire tree will be more useful.
> >Also, to make sure we don't have selinux issue on the hypervisor, you
> can change
> >selinux to permissive mode:
>
> > setenforce 0
>
> This is the first thing I did and the systems were still in permissive
> when I tried again.I'm 99.99% sure it's not selinux.
>
>
> >And then try again. If this was selinux issue the permission denied
> issue will disappear.
> >If this is the case please provide the output of:
>
> > ausearh -m AVC -ts today
>
> >If the issue still exists, we eliminated selinux, and you can enable it
> again:
>
> > setenforce 1
>
> [root@ovirt3 ~]# ausearch -m AVC -ts today
> <no matches>
> [root@ovirt2 ~]# ausearch -m AVC -ts today
> <no matches>
> [root@ovirt1 ~]# ausearch -m AVC -ts today
> <no matches>
>
So this is not selinux on the hypervisor. I wonder if it can be selinux on
the gluster side?
> I have a vague feeling that the issue is related to gluster v6.5 to 6.6
> upgrade which I several days before... So if any logs are needed (or debug
> enabled), just mention.
>
If this is the last change, and it worked before, most likely.
Nir
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NE...
11:10 p.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi Krutika,
thanks for your assistance.Let me summarize some info about the volume:
Volume Name: data_fastType: ReplicateVolume ID:
378804bf-2975-44d8-84c2-b541aa87f9efStatus: StartedSnapshot Count: 0Number of Bricks: 1 x
(2 + 1) = 3Transport-type: tcpBricks:Brick1:
gluster1:/gluster_bricks/data_fast/data_fastBrick2:
gluster2:/gluster_bricks/data_fast/data_fastBrick3:
ovirt3:/gluster_bricks/data_fast/data_fast (arbiter)Options
Reconfigured:performance.client-io-threads: onnfs.disable: ontransport.address-family:
inetperformance.quick-read: offperformance.read-ahead: offperformance.io-cache:
offperformance.low-prio-threads: 32network.remote-dio: oncluster.eager-lock:
enablecluster.quorum-type: autocluster.server-quorum-type:
servercluster.data-self-heal-algorithm: fullcluster.locking-scheme:
granularcluster.shd-max-threads: 8cluster.shd-wait-qlength: 10000features.shard:
onuser.cifs: offcluster.choose-local: onclient.event-threads: 4server.event-threads:
4storage.owner-uid: 36storage.owner-gid: 36performance.strict-o-direct:
onnetwork.ping-timeout: 30cluster.granular-entry-heal:
enablecluster.enable-shared-storage: enable
[root@ovirt1 ~]# gluster volume get engine all | grep shardfeatures.shard
onfeatures.shard-block-size 64MBfeatures.shard-lru-limit
16384features.shard-deletion-rate 100
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi All,
another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission
denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c
fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission
denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4
fd=0x7fc63c01a058 (Permission denied)
Anyone got an idea why is it happening?I checked user/group and selinux permissions - all
OK
Can you share the commands (and output) used to check this?I first
thought that the file is cached in memory and that's why vdsm user can read the file ,
but the following shows opposite:
[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# lltotal 562145-rw-rw----. 1 vdsm kvm 5368709120 Nov
12 23:29 5b1d3113-5cca-4582-9029-634b16338a2f-rw-rw----. 1 vdsm kvm 1048576 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.lease-rw-r--r--. 1 vdsm kvm 313 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.meta[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]#
pwd/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always use direct
I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progressdd: error
reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied...
16+0 records in16+0 records out67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>Seems like it could read upto ~67MB successfully before it encountered
'Permission denied' errors. Assuming a shard-block-size >of 64MB, looks like
all the shards under /.shard could not be accessed.>Could you share the following
pieces of information:>1. brick logs of data_fastAttached in data_fast-brick-logs.tgz
2. ls -la of .shard relative to the bricks (NOT the mount) on all the
bricks of data_fast
Not sure if I understood you correctly, so I ran "ls -lad
/gluster_bricks/data_fast/data_fast/.shard". If it's not what you wanted to see -
just correct me.I have run multiple "find" commands with "-exec chown
vdsm:kvm {} \;" , just to be sure that this is not happening.>3. and ls -la of all
shards under .shard of data_fast (perhaps a handful of them have root permission assigned
somehow which is causing access to be denied? Perhaps while resolving pending heals
manually? )
All shards seem to be owned by "vdsm:kvm" with 660.
-Krutika
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct bs=8M
status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for this test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5356126208 bytes
(5.4 GB) copied, 12.061393 s, 444 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.0876 s, 444 MB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]#
sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M
status=progress3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s1280+0 records
in1280+0 records out5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches [root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5171576832 bytes
(5.2 GB) copied, 12.071837 s, 428 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do that.
Smells like issue on gluster side.
I would try this on the hypervisor to check what vdsm/qemu see:
$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_pathI'm
attaching the output of the find I run, but this one should be enough:[root@ovirt1 ~]#
find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/ -not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and selinux labelof the
entire tree will be more useful.
Also, to make sure we don't have selinux issue on the hypervisor,
you can change>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive when I tried
again.I'm 99.99% sure it's not selinux.
And then try again. If this was selinux issue the permission denied
issue will disappear.>If this is the case please provide the output of:
>
ausearh -m AVC -ts today
If the issue still exists, we eliminated selinux, and you can enable
it again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today<no matches>[root@ovirt2 ~]# ausearch -m
AVC -ts today<no matches>[root@ovirt1 ~]# ausearch -m AVC -ts today<no
matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on the gluster
side?
I have a vague feeling that the issue is related to gluster v6.5 to 6.6 upgrade which I
several days before... So if any logs are needed (or debug enabled), just mention.
If this is the last change, and it worked before, most likely.
Nir_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NE...
11:45 p.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi Krutika,
I have enabled TRACE log level for the volume data_fast,
but the issue is not much clear:FUSE reports:
[2019-11-25 21:31:53.478130] I [MSGID: 133022] [shard.c:3674:shard_delete_shards]
0-data_fast-shard: Deleted shards of gfid=6d9ed2e5-d4f2-4749-839b-2f13a68ed472 from
backend[2019-11-25 21:32:43.564694] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-25 21:32:43.565653] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-25 21:32:43.565689] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-25 21:32:43.565770] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-25 21:32:43.565858] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 279: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fbf40005ea8 (Permission denied)
While the BRICK logs on ovirt1/gluster1 report:2019-11-25 21:32:43.564177] D [MSGID: 0]
[io-threads.c:376:iot_schedule] 0-data_fast-io-threads: LOOKUP scheduled as fast priority
fop[2019-11-25 21:32:43.564194] T [MSGID: 0] [defaults.c:2008:default_lookup_resume]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-io-threads to
data_fast-upcall[2019-11-25 21:32:43.564206] T [MSGID: 0] [upcall.c:790:up_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-upcall to
data_fast-leases[2019-11-25 21:32:43.564215] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-leases to
data_fast-read-only[2019-11-25 21:32:43.564222] T [MSGID: 0]
[defaults.c:2766:default_lookup] 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding
from data_fast-read-only to data_fast-worm[2019-11-25 21:32:43.564230] T [MSGID: 0]
[defaults.c:2766:default_lookup] 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding
from data_fast-worm to data_fast-locks[2019-11-25 21:32:43.564241] T [MSGID: 0]
[posix.c:2897:pl_lookup] 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from
data_fast-locks to data_fast-access-control[2019-11-25 21:32:43.564254] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client:
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) [Permission
denied][2019-11-25 21:32:43.564268] D [MSGID: 0] [posix-acl.c:1057:posix_acl_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-access-control returned -1 error:
Permission denied [Permission denied][2019-11-25 21:32:43.564279] D [MSGID: 0]
[posix.c:2888:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-locks
returned -1 error: Permission denied [Permission denied][2019-11-25 21:32:43.564289] D
[MSGID: 0] [upcall.c:769:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fc02c00bbf8,
data_fast-upcall returned -1 error: Permission denied [Permission denied][2019-11-25
21:32:43.564302] D [MSGID: 0] [defaults.c:1349:default_lookup_cbk] 0-stack-trace:
stack-address: 0x7fc02c00bbf8, data_fast-io-threads returned -1 error: Permission denied
[Permission denied][2019-11-25 21:32:43.564313] T [marker.c:2918:marker_lookup_cbk]
0-data_fast-marker: lookup failed with Permission denied[2019-11-25 21:32:43.564320] D
[MSGID: 0] [marker.c:2955:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fc02c00bbf8,
data_fast-marker returned -1 error: Permission denied [Permission denied][2019-11-25
21:32:43.564334] D [MSGID: 0] [index.c:2070:index_lookup_cbk] 0-stack-trace:
stack-address: 0x7fc02c00bbf8, data_fast-index returned -1 error: Permission denied
[Permission denied]
В понеделник, 25 ноември 2019 г., 23:10:41 ч. Гринуич+2, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
thanks for your assistance.Let me summarize some info about the volume:
Volume Name: data_fastType: ReplicateVolume ID:
378804bf-2975-44d8-84c2-b541aa87f9efStatus: StartedSnapshot Count: 0Number of Bricks: 1 x
(2 + 1) = 3Transport-type: tcpBricks:Brick1:
gluster1:/gluster_bricks/data_fast/data_fastBrick2:
gluster2:/gluster_bricks/data_fast/data_fastBrick3:
ovirt3:/gluster_bricks/data_fast/data_fast (arbiter)Options
Reconfigured:performance.client-io-threads: onnfs.disable: ontransport.address-family:
inetperformance.quick-read: offperformance.read-ahead: offperformance.io-cache:
offperformance.low-prio-threads: 32network.remote-dio: oncluster.eager-lock:
enablecluster.quorum-type: autocluster.server-quorum-type:
servercluster.data-self-heal-algorithm: fullcluster.locking-scheme:
granularcluster.shd-max-threads: 8cluster.shd-wait-qlength: 10000features.shard:
onuser.cifs: offcluster.choose-local: onclient.event-threads: 4server.event-threads:
4storage.owner-uid: 36storage.owner-gid: 36performance.strict-o-direct:
onnetwork.ping-timeout: 30cluster.granular-entry-heal:
enablecluster.enable-shared-storage: enable
[root@ovirt1 ~]# gluster volume get engine all | grep shardfeatures.shard
onfeatures.shard-block-size 64MBfeatures.shard-lru-limit
16384features.shard-deletion-rate 100
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi All,
another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission
denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c
fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission
denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4
fd=0x7fc63c01a058 (Permission denied)
Anyone got an idea why is it happening?I checked user/group and selinux permissions - all
OK
Can you share the commands (and output) used to check this?I first
thought that the file is cached in memory and that's why vdsm user can read the file ,
but the following shows opposite:
[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# lltotal 562145-rw-rw----. 1 vdsm kvm 5368709120 Nov
12 23:29 5b1d3113-5cca-4582-9029-634b16338a2f-rw-rw----. 1 vdsm kvm 1048576 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.lease-rw-r--r--. 1 vdsm kvm 313 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.meta[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]#
pwd/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always use direct
I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progressdd: error
reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied...
16+0 records in16+0 records out67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>Seems like it could read upto ~67MB successfully before it encountered
'Permission denied' errors. Assuming a shard-block-size >of 64MB, looks like
all the shards under /.shard could not be accessed.>Could you share the following
pieces of information:>1. brick logs of data_fastAttached in data_fast-brick-logs.tgz
2. ls -la of .shard relative to the bricks (NOT the mount) on all the
bricks of data_fast
Not sure if I understood you correctly, so I ran "ls -lad
/gluster_bricks/data_fast/data_fast/.shard". If it's not what you wanted to see -
just correct me.I have run multiple "find" commands with "-exec chown
vdsm:kvm {} \;" , just to be sure that this is not happening.>3. and ls -la of all
shards under .shard of data_fast (perhaps a handful of them have root permission assigned
somehow which is causing access to be denied? Perhaps while resolving pending heals
manually? )
All shards seem to be owned by "vdsm:kvm" with 660.
-Krutika
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct bs=8M
status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for this test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5356126208 bytes
(5.4 GB) copied, 12.061393 s, 444 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.0876 s, 444 MB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]#
sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M
status=progress3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s1280+0 records
in1280+0 records out5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches [root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5171576832 bytes
(5.2 GB) copied, 12.071837 s, 428 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do that.
Smells like issue on gluster side.
I would try this on the hypervisor to check what vdsm/qemu see:
$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_pathI'm
attaching the output of the find I run, but this one should be enough:[root@ovirt1 ~]#
find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/ -not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and selinux labelof the
entire tree will be more useful.
Also, to make sure we don't have selinux issue on the hypervisor,
you can change>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive when I tried
again.I'm 99.99% sure it's not selinux.
And then try again. If this was selinux issue the permission denied
issue will disappear.>If this is the case please provide the output of:
>
ausearh -m AVC -ts today
If the issue still exists, we eliminated selinux, and you can enable
it again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today<no matches>[root@ovirt2 ~]# ausearch -m
AVC -ts today<no matches>[root@ovirt1 ~]# ausearch -m AVC -ts today<no
matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on the gluster
side?
I have a vague feeling that the issue is related to gluster v6.5 to 6.6 upgrade which I
several days before... So if any logs are needed (or debug enabled), just mention.
If this is the last change, and it worked before, most likely.
Nir_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NE...
10:41 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi Nir,All,
it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached screenshot of
oVirt running on v7 gluster).It seems strange that both my serious issues with oVirt are
related to gluster issue (1st gluster v3 to v5 migration and now this one).
I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all nodes.Now both
Engine and all my VMs are back online - so if you hit issues with 6.6 , you should give a
try to 7.0 (and even 7.1 is coming soon) before deciding to wipe everything.
@Krutika,
I guess you will ask for the logs, so let's switch to gluster-users about this one ?
Best Regards,Strahil Nikolov
В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
I have enabled TRACE log level for the volume data_fast,
but the issue is not much clear:FUSE reports:
[2019-11-25 21:31:53.478130] I [MSGID: 133022] [shard.c:3674:shard_delete_shards]
0-data_fast-shard: Deleted shards of gfid=6d9ed2e5-d4f2-4749-839b-2f13a68ed472 from
backend[2019-11-25 21:32:43.564694] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-25 21:32:43.565653] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-25 21:32:43.565689] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-25 21:32:43.565770] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-25 21:32:43.565858] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 279: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fbf40005ea8 (Permission denied)
While the BRICK logs on ovirt1/gluster1 report:2019-11-25 21:32:43.564177] D [MSGID: 0]
[io-threads.c:376:iot_schedule] 0-data_fast-io-threads: LOOKUP scheduled as fast priority
fop[2019-11-25 21:32:43.564194] T [MSGID: 0] [defaults.c:2008:default_lookup_resume]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-io-threads to
data_fast-upcall[2019-11-25 21:32:43.564206] T [MSGID: 0] [upcall.c:790:up_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-upcall to
data_fast-leases[2019-11-25 21:32:43.564215] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-leases to
data_fast-read-only[2019-11-25 21:32:43.564222] T [MSGID: 0]
[defaults.c:2766:default_lookup] 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding
from data_fast-read-only to data_fast-worm[2019-11-25 21:32:43.564230] T [MSGID: 0]
[defaults.c:2766:default_lookup] 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding
from data_fast-worm to data_fast-locks[2019-11-25 21:32:43.564241] T [MSGID: 0]
[posix.c:2897:pl_lookup] 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from
data_fast-locks to data_fast-access-control[2019-11-25 21:32:43.564254] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client:
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) [Permission
denied][2019-11-25 21:32:43.564268] D [MSGID: 0] [posix-acl.c:1057:posix_acl_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-access-control returned -1 error:
Permission denied [Permission denied][2019-11-25 21:32:43.564279] D [MSGID: 0]
[posix.c:2888:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-locks
returned -1 error: Permission denied [Permission denied][2019-11-25 21:32:43.564289] D
[MSGID: 0] [upcall.c:769:up_lookup_cbk] 0-stack-trace: stack-address: 0x7fc02c00bbf8,
data_fast-upcall returned -1 error: Permission denied [Permission denied][2019-11-25
21:32:43.564302] D [MSGID: 0] [defaults.c:1349:default_lookup_cbk] 0-stack-trace:
stack-address: 0x7fc02c00bbf8, data_fast-io-threads returned -1 error: Permission denied
[Permission denied][2019-11-25 21:32:43.564313] T [marker.c:2918:marker_lookup_cbk]
0-data_fast-marker: lookup failed with Permission denied[2019-11-25 21:32:43.564320] D
[MSGID: 0] [marker.c:2955:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7fc02c00bbf8,
data_fast-marker returned -1 error: Permission denied [Permission denied][2019-11-25
21:32:43.564334] D [MSGID: 0] [index.c:2070:index_lookup_cbk] 0-stack-trace:
stack-address: 0x7fc02c00bbf8, data_fast-index returned -1 error: Permission denied
[Permission denied]
В понеделник, 25 ноември 2019 г., 23:10:41 ч. Гринуич+2, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
thanks for your assistance.Let me summarize some info about the volume:
Volume Name: data_fastType: ReplicateVolume ID:
378804bf-2975-44d8-84c2-b541aa87f9efStatus: StartedSnapshot Count: 0Number of Bricks: 1 x
(2 + 1) = 3Transport-type: tcpBricks:Brick1:
gluster1:/gluster_bricks/data_fast/data_fastBrick2:
gluster2:/gluster_bricks/data_fast/data_fastBrick3:
ovirt3:/gluster_bricks/data_fast/data_fast (arbiter)Options
Reconfigured:performance.client-io-threads: onnfs.disable: ontransport.address-family:
inetperformance.quick-read: offperformance.read-ahead: offperformance.io-cache:
offperformance.low-prio-threads: 32network.remote-dio: oncluster.eager-lock:
enablecluster.quorum-type: autocluster.server-quorum-type:
servercluster.data-self-heal-algorithm: fullcluster.locking-scheme:
granularcluster.shd-max-threads: 8cluster.shd-wait-qlength: 10000features.shard:
onuser.cifs: offcluster.choose-local: onclient.event-threads: 4server.event-threads:
4storage.owner-uid: 36storage.owner-gid: 36performance.strict-o-direct:
onnetwork.ping-timeout: 30cluster.granular-entry-heal:
enablecluster.enable-shared-storage: enable
[root@ovirt1 ~]# gluster volume get engine all | grep shardfeatures.shard
onfeatures.shard-block-size 64MBfeatures.shard-lru-limit
16384features.shard-deletion-rate 100
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi All,
another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission
denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c
fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission
denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4
fd=0x7fc63c01a058 (Permission denied)
Anyone got an idea why is it happening?I checked user/group and selinux permissions - all
OK
Can you share the commands (and output) used to check this?I first
thought that the file is cached in memory and that's why vdsm user can read the file ,
but the following shows opposite:
[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# lltotal 562145-rw-rw----. 1 vdsm kvm 5368709120 Nov
12 23:29 5b1d3113-5cca-4582-9029-634b16338a2f-rw-rw----. 1 vdsm kvm 1048576 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.lease-rw-r--r--. 1 vdsm kvm 313 Nov 11
14:11 5b1d3113-5cca-4582-9029-634b16338a2f.meta[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]#
pwd/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always use direct
I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progressdd: error
reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied...
16+0 records in16+0 records out67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>Seems like it could read upto ~67MB successfully before it encountered
'Permission denied' errors. Assuming a shard-block-size >of 64MB, looks like
all the shards under /.shard could not be accessed.>Could you share the following
pieces of information:>1. brick logs of data_fastAttached in data_fast-brick-logs.tgz
2. ls -la of .shard relative to the bricks (NOT the mount) on all the
bricks of data_fast
Not sure if I understood you correctly, so I ran "ls -lad
/gluster_bricks/data_fast/data_fast/.shard". If it's not what you wanted to see -
just correct me.I have run multiple "find" commands with "-exec chown
vdsm:kvm {} \;" , just to be sure that this is not happening.>3. and ls -la of all
shards under .shard of data_fast (perhaps a handful of them have root permission assigned
somehow which is causing access to be denied? Perhaps while resolving pending heals
manually? )
All shards seem to be owned by "vdsm:kvm" with 660.
-Krutika
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct bs=8M
status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for this test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5356126208 bytes
(5.4 GB) copied, 12.061393 s, 444 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.0876 s, 444 MB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]#
sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M
status=progress3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s1280+0 records
in1280+0 records out5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s[root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches [root@ovirt1
94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5171576832 bytes
(5.2 GB) copied, 12.071837 s, 428 MB/s1280+0 records in1280+0 records out5368709120 bytes
(5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do that.
Smells like issue on gluster side.
I would try this on the hypervisor to check what vdsm/qemu see:
$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_pathI'm
attaching the output of the find I run, but this one should be enough:[root@ovirt1 ~]#
find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/ -not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and selinux labelof the
entire tree will be more useful.
Also, to make sure we don't have selinux issue on the hypervisor,
you can change>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive when I tried
again.I'm 99.99% sure it's not selinux.
And then try again. If this was selinux issue the permission denied
issue will disappear.>If this is the case please provide the output of:
>
ausearh -m AVC -ts today
If the issue still exists, we eliminated selinux, and you can enable
it again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today<no matches>[root@ovirt2 ~]# ausearch -m
AVC -ts today<no matches>[root@ovirt1 ~]# ausearch -m AVC -ts today<no
matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on the gluster
side?
I have a vague feeling that the issue is related to gluster v6.5 to 6.6 upgrade which I
several days before... So if any logs are needed (or debug enabled), just mention.
If this is the last change, and it worked before, most likely.
Nir_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NE...
8:18 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Sorry about the late response.
I looked at the logs. These errors are originating from posix-acl
translator -
*[2019-11-17 07:55:47.090065] E [MSGID: 115050]
[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496:
LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6
(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6),
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
error-xlator: data_fast-access-control [Permission denied][2019-11-17
07:55:47.090174] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:36,gid:36,perm:1,ngrps:3),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050]
[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497:
LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7
(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7),
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
error-xlator: data_fast-access-control [Permission denied][2019-11-17
07:55:47.090299] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:36,gid:36,perm:1,ngrps:3),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]*
Jiffin/Raghavendra Talur,
Can you help?
-Krutika
On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi Nir,All,
it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached
screenshot of oVirt running on v7 gluster).
It seems strange that both my serious issues with oVirt are related to
gluster issue (1st gluster v3 to v5 migration and now this one).
I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all
nodes.
Now both Engine and all my VMs are back online - so if you hit issues with
6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before
deciding to wipe everything.
@Krutika,
I guess you will ask for the logs, so let's switch to gluster-users about
this one ?
Best Regards,
Strahil Nikolov
В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov <
hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
I have enabled TRACE log level for the volume data_fast,
but the issue is not much clear:
FUSE reports:
[2019-11-25 21:31:53.478130] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=6d9ed2e5-d4f2-4749-839b-2f1
3a68ed472 from backend
[2019-11-25 21:32:43.564694] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565653] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565689] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565770] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-25 21:32:43.565858] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 279: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fbf40005ea8 (Permission denied)
While the BRICK logs on ovirt1/gluster1 report:
2019-11-25 21:32:43.564177] D [MSGID: 0] [io-threads.c:376:iot_schedule]
0-data_fast-io-threads: LOOKUP scheduled as fast priority fop
[2019-11-25 21:32:43.564194] T [MSGID: 0]
[defaults.c:2008:default_lookup_resume] 0-stack-trace: stack-address:
0x7fc02c00bbf8, winding from data_fast-io-threads to data_fast-upcall
[2019-11-25 21:32:43.564206] T [MSGID: 0] [upcall.c:790:up_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-upcall
to data_fast-leases
[2019-11-25 21:32:43.564215] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-leases
to data_fast-read-only
[2019-11-25 21:32:43.564222] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from
data_fast-read-only to data_fast-worm
[2019-11-25 21:32:43.564230] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-worm
to data_fast-locks
[2019-11-25 21:32:43.564241] T [MSGID: 0] [posix.c:2897:pl_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-locks
to data_fast-access-control
[2019-11-25 21:32:43.564254] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]
[2019-11-25 21:32:43.564268] D [MSGID: 0]
[posix-acl.c:1057:posix_acl_lookup] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-access-control returned -1 error: Permission
denied [Permission denied]
[2019-11-25 21:32:43.564279] D [MSGID: 0] [posix.c:2888:pl_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-locks returned -1
error: Permission denied [Permission denied]
[2019-11-25 21:32:43.564289] D [MSGID: 0] [upcall.c:769:up_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-upcall returned -1
error: Permission denied [Permission denied]
[2019-11-25 21:32:43.564302] D [MSGID: 0]
[defaults.c:1349:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-io-threads returned -1 error: Permission denied
[Permission denied]
[2019-11-25 21:32:43.564313] T [marker.c:2918:marker_lookup_cbk]
0-data_fast-marker: lookup failed with Permission denied
[2019-11-25 21:32:43.564320] D [MSGID: 0]
[marker.c:2955:marker_lookup_cbk] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-marker returned -1 error: Permission denied
[Permission denied]
[2019-11-25 21:32:43.564334] D [MSGID: 0] [index.c:2070:index_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-index returned -1
error: Permission denied [Permission denied]
В понеделник, 25 ноември 2019 г., 23:10:41 ч. Гринуич+2, Strahil Nikolov <
hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
thanks for your assistance.
Let me summarize some info about the volume:
Volume Name: data_fast
Type: Replicate
Volume ID: 378804bf-2975-44d8-84c2-b541aa87f9ef
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/gluster_bricks/data_fast/data_fast
Brick2: gluster2:/gluster_bricks/data_fast/data_fast
Brick3: ovirt3:/gluster_bricks/data_fast/data_fast (arbiter)
Options Reconfigured:
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: on
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
cluster.choose-local: on
client.event-threads: 4
server.event-threads: 4
storage.owner-uid: 36
storage.owner-gid: 36
performance.strict-o-direct: on
network.ping-timeout: 30
cluster.granular-entry-heal: enable
cluster.enable-shared-storage: enable
[root@ovirt1 ~]# gluster volume get engine all | grep shard
features.shard on
features.shard-block-size 64MB
features.shard-lru-limit 16384
features.shard-deletion-rate 100
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi All,
another clue in the logs :
[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536798] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536959] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.537007] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1
gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission
denied)
[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
[2019-11-21 00:30:13.132756] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.132824] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133217] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133238] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c
[Permission denied]
[2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1
gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission
denied)
[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489520] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489669] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489717] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4
[Permission denied]
[2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1
gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission
denied)
Anyone got an idea why is it happening?
I checked user/group and selinux permissions - all OK
>Can you share the commands (and output) used to check this?
I first thought that the file is cached in memory and that's why vdsm user
can read the file , but the following shows opposite:
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# ll
total 562145
-rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29
5b1d3113-5cca-4582-9029-634b16338a2f
-rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.lease
-rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.meta
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always
use direct I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
dd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied...
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>Seems like it could read upto ~67MB successfully before it encountered
'Permission denied' errors. Assuming a shard-block-size >of 64MB, looks
like all the shards under /.shard could not be accessed.
>Could you share the following pieces of information:
>1. brick logs of data_fast
Attached in data_fast-brick-logs.tgz
>2. ls -la of .shard relative to the bricks (NOT the mount) on all the
bricks of data_fast
Not sure if I understood you correctly, so I ran "ls -lad
/gluster_bricks/data_fast/data_fast/.shard". If it's not what you wanted to
see - just correct me.
I have run multiple "find" commands with "-exec chown vdsm:kvm {} \;"
,
just to be sure that this is not happening.
>3. and ls -la of all shards under .shard of data_fast (perhaps a handful
of them have root permission assigned somehow which is causing access to be
denied? Perhaps while resolving pending heals manually? )
All shards seem to be owned by "vdsm:kvm" with 660.
-Krutika
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct
bs=8M status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for
this test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do
that.
Smells like issue on gluster side.
>I would try this on the hypervisor to check what vdsm/qemu see:
>$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path
I'm attaching the output of the find I run, but this one should be enough:
[root@ovirt1 ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/
-not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and
selinux label
of the entire tree will be more useful.
>Also, to make sure we don't have selinux issue on the hypervisor, you can
change
>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive
when I tried again.I'm 99.99% sure it's not selinux.
>And then try again. If this was selinux issue the permission denied issue
will disappear.
>If this is the case please provide the output of:
> ausearh -m AVC -ts today
>If the issue still exists, we eliminated selinux, and you can enable it
again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt2 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt1 ~]# ausearch -m AVC -ts today
<no matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on
the gluster side?
I have a vague feeling that the issue is related to gluster v6.5 to 6.6
upgrade which I several days before... So if any logs are needed (or debug
enabled), just mention.
If this is the last change, and it worked before, most likely.
Nir
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NE...
11:32 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi Krutika,
Apparently, in context acl info got corrupted see brick logs
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]
which resulted in the situation. There was one bug similar was reported
https://bugzilla.redhat.com/show_bug.cgi?id=1668286 and
it got fixed in 6.6 release IMO https://review.gluster.org/#/c/glusterfs/+/23233/. But
here he mentioned he saw the issue when
he upgraded from 6.5 to 6.6
One way to workaround is to perform a dummy setfacl(preferably using root) on the
corrupted files which will forcefully fetch the acl
info again from backend and update the context. Another approach to restart brick
process(kill and vol start force)
Regards,
Jiffin
----- Original Message -----
From: "Krutika Dhananjay" <kdhananj(a)redhat.com>
To: "Strahil Nikolov" <hunter86_bg(a)yahoo.com>, "Jiffin Thottan"
<jthottan(a)redhat.com>, "raghavendra talur" <rtalur(a)redhat.com>
Cc: "Nir Soffer" <nsoffer(a)redhat.com>, "Rafi Kavungal Chundattu
Parambil" <rkavunga(a)redhat.com>, "users" <users(a)ovirt.org>,
"gluster-user" <gluster-users(a)gluster.org>
Sent: Monday, December 2, 2019 11:48:22 AM
Subject: Re: [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available
for testing
Sorry about the late response.
I looked at the logs. These errors are originating from posix-acl
translator -
*[2019-11-17 07:55:47.090065] E [MSGID: 115050]
[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496:
LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6
(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6),
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
error-xlator: data_fast-access-control [Permission denied][2019-11-17
07:55:47.090174] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:36,gid:36,perm:1,ngrps:3),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050]
[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497:
LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7
(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7),
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
error-xlator: data_fast-access-control [Permission denied][2019-11-17
07:55:47.090299] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:36,gid:36,perm:1,ngrps:3),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]*
Jiffin/Raghavendra Talur,
Can you help?
-Krutika
On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi Nir,All,
it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached
screenshot of oVirt running on v7 gluster).
It seems strange that both my serious issues with oVirt are related to
gluster issue (1st gluster v3 to v5 migration and now this one).
I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all
nodes.
Now both Engine and all my VMs are back online - so if you hit issues with
6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before
deciding to wipe everything.
@Krutika,
I guess you will ask for the logs, so let's switch to gluster-users about
this one ?
Best Regards,
Strahil Nikolov
В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov <
hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
I have enabled TRACE log level for the volume data_fast,
but the issue is not much clear:
FUSE reports:
[2019-11-25 21:31:53.478130] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=6d9ed2e5-d4f2-4749-839b-2f1
3a68ed472 from backend
[2019-11-25 21:32:43.564694] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565653] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565689] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565770] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-25 21:32:43.565858] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 279: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fbf40005ea8 (Permission denied)
While the BRICK logs on ovirt1/gluster1 report:
2019-11-25 21:32:43.564177] D [MSGID: 0] [io-threads.c:376:iot_schedule]
0-data_fast-io-threads: LOOKUP scheduled as fast priority fop
[2019-11-25 21:32:43.564194] T [MSGID: 0]
[defaults.c:2008:default_lookup_resume] 0-stack-trace: stack-address:
0x7fc02c00bbf8, winding from data_fast-io-threads to data_fast-upcall
[2019-11-25 21:32:43.564206] T [MSGID: 0] [upcall.c:790:up_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-upcall
to data_fast-leases
[2019-11-25 21:32:43.564215] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-leases
to data_fast-read-only
[2019-11-25 21:32:43.564222] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from
data_fast-read-only to data_fast-worm
[2019-11-25 21:32:43.564230] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-worm
to data_fast-locks
[2019-11-25 21:32:43.564241] T [MSGID: 0] [posix.c:2897:pl_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-locks
to data_fast-access-control
[2019-11-25 21:32:43.564254] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]
[2019-11-25 21:32:43.564268] D [MSGID: 0]
[posix-acl.c:1057:posix_acl_lookup] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-access-control returned -1 error: Permission
denied [Permission denied]
[2019-11-25 21:32:43.564279] D [MSGID: 0] [posix.c:2888:pl_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-locks returned -1
error: Permission denied [Permission denied]
[2019-11-25 21:32:43.564289] D [MSGID: 0] [upcall.c:769:up_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-upcall returned -1
error: Permission denied [Permission denied]
[2019-11-25 21:32:43.564302] D [MSGID: 0]
[defaults.c:1349:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-io-threads returned -1 error: Permission denied
[Permission denied]
[2019-11-25 21:32:43.564313] T [marker.c:2918:marker_lookup_cbk]
0-data_fast-marker: lookup failed with Permission denied
[2019-11-25 21:32:43.564320] D [MSGID: 0]
[marker.c:2955:marker_lookup_cbk] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-marker returned -1 error: Permission denied
[Permission denied]
[2019-11-25 21:32:43.564334] D [MSGID: 0] [index.c:2070:index_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-index returned -1
error: Permission denied [Permission denied]
В понеделник, 25 ноември 2019 г., 23:10:41 ч. Гринуич+2, Strahil Nikolov <
hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
thanks for your assistance.
Let me summarize some info about the volume:
Volume Name: data_fast
Type: Replicate
Volume ID: 378804bf-2975-44d8-84c2-b541aa87f9ef
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/gluster_bricks/data_fast/data_fast
Brick2: gluster2:/gluster_bricks/data_fast/data_fast
Brick3: ovirt3:/gluster_bricks/data_fast/data_fast (arbiter)
Options Reconfigured:
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: on
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
cluster.choose-local: on
client.event-threads: 4
server.event-threads: 4
storage.owner-uid: 36
storage.owner-gid: 36
performance.strict-o-direct: on
network.ping-timeout: 30
cluster.granular-entry-heal: enable
cluster.enable-shared-storage: enable
[root@ovirt1 ~]# gluster volume get engine all | grep shard
features.shard on
features.shard-block-size 64MB
features.shard-lru-limit 16384
features.shard-deletion-rate 100
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi All,
another clue in the logs :
[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536798] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536959] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.537007] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1
gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission
denied)
[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
[2019-11-21 00:30:13.132756] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.132824] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133217] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133238] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c
[Permission denied]
[2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1
gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission
denied)
[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489520] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489669] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489717] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4
[Permission denied]
[2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1
gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission
denied)
Anyone got an idea why is it happening?
I checked user/group and selinux permissions - all OK
>Can you share the commands (and output) used to check this?
I first thought that the file is cached in memory and that's why vdsm user
can read the file , but the following shows opposite:
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# ll
total 562145
-rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29
5b1d3113-5cca-4582-9029-634b16338a2f
-rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.lease
-rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.meta
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always
use direct I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
dd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied...
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>Seems like it could read upto ~67MB successfully before it encountered
'Permission denied' errors. Assuming a shard-block-size >of 64MB, looks
like all the shards under /.shard could not be accessed.
>Could you share the following pieces of information:
>1. brick logs of data_fast
Attached in data_fast-brick-logs.tgz
>2. ls -la of .shard relative to the bricks (NOT the mount) on all the
bricks of data_fast
Not sure if I understood you correctly, so I ran "ls -lad
/gluster_bricks/data_fast/data_fast/.shard". If it's not what you wanted to
see - just correct me.
I have run multiple "find" commands with "-exec chown vdsm:kvm {} \;"
,
just to be sure that this is not happening.
>3. and ls -la of all shards under .shard of data_fast (perhaps a handful
of them have root permission assigned somehow which is causing access to be
denied? Perhaps while resolving pending heals manually? )
All shards seem to be owned by "vdsm:kvm" with 660.
-Krutika
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct
bs=8M status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for
this test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do
that.
Smells like issue on gluster side.
>I would try this on the hypervisor to check what vdsm/qemu see:
>$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path
I'm attaching the output of the find I run, but this one should be enough:
[root@ovirt1 ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/
-not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and
selinux label
of the entire tree will be more useful.
>Also, to make sure we don't have selinux issue on the hypervisor, you can
change
>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive
when I tried again.I'm 99.99% sure it's not selinux.
>And then try again. If this was selinux issue the permission denied issue
will disappear.
>If this is the case please provide the output of:
> ausearh -m AVC -ts today
>If the issue still exists, we eliminated selinux, and you can enable it
again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt2 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt1 ~]# ausearch -m AVC -ts today
<no matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on
the gluster side?
I have a vague feeling that the issue is related to gluster v6.5 to 6.6
upgrade which I several days before... So if any logs are needed (or debug
enabled), just mention.
If this is the last change, and it worked before, most likely.
Nir
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NE...
1:57 a.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
Hi Jiffin,
I'm observing the same behaviour like last time .This time after upgrading from
gluster v 7.0 to 7.2 as follows:
[2020-01-30 23:42:26.039822] T [MSGID: 0] [posix.c:2994:pl_lookup] 0-stack-trace:
stack-address: 0x7f7b50003958, winding from data_fast-locks to
data_fast-access-control [2020-01-30 23:42:26.039837] I [MSGID: 139001]
[posix-acl.c:262:posix_acl_log_permit_denied] 0-data_fast-access-control: client:
CTX_ID:868c1fba-1f38-4e91-a664-6e49fddb3f27-GRAPH_ID:0-PID:9525-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-1-RECON_NO:-1,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) [Permission
denied][2020-01-30 23:42:26.039857] D [MSGID: 0] [posix-acl.c:1055:posix_acl_lookup]
0-stack-trace: stack-address: 0x7f7b50003958, data_fast-access-control returned -1 error:
Permission denied [Permission denied][2020-01-30 23:42:26.039868] D [MSGID: 0]
[posix.c:2985:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958, data_fast-locks
returned -1 error: Permission denied [Permission denied][2020-01-30 23:42:26.039879] D
[MSGID: 0] [upcall.c:756:up_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958,
data_fast-upcall returned -1 error: Permission denied [Permission denied][2020-01-30
23:42:26.039889] D [MSGID: 0] [defaults.c:1349:default_lookup_cbk] 0-stack-trace:
stack-address: 0x7f7b50003958, data_fast-io-threads returned -1 error: Permission denied
[Permission denied][2020-01-30 23:42:26.039901] T [marker.c:2907:marker_lookup_cbk]
0-data_fast-marker: lookup failed with Permission denied[2020-01-30 23:42:26.039908] D
[MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958,
data_fast-marker returned -1 error: Permission denied [Permission denied][2020-01-30
23:42:26.039922] D [MSGID: 0] [index.c:2070:index_lookup_cbk] 0-stack-trace:
stack-address: 0x7f7b50003958, data_fast-index returned -1 error: Permission denied
[Permission denied][2020-01-30 23:42:26.039933] D [MSGID: 0]
[io-stats.c:2228:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958,
/gluster_bricks/data_fast/data_fast returned -1 error: Permission denied [Permission
denied][2020-01-30 23:42:26.039949] E [MSGID: 115050]
[server-rpc-fops_v2.c:157:server4_lookup_cbk] 0-data_fast-server: 16268: LOOKUP
/.shard/f476698a-d8d2-4ab2-b9c4-4c276c2eef43.79
(be318638-e8a0-4c6d-977d-7a937aa84806/f476698a-d8d2-4ab2-b9c4-4c276c2eef43.79), client:
CTX_ID:868c1fba-1f38-4e91-a664-6e49fddb3f27-GRAPH_ID:0-PID:9525-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-1-RECON_NO:-1,
error-xlator: data_fast-access-control [Permission denied][2020-01-30 23:42:26.039964] T
[rpcsvc.c:1529:rpcsvc_submit_generic] 0-rpc-service: Tx message: 344[2020-01-30
23:42:26.039971] T [rpcsvc.c:1065:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen
368, payload: 344, rpc hdr: 24[2020-01-30 23:42:26.040007] T
[rpcsvc.c:1581:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID:
0x24c1, Program: GlusterFS 4.x v1, ProgVers: 400, Proc: 27) to rpc-transport
(tcp.data_fast-server)
Are you sure it's fixed?If yes, then this looks like a regresion.
By the way, I'm not using ACL - how can I permanently disable it ,so even if ACL gets
corrupted -> my oVirt will survive ?
Thanks in advance for your answer.
Best Regards,Strahil Nikolov
В сряда, 4 декември 2019 г., 13:28:39 ч. Гринуич+2, Jiffin Thottan
<jthottan(a)redhat.com> написа:
Hi Krutika,
Apparently, in context acl info got corrupted see brick logs
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]
which resulted in the situation. There was one bug similar was reported
https://bugzilla.redhat.com/show_bug.cgi?id=1668286 and
it got fixed in 6.6 release IMO https://review.gluster.org/#/c/glusterfs/+/23233/. But
here he mentioned he saw the issue when
he upgraded from 6.5 to 6.6
One way to workaround is to perform a dummy setfacl(preferably using root) on the
corrupted files which will forcefully fetch the acl
info again from backend and update the context. Another approach to restart brick
process(kill and vol start force)
Regards,
Jiffin
----- Original Message -----
From: "Krutika Dhananjay" <kdhananj(a)redhat.com>
To: "Strahil Nikolov" <hunter86_bg(a)yahoo.com>, "Jiffin Thottan"
<jthottan(a)redhat.com>, "raghavendra talur" <rtalur(a)redhat.com>
Cc: "Nir Soffer" <nsoffer(a)redhat.com>, "Rafi Kavungal Chundattu
Parambil" <rkavunga(a)redhat.com>, "users" <users(a)ovirt.org>,
"gluster-user" <gluster-users(a)gluster.org>
Sent: Monday, December 2, 2019 11:48:22 AM
Subject: Re: [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available
for testing
Sorry about the late response.
I looked at the logs. These errors are originating from posix-acl
translator -
*[2019-11-17 07:55:47.090065] E [MSGID: 115050]
[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496:
LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6
(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6),
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
error-xlator: data_fast-access-control [Permission denied][2019-11-17
07:55:47.090174] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:36,gid:36,perm:1,ngrps:3),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050]
[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497:
LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7
(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7),
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
error-xlator: data_fast-access-control [Permission denied][2019-11-17
07:55:47.090299] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:36,gid:36,perm:1,ngrps:3),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]*
Jiffin/Raghavendra Talur,
Can you help?
-Krutika
On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi Nir,All,
it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached
screenshot of oVirt running on v7 gluster).
It seems strange that both my serious issues with oVirt are related to
gluster issue (1st gluster v3 to v5 migration and now this one).
I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all
nodes.
Now both Engine and all my VMs are back online - so if you hit issues with
6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before
deciding to wipe everything.
@Krutika,
I guess you will ask for the logs, so let's switch to gluster-users about
this one ?
Best Regards,
Strahil Nikolov
В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov <
hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
I have enabled TRACE log level for the volume data_fast,
but the issue is not much clear:
FUSE reports:
[2019-11-25 21:31:53.478130] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=6d9ed2e5-d4f2-4749-839b-2f1
3a68ed472 from backend
[2019-11-25 21:32:43.564694] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565653] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565689] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-25 21:32:43.565770] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-25 21:32:43.565858] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 279: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fbf40005ea8 (Permission denied)
While the BRICK logs on ovirt1/gluster1 report:
2019-11-25 21:32:43.564177] D [MSGID: 0] [io-threads.c:376:iot_schedule]
0-data_fast-io-threads: LOOKUP scheduled as fast priority fop
[2019-11-25 21:32:43.564194] T [MSGID: 0]
[defaults.c:2008:default_lookup_resume] 0-stack-trace: stack-address:
0x7fc02c00bbf8, winding from data_fast-io-threads to data_fast-upcall
[2019-11-25 21:32:43.564206] T [MSGID: 0] [upcall.c:790:up_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-upcall
to data_fast-leases
[2019-11-25 21:32:43.564215] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-leases
to data_fast-read-only
[2019-11-25 21:32:43.564222] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from
data_fast-read-only to data_fast-worm
[2019-11-25 21:32:43.564230] T [MSGID: 0] [defaults.c:2766:default_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-worm
to data_fast-locks
[2019-11-25 21:32:43.564241] T [MSGID: 0] [posix.c:2897:pl_lookup]
0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-locks
to data_fast-access-control
[2019-11-25 21:32:43.564254] I [MSGID: 139001]
[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
client:
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
req(uid:107,gid:107,perm:1,ngrps:4),
ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
[Permission denied]
[2019-11-25 21:32:43.564268] D [MSGID: 0]
[posix-acl.c:1057:posix_acl_lookup] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-access-control returned -1 error: Permission
denied [Permission denied]
[2019-11-25 21:32:43.564279] D [MSGID: 0] [posix.c:2888:pl_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-locks returned -1
error: Permission denied [Permission denied]
[2019-11-25 21:32:43.564289] D [MSGID: 0] [upcall.c:769:up_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-upcall returned -1
error: Permission denied [Permission denied]
[2019-11-25 21:32:43.564302] D [MSGID: 0]
[defaults.c:1349:default_lookup_cbk] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-io-threads returned -1 error: Permission denied
[Permission denied]
[2019-11-25 21:32:43.564313] T [marker.c:2918:marker_lookup_cbk]
0-data_fast-marker: lookup failed with Permission denied
[2019-11-25 21:32:43.564320] D [MSGID: 0]
[marker.c:2955:marker_lookup_cbk] 0-stack-trace: stack-address:
0x7fc02c00bbf8, data_fast-marker returned -1 error: Permission denied
[Permission denied]
[2019-11-25 21:32:43.564334] D [MSGID: 0] [index.c:2070:index_lookup_cbk]
0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-index returned -1
error: Permission denied [Permission denied]
В понеделник, 25 ноември 2019 г., 23:10:41 ч. Гринуич+2, Strahil Nikolov <
hunter86_bg(a)yahoo.com> написа:
Hi Krutika,
thanks for your assistance.
Let me summarize some info about the volume:
Volume Name: data_fast
Type: Replicate
Volume ID: 378804bf-2975-44d8-84c2-b541aa87f9ef
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster1:/gluster_bricks/data_fast/data_fast
Brick2: gluster2:/gluster_bricks/data_fast/data_fast
Brick3: ovirt3:/gluster_bricks/data_fast/data_fast (arbiter)
Options Reconfigured:
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: on
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
cluster.choose-local: on
client.event-threads: 4
server.event-threads: 4
storage.owner-uid: 36
storage.owner-gid: 36
performance.strict-o-direct: on
network.ping-timeout: 30
cluster.granular-entry-heal: enable
cluster.enable-shared-storage: enable
[root@ovirt1 ~]# gluster volume get engine all | grep shard
features.shard on
features.shard-block-size 64MB
features.shard-lru-limit 16384
features.shard-deletion-rate 100
On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <nsoffer(a)redhat.com> wrote:
On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com>
wrote:
Hi All,
another clue in the logs :
[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536798] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536959] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.537007] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1
gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission
denied)
[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
[2019-11-21 00:30:13.132756] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.132824] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133217] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133238] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c
[Permission denied]
[2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1
gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission
denied)
[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489520] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489669] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path:
/.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489717] E [MSGID: 133010]
[shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4
[Permission denied]
[2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1
gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission
denied)
Anyone got an idea why is it happening?
I checked user/group and selinux permissions - all OK
>Can you share the commands (and output) used to check this?
I first thought that the file is cached in memory and that's why vdsm user
can read the file , but the following shows opposite:
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# ll
total 562145
-rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29
5b1d3113-5cca-4582-9029-634b16338a2f
-rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.lease
-rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11
5b1d3113-5cca-4582-9029-634b16338a2f.meta
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always
use direct I/O.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
dd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied...
16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
>Seems like it could read upto ~67MB successfully before it encountered
'Permission denied' errors. Assuming a shard-block-size >of 64MB, looks
like all the shards under /.shard could not be accessed.
>Could you share the following pieces of information:
>1. brick logs of data_fast
Attached in data_fast-brick-logs.tgz
>2. ls -la of .shard relative to the bricks (NOT the mount) on all the
bricks of data_fast
Not sure if I understood you correctly, so I ran "ls -lad
/gluster_bricks/data_fast/data_fast/.shard". If it's not what you wanted to
see - just correct me.
I have run multiple "find" commands with "-exec chown vdsm:kvm {} \;"
,
just to be sure that this is not happening.
>3. and ls -la of all shards under .shard of data_fast (perhaps a handful
of them have root permission assigned somehow which is causing access to be
denied? Perhaps while resolving pending heals manually? )
All shards seem to be owned by "vdsm:kvm" with 660.
-Krutika
And dd continue to read data?!
I have never seen anything like this.
It will be helpful to run this with strace:
strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct
bs=8M status=progress
And share dd.strace.
Logs in /var/log/glusterfs/exportname.log will contain useful info for
this test.
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 >
/proc/sys/vm/drop_caches
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd
if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s
As you can see , once root user reads the file -> vdsm user can also do
that.
Smells like issue on gluster side.
>I would try this on the hypervisor to check what vdsm/qemu see:
>$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path
I'm attaching the output of the find I run, but this one should be enough:
[root@ovirt1 ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/
-not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and
selinux label
of the entire tree will be more useful.
>Also, to make sure we don't have selinux issue on the hypervisor, you can
change
>selinux to permissive mode:
> setenforce 0
This is the first thing I did and the systems were still in permissive
when I tried again.I'm 99.99% sure it's not selinux.
>And then try again. If this was selinux issue the permission denied issue
will disappear.
>If this is the case please provide the output of:
> ausearh -m AVC -ts today
>If the issue still exists, we eliminated selinux, and you can enable it
again:
> setenforce 1
[root@ovirt3 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt2 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt1 ~]# ausearch -m AVC -ts today
<no matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on
the gluster side?
I have a vague feeling that the issue is related to gluster v6.5 to 6.6
upgrade which I several days before... So if any logs are needed (or debug
enabled), just mention.
If this is the last change, and it worked before, most likely.
Nir
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NE...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/UJLBNDFKU27...
10:27 p.m.
New subject: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing
On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi All,
another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation
failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission
denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae
fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022]
[shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission
denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c
fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031]
[client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation
failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W
[MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:
remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6
(00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E
[MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on
shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission
denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]
0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4
fd=0x7fc63c01a058 (Permission denied)
Anyone got an idea why is it happening?I checked user/group and selinux permissions - all
OKI even replaced one of the disks and healed , but the result is the same for all my
VMs.
Have you checked the permission for user/group are set correctly
across all the bricks in the cluster?Seem OK:
root@ovirt1 ~]# for i in ovirt{1..3};
do echo $i; ssh $i "find /gluster_bricks/*/*/[0-9]* -not -user vdsm -not -type l
-print" ; echo;echo; doneovirt1
ovirt2
ovirt3
What does ls -la on the images directory from mount of the volume show
you.Attached the output.
Adding Krutika and Rafi as they ran into a similar issue in the past.Most probably the
root's dd is putting the image into the ram ,and then the second dd (via sudo -u vdsm)
gets it from linux cache...
Best Regards,Strahil Nikolov
В сряда, 20 ноември 2019 г., 18:17:18 ч. Гринуич+2, Strahil Nikolov
<hunter86_bg(a)yahoo.com> написа:
Hello All,
my engine is back online , but I'm still having difficulties to make vdsm powerup the
systems.I think that the events generated today can lead me to the right direction(just an
example , many more are there):
VDSM ovirt3.localdomain command SpmStatusVDS failed: Cannot inquire
Lease(name='SDM',
path=u'/rhev/data-center/mnt/glusterSD/gluster1:_data__fast3/ecc3bf0e-8214-45c1-98a6-0afa642e591f/dom_md/leases',
offset=1048576): (2, 'Sanlock get hosts failure', 'No such file or
directory')
I will try to collect a fresh log and see what is it complaining about this time.
Best Regards,Strahil Nikolov
Hi Sahina,
I have a strange situation:
1. When I try to access the file via 'sudo -u vdsm dd if=disk of=test bs=4M' the
command fails on aprox 60MB.
2. If I run same command as root , remove the file and then run again via vdsm user ->
this time no i/o error reported.
My guess is that I need to check what's going on the bricks
themselve ...
Best Regards,
Strahil Nikolov
В вторник, 19 ноември 2019 г., 0:02:16 ч. Гринуич-5, Sahina Bose
<sabose(a)redhat.com> написа:
On Tue, Nov 19, 2019 at 10:10 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Hi Sahina,
Sadly engine logs have no errors.I've got only an I/O error, but in the debug of the
vdsm I can clearly see that "qemu-img" is giving an "OK".During the
upgrade I got some metadata files pending heal, but I have recovered the conflict manually
and should be OK.Today I have defined one of the VMs manually (virsh define) and then
started it , but the issue is the same.It seems to be storage-related issue,as VMs that
are on specific domain can be started , but most of my VMs are on the fast storage domains
and none of them can be started.
After the gluster snapshot restore , the engine is having issues and I have to separately
investigate that (as I poweroff my HostedEngine before creating the snapshot).
The logs can be find at
: https://drive.google.com/open?id=1VAZFZWWrpimDeVuZT0sWFVXy76scr4NM
Any ideas where to look at , as I can definitely read (using "dd if=disk" or
qemu-img info) the disks of the rhel7 VM ?
The vdsm logs have this:2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') abnormal vm stop device
ua-94f763e9-fd96-4bee-a6b2-31af841a918b error eother (vm:5075)
2019-11-17 10:21:23,892+0200 INFO (libvirt/events) [virt.vm]
(vmId='b3c4d84a-9784-470c-b70e-7ad7cc45e913') CPU stopped: onIOError (vm:6062)
2019-11-17 10:21:23,893+0200 DEBUG (libvirt/events) [jsonrpc.Notification] Sending event
{"params": {"notify_time": 4356025830,
"b3c4d84a-9784-470c-b70e-7ad7cc45e913": {"status":
"WaitForLaunch", "ioerror": {"alias":
"ua-94f763e9-fd96-4bee-a6b2-31af841a918b", "name": "sda",
"path":
"/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f"},
"pauseCode": "EOTHER"}}, "jsonrpc": "2.0",
"method": "|virt|VM_status|b3c4d84a-9784-470c-b70e-7ad7cc45e913"}
(__init__:181)
Can you check the permissions of the file
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b/5b1d3113-5cca-4582-9029-634b16338a2f.
Was it reset after upgrade?
Are you able to copy this file to a different location and try running a VM with this
image?
Any errors in the mount log of gluster1:_data__fast volume?
Best Regards,Strahil Nikolov
В понеделник, 18 ноември 2019 г., 11:38:13 ч. Гринуич+2, Sahina Bose
<sabose(a)redhat.com> написа:
On Mon, Nov 18, 2019 at 2:58 PM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
+Sahina Bose +Gobinda Das +Nir Soffer +Tal Nisan can you please help here?
Il giorno dom 17 nov 2019 alle ore 16:00 Strahil Nikolov <hunter86_bg(a)yahoo.com> ha
scritto:
So far,
I have rolled back the engine and the 3 hosts - still cannot manipulate the storage.It
seems that gluster itself is working, but vdsm and the oVirt stack cannot access the
storage - cannot create new VM disks, cannot start a VM and I'm on the verge of
redeploy.
Any errors in vdsm logs? engine logs?
Best Regards,Strahil Nikolov
В събота, 16 ноември 2019 г., 15:40:25 ч. Гринуич+2, Strahil
<hunter86_bg(a)yahoo.com> написа:
I got upgraded to RC3 and now cannot power any VM .
Constantly getting I/O error, but checking at gluster level - I can dd from each disk or
even create a new one.
Removing the HighAvailability doesn't help.
I guess I should restore the engine from the gluster snapshot and rollback via 'yum
history undo last'.
Does anyone else have my issues ?
Best Regards,
Strahil Nikolov
On Nov 13, 2019 15:31, Sandro Bonazzola <sbonazzo(a)redhat.com> wrote:
Il giorno mer 13 nov 2019 alle ore 14:25 Sandro Bonazzola <sbonazzo(a)redhat.com> ha
scritto:
Il giorno mer 13 nov 2019 alle ore 13:56 Florian Schmid <fschmid(a)ubimet.com> ha
scritto:
Hello,
I have a question about bugs, which are flagged as [downstream clone - 4.3.7], but are not
yet released.
I'm talking about this bug:https://bugzilla.redhat.com/show_bug.cgi?id=1749202
I can't see it in 4.3.7 release notes. Will it be included in a further release
candidate? This fix is very important I think and I can't upgrade yet because of this
bug.
Looking at the bug, the fix was done with $ git tag --contains
12bd5cb1fe7c95e29b4065fca968913722fe9eaaovirt-engine-4.3.6.6
ovirt-engine-4.3.6.7
ovirt-engine-4.3.7.0
ovirt-engine-4.3.7.1
So the fix is already included in release oVirt 4.3.6.
Sent a fix to 4.3.6 release notes: https://github.com/oVirt/ovirt-site/pull/2143. @Ryan
Barry can you please review?
BR Florian Schmid
Von: "Sandro Bonazzola" <sbonazzo(a)redhat.com>
An: "users" <users(a)ovirt.org>
Gesendet: Mittwoch, 13. November 2019 13:34:59
Betreff: [ovirt-users] [ANN] oVirt 4.3.7 Third Release Candidate is now available for
testing
The oVirt Project is pleased to announce the availability of the oVirt 4.3.7 Third Release
Candidate for testing, as of November 13th, 2019.
This update is a release candidate of the seventh in a series of stabilization updates to
the 4.3 series.
This is pre-release software. This pre-release should not to be used in production.
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures for:
* Red Hat Enterprise Linux 7.7 or later (but <8)
* CentOS Linux (or similar) 7.7 or later (but <8)
* oVirt Node 4.3 (available for x86_64 only) has been built consuming CentOS 7.7 Release
See the release notes [1] for known issues, new features and bugs fixed.
While testing this release candidate please note that oVirt node now includes:
- ansible 2.9.0
- GlusterFS 6.6
Notes:
- oVirt Appliance is already available
- oVirt Node is already available
Additional Resources:
* Read more about the oVirt 4.3.7 release highlights: http://www.ovirt.org/release/4.3.7/
* Get more oVirt Project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.3.7/
[2] http://resources.ovirt.org/pub/ovirt-4.3-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/24QUREJPZHT...
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA
sbonazzo(a)redhat.com
| |
Red Hat respects your work life balance. Therefore there is no need to answer this email
out of your office hours.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3ZPGKJ4JJWV...
1751
days inactive
1826
days old
20 comments
7 participants
participants (7)
-
Jiffin Thottan -
Krutika Dhananjay -
Nir Soffer -
Sahina Bose -
Sandro Bonazzola -
Strahil -
Strahil Nikolov