On 7/14/22 00:34, Strahil Nikolov wrote:
Well... not yet.
Check if the engine detects the volumes and verify again that all
glustereventsd work.
I would even consider restarting the engine, just to be on the safe side.
engine restarted (I also yum updated it before), glustereventsd is
running on all hosts, selinux's port label is set. Still only one brick
is up and two are in unknown state in manager. See the screenshot.
What is your oVirt version ? Maybe an update could solve your
problem.
latest in 4.4 version -> 4.4.10.7-1.el8. Engine and hosts are Rocky
Linux 8.6 based.
ale here is output of select * from gluster_volume_bricks; I dont know
if it is relevant but it shows old (2022-06-12 14:32:46.476558+02) dates
in _update_date filed in part of UNKNOWN state bricks. Also UP state
bricks has timestamp i past (around 1 day), is it normal?
https://pastebin.com/hQnj1en3
Could you suggest some relevant strings in engine.log | vdsm.log I could
looks for? My blind grepping just reveleated
grep "GLUSTER" /var/log/ovirt-engine/engine.log
2022-07-13 13:10:19,263+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-3) [] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086),
Detected change in status of brick
10.0.4.11:/gluster_bricks/engine/engine of volume engine of cluster
McHosting from UNKNOWN to UP via gluster event.
2022-07-13 13:10:19,492+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-3) [] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086),
Detected change in status of brick 10.0.4.11:/gluster_bricks/vms/vms of
volume vms of cluster McHosting from UNKNOWN to UP via gluster event.
2022-07-13 13:10:21,185+02 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-3) [] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086),
Detected change in status of brick 10.0.4.11:/gluster_bricks/vms2/vms2
of volume vms of cluster McHosting from UNKNOWN to UP via gluster event.
it match the engine log and also time I run glustereventsd service on
host 10.0.4.11
but I see also repeatedly
2022-07-13 14:14:14,440+02 WARN
[org.ovirt.engine.core.bll.UpdateClusterCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-88)
[5fb948bc] Validation of action 'UpdateCluster' failed for user SYSTEM.
Reasons:
VAR__TYPE__CLUSTER,VAR__ACTION__UPDATE,CLUSTER_CANNOT_DISABLE_GLUSTER_WHEN_CLUSTER_CONTAINS_VOLUMES
so it looks to me like there is some action running which wants disable
gluster services on cluster and it cannot (and it is right!). But it
probably blocks that gluster services checkbox in cluster settings in
manager. What do you think?
Cheers, Jiri
Best Regards,
Strahil Nikolov
On Wed, Jul 13, 2022 at 17:05, Jiří Sléžka
<jiri.slezka(a)slu.cz> wrote:
On 7/13/22 14:53, Jiří Sléžka wrote:
> On 7/12/22 22:28, Strahil Nikolov wrote:
>> glustereventad will notify the engine when something changes -
like a
>> new volume is created from the cli (or bad things happened ;) ),
so it
>> should be running. >
>> You can use the workaround from the github issue and reatart the
>> glustereventsd service.
>
> ok, workaround applied, glustereventsd service enabled and
started on
> all hosts.
>
> I can see this log entry in volume Events
>
> Detected change in status of brick
> 10.0.4.11:/gluster_bricks/engine/engine of volume engine of cluster
> McHosting from UNKNOWN to UP via gluster event.
>
> but Bricks tab shows still two (.12 and .13) of three bricks in
Unknown
> state. From command line point of view all bricks are up and healthy.
>
> it looks like engine thinks that gluster service is disabled in
cluster
> but I cannot enable it because checkbox is disabled. In my other (FC
> based) oVirt instance Gluster Service checkbox is not selected
but not
> disabled. So I am interested what could make that checkbox
inactive...
well, on db side it looks like cluster has gluster_service disabled...
engine=# select virt_service, gluster_service from cluster;
virt_service | gluster_service
--------------+-----------------
t | f
(1 row)
still don't know why the checkbox is disabled. Would it be safe to
enabled gluster_service directly in db? I suppose no... :-)
Cheers,
Jiri
>
>> For the vdsm, you can always run
>> '/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start' which is
executed
>> by the vdsmd.service before every start (ExecStartPre stanza)
and see
>> if it complains about something.
>
> [root@ovirt-hci03 <mailto:root@ovirt-hci03> ~]#
/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start
> vdsm: Running mkdirs
> vdsm: Running configure_vdsm_logs
> vdsm: Running run_init_hooks
> vdsm: Running check_is_configured
> sanlock is configured for vdsm
> lvm is configured for vdsm
> abrt is already configured for vdsm
> Managed volume database is already configured
> Current revision of multipath.conf detected, preserving
> libvirt is already configured for vdsm
> vdsm: Running validate_configuration
> SUCCESS: ssl configured to true. No conflicts
> vdsm: Running prepare_transient_repository
> vdsm: Running syslog_available
> vdsm: Running nwfilter
> vdsm: Running dummybr
> vdsm: Running tune_system
> vdsm: Running test_space
> vdsm: Running test_lo
>
> retcode 0, all looks ok...
>
> Cheers,
>
> Jiri
>
>>
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> On Tue, Jul 12, 2022 at 11:12, Jiří Sléžka
>> <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz>> wrote:
>> On 7/11/22 16:22, Jiří Sléžka wrote:
>> > On 7/11/22 15:57, Strahil Nikolov wrote:
>> >> Can you check for AVC denials and the error message like the
>> described
>> >> in
>> >>
>>
>>
https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183
<
https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183>
>>
>>
<
https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183
<
https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183>
>> >?
>> >
>> > thanks for reply, there are two unrelated (qemu-kvm) avc
denials
>> logged
>> > (related probably to sanlock recovery)
>> >
>> > also I cannot find glustereventsd in any related log...
is it
>> really
>> > used by vdsm-gluster?
>> >
>> > this service runs on no hosts
>> >
>> > systemctl status glustereventsd
>> > ● glustereventsd.service - Gluster Events Notifier
>> > Loaded: loaded
>> (/usr/lib/systemd/system/glustereventsd.service;
>> > disabled; vendor preset: disabled)
>> > Active: inactive (dead)
>>
>> it looks like root of the problem is that Gluster service is
>> disabled in
>> cluster settings and cannot be enabled. But it was enabled
before...
>> also I have to manually install vdsm-gluster when I
(re)install new
>> host, but bricks from this host are in unknown state in
admin. Maybe
>> vdsm-gluster is not correctly configured? Maybe
glustereventsd is not
>> running? I am just guessing...
>>
>> I have no access to other HCI installation so I cannot compare
>> differences.
>>
>> I would be really happy if someone could tell me what
circumstances
>> could disable Gluster service checkbox in admin and how to
enable it
>> again...
>>
>> Cheers,
>>
>> Jiri
>>
>>
>> >
>> > Cheers,
>> >
>> > Jiri
>> >
>> >
>> >>
>> >>
>> >> Best Regards,
>> >> Strahil Nikolov
>> >>
>> >> On Mon, Jul 11, 2022 at 16:44, Jiří Sléžka
>> >> <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz>
<mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote:
>> >> Hello,
>> >>
>> >> On 7/11/22 14:34, Strahil Nikolov wrote:
>> >> > Can you check something on the host:
>> >> > cat /etc/glusterfs/eventsconfig.json
>> >>
>> >> cat /etc/glusterfs/eventsconfig.json
>> >> {
>> >> "log-level": "INFO",
>> >> "port": 24009,
>> >> "disable-events-log": false
>> >> }
>> >>
>> >>
>> >> > semanage port -l | grep $(awk -F ':'
'/port/
>> {gsub(",","",$2);
>> >> print
>> >> > $2}' /etc/glusterfs/eventsconfig.json)
>> >>
>> >> semanage port -l | grep 24009
>> >>
>> >> returns empty set, it looks like this port is not
labeled
>> >>
>> >> Cheers,
>> >>
>> >> Jiri
>> >>
>> >> >
>> >> > Best Regards,
>> >> > Strahil Nikolov
>> >> > В понеделник, 11 юли 2022 г., 02:18:57 ч.
Гринуич+3, Jiří
>> Sléžka
>> >> > <jiri.slezka(a)slu.cz
<mailto:jiri.slezka@slu.cz>
<mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>
>> <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>
<mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> написа:
>> >> >
>> >> >
>> >> > Hi,
>> >> >
>> >> > I would like to change CPU Type in my oVirt
4.4.10 HCI
>> cluster
>> >> (based on
>> >> > 3 glusterfs/virt hosts). When I try to I got this
error
>> >> >
>> >> > Error while executing action: Cannot disable
gluster
>> service on
>> >> the
>> >> > cluster as it contains volumes.
>> >> >
>> >> > As I remember I had Gluster Service enabled on
this
>> cluster but
>> >> now both
>> >> > (Enable Virt Services and Enable Gluster Service)
>> checkboxes are
>> >> grayed
>> >> > out and Gluster Service is unchecked.
>> >> >
>> >> > Also Storage / Volumes displays my volumes...
well,
>> displays one
>> >> brick
>> >> > on particular host in unknown state (? mark)
which is new
>> >> situation. As
>> >> > I can see from command line all bricks are online,
no
>> healing in
>> >> > progress, all looks good...
>> >> >
>> >> > I am not sure if the second issue is relevant to
first
>> one so main
>> >> > question is how can I (re)enable gluster service
in my
>> cluster?
>> >> >
>> >> > Thanks in advance,
>> >> >
>> >> > Jiri
>> >> > _______________________________________________
>> >> > Users mailing list -- users(a)ovirt.org
<mailto:users@ovirt.org>
>> <mailto:users@ovirt.org <mailto:users@ovirt.org>>
<mailto:users@ovirt.org <mailto:users@ovirt.org>
>> <mailto:users@ovirt.org <mailto:users@ovirt.org>>>
>> >> <mailto:users@ovirt.org
<mailto:users@ovirt.org>
<mailto:users@ovirt.org <mailto:users@ovirt.org>>
>> <mailto:users@ovirt.org <mailto:users@ovirt.org>
<mailto:users@ovirt.org <mailto:users@ovirt.org>>>>
>> >> > To unsubscribe send an email to
users-leave(a)ovirt.org <mailto:users-leave@ovirt.org>
>> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>>
>> >> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>>>
>> >>
>> >> > <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>
>> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>
>> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>>>>
>> >> > Privacy Statement:
>>
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>
>> <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>>
>> >> <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>
>> <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html >>>
>> >> > <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>
>> <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>>
>> >> <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>
>> <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>>>>
>> >> > oVirt Code of Conduct:
>> >> >
>>
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>
>> <
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>>
>> >>
<
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>
>> <
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/ >>>
>> >> >
>> <
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>
>> <
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>>
>> >>
<
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>
>> <
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>>>>
>> >> > List Archives:
>> >> >
>> >>
>> >>
>>
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>
>>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>
>>
>> >
>> >>
>> >>
>> >>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>
>>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>
>>
>> >
>> >>
>> >> >
>> >> >
>> >>
>> >>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>
>>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>
>>
>> >
>> >>
>> >>
>> >>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>
>>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ...
>>
>>
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Users mailing list -- users(a)ovirt.org
<mailto:users@ovirt.org> <mailto:users@ovirt.org
<mailto:users@ovirt.org>>
>> > To unsubscribe send an email to users-leave(a)ovirt.org
<mailto:users-leave@ovirt.org>
>> <mailto:users-leave@ovirt.org
<mailto:users-leave@ovirt.org>>
>> > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>
>> <
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>>
>> > oVirt Code of Conduct:
>>
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>
>> <
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>>
>> > List Archives:
>>
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2...
>
>>
>>
>>
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2...
>>
>>
>
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org>
> To unsubscribe send an email to users-leave(a)ovirt.org
<mailto:users-leave@ovirt.org>
> Privacy Statement:
https://www.ovirt.org/privacy-policy.html
<
https://www.ovirt.org/privacy-policy.html>
> oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
<
https://www.ovirt.org/community/about/community-guidelines/>
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YVDPYIPFHHF...
<
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YVDPYIPFHHF...