[ovirt-users] Re: gluster service on the cluster is unchecked on hci cluster

On 7/14/22 00:34, Strahil Nikolov wrote: > Well... not yet. > Check if the engine detects the volumes and verify again that all > glustereventsd work. > > I would even consider restarting the engine, just to be on the safe side. engine restarted (I also yum updated it before), glustereventsd is running on all hosts, selinux's port label is set. Still only one brick is up and two are in unknown state in manager. See the screenshot. > What is your oVirt version ? Maybe an update could solve your problem. latest in 4.4 version -> 4.4.10.7-1.el8. Engine and hosts are Rocky Linux 8.6 based. ale here is output of select * from gluster_volume_bricks; I dont know if it is relevant but it shows old (2022-06-12 14:32:46.476558+02) dates in _update_date filed in part of UNKNOWN state bricks. Also UP state bricks has timestamp i past (around 1 day), is it normal? https://pastebin.com/hQnj1en3 Could you suggest some relevant strings in engine.log | vdsm.log I could looks for? My blind grepping just reveleated grep "GLUSTER" /var/log/ovirt-engine/engine.log 2022-07-13 13:10:19,263+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick 10.0.4.11:/gluster_bricks/engine/engine of volume engine of cluster McHosting from UNKNOWN to UP via gluster event. 2022-07-13 13:10:19,492+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick 10.0.4.11:/gluster_bricks/vms/vms of volume vms of cluster McHosting from UNKNOWN to UP via gluster event. 2022-07-13 13:10:21,185+02 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [] EVENT_ID: GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick 10.0.4.11:/gluster_bricks/vms2/vms2 of volume vms of cluster McHosting from UNKNOWN to UP via gluster event. it match the engine log and also time I run glustereventsd service on host 10.0.4.11 but I see also repeatedly 2022-07-13 14:14:14,440+02 WARN [org.ovirt.engine.core.bll.UpdateClusterCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-88) [5fb948bc] Validation of action 'UpdateCluster' failed for user SYSTEM. Reasons: VAR__TYPE__CLUSTER,VAR__ACTION__UPDATE,CLUSTER_CANNOT_DISABLE_GLUSTER_WHEN_CLUSTER_CONTAINS_VOLUMES so it looks to me like there is some action running which wants disable gluster services on cluster and it cannot (and it is right!). But it probably blocks that gluster services checkbox in cluster settings in manager. What do you think?

Cheers, Jiri > > Best Regards, > Strahil Nikolov > >     On Wed, Jul 13, 2022 at 17:05, Jiří Sléžka >     <jiri.slezka(a)slu.cz&gt; wrote: >     On 7/13/22 14:53, Jiří Sléžka wrote: >      > On 7/12/22 22:28, Strahil Nikolov wrote: >      >> glustereventad will notify the engine when something changes - >     like a >      >> new volume is created from the cli (or bad things happened ;) ), >     so it >      >> should be running. > >      >> You can use the workaround from the github issue and reatart the >      >> glustereventsd service. >      > >      > ok, workaround applied, glustereventsd service enabled and >     started on >      > all hosts. >      > >      > I can see this log entry in volume Events >      > >      > Detected change in status of brick >      > 10.0.4.11:/gluster_bricks/engine/engine of volume engine of > cluster >      > McHosting from UNKNOWN to UP via gluster event. >      > >      > but Bricks tab shows still two (.12 and .13) of three bricks in >     Unknown >      > state. From command line point of view all bricks are up and > healthy. >      > >      > it looks like engine thinks that gluster service is disabled in >     cluster >      > but I cannot enable it because checkbox is disabled. In my > other (FC >      > based) oVirt instance Gluster Service checkbox is not selected >     but not >      > disabled. So I am interested what could make that checkbox >     inactive... > >     well, on db side it looks like cluster has gluster_service > disabled... > >     engine=# select virt_service, gluster_service from cluster; >        virt_service | gluster_service >     --------------+----------------- >        t            | f >     (1 row) > >     still don't know why the checkbox is disabled. Would it be safe to >     enabled gluster_service directly in db? I suppose no... :-) > >     Cheers, > >     Jiri > > >      > >      >> For the vdsm, you can always run >      >> '/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start' which is >     executed >      >> by the vdsmd.service before every start (ExecStartPre stanza) >     and see >      >> if it complains about something. >      > >      > [root@ovirt-hci03 <mailto:root@ovirt-hci03> ~]# >     /usr/libexec/vdsm/vdsmd_init_common.sh --pre-start >      > vdsm: Running mkdirs >      > vdsm: Running configure_vdsm_logs >      > vdsm: Running run_init_hooks >      > vdsm: Running check_is_configured >      > sanlock is configured for vdsm >      > lvm is configured for vdsm >      > abrt is already configured for vdsm >      > Managed volume database is already configured >      > Current revision of multipath.conf detected, preserving >      > libvirt is already configured for vdsm >      > vdsm: Running validate_configuration >      > SUCCESS: ssl configured to true. No conflicts >      > vdsm: Running prepare_transient_repository >      > vdsm: Running syslog_available >      > vdsm: Running nwfilter >      > vdsm: Running dummybr >      > vdsm: Running tune_system >      > vdsm: Running test_space >      > vdsm: Running test_lo >      > >      > retcode 0, all looks ok... >      > >      > Cheers, >      > >      > Jiri >      > >      >> >      >> >      >> Best Regards, >      >> Strahil Nikolov >      >> >      >>     On Tue, Jul 12, 2022 at 11:12, Jiří Sléžka >      >>     <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz>> wrote: >      >>     On 7/11/22 16:22, Jiří Sléžka wrote: >      >>      > On 7/11/22 15:57, Strahil Nikolov wrote: >      >>      >> Can you check for AVC denials and the error message > like the >      >>     described >      >>      >> in >      >>      >> >      >> >      >> > > https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183 > > <https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183> >      >> >      >> > > <https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183 > > <https://github.com/gluster/glusterfs-selinux/issues/27#issue-1097225183> >      >>      >? >      >>      > >      >>      > thanks for reply, there are two unrelated (qemu-kvm) avc >     denials >      >>     logged >      >>      > (related probably to sanlock recovery) >      >>      > >      >>      > also I cannot find glustereventsd in any related log... >     is it >      >> really >      >>      > used by vdsm-gluster? >      >>      > >      >>      > this service runs on no hosts >      >>      > >      >>      > systemctl status glustereventsd >      >>      > ● glustereventsd.service - Gluster Events Notifier >      >>      >     Loaded: loaded >      >> (/usr/lib/systemd/system/glustereventsd.service; >      >>      > disabled; vendor preset: disabled) >      >>      >     Active: inactive (dead) >      >> >      >>     it looks like root of the problem is that Gluster service is >      >>     disabled in >      >>     cluster settings and cannot be enabled. But it was enabled >     before... >      >>     also I have to manually install vdsm-gluster when I >     (re)install new >      >>     host, but bricks from this host are in unknown state in >     admin. Maybe >      >>     vdsm-gluster is not correctly configured? Maybe >     glustereventsd is not >      >>     running? I am just guessing... >      >> >      >>     I have no access to other HCI installation so I cannot > compare >      >>     differences. >      >> >      >>     I would be really happy if someone could tell me what >     circumstances >      >>     could disable Gluster service checkbox in admin and how to >     enable it >      >>     again... >      >> >      >>     Cheers, >      >> >      >>     Jiri >      >> >      >> >      >>      > >      >>      > Cheers, >      >>      > >      >>      > Jiri >      >>      > >      >>      > >      >>      >> >      >>      >> >      >>      >> Best Regards, >      >>      >> Strahil Nikolov >      >>      >> >      >>      >>     On Mon, Jul 11, 2022 at 16:44, Jiří Sléžka >      >>      >>     <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz> >     <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>> wrote: >      >>      >>     Hello, >      >>      >> >      >>      >>     On 7/11/22 14:34, Strahil Nikolov wrote: >      >>      >>      > Can you check something on the host: >      >>      >>      > cat /etc/glusterfs/eventsconfig.json >      >>      >> >      >>      >>     cat /etc/glusterfs/eventsconfig.json >      >>      >>     { >      >>      >>          "log-level": "INFO", >      >>      >>          "port": 24009, >      >>      >>          "disable-events-log": false >      >>      >>     } >      >>      >> >      >>      >> >      >>      >>      > semanage port -l | grep $(awk -F ':' '/port/ >      >>     {gsub(",","",$2); >      >>      >> print >      >>      >>      > $2}' /etc/glusterfs/eventsconfig.json) >      >>      >> >      >>      >>     semanage port -l | grep 24009 >      >>      >> >      >>      >>     returns empty set, it looks like this port is not >     labeled >      >>      >> >      >>      >>     Cheers, >      >>      >> >      >>      >>     Jiri >      >>      >> >      >>      >>      > >      >>      >>      > Best Regards, >      >>      >>      > Strahil Nikolov >      >>      >>      > В понеделник, 11 юли 2022 г., 02:18:57 ч. >     Гринуич+3, Jiří >      >>     Sléžka >      >>      >>      > <jiri.slezka(a)slu.cz <mailto:jiri.slezka@slu.cz> >     <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>> >      >>     <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz> >     <mailto:jiri.slezka@slu.cz <mailto:jiri.slezka@slu.cz>>>> написа: >      >>      >>      > >      >>      >>      > >      >>      >>      > Hi, >      >>      >>      > >      >>      >>      > I would like to change CPU Type in my oVirt >     4.4.10 HCI >      >>     cluster >      >>      >>     (based on >      >>      >>      > 3 glusterfs/virt hosts). When I try to I got this >     error >      >>      >>      > >      >>      >>      > Error while executing action: Cannot disable > gluster >      >>     service on >      >>      >> the >      >>      >>      > cluster as it contains volumes. >      >>      >>      > >      >>      >>      > As I remember I had Gluster Service enabled on > this >      >>     cluster but >      >>      >>     now both >      >>      >>      > (Enable Virt Services and Enable Gluster Service) >      >>     checkboxes are >      >>      >>     grayed >      >>      >>      > out and Gluster Service is unchecked. >      >>      >>      > >      >>      >>      > Also Storage / Volumes displays my volumes... > well, >      >>     displays one >      >>      >>     brick >      >>      >>      > on particular host in unknown state (? mark) >     which is new >      >>      >>     situation. As >      >>      >>      > I can see from command line all bricks are > online, no >      >>     healing in >      >>      >>      > progress, all looks good... >      >>      >>      > >      >>      >>      > I am not sure if the second issue is relevant to >     first >      >>     one so main >      >>      >>      > question is how can I (re)enable gluster service >     in my >      >>     cluster? >      >>      >>      > >      >>      >>      > Thanks in advance, >      >>      >>      > >      >>      >>      > Jiri >      >>      >>      > _______________________________________________ >      >>      >>      > Users mailing list -- users(a)ovirt.org >     <mailto:users@ovirt.org> >      >>     <mailto:users@ovirt.org <mailto:users@ovirt.org>> >     <mailto:users@ovirt.org <mailto:users@ovirt.org> >      >>     <mailto:users@ovirt.org <mailto:users@ovirt.org>>> >      >>      >>     <mailto:users@ovirt.org <mailto:users@ovirt.org> >     <mailto:users@ovirt.org <mailto:users@ovirt.org>> >      >>     <mailto:users@ovirt.org <mailto:users@ovirt.org> >     <mailto:users@ovirt.org <mailto:users@ovirt.org>>>> >      >>      >>      > To unsubscribe send an email to >     users-leave(a)ovirt.org <mailto:users-leave@ovirt.org> >      >>     <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >      >>      >>     <mailto:users-leave@ovirt.org >     <mailto:users-leave@ovirt.org> <mailto:users-leave@ovirt.org >     <mailto:users-leave@ovirt.org>>> >      >>      >> >      >>      >>      > <mailto:users-leave@ovirt.org >     <mailto:users-leave@ovirt.org> >      >>     <mailto:users-leave@ovirt.org >     <mailto:users-leave@ovirt.org>> <mailto:users-leave@ovirt.org >     <mailto:users-leave@ovirt.org> >      >>     <mailto:users-leave@ovirt.org > <mailto:users-leave@ovirt.org>>>> >      >>      >>      > Privacy Statement: >      >> https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html> >      >>     <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html>> >      >>      >>     <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html> >      >>     <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html >>> >      >>      >>      > <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html> >      >>     <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html>> >      >>      >>     <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html> >      >>     <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html>>>> >      >>      >>      > oVirt Code of Conduct: >      >>      >>      > >      >> https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/> >      >>     <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/>> >      >>      >> > <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/> >      >>     <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/ >>> >      >>      >>      > >      >>     <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/> >      >>     <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/>> >      >>      >> > <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/> >      >>     <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/>>>> >      >>      >>      > List Archives: >      >>      >>      > >      >>      >> >      >>      >> >      >> >      >> > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > >      > >      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > >      > >      >> >      >>      > >      >>      >> >      >>      >> >      >>      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > >      > >      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > >      > >      >> >      >>      > >      >>      >> >      >>      >>      > >      >>      >>      > >      >>      >> >      >>      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > >      > >      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > >      > >      >> >      >>      > >      >>      >> >      >>      >> >      >>      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > >      > >      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/S4NVCQ33ZSJ... > > >      >> >      >> >      >>      >> >      >>      >> >      >>      > >      >>      > >      >>      > _______________________________________________ >      >>      > Users mailing list -- users(a)ovirt.org >     <mailto:users@ovirt.org> <mailto:users@ovirt.org >     <mailto:users@ovirt.org>> >      >>      > To unsubscribe send an email to users-leave(a)ovirt.org >     <mailto:users-leave@ovirt.org> >      >>     <mailto:users-leave@ovirt.org <mailto:users-leave@ovirt.org>> >      >>      > Privacy Statement: >     https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html> >      >>     <https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html>> >      >>      > oVirt Code of Conduct: >      >> https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/> >      >>     <https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/>> >      >>      > List Archives: >      >> >      >> > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2... > >      > >      >> >      >> >      >> > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/KJZMKJCPGC2... > > >      >> >      >> >      > >      > >      > _______________________________________________ >      > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >      > To unsubscribe send an email to users-leave(a)ovirt.org >     <mailto:users-leave@ovirt.org> >      > Privacy Statement: https://www.ovirt.org/privacy-policy.html >     <https://www.ovirt.org/privacy-policy.html> >      > oVirt Code of Conduct: >     https://www.ovirt.org/community/about/community-guidelines/ >     <https://www.ovirt.org/community/about/community-guidelines/> >      > List Archives: > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/YVDPYIPFHHF... > > > <https://lists.ovirt.org/archives/list/users@ovirt.org/message/YVDPYIPFHHF... > > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LA5YEMMY4UW...