Hello Team, We are running a setup of 3-way replica HC gluster setup configured during the initial deployment from the cockpit console using ansible. NODE1 - /dev/sda (OS) - /dev/sdb ( Gluster Bricks ) * /gluster_bricks/engine/engine/ * /gluster_bricks/data/data/ * /gluster_bricks/vmstore/vmstore/ NODE2 and NODE3 with a similar setup. Hosted engine was running on node2. - While moving NODE1 to maintenance mode along with stopping the gluster service as it prompts before, Hosted engine instantly went down. - I start the gluster service back on node1 and start the hosted engine again and found hosted engine started properly but getting crashed again and again within frames of second after a successful start because HE itself stopping glusterd on node1. (not sure) but cross-verified by checking glusterd status. *Is it possible to clear pending tasks or not let the HE to stop glusterd on node1?* *Or we can start the HE using other gluster node?* https://paste.fedoraproject.org/paste/Qu2tSHuF-~G4GjGmstV6mg -- ABHISHEK SAHNI IISER Bhopal
On Tue, Dec 4, 2018 at 11:32 AM Abhishek Sahni <abhishek.sahni1991@gmail.com> wrote:
Hello Team,
We are running a setup of 3-way replica HC gluster setup configured during the initial deployment from the cockpit console using ansible.
NODE1 - /dev/sda (OS) - /dev/sdb ( Gluster Bricks ) * /gluster_bricks/engine/engine/ * /gluster_bricks/data/data/ * /gluster_bricks/vmstore/vmstore/
NODE2 and NODE3 with a similar setup.
Hosted engine was running on node2.
- While moving NODE1 to maintenance mode along with stopping the gluster service as it prompts before, Hosted engine instantly went down.
- I start the gluster service back on node1 and start the hosted engine again and found hosted engine started properly but getting crashed again and again within frames of second after a successful start because HE itself stopping glusterd on node1. (not sure) but cross-verified by checking glusterd status.
*Is it possible to clear pending tasks or not let the HE to stop glusterd on node1?*
*Or we can start the HE using other gluster node?*
https://paste.fedoraproject.org/paste/Qu2tSHuF-~G4GjGmstV6mg
The Hosted Engine storage domain should have the backup-volfile-servers mount options specified, so that even if the node used to mount the gluster volume initially goes down, it can try with an alternate server. Can you check if this is set? ( check /etc/ovirt-hosted-engine/hosted-engine.conf or hosted-engine --get-shared-config mnt_options --type=he_shared) If not set, you can update the mount options for HE domain using hosted-engine --set-shared-config mnt_options="backup-volfile-servers=<server2>:<server3>" --type=he_shared
--
ABHISHEK SAHNI IISER Bhopal
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/26C32RPGG6OF7L...
Hello Sahina, [root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared mnt_options : , type : he_shared [root@node2 ~]# [root@node2 ~]# hosted-engine --set-shared-config mnt_options backup-volfile-servers=192.168.2.1:192.168.2.1 --type=he_shared [root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared mnt_options : backup-volfile-servers=192.168.2.1:192.168.2.1, type : he_shared I have restarted the broker and agent on node2 and try to start HE vm but still it is starting, seems like bricks are still not getting mounted from backup-volfile-servers. Thanks in advance,
Corrections, Hello Sahina, [root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared mnt_options : , type : he_shared [root@node2 ~]# [root@node2 ~]# hosted-engine --set-shared-config mnt_options backup-volfile-servers=192.168.2.2:192.168.2.3 --type=he_shared [root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared mnt_options : backup-volfile-servers=192.168.2.2:192.168.2.3, type : he_shared I have restarted the broker and agent on node2 and try to start HE vm but still it is not starting, seems like bricks are still not getting mounted from backup-volfile-servers. Thanks in advance, On Tue, Dec 4, 2018 at 2:19 PM Abhishek Sahni <abhishek.sahni1991@gmail.com> wrote:
Hello Sahina,
[root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared
mnt_options : , type : he_shared
[root@node2 ~]#
[root@node2 ~]# hosted-engine --set-shared-config mnt_options backup-volfile-servers=192.168.2.1:192.168.2.1 --type=he_shared
[root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared
mnt_options : backup-volfile-servers=192.168.2.1:192.168.2.1, type : he_shared
I have restarted the broker and agent on node2 and try to start HE vm but still it is starting, seems like bricks are still not getting mounted from backup-volfile-servers.
Thanks in advance,
-- ABHISHEK SAHNI Mob : +91-990-701-5143
Hello, I can confirm hosted engine itself is stopping glusterd service on the node1, - Is it possible to remove the pending task from DB - as I am able to halt the engine vm from failing by stopping ovirt-engine service via ssh? - Is it possible to stop gluster service on the cluster ( without having GUI access) so that engine will not manage host gluster service? On Tue, Dec 4, 2018 at 2:22 PM Abhishek Sahni <abhishek.sahni1991@gmail.com> wrote:
Corrections,
Hello Sahina,
[root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared
mnt_options : , type : he_shared
[root@node2 ~]#
[root@node2 ~]# hosted-engine --set-shared-config mnt_options backup-volfile-servers=192.168.2.2:192.168.2.3 --type=he_shared
[root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared
mnt_options : backup-volfile-servers=192.168.2.2:192.168.2.3, type : he_shared
I have restarted the broker and agent on node2 and try to start HE vm but still it is not starting, seems like bricks are still not getting mounted from backup-volfile-servers.
Thanks in advance,
On Tue, Dec 4, 2018 at 2:19 PM Abhishek Sahni < abhishek.sahni1991@gmail.com> wrote:
Hello Sahina,
[root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared
mnt_options : , type : he_shared
[root@node2 ~]#
[root@node2 ~]# hosted-engine --set-shared-config mnt_options backup-volfile-servers=192.168.2.1:192.168.2.1 --type=he_shared
[root@node2 ~]# hosted-engine --get-shared-config mnt_options --type=he_shared
mnt_options : backup-volfile-servers=192.168.2.1:192.168.2.1, type : he_shared
I have restarted the broker and agent on node2 and try to start HE vm but still it is starting, seems like bricks are still not getting mounted from backup-volfile-servers.
Thanks in advance,
--
ABHISHEK SAHNI Mob : +91-990-701-5143
-- ABHISHEK SAHNI Mob : +91-990-701-5143
participants (2)
-
Abhishek Sahni -
Sahina Bose