rebooting an ovirt cluster
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes? Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues. 1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works. 2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start. This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on. So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine? -- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people.
So why is this the default behavior of the ovirt Node distro? On Sun, Feb 3, 2019 at 5:16 PM Strahil <hunter86_bg@yahoo.com> wrote:
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start. This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
If you use fstab - prepare for pain... Systemd mounts are more effective. Here is a sample:
[root@ovirt1 ~]# systemctl cat gluster_bricks-engine.mount # /etc/systemd/system/gluster_bricks-engine.mount [Unit] Description=Mount glusterfs brick - ENGINE Requires = vdo.service After = vdo.service Before = glusterd.service Conflicts = umount.target
[Mount] What=/dev/mapper/gluster_vg_md0-gluster_lv_engine Where=/gluster_bricks/engine Type=xfs Options=inode64,noatime,nodiratime
[Install] WantedBy=glusterd.service
[root@ovirt1 ~]# systemctl cat glusterd.service # /etc/systemd/system/glusterd.service [Unit] Description=GlusterFS, a clustered file-system server Requires=rpcbind.service gluster_bricks-engine.mount gluster_bricks-data.mount After=network.target rpcbind.service gluster_bricks-engine.mount Before=network-online.target
[Service] Type=forking PIDFile=/var/run/glusterd.pid LimitNOFILE=65536 Environment="LOG_LEVEL=INFO" EnvironmentFile=-/etc/sysconfig/glusterd ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVELKillMode=process SuccessExitStatus=15
[Install] WantedBy=multi-user.target
# /etc/systemd/system/glusterd.service.d/99-cpu.conf [Service] CPUAccounting=yes Slice=glusterfs.slice
Note : Some of the 'After=' and 'Requires=' entries were removed during copy-pasting.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
Expected -everything to be up and running. Root cause , the system's fstab generator starts after cluster tries to start the bricks - and of course fails. Then everything on the chain fails.
Just use systemd's mount entries ( I have added automount also) and you won't have such issues.
Best Regards, Strahil Nikolov
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people.
On Sat, Feb 2, 2019 at 7:32 PM feral <blistovmhz@gmail.com> wrote:
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes?
Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues.
1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works.
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start.
This is not required at all. Are you sure that your cluster is not set in global maintenance mode? Can you please share /var/log/ovirt-hosted-engine-ha/agent.log and broker.log from your hosts?
This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIADNRZRXTPTRG...
I think I found the answer to glusterd not starting. https://bugzilla.redhat.com/show_bug.cgi?id=1472267 Apparently the version of gluster (3.12.15) that comes packaged with ovirt-node 4.2.8 has a known issue where gluster tries to come up before networking, fails, and crashes. This was fixed in gluster 3.13.0 (apparently). Do devs paruse this list? Any chance someone who can update the gluster package might read this? On Mon, Feb 4, 2019 at 2:38 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sat, Feb 2, 2019 at 7:32 PM feral <blistovmhz@gmail.com> wrote:
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes?
Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues.
1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works.
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start.
This is not required at all. Are you sure that your cluster is not set in global maintenance mode? Can you please share /var/log/ovirt-hosted-engine-ha/agent.log and broker.log from your hosts?
This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIADNRZRXTPTRG...
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people.
On Mon, Feb 4, 2019 at 4:15 PM feral <blistovmhz@gmail.com> wrote:
I think I found the answer to glusterd not starting. https://bugzilla.redhat.com/show_bug.cgi?id=1472267
Apparently the version of gluster (3.12.15) that comes packaged with ovirt-node 4.2.8 has a known issue where gluster tries to come up before networking, fails, and crashes. This was fixed in gluster 3.13.0 (apparently). Do devs paruse this list?
Yes :)
Any chance someone who can update the gluster package might read this?
+Sahina might be able to help The developers list is https://lists.ovirt.org/archives/list/devel@ovirt.org/
On Mon, Feb 4, 2019 at 2:38 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sat, Feb 2, 2019 at 7:32 PM feral <blistovmhz@gmail.com> wrote:
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes?
Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues.
1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works.
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start.
This is not required at all. Are you sure that your cluster is not set in global maintenance mode? Can you please share /var/log/ovirt-hosted-engine-ha/agent.log and broker.log from your hosts?
This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIADNRZRXTPTRG...
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WNN3PJFFP4VU5Y...
-- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
Il giorno mar 5 feb 2019 alle ore 02:53 Greg Sheremeta <gshereme@redhat.com> ha scritto:
On Mon, Feb 4, 2019 at 4:15 PM feral <blistovmhz@gmail.com> wrote:
I think I found the answer to glusterd not starting. https://bugzilla.redhat.com/show_bug.cgi?id=1472267
Apparently the version of gluster (3.12.15) that comes packaged with ovirt-node 4.2.8 has a known issue where gluster tries to come up before networking, fails, and crashes. This was fixed in gluster 3.13.0 (apparently).
May I suggest to upgrade to 4.3.0? It ships Gluster 5 which should include the needed fixes.
Do devs paruse this list?
Yes :)
Any chance someone who can update the gluster package might read this?
+Sahina might be able to help The developers list is https://lists.ovirt.org/archives/list/devel@ovirt.org/
On Mon, Feb 4, 2019 at 2:38 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sat, Feb 2, 2019 at 7:32 PM feral <blistovmhz@gmail.com> wrote:
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes?
Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues.
1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works.
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start.
This is not required at all. Are you sure that your cluster is not set in global maintenance mode? Can you please share /var/log/ovirt-hosted-engine-ha/agent.log and broker.log from your hosts?
This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIADNRZRXTPTRG...
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WNN3PJFFP4VU5Y...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
-- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig>
ha! I was all worried I was completely tarding out as I've never heard of 4.3... Released two days ago... On Mon, Feb 4, 2019 at 11:53 PM Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il giorno mar 5 feb 2019 alle ore 02:53 Greg Sheremeta < gshereme@redhat.com> ha scritto:
On Mon, Feb 4, 2019 at 4:15 PM feral <blistovmhz@gmail.com> wrote:
I think I found the answer to glusterd not starting. https://bugzilla.redhat.com/show_bug.cgi?id=1472267
Apparently the version of gluster (3.12.15) that comes packaged with ovirt-node 4.2.8 has a known issue where gluster tries to come up before networking, fails, and crashes. This was fixed in gluster 3.13.0 (apparently).
May I suggest to upgrade to 4.3.0? It ships Gluster 5 which should include the needed fixes.
Do devs paruse this list?
Yes :)
Any chance someone who can update the gluster package might read this?
+Sahina might be able to help The developers list is https://lists.ovirt.org/archives/list/devel@ovirt.org/
On Mon, Feb 4, 2019 at 2:38 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sat, Feb 2, 2019 at 7:32 PM feral <blistovmhz@gmail.com> wrote:
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes?
Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues.
1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works.
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start.
This is not required at all. Are you sure that your cluster is not set in global maintenance mode? Can you please share /var/log/ovirt-hosted-engine-ha/agent.log and broker.log from your hosts?
This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIADNRZRXTPTRG...
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WNN3PJFFP4VU5Y...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com <https://red.ht/sig>
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people.
On Tue, Feb 5, 2019 at 7:23 AM Greg Sheremeta <gshereme@redhat.com> wrote:
On Mon, Feb 4, 2019 at 4:15 PM feral <blistovmhz@gmail.com> wrote:
I think I found the answer to glusterd not starting. https://bugzilla.redhat.com/show_bug.cgi?id=1472267
Apparently the version of gluster (3.12.15) that comes packaged with ovirt-node 4.2.8 has a known issue where gluster tries to come up before networking, fails, and crashes. This was fixed in gluster 3.13.0 (apparently). Do devs paruse this list?
Yes :)
Any chance someone who can update the gluster package might read this?
+Sahina might be able to help The developers list is https://lists.ovirt.org/archives/list/devel@ovirt.org/
On 4.2, we're stuck with glusterfs 3.12 due to dependency on gluster-gnfs. The bug you refer to is hit only when a hostname changes or one of the network interfaces is down and brick path cannot be resolved. What's the error in glusterd.log for the failure to start?
On Mon, Feb 4, 2019 at 2:38 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sat, Feb 2, 2019 at 7:32 PM feral <blistovmhz@gmail.com> wrote:
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes?
Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues.
1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works.
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start.
This is not required at all. Are you sure that your cluster is not set in global maintenance mode? Can you please share /var/log/ovirt-hosted-engine-ha/agent.log and broker.log from your hosts?
This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIADNRZRXTPTRG...
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WNN3PJFFP4VU5Y...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
Incidentally, at least so far, ovirt-node-4.3 is going much better (for the installation anyway). The documentation for hyperconverged does not mention anything about setting up cockpit though, so you have to manually enable and start it, and if you're not using ipv6, you have to modify the cockpit.socket to forcibly enable ipv4. On Tue, Feb 5, 2019 at 8:27 AM Sahina Bose <sabose@redhat.com> wrote:
On Tue, Feb 5, 2019 at 7:23 AM Greg Sheremeta <gshereme@redhat.com> wrote:
On Mon, Feb 4, 2019 at 4:15 PM feral <blistovmhz@gmail.com> wrote:
I think I found the answer to glusterd not starting. https://bugzilla.redhat.com/show_bug.cgi?id=1472267
Apparently the version of gluster (3.12.15) that comes packaged with ovirt-node 4.2.8 has a known issue where gluster tries to come up before networking, fails, and crashes. This was fixed in gluster 3.13.0 (apparently). Do devs paruse this list?
Yes :)
Any chance someone who can update the gluster package might read this?
+Sahina might be able to help The developers list is https://lists.ovirt.org/archives/list/devel@ovirt.org/
On 4.2, we're stuck with glusterfs 3.12 due to dependency on gluster-gnfs.
The bug you refer to is hit only when a hostname changes or one of the network interfaces is down and brick path cannot be resolved. What's the error in glusterd.log for the failure to start?
On Mon, Feb 4, 2019 at 2:38 AM Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sat, Feb 2, 2019 at 7:32 PM feral <blistovmhz@gmail.com> wrote:
How is an oVirt hyperconverged cluster supposed to come back to life after a power outage to all 3 nodes?
Running ovirt-node (ovirt-node-ng-installer-4.2.0-2019013006.el7.iso) to get things going, but I've run into multiple issues.
1. During the gluster setup, the volume sizes I specify, are not reflected in the deployment configuration. The auto-populated values are used every time. I manually hacked on the config to get the volume sizes correct. I also noticed if I create the deployment config with "sdb" by accident, but click back and change it to "vdb", again, the changes are not reflected in the config. My deployment config does seem to work. All volumes are created (though the xfs options used don't make sense as you end up with stripe sizes that aren't a multiple of the block size). Once gluster is deployed, I deploy the hosted engine, and everything works.
2. Reboot all nodes. I was testing for power outage response. All nodes come up, but glusterd is not running (seems to have failed for some reason). I can manually restart glusterd on all nodes and it comes up and starts communicating normally. However, the engine does not come online. So I figure out where it last lived, and try to start it manually through the web interface. This fails because vdsm-ovirtmgmt is not up. I figured out the correct way to start up the engine would be through the cli via hosted-engine --vm-start.
This is not required at all. Are you sure that your cluster is not set in global maintenance mode? Can you please share /var/log/ovirt-hosted-engine-ha/agent.log and broker.log from your hosts?
This does work, but it takes a very long time, and it usually starts up on any node other than the one I told it to start on.
So I guess two (or three) questions. What is the expected operation after a full cluster reboot (ie: in the event of a power failure)? Why doesn't the engine start automatically, and what might be causing glusterd to fail, when it can be restarted manually and works fine?
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RIADNRZRXTPTRG...
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people. _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/WNN3PJFFP4VU5Y...
--
GREG SHEREMETA
SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
Red Hat NA
gshereme@redhat.com IRC: gshereme <https://red.ht/sig>
-- _____ Fact: 1. Ninjas are mammals. 2. Ninjas fight ALL the time. 3. The purpose of the ninja is to flip out and kill people.
participants (6)
-
feral -
Greg Sheremeta -
Sahina Bose -
Sandro Bonazzola -
Simone Tiraboschi -
Strahil