iscsi data domain when engine is down
We have an ovirt 4.1 cluster with an iSCSI data domain. If I shut down the entire cluster and just boot the hosts, none of the hosts login to their iSCSI sessions until the engine comes up. Without logging into the sessions, sanlock doesn't obtain any leases and obviously none of the VMs start. I'm sure there's something I'm missing, as it looks like it should be possible to run a hosted engine on a cluster using an iSCSI data domain. Any pointers or suggestions would be greatly appreciated. Many thanks, Devin
Hi, what kind of setup you have? hosted engine just runs on nfs or gluster afaik. regards, 2017-03-10 15:22 GMT-03:00 Devin A. Bougie <devin.bougie@cornell.edu>:
We have an ovirt 4.1 cluster with an iSCSI data domain. If I shut down the entire cluster and just boot the hosts, none of the hosts login to their iSCSI sessions until the engine comes up. Without logging into the sessions, sanlock doesn't obtain any leases and obviously none of the VMs start.
I'm sure there's something I'm missing, as it looks like it should be possible to run a hosted engine on a cluster using an iSCSI data domain.
Any pointers or suggestions would be greatly appreciated.
Many thanks, Devin _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mar 10, 2017, at 1:28 PM, Juan Pablo <pablo.localhost@gmail.com> wrote:
Hi, what kind of setup you have? hosted engine just runs on nfs or gluster afaik.
Thanks for replying, Juan. I was under the impression that the hosted engine would run on an iSCSI data domain, based on http://www.ovirt.org/develop/release-management/features/engine/self-hosted-... and the fact that "hosted-engine --deploy" does give you the option to choose iscsi storage (but only one path, as far as I can tell). I certainly could manage the iSCSI sessions outside of ovirt / vdsm, but wasn't sure if that would cause problems or if that was all that's needed to allow the hosted engine to boot automatically on an iSCSI data domain. Thanks again, Devin
2017-03-10 15:22 GMT-03:00 Devin A. Bougie <devin.bougie@cornell.edu>: We have an ovirt 4.1 cluster with an iSCSI data domain. If I shut down the entire cluster and just boot the hosts, none of the hosts login to their iSCSI sessions until the engine comes up. Without logging into the sessions, sanlock doesn't obtain any leases and obviously none of the VMs start.
I'm sure there's something I'm missing, as it looks like it should be possible to run a hosted engine on a cluster using an iSCSI data domain.
Any pointers or suggestions would be greatly appreciated.
Many thanks, Devin _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Once upon a time, Devin A. Bougie <devin.bougie@cornell.edu> said:
Thanks for replying, Juan. I was under the impression that the hosted engine would run on an iSCSI data domain, based on http://www.ovirt.org/develop/release-management/features/engine/self-hosted-... and the fact that "hosted-engine --deploy" does give you the option to choose iscsi storage (but only one path, as far as I can tell).
Hosted engine runs fine on iSCSI since oVirt 3.5. It needs a separate target from VM storage, but then that access is managed by the hosted engine HA system. If all the engine hosts are shut down together, it will take a bit after boot for the HA system to converge and try to bring the engine back online (including logging in to the engine iSCSI LUN). You can force this on one host by running "hosted-engine --vm-start". -- Chris Adams <cma@cmadams.net>
"Hosted engine runs fine on iSCSI since oVirt 3.5.", I have similar configuration and also I have got problem with resolve the hosted-engine storage by vdsm but the cause is that I do know how to edit iSCSI settings. I suspect that there is only one path to target not four :( Devin what is the output of "hosted-engine --vm-status"? 2017-03-11 16:59 GMT+01:00 Chris Adams <cma@cmadams.net>:
Thanks for replying, Juan. I was under the impression that the hosted engine would run on an iSCSI data domain, based on http://www.ovirt.org/develop/release-management/features/ engine/self-hosted-engine-iscsi-support/ and the fact that "hosted-engine --deploy" does give you the option to choose iscsi storage (but only one
Once upon a time, Devin A. Bougie <devin.bougie@cornell.edu> said: path, as far as I can tell).
Hosted engine runs fine on iSCSI since oVirt 3.5. It needs a separate target from VM storage, but then that access is managed by the hosted engine HA system.
If all the engine hosts are shut down together, it will take a bit after boot for the HA system to converge and try to bring the engine back online (including logging in to the engine iSCSI LUN). You can force this on one host by running "hosted-engine --vm-start".
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mar 11, 2017, at 10:59 AM, Chris Adams <cma@cmadams.net> wrote:
Hosted engine runs fine on iSCSI since oVirt 3.5. It needs a separate target from VM storage, but then that access is managed by the hosted engine HA system.
Thanks so much, Chris. It sounds like that is exactly what I was missing. It would be great to know how to add multiple paths to the hosted engine's iSCSI target, but hopefully I can figure that out once I have things up and running. Thanks again, Devin
If all the engine hosts are shut down together, it will take a bit after boot for the HA system to converge and try to bring the engine back online (including logging in to the engine iSCSI LUN). You can force this on one host by running "hosted-engine --vm-start".
-- Chris Adams <cma@cmadams.net>
Once upon a time, Devin A. Bougie <devin.bougie@cornell.edu> said:
On Mar 11, 2017, at 10:59 AM, Chris Adams <cma@cmadams.net> wrote:
Hosted engine runs fine on iSCSI since oVirt 3.5. It needs a separate target from VM storage, but then that access is managed by the hosted engine HA system.
Thanks so much, Chris. It sounds like that is exactly what I was missing.
It would be great to know how to add multiple paths to the hosted engine's iSCSI target, but hopefully I can figure that out once I have things up and running.
oVirt doesn't currently support adding paths to an existing storage domain; they all have to be selected when the domain is created. Since the hosted engine setup doesn't support that, there's no way to add additional paths after the fact. I think adding paths to a regular storage domain is something that is being looked at (believe I read that on the list), so maybe if that gets added, support for adding to the hosted engine domain will be added as well. I have a script that gets run out of cron periodically that looks at sessions and configured paths and tries to add them if necessary. I manually created the config for the additional paths with iscsiadm. It works, although I'm not sure that the hosted engine HA is really "happy" with what I did. :) -- Chris Adams <cma@cmadams.net>
OK, but what your script do, only add paths by the iscsiadm command, but the question is if hosted-engine can see it. I do not know how to add an extra path, for example, when I congirured hosted-engine during installation there was only one path in the target, but now there are four. So how can I verify how many paths are now, and how eventually change it. 2017-03-12 0:32 GMT+01:00 Chris Adams <cma@cmadams.net>:
On Mar 11, 2017, at 10:59 AM, Chris Adams <cma@cmadams.net> wrote:
Hosted engine runs fine on iSCSI since oVirt 3.5. It needs a separate target from VM storage, but then that access is managed by the hosted engine HA system.
Thanks so much, Chris. It sounds like that is exactly what I was missing.
It would be great to know how to add multiple paths to the hosted engine's iSCSI target, but hopefully I can figure that out once I have
Once upon a time, Devin A. Bougie <devin.bougie@cornell.edu> said: things up and running.
oVirt doesn't currently support adding paths to an existing storage domain; they all have to be selected when the domain is created. Since the hosted engine setup doesn't support that, there's no way to add additional paths after the fact. I think adding paths to a regular storage domain is something that is being looked at (believe I read that on the list), so maybe if that gets added, support for adding to the hosted engine domain will be added as well.
I have a script that gets run out of cron periodically that looks at sessions and configured paths and tries to add them if necessary. I manually created the config for the additional paths with iscsiadm. It works, although I'm not sure that the hosted engine HA is really "happy" with what I did. :)
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Once upon a time, Marcin Kruk <askifyouneed@gmail.com> said:
OK, but what your script do, only add paths by the iscsiadm command, but the question is if hosted-engine can see it. I do not know how to add an extra path, for example, when I congirured hosted-engine during installation there was only one path in the target, but now there are four. So how can I verify how many paths are now, and how eventually change it.
oVirt access iSCSI storage through multipath devices, so adding a path to the multipath device will work. Adding an additional path with iscsiadm causes multipathd to recognize it; you can verify that with "multipath -ll". -- Chris Adams <cma@cmadams.net>
Are you 1000% sure? Because during startup vmsd the comman iscsiadm <parameters> <target> is executed, In my opinion vmsd should rely on /var/lib/iscsi settings. 2017-03-12 15:02 GMT+01:00 Chris Adams <cma@cmadams.net>:
Once upon a time, Marcin Kruk <askifyouneed@gmail.com> said:
OK, but what your script do, only add paths by the iscsiadm command, but the question is if hosted-engine can see it. I do not know how to add an extra path, for example, when I congirured hosted-engine during installation there was only one path in the target, but now there are four. So how can I verify how many paths are now, and how eventually change it.
oVirt access iSCSI storage through multipath devices, so adding a path to the multipath device will work. Adding an additional path with iscsiadm causes multipathd to recognize it; you can verify that with "multipath -ll". -- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
It's easy enough to verify; just SSH in to the physical machine running the hosted engine and look. You can see the qemu command line, follow the links to the block device, then look at the LVM physical volume and see that it is a multipath device. Once upon a time, Marcin Kruk <askifyouneed@gmail.com> said:
Are you 1000% sure? Because during startup vmsd the comman iscsiadm <parameters> <target> is executed, In my opinion vmsd should rely on /var/lib/iscsi settings.
2017-03-12 15:02 GMT+01:00 Chris Adams <cma@cmadams.net>:
Once upon a time, Marcin Kruk <askifyouneed@gmail.com> said:
OK, but what your script do, only add paths by the iscsiadm command, but the question is if hosted-engine can see it. I do not know how to add an extra path, for example, when I congirured hosted-engine during installation there was only one path in the target, but now there are four. So how can I verify how many paths are now, and how eventually change it.
oVirt access iSCSI storage through multipath devices, so adding a path to the multipath device will work. Adding an additional path with iscsiadm causes multipathd to recognize it; you can verify that with "multipath -ll". -- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Chris Adams <cma@cmadams.net>
but VDSM when it starts, it runs command: /usr/sbin/iscsiadm -m node -T iqn.1984-05.com... -I default -p <IP>:3260,1 -l and when the above IP address is unavailable (I unpluged array cable for testing HA), it hangs and cannot read vd.conf file, so I do not know from where VDSM get above IP addres and why it tires to logon to iSCSI share itself. Because iSCSI daemon does it correctly and do not create sessions where the link is not active. 2017-03-13 16:15 GMT+01:00 Chris Adams <cma@cmadams.net>:
It's easy enough to verify; just SSH in to the physical machine running the hosted engine and look. You can see the qemu command line, follow the links to the block device, then look at the LVM physical volume and see that it is a multipath device.
Once upon a time, Marcin Kruk <askifyouneed@gmail.com> said:
Are you 1000% sure? Because during startup vmsd the comman iscsiadm <parameters> <target> is executed, In my opinion vmsd should rely on /var/lib/iscsi settings.
2017-03-12 15:02 GMT+01:00 Chris Adams <cma@cmadams.net>:
Once upon a time, Marcin Kruk <askifyouneed@gmail.com> said:
OK, but what your script do, only add paths by the iscsiadm command, but the question is if hosted-engine can see it. I do not know how to add an extra path, for example, when I congirured hosted-engine during installation there was only one path in the target, but now there are four. So how can I verify how many paths are now, and how eventually change it.
oVirt access iSCSI storage through multipath devices, so adding a path to the multipath device will work. Adding an additional path with iscsiadm causes multipathd to recognize it; you can verify that with "multipath -ll". -- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (4)
-
Chris Adams -
Devin A. Bougie -
Juan Pablo -
Marcin Kruk