Recognize HE iSCSI volume size change
I'm testing upgrading an oVirt 3.5 setup, and I have run into a problem when going from 3.5 to 3.6 on a physical machine configured for the hosted engine. I upgraded the engine itself okay, but when I upgraded the first physical machine, it cannot be re-activated; it gets an error connecting to the storage domain. Checking the logs, it looks like it is looping trying to create a new LV in the HE VG. I assume this is for moving the HE config to the shared storage? It is failing because it is trying to create a 1G LV, but the VG only has 512M free space. I extended the iSCSI volume, but there doesn't appear to be anyway to get the HE nodes to recognize this; they both still see the original size, no matter what I try. Is there a way to get them to see the larger PV, so the new LV(s) can be created? -- Chris Adams <cma@cmadams.net>
Hi Chris. We added this feature to newer versions of oVirt (see the feature page[1]). The easiest way to work around this problem might be to add an additional LUN to this domain if you are able to do it. If not, it looks like you would need to manually reconnect the host to the domain, to a pvresize to the new size. I am not sure if any engine DB updates will also be required. Nir and Fred worked on this feature and might be able to assist you further. [1] https://www.ovirt.org/develop/release-management/features/storage/lun-resize... On Fri, Feb 24, 2017 at 9:00 AM, Chris Adams <cma@cmadams.net> wrote:
I'm testing upgrading an oVirt 3.5 setup, and I have run into a problem when going from 3.5 to 3.6 on a physical machine configured for the hosted engine. I upgraded the engine itself okay, but when I upgraded the first physical machine, it cannot be re-activated; it gets an error connecting to the storage domain.
Checking the logs, it looks like it is looping trying to create a new LV in the HE VG. I assume this is for moving the HE config to the shared storage? It is failing because it is trying to create a 1G LV, but the VG only has 512M free space.
I extended the iSCSI volume, but there doesn't appear to be anyway to get the HE nodes to recognize this; they both still see the original size, no matter what I try. Is there a way to get them to see the larger PV, so the new LV(s) can be created?
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Adam Litke
I did see that page, but... I can't get there from here. I can't get upgraded from 3.5 until I get past this problem, and on 3.5, the hosted engine storage domain isn't included in the normal UI at all. I think I did get this working though; both servers had kernel messages that they saw the LUN resize, but they didn't actually change the block device size to reflect that. After rebooting each server (separately), lsblk showed the new size on both, and a manual pvresize on both shows the increased VG size. On to testing 3.6 upgrade again! Once upon a time, Adam Litke <alitke@redhat.com> said:
Hi Chris. We added this feature to newer versions of oVirt (see the feature page[1]). The easiest way to work around this problem might be to add an additional LUN to this domain if you are able to do it. If not, it looks like you would need to manually reconnect the host to the domain, to a pvresize to the new size. I am not sure if any engine DB updates will also be required. Nir and Fred worked on this feature and might be able to assist you further.
[1] https://www.ovirt.org/develop/release-management/features/storage/lun-resize...
On Fri, Feb 24, 2017 at 9:00 AM, Chris Adams <cma@cmadams.net> wrote:
I'm testing upgrading an oVirt 3.5 setup, and I have run into a problem when going from 3.5 to 3.6 on a physical machine configured for the hosted engine. I upgraded the engine itself okay, but when I upgraded the first physical machine, it cannot be re-activated; it gets an error connecting to the storage domain.
Checking the logs, it looks like it is looping trying to create a new LV in the HE VG. I assume this is for moving the HE config to the shared storage? It is failing because it is trying to create a 1G LV, but the VG only has 512M free space.
I extended the iSCSI volume, but there doesn't appear to be anyway to get the HE nodes to recognize this; they both still see the original size, no matter what I try. Is there a way to get them to see the larger PV, so the new LV(s) can be created?
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Adam Litke
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Chris Adams <cma@cmadams.net>
Chris, Sorry to chime in late, but this might help you (or someone with a similar upgrade situation) in the future. The "rescan-scsi-bus.sh" script from the sg3-utils package can handle the update for the size increase without the reboot. Just need to run with the "-s" param to look for resized LUNs like the below (run from a CentOS7.3 server): ``` # rescan-scsi-bus.sh -s Scanning SCSI subsystem for new devices *Searching for resized LUNs * 0 new or changed device(s) found. 0 remapped or resized device(s) found. 0 device(s) removed. ``` Cheers Jeff On Fri, Feb 24, 2017 at 9:20 AM, Chris Adams <cma@cmadams.net> wrote:
I did see that page, but... I can't get there from here. I can't get upgraded from 3.5 until I get past this problem, and on 3.5, the hosted engine storage domain isn't included in the normal UI at all.
I think I did get this working though; both servers had kernel messages that they saw the LUN resize, but they didn't actually change the block device size to reflect that. After rebooting each server (separately), lsblk showed the new size on both, and a manual pvresize on both shows the increased VG size.
On to testing 3.6 upgrade again!
Once upon a time, Adam Litke <alitke@redhat.com> said:
Hi Chris. We added this feature to newer versions of oVirt (see the feature page[1]). The easiest way to work around this problem might be to add an additional LUN to this domain if you are able to do it. If not, it looks like you would need to manually reconnect the host to the domain, to a pvresize to the new size. I am not sure if any engine DB updates will also be required. Nir and Fred worked on this feature and might be able to assist you further.
[1] https://www.ovirt.org/develop/release-management/features/ storage/lun-resize/
On Fri, Feb 24, 2017 at 9:00 AM, Chris Adams <cma@cmadams.net> wrote:
I'm testing upgrading an oVirt 3.5 setup, and I have run into a problem when going from 3.5 to 3.6 on a physical machine configured for the hosted engine. I upgraded the engine itself okay, but when I upgraded the first physical machine, it cannot be re-activated; it gets an error connecting to the storage domain.
Checking the logs, it looks like it is looping trying to create a new LV in the HE VG. I assume this is for moving the HE config to the shared storage? It is failing because it is trying to create a 1G LV, but the VG only has 512M free space.
I extended the iSCSI volume, but there doesn't appear to be anyway to get the HE nodes to recognize this; they both still see the original size, no matter what I try. Is there a way to get them to see the larger PV, so the new LV(s) can be created?
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Adam Litke
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Fri, Feb 24, 2017 at 7:36 PM, Jeff Burns <admiraljkb@gmail.com> wrote:
Chris,
Sorry to chime in late, but this might help you (or someone with a similar upgrade situation) in the future. The "rescan-scsi-bus.sh" script from the sg3-utils package can handle the update for the size increase without the reboot. Just need to run with the "-s" param to look for resized LUNs like the below (run from a CentOS7.3 server):
This is not enough, you need to resize the multiapth mapping on all hosts, resize the pv using the LUN (must be done by the SPM), and invalidate vdsm lvm cache on all hosts, so they go to storage and see the new size of the pv. Nir
``` # rescan-scsi-bus.sh -s Scanning SCSI subsystem for new devices Searching for resized LUNs 0 new or changed device(s) found. 0 remapped or resized device(s) found. 0 device(s) removed. ```
Cheers Jeff
On Fri, Feb 24, 2017 at 9:20 AM, Chris Adams <cma@cmadams.net> wrote:
I did see that page, but... I can't get there from here. I can't get upgraded from 3.5 until I get past this problem, and on 3.5, the hosted engine storage domain isn't included in the normal UI at all.
I think I did get this working though; both servers had kernel messages that they saw the LUN resize, but they didn't actually change the block device size to reflect that. After rebooting each server (separately), lsblk showed the new size on both, and a manual pvresize on both shows the increased VG size.
On to testing 3.6 upgrade again!
Once upon a time, Adam Litke <alitke@redhat.com> said:
Hi Chris. We added this feature to newer versions of oVirt (see the feature page[1]). The easiest way to work around this problem might be to add an additional LUN to this domain if you are able to do it. If not, it looks like you would need to manually reconnect the host to the domain, to a pvresize to the new size. I am not sure if any engine DB updates will also be required. Nir and Fred worked on this feature and might be able to assist you further.
[1]
https://www.ovirt.org/develop/release-management/features/storage/lun-resize...
On Fri, Feb 24, 2017 at 9:00 AM, Chris Adams <cma@cmadams.net> wrote:
I'm testing upgrading an oVirt 3.5 setup, and I have run into a problem when going from 3.5 to 3.6 on a physical machine configured for the hosted engine. I upgraded the engine itself okay, but when I upgraded the first physical machine, it cannot be re-activated; it gets an error connecting to the storage domain.
Checking the logs, it looks like it is looping trying to create a new LV in the HE VG. I assume this is for moving the HE config to the shared storage? It is failing because it is trying to create a 1G LV, but the VG only has 512M free space.
I extended the iSCSI volume, but there doesn't appear to be anyway to get the HE nodes to recognize this; they both still see the original size, no matter what I try. Is there a way to get them to see the larger PV, so the new LV(s) can be created?
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Adam Litke
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Nir, Ah, I see. Makes sense, and thanks for chiming in with the correction so quickly. Bummer a quick rescan won't work for this scenario. Thanks, Jeff On Fri, Feb 24, 2017 at 11:50 AM, Nir Soffer <nsoffer@redhat.com> wrote:
Chris,
Sorry to chime in late, but this might help you (or someone with a similar upgrade situation) in the future. The "rescan-scsi-bus.sh" script from
sg3-utils package can handle the update for the size increase without the reboot. Just need to run with the "-s" param to look for resized LUNs
On Fri, Feb 24, 2017 at 7:36 PM, Jeff Burns <admiraljkb@gmail.com> wrote: the like
the below (run from a CentOS7.3 server):
This is not enough, you need to resize the multiapth mapping on all hosts, resize the pv using the LUN (must be done by the SPM), and invalidate vdsm lvm cache on all hosts, so they go to storage and see the new size of the pv.
Nir
``` # rescan-scsi-bus.sh -s Scanning SCSI subsystem for new devices Searching for resized LUNs 0 new or changed device(s) found. 0 remapped or resized device(s) found. 0 device(s) removed. ```
Cheers Jeff
On Fri, Feb 24, 2017 at 9:20 AM, Chris Adams <cma@cmadams.net> wrote:
I did see that page, but... I can't get there from here. I can't get upgraded from 3.5 until I get past this problem, and on 3.5, the hosted engine storage domain isn't included in the normal UI at all.
I think I did get this working though; both servers had kernel messages that they saw the LUN resize, but they didn't actually change the block device size to reflect that. After rebooting each server (separately), lsblk showed the new size on both, and a manual pvresize on both shows the increased VG size.
On to testing 3.6 upgrade again!
Once upon a time, Adam Litke <alitke@redhat.com> said:
Hi Chris. We added this feature to newer versions of oVirt (see the feature page[1]). The easiest way to work around this problem might
be
to add an additional LUN to this domain if you are able to do it. If not, it looks like you would need to manually reconnect the host to the domain, to a pvresize to the new size. I am not sure if any engine DB updates will also be required. Nir and Fred worked on this feature and might be able to assist you further.
[1]
https://www.ovirt.org/develop/release-management/features/ storage/lun-resize/
On Fri, Feb 24, 2017 at 9:00 AM, Chris Adams <cma@cmadams.net> wrote:
I'm testing upgrading an oVirt 3.5 setup, and I have run into a problem when going from 3.5 to 3.6 on a physical machine configured for the hosted engine. I upgraded the engine itself okay, but when I upgraded the first physical machine, it cannot be re-activated; it gets an error connecting to the storage domain.
Checking the logs, it looks like it is looping trying to create a new LV in the HE VG. I assume this is for moving the HE config to the shared storage? It is failing because it is trying to create a 1G LV, but the VG only has 512M free space.
I extended the iSCSI volume, but there doesn't appear to be anyway to get the HE nodes to recognize this; they both still see the original size, no matter what I try. Is there a way to get them to see the larger PV, so the new LV(s) can be created?
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Adam Litke
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Once upon a time, Nir Soffer <nsoffer@redhat.com> said:
This is not enough, you need to resize the multiapth mapping on all hosts, resize the pv using the LUN (must be done by the SPM), and invalidate vdsm lvm cache on all hosts, so they go to storage and see the new size of the pv.
I did all of this except invaliding the vdsm lvm cache - how would I do that? I will say just doing up to the pvresize worked in my test environment (but I might have just been lucky). -- Chris Adams <cma@cmadams.net>
On Fri, Feb 24, 2017 at 8:33 PM, Chris Adams <cma@cmadams.net> wrote:
Once upon a time, Nir Soffer <nsoffer@redhat.com> said:
This is not enough, you need to resize the multiapth mapping on all hosts, resize the pv using the LUN (must be done by the SPM), and invalidate vdsm lvm cache on all hosts, so they go to storage and see the new size of the pv.
I did all of this except invaliding the vdsm lvm cache - how would I do that?
The easiest way is to restart vdsm on all hosts. I think the complete flow on 3.5 can work like this: 1. stop ovirt-engine, so it will not try to restart vdsm on any host 2. stop vdsm on all hosts 3. rescan scsi bus, resizing luns on all hosts 4. pvresize the pv from one on the host 5. start vdsm on all hosts 6. start ovirt-engine This will allow resize while the storage is online and vms are running. If you are ok with some downtime, you can simplify: 1. stop vms using this storage domain 2. put the storage domain to maintenance 3. rescan scsi bug, resizing luns 4. pvresize the pv 5. active the storage domain
I will say just doing up to the pvresize worked in my test environment (but I might have just been lucky).
Probably.
-- Chris Adams <cma@cmadams.net> _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Once upon a time, Nir Soffer <nsoffer@redhat.com> said:
I think the complete flow on 3.5 can work like this:
1. stop ovirt-engine, so it will not try to restart vdsm on any host 2. stop vdsm on all hosts 3. rescan scsi bus, resizing luns on all hosts 4. pvresize the pv from one on the host 5. start vdsm on all hosts 6. start ovirt-engine
This will allow resize while the storage is online and vms are running.
Thanks! -- Chris Adams <cma@cmadams.net>
participants (4)
-
Adam Litke -
Chris Adams -
Jeff Burns -
Nir Soffer