- Users - oVirt List Archives

Re: Does cluster upgrade wait for heal before proceeding to next host?
by Strahil 17 Sep '19

17 Sep '19

As far as I know, The UI allows the administrator to override the gluster status, although I always use that protection. In my opinion gluster health (especially in my setup - replica2 arbiter1) is very important. Best Regards, Strahil NikolovOn Sep 17, 2019 00:47, Jayme <jaymef(a)gmail.com> wrote: > > Strahil, yes this is similar to the approach I have used to upgrade my hci cluster. This thread is in regards to the ui cluster upgrade procedure. It fails to update after the first host is rebooted because the gluster volume is still in a healing state and the attempt to set the second host to maintenance fails because quorum is not met. > > On Mon, Sep 16, 2019 at 1:41 PM Strahil <hunter86_bg(a)yahoo.com> wrote: >> >> I keep reading this chain and I still don't get what/who should wait for the cluster to heal... >> Is there some kind of built-in autopatching feature? >> >> Here is my approach: >> 1. Set global maintenance >> 2. Power off the engine >> 3. Create a gluster snapshot of the engine's volume >> 4. Power on engine manually >> 5. Check engine status >> 6. Upgrade engine >> 7. Upgrade engine's OS >> 8. Reboot engine and check health >> 9. Remove global maintenance >> 10. Set a host into local maintenance (evacuate all VMs) >> 11. Use UI to patch the host (enable autoreboot) >> 12. When host is up - login and check gluster volumes' heal status >> 13. Remove maintenance for the host and repeate for the rest of the cluster. >> >> I realize that for large clusters this approach is tedious and an automatic approach can be scripted. >> >> Best Regards, >> Strahil Nikolov >> >> On Sep 16, 2019 11:02, Kaustav Majumder <kmajumde(a)redhat.com> wrote: >>> >>> Hi Jayme, >>> It would be great if you could raise a bug regarding the same. >>> >>> On Wed, Sep 11, 2019 at 5:05 PM Jayme <jaymef(a)gmail.com> wrote: >>>> >>>> This sounds similar to the issue I hit with the cluster upgrade process in my environment. I have large 2tb ssds and most of my vms are several hundred Gbs in size. The heal process after host reboot can take 5-10 minutes to complete. I may be able to address this with better gluster tuning. >>>> >>>> Either way the upgrade process should be aware of the heal status and wait for it to complete before attempting to move on to the next host. >>>> >>>> >>>> On Wed, Sep 11, 2019 at 3:53 AM Sahina Bose <sabose(a)redhat.com> wrote: >>>>> >>>>> >>>>> >>>>> On Fri, Aug 9, 2019 at 3:41 PM Martin Perina <mperina(a)redhat.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Aug 8, 2019 at 10:25 AM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Il giorno mar 6 ago 2019 alle ore 23:17 Jayme <jaymef(a)gmail.com> ha scritto: >>>>>>>> >>>>>>>> I’m aware of the heal process but it’s unclear to me if the update continues to run while the volumes are healing and resumes when they are done. There doesn’t seem to be any indication in the ui (unless I’m mistaken) >>>>>>> >>>>>>> >>>>>>> Adding @Martin Perina , @Sahina Bose and

1 0

Re: Does cluster upgrade wait for heal before proceeding to next host?
by Strahil 16 Sep '19

16 Sep '19

I keep reading this chain and I still don't get what/who should wait for the cluster to heal... Is there some kind of built-in autopatching feature? Here is my approach: 1. Set global maintenance 2. Power off the engine 3. Create a gluster snapshot of the engine's volume 4. Power on engine manually 5. Check engine status 6. Upgrade engine 7. Upgrade engine's OS 8. Reboot engine and check health 9. Remove global maintenance 10. Set a host into local maintenance (evacuate all VMs) 11. Use UI to patch the host (enable autoreboot) 12. When host is up - login and check gluster volumes' heal status 13. Remove maintenance for the host and repeate for the rest of the cluster. I realize that for large clusters this approach is tedious and an automatic approach can be scripted. Best Regards, Strahil NikolovOn Sep 16, 2019 11:02, Kaustav Majumder <kmajumde(a)redhat.com> wrote: > > Hi Jayme, > It would be great if you could raise a bug regarding the same. > > On Wed, Sep 11, 2019 at 5:05 PM Jayme <jaymef(a)gmail.com> wrote: >> >> This sounds similar to the issue I hit with the cluster upgrade process in my environment. I have large 2tb ssds and most of my vms are several hundred Gbs in size. The heal process after host reboot can take 5-10 minutes to complete. I may be able to address this with better gluster tuning. >> >> Either way the upgrade process should be aware of the heal status and wait for it to complete before attempting to move on to the next host. >> >> >> On Wed, Sep 11, 2019 at 3:53 AM Sahina Bose <sabose(a)redhat.com> wrote: >>> >>> >>> >>> On Fri, Aug 9, 2019 at 3:41 PM Martin Perina <mperina(a)redhat.com> wrote: >>>> >>>> >>>> >>>> On Thu, Aug 8, 2019 at 10:25 AM Sandro Bonazzola <sbonazzo(a)redhat.com> wrote: >>>>> >>>>> >>>>> >>>>> Il giorno mar 6 ago 2019 alle ore 23:17 Jayme <jaymef(a)gmail.com> ha scritto: >>>>>> >>>>>> I’m aware of the heal process but it’s unclear to me if the update continues to run while the volumes are healing and resumes when they are done. There doesn’t seem to be any indication in the ui (unless I’m mistaken) >>>>> >>>>> >>>>> Adding @Martin Perina , @Sahina Bose and @Laura Wright on this, hyperconverged deployments using cluster upgrade command would probably need some improvement. >>>> >>>> >>>> The cluster upgrade process continues to the 2nd host after the 1st host becomes Up. If 2nd host then fails to switch to maintenance, we stop the upgrade process to prevent breakage. >>>> Sahina, is gluster healing process status exposed in RESTAPI? If so, does it makes sense to wait for healing to be finished before trying to move next host to maintenance? Or any other ideas how to improve? >>> >>> >>> I need to cross-check this, if we expose the heal count in the gluster bricks. Moving a host to maintenance does check if there are pending heal entries or possibility of quorum loss. And this would prevent the additional hosts to upgrade. >>> +Gobinda Das +Sachidananda URS >>> >>>>> >>>>> >>>>>> >>>>>> >>>>>> On Tue, Aug 6, 2019 at 6:06 PM Robert O'Kane <okane(a)khm.de> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> Often(?), updates to a hypervisor that also has (provides) a Gluster >>>>>>> brick takes the hypervisor offline (updates often require a reboot). >>>>>>> >>>>>>> This reboot then makes the brick "out of sync" and it has to be resync'd. >>>>>>> >>>>>>> I find it a "feature" than another host that is also part of a gluster >>>>>>> domain can not be updated (rebooted) before all the bricks are updated >>>>>>> in order to guarantee there is not data loss. It is called Quorum, or? >>>>>>> >>>>>>> Always let the heal process end. Then the next update can start. >>>>>>> For me there is ALWAYS a healing time before Gluster is happy again. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Robert O'Kane >>>>>>> >>>>>>> >>>>>>> Am 06.08.2019 um 16:38 schrieb Shani Leviim: >>>>>>> > Hi Jayme, >>>>>>> > I can't recall such a healing time. >>>>>>> > Can you please retry and attach the engine & vdsm logs so we'll be smarter? >>>>>>> > >>>>>>> > *Regards, >>>>>>> > * >>>>>>> > *Shani Leviim >>>>>>> > * >>>>>>> > >>>>>>> > >>>>>>> > On Tue, Aug 6, 2019 at 5:24 PM Jayme <jaymef(a)gmail.com >>>>>>> > <mailto:jaymef@gmail.com>> wrote: >>>>>>> > >>>>>>> > I've yet to have cluster upgrade finish updating my three host HCI >>>>>>> > cluster. The most recent try was today moving from oVirt 4.3.3 to >>>>>>> > 4.3.5.5. The first host updates normally, but when it moves on to >>>>>>> > the second host it fails to put it in maintenance and the cluster >>>>>>> > upgrade stops. >>>>>>> > >>>>>>> > I suspect this is due to that fact that after my hosts are updated >>>>>>> > it takes 10 minutes or more for all volumes to sync/heal. I have >>>>>>> > 2Tb SSDs. >>>>>>> > >>>>>>> > Does the cluster upgrade process take heal time in to account before >>>>>>> > attempting to place the next host in maintenance to upgrade it? Or >>>>>>> > is there something else that may be at fault here, or perhaps a >>>>>>> > reason why the heal process takes 10 minutes after reboot to complete? >>>>>>> > _______________________________________________ >>>>>>> > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >>>>>>> > To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>> > <mailto:users-leave@ovirt.org> >>>>>>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> > oVirt Code of Conduct: >>>>>>> > https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> > List Archives: >>>>>>> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/5XM3QB3364ZYI… >>>>>>> > >>>>>>> > >>>>>>> > _______________________________________________ >>>>>>> > Users mailing list -- users(a)ovirt.org >>>>>>> > To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GBX3L23MWGMTF… >>>>>>> > >>>>>>> >>>>>>> -- >>>>>>> Systems Administrator >>>>>>> Kunsthochschule für Medien Köln >>>>>>> Peter-Welter-Platz 2 >>>>>>> 50676 Köln >>>>>>> _______________________________________________ >>>>>>> Users mailing list -- users(a)ovirt.org >>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OBAHFFFTDOI7L… >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list -- users(a)ovirt.org >>>>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/T27ROHWZPJL47… >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Sandro Bonazzola >>>>> >>>>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV >>>>> >>>>> Red Hat EMEA >>>>> >>>>> sbonazzo(a)redhat.com >>>>> >>>>> Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. >>>> >>>> >>>> >>>> -- >>>> Martin Perina >>>> Manager, Software Engineering >>>> Red Hat Czech s.r.o. >> >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/I4KLDBPYBCQEM… > > > > -- > > Thanks, > > Kaustav Majumder

2 1

Cannot Activate iSCSI Storage Domain After Errors
by Clint Boggio 16 Sep '19

16 Sep '19

Good Day To all; oVirt 4.3.4.3-1.el7 6 Dell R710 oVirt Nodes 10 gig Infiniband iSCSI Standalone Engine After an error during scheduled automated VM backkup, the iSCSi datastore that the VMs beinng backed up lived on suddenly went inactive and refuse to activate. The backup job did not finish as the source datastore went inactive but the task did clear from the console after i restarted the ovirt-engine service. I have tried: 1. Restart the ovirt-engine service 2. Restarted the engine physical server. 3. Tried to migrate the VM disks to a different datastore (failure) The virtual machines that are currently living on the inactive datastore are still running and accessible. Trying to reboot one of those VM as a test fails as the VM will not come up due to the system believing that the datastore is inactive. From engin.log: 2019-09-15 17:09:08,787-05 INFO [org.ovirt.engine.core.bll.storage.disk.RemoveDiskCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] Removing 'Wildix4-PBX_snapshot_metadata' disk 2019-09-15 17:09:13,399-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] EVENT_ID: IRS_BROKER_COMMAND_FAILURE(10,803), VDSM command DeleteImageGroupVDS failed: General Storage Exception: ('5 [\' Logical volume d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\'] [\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining activation change with other commands is not advised.\', \' /dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset 19198423553536\', " Couldn\'t read volume group metadata from /dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on /dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for VG.\', \' Failed to read metadata summary from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group "d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process volume group d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\': ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'], parent=\'00000000-0000-0000-0000-000000000000\')}',) 2019-09-15 17:09:13,399-05 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] IrsBroker::Failed::DeleteImageGroupVDS: IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = General Storage Exception: ('5 [\' Logical volume d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\'] [\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining activation change with other commands is not advised.\', \' /dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset 19198423553536\', " Couldn\'t read volume group metadata from /dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on /dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for VG.\', \' Failed to read metadata summary from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group "d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process volume group d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\': ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'], parent=\'00000000-0000-0000-0000-000000000000\')}',), code = 200 2019-09-15 17:09:13,463-05 ERROR [org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [fbdd9b7] Command 'org.ovirt.engine.core.bll.storage.disk.image.RemoveImageCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSErrorException: IRSGenericException: IRSErrorException: Failed to DeleteImageGroupVDS, error = General Storage Exception: ('5 [\' Logical volume d1c5f4da-9024-4599-9135-4dabafe86cf8/c5f64501-142b-4a38-9b9c-faf283a64b06 changed.\'] [\' WARNING: This metadata update is NOT backed up.\', \' WARNING: Combining activation change with other commands is not advised.\', \' /dev/mapper/36001405f448f60ecea7479f9d920ac51: Checksum error at offset 19198423553536\', " Couldn\'t read volume group metadata from /dev/mapper/36001405f448f60ecea7479f9d920ac51.", \' Metadata location on /dev/mapper/36001405f448f60ecea7479f9d920ac51 at 19198423553536 has invalid summary for VG.\', \' Failed to read metadata summary from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Failed to scan VG from /dev/mapper/36001405f448f60ecea7479f9d920ac51\', \' Volume group "d1c5f4da-9024-4599-9135-4dabafe86cf8" not found\', \' Cannot process volume group d1c5f4da-9024-4599-9135-4dabafe86cf8\']\nd1c5f4da-9024-4599-9135-4dabafe86cf8/{\'c5f64501-142b-4a38-9b9c-faf283a64b06\': ImgsPar(imgs=[\'aadbfed3-af54-4400-af79-65d09b74c687\'], parent=\'00000000-0000-0000-0000-000000000000\')}',), code = 200 (Failed with error StorageException and code 200) <metadata xmlns:ns0="http://ovirt.org/vm/tune/1.0" xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> Any help or input would be greaatly appreciated.

1 0

Re: Virt-Viewer issue
by Staniforth, Paul 16 Sep '19

16 Sep '19

Hello, you should use remote-viewer, it's part of the virt-viewer package. Paul Staniforth School of Built Environment Engineering and Computing. Leeds Beckett University Networked Systems Analyst, Research and Engineering Support. tel: +44 (0)113 28123754 email: p.staniforth(a)leedsbeckett.ac.uk ________________________________ From: Lozada, Agustin T <Agustin.Lozada(a)centerpointenergy.com> Sent: 13 September 2019 15:38 To: users(a)ovirt.org Subject: [ovirt-users] Virt-Viewer issue Im building my first VM in ovirt with gluster and Im getting this issue when I'm opening a console session using virt-viewer. Does anyone has encountered this issue? [cid:image001.png@01D56A15.FECFDAF0] To view the terms under which this email is distributed, please go to:- http://leedsbeckett.ac.uk/disclaimer/email/

1 0

Does cluster upgrade wait for heal before proceeding to next host?
by Jayme 16 Sep '19

16 Sep '19

I've yet to have cluster upgrade finish updating my three host HCI cluster. The most recent try was today moving from oVirt 4.3.3 to 4.3.5.5. The first host updates normally, but when it moves on to the second host it fails to put it in maintenance and the cluster upgrade stops. I suspect this is due to that fact that after my hosts are updated it takes 10 minutes or more for all volumes to sync/heal. I have 2Tb SSDs. Does the cluster upgrade process take heal time in to account before attempting to place the next host in maintenance to upgrade it? Or is there something else that may be at fault here, or perhaps a reason why the heal process takes 10 minutes after reboot to complete?

8 9

CentOS 6 in oVirt
by Dominik Holler 16 Sep '19

16 Sep '19

Hello, are the official CentOS 6 cloud images known to work in oVirt? I tried, but they seem not to work in oVirt [1], but on plain libvirt. Are there any pitfalls known during using them in oVirt? Thanks Dominik [1] https://gerrit.ovirt.org/#/c/101117/

2 2

VMs stuck at Migrating status in Dashboard- Trying to cancel migration will yield "cannot cancel migration for non-migrating vm"
by paul.christian.suba＠cevalogistics.com 16 Sep '19

16 Sep '19

Saw this problem initially on one host. Host was stuck in entering maintenance mode since it could not migrate a VM from itself. Manually shutdown that VM and also restarted the host. After restart, some VMs are stuck in a migrating status while trying to "migrate" to the rebooted host. oVirt Node 4.3.1

2 2

How do I make delta clones ? OVM (aka Xen) vs Ovirt
by Tim Tuck 15 Sep '19

15 Sep '19

Hi all, I'm new to Ovirt coming from a OVM ( aka Xen ) environment and I'm trying to work out how to do similar things. In my OVM environment I have VMs that are 100GB in size. These VMs are templates that are used for running customer workshops so we create and destroy them quite often. In OVM I can clone 10 x 100GB VM's in about 1 minute, the cloning process just creates a "delta" VM rather than copying the entire 100GB. Those VM's can then be run and the on-disk size of the delta VM grows as it runs and deviates from the master. The same goes for templates, creating a 100GB VM from a template takes only a few seconds. Creating a template from a VM only takes a few seconds. My problem is that I cant seem to do any of this using Ovirt. it appears that there is no way to tell the cloning process " I want 10" like I can in OVM - am I missing something here? In my tests I find that even if a "snapshot" is made, the entire VM is copied rather than creating a delta. Creating a VM from a template is no different in that it copies the entire VM. So... Is it possible for Ovirt to create multiple clones on request ? Can Ovirt do "delta" copies rather than copy the entire VM ? If Ovirt can do these things, how do I set it up to to do that ? Any help appreciated. thanks Tim --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

2 1

Virt-Viewer issue
by Lozada, Agustin T 13 Sep '19

13 Sep '19

Im building my first VM in ovirt with gluster and Im getting this issue when I'm opening a console session using virt-viewer. Does anyone has encountered this issue? [cid:image001.png@01D56A15.FECFDAF0]

1 0

Re: VMs stuck at Migrating status in Dashboard- Trying to cancel migration will yield "cannot cancel migration for non-migrating vm"
by Strahil 13 Sep '19

13 Sep '19

Have you marked your host as 'recently rebooted' (or whatever was the name in the UI)? Best Regards, Strahil NikolovOn Sep 12, 2019 18:15, paul.christian.suba(a)cevalogistics.com wrote: > > Saw this problem initially on one host. Host was stuck in entering maintenance mode since it could not migrate a VM from itself. > Manually shutdown that VM and also restarted the host. > After restart, some VMs are stuck in a migrating status while trying to "migrate" to the rebooted host. > > oVirt Node 4.3.1 > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QFMNWH63FQIVZ…

1 0