April 2019 - Users - oVirt List Archives

Can't create pool after update to 4.3.3
by kiv＠intercom.pro 22 Apr '19

22 Apr '19

Hi all. After update my oVirt to 4.3.3 i can't create new pool. I make new template and trying to create new pool - but can't select any template, selection field is empty. I have this error: Uncaught exception occurred. Please try reloading the page. Details: Exception caught: undefined Please have your administrator check the UI logs

3 4

4.3.x upgrade and issues with OVN
by Charles Weber 22 Apr '19

22 Apr '19

Hi everyone, We have had a pair of Ovirt clusters at work, starting with the 3.x. I replaced last year with 4.x cluster on new machines. 4.2 worked great, but when I upgraded to 4.3.2 and now .3 I immediately ran into host networking issues resulting in hung or unable totally to migrate VMs. Configuration notes: 1. I use commercial star.x.x.x SSL cert on engine installed per Ovirt.org instructions 2. I have 2 untagged NICs assigned to ovirtmgmt and Public, both on the same IP range network. I also have another set of VLANs tagged on various NICs. 1 cluster uses HP blades with lots of NICs, the other uses supermicro 1Us with only 2 NICs. 3. I upgraded both engines and hosts to 4.3.2 from 4.2.8. I started seeing hung migrations cleared by restarting vdsm. 4. All computers involved run current CentOS 7.6 5. Both clusters use iscsi on dedicated tagged storage VLAN, seems fine. 6. I upgraded 1 engine and 1 host to 4.3.3 and things got worse. I have not updated any hosts or the second engine since. 7. The 4 hosts on the 4.3.3 engine now have out of sync error that refuses to clear for my Public network. 8. 2 OV 4.3.2 nodes had the following errors with 2 ovn ports. Here is example, perhaps related to the genev_sys_6081 error. ovn-d6eaa1-0: attempting to add tunnel port with same config as port 'ovn-f0f789-0' (::->137.187.160.13 ovn-877214-0: attempting to add tunnel port with same config as port 'ovn-483528-0' 9. I deleted and uninstalled all ovirt related rpms on one node, then did clean node install using latest 4.3 release. Same errors. 10. I downloaded latest node iso, installed on same host, upgraded to to 4.3.3.1 node and joined cluster. The node installation has no errors and can migrate to my other hosts. All networks are in sync. The migration hangs. Restarting VDSMD clears the hung migration. 11. None of the other hosts can migrate VMs to the new node. Here are excerpts from 3.2 node on 3.3 engine log file. Apr 17, 2019, 2:43:04 PM Check for available updates on host BRCVN3 was completed successfully with message 'ovirt-host, cockpit-ovirt-dashboard, vdsm-client, ovirt-release43, ovirt-host, ovirt-hosted-engine-setup, vdsm-api, vdsm-jsonrpc, ovirt-ansible-hosted-engine-setup, ovirt-host-dependencies and 12 others. To see all packages check engine.log.'. oVirt Apr 17, 2019, 2:43:04 PM Host BRCVN3 has available updates: ovirt-host, cockpit-ovirt-dashboard, vdsm-client, ovirt-release43, ovirt-host, ovirt-hosted-engine-setup, vdsm-api, vdsm-jsonrpc, ovirt-ansible-hosted-engine-setup, ovirt-host-dependencies and 12 others. To see all packages check engine.log.. oVirt Apr 17, 2019, 2:27:39 PM Host BRCVN3's following network(s) are not synchronized with their Logical Network configuration: Public. oVirt Apr 16, 2019, 5:53:05 PM Failed to sync all host BRCVN3 networks c1ba631c-7be5-4be0-abbb-a37b6bb7d26d oVirt Apr 16, 2019, 5:53:05 PM (1/1): Failed to apply changes on host BRCVN3. (User: admin@internal-authz) 455a260f oVirt Apr 16, 2019, 5:53:05 PM VDSM BRCVN3 command HostSetupNetworksVDS failed: Bridge Public has interfaces set([u'vnet19', u'vnet12', u'vnet10', u'vnet11', u'vnet16', u'vnet17', u'vnet14', u'vnet15', u'vnet0', u'vnet2', u'vnet3', u'vnet4', u'vnet7', u'vnet8', u'vnet9']) connected oVirt Apr 16, 2019, 5:52:59 PM (1/1): Applying network's changes on host BRCVN3. (User: admin@internal-authz) 455a260f oVirt Apr 16, 2019, 5:32:07 PM Check for available updates on host BRCVN3 was completed successfully with message 'no updates found.'. oVirt Apr 16, 2019, 5:16:40 PM Host BRCVN3's following network(s) are not synchronized with their Logical Network configuration: Public. oVirt Apr 16, 2019, 3:33:36 PM Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4). bd21f294-0170-47d1-845e-c64c52abbab4 oVirt Apr 16, 2019, 3:33:36 PM Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User: admin@internal-authz). bd21f294-0170-47d1-845e-c64c52abbab4 oVirt Apr 16, 2019, 2:49:36 PM Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4). 7c04f7a6-8f88-4c64-99f9-e621724fc7ff oVirt Apr 16, 2019, 2:49:36 PM Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User: admin@internal-authz). 7c04f7a6-8f88-4c64-99f9-e621724fc7ff oVirt Apr 16, 2019, 2:22:03 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 2:11:02 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 2:10:02 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 2:05:46 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 2:03:16 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 2:02:16 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 2:00:16 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 1:59:01 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 1:57:31 PM VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error: {'reason': '[Errno 19] genev_sys_6081 is not present in the system'} oVirt Apr 16, 2019, 1:57:05 PM

2 1

Re: Help with 4.3.3 re-setup
by Strahil 21 Apr '19

21 Apr '19

Hi Michael, I know how you feel abiut VDO. I wass in the same boat until I starrted using systemd's mount units.Since then I do not have any issues. For now try the following: Boot 1 host via rescue dvd , chroot into the system and install the vdo and kvdo packages (maybe reinstall is more correct) . Once then, try to run the vdo service (still in chroot) and try to make it work. Then, just rebuild the initramfs (dracut -f -k kernel-version) and reboot. Check if your VDO, bricks and Gluster are operational again. If yes, do the other nodes. Once done - run 'gluster volume heal my-volume full' (replace my-volume with actual name). This will force gluster to sync all bricks. P.S.: Now, I'm using gluster snapshot for recovery between engine upgrades (HostedEngine is stopped while makking the snapahot). Best Regards, Strahil NikolovOn Apr 21, 2019 20:44, michael(a)wanderingmad.com wrote: > > I updated my ovirt from 4.3.2 to 4.3.3 and everything went bad, my gluster volumes were constantly dropping out and getting unsynced and everything was just running slower than molasses. So I thought I would reload the hosts from ovirt node to Centos and re-do the storage, thinking that may help. I tried using centos, but I was never able to add centos hosts back to the ovirt engine due to a gluster error in the logs, so I went and downloaded ovirt node 4.3.3, and now after spending an hour reloading all the servers with ovirt node, One of the nodes I can't access the cockpit on, and the other two have broken VDO modules, the modules are installed but don't exist in the kernel, and when I removed the module, I lost all network configuration, and when I re-installed it, I get a scriptlet error on compilation and there is still a missing VDO module. This is now the 3rd time i've hit this VDO issue and i'm really frustrated right now, should I just go back to 4.3.2? I just redownloa > ded the latest 4.3.3 ISO today. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QA2SMYRZCXQYR…

1 0

Help with 4.3.3 re-setup
by michael＠wanderingmad.com 21 Apr '19

21 Apr '19

I updated my ovirt from 4.3.2 to 4.3.3 and everything went bad, my gluster volumes were constantly dropping out and getting unsynced and everything was just running slower than molasses. So I thought I would reload the hosts from ovirt node to Centos and re-do the storage, thinking that may help. I tried using centos, but I was never able to add centos hosts back to the ovirt engine due to a gluster error in the logs, so I went and downloaded ovirt node 4.3.3, and now after spending an hour reloading all the servers with ovirt node, One of the nodes I can't access the cockpit on, and the other two have broken VDO modules, the modules are installed but don't exist in the kernel, and when I removed the module, I lost all network configuration, and when I re-installed it, I get a scriptlet error on compilation and there is still a missing VDO module. This is now the 3rd time i've hit this VDO issue and i'm really frustrated right now, should I just go back to 4.3.2? I just redownloa ded the latest 4.3.3 ISO today.

1 0

Issues post-upgrade from 4.2.8 to 4.3.2
by Vrgotic, Marko 21 Apr '19

21 Apr '19

Dear oVirt team, We have 3 Hosts HA Cluster with SHE engine. Storage is NetApp. Recently we have executed an upgrade from 4.2.8 to 4.3.2. Since then we are seeing strange behavior when trying to put Hosts to Maintenance or manually Migrate VMs between Hosts. The most repeated WARN/ERROR I am observing is following: The UI is constantly throwing an exception when trying to put HA host to maintenance or increase the SPM level. The repeated log line from engine.log I see, with regards or failing to migrate all hosts is: 2019-04-18 13:05:30,760Z WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [63d33dd1] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-02,$filterName Memory,$availableMem 3824,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-01,$filterName Memory,$availableMem 0,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-02,$filterName Memory,$availableMem 3824,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-01,$filterName Memory,$availableMem 0,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL But there is more than enough. Also from UI.log: 2019-04-18 14:05:09,007Z ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-114) [] Permutation name: 0D2DB7A91B469CC36C64386E5632FAC5 2019-04-18 14:05:09,007Z ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-114) [] Uncaught exception: com.google.gwt.event.shared.Umbrell aException: Exception caught: (TypeError) : oab(...) is null at java.lang.Throwable.Throwable(Throwable.java:70) [rt.jar:1.8.0_201] at java.lang.RuntimeException.RuntimeException(RuntimeException.java:32) [rt.jar:1.8.0_201] at com.google.web.bindery.event.shared.UmbrellaException.UmbrellaException(UmbrellaException.java:64) [gwt-servlet.jar:] at com.google.gwt.event.shared.UmbrellaException.UmbrellaException(UmbrellaException.java:25) [gwt-servlet.jar:] at com.google.gwt.event.shared.HandlerManager.$fireEvent(HandlerManager.java:117) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.$fireEvent(Widget.java:127) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.fireEvent(Widget.java:127) [gwt-servlet.jar:] at com.google.gwt.event.dom.client.DomEvent.fireNativeEvent(DomEvent.java:110) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.$onBrowserEvent(Widget.java:163) [gwt-servlet.jar:] at com.google.gwt.user.client.ui.Widget.onBrowserEvent(Widget.java:163) [gwt-servlet.jar:] at com.google.gwt.user.client.DOM.dispatchEvent(DOM.java:1415) [gwt-servlet.jar:] at com.google.gwt.user.client.impl.DOMImplStandard.dispatchEvent(DOMImplStandard.java:312) [gwt-servlet.jar:] at com.google.gwt.core.client.impl.Impl.apply(Impl.java:236) [gwt-servlet.jar:] at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:275) [gwt-servlet.jar:] at Unknown.Su/<(https://ovirt-staging-engine.avinity.tv/ovirt-engine/webadmin/?locale=en_US… line 9 > scriptElement) at Unknown.anonymous(Unknown) Caused by: com.google.gwt.core.client.JavaScriptException: (TypeError) : oab(...) is null at org.ovirt.engine.ui.uicommonweb.models.clusters.ClusterGuideModel.$onAddHost(ClusterGuideModel.java:533) at org.ovirt.engine.ui.uicommonweb.models.clusters.ClusterGuideModel.executeCommand(ClusterGuideModel.java:617) at org.ovirt.engine.ui.uicommonweb.UICommand.$execute(UICommand.java:163) at org.ovirt.engine.ui.common.presenter.AbstractModelBoundPopupPresenterWidget.$lambda$4(AbstractModelBoundPopupPresenterWidget.java:306) at org.ovirt.engine.ui.common.presenter.AbstractModelBoundPopupPresenterWidget$lambda$4$Type.onClick(AbstractModelBoundPopupPresenterWidget.java:306) at com.google.gwt.event.dom.client.ClickEvent.dispatch(ClickEvent.java:55) [gwt-servlet.jar:] at com.google.gwt.event.shared.GwtEvent.dispatch(GwtEvent.java:76) [gwt-servlet.jar:] at com.google.web.bindery.event.shared.SimpleEventBus.$doFire(SimpleEventBus.java:173) [gwt-servlet.jar:] ... 12 more Can you give me some advice on how to proceed? This started happening since upgrade. Kindly awaiting your reply, Marko Vrgotic

1 0

VM Snapshots not erasable and not bootable
by Jonathan Baecker 21 Apr '19

21 Apr '19

Hello, I make automatically backups of my VMs and last night there was making some new one. But somehow ovirt could not delete the snapshots anymore, in the log it show that it tried the hole day to delete them but they had to wait until the merge command was done. In the evening the host was totally crashed and started again. Now I can not delete the snapshots manually and I can also not start the VMs anymore. In the web interface I get the message: VM timetrack is down with error. Exit message: Bad volume specification {'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': '0'}, 'serial': 'fd3b80fd-49ad-44ac-9efd-1328300582cd', 'index': 0, 'iface': 'scsi', 'apparentsize': '1572864', 'specParams': {}, 'cache': 'none', 'imageID': 'fd3b80fd-49ad-44ac-9efd-1328300582cd', 'truesize': '229888', 'type': 'disk', 'domainID': '9c3f06cf-7475-448e-819b-f4f52fa7d782', 'reqsize': '0', 'format': 'cow', 'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'device': 'disk', 'path': '/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fd3b80fd-49ad-44ac-9efd-1328300582cd/47c0f42e-8bda-4e3f-8337-870899238788', 'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID': '47c0f42e-8bda-4e3f-8337-870899238788', 'diskType': 'file', 'alias': 'ua-fd3b80fd-49ad-44ac-9efd-1328300582cd', 'discard': False}. When I check the path permission is correct and there are also files in it. Is there any ways to fix that? Or to prevent this issue in the future? In the attachment I send also the engine.log Regards Jonathan

2 10

Re: Ovirt gluster arbiter within hosted VM
by Strahil 20 Apr '19

20 Apr '19

With GlusterD2 , you can use thin arbiter - which can be deployed in the cloud. Do not try to setup regular arbiter far away from the data bicks or yoir performance will be awful. Best Regards, Strahil NikolovOn Apr 19, 2019 13:28, Alex K <rightkicktech(a)gmail.com> wrote: > > > > On Fri, Apr 19, 2019 at 1:14 PM Scott Worthington <scott.c.worthington(a)gmail.com> wrote: >> >> And where, magically, is that node's storage going to live? > > In the disk of the VM. Let me know if I need to clarify further. >> >> >> You can't fake a proper setup of gluster. > > Not trying to fake it. Trying to find a solution with the available options. >> >> >> On Fri, Apr 19, 2019, 5:00 AM Alex K <rightkicktech(a)gmail.com> wrote: >>> >>> Hi all, >>> >>> I have a two node hyper-converged setup which are causing me split-brains when network issues are encountered. Since I cannot add a third hardware node, I was thinking to add a dedicated guest VM hosted in same hyper-converged cluster which would do the arbiter for the volumes. >>> >>> What do you think about this setup in regards to stability and performance? >>> I am running ovirt 4.2. >>> >>> Thanx, >>> Alex >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LTDOAMBKVE5BP…

2 1

Re: Ovirt gluster arbiter within hosted VM
by Strahil 20 Apr '19

20 Apr '19

As long as the VM is not hosted by those 2 hyperconverged nodes - there should be no problem. Another option is a small machine with a single SSD. I'm using Lenovo Tiny M-series as an arbiter. Best Regards, Strahil NikolovOn Apr 19, 2019 12:59, Alex K <rightkicktech(a)gmail.com> wrote: > > Hi all, > > I have a two node hyper-converged setup which are causing me split-brains when network issues are encountered. Since I cannot add a third hardware node, I was thinking to add a dedicated guest VM hosted in same hyper-converged cluster which would do the arbiter for the volumes. > > What do you think about this setup in regards to stability and performance? > I am running ovirt 4.2. > > Thanx, > Alex

1 0

Replica 3 distribute-replicated - data placement and fault tolerance
by Leo David 19 Apr '19

19 Apr '19

Hello Everyone, I did some fio performance tests on a particular vm and I have noticed things that I do not understand regarding how data is placed along the bricks. I am sure this is a lack of knowledge, but I would really appreciate any help in understanding this, I did a bit of research on the internet, but just could't find something relevant. I have one replica 3 distributed-replicated arbitrated volume, across 18 bricks ( 9 nodes, 2 jbods per node ). The volume was created as: node1 - brick1, node-2, brick1,.....node9-brick1, node1 - brick2, node-2, brick2,.....node9-brick2 As far as i've understood, under the hood there are sets of 3 times replicated data ( subvolumes ) which are assigned to the first 3 bricks, next set of replicated data to the next set of 3 bricks, and so on.. Now, I have this testing vm runign one node one. When I've started the fio test, i've noticed incresed gluster traffic from node 1 to node2,4 and 5 So I assumed that the vm disk is data resides on a subvolume allocated to bricks from these hosts. Then I have migrated the vm on node 2, and did the same test. Now the increased treffic is generated from node2 to node1, 4, and 5.. What I do not understand is: - why gluster client ( ovirt host ) is sending data to 3 bricks if the volume is arbitrated - shouldn't it send only to 2 of them ? - why are there 4 brick implicated in this subvolume - what would it be the fault tolerance level in this setup ,ie: how many hosts can I take down and still having the volume serving IO requests; can they be random ? I am sorry for my lack of knowledge, I am just trying to understand what is happening so I can deploy a decent proper setup of an hci environment. Thank you, Leo -- Best regards, Leo David

1 0

Re: Ovirt gluster arbiter within hosted VM
by Alex K 19 Apr '19

19 Apr '19

On Fri, Apr 19, 2019 at 1:14 PM Scott Worthington < scott.c.worthington(a)gmail.com> wrote: > And where, magically, is that node's storage going to live? > In the disk of the VM. Let me know if I need to clarify further. > > You can't fake a proper setup of gluster. > Not trying to fake it. Trying to find a solution with the available options. > > On Fri, Apr 19, 2019, 5:00 AM Alex K <rightkicktech(a)gmail.com> wrote: > >> Hi all, >> >> I have a two node hyper-converged setup which are causing me split-brains >> when network issues are encountered. Since I cannot add a third hardware >> node, I was thinking to add a dedicated guest VM hosted in same >> hyper-converged cluster which would do the arbiter for the volumes. >> >> What do you think about this setup in regards to stability and >> performance? >> I am running ovirt 4.2. >> >> Thanx, >> Alex >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org >> To unsubscribe send an email to users-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/LTDOAMBKVE5BP… >> >

1 0