Can't create pool after update to 4.3.3
by kiv＠intercom.pro
Hi all.
After update my oVirt to 4.3.3 i can't create new pool.
I make new template and trying to create new pool - but can't select any template, selection field is empty.
I have this error:
Uncaught exception occurred. Please try reloading the page. Details: Exception caught: undefined
Please have your administrator check the UI logs
5 years, 7 months
4.3.x upgrade and issues with OVN
by Charles Weber
Hi everyone,
We have had a pair of Ovirt clusters at work, starting with the 3.x. I
replaced last year with 4.x cluster on new machines. 4.2 worked great, but
when I upgraded to 4.3.2 and now .3 I immediately ran into host networking
issues resulting in hung or unable totally to migrate VMs.
Configuration notes:
1. I use commercial star.x.x.x SSL cert on engine installed per Ovirt.org
instructions
2. I have 2 untagged NICs assigned to ovirtmgmt and Public, both on the
same IP range network. I also have another set of VLANs tagged on various
NICs. 1 cluster uses HP blades with lots of NICs, the other uses supermicro
1Us with only 2 NICs.
3. I upgraded both engines and hosts to 4.3.2 from 4.2.8. I started seeing
hung migrations cleared by restarting vdsm.
4. All computers involved run current CentOS 7.6
5. Both clusters use iscsi on dedicated tagged storage VLAN, seems fine.
6. I upgraded 1 engine and 1 host to 4.3.3 and things got worse. I have not
updated any hosts or the second engine since.
7. The 4 hosts on the 4.3.3 engine now have out of sync error that refuses
to clear for my Public network.
8. 2 OV 4.3.2 nodes had the following errors with 2 ovn ports. Here is
example, perhaps related to the genev_sys_6081 error.
ovn-d6eaa1-0: attempting to add tunnel port with same config as port
'ovn-f0f789-0' (::->137.187.160.13
ovn-877214-0: attempting to add tunnel port with same config as port
'ovn-483528-0'
9. I deleted and uninstalled all ovirt related rpms on one node, then did
clean node install using latest 4.3 release. Same errors.
10. I downloaded latest node iso, installed on same host, upgraded to to
4.3.3.1 node and joined cluster. The node installation has no errors and
can migrate to my other hosts. All networks are in sync. The migration
hangs. Restarting VDSMD clears the hung migration.
11. None of the other hosts can migrate VMs to the new node.
Here are excerpts from 3.2 node on 3.3 engine log file.
Apr 17, 2019, 2:43:04 PM
Check for available updates on host BRCVN3 was completed successfully with
message 'ovirt-host, cockpit-ovirt-dashboard, vdsm-client, ovirt-release43,
ovirt-host, ovirt-hosted-engine-setup, vdsm-api, vdsm-jsonrpc,
ovirt-ansible-hosted-engine-setup, ovirt-host-dependencies and 12 others.
To see all packages check engine.log.'.
oVirt
Apr 17, 2019, 2:43:04 PM
Host BRCVN3 has available updates: ovirt-host, cockpit-ovirt-dashboard,
vdsm-client, ovirt-release43, ovirt-host, ovirt-hosted-engine-setup,
vdsm-api, vdsm-jsonrpc, ovirt-ansible-hosted-engine-setup,
ovirt-host-dependencies and 12 others. To see all packages check
engine.log..
oVirt
Apr 17, 2019, 2:27:39 PM
Host BRCVN3's following network(s) are not synchronized with their Logical
Network configuration: Public.
oVirt
Apr 16, 2019, 5:53:05 PM
Failed to sync all host BRCVN3 networks
c1ba631c-7be5-4be0-abbb-a37b6bb7d26d
oVirt
Apr 16, 2019, 5:53:05 PM
(1/1): Failed to apply changes on host BRCVN3. (User: admin@internal-authz)
455a260f
oVirt
Apr 16, 2019, 5:53:05 PM
VDSM BRCVN3 command HostSetupNetworksVDS failed: Bridge Public has
interfaces set([u'vnet19', u'vnet12', u'vnet10', u'vnet11', u'vnet16',
u'vnet17', u'vnet14', u'vnet15', u'vnet0', u'vnet2', u'vnet3', u'vnet4',
u'vnet7', u'vnet8', u'vnet9']) connected
oVirt
Apr 16, 2019, 5:52:59 PM
(1/1): Applying network's changes on host BRCVN3. (User:
admin@internal-authz)
455a260f
oVirt
Apr 16, 2019, 5:32:07 PM
Check for available updates on host BRCVN3 was completed successfully with
message 'no updates found.'.
oVirt
Apr 16, 2019, 5:16:40 PM
Host BRCVN3's following network(s) are not synchronized with their Logical
Network configuration: Public.
oVirt
Apr 16, 2019, 3:33:36 PM
Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4).
bd21f294-0170-47d1-845e-c64c52abbab4
oVirt
Apr 16, 2019, 3:33:36 PM
Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User:
admin@internal-authz).
bd21f294-0170-47d1-845e-c64c52abbab4
oVirt
Apr 16, 2019, 2:49:36 PM
Migration failed (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4).
7c04f7a6-8f88-4c64-99f9-e621724fc7ff
oVirt
Apr 16, 2019, 2:49:36 PM
Migration started (VM: Hwebdev, Source: BRCVN3, Destination: BRCVN4, User:
admin@internal-authz).
7c04f7a6-8f88-4c64-99f9-e621724fc7ff
oVirt
Apr 16, 2019, 2:22:03 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:11:02 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:10:02 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:05:46 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:03:16 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:02:16 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 2:00:16 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 1:59:01 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 1:57:31 PM
VDSM BRCVN3 command Get Host Statistics failed: Internal JSON-RPC error:
{'reason': '[Errno 19] genev_sys_6081 is not present in the system'}
oVirt
Apr 16, 2019, 1:57:05 PM
5 years, 7 months
Re: Help with 4.3.3 re-setup
by Strahil
Hi Michael,
I know how you feel abiut VDO. I wass in the same boat until I starrted using systemd's mount units.Since then I do not have any issues.
For now try the following:
Boot 1 host via rescue dvd , chroot into the system and install the vdo and kvdo packages (maybe reinstall is more correct) .
Once then, try to run the vdo service (still in chroot) and try to make it work.
Then, just rebuild the initramfs (dracut -f -k kernel-version) and reboot.
Check if your VDO, bricks and Gluster are operational again.
If yes, do the other nodes.
Once done - run 'gluster volume heal my-volume full' (replace my-volume with actual name).
This will force gluster to sync all bricks.
P.S.: Now, I'm using gluster snapshot for recovery between engine upgrades (HostedEngine is stopped while makking the snapahot).
Best Regards,
Strahil NikolovOn Apr 21, 2019 20:44, michael(a)wanderingmad.com wrote:
>
> I updated my ovirt from 4.3.2 to 4.3.3 and everything went bad, my gluster volumes were constantly dropping out and getting unsynced and everything was just running slower than molasses. So I thought I would reload the hosts from ovirt node to Centos and re-do the storage, thinking that may help. I tried using centos, but I was never able to add centos hosts back to the ovirt engine due to a gluster error in the logs, so I went and downloaded ovirt node 4.3.3, and now after spending an hour reloading all the servers with ovirt node, One of the nodes I can't access the cockpit on, and the other two have broken VDO modules, the modules are installed but don't exist in the kernel, and when I removed the module, I lost all network configuration, and when I re-installed it, I get a scriptlet error on compilation and there is still a missing VDO module. This is now the 3rd time i've hit this VDO issue and i'm really frustrated right now, should I just go back to 4.3.2? I just redownloa
> ded the latest 4.3.3 ISO today.
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QA2SMYRZCXQ...
5 years, 7 months
Help with 4.3.3 re-setup
by michael＠wanderingmad.com
I updated my ovirt from 4.3.2 to 4.3.3 and everything went bad, my gluster volumes were constantly dropping out and getting unsynced and everything was just running slower than molasses. So I thought I would reload the hosts from ovirt node to Centos and re-do the storage, thinking that may help. I tried using centos, but I was never able to add centos hosts back to the ovirt engine due to a gluster error in the logs, so I went and downloaded ovirt node 4.3.3, and now after spending an hour reloading all the servers with ovirt node, One of the nodes I can't access the cockpit on, and the other two have broken VDO modules, the modules are installed but don't exist in the kernel, and when I removed the module, I lost all network configuration, and when I re-installed it, I get a scriptlet error on compilation and there is still a missing VDO module. This is now the 3rd time i've hit this VDO issue and i'm really frustrated right now, should I just go back to 4.3.2? I just redownloa
ded the latest 4.3.3 ISO today.
5 years, 7 months
Issues post-upgrade from 4.2.8 to 4.3.2
by Vrgotic, Marko
Dear oVirt team,
We have 3 Hosts HA Cluster with SHE engine.
Storage is NetApp.
Recently we have executed an upgrade from 4.2.8 to 4.3.2.
Since then we are seeing strange behavior when trying to put Hosts to Maintenance or manually Migrate VMs between Hosts.
The most repeated WARN/ERROR I am observing is following:
The UI is constantly throwing an exception when trying to put HA host to maintenance or increase the SPM level.
The repeated log line from engine.log I see, with regards or failing to migrate all hosts is:
2019-04-18 13:05:30,760Z WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [63d33dd1] Validation of action 'MigrateVm' failed for user SYSTEM. Reasons: VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-02,$filterName Memory,$availableMem 3824,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-01,$filterName Memory,$availableMem 0,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,SCHEDULING_ALL_HOSTS_FILTERED_OUT,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-02,$filterName Memory,$availableMem 3824,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL,VAR__FILTERTYPE__INTERNAL,$hostName ovirt-staging-hv-01,$filterName Memory,$availableMem 0,VAR__DETAIL__NOT_ENOUGH_MEMORY,SCHEDULING_HOST_FILTERED_REASON_WITH_DETAIL
But there is more than enough.
Also from UI.log:
2019-04-18 14:05:09,007Z ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-114) [] Permutation name: 0D2DB7A91B469CC36C64386E5632FAC5
2019-04-18 14:05:09,007Z ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-114) [] Uncaught exception: com.google.gwt.event.shared.Umbrell
aException: Exception caught: (TypeError) : oab(...) is null
at java.lang.Throwable.Throwable(Throwable.java:70) [rt.jar:1.8.0_201]
at java.lang.RuntimeException.RuntimeException(RuntimeException.java:32) [rt.jar:1.8.0_201]
at com.google.web.bindery.event.shared.UmbrellaException.UmbrellaException(UmbrellaException.java:64) [gwt-servlet.jar:]
at com.google.gwt.event.shared.UmbrellaException.UmbrellaException(UmbrellaException.java:25) [gwt-servlet.jar:]
at com.google.gwt.event.shared.HandlerManager.$fireEvent(HandlerManager.java:117) [gwt-servlet.jar:]
at com.google.gwt.user.client.ui.Widget.$fireEvent(Widget.java:127) [gwt-servlet.jar:]
at com.google.gwt.user.client.ui.Widget.fireEvent(Widget.java:127) [gwt-servlet.jar:]
at com.google.gwt.event.dom.client.DomEvent.fireNativeEvent(DomEvent.java:110) [gwt-servlet.jar:]
at com.google.gwt.user.client.ui.Widget.$onBrowserEvent(Widget.java:163) [gwt-servlet.jar:]
at com.google.gwt.user.client.ui.Widget.onBrowserEvent(Widget.java:163) [gwt-servlet.jar:]
at com.google.gwt.user.client.DOM.dispatchEvent(DOM.java:1415) [gwt-servlet.jar:]
at com.google.gwt.user.client.impl.DOMImplStandard.dispatchEvent(DOMImplStandard.java:312) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.apply(Impl.java:236) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:275) [gwt-servlet.jar:]
at Unknown.Su/<(https://ovirt-staging-engine.avinity.tv/ovirt-engine/webadmin/?locale=en_... line 9 > scriptElement)
at Unknown.anonymous(Unknown)
Caused by: com.google.gwt.core.client.JavaScriptException: (TypeError) : oab(...) is null
at org.ovirt.engine.ui.uicommonweb.models.clusters.ClusterGuideModel.$onAddHost(ClusterGuideModel.java:533)
at org.ovirt.engine.ui.uicommonweb.models.clusters.ClusterGuideModel.executeCommand(ClusterGuideModel.java:617)
at org.ovirt.engine.ui.uicommonweb.UICommand.$execute(UICommand.java:163)
at org.ovirt.engine.ui.common.presenter.AbstractModelBoundPopupPresenterWidget.$lambda$4(AbstractModelBoundPopupPresenterWidget.java:306)
at org.ovirt.engine.ui.common.presenter.AbstractModelBoundPopupPresenterWidget$lambda$4$Type.onClick(AbstractModelBoundPopupPresenterWidget.java:306)
at com.google.gwt.event.dom.client.ClickEvent.dispatch(ClickEvent.java:55) [gwt-servlet.jar:]
at com.google.gwt.event.shared.GwtEvent.dispatch(GwtEvent.java:76) [gwt-servlet.jar:]
at com.google.web.bindery.event.shared.SimpleEventBus.$doFire(SimpleEventBus.java:173) [gwt-servlet.jar:]
... 12 more
Can you give me some advice on how to proceed? This started happening since upgrade.
Kindly awaiting your reply,
Marko Vrgotic
5 years, 7 months
VM Snapshots not erasable and not bootable
by Jonathan Baecker
Hello,
I make automatically backups of my VMs and last night there was making
some new one. But somehow ovirt could not delete the snapshots anymore,
in the log it show that it tried the hole day to delete them but they
had to wait until the merge command was done.
In the evening the host was totally crashed and started again. Now I can
not delete the snapshots manually and I can also not start the VMs
anymore. In the web interface I get the message:
VM timetrack is down with error. Exit message: Bad volume specification
{'address': {'bus': '0', 'controller': '0', 'type': 'drive', 'target':
'0', 'unit': '0'}, 'serial': 'fd3b80fd-49ad-44ac-9efd-1328300582cd',
'index': 0, 'iface': 'scsi', 'apparentsize': '1572864', 'specParams':
{}, 'cache': 'none', 'imageID': 'fd3b80fd-49ad-44ac-9efd-1328300582cd',
'truesize': '229888', 'type': 'disk', 'domainID':
'9c3f06cf-7475-448e-819b-f4f52fa7d782', 'reqsize': '0', 'format': 'cow',
'poolID': '59ef3a18-002f-02d1-0220-000000000124', 'device': 'disk',
'path':
'/rhev/data-center/59ef3a18-002f-02d1-0220-000000000124/9c3f06cf-7475-448e-819b-f4f52fa7d782/images/fd3b80fd-49ad-44ac-9efd-1328300582cd/47c0f42e-8bda-4e3f-8337-870899238788',
'propagateErrors': 'off', 'name': 'sda', 'bootOrder': '1', 'volumeID':
'47c0f42e-8bda-4e3f-8337-870899238788', 'diskType': 'file', 'alias':
'ua-fd3b80fd-49ad-44ac-9efd-1328300582cd', 'discard': False}.
When I check the path permission is correct and there are also files in it.
Is there any ways to fix that? Or to prevent this issue in the future?
In the attachment I send also the engine.log
Regards
Jonathan
5 years, 7 months
Re: Ovirt gluster arbiter within hosted VM
by Strahil
With GlusterD2 , you can use thin arbiter - which can be deployed in the cloud.
Do not try to setup regular arbiter far away from the data bicks or yoir performance will be awful.
Best Regards,
Strahil NikolovOn Apr 19, 2019 13:28, Alex K <rightkicktech(a)gmail.com> wrote:
>
>
>
> On Fri, Apr 19, 2019 at 1:14 PM Scott Worthington <scott.c.worthington(a)gmail.com> wrote:
>>
>> And where, magically, is that node's storage going to live?
>
> In the disk of the VM. Let me know if I need to clarify further.
>>
>>
>> You can't fake a proper setup of gluster.
>
> Not trying to fake it. Trying to find a solution with the available options.
>>
>>
>> On Fri, Apr 19, 2019, 5:00 AM Alex K <rightkicktech(a)gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> I have a two node hyper-converged setup which are causing me split-brains when network issues are encountered. Since I cannot add a third hardware node, I was thinking to add a dedicated guest VM hosted in same hyper-converged cluster which would do the arbiter for the volumes.
>>>
>>> What do you think about this setup in regards to stability and performance?
>>> I am running ovirt 4.2.
>>>
>>> Thanx,
>>> Alex
>>> _______________________________________________
>>> Users mailing list -- users(a)ovirt.org
>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LTDOAMBKVE5...
5 years, 7 months
Re: Ovirt gluster arbiter within hosted VM
by Strahil
As long as the VM is not hosted by those 2 hyperconverged nodes - there should be no problem.
Another option is a small machine with a single SSD.
I'm using Lenovo Tiny M-series as an arbiter.
Best Regards,
Strahil NikolovOn Apr 19, 2019 12:59, Alex K <rightkicktech(a)gmail.com> wrote:
>
> Hi all,
>
> I have a two node hyper-converged setup which are causing me split-brains when network issues are encountered. Since I cannot add a third hardware node, I was thinking to add a dedicated guest VM hosted in same hyper-converged cluster which would do the arbiter for the volumes.
>
> What do you think about this setup in regards to stability and performance?
> I am running ovirt 4.2.
>
> Thanx,
> Alex
5 years, 7 months
Replica 3 distribute-replicated - data placement and fault tolerance
by Leo David
Hello Everyone,
I did some fio performance tests on a particular vm and I have noticed
things that I do not understand regarding how data is placed along the
bricks. I am sure this is a lack of knowledge, but I would really
appreciate any help in understanding this, I did a bit of research on the
internet, but just could't find something relevant.
I have one replica 3 distributed-replicated arbitrated volume, across 18
bricks ( 9 nodes, 2 jbods per node ).
The volume was created as:
node1 - brick1, node-2, brick1,.....node9-brick1, node1 - brick2, node-2,
brick2,.....node9-brick2
As far as i've understood, under the hood there are sets of 3 times
replicated data ( subvolumes ) which are assigned to the first 3 bricks,
next set of replicated data to the next set of 3 bricks, and so on..
Now, I have this testing vm runign one node one.
When I've started the fio test, i've noticed incresed gluster traffic from
node 1 to node2,4 and 5
So I assumed that the vm disk is data resides on a subvolume allocated to
bricks from these hosts.
Then I have migrated the vm on node 2, and did the same test. Now the
increased treffic is generated from node2 to node1, 4, and 5..
What I do not understand is:
- why gluster client ( ovirt host ) is sending data to 3 bricks if the
volume is arbitrated - shouldn't it send only to 2 of them ?
- why are there 4 brick implicated in this subvolume
- what would it be the fault tolerance level in this setup ,ie: how many
hosts can I take down and still having the volume serving IO requests; can
they be random ?
I am sorry for my lack of knowledge, I am just trying to understand what
is happening so I can deploy a decent proper setup of an hci environment.
Thank you,
Leo
--
Best regards, Leo David
5 years, 7 months
Re: Ovirt gluster arbiter within hosted VM
by Alex K
On Fri, Apr 19, 2019 at 1:14 PM Scott Worthington <
scott.c.worthington(a)gmail.com> wrote:
> And where, magically, is that node's storage going to live?
>
In the disk of the VM. Let me know if I need to clarify further.
>
> You can't fake a proper setup of gluster.
>
Not trying to fake it. Trying to find a solution with the available
options.
>
> On Fri, Apr 19, 2019, 5:00 AM Alex K <rightkicktech(a)gmail.com> wrote:
>
>> Hi all,
>>
>> I have a two node hyper-converged setup which are causing me split-brains
>> when network issues are encountered. Since I cannot add a third hardware
>> node, I was thinking to add a dedicated guest VM hosted in same
>> hyper-converged cluster which would do the arbiter for the volumes.
>>
>> What do you think about this setup in regards to stability and
>> performance?
>> I am running ovirt 4.2.
>>
>> Thanx,
>> Alex
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/LTDOAMBKVE5...
>>
>
5 years, 7 months
