I managed to fix the error with HA broker and agent continually crashing.  I found that it was not a permissions problem on the path mentioned in the log:

/var/run/vdsm/storage/
248f46f0-d793-4581-9810-c9d965e2f286/14a20941-1b84-4b82-be8f-ace38d7c037a/8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8

but this folder had wrong permissions:

_exports_hosted__engine/248f46f0-d793-4581-9810-c9d965e2f286/images/19f114a1-e1c3-41c5-9fcb-b6099612d012

That seems to have resolved the agent and broker problem.  The UI error when clicking network interfaces or snapshots for hosted engine VM still appears to exist.  Here is the ui log from that:

Network interfaces tab for hosted engine VM:

2018-01-14 14:12:06,781-04 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-19) [] Permutation name: C1D1FEFE5DCAE683818762C75C501D92
2018-01-14 14:12:06,781-04 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-19) [] Uncaught exception: com.google.gwt.core.client.JavaScriptException: (TypeError) : Cannot read property 'a' of null
at Unknown.new pvp(webadmin-141.js)
at org.ovirt.engine.ui.webadmin.section.main.view.tab.virtualMachine.SubTabVirtualMachineNetworkInterfaceView.$createListViewItem(SubTabVirtualMachineNetworkInterfaceView.java:69)
at org.ovirt.engine.ui.webadmin.section.main.view.tab.virtualMachine.SubTabVirtualMachineNetworkInterfaceView.createListViewItem(SubTabVirtualMachineNetworkInterfaceView.java:69)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView.$updateInfoPanel(PatternflyListView.java:137)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView.$lambda$1(PatternflyListView.java:63)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView$lambda$1$Type.eventRaised(PatternflyListView.java:63)
at org.ovirt.engine.ui.uicompat.Event.$raise(Event.java:99)
at org.ovirt.engine.ui.uicommonweb.models.SearchableListModel.$setItems(SearchableListModel.java:739)
at org.ovirt.engine.ui.uicommonweb.models.vms.VmInterfaceListModel.$lambda$2(VmInterfaceListModel.java:143)
at org.ovirt.engine.ui.uicommonweb.models.vms.VmInterfaceListModel$lambda$2$Type.executed(VmInterfaceListModel.java:143)
at org.ovirt.engine.ui.frontend.Frontend$2.$onSuccess(Frontend.java:319) [frontend.jar:]
at org.ovirt.engine.ui.frontend.Frontend$2.onSuccess(Frontend.java:319) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.$onSuccess(OperationProcessor.java:170) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.OperationProcessor$2.onSuccess(OperationProcessor.java:170) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.$onSuccess(GWTRPCCommunicationProvider.java:270) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.onSuccess(GWTRPCCommunicationProvider.java:270) [frontend.jar:]
at com.google.gwt.user.client.rpc.impl.RequestCallbackAdapter.onResponseReceived(RequestCallbackAdapter.java:198) [gwt-servlet.jar:]
at com.google.gwt.http.client.Request.$fireOnResponseReceived(Request.java:233) [gwt-servlet.jar:]
at com.google.gwt.http.client.RequestBuilder$1.onReadyStateChange(RequestBuilder.java:409) [gwt-servlet.jar:]
at Unknown.eval(webadmin-0.js)
at com.google.gwt.core.client.impl.Impl.apply(Impl.java:236) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:275) [gwt-servlet.jar:]
at Unknown.eval(webadmin-0.js)



snapshots tab for hosted engine vm:

2018-01-14 14:12:55,628-04 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-14) [] Permutation name: C1D1FEFE5DCAE683818762C75C501D92
2018-01-14 14:12:55,628-04 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-14) [] Uncaught exception: com.google.gwt.core.client.JavaScriptException: (TypeError) : Cannot read property 'a' of null
at org.ovirt.engine.ui.common.widget.uicommon.vm.VmSnapshotListViewItem.$createNicsItemContainerPanel(VmSnapshotListViewItem.java:90)
at org.ovirt.engine.ui.common.widget.uicommon.vm.VmSnapshotListViewItem.$updateValues(VmSnapshotListViewItem.java:387)
at Unknown.new Swp(webadmin-143.js)
at org.ovirt.engine.ui.webadmin.section.main.view.tab.virtualMachine.SubTabVirtualMachineSnapshotView.$createListViewItem(SubTabVirtualMachineSnapshotView.java:68)
at org.ovirt.engine.ui.webadmin.section.main.view.tab.virtualMachine.SubTabVirtualMachineSnapshotView.createListViewItem(SubTabVirtualMachineSnapshotView.java:68)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView.$updateInfoPanel(PatternflyListView.java:137)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView.$lambda$1(PatternflyListView.java:63)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView$lambda$1$Type.eventRaised(PatternflyListView.java:63)
at org.ovirt.engine.ui.uicompat.Event.$raise(Event.java:99)
at org.ovirt.engine.ui.uicommonweb.models.SearchableListModel.$setItems(SearchableListModel.java:739)
at org.ovirt.engine.ui.uicommonweb.models.vms.VmSnapshotListModel.$updateItems(VmSnapshotListModel.java:230)
at org.ovirt.engine.ui.uicommonweb.models.vms.VmSnapshotListModel.setItems(VmSnapshotListModel.java:209)
at org.ovirt.engine.ui.uicommonweb.models.SearchableListModel$SetItemsAsyncQuery$1.$onSuccess(SearchableListModel.java:902)
at org.ovirt.engine.ui.uicommonweb.models.SearchableListModel$SetItemsAsyncQuery$1.onSuccess(SearchableListModel.java:902)
at org.ovirt.engine.ui.frontend.Frontend$1.$onSuccess(Frontend.java:227) [frontend.jar:]
at org.ovirt.engine.ui.frontend.Frontend$1.onSuccess(Frontend.java:227) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.OperationProcessor$1.$onSuccess(OperationProcessor.java:133) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.OperationProcessor$1.onSuccess(OperationProcessor.java:133) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.$onSuccess(GWTRPCCommunicationProvider.java:270) [frontend.jar:]
at org.ovirt.engine.ui.frontend.communication.GWTRPCCommunicationProvider$5$1.onSuccess(GWTRPCCommunicationProvider.java:270) [frontend.jar:]
at com.google.gwt.user.client.rpc.impl.RequestCallbackAdapter.onResponseReceived(RequestCallbackAdapter.java:198) [gwt-servlet.jar:]
at com.google.gwt.http.client.Request.$fireOnResponseReceived(Request.java:233) [gwt-servlet.jar:]
at com.google.gwt.http.client.RequestBuilder$1.onReadyStateChange(RequestBuilder.java:409) [gwt-servlet.jar:]
at Unknown.eval(webadmin-0.js)
at com.google.gwt.core.client.impl.Impl.apply(Impl.java:236) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:275) [gwt-servlet.jar:]
at Unknown.eval(webadmin-0.js)

2018-01-14 14:12:55,673-04 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-23) [] Permutation name: C1D1FEFE5DCAE683818762C75C501D92
2018-01-14 14:12:55,674-04 ERROR [org.ovirt.engine.ui.frontend.server.gwt.OvirtRemoteLoggingService] (default task-23) [] Uncaught exception: com.google.gwt.event.shared.UmbrellaException: Exception caught: (TypeError) : Cannot read property 'a' of null
at java.lang.Throwable.Throwable(Throwable.java:70) [rt.jar:1.8.0_151]
at java.lang.RuntimeException.RuntimeException(RuntimeException.java:32) [rt.jar:1.8.0_151]
at com.google.web.bindery.event.shared.UmbrellaException.UmbrellaException(UmbrellaException.java:64) [gwt-servlet.jar:]
at Unknown.new a0(webadmin-0.js)
at com.google.gwt.event.shared.HandlerManager.$fireEvent(HandlerManager.java:117) [gwt-servlet.jar:]
at com.google.gwt.view.client.SelectionChangeEvent.fire(SelectionChangeEvent.java:67) [gwt-servlet.jar:]
at com.google.gwt.view.client.SingleSelectionModel.$resolveChanges(SingleSelectionModel.java:118) [gwt-servlet.jar:]
at com.google.gwt.view.client.SingleSelectionModel.fireSelectionChangeEvent(SingleSelectionModel.java:107) [gwt-servlet.jar:]
at com.google.gwt.view.client.SelectionModel$AbstractSelectionModel$1.execute(SelectionModel.java:128) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.SchedulerImpl.runScheduledTasks(SchedulerImpl.java:167) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.SchedulerImpl.$flushFinallyCommands(SchedulerImpl.java:272) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.exit(Impl.java:313) [gwt-servlet.jar:]
at com.google.gwt.core.client.impl.Impl.entry0(Impl.java:275) [gwt-servlet.jar:]
at Unknown.eval(webadmin-0.js)
Caused by: com.google.gwt.core.client.JavaScriptException: (TypeError) : Cannot read property 'a' of null
at org.ovirt.engine.ui.common.widget.uicommon.vm.VmSnapshotListViewItem.$createNicsItemContainerPanel(VmSnapshotListViewItem.java:90)
at org.ovirt.engine.ui.common.widget.uicommon.vm.VmSnapshotListViewItem.$updateValues(VmSnapshotListViewItem.java:387)
at Unknown.new Swp(webadmin-143.js)
at org.ovirt.engine.ui.webadmin.section.main.view.tab.virtualMachine.SubTabVirtualMachineSnapshotView.$createListViewItem(SubTabVirtualMachineSnapshotView.java:68)
at org.ovirt.engine.ui.webadmin.section.main.view.tab.virtualMachine.SubTabVirtualMachineSnapshotView.createListViewItem(SubTabVirtualMachineSnapshotView.java:68)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView.$updateInfoPanel(PatternflyListView.java:137)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView.$processSelectionChanged(PatternflyListView.java:126)
at org.ovirt.engine.ui.common.widget.listgroup.PatternflyListView$lambda$0$Type.onSelectionChange(PatternflyListView.java:59)
at com.google.gwt.view.client.SelectionChangeEvent.dispatch(SelectionChangeEvent.java:98) [gwt-servlet.jar:]
at com.google.gwt.event.shared.GwtEvent.dispatch(GwtEvent.java:76) [gwt-servlet.jar:]
at com.google.web.bindery.event.shared.SimpleEventBus.$doFire(SimpleEventBus.java:173) [gwt-servlet.jar:]
... 10 more

On Sun, Jan 14, 2018 at 2:09 PM, Jayme <jaymef@gmail.com> wrote:
I managed to fix the error with HA broker and agent continually crashing.  I found that it was not a permissions problem on the path mentioned in the log:


On Sun, Jan 14, 2018 at 2:07 PM, Greg Sheremeta <gshereme@redhat.com> wrote:


On Sun, Jan 14, 2018 at 8:50 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Sun, Jan 14, 2018 at 3:46 PM, Yedidyah Bar David <didi@redhat.com> wrote:
> On Sun, Jan 14, 2018 at 3:37 PM, Jayme <jaymef@gmail.com> wrote:
>> First, apologies for all the posts to this list lately, I've been having a
>> heck of a time after 4.2 upgrade and you've been helpful, I appreciate that.
>>
>> Since 4.2 upgrade I'm experiencing a few problems that I'm trying to debug.
>>
>> Current status is engine and all hosts are upgraded to 4.2, and cluster and
>> domain set to 4.2 compatibility.  Hosted Engine VM is running and ui
>> accessible etc, all VMs on hosts are running but no HA service.  Web UI is
>> giving a few errors when checking network and snapshots on the hosted engine
>> VM only, it doesn't give errors on any of the others VMs that I spot
>> checked.
>>
>> 1. HA-agent and HA-broker are continually crashing on all three hosts over
>> and over every few seconds.  I sent an email to users list with more details
>> on this problem but unfortunately haven't heard anything back yet.  The
>> general error in the logs seems to be:
>> VolumeDoesNotExist(leafUUID)#012VolumeDoesNotExist: Volume does not exist:
>> (u'8582bdfc-ef54-47af-9f1e-f5b7ec1f1cf8',) --  What?  Volume doesn't exist,
>> why not?
>
> If agent/broker logs do not reveal this, the next step is usually checking
> vdsm logs and/or system logs. Can you please check/share these? Thanks.
>
>>
>> 2. Error when clicking "network interfaces" in the web gui for the hosted VM
>> engine.
>>
>> 3. Similar to #2 above an error is given when clicking "snapshots" in the
>> web gui for the hosted engine VM.
>>
>> The errors for #2 and #3 are generic "cannot read property 'a' of null".
>> I've read previous postings on ovirt-mailing list that suggest you can
>> install debug-info package to get a human readable error.. but this package
>> does not seem to be compatible with 4.2, it expects 4.1:  Requires:
>> "ovirt-engine-webadmin-portal = 4.1.2.2-1.el7.centos" -- Perhaps this
>> package is no longer required?

Indeed: https://bugzilla.redhat.com/show_bug.cgi?id=1431545


Right. ui.log will include the same stack trace that it gives in the browser -- just deobfuscated (automatically now).
 
>>  I do see some additional details in the
>> ui.log that I can post if helpful.

So, yes, please share it.
 
>>
>> There is obviously something odd going on here with the hosted engine VM.
>> All three errors appear to related to a problem with it, although it is
>> indeed up and running.   I'd really like to get HA broker and agent back up
>> and running, and fix these GUI errors related to hosted engine VM.  All
>> three problems may be connected to one common issue?
>>
>> Thanks in advance!
>>
>>
>>
>> _______________________________________________
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>
>
>
> --
> Didi



--
Didi
_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

gshereme@redhat.com    IRC: gshereme