1:12 p.m.
Hi guys,
I see to be having an issue starting VMs with the latest master. Whenever I
try to start a VM I get null pointer exception. And the VM doesn't start. I
have debugged the engine, and it appears that the null pointer happens after
the engine tries to connect to the host. In the stack trace I see
SSLPeerUnverifiedException, so it appears something went wrong with a
certificate somewhere.
I have put my hosts in maintaince and re-enrolled the certificate, but that
doesn't appear to be helping at all. Any other place I need to look at to make
sure the engine can talk to the hosts? This appears to have started after I
upgraded Wildfly to 11, so it is possible it has something to do with that as
well.
Any help figuring this out would be appreciated.
Alexander
2:31 p.m.
On Monday, September 25, 2017 3:24:14 PM EDT Roy Golan wrote:
what JRE are you using? any change with that?
So I just figured out the problem, and its really strange. It has nothing to
do with the SSL as the stack trace is mentioning. I manually stepped through
the code to see what was going on and it turns out it is failing in
FeatureSupported.java in supportedInConfig call from hotPlugMemory.
The Config.<Map>getValue(feature, version.getValue()) (version is 4.2) is
returning a map containing x86=true and ppc=true. But then it compares this to
ArchitectureType.name() it returns null, because .name() return x86_64. No it
appears that sometime during the last few months we dropped the _64 in the
ArchitectureType, or at least in the database.
As soon as I added a vdc_options tha contains x86_64 value for that key it
started working. Now I have checked with Greg who has a fresh database that he
can start VMs no problem, and his database contains x86 instead of x86_64.
2 a.m.
On Mon, Sep 25, 2017 at 10:08 PM, Roy Golan <rgolan(a)redhat.com> wrote:
It looks a lot like introduced here: https://gerrit.ovirt.org/#/c/81464/
@Mirka: what you think?
2:17 a.m.
From 4.0 architecture family was renamed in
script
04_00_0080_rename_architecture_family. So
'HotPlugCpuSupported', 'HotUnplugCpuSupported',
'HotPlugMemorySupported', 'HotUnplugMemorySupported',
'IsMigrationSupported',
'IsMemorySnapshotSupported' and 'IsSuspendSupported' are all in db with
x86
not x86_64. In my point of view nothing wrong with that particular line in
[1].
Could be that somewhere in code is not used architecture family, but host
architecture, when asked for value of this ConfigValues. But that would
throw exception even before my patch, because
'{"x86:"true","ppc":"true"}'
was default value for HotPlugMemorySupported.
[1]
*https://gerrit.ovirt.org/#/c/81464/7/packaging/dbscripts/upgrade/pre_upgrade/0000_config.sql
<https://gerrit.ovirt.org/#/c/81464/7/packaging/dbscripts/upgrade/pre_upgr...
On Tue, Sep 26, 2017 at 9:00 AM, Tomas Jelinek <tjelinek(a)redhat.com> wrote:
2:26 a.m.
On Tue, Sep 26, 2017 at 9:17 AM, Miroslava Voglova <mvoglova(a)redhat.com>
wrote:
I see a code path where the cluster arch can be set to x86_64 - it is
always executed for external VMs (imported from external provider or
unmanaged). It does not happen all the time, it is only a fallback if the
arch type is not known/reported etc.
@Alexander: by any chance, was this VM an unmanaged one? Or imported? In
logs you should find something like:
"Illegal architecture type: {}, replacing with x86_64" or "null
architecture type, replacing with x86_64, {}".
Also, if you create a new VM, can you start it?
3:12 a.m.
I have same issue with new VM :
2017-09-26 11:07:59,255+03 ERROR
[org.ovirt.engine.core.bll.GetArchitectureCapabilitiesQuery] (default
task-2) [c066bc1d-4048-4b5f-bdb6-74fd813aa82e] Query
'GetArchitectureCapabilitiesQuery' failed: null
2017-09-26 11:07:59,255+03 ERROR
[org.ovirt.engine.core.bll.GetArchitectureCapabilitiesQuery] (default
task-2) [c066bc1d-4048-4b5f-bdb6-74fd813aa82e] Exception:
java.lang.NullPointerException
at
org.ovirt.engine.core.common.FeatureSupported.supportedInConfig(FeatureSupported.java:23)
[common.jar:]
at
org.ovirt.engine.core.common.FeatureSupported.hotUnplugMemory(FeatureSupported.java:43)
[common.jar:]
at
org.ovirt.engine.core.bll.GetArchitectureCapabilitiesQuery.isSupported(GetArchitectureCapabilitiesQuery.java:66)
[bll.jar:]
at
org.ovirt.engine.core.bll.GetArchitectureCapabilitiesQuery.getMap(GetArchitectureCapabilitiesQuery.java:36)
[bll.jar:]
at
org.ovirt.engine.core.bll.GetArchitectureCapabilitiesQuery.executeQueryCommand(GetArchitectureCapabilitiesQuery.java:22)
[bll.jar:]
at
org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106)
[bll.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14)
[bll.jar:]
Then:
2017-09-26 11:08:08,397+03 ERROR
[org.ovirt.engine.core.vdsbroker.CreateVDSCommand]
(org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-11)
[9eb76d15-2bd5-49a9-a7d8-c73cfd55282c] Failed to create VM: null
2017-09-26 11:08:08,398+03 ERROR
[org.ovirt.engine.core.vdsbroker.CreateVDSCommand]
(org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-11)
[9eb76d15-2bd5-49a9-a7d8-c73cfd55282c] Command 'CreateVDSCommand(
CreateVDSCommandParameters:{hostId='b6b6a226-8d4f-4929-85d1-b218eceee99e',
vmId='f15ddd07-408b-4665-aede-a9efc5716dc7', vm='VM
[FEDORA_CINDER]'})'
execution failed: java.lang.NullPointerException
On Tue, Sep 26, 2017 at 10:26 AM, Tomas Jelinek <tjelinek(a)redhat.com> wrote:
4:44 a.m.
So we found where is problem. There was patch before some time that was
reverted and had two badly formatted jsons in value of vdc_option. If you
have db from that time, the bug will appear, because the values are not
updated (because they are already in db).
To fix this, its enough to change 'HotPlugMemorySupported' to value
'{"x86":"true","ppc":"true"}' for versions
4.0 - 4.2 and
'HotUnplugMemorySupported' to
'{"x86":"true","ppc":"true"}' for 4.2.
Will do the patch that will include this update in 0000_config in case
anyone else had the same problem.
On Tue, Sep 26, 2017 at 10:12 AM, Fred Rolland <frolland(a)redhat.com> wrote:
5:04 a.m.
Promised patch: https://gerrit.ovirt.org/#/c/82203/
On Tue, Sep 26, 2017 at 11:44 AM, Miroslava Voglova <mvoglova(a)redhat.com>
wrote:
6:58 a.m.
On Tuesday, September 26, 2017 3:26:44 AM EDT Tomas Jelinek wrote:
No its an old database though from pre 4.0 times. These VMs have never been
unmanaged or imported from external providers. I did not see that in the log,
I had to manuall step through the code to end up in the right place that
causes the NPE. Like I said before line 23 in FeatureSupported.java is the
culprit IMO. It does:
String value = archOptions.get(arch.name());
arch is ArchitectureType, and arch.name returns x86_64, and if I understand
right they should have done arch.getFamily().name() which does happen 2 lines
below it. Honestly I don't understand how any VMs are able to run with the
code like that since they all check to see if you can do memory hot plug
before starting, and that check runs through this piece of code, which based
on the contents of [1] should return an npe since the database should not
contain the x86_64 entries.
8:08 a.m.
On Tue, Sep 26, 2017 at 1:58 PM, Alexander Wels <awels(a)redhat.com> wrote:
the reason it did not work is that there was a syntactic error in the
vdc_options table causing the
Config.<Map>getValue(feature, version.getValue()); to return null.
The VMs normally run, because if there is no entry for x86_64 than it
checks x86 two lines below.
2717
days inactive
2718
days old
14 comments
5 participants
participants (5)
-
Alexander Wels -
Fred Rolland -
Miroslava Voglova -
Roy Golan -
Tomas Jelinek