11:55 a.m.
Dear All,
There appears to be an issue with the host install on these, there's quite a lot of
errors being kicked out into the logs as they cycle through attempting to activate,
failing, and then failing to go into maintenance. I can't remove the hosts from the
engine to attempt to reinstall them.
Issue presented from having a statically configured network on the ovritmgmt network on
the hosts before running the host install. There are SQL errors (FK missing) and assorted
goodness in there.
Any help greatly appreciated, not sure what the next step forward is from here
Engine.log on dropbox:
https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
PS. Sorry if this comes through a few times, mailing list membership seems to be having a
funny turn this morning.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
1:37 p.m.
Hi Callum
Looks like you hitting https://bugzilla.redhat.com/show_bug.cgi?id=1570388
Do you have a required network in your env(except ovirtmgmt)? try to
uncheck the 'required' from this network, this will solve the issue(via
Clusters>network or Networks>Cluster).
The bug was fixed in ovirt-engine-4.2.3.4
Cheers)
On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
Dear All,
There appears to be an issue with the host install on these, there's quite
a lot of errors being kicked out into the logs as they cycle through
attempting to activate, failing, and then failing to go into maintenance. I
can't remove the hosts from the engine to attempt to reinstall them.
Issue presented from having a statically configured network on the
ovritmgmt network on the hosts before running the host install. There are
SQL errors (FK missing) and assorted goodness in there.
Any help greatly appreciated, not sure what the next step forward is from
here
Engine.log on dropbox:
https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
PS. Sorry if this comes through a few times, mailing list membership seems
to be having a funny turn this morning.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum(a)well.ox.ac.uk
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
2:39 p.m.
New subject: Host install failed - cannot set maintenance or remove
Dear Michael,
Good to know - didn't know about that path to configure networks.
What's the procedure to remove the failing hosts and re-install? I can't get them
into maintenance mode to then install them.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 12:37, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
Hi Callum
Looks like you hitting https://bugzilla.redhat.com/show_bug.cgi?id=1570388
Do you have a required network in your env(except ovirtmgmt)? try to uncheck the
'required' from this network, this will solve the issue(via Clusters>network or
Networks>Cluster).
The bug was fixed in ovirt-engine-4.2.3.4
Cheers)
On Tue, May 8, 2018 at 12:55 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear All,
There appears to be an issue with the host install on these, there's quite a lot of
errors being kicked out into the logs as they cycle through attempting to activate,
failing, and then failing to go into maintenance. I can't remove the hosts from the
engine to attempt to reinstall them.
Issue presented from having a statically configured network on the ovritmgmt network on
the hosts before running the host install. There are SQL errors (FK missing) and assorted
goodness in there.
Any help greatly appreciated, not sure what the next step forward is from here
Engine.log on dropbox:
https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
PS. Sorry if this comes through a few times, mailing list membership seems to be having a
funny turn this morning.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
![https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...](https://www.redhat.com/files/brand/email/sig-redhat.png]%3Chttps://red.ht/sig%3E){kind=link}
2:44 p.m.
You should only remove the required property from the network, this will
release the host from the activating cycle in which it's stuck now.
On Tue, May 8, 2018 at 3:39 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
Dear Michael,
Good to know - didn't know about that path to configure networks.
What's the procedure to remove the failing hosts and re-install? I can't
get them into maintenance mode to then install them.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum(a)well.ox.ac.uk
On 8 May 2018, at 12:37, Michael Burman <mburman(a)redhat.com> wrote:
Hi Callum
Looks like you hitting https://bugzilla.redhat.com/
show_bug.cgi?id=1570388
Do you have a required network in your env(except ovirtmgmt)? try to
uncheck the 'required' from this network, this will solve the issue(via
Clusters>network or Networks>Cluster).
The bug was fixed in ovirt-engine-4.2.3.4
Cheers)
On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum(a)well.ox.ac.uk>
wrote:
> Dear All,
>
> There appears to be an issue with the host install on these, there's
> quite a lot of errors being kicked out into the logs as they cycle through
> attempting to activate, failing, and then failing to go into maintenance. I
> can't remove the hosts from the engine to attempt to reinstall them.
>
> Issue presented from having a statically configured network on the
> ovritmgmt network on the hosts before running the host install. There are
> SQL errors (FK missing) and assorted goodness in there.
>
> Any help greatly appreciated, not sure what the next step forward is from
> here
>
> Engine.log on dropbox:
> https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
>
> PS. Sorry if this comes through a few times, mailing list membership
> seems to be having a funny turn this morning.
>
> Regards,
> Callum
>
> --
>
> Callum Smith
> Research Computing Core
> Wellcome Trust Centre for Human Genetics
> University of Oxford
> e. callum(a)well.ox.ac.uk
>
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
>
>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
3:42 p.m.
New subject: Host install failed - cannot set maintenance or remove
Dear Michael,
No luck, the cycling is still happening. This is with a hard reset of the host-engine vm,
the host, updating the node versions on everything.
Would it be really bad to just delete the nodes from the DB and clean install them?
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 13:44, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
You should only remove the required property from the network, this will release the host
from the activating cycle in which it's stuck now.
On Tue, May 8, 2018 at 3:39 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear Michael,
Good to know - didn't know about that path to configure networks.
What's the procedure to remove the failing hosts and re-install? I can't get them
into maintenance mode to then install them.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 12:37, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
Hi Callum
Looks like you hitting https://bugzilla.redhat.com/show_bug.cgi?id=1570388
Do you have a required network in your env(except ovirtmgmt)? try to uncheck the
'required' from this network, this will solve the issue(via Clusters>network or
Networks>Cluster).
The bug was fixed in ovirt-engine-4.2.3.4
Cheers)
On Tue, May 8, 2018 at 12:55 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear All,
There appears to be an issue with the host install on these, there's quite a lot of
errors being kicked out into the logs as they cycle through attempting to activate,
failing, and then failing to go into maintenance. I can't remove the hosts from the
engine to attempt to reinstall them.
Issue presented from having a statically configured network on the ovritmgmt network on
the hosts before running the host install. There are SQL errors (FK missing) and assorted
goodness in there.
Any help greatly appreciated, not sure what the next step forward is from here
Engine.log on dropbox:
https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
PS. Sorry if this comes through a few times, mailing list membership seems to be having a
funny turn this morning.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
3:50 p.m.
Have you removed the required property?
Do you have other required networks in this cluster except ovirtmgmt?
Do you see event massages complaining that some network/s is missing in the
cluster?
What versions are your engine and vdsm?
Alona, am i missing something?
On Tue, May 8, 2018 at 4:42 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
Dear Michael,
No luck, the cycling is still happening. This is with a hard reset of the
host-engine vm, the host, updating the node versions on everything.
Would it be really bad to just delete the nodes from the DB and clean
install them?
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum(a)well.ox.ac.uk
On 8 May 2018, at 13:44, Michael Burman <mburman(a)redhat.com> wrote:
You should only remove the required property from the network, this will
release the host from the activating cycle in which it's stuck now.
On Tue, May 8, 2018 at 3:39 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
> Dear Michael,
>
> Good to know - didn't know about that path to configure networks.
>
> What's the procedure to remove the failing hosts and re-install? I can't
> get them into maintenance mode to then install them.
>
> Regards,
> Callum
>
> --
>
> Callum Smith
> Research Computing Core
> Wellcome Trust Centre for Human Genetics
> University of Oxford
> e. callum(a)well.ox.ac.uk
>
> On 8 May 2018, at 12:37, Michael Burman <mburman(a)redhat.com> wrote:
>
> Hi Callum
> Looks like you hitting https://bugzilla.redhat.com/sh
> ow_bug.cgi?id=1570388
> Do you have a required network in your env(except ovirtmgmt)? try to
> uncheck the 'required' from this network, this will solve the issue(via
> Clusters>network or Networks>Cluster).
> The bug was fixed in ovirt-engine-4.2.3.4
>
> Cheers)
>
> On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum(a)well.ox.ac.uk>
> wrote:
>
>> Dear All,
>>
>> There appears to be an issue with the host install on these, there's
>> quite a lot of errors being kicked out into the logs as they cycle through
>> attempting to activate, failing, and then failing to go into maintenance. I
>> can't remove the hosts from the engine to attempt to reinstall them.
>>
>> Issue presented from having a statically configured network on the
>> ovritmgmt network on the hosts before running the host install. There are
>> SQL errors (FK missing) and assorted goodness in there.
>>
>> Any help greatly appreciated, not sure what the next step forward is
>> from here
>>
>> Engine.log on dropbox:
>> https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
>>
>> PS. Sorry if this comes through a few times, mailing list membership
>> seems to be having a funny turn this morning.
>>
>> Regards,
>> Callum
>>
>> --
>>
>> Callum Smith
>> Research Computing Core
>> Wellcome Trust Centre for Human Genetics
>> University of Oxford
>> e. callum(a)well.ox.ac.uk
>>
>>
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>>
>>
>
>
> --
> Michael Burman
>
> Senior Quality engineer - rhv network - redhat israel
> Red Hat
>
> <https://www.redhat.com/>
>
> mburman(a)redhat.com M: 0545355725 IM: mburman
> <https://red.ht/sig>
>
>
>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
4:03 p.m.
New subject: Host install failed - cannot set maintenance or remove
I've attached a couple of screenshots, the hosts seem to not succeed in going into
non-operational mode. Required network is now unticked. Storage is definitely accessible
by all nodes + engine.
The moment they succeed into going non-operational, they cycle back to activating.
It's very tempting to run a DELETE FROM `vds` WHERE `vds_name` =
'virtA003.cluster';
2018-05-08 14:54:32,351+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (E
E-ManagedThreadFactory-engineScheduled-Thread-30) [] FINISH,
GetHardwareInfoAsyncVDSCommand, log id: 313cfe33
2018-05-08 14:54:32,353+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory
-engineScheduled-Thread-44) [31fd8e91] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[
04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS_INIT]', sharedLocks=''}'
2018-05-08 14:54:32,385+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFact
ory-engineScheduled-Thread-30) [32499b7b] Running command: SetNonOperationalVdsCommand
internal: true. Entitie
s affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDS
2018-05-08 14:54:32,388+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFac
tory-engineScheduled-Thread-30) [32499b7b] START, SetVdsStatusVDSCommand(HostName =
virtA002.cluster, SetVdsSt
atusVDSCommandParameters:{hostId='e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7',
status='NonOperational', nonOperation
alReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false',
maintenanceReason='null'}), log id: 3b5138fd
2018-05-08 14:54:32,705+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFac
tory-engineScheduled-Thread-36) [3c65dca0] FINISH, SetVdsStatusVDSCommand, log id:
44441e64
2018-05-08 14:54:32,729+01 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFact
ory-engineScheduled-Thread-36) [3c65dca0] Host 'virtA003.cluster' is set to
Non-Operational, it is missing the
following networks: 'ovirtmgmt'
2018-05-08 14:54:32,740+01 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-Ma
nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
irtA003.cluster does not comply with the cluster Default networks, the following networks
are missing on host:
'ovirtmgmt'
2018-05-08 14:54:32,743+01 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler]
(EE-ManagedThreadFactory-eng
ineScheduled-Thread-36) [3c65dca0] Exception:
org.springframework.jdbc.CannotGetJdbcConnectionException: Could
not get JDBC Connection; nested exception is java.sql.SQLException:
javax.resource.ResourceException: IJ00045
7: Unchecked throwable in managedConnectionReconnected()
cl=org.jboss.jca.core.connectionmanager.listener.TxCo
nnectionListener@769d34fc[state=NORMAL managed
connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnec
tion@1444dfba connection handles=0 lastReturned=1525787672740 lastValidated=1525786889842
lastCheckedOut=15257
87372702 trackByTx=false
pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp=Semaphor
eConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=ENGINEDataSource]
xaResource=LocalXAResourceImpl@3aa2
9b7a[connectionListener=769d34fc connectionManager=5d6abc6c warned=false currentXid=null
productName=PostgreSQ
L productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
[spring-
jdbc.jar:4.3.9.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619)
[spring-jdbc.jar:4.3.9.RE
LEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:766)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallIntern
al(PostgresDbEngineDialect.java:152) [dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(Postgre
sDbEngineDialect.java:118) [dal.jar:]
at
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198)
[spring-jdbc.j
ar:4.3.9.RELEASE]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:1
35) [dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.ja
va:105) [dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:9
7) [dal.jar:]
at org.ovirt.engine.core.dao.JobDaoImpl.checkIfJobHasTasks(JobDaoImpl.java:149)
[dal.jar:]
at
org.ovirt.engine.core.bll.job.ExecutionHandler.checkIfJobHasTasks(ExecutionHandler.java:893)
[bll.j
ar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1368)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:400)
[bll.jar:]
at
org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecuto
r.java:13) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:468) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:450) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:656)
[bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_161]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0
_161]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterce
ptor.java:78)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterce
ptor.java:88)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsIntercepto
r.java:101)
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFa
ctory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionT
imeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextIntercep
tor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDi
spatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvoc
ation(SingletonComponentInstanceAssociationInterceptor.java:53)
[wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255)
[wildfly-ejb3-11.
0.0.Final.jar:11.0.0.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:381)
[wildfly-ejb3-11.0.0.Fina
l.jar:11.0.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244)
[wildfly-ejb3-11
.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScop
eActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivat
ionInterceptor.java:89)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(Curr
entInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterce
ptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.
java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInter
ceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInt
erceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.ja
va:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor
.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.
java:60)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
at
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
at
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown
Source) [bll
.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_161]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0
_161]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
at
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433)
[weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.
java:127) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPoi
ntPropagatingEnterpriseTargetBeanInstance.java:67)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
[weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
at
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Prox
y$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:]
at
org.ovirt.engine.core.bll.VdsEventListener.vdsNonOperational(VdsEventListener.java:303)
[bll.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.setNonOperational(HostNe
tworkTopologyPersisterImpl.java:340) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.enforceNetworkCompliance
(HostNetworkTopologyPersisterImpl.java:121) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.lambda$persistAndEnforce
NetworkCompliance$0(HostNetworkTopologyPersisterImpl.java:99) [vdsbroker.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSuppo
rt.java:202) [utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.jav
a:137) [utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:1
05) [utils.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:93) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:154) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.processRefreshCapabilitiesResponse(VdsManager.java:794)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring$BeforeFirstRefreshTreatmentCallback.onRes
ponse(HostMonitoring.java:767) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand$GetCapabilitiesVDSCommandC
allback.onResponse(GetCapabilitiesAsyncVDSCommand.java:45) [vdsbroker.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.lambda$processResponse$1(JsonRpcClient.java:182)
[vdsm-
jsonrpc-java-client.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_161]
at
org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141)
[jav
ax.enterprise.concurrent-1.0.jar:]
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383)
[javax.enterprise.concurrent-1.0.jar:]
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.run(ManagedScheduledThreadPoolExecutor.java:532)
[javax.enterprise.concurrent-1.0.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
at
org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactory
Impl.java:250) [javax.enterprise.concurrent-1.0.jar:]
at
org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronMana
gedThreadFactory.java:78)
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked
throwable in managedCo
nnectionReconnected()
cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NOR
MAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba
connection handles=0
lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=1525787372702
trackByTx=false pool=org.j
boss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4
mcp=SemaphoreConcurrentLinkedQueueManagedConnec
tionPool@78b9d77[pool=ENGINEDataSource]
xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc co
nnectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQL
productVersion=9.5.9 jndiName=jav
a:/ENGINEDataSource] txSync=null]
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
at
org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.jav
a:64)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
[spri
ng-jdbc.jar:4.3.9.RELEASE]
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
[spring-
jdbc.jar:4.3.9.RELEASE]
... 107 more
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in
managedConnectionReconnected() c
l=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NORMAL
managed connection=
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba connection handles=0
lastReturned=1525787672
740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false
pool=org.jboss.jca.core.connectio
nmanager.pool.strategy.OnePool@55129ed4
mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=E
NGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc
connectionManager=5d6abc6
c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9
jndiName=java:/ENGINEDataSource] tx
Sync=null]
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractC
onnectionManager.java:975)
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectio
nManager.java:792)
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
... 110 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on
entering meta-aware object
at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:561)
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:970)
... 112 more
Caused by: java.lang.IllegalStateException: Transaction Local transaction
(delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2
status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA
transaction provider) is not active STATUS_ROLLEDBACK
at
org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:296)
at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:554)
... 113 more
2018-05-08 14:54:32,749+01 INFO
[org.ovirt.engine.core.utils.transaction.TransactionSupport]
(EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] transaction rolled back
2018-05-08 14:54:32,749+01 ERROR
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] Unable to
RefreshCapabilities beforeFirstRefreshTreatment: IllegalStateException: Transaction Local
transaction (delegate=TransactionImple < ac, BasicAction:
0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local
transaction context for provider JBoss JTA transaction provider) is not active
STATUS_ROLLEDBACK
2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock acquired, from now a
monitoring of host will be skipped for host 'virtA003.cluster' from data-center
'Default'
2018-05-08 14:54:32,768+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] START,
SetVdsStatusVDSCommand(HostName = virtA003.cluster,
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
status='Unassigned', nonOperationalReason='NONE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
635dd4c4
2018-05-08 14:54:32,771+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] FINISH,
SetVdsStatusVDSCommand, log id: 635dd4c4
2018-05-08 14:54:32,774+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Activate host finished.
Lock released. Monitoring can run now for host 'virtA003.cluster' from data-center
'Default'
2018-05-08 14:54:32,775+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock freed to object
'EngineLock:{exclusiveLocks='[04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS]',
sharedLocks=''}'
2018-05-08 14:54:35,736+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [] START,
GetHardwareInfoAsyncVDSCommand(HostName = virtA003.cluster,
VdsIdAndVdsVDSCommandParametersBase:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
vds='Host[virtA003.cluster,04b8698b-fbfe-4efe-afa5-2cb604cbdb3d]'}), log id:
710291f0
2018-05-08 14:54:35,736+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [] FINISH,
GetHardwareInfoAsyncVDSCommand, log id: 710291f0
2018-05-08 14:54:35,774+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
04b8698b-fbfe-4efe-afa5-2cb604cbdb3d Type: VDS
2018-05-08 14:54:35,776+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] START,
SetVdsStatusVDSCommand(HostName = virtA003.cluster,
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
707d7be2
2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering 1 hosts
2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering hosts id:
e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 , name : virtA002.cluster
2018-05-08 14:55:00,042+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Lock Acquired to object
'EngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS]',
sharedLocks=''}'
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Running command:
ActivateVdsCommand internal: true. Entities affected : ID:
e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDSAction group MANIPULATE_HOST with role type
ADMIN
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Before acquiring lock in
order to prevent monitoring for host 'virtA002.cluster' from data-center
'Default'
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and
wait lock
'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]',
sharedLocks=''}'
2018-05-08 14:55:02,410+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and
wait lock
'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]',
sharedLocks=''}'
Regards,
Callum
[cid:672081BE-E695-46B2-965B-D34194DE1D60@well.ox.ac.uk][cid:030D44DD-3DED-412D-856D-E7A458E495AE@well.ox.ac.uk]
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 14:50, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
Have you removed the required property?
Do you have other required networks in this cluster except ovirtmgmt?
Do you see event massages complaining that some network/s is missing in the cluster?
What versions are your engine and vdsm?
Alona, am i missing something?
On Tue, May 8, 2018 at 4:42 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear Michael,
No luck, the cycling is still happening. This is with a hard reset of the host-engine vm,
the host, updating the node versions on everything.
Would it be really bad to just delete the nodes from the DB and clean install them?
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 13:44, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
You should only remove the required property from the network, this will release the host
from the activating cycle in which it's stuck now.
On Tue, May 8, 2018 at 3:39 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear Michael,
Good to know - didn't know about that path to configure networks.
What's the procedure to remove the failing hosts and re-install? I can't get them
into maintenance mode to then install them.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 12:37, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
Hi Callum
Looks like you hitting https://bugzilla.redhat.com/show_bug.cgi?id=1570388
Do you have a required network in your env(except ovirtmgmt)? try to uncheck the
'required' from this network, this will solve the issue(via Clusters>network or
Networks>Cluster).
The bug was fixed in ovirt-engine-4.2.3.4
Cheers)
On Tue, May 8, 2018 at 12:55 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear All,
There appears to be an issue with the host install on these, there's quite a lot of
errors being kicked out into the logs as they cycle through attempting to activate,
failing, and then failing to go into maintenance. I can't remove the hosts from the
engine to attempt to reinstall them.
Issue presented from having a statically configured network on the ovritmgmt network on
the hosts before running the host install. There are SQL errors (FK missing) and assorted
goodness in there.
Any help greatly appreciated, not sure what the next step forward is from here
Engine.log on dropbox:
https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
PS. Sorry if this comes through a few times, mailing list membership seems to be having a
funny turn this morning.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
4:17 p.m.
Ok,
2018-05-08 14:54:32,740+01 WARN [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (EE-Ma
nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
irtA003.cluster does not comply with the cluster Default networks, the
following networks are missing on host:
'ovirtmgmt'
Now ovirtmgmt is missing, you can do two things:
1) GO to Setup networks dialogue(under host) and drag the ovirtmgmt on the
active NIC, this should work. If not
2) You should be able to set to maintenance now and re-install the host
On Tue, May 8, 2018 at 5:03 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
I've attached a couple of screenshots, the hosts seem to not
succeed in
going into non-operational mode. Required network is now unticked. Storage
is definitely accessible by all nodes + engine.
The moment they succeed into going non-operational, they cycle back to
activating.
It's very tempting to run a DELETE FROM `vds` WHERE `vds_name` =
'virtA003.cluster';
2018-05-08 14:54:32,351+01 INFO [org.ovirt.engine.core.
vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (E
E-ManagedThreadFactory-engineScheduled-Thread-30) [] FINISH,
GetHardwareInfoAsyncVDSCommand, log id: 313cfe33
2018-05-08 14:54:32,353+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory
-engineScheduled-Thread-44) [31fd8e91] Failed to acquire lock and wait
lock 'HostEngineLock:{exclusiveLocks='[
04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS_INIT]', sharedLocks=''}'
2018-05-08 14:54:32,385+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFact
ory-engineScheduled-Thread-30) [32499b7b] Running command:
SetNonOperationalVdsCommand internal: true. Entitie
s affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDS
2018-05-08 14:54:32,388+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFac
tory-engineScheduled-Thread-30) [32499b7b] START, SetVdsStatusVDSCommand(HostName
= virtA002.cluster, SetVdsSt
atusVDSCommandParameters:{hostId='e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7',
status='NonOperational', nonOperation
alReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false',
maintenanceReason='null'}), log id: 3b5138fd
2018-05-08 14:54:32,705+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFac
tory-engineScheduled-Thread-36) [3c65dca0] FINISH,
SetVdsStatusVDSCommand, log id: 44441e64
2018-05-08 14:54:32,729+01 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFact
ory-engineScheduled-Thread-36) [3c65dca0] Host 'virtA003.cluster' is set
to Non-Operational, it is missing the
following networks: 'ovirtmgmt'
2018-05-08 14:54:32,740+01 WARN [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (EE-Ma
nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
irtA003.cluster does not comply with the cluster Default networks, the
following networks are missing on host:
'ovirtmgmt'
2018-05-08 14:54:32,743+01 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler]
(EE-ManagedThreadFactory-eng
ineScheduled-Thread-36) [3c65dca0] Exception: org.springframework.jdbc.
CannotGetJdbcConnectionException: Could
not get JDBC Connection; nested exception is java.sql.SQLException:
javax.resource.ResourceException: IJ00045
7: Unchecked throwable in managedConnectionReconnected()
cl=org.jboss.jca.core.connectionmanager.listener.TxCo
nnectionListener@769d34fc[state=NORMAL managed connection=org.jboss.jca.
adapters.jdbc.local.LocalManagedConnec
tion@1444dfba connection handles=0 lastReturned=1525787672740
lastValidated=1525786889842 lastCheckedOut=15257
87372702 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.
strategy.OnePool@55129ed4 mcp=Semaphor
eConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=ENGINEDataSource]
xaResource=LocalXAResourceImpl@3aa2
9b7a[connectionListener=769d34fc connectionManager=5d6abc6c warned=false
currentXid=null productName=PostgreSQ
L productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
at org.springframework.jdbc.datasource.DataSourceUtils.
getConnection(DataSourceUtils.java:80) [spring-
jdbc.jar:4.3.9.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619)
[spring-jdbc.jar:4.3.9.RE
LEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:766)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$
PostgresSimpleJdbcCall.executeCallIntern
al(PostgresDbEngineDialect.java:152) [dal.jar:]
at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$
PostgresSimpleJdbcCall.doExecute(Postgre
sDbEngineDialect.java:118) [dal.jar:]
at
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198)
[spring-jdbc.j
ar:4.3.9.RELEASE]
at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.
executeImpl(SimpleJdbcCallsHandler.java:1
35) [dal.jar:]
at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.
executeReadList(SimpleJdbcCallsHandler.ja
va:105) [dal.jar:]
at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.
executeRead(SimpleJdbcCallsHandler.java:9
7) [dal.jar:]
at org.ovirt.engine.core.dao.JobDaoImpl.checkIfJobHasTasks(JobDaoImpl.java:149)
[dal.jar:]
at org.ovirt.engine.core.bll.job.ExecutionHandler.
checkIfJobHasTasks(ExecutionHandler.java:893) [bll.j
ar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1368)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:400)
[bll.jar:]
at org.ovirt.engine.core.bll.executor.
DefaultBackendActionExecutor.execute(DefaultBackendActionExecuto
r.java:13) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:468)
[bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:450)
[bll.jar:]
at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:656)
[bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0
_161]
at java.lang.reflect.Method.invoke(Method.java:498)
[rt.jar:1.8.0_161]
at org.jboss.as.ee.component.ManagedReferenceMethodIntercep
tor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext$Invocation.
proceed(InterceptorContext.java:509)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.
delegateInterception(Jsr299BindingsInterce
ptor.java:78)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.
doMethodInterception(Jsr299BindingsInterce
ptor.java:88)
at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.
processInvocation(Jsr299BindingsIntercepto
r.java:101)
at org.jboss.as.ee.component.interceptors.
UserInterceptorFactory$1.processInvocation(UserInterceptorFa
ctory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.component.invocationmetrics.
ExecutionTimeInterceptor.processInvocation(ExecutionT
imeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.
processInvocation(ConcurrentContextIntercep
tor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.invocation.InitialInterceptor.processInvocation(
InitialInterceptor.java:40)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(
ChainedInterceptor.java:53)
at org.jboss.as.ee.component.interceptors.
ComponentDispatcherInterceptor.processInvocation(ComponentDi
spatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.component.singleton.
SingletonComponentInstanceAssociationInterceptor.processInvoc
ation(SingletonComponentInstanceAssociationInterceptor.java:53)
[wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255)
[wildfly-ejb3-11.
0.0.Final.jar:11.0.0.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:381)
[wildfly-ejb3-11.0.0.Fina
l.jar:11.0.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244)
[wildfly-ejb3-11
.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext$Invocation.
proceed(InterceptorContext.java:509)
at org.jboss.weld.ejb.AbstractEJBRequestScopeActivat
ionInterceptor.aroundInvoke(AbstractEJBRequestScop
eActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.
jar:2.4.3.Final]
at org.jboss.as.weld.ejb.EjbRequestScopeActivationInter
ceptor.processInvocation(EjbRequestScopeActivat
ionInterceptor.java:89)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.
CurrentInvocationContextInterceptor.processInvocation(Curr
entInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.
jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.component.invocationmetrics.
WaitTimeInterceptor.processInvocation(WaitTimeInterce
ptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.security.SecurityContextInterceptor.
processInvocation(SecurityContextInterceptor.
java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.deployment.processors.
StartupAwaitInterceptor.processInvocation(StartupAwaitInter
ceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.
ShutDownInterceptorFactory$1.processInvocation(ShutDownInt
erceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.
processInvocation(LoggingInterceptor.ja
va:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.as.ee.component.NamespaceContextInterceptor.
processInvocation(NamespaceContextInterceptor
.java:50)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.invocation.ContextClassLoaderInterceptor.
processInvocation(ContextClassLoaderInterceptor.
java:60)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext.run(
InterceptorContext.java:438)
at org.wildfly.security.manager.WildFlySecurityManager.doChecked(
WildFlySecurityManager.java:609)
at org.jboss.invocation.AccessCheckingInterceptor.
processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation.InterceptorContext.proceed(
InterceptorContext.java:422)
at org.jboss.invocation.ChainedInterceptor.processInvocation(
ChainedInterceptor.java:53)
at org.jboss.as.ee.component.ViewService$View.invoke(
ViewService.java:198)
at org.jboss.as.ee.component.ViewDescription$1.processInvocation(
ViewDescription.java:185)
at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(
ProxyInvocationHandler.java:81)
at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$
view3.runInternalAction(Unknown Source) [bll
.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_161]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0
_161]
at java.lang.reflect.Method.invoke(Method.java:498)
[rt.jar:1.8.0_161]
at org.jboss.weld.util.reflection.Reflections.
invokeAndUnwrap(Reflections.java:433) [weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandl
er.invoke(EnterpriseBeanProxyMethodHandler.
java:127) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(
EnterpriseTargetBeanInstance.java:56)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnter
priseTargetBeanInstance.invoke(InjectionPoi
ntPropagatingEnterpriseTargetBeanInstance.java:67)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
[weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$
BackendInternal$BackendLocal$2049259618$Prox
y$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:]
at
org.ovirt.engine.core.bll.VdsEventListener.vdsNonOperational(VdsEventListener.java:303)
[bll.jar:]
at org.ovirt.engine.core.vdsbroker.vdsbroker.
HostNetworkTopologyPersisterImpl.setNonOperational(HostNe
tworkTopologyPersisterImpl.java:340) [vdsbroker.jar:]
at org.ovirt.engine.core.vdsbroker.vdsbroker.
HostNetworkTopologyPersisterImpl.enforceNetworkCompliance
(HostNetworkTopologyPersisterImpl.java:121) [vdsbroker.jar:]
at org.ovirt.engine.core.vdsbroker.vdsbroker.
HostNetworkTopologyPersisterImpl.lambda$persistAndEnforce
NetworkCompliance$0(HostNetworkTopologyPersisterImpl.java:99)
[vdsbroker.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.
executeInNewTransaction(TransactionSuppo
rt.java:202) [utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.
executeInRequired(TransactionSupport.jav
a:137) [utils.jar:]
at org.ovirt.engine.core.utils.transaction.TransactionSupport.
executeInScope(TransactionSupport.java:1
05) [utils.jar:]
at org.ovirt.engine.core.vdsbroker.vdsbroker.
HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:93) [vdsbroker.jar:]
at org.ovirt.engine.core.vdsbroker.vdsbroker.
HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:154) [vdsbroker.jar:]
at org.ovirt.engine.core.vdsbroker.VdsManager.
processRefreshCapabilitiesResponse(VdsManager.java:794)
[vdsbroker.jar:]
at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring$
BeforeFirstRefreshTreatmentCallback.onRes
ponse(HostMonitoring.java:767) [vdsbroker.jar:]
at org.ovirt.engine.core.vdsbroker.vdsbroker.
GetCapabilitiesAsyncVDSCommand$GetCapabilitiesVDSCommandC
allback.onResponse(GetCapabilitiesAsyncVDSCommand.java:45)
[vdsbroker.jar:]
at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.lambda$
processResponse$1(JsonRpcClient.java:182) [vdsm-
jsonrpc-java-client.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_161]
at org.glassfish.enterprise.concurrent.internal.
ManagedFutureTask.run(ManagedFutureTask.java:141) [jav
ax.enterprise.concurrent-1.0.jar:]
at org.glassfish.enterprise.concurrent.internal.
ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383)
[javax.enterprise.concurrent-1.0.jar:]
at org.glassfish.enterprise.concurrent.internal.
ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.run(ManagedScheduledThreadPoolExecutor.java:532)
[javax.enterprise.concurrent-1.0.jar:]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_161]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$
ManagedThread.run(ManagedThreadFactory
Impl.java:250) [javax.enterprise.concurrent-1.0.jar:]
at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$
ElytronManagedThread.run(ElytronMana
gedThreadFactory.java:78)
Caused by: java.sql.SQLException: javax.resource.ResourceException:
IJ000457: Unchecked throwable in managedCo
nnectionReconnected() cl=org.jboss.jca.core.connectionmanager.listener.
TxConnectionListener@769d34fc[state=NOR
MAL managed connection=org.jboss.jca.adapters.jdbc.local.
LocalManagedConnection@1444dfba connection handles=0
lastReturned=1525787672740 lastValidated=1525786889842
lastCheckedOut=1525787372702 trackByTx=false pool=org.j
boss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp=
SemaphoreConcurrentLinkedQueueManagedConnec
tionPool@78b9d77[pool=ENGINEDataSource] xaResource=
LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc co
nnectionManager=5d6abc6c warned=false currentXid=null
productName=PostgreSQL productVersion=9.5.9 jndiName=jav
a:/ENGINEDataSource] txSync=null]
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(
WrapperDataSource.java:146)
at org.jboss.as.connector.subsystems.datasources.
WildFlyDataSource.getConnection(WildFlyDataSource.jav
a:64)
at org.springframework.jdbc.datasource.DataSourceUtils.
doGetConnection(DataSourceUtils.java:111) [spri
ng-jdbc.jar:4.3.9.RELEASE]
at org.springframework.jdbc.datasource.DataSourceUtils.
getConnection(DataSourceUtils.java:77) [spring-
jdbc.jar:4.3.9.RELEASE]
... 107 more
Caused by: javax.resource.ResourceException: IJ000457: Unchecked
throwable in managedConnectionReconnected() c
l=org.jboss.jca.core.connectionmanager.listener.
TxConnectionListener@769d34fc[state=NORMAL managed connection=
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba
connection handles=0 lastReturned=1525787672
740 lastValidated=1525786889842 lastCheckedOut=1525787372702
trackByTx=false pool=org.jboss.jca.core.connectio
nmanager.pool.strategy.OnePool@55129ed4 mcp=SemaphoreConcurrentLinkedQueue
ManagedConnectionPool@78b9d77[pool=E
NGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc
connectionManager=5d6abc6
c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9
jndiName=java:/ENGINEDataSource] tx
Sync=null]
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.
reconnectManagedConnection(AbstractC
onnectionManager.java:975)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.
allocateConnection(AbstractConnectio
nManager.java:792)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(
WrapperDataSource.java:138)
... 110 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist
in transaction on entering meta-aware object
at org.jboss.jca.core.connectionmanager.tx.
TxConnectionManagerImpl.managedConnectionReconnected(
TxConnectionManagerImpl.java:561)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.
reconnectManagedConnection(AbstractConnectionManager.java:970)
... 112 more
Caused by: java.lang.IllegalStateException: Transaction Local transaction
(delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2
status: ActionStatus.ABORTED >, owner=Local transaction context for
provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK
at org.jboss.jca.core.connectionmanager.listener.
TxConnectionListener.enlist(TxConnectionListener.java:296)
at org.jboss.jca.core.connectionmanager.tx.
TxConnectionManagerImpl.managedConnectionReconnected(
TxConnectionManagerImpl.java:554)
... 113 more
2018-05-08 14:54:32,749+01 INFO
[org.ovirt.engine.core.utils.transaction.TransactionSupport]
(EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0]
transaction rolled back
2018-05-08 14:54:32,749+01 ERROR [org.ovirt.engine.core.
vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-36)
[3c65dca0] Unable to RefreshCapabilities beforeFirstRefreshTreatment:
IllegalStateException: Transaction Local transaction
(delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2
status: ActionStatus.ABORTED >, owner=Local transaction context for
provider JBoss JTA transaction provider) is not active STATUS_ROLLEDBACK
2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock
acquired, from now a monitoring of host will be skipped for host
'virtA003.cluster' from data-center 'Default'
2018-05-08 14:54:32,768+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] START,
SetVdsStatusVDSCommand(HostName = virtA003.cluster,
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
status='Unassigned', nonOperationalReason='NONE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
635dd4c4
2018-05-08 14:54:32,771+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] FINISH,
SetVdsStatusVDSCommand, log id: 635dd4c4
2018-05-08 14:54:32,774+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Activate
host finished. Lock released. Monitoring can run now for host
'virtA003.cluster' from data-center 'Default'
2018-05-08 14:54:32,775+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock freed
to object
'EngineLock:{exclusiveLocks='[04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS]',
sharedLocks=''}'
2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core.
vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [] START,
GetHardwareInfoAsyncVDSCommand(HostName = virtA003.cluster,
VdsIdAndVdsVDSCommandParametersBase:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
vds='Host[virtA003.cluster,04b8698b-fbfe-4efe-afa5-2cb604cbdb3d]'}), log
id: 710291f0
2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core.
vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [] FINISH,
GetHardwareInfoAsyncVDSCommand, log id: 710291f0
2018-05-08 14:54:35,774+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] Running
command: SetNonOperationalVdsCommand internal: true. Entities affected :
ID: 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d Type: VDS
2018-05-08 14:54:35,776+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] START,
SetVdsStatusVDSCommand(HostName = virtA003.cluster,
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
707d7be2
2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering 1
hosts
2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering
hosts id: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 , name : virtA002.cluster
2018-05-08 14:55:00,042+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Lock
Acquired to object
'EngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS]',
sharedLocks=''}'
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Running
command: ActivateVdsCommand internal: true. Entities affected : ID:
e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDSAction group
MANIPULATE_HOST with role type ADMIN
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Before
acquiring lock in order to prevent monitoring for host 'virtA002.cluster'
from data-center 'Default'
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to
acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e-
b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}'
2018-05-08 14:55:02,410+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to
acquire lock and wait lock 'HostEngineLock:{exclusiveLocks='[e8daef6e-
b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]', sharedLocks=''}'
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum(a)well.ox.ac.uk
On 8 May 2018, at 14:50, Michael Burman <mburman(a)redhat.com> wrote:
Have you removed the required property?
Do you have other required networks in this cluster except ovirtmgmt?
Do you see event massages complaining that some network/s is missing in
the cluster?
What versions are your engine and vdsm?
Alona, am i missing something?
On Tue, May 8, 2018 at 4:42 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
> Dear Michael,
>
> No luck, the cycling is still happening. This is with a hard reset of the
> host-engine vm, the host, updating the node versions on everything.
>
> Would it be really bad to just delete the nodes from the DB and clean
> install them?
>
> Regards,
> Callum
>
> --
>
> Callum Smith
> Research Computing Core
> Wellcome Trust Centre for Human Genetics
> University of Oxford
> e. callum(a)well.ox.ac.uk
>
> On 8 May 2018, at 13:44, Michael Burman <mburman(a)redhat.com> wrote:
>
> You should only remove the required property from the network, this will
> release the host from the activating cycle in which it's stuck now.
>
> On Tue, May 8, 2018 at 3:39 PM, Callum Smith <callum(a)well.ox.ac.uk>
> wrote:
>
>> Dear Michael,
>>
>> Good to know - didn't know about that path to configure networks.
>>
>> What's the procedure to remove the failing hosts and re-install? I can't
>> get them into maintenance mode to then install them.
>>
>> Regards,
>> Callum
>>
>> --
>>
>> Callum Smith
>> Research Computing Core
>> Wellcome Trust Centre for Human Genetics
>> University of Oxford
>> e. callum(a)well.ox.ac.uk
>>
>> On 8 May 2018, at 12:37, Michael Burman <mburman(a)redhat.com> wrote:
>>
>> Hi Callum
>> Looks like you hitting https://bugzilla.redhat.com/sh
>> ow_bug.cgi?id=1570388
>> Do you have a required network in your env(except ovirtmgmt)? try to
>> uncheck the 'required' from this network, this will solve the issue(via
>> Clusters>network or Networks>Cluster).
>> The bug was fixed in ovirt-engine-4.2.3.4
>>
>> Cheers)
>>
>> On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum(a)well.ox.ac.uk>
>> wrote:
>>
>>> Dear All,
>>>
>>> There appears to be an issue with the host install on these, there's
>>> quite a lot of errors being kicked out into the logs as they cycle through
>>> attempting to activate, failing, and then failing to go into maintenance. I
>>> can't remove the hosts from the engine to attempt to reinstall them.
>>>
>>> Issue presented from having a statically configured network on the
>>> ovritmgmt network on the hosts before running the host install. There are
>>> SQL errors (FK missing) and assorted goodness in there.
>>>
>>> Any help greatly appreciated, not sure what the next step forward is
>>> from here
>>>
>>> Engine.log on dropbox:
>>> https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
>>>
>>> PS. Sorry if this comes through a few times, mailing list membership
>>> seems to be having a funny turn this morning.
>>>
>>> Regards,
>>> Callum
>>>
>>> --
>>>
>>> Callum Smith
>>> Research Computing Core
>>> Wellcome Trust Centre for Human Genetics
>>> University of Oxford
>>> e. callum(a)well.ox.ac.uk
>>>
>>>
>>> _______________________________________________
>>> Users mailing list -- users(a)ovirt.org
>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>
>>>
>>
>>
>> --
>> Michael Burman
>>
>> Senior Quality engineer - rhv network - redhat israel
>> Red Hat
>>
>> <https://www.redhat.com/>
>>
>> mburman(a)redhat.com M: 0545355725 IM: mburman
>> <https://red.ht/sig>
>>
>>
>>
>
>
> --
> Michael Burman
>
> Senior Quality engineer - rhv network - redhat israel
> Red Hat
>
> <https://www.redhat.com/>
>
> mburman(a)redhat.com M: 0545355725 IM: mburman
> <https://red.ht/sig>
>
>
>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
4:20 p.m.
New subject: Host install failed - cannot set maintenance or remove
At the moment network interfaces when i look at the host in the engine has no items to
display. (install really didn't go well for these hosts).
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
[cid:91BFA625-D4E4-4626-B00A-C3B92D035D53@well.ox.ac.uk][cid:EFC3BB26-9DEC-40AD-A0AA-B703D5C96590@well.ox.ac.uk]
On 8 May 2018, at 15:17, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
Ok,
2018-05-08 14:54:32,740+01 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-Ma
nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
irtA003.cluster does not comply with the cluster Default networks, the following networks
are missing on host:
'ovirtmgmt'
Now ovirtmgmt is missing, you can do two things:
1) GO to Setup networks dialogue(under host) and drag the ovirtmgmt on the active NIC,
this should work. If not
2) You should be able to set to maintenance now and re-install the host
On Tue, May 8, 2018 at 5:03 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
I've attached a couple of screenshots, the hosts seem to not succeed in going into
non-operational mode. Required network is now unticked. Storage is definitely accessible
by all nodes + engine.
The moment they succeed into going non-operational, they cycle back to activating.
It's very tempting to run a DELETE FROM `vds` WHERE `vds_name` =
'virtA003.cluster';
2018-05-08 14:54:32,351+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (E
E-ManagedThreadFactory-engineScheduled-Thread-30) [] FINISH,
GetHardwareInfoAsyncVDSCommand, log id: 313cfe33
2018-05-08 14:54:32,353+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory
-engineScheduled-Thread-44) [31fd8e91] Failed to acquire lock and wait lock
'HostEngineLock:{exclusiveLocks='[
04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS_INIT]', sharedLocks=''}'
2018-05-08 14:54:32,385+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFact
ory-engineScheduled-Thread-30) [32499b7b] Running command: SetNonOperationalVdsCommand
internal: true. Entitie
s affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDS
2018-05-08 14:54:32,388+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFac
tory-engineScheduled-Thread-30) [32499b7b] START, SetVdsStatusVDSCommand(HostName =
virtA002.cluster, SetVdsSt
atusVDSCommandParameters:{hostId='e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7',
status='NonOperational', nonOperation
alReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false',
maintenanceReason='null'}), log id: 3b5138fd
2018-05-08 14:54:32,705+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFac
tory-engineScheduled-Thread-36) [3c65dca0] FINISH, SetVdsStatusVDSCommand, log id:
44441e64
2018-05-08 14:54:32,729+01 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFact
ory-engineScheduled-Thread-36) [3c65dca0] Host 'virtA003.cluster' is set to
Non-Operational, it is missing the
following networks: 'ovirtmgmt'
2018-05-08 14:54:32,740+01 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-Ma
nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
irtA003.cluster does not comply with the cluster Default networks, the following networks
are missing on host:
'ovirtmgmt'
2018-05-08 14:54:32,743+01 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler]
(EE-ManagedThreadFactory-eng
ineScheduled-Thread-36) [3c65dca0] Exception:
org.springframework.jdbc.CannotGetJdbcConnectionException: Could
not get JDBC Connection; nested exception is java.sql.SQLException:
javax.resource.ResourceException: IJ00045
7: Unchecked throwable in managedConnectionReconnected()
cl=org.jboss.jca.core.connectionmanager.listener.TxCo
nnectionListener@769d34fc[state=NORMAL managed
connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnec
tion@1444dfba connection handles=0 lastReturned=1525787672740 lastValidated=1525786889842
lastCheckedOut=15257
87372702 trackByTx=false
pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4 mcp=Semaphor
eConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=ENGINEDataSource]
xaResource=LocalXAResourceImpl@3aa2
9b7a[connectionListener=769d34fc connectionManager=5d6abc6c warned=false currentXid=null
productName=PostgreSQ
L productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
[spring-
jdbc.jar:4.3.9.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619)
[spring-jdbc.jar:4.3.9.RE<http://4.3.9.re/>
LEASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:766)
[spring-jdbc.jar:4.3.9.RELE
ASE]
at
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallIntern
al(PostgresDbEngineDialect.java:152) [dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(Postgre
sDbEngineDialect.java:118) [dal.jar:]
at
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198)
[spring-jdbc.j
ar:4.3.9.RELEASE]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:1
35) [dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.ja
va:105) [dal.jar:]
at
org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeRead(SimpleJdbcCallsHandler.java:9
7) [dal.jar:]
at org.ovirt.engine.core.dao.JobDaoImpl.checkIfJobHasTasks(JobDaoImpl.java:149)
[dal.jar:]
at
org.ovirt.engine.core.bll.job.ExecutionHandler.checkIfJobHasTasks(ExecutionHandler.java:893)
[bll.j
ar:]
at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1368)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:400)
[bll.jar:]
at
org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecuto
r.java:13) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:468) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:450) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:656)
[bll.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_161]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0
_161]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
at
org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterce
ptor.java:78)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterce
ptor.java:88)
at
org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsIntercepto
r.java:101)
at
org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFa
ctory.java:63)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionT
imeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextIntercep
tor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at
org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDi
spatcherInterceptor.java:52)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvoc
ation(SingletonComponentInstanceAssociationInterceptor.java:53)
[wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255)
[wildfly-ejb3-11.
0.0.Final.jar:11.0.0.Final]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:381)
[wildfly-ejb3-11.0.0.Fina
l.jar:11.0.0.Final]
at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244)
[wildfly-ejb3-11
.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)
at
org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScop
eActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivat
ionInterceptor.java:89)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(Curr
entInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterce
ptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.
java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInter
ceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInt
erceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.ja
va:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor
.java:50)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.
java:60)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)
at
org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)
at
org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
at
org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)
at
org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
at
org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)
at
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown
Source) [bll
.jar:]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_161]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_161]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0
_161]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_161]
at
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433)
[weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.
java:127) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPoi
ntPropagatingEnterpriseTargetBeanInstance.java:67)
[weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
[weld-core-impl-2.
4.3.Final.jar:2.4.3.Final]
at
org.ovirt.engine.core.bll.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Prox
y$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:]
at
org.ovirt.engine.core.bll.VdsEventListener.vdsNonOperational(VdsEventListener.java:303)
[bll.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.setNonOperational(HostNe
tworkTopologyPersisterImpl.java:340) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.enforceNetworkCompliance
(HostNetworkTopologyPersisterImpl.java:121) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.lambda$persistAndEnforce
NetworkCompliance$0(HostNetworkTopologyPersisterImpl.java:99) [vdsbroker.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSuppo
rt.java:202) [utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.jav
a:137) [utils.jar:]
at
org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:1
05) [utils.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:93) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopologyPersisterImpl.persistAndEnforceNetwork
Compliance(HostNetworkTopologyPersisterImpl.java:154) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.processRefreshCapabilitiesResponse(VdsManager.java:794)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring$BeforeFirstRefreshTreatmentCallback.onRes
ponse(HostMonitoring.java:767) [vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand$GetCapabilitiesVDSCommandC
allback.onResponse(GetCapabilitiesAsyncVDSCommand.java:45) [vdsbroker.jar:]
at
org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.lambda$processResponse$1(JsonRpcClient.java:182)
[vdsm-
jsonrpc-java-client.jar:]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_161]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_161]
at
org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:141)
[jav
ax.enterprise.concurrent-1.0.jar:]
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383)
[javax.enterprise.concurrent-1.0.jar:]
at
org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFut
ureTask.run(ManagedScheduledThreadPoolExecutor.java:532)
[javax.enterprise.concurrent-1.0.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
at
org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactory
Impl.java:250) [javax.enterprise.concurrent-1.0.jar:]
at
org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronMana
gedThreadFactory.java:78)
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000457: Unchecked
throwable in managedCo
nnectionReconnected()
cl=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NOR
MAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba
connection handles=0
lastReturned=1525787672740 lastValidated=1525786889842 lastCheckedOut=1525787372702
trackByTx=false pool=org.j
boss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4
mcp=SemaphoreConcurrentLinkedQueueManagedConnec
tionPool@78b9d77[pool=ENGINEDataSource]
xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc co
nnectionManager=5d6abc6c warned=false currentXid=null productName=PostgreSQL
productVersion=9.5.9 jndiName=jav
a:/ENGINEDataSource] txSync=null]
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
at
org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.jav
a:64)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
[spri
ng-jdbc.jar:4.3.9.RELEASE]
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
[spring-
jdbc.jar:4.3.9.RELEASE]
... 107 more
Caused by: javax.resource.ResourceException: IJ000457: Unchecked throwable in
managedConnectionReconnected() c
l=org.jboss.jca.core.connectionmanager.listener.TxConnectionListener@769d34fc[state=NORMAL
managed connection=
org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba connection handles=0
lastReturned=1525787672
740 lastValidated=1525786889842 lastCheckedOut=1525787372702 trackByTx=false
pool=org.jboss.jca.core.connectio
nmanager.pool.strategy.OnePool@55129ed4
mcp=SemaphoreConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=E
NGINEDataSource] xaResource=LocalXAResourceImpl@3aa29b7a[connectionListener=769d34fc
connectionManager=5d6abc6
c warned=false currentXid=null productName=PostgreSQL productVersion=9.5.9
jndiName=java:/ENGINEDataSource] tx
Sync=null]
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractC
onnectionManager.java:975)
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectio
nManager.java:792)
at
org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
... 110 more
Caused by: javax.resource.ResourceException: IJ000461: Could not enlist in transaction on
entering meta-aware object
at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:561)
at
org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:970)
... 112 more
Caused by: java.lang.IllegalStateException: Transaction Local transaction
(delegate=TransactionImple < ac, BasicAction: 0:ffffc0a840fd:-27da22a6:5af1a747:8b2
status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA
transaction provider) is not active STATUS_ROLLEDBACK
at
org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:296)
at
org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:554)
... 113 more
2018-05-08 14:54:32,749+01 INFO
[org.ovirt.engine.core.utils.transaction.TransactionSupport]
(EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] transaction rolled back
2018-05-08 14:54:32,749+01 ERROR
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] Unable to
RefreshCapabilities beforeFirstRefreshTreatment: IllegalStateException: Transaction Local
transaction (delegate=TransactionImple < ac, BasicAction:
0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >, owner=Local
transaction context for provider JBoss JTA transaction provider) is not active
STATUS_ROLLEDBACK
2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock acquired, from now a
monitoring of host will be skipped for host 'virtA003.cluster' from data-center
'Default'
2018-05-08 14:54:32,768+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] START,
SetVdsStatusVDSCommand(HostName = virtA003.cluster,
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
status='Unassigned', nonOperationalReason='NONE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
635dd4c4
2018-05-08 14:54:32,771+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] FINISH,
SetVdsStatusVDSCommand, log id: 635dd4c4
2018-05-08 14:54:32,774+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Activate host finished.
Lock released. Monitoring can run now for host 'virtA003.cluster' from data-center
'Default'
2018-05-08 14:54:32,775+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock freed to object
'EngineLock:{exclusiveLocks='[04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS]',
sharedLocks=''}'
2018-05-08 14:54:35,736+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [] START,
GetHardwareInfoAsyncVDSCommand(HostName = virtA003.cluster,
VdsIdAndVdsVDSCommandParametersBase:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
vds='Host[virtA003.cluster,04b8698b-fbfe-4efe-afa5-2cb604cbdb3d]'}), log id:
710291f0
2018-05-08 14:54:35,736+01 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [] FINISH,
GetHardwareInfoAsyncVDSCommand, log id: 710291f0
2018-05-08 14:54:35,774+01 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
04b8698b-fbfe-4efe-afa5-2cb604cbdb3d Type: VDS
2018-05-08 14:54:35,776+01 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] START,
SetVdsStatusVDSCommand(HostName = virtA003.cluster,
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
707d7be2
2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering 1 hosts
2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering hosts id:
e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 , name : virtA002.cluster
2018-05-08 14:55:00,042+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Lock Acquired to object
'EngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS]',
sharedLocks=''}'
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Running command:
ActivateVdsCommand internal: true. Entities affected : ID:
e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDSAction group MANIPULATE_HOST with role type
ADMIN
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Before acquiring lock in
order to prevent monitoring for host 'virtA002.cluster' from data-center
'Default'
2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and
wait lock
'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]',
sharedLocks=''}'
2018-05-08 14:55:02,410+01 INFO [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to acquire lock and
wait lock
'HostEngineLock:{exclusiveLocks='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]',
sharedLocks=''}'
Regards,
Callum
<Screen Shot 2018-05-08 at 14.52.20.png><Screen Shot 2018-05-08 at
14.53.27.png>
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 14:50, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
Have you removed the required property?
Do you have other required networks in this cluster except ovirtmgmt?
Do you see event massages complaining that some network/s is missing in the cluster?
What versions are your engine and vdsm?
Alona, am i missing something?
On Tue, May 8, 2018 at 4:42 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear Michael,
No luck, the cycling is still happening. This is with a hard reset of the host-engine vm,
the host, updating the node versions on everything.
Would it be really bad to just delete the nodes from the DB and clean install them?
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 13:44, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
You should only remove the required property from the network, this will release the host
from the activating cycle in which it's stuck now.
On Tue, May 8, 2018 at 3:39 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear Michael,
Good to know - didn't know about that path to configure networks.
What's the procedure to remove the failing hosts and re-install? I can't get them
into maintenance mode to then install them.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
On 8 May 2018, at 12:37, Michael Burman
<mburman@redhat.com<mailto:mburman@redhat.com>> wrote:
Hi Callum
Looks like you hitting https://bugzilla.redhat.com/show_bug.cgi?id=1570388
Do you have a required network in your env(except ovirtmgmt)? try to uncheck the
'required' from this network, this will solve the issue(via Clusters>network or
Networks>Cluster).
The bug was fixed in ovirt-engine-4.2.3.4
Cheers)
On Tue, May 8, 2018 at 12:55 PM, Callum Smith
<callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>> wrote:
Dear All,
There appears to be an issue with the host install on these, there's quite a lot of
errors being kicked out into the logs as they cycle through attempting to activate,
failing, and then failing to go into maintenance. I can't remove the hosts from the
engine to attempt to reinstall them.
Issue presented from having a statically configured network on the ovritmgmt network on
the hosts before running the host install. There are SQL errors (FK missing) and assorted
goodness in there.
Any help greatly appreciated, not sure what the next step forward is from here
Engine.log on dropbox:
https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
PS. Sorry if this comes through a few times, mailing list membership seems to be having a
funny turn this morning.
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum@well.ox.ac.uk<mailto:callum@well.ox.ac.uk>
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman@redhat.com<mailto:mburman@redhat.com> M: 0545355725<tel:0545355725>
IM: mburman
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
7:58 a.m.
Ok, can you switch the host to maintenance?
On Tue, May 8, 2018 at 5:20 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
At the moment network interfaces when i look at the host in the
engine has
no items to display. (install really didn't go well for these hosts).
Regards,
Callum
--
Callum Smith
Research Computing Core
Wellcome Trust Centre for Human Genetics
University of Oxford
e. callum(a)well.ox.ac.uk
On 8 May 2018, at 15:17, Michael Burman <mburman(a)redhat.com> wrote:
Ok,
2018-05-08 14:54:32,740+01 WARN [org.ovirt.engine.core.dal.db
broker.auditloghandling.AuditLogDirector] (EE-Ma
nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID:
VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
irtA003.cluster does not comply with the cluster Default networks, the
following networks are missing on host:
'ovirtmgmt'
Now ovirtmgmt is missing, you can do two things:
1) GO to Setup networks dialogue(under host) and drag the ovirtmgmt on the
active NIC, this should work. If not
2) You should be able to set to maintenance now and re-install the host
On Tue, May 8, 2018 at 5:03 PM, Callum Smith <callum(a)well.ox.ac.uk> wrote:
> I've attached a couple of screenshots, the hosts seem to not succeed in
> going into non-operational mode. Required network is now unticked. Storage
> is definitely accessible by all nodes + engine.
>
> The moment they succeed into going non-operational, they cycle back to
> activating.
>
> It's very tempting to run a DELETE FROM `vds` WHERE `vds_name` =
> 'virtA003.cluster';
>
> 2018-05-08 14:54:32,351+01 INFO [org.ovirt.engine.core.vdsbro
> ker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (E
> E-ManagedThreadFactory-engineScheduled-Thread-30) [] FINISH,
> GetHardwareInfoAsyncVDSCommand, log id: 313cfe33
> 2018-05-08 14:54:32,353+01 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (EE-ManagedThreadFactory
> -engineScheduled-Thread-44) [31fd8e91] Failed to acquire lock and wait
> lock 'HostEngineLock:{exclusiveLocks='[
> 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS_INIT]', sharedLocks=''}'
> 2018-05-08 14:54:32,385+01 INFO
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
> (EE-ManagedThreadFact
> ory-engineScheduled-Thread-30) [32499b7b] Running command:
> SetNonOperationalVdsCommand internal: true. Entitie
> s affected : ID: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDS
> 2018-05-08 14:54:32,388+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (EE-ManagedThreadFac
> tory-engineScheduled-Thread-30) [32499b7b] START,
> SetVdsStatusVDSCommand(HostName = virtA002.cluster, SetVdsSt
> atusVDSCommandParameters:{hostId='e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7',
> status='NonOperational', nonOperation
> alReason='NETWORK_UNREACHABLE', stopSpmFailureLogged='false',
> maintenanceReason='null'}), log id: 3b5138fd
> 2018-05-08 14:54:32,705+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (EE-ManagedThreadFac
> tory-engineScheduled-Thread-36) [3c65dca0] FINISH,
> SetVdsStatusVDSCommand, log id: 44441e64
> 2018-05-08 14:54:32,729+01 ERROR
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
> (EE-ManagedThreadFact
> ory-engineScheduled-Thread-36) [3c65dca0] Host 'virtA003.cluster' is set
> to Non-Operational, it is missing the
> following networks: 'ovirtmgmt'
> 2018-05-08 14:54:32,740+01 WARN [org.ovirt.engine.core.dal.db
> broker.auditloghandling.AuditLogDirector] (EE-Ma
> nagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] EVENT_ID:
> VDS_SET_NONOPERATIONAL_NETWORK(519), Host v
> irtA003.cluster does not comply with the cluster Default networks, the
> following networks are missing on host:
> 'ovirtmgmt'
> 2018-05-08 14:54:32,743+01 ERROR [org.ovirt.engine.core.bll.job.ExecutionHandler]
> (EE-ManagedThreadFactory-eng
> ineScheduled-Thread-36) [3c65dca0] Exception:
> org.springframework.jdbc.CannotGetJdbcConnectionException: Could
> not get JDBC Connection; nested exception is java.sql.SQLException:
> javax.resource.ResourceException: IJ00045
> 7: Unchecked throwable in managedConnectionReconnected()
> cl=org.jboss.jca.core.connectionmanager.listener.TxCo
> nnectionListener@769d34fc[state=NORMAL managed
> connection=org.jboss.jca.adapters.jdbc.local.LocalManagedConnec
> tion@1444dfba connection handles=0 lastReturned=1525787672740
> lastValidated=1525786889842 lastCheckedOut=15257
> 87372702 trackByTx=false pool=org.jboss.jca.core.connec
> tionmanager.pool.strategy.OnePool@55129ed4 mcp=Semaphor
> eConcurrentLinkedQueueManagedConnectionPool@78b9d77[pool=ENGINEDataSource]
> xaResource=LocalXAResourceImpl@3aa2
> 9b7a[connectionListener=769d34fc connectionManager=5d6abc6c warned=false
> currentXid=null productName=PostgreSQ
> L productVersion=9.5.9 jndiName=java:/ENGINEDataSource] txSync=null]
> at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
> [spring-
> jdbc.jar:4.3.9.RELEASE]
> at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:619)
> [spring-jdbc.jar:4.3.9.RE <http://4.3.9.re/>
> LEASE]
> at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:684)
> [spring-jdbc.jar:4.3.9.RELE
> ASE]
> at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:716)
> [spring-jdbc.jar:4.3.9.RELE
> ASE]
> at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:766)
> [spring-jdbc.jar:4.3.9.RELE
> ASE]
> at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$P
> ostgresSimpleJdbcCall.executeCallIntern
> al(PostgresDbEngineDialect.java:152) [dal.jar:]
> at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$P
> ostgresSimpleJdbcCall.doExecute(Postgre
> sDbEngineDialect.java:118) [dal.jar:]
> at
org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198)
> [spring-jdbc.j
> ar:4.3.9.RELEASE]
> at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.ex
> ecuteImpl(SimpleJdbcCallsHandler.java:1
> 35) [dal.jar:]
> at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.ex
> ecuteReadList(SimpleJdbcCallsHandler.ja
> va:105) [dal.jar:]
> at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.ex
> ecuteRead(SimpleJdbcCallsHandler.java:9
> 7) [dal.jar:]
> at
org.ovirt.engine.core.dao.JobDaoImpl.checkIfJobHasTasks(JobDaoImpl.java:149)
> [dal.jar:]
> at org.ovirt.engine.core.bll.job.ExecutionHandler.checkIfJobHas
> Tasks(ExecutionHandler.java:893) [bll.j
> ar:]
> at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1368)
> [bll.jar:]
> at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:400)
> [bll.jar:]
> at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecu
> tor.execute(DefaultBackendActionExecuto
> r.java:13) [bll.jar:]
> at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:468)
> [bll.jar:]
> at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:450)
> [bll.jar:]
> at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:656)
> [bll.jar:]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_161]
> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_161]
> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0
> _161]
> at java.lang.reflect.Method.invoke(Method.java:498)
> [rt.jar:1.8.0_161]
> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.
> processInvocation(ManagedReferenceMethodInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(
> InterceptorContext.java:509)
> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.del
> egateInterception(Jsr299BindingsInterce
> ptor.java:78)
> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doM
> ethodInterception(Jsr299BindingsInterce
> ptor.java:88)
> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.pro
> cessInvocation(Jsr299BindingsIntercepto
> r.java:101)
> at org.jboss.as.ee.component.interceptors.UserInterceptorFactor
> y$1.processInvocation(UserInterceptorFa
> ctory.java:63)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeI
> nterceptor.processInvocation(ExecutionT
> imeInterceptor.java:43) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.proc
> essInvocation(ConcurrentContextIntercep
> tor.java:45) [wildfly-ee-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.invocation.InitialInterceptor.processInvocation(In
> itialInterceptor.java:40)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch
> ainedInterceptor.java:53)
> at org.jboss.as.ee.component.interceptors.ComponentDispatcherIn
> terceptor.processInvocation(ComponentDi
> spatcherInterceptor.java:52)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.component.singleton.SingletonComponentInst
> anceAssociationInterceptor.processInvoc
> ation(SingletonComponentInstanceAssociationInterceptor.java:53)
> [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:255)
> [wildfly-ejb3-11.
> 0.0.Final.jar:11.0.0.Final]
> at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:381)
> [wildfly-ejb3-11.0.0.Fina
> l.jar:11.0.0.Final]
> at
org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244)
> [wildfly-ejb3-11
> .0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.invocation.InterceptorContext$Invocation.proceed(
> InterceptorContext.java:509)
> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationIntercep
> tor.aroundInvoke(AbstractEJBRequestScop
> eActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.ja
> r:2.4.3.Final]
> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.
> processInvocation(EjbRequestScopeActivat
> ionInterceptor.java:89)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationCo
> ntextInterceptor.processInvocation(Curr
> entInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.Final.jar
> :11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterc
> eptor.processInvocation(WaitTimeInterce
> ptor.java:47) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.security.SecurityContextInterceptor.proces
> sInvocation(SecurityContextInterceptor.
> java:100) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.deployment.processors.StartupAwaitIntercep
> tor.processInvocation(StartupAwaitInter
> ceptor.java:22) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptor
> Factory$1.processInvocation(ShutDownInt
> erceptorFactory.java:64) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.
> processInvocation(LoggingInterceptor.ja
> va:67) [wildfly-ejb3-11.0.0.Final.jar:11.0.0.Final]
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.as.ee.component.NamespaceContextInterceptor.proces
> sInvocation(NamespaceContextInterceptor
> .java:50)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.invocation.ContextClassLoaderInterceptor.processIn
> vocation(ContextClassLoaderInterceptor.
> java:60)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.invocation.InterceptorContext.run(InterceptorConte
> xt.java:438)
> at org.wildfly.security.manager.WildFlySecurityManager.doChecke
> d(WildFlySecurityManager.java:609)
> at org.jboss.invocation.AccessCheckingInterceptor.processInvoca
> tion(AccessCheckingInterceptor.java:57)
> at org.jboss.invocation.InterceptorContext.proceed(InterceptorC
> ontext.java:422)
> at org.jboss.invocation.ChainedInterceptor.processInvocation(Ch
> ainedInterceptor.java:53)
> at org.jboss.as.ee.component.ViewService$View.invoke(ViewServic
> e.java:198)
> at org.jboss.as.ee.component.ViewDescription$1.processInvocatio
> n(ViewDescription.java:185)
> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(Prox
> yInvocationHandler.java:81)
> at
org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view3.runInternalAction(Unknown
> Source) [bll
> .jar:]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_161]
> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_161]
> at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0
> _161]
> at java.lang.reflect.Method.invoke(Method.java:498)
> [rt.jar:1.8.0_161]
> at
org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433)
> [weld-core-impl-2.
> 4.3.Final.jar:2.4.3.Final]
> at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.
> invoke(EnterpriseBeanProxyMethodHandler.
> java:127) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
> at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invok
> e(EnterpriseTargetBeanInstance.java:56)
> [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
> at org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpris
> eTargetBeanInstance.invoke(InjectionPoi
> ntPropagatingEnterpriseTargetBeanInstance.java:67)
> [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]
> at
org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
> [weld-core-impl-2.
> 4.3.Final.jar:2.4.3.Final]
> at org.ovirt.engine.core.bll.BackendCommandObjectsHandler$Backe
> ndInternal$BackendLocal$2049259618$Prox
> y$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) [bll.jar:]
> at
org.ovirt.engine.core.bll.VdsEventListener.vdsNonOperational(VdsEventListener.java:303)
> [bll.jar:]
> at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopolog
> yPersisterImpl.setNonOperational(HostNe
> tworkTopologyPersisterImpl.java:340) [vdsbroker.jar:]
> at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopolog
> yPersisterImpl.enforceNetworkCompliance
> (HostNetworkTopologyPersisterImpl.java:121) [vdsbroker.jar:]
> at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopolog
> yPersisterImpl.lambda$persistAndEnforce
> NetworkCompliance$0(HostNetworkTopologyPersisterImpl.java:99)
> [vdsbroker.jar:]
> at org.ovirt.engine.core.utils.transaction.TransactionSupport.e
> xecuteInNewTransaction(TransactionSuppo
> rt.java:202) [utils.jar:]
> at org.ovirt.engine.core.utils.transaction.TransactionSupport.e
> xecuteInRequired(TransactionSupport.jav
> a:137) [utils.jar:]
> at org.ovirt.engine.core.utils.transaction.TransactionSupport.e
> xecuteInScope(TransactionSupport.java:1
> 05) [utils.jar:]
> at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopolog
> yPersisterImpl.persistAndEnforceNetwork
> Compliance(HostNetworkTopologyPersisterImpl.java:93) [vdsbroker.jar:]
> at org.ovirt.engine.core.vdsbroker.vdsbroker.HostNetworkTopolog
> yPersisterImpl.persistAndEnforceNetwork
> Compliance(HostNetworkTopologyPersisterImpl.java:154) [vdsbroker.jar:]
> at org.ovirt.engine.core.vdsbroker.VdsManager.processRefreshCap
> abilitiesResponse(VdsManager.java:794)
> [vdsbroker.jar:]
> at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring$Be
> foreFirstRefreshTreatmentCallback.onRes
> ponse(HostMonitoring.java:767) [vdsbroker.jar:]
> at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsy
> ncVDSCommand$GetCapabilitiesVDSCommandC
> allback.onResponse(GetCapabilitiesAsyncVDSCommand.java:45)
> [vdsbroker.jar:]
> at org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.lambda$processRe
> sponse$1(JsonRpcClient.java:182) [vdsm-
> jsonrpc-java-client.jar:]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [rt.jar:1.8.0_161]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [rt.jar:1.8.0_161]
> at org.glassfish.enterprise.concurrent.internal.ManagedFutureTa
> sk.run(ManagedFutureTask.java:141) [jav
> ax.enterprise.concurrent-1.0.jar:]
> at org.glassfish.enterprise.concurrent.internal.ManagedSchedule
> dThreadPoolExecutor$ManagedScheduledFut
> ureTask.access$101(ManagedScheduledThreadPoolExecutor.java:383)
> [javax.enterprise.concurrent-1.0.jar:]
> at org.glassfish.enterprise.concurrent.internal.ManagedSchedule
> dThreadPoolExecutor$ManagedScheduledFut
> ureTask.run(ManagedScheduledThreadPoolExecutor.java:532)
> [javax.enterprise.concurrent-1.0.jar:]
> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [rt.jar:1.8.0_161]
> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [rt.jar:1.8.0_161]
> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_161]
> at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl
> $ManagedThread.run(ManagedThreadFactory
> Impl.java:250) [javax.enterprise.concurrent-1.0.jar:]
> at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFacto
> ry$ElytronManagedThread.run(ElytronMana
> gedThreadFactory.java:78)
> Caused by: java.sql.SQLException: javax.resource.ResourceException:
> IJ000457: Unchecked throwable in managedCo
> nnectionReconnected() cl=org.jboss.jca.core.connecti
> onmanager.listener.TxConnectionListener@769d34fc[state=NOR
> MAL managed connection=org.jboss.jca.adapters.jdbc.local.LocalManagedCon
> nection@1444dfba connection handles=0
> lastReturned=1525787672740 lastValidated=1525786889842
> lastCheckedOut=1525787372702 trackByTx=false pool=org.j
> boss.jca.core.connectionmanager.pool.strategy.OnePool@55129ed4
> mcp=SemaphoreConcurrentLinkedQueueManagedConnec
> tionPool@78b9d77[pool=ENGINEDataSource] xaResource=LocalXAResourceImpl
> @3aa29b7a[connectionListener=769d34fc co
> nnectionManager=5d6abc6c warned=false currentXid=null
> productName=PostgreSQL productVersion=9.5.9 jndiName=jav
> a:/ENGINEDataSource] txSync=null]
> at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(
> WrapperDataSource.java:146)
> at org.jboss.as.connector.subsystems.datasources.WildFlyDataSou
> rce.getConnection(WildFlyDataSource.jav
> a:64)
> at org.springframework.jdbc.datasource.DataSourceUtils.doGetCon
> nection(DataSourceUtils.java:111) [spri
> ng-jdbc.jar:4.3.9.RELEASE]
> at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
> [spring-
> jdbc.jar:4.3.9.RELEASE]
> ... 107 more
> Caused by: javax.resource.ResourceException: IJ000457: Unchecked
> throwable in managedConnectionReconnected() c
> l=org.jboss.jca.core.connectionmanager.listener.TxConnection
> Listener@769d34fc[state=NORMAL managed connection=
> org.jboss.jca.adapters.jdbc.local.LocalManagedConnection@1444dfba
> connection handles=0 lastReturned=1525787672
> 740 lastValidated=1525786889842 lastCheckedOut=1525787372702
> trackByTx=false pool=org.jboss.jca.core.connectio
> nmanager.pool.strategy.OnePool@55129ed4 mcp=SemaphoreConcurrentLinkedQ
> ueueManagedConnectionPool@78b9d77[pool=E
> NGINEDataSource] xaResource=LocalXAResourceImpl
> @3aa29b7a[connectionListener=769d34fc connectionManager=5d6abc6
> c warned=false currentXid=null productName=PostgreSQL
> productVersion=9.5.9 jndiName=java:/ENGINEDataSource] tx
> Sync=null]
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManag
> er.reconnectManagedConnection(AbstractC
> onnectionManager.java:975)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManag
> er.allocateConnection(AbstractConnectio
> nManager.java:792)
> at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(
> WrapperDataSource.java:138)
> ... 110 more
> Caused by: javax.resource.ResourceException: IJ000461: Could not enlist
> in transaction on entering meta-aware object
> at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerI
> mpl.managedConnectionReconnected(TxConnectionManagerImpl.java:561)
> at org.jboss.jca.core.connectionmanager.AbstractConnectionManag
> er.reconnectManagedConnection(AbstractConnectionManager.java:970)
> ... 112 more
> Caused by: java.lang.IllegalStateException: Transaction Local
> transaction (delegate=TransactionImple < ac, BasicAction:
> 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >,
> owner=Local transaction context for provider JBoss JTA transaction
> provider) is not active STATUS_ROLLEDBACK
> at org.jboss.jca.core.connectionmanager.listener.TxConnectionLi
> stener.enlist(TxConnectionListener.java:296)
> at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerI
> mpl.managedConnectionReconnected(TxConnectionManagerImpl.java:554)
> ... 113 more
>
> 2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.utils.
> transaction.TransactionSupport] (EE-ManagedThreadFactory-engineScheduled-Thread-36)
> [3c65dca0] transaction rolled back
> 2018-05-08 14:54:32,749+01 ERROR
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
> (EE-ManagedThreadFactory-engineScheduled-Thread-36) [3c65dca0] Unable to
> RefreshCapabilities beforeFirstRefreshTreatment: IllegalStateException:
> Transaction Local transaction (delegate=TransactionImple < ac, BasicAction:
> 0:ffffc0a840fd:-27da22a6:5af1a747:8b2 status: ActionStatus.ABORTED >,
> owner=Local transaction context for provider JBoss JTA transaction
> provider) is not active STATUS_ROLLEDBACK
> 2018-05-08 14:54:32,749+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock
> acquired, from now a monitoring of host will be skipped for host
> 'virtA003.cluster' from data-center 'Default'
> 2018-05-08 14:54:32,768+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] START,
> SetVdsStatusVDSCommand(HostName = virtA003.cluster,
>
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
> status='Unassigned', nonOperationalReason='NONE',
> stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
635dd4c4
> 2018-05-08 14:54:32,771+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] FINISH,
> SetVdsStatusVDSCommand, log id: 635dd4c4
> 2018-05-08 14:54:32,774+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Activate
> host finished. Lock released. Monitoring can run now for host
> 'virtA003.cluster' from data-center 'Default'
> 2018-05-08 14:54:32,775+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-44) [31fd8e91] Lock
> freed to object 'EngineLock:{exclusiveLocks='[
> 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d=VDS]', sharedLocks=''}'
> 2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core.vdsbro
> ker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] START,
> GetHardwareInfoAsyncVDSCommand(HostName = virtA003.cluster,
>
VdsIdAndVdsVDSCommandParametersBase:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
> vds='Host[virtA003.cluster,04b8698b-fbfe-4efe-afa5-2cb604cbdb3d]'}), log
> id: 710291f0
> 2018-05-08 14:54:35,736+01 INFO [org.ovirt.engine.core.vdsbro
> ker.vdsbroker.GetHardwareInfoAsyncVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-31) [] FINISH,
> GetHardwareInfoAsyncVDSCommand, log id: 710291f0
> 2018-05-08 14:54:35,774+01 INFO
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] Running
> command: SetNonOperationalVdsCommand internal: true. Entities affected :
> ID: 04b8698b-fbfe-4efe-afa5-2cb604cbdb3d Type: VDS
> 2018-05-08 14:54:35,776+01 INFO
[org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-31) [28856dde] START,
> SetVdsStatusVDSCommand(HostName = virtA003.cluster,
>
SetVdsStatusVDSCommandParameters:{hostId='04b8698b-fbfe-4efe-afa5-2cb604cbdb3d',
> status='NonOperational', nonOperationalReason='NETWORK_UNREACHABLE',
> stopSpmFailureLogged='false', maintenanceReason='null'}), log id:
707d7be2
> 2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
> (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering 1
> hosts
> 2018-05-08 14:55:00,005+01 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager]
> (EE-ManagedThreadFactory-engineScheduled-Thread-39) [] Autorecovering
> hosts id: e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 , name : virtA002.cluster
> 2018-05-08 14:55:00,042+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Lock
> Acquired to object 'EngineLock:{exclusiveLocks='[
> e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS]', sharedLocks=''}'
> 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Running
> command: ActivateVdsCommand internal: true. Entities affected : ID:
> e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7 Type: VDSAction group
> MANIPULATE_HOST with role type ADMIN
> 2018-05-08 14:55:00,056+01 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand]
> (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Before
> acquiring lock in order to prevent monitoring for host 'virtA002.cluster'
> from data-center 'Default'
> 2018-05-08 14:55:00,056+01 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to
> acquire lock and wait lock 'HostEngineLock:{exclusiveLock
> s='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]',
sharedLocks=''}'
> 2018-05-08 14:55:02,410+01 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (EE-ManagedThreadFactory-engineScheduled-Thread-39) [1a440f59] Failed to
> acquire lock and wait lock 'HostEngineLock:{exclusiveLock
> s='[e8daef6e-b098-4ed0-8b4b-5865bbbf5dc7=VDS_INIT]',
sharedLocks=''}'
>
> Regards,
> Callum
> <Screen Shot 2018-05-08 at 14.52.20.png><Screen Shot 2018-05-08 at
> 14.53.27.png>
> --
>
> Callum Smith
> Research Computing Core
> Wellcome Trust Centre for Human Genetics
> University of Oxford
> e. callum(a)well.ox.ac.uk
>
> On 8 May 2018, at 14:50, Michael Burman <mburman(a)redhat.com> wrote:
>
> Have you removed the required property?
> Do you have other required networks in this cluster except ovirtmgmt?
> Do you see event massages complaining that some network/s is missing in
> the cluster?
> What versions are your engine and vdsm?
>
> Alona, am i missing something?
>
>
>
> On Tue, May 8, 2018 at 4:42 PM, Callum Smith <callum(a)well.ox.ac.uk>
> wrote:
>
>> Dear Michael,
>>
>> No luck, the cycling is still happening. This is with a hard reset of
>> the host-engine vm, the host, updating the node versions on everything.
>>
>> Would it be really bad to just delete the nodes from the DB and clean
>> install them?
>>
>> Regards,
>> Callum
>>
>> --
>>
>> Callum Smith
>> Research Computing Core
>> Wellcome Trust Centre for Human Genetics
>> University of Oxford
>> e. callum(a)well.ox.ac.uk
>>
>> On 8 May 2018, at 13:44, Michael Burman <mburman(a)redhat.com> wrote:
>>
>> You should only remove the required property from the network, this will
>> release the host from the activating cycle in which it's stuck now.
>>
>> On Tue, May 8, 2018 at 3:39 PM, Callum Smith <callum(a)well.ox.ac.uk>
>> wrote:
>>
>>> Dear Michael,
>>>
>>> Good to know - didn't know about that path to configure networks.
>>>
>>> What's the procedure to remove the failing hosts and re-install? I
>>> can't get them into maintenance mode to then install them.
>>>
>>> Regards,
>>> Callum
>>>
>>> --
>>>
>>> Callum Smith
>>> Research Computing Core
>>> Wellcome Trust Centre for Human Genetics
>>> University of Oxford
>>> e. callum(a)well.ox.ac.uk
>>>
>>> On 8 May 2018, at 12:37, Michael Burman <mburman(a)redhat.com> wrote:
>>>
>>> Hi Callum
>>> Looks like you hitting https://bugzilla.redhat.com/sh
>>> ow_bug.cgi?id=1570388
>>> Do you have a required network in your env(except ovirtmgmt)? try to
>>> uncheck the 'required' from this network, this will solve the
issue(via
>>> Clusters>network or Networks>Cluster).
>>> The bug was fixed in ovirt-engine-4.2.3.4
>>>
>>> Cheers)
>>>
>>> On Tue, May 8, 2018 at 12:55 PM, Callum Smith <callum(a)well.ox.ac.uk>
>>> wrote:
>>>
>>>> Dear All,
>>>>
>>>> There appears to be an issue with the host install on these, there's
>>>> quite a lot of errors being kicked out into the logs as they cycle
through
>>>> attempting to activate, failing, and then failing to go into maintenance.
I
>>>> can't remove the hosts from the engine to attempt to reinstall them.
>>>>
>>>> Issue presented from having a statically configured network on the
>>>> ovritmgmt network on the hosts before running the host install. There
are
>>>> SQL errors (FK missing) and assorted goodness in there.
>>>>
>>>> Any help greatly appreciated, not sure what the next step forward is
>>>> from here
>>>>
>>>> Engine.log on dropbox:
>>>> https://www.dropbox.com/s/82iem0ov869yh32/engine.log-20180508.zip?dl=0
>>>>
>>>> PS. Sorry if this comes through a few times, mailing list membership
>>>> seems to be having a funny turn this morning.
>>>>
>>>> Regards,
>>>> Callum
>>>>
>>>> --
>>>>
>>>> Callum Smith
>>>> Research Computing Core
>>>> Wellcome Trust Centre for Human Genetics
>>>> University of Oxford
>>>> e. callum(a)well.ox.ac.uk
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list -- users(a)ovirt.org
>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>>
>>>>
>>>
>>>
>>> --
>>> Michael Burman
>>>
>>> Senior Quality engineer - rhv network - redhat israel
>>> Red Hat
>>>
>>> <https://www.redhat.com/>
>>>
>>> mburman(a)redhat.com M: 0545355725 IM: mburman
>>> <https://red.ht/sig>
>>>
>>>
>>>
>>
>>
>> --
>> Michael Burman
>>
>> Senior Quality engineer - rhv network - redhat israel
>> Red Hat
>>
>> <https://www.redhat.com/>
>>
>> mburman(a)redhat.com M: 0545355725 IM: mburman
>> <https://red.ht/sig>
>>
>>
>>
>
>
> --
> Michael Burman
>
> Senior Quality engineer - rhv network - redhat israel
> Red Hat
>
> <https://www.redhat.com/>
>
> mburman(a)redhat.com M: 0545355725 IM: mburman
> <https://red.ht/sig>
>
>
>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com/>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
--
Michael Burman
Senior Quality engineer - rhv network - redhat israel
Red Hat
<https://www.redhat.com>
mburman(a)redhat.com M: 0545355725 IM: mburman
<https://red.ht/sig>
8:27 a.m.
Hi Callum,
For some reason, when installing the host the management network wasn't
attached to the host.
Since the management network is required, due to
https://bugzilla.redhat.com/1570388
<https://bugzilla.redhat.com/show_bug.cgi?id=1570388> the host stuck in non
operational <-> activating states.
You have two options -
1. Updating ovirt to ovirt-engine-4.2.3.4 and re-installing the host.
2. Trying to understand why the management network wasn't attached to the
host at the first place.
If you choose option 2 please attach the full engine logs (engine.log,
server.log) and vdsm logs (vdsm.log, supervdsm.log).
Thanks,
Alona.
2184
days inactive
2185
days old
10 comments
3 participants
participants (3)
-
Alona Kaplan -
Callum Smith -
Michael Burman