[ovirt-devel] VDSM stopped while adding host to newly created cluster
Christopher Pereira
kripper at imatronix.cl
Fri Mar 27 13:26:53 UTC 2015
On 27-03-2015 7:03, Dan Kenigsberg wrote:
> On Thu, Mar 26, 2015 at 06:16:24PM -0300, Christopher Pereira wrote:
>> Continuing with the 3.6 Night Builds testing...
>>
>> While hosted-engine-setup was adding the host to the newly created cluster,
>> VDSM crashed, probably because the gluster engine storage disappeared as in
>> BZ 1201355 [1]
>>
>> Facts:
>> - the engine storage (/rhev/data-center/mmt/...) was umounted during
>> this process
>> - another mount of the same volume was still mounted after the VDSM
>> crash (maybe the problem is not related with gluster)
> What exactly happened to vdsm? Did the process die? Why? Was it stopped?
> did it segfault? Did it stop responding? Can you share vdsm.log and
> /var/log/message showing what happened during the crash?
Hi Dan,
You will find relevants logs here:
https://bugzilla.redhat.com/show_bug.cgi?id=1201355#c4
Summary:
1) During setup, VDSM receives a SIGTERM:
MainThread::DEBUG::2015-03-26
18:36:56,767::vdsm::66::vds::(sigtermHandler) Received signal 15
Maybe the activation process installs VDSM and/or restarts it.
2) Since the gluster storage is mounted from a VDSM ChildProcess, it
disappears when VDSM stops.
Thus, the VM is paused and will never resume (even after remounting the
storage, because the paused QEMU process keeps invalid file descriptors):
https://bugzilla.redhat.com/show_bug.cgi?id=1058300
https://bugzilla.redhat.com/show_bug.cgi?id=1172905
3) After the VDSM stopped, it's not possible to restart it since you
will get an "invalid lockspace" in sanlock.
This can be solved with hosted-engine --start-pool.
4) You will be able to reproduce the VDSM sigterm with less effort (no
need to re-deploy) by accessing the engine portal and reactivating the host.
You will see that VDSM gets stopped and the storage lost.
As a workarround to avoid the storage to get lost, you can mount it
manually so that it doesn't relay on the VDSM ChildProcess.
Questions:
1) I'm affraid that by activating the host manually after an interrupted
setup I may be skipping some special configurations.
Is there any difference between activating the host manually from the
web-manager and activating the host with the setup script?
How can I complete the setup manually?
Status:
I'm still unable to activate the host manually, because engine is now
having problems with the JsonRPC communcation:
2015-03-27 10:11:54,889 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
Reactor) [] Connecting to h2.imatronix.com/209.126.105.36
2015-03-27 10:11:54,893 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor)
[] *Unable to process messages*
2015-03-27 10:11:54,893 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand]
(DefaultQuartzScheduler_Worker-96) [] Command
'ListVDSCommand(HostName = h2, HostId =
46d4659a-4efe-4427-aa68-a4536508fa08,
vds=Host[h2,46d4659a-4efe-4427-aa68-a4536508fa08])' execution
failed: VDSGenericException: VDSNetworkException: General SSLEngine
problem
2015-03-27 10:11:54,894 ERROR
[org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl]
(DefaultQuartzScheduler_Worker-96) [] Failed to invoke scheduled
method vmsMonitoring: null
2015-03-27 10:11:57,894 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
Reactor) [] Connecting to h2.imatronix.com/209.126.105.36
2015-03-27 10:11:57,897 ERROR
[org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor)
[] *Unable to process messages*
2015-03-27 10:11:57,897 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95)
[] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return
value
'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc at 79313585'
2015-03-27 10:11:57,898 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95)
[] HostName = h2
2015-03-27 10:11:57,898 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-95)
[] Command 'GetCapabilitiesVDSCommand(HostName = h2, HostId =
46d4659a-4efe-4427-aa68-a4536508fa08,
vds=Host[h2,46d4659a-4efe-4427-aa68-a4536508fa08])' execution
failed: VDSGenericException: VDSNetworkException: *General SSLEngine
problem*
2015-03-27 10:11:57,898 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-95) [] Failure to refresh Vds runtime
info: VDSGenericException: VDSNetworkException: General SSLEngine
problem
2015-03-27 10:11:57,898 ERROR
[org.ovirt.engine.core.vdsbroker.HostMonitoring]
(DefaultQuartzScheduler_Worker-95) [] Exception:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException:
VDSGenericException: VDSNetworkException: General SSLEngine problem
at
org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:183)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:55)
[vdsbroker.jar:]
at
org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33)
[dal.jar:]
at
org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:465)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:587)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:111)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.HostMonitoring.refresh(HostMonitoring.java:76)
[vdsbroker.jar:]
at
org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:199)
[vdsbroker.jar:]
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown
Source) [:1.7.0_75]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.7.0_75]
at java.lang.reflect.Method.invoke(Method.java:606)
[rt.jar:1.7.0_75]
at
org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81)
[scheduler.jar:]
at
org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52)
[scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
[quartz.jar:]
at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
[quartz.jar:]
2015-03-27 10:11:57,899 WARN
[org.ovirt.engine.core.vdsbroker.VdsManager]
(DefaultQuartzScheduler_Worker-95) [] Failed to refresh VDS, network
error, continuing, vds='h2'(46d4659a-4efe-4427-aa68-a4536508fa08):
VDSGenericException: VDSNetworkException: *General SSLEngine problem*
[...]
On the VDSM side, we have:
clientIFinit::DEBUG::2015-03-27
10:11:53,098::task::592::Storage.TaskManager.Task::(_updateState)
Task=`87ed5b66-3abb-4edc-aec3-59f071b33276`::moving from state init
-> state preparing
clientIFinit::INFO::2015-03-27
10:11:53,098::logUtils::48::dispatcher::(wrapper) Run and protect:
getConnectedStoragePoolsList(options=None)
clientIFinit::INFO::2015-03-27
10:11:53,098::logUtils::51::dispatcher::(wrapper) Run and protect:
getConnectedStoragePoolsList, Return response: {'poollist': []}
clientIFinit::DEBUG::2015-03-27
10:11:53,098::task::1188::Storage.TaskManager.Task::(prepare)
Task=`87ed5b66-3abb-4edc-aec3-59f071b33276`::finished: {'poollist': []}
clientIFinit::DEBUG::2015-03-27
10:11:53,098::task::592::Storage.TaskManager.Task::(_updateState)
Task=`87ed5b66-3abb-4edc-aec3-59f071b33276`::moving from state
preparing -> state finished
clientIFinit::DEBUG::2015-03-27
10:11:53,098::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
clientIFinit::DEBUG::2015-03-27
10:11:53,098::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
clientIFinit::DEBUG::2015-03-27
10:11:53,098::task::990::Storage.TaskManager.Task::(_decref)
Task=`87ed5b66-3abb-4edc-aec3-59f071b33276`::ref 0 aborting False
Detector thread::DEBUG::2015-03-27
10:11:53,450::protocoldetector::201::vds.MultiProtocolAcceptor::(_add_connection)
*Adding connection 209.239.124.8:54218*
Detector thread::DEBUG::2015-03-27
10:11:53,459::protocoldetector::225::vds.MultiProtocolAcceptor::(_process_handshake)
*Error during handshake: sslv3 alert certificate unknown*
Detector thread::DEBUG::2015-03-27
10:11:53,459::protocoldetector::215::vds.MultiProtocolAcceptor::(_remove_connection)
Removing connection 209.239.124.8:54218
Detector thread::DEBUG::2015-03-27
10:11:55,249::protocoldetector::201::vds.MultiProtocolAcceptor::(_add_connection)
Adding connection 209.126.113.73:54119
Detector thread::DEBUG::2015-03-27
10:11:55,252::protocoldetector::225::vds.MultiProtocolAcceptor::(_process_handshake)
Error during handshake: unexpected eof
Detector thread::DEBUG::2015-03-27
10:11:55,252::protocoldetector::215::vds.MultiProtocolAcceptor::(_remove_connection)
Removing connection 209.126.113.73:54119
Detector thread::DEBUG::2015-03-27
10:11:56,582::protocoldetector::201::vds.MultiProtocolAcceptor::(_add_connection)
Adding connection 209.239.124.8:39606
Detector thread::DEBUG::2015-03-27
10:11:56,629::protocoldetector::225::vds.MultiProtocolAcceptor::(_process_handshake)
Error during handshake: sslv3 alert certificate unknown
Detector thread::DEBUG::2015-03-27
10:11:56,629::protocoldetector::215::vds.MultiProtocolAcceptor::(_remove_connection)
Removing connection 209.239.124.8:39606
clientIFinit::DEBUG::2015-03-27
10:11:58,104::task::592::Storage.TaskManager.Task::(_updateState)
Task=`9e6db6dc-3ce0-4e93-8ddd-2aa1d09fa687`::moving from state init
-> state preparing
clientIFinit::INFO::2015-03-27
10:11:58,104::logUtils::48::dispatcher::(wrapper) Run and protect:
getConnectedStoragePoolsList(options=None)
clientIFinit::INFO::2015-03-27
10:11:58,104::logUtils::51::dispatcher::(wrapper) Run and protect:
getConnectedStoragePoolsList, Return response: {'poollist': []}
clientIFinit::DEBUG::2015-03-27
10:11:58,104::task::1188::Storage.TaskManager.Task::(prepare)
Task=`9e6db6dc-3ce0-4e93-8ddd-2aa1d09fa687`::finished: {'poollist': []}
clientIFinit::DEBUG::2015-03-27
10:11:58,104::task::592::Storage.TaskManager.Task::(_updateState)
Task=`9e6db6dc-3ce0-4e93-8ddd-2aa1d09fa687`::moving from state
preparing -> state finished
clientIFinit::DEBUG::2015-03-27
10:11:58,104::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources {}
clientIFinit::DEBUG::2015-03-27
10:11:58,104::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
clientIFinit::DEBUG::2015-03-27
10:11:58,104::task::990::Storage.TaskManager.Task::(_decref)
Task=`9e6db6dc-3ce0-4e93-8ddd-2aa1d09fa687`::ref 0 aborting False
[...]
I guess this is related to an invalid certificate or some protocol
version missmatch.
How can I fix it?
Regards,
Christopher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20150327/d32bebf9/attachment-0001.html>
More information about the Devel
mailing list