9:17 a.m.
After changing the ownership the engine is up!!
thanks for your help!!!:)
On Tue, Mar 19, 2019 at 3:25 PM Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
On Tue, Mar 19, 2019 at 2:21 PM ada per <adaper3(a)gmail.com> wrote:
> Thanks for you reply.
>
> Can you please provide step by step instructions on how to upgrade the
> vdsm from a node command line?
>
Can you please report the version of vdsm you are using?
then check the ownership of
/rhev/data-center/00000000-0000-0000-0000-000000000000/05b2b2d5-a80e-4622-9410-8e1e9d362f3f/images/bb890447-f1f7-46af-8e57-543d61f0bd08/81685d19-0060-4f5d-a4cd-c5efa24aecfe
if it's not vdsm:kvm, change it and then try again with hosted-engine
--vm-start
>
> On Tue, Mar 19, 2019 at 2:49 PM Simone Tiraboschi <stirabos(a)redhat.com>
> wrote:
>
>> Hi Ada,
>> here the error:
>>
>> 2019-03-19 14:08:25,833+0200 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer]
>> RPC call Host.getStorageRepoStats succeeded in 0.00 seconds (__init__:312)
>> 2019-03-19 14:08:25,839+0200 INFO (vm/a492d2eb) [vdsm.api] FINISH
>> prepareImage error=Volume does not exist:
>> (u'81685d19-0060-4f5d-a4cd-c5efa24aecfe',) from=internal,
>> task_id=dc8fbf34-8d7e-47a3-8e02-0a5e5cb90257 (api:52)
>> 2019-03-19 14:08:25,839+0200 ERROR (vm/a492d2eb)
>> [storage.TaskManager.Task] (Task='dc8fbf34-8d7e-47a3-8e02-0a5e5cb90257')
>> Unexpected error (task:875)
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
>> 882, in _run
>> return fn(*args, **kargs)
>> File "<string>", line 2, in prepareImage
>> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50,
>> in method
>> ret = func(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line
>> 3199, in prepareImage
>> legality = dom.produceVolume(imgUUID, volUUID).getLegality()
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line
822,
>> in produceVolume
>> volUUID)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line
>> 801, in __init__
>> self._manifest = self.manifestClass(repoPath, sdUUID, imgUUID,
>> volUUID)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py",
>> line 71, in __init__
>> volUUID)
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line
>> 86, in __init__
>> self.validate()
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/volume.py", line
>> 112, in validate
>> self.validateVolumePath()
>> File "/usr/lib/python2.7/site-packages/vdsm/storage/fileVolume.py",
>> line 131, in validateVolumePath
>> raise se.VolumeDoesNotExist(self.volUUID)
>> VolumeDoesNotExist: Volume does not exist:
>> (u'81685d19-0060-4f5d-a4cd-c5efa24aecfe',)
>> 2019-03-19 14:08:25,840+0200 INFO (vm/a492d2eb)
>> [storage.TaskManager.Task] (Task='dc8fbf34-8d7e-47a3-8e02-0a5e5cb90257')
>> aborting: Task is aborted: "Volume does not exist:
>> (u'81685d19-0060-4f5d-a4cd-c5efa24aecfe',)" - code 201 (task:1181)
>> 2019-03-19 14:08:25,840+0200 ERROR (vm/a492d2eb) [storage.Dispatcher]
>> FINISH prepareImage error=Volume does not exist:
>> (u'81685d19-0060-4f5d-a4cd-c5efa24aecfe',) (dispatcher:83)
>>
>> I think it's still https://bugzilla.redhat.com/1666795
>> <https://bugzilla.redhat.com/show_bug.cgi?id=1666795>
>>
>> Can you please try updating vdsm to vdsm-4.30.10 since the bug is
>> reported as solved in that version?
>>
>>
>>
>>
>> On Tue, Mar 19, 2019 at 12:30 PM ada per <adaper3(a)gmail.com> wrote:
>>
>>> an vdsm:
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 19, 2019 at 1:24 PM ada per <adaper3(a)gmail.com> wrote:
>>>
>>>> Thank you! please see attached files:
>>>>
>>>> On Tue, Mar 19, 2019 at 12:52 PM Simone Tiraboschi <
>>>> stirabos(a)redhat.com> wrote:
>>>>
>>>>> Can you please check/attach also
>>>>> /var/log/ovirt-hosted-engine-ha/broker.log and /var/log/vdsm/vdsm.log
?
>>>>>
>>>>> On Tue, Mar 19, 2019 at 11:36 AM ada per <adaper3(a)gmail.com>
wrote:
>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> For a strange reason the hosted engine went down and I cannot
>>>>>> restart it. I tried manually restarting it without any success
can you
>>>>>> please advice?
>>>>>>
>>>>>> For all the nodes the engine status is the same as the one
below.
>>>>>> --== Host nodex. (id: 6) status ==--
>>>>>> conf_on_shared_storage : True
>>>>>> Status up-to-date : True
>>>>>> Hostname : nodex
>>>>>> Host ID : 6
>>>>>> Engine status : {"reason":
"bad vm status",
>>>>>> "health": "bad", "vm":
"down_unexpected", "detail": "Down"}
>>>>>> Score : 3400
>>>>>> stopped : False
>>>>>> Local maintenance : False
>>>>>> crc32 : 323a9f45
>>>>>> local_conf_timestamp : 2648874
>>>>>> Host timestamp : 2648874
>>>>>> Extra metadata (valid at timestamp):
>>>>>> metadata_parse_version=1
>>>>>> metadata_feature_version=1
>>>>>> timestamp=2648874 (Tue Mar 19 12:25:44 2019)
>>>>>> host-id=6
>>>>>> score=3400
>>>>>> vm_conf_refresh_time=2648874 (Tue Mar 19 12:25:44 2019)
>>>>>> conf_on_shared_storage=True
>>>>>> maintenance=False
>>>>>> state=GlobalMaintenance
>>>>>> stopped=False
>>>>>>
>>>>>> When I try the commands
>>>>>> root@node5# hosted-engine --vm-shutdown
>>>>>> I ge the response:
>>>>>> root@node5# Command VM.shutdown with args {'delay':
'120',
>>>>>> 'message': 'VM is shutting down!',
'vmID':
>>>>>> 'a492d2eb-1dfd-470d-a141-3e55d2189275'} failed:(code=1,
message=Virtual
>>>>>> machine does not exist)
>>>>>>
>>>>>> But when I run : hosted-engine --vm-start
>>>>>> I get the response: VM exists and is down, cleaning up and
restarting
>>>>>>
>>>>>>
>>>>>>
>>>>>> Below you can see the # journalctl -u ovirt-ha-agent logs
>>>>>>
>>>>>> Mar 14 12:04:42 node7. ovirt-ha-agent[4134]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Unhandled
>>>>>> monitoring loop exception
>>>>>>
Traceback
>>>>>> (most recent call last):
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 430, in start_monitoring
>>>>>>
>>>>>> self._monitoring_loop()
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 449, in _monitoring_loop
>>>>>>
for
>>>>>> old_state, state, delay in self.fsm:
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py",
>>>>>> line 127, in next
>>>>>>
>>>>>> new_data = self.refresh(self._state.data)
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py",
>>>>>> line 81, in refresh
>>>>>>
>>>>>> stats.update(self.hosted_engine.collect_stats())
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 737, in collect_stats
>>>>>>
>>>>>> all_stats = self._broker.get_stats_from_storage()
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 143, in get_stats_from_storage
>>>>>>
result
>>>>>> = self._proxy.get_stats()
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in
__call__
>>>>>>
return
>>>>>> self.__send(self.__name, args)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in
__request
>>>>>>
>>>>>> verbose=self.__verbose
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in
request
>>>>>>
return
>>>>>> self.single_request(host, handler, request_body, verbose)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in
single_request
>>>>>>
>>>>>> self.send_content(h, request_body)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in
send_content
>>>>>>
>>>>>> connection.endheaders(request_body)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/httplib.py", line 1037, in
endheaders
>>>>>>
>>>>>> self._send_output(message_body)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/httplib.py", line 881, in
_send_output
>>>>>>
>>>>>> self.send(msg)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/httplib.py", line 843, in send
>>>>>>
>>>>>> self.connect()
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
>>>>>> line 52, in connect
>>>>>>
>>>>>> self.sock.connect(base64.b16decode(self.host))
>>>>>> File
>>>>>> "/usr/lib64/python2.7/socket.py", line 224, in meth
>>>>>>
return
>>>>>> getattr(self._sock,name)(*args)
>>>>>> error:
>>>>>> [Errno 2] No such file or directory
>>>>>> Mar 14 12:04:42 node7. ovirt-ha-agent[4134]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most
recent call
>>>>>> last):
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
>>>>>> line 131, in _run_agent
>>>>>>
return
>>>>>> action(he)
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
>>>>>> line 55, in action_proper
>>>>>>
return
>>>>>> he.start_monitoring()
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 437, in start_monitoring
>>>>>>
>>>>>> self.publish(stopped)
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 337, in publish
>>>>>>
>>>>>> self._push_to_storage(blocks)
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 708, in _push_to_storage
>>>>>>
>>>>>> self._broker.put_stats_on_storage(self.host_id, blocks)
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 113, in put_stats_on_storage
>>>>>>
>>>>>> self._proxy.put_stats(host_id, xmlrpclib.Binary(data))
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in
__call__
>>>>>>
return
>>>>>> self.__send(self.__name, args)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in
__request
>>>>>>
>>>>>> verbose=self.__verbose
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in
request
>>>>>>
return
>>>>>> self.single_request(host, handler, request_body, verbose)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1301, in
single_request
>>>>>>
>>>>>> self.send_content(h, request_body)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/xmlrpclib.py", line 1448, in
send_content
>>>>>>
>>>>>> connection.endheaders(request_body)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/httplib.py", line 1037, in
endheaders
>>>>>>
>>>>>> self._send_output(message_body)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/httplib.py", line 881, in
_send_output
>>>>>>
>>>>>> self.send(msg)
>>>>>> File
>>>>>> "/usr/lib64/python2.7/httplib.py", line 843, in send
>>>>>>
>>>>>> self.connect()
>>>>>> File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py",
>>>>>> line 52, in connect
>>>>>>
>>>>>> self.sock.connect(base64.b16decode(self.host))
>>>>>> File
>>>>>> "/usr/lib64/python2.7/socket.py", line 224, in meth
>>>>>>
return
>>>>>> getattr(self._sock,name)(*args)
>>>>>> error:
>>>>>> [Errno 2] No such file or directory
>>>>>> Mar 14 12:04:42 node7. ovirt-ha-agent[4134]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart
agent
>>>>>> Mar 14 12:04:42 node7. systemd[1]: ovirt-ha-agent.service: main
>>>>>> process exited, code=exited, status=157/n/a
>>>>>> Mar 14 12:04:42 node7. systemd[1]: Unit ovirt-ha-agent.service
>>>>>> entered failed state.
>>>>>> Mar 14 12:04:42 node7. systemd[1]: ovirt-ha-agent.service
failed.
>>>>>> Mar 14 12:04:52 node7. systemd[1]: ovirt-ha-agent.service
holdoff
>>>>>> time over, scheduling restart.
>>>>>> Mar 14 12:04:52 node7. systemd[1]: Stopped oVirt Hosted Engine
High
>>>>>> Availability Monitoring Agent.
>>>>>> Mar 14 12:04:52 node7. systemd[1]: Started oVirt Hosted Engine
High
>>>>>> Availability Monitoring Agent.
>>>>>> Mar 14 12:04:52 node7. ovirt-ha-agent[31765]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Failed to
>>>>>> start necessary monitors
>>>>>> Mar 14 12:04:52 node7. ovirt-ha-agent[31765]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most
recent call
>>>>>> last):
>>>>>>
File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
>>>>>> line 131, in _run_agent
>>>>>>
>>>>>> return action(he)
>>>>>>
File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
>>>>>> line 55, in action_proper
>>>>>>
>>>>>> return he.start_monitoring()
>>>>>>
File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 413, in start_monitoring
>>>>>>
>>>>>> self._initialize_broker()
>>>>>>
File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
>>>>>> line 537, in _initialize_broker
>>>>>>
>>>>>> m.get('options', {}))
>>>>>>
File
>>>>>>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
>>>>>> line 86, in start_monitor
>>>>>>
>>>>>> ).format(t=type, o=options, e=e)
>>>>>>
>>>>>> RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker:
>>>>>> [Errno 2] No such file or directory, [monitor: 'ping',
options: {'addr': '19
>>>>>> Mar 14 12:04:52 node7. ovirt-ha-agent[31765]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart
agent
>>>>>> Mar 14 12:04:52 node7. systemd[1]: ovirt-ha-agent.service: main
>>>>>> process exited, code=exited, status=157/n/a
>>>>>> Mar 14 12:04:52 node7. systemd[1]: Unit ovirt-ha-agent.service
>>>>>> entered failed state.
>>>>>> Mar 14 12:04:52 node7. systemd[1]: ovirt-ha-agent.service
failed.
>>>>>> Mar 14 12:05:02 node7. systemd[1]: ovirt-ha-agent.service
holdoff
>>>>>> time over, scheduling restart.
>>>>>> Mar 14 12:05:02 node7. systemd[1]: Stopped oVirt Hosted Engine
High
>>>>>> Availability Monitoring Agent.
>>>>>> Mar 14 12:05:02 node7. systemd[1]: Started oVirt Hosted Engine
High
>>>>>> Availability Monitoring Agent.
>>>>>> Mar 14 12:06:55 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Failed to
>>>>>> stop engine vm with /usr/sbin/hosted-engine --vm-poweroff: Co
>>>>>>
(code=1,
>>>>>> message=Virtual machine does not exist: {'vmId':
>>>>>> u'a492d2eb-1dfd-470d-a141-3e55d2189275'})
>>>>>> Mar 14 12:06:55 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Failed to
>>>>>> stop engine VM: Command VM.destroy with args {'vmID':
'a492d2
>>>>>>
(code=1,
>>>>>> message=Virtual machine does not exist: {'vmId':
>>>>>> u'a492d2eb-1dfd-470d-a141-3e55d2189275'})
>>>>>> Mar 15 14:28:16 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:28:36 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:29:00 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:29:22 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:29:44 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:30:06 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:30:28 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:30:50 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:31:12 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:31:33 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:31:56 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:32:18 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:32:40 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> Mar 15 14:33:02 node7. ovirt-ha-agent[31822]: ovirt-ha-agent
>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR
Engine VM
>>>>>> stopped on localhost
>>>>>> _______________________________________________
>>>>>> Users mailing list -- users(a)ovirt.org
>>>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>>> oVirt Code of Conduct:
>>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>>> List Archives:
>>>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/NS2SASAK66T...
>>>>>>
>>>>>