Unable to get HE up after update
Hi, i've been discussing this issue over at IRC (#ovirt on irc.oftc.net)) and was suggested to send an email with logs the ML. The problem i have is that the HE is a loop getting up and down ( ( i assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a couple of VMs i use daily with SPICE and remoteviewer and i hope to get them up and running again. Due to this, i've put it in global maintenance mode. I'm attaching the server log in the hope of getting some help and can provide more details if needed to debug the issue further. The log is too big for fpaste so it's to be found here: https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached here. Thanks! -- Susinthiran Sithamparanathan
Can you please provide the HA agent logs? On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i've been discussing this issue over at IRC (#ovirt on irc.oftc.net)) and was suggested to send an email with logs the ML.
The problem i have is that the HE is a loop getting up and down ( ( i assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a couple of VMs i use daily with SPICE and remoteviewer and i hope to get them up and running again. Due to this, i've put it in global maintenance mode. I'm attaching the server log in the hope of getting some help and can provide more details if needed to debug the issue further.
The log is too big for fpaste so it's to be found here: https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached here.
Thanks!
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Sure, here it is: https://my.owndrive.com/index.php/s/MFoFyKJVLjzezey On Sun, Oct 9, 2016 at 3:19 PM, Doron Fediuck <dfediuck@redhat.com> wrote:
Can you please provide the HA agent logs?
On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i've been discussing this issue over at IRC (#ovirt on irc.oftc.net)) and was suggested to send an email with logs the ML.
The problem i have is that the HE is a loop getting up and down ( ( i assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a couple of VMs i use daily with SPICE and remoteviewer and i hope to get them up and running again. Due to this, i've put it in global maintenance mode. I'm attaching the server log in the hope of getting some help and can provide more details if needed to debug the issue further.
The log is too big for fpaste so it's to be found here: https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached here.
Thanks!
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Susinthiran Sithamparanathan
On Sun, Oct 9, 2016 at 5:13 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Sure, here it is: https://my.owndrive.com/index.php/s/MFoFyKJVLjzezey
The agent is periodically restarting the engine VM but from the logs I don't see why. Can you please set the agent in debug mode and share again its logs? You have to edit /etc/ovirt-hosted-engine-ha/agent-log.conf changing from [logger_root] level=INFO to [logger_root] level=DEBUG and then restart ovirt-ha-agent.
On Sun, Oct 9, 2016 at 3:19 PM, Doron Fediuck <dfediuck@redhat.com> wrote:
Can you please provide the HA agent logs?
On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i've been discussing this issue over at IRC (#ovirt on irc.oftc.net)) and was suggested to send an email with logs the ML.
The problem i have is that the HE is a loop getting up and down ( ( i assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a couple of VMs i use daily with SPICE and remoteviewer and i hope to get them up and running again. Due to this, i've put it in global maintenance mode. I'm attaching the server log in the hope of getting some help and can provide more details if needed to debug the issue further.
The log is too big for fpaste so it's to be found here: https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached here.
Thanks!
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Oct 10, 2016 at 10:56 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sun, Oct 9, 2016 at 5:13 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Sure, here it is: https://my.owndrive.com/index.php/s/MFoFyKJVLjzezey
The agent is periodically restarting the engine VM but from the logs I don't see why.
Also it keeps doing: MainThread::INFO::2016-10-09 17:06:01,025::hosted_engine::612::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2016-10-09 17:06:05,118::hosted_engine::639::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2016-10-09 17:06:05,131::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,459::storage_server::225::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,496::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2016-10-09 17:06:13,737::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2016-10-09 17:06:13,737::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images Does this make sense, Simone? Please check/share also /var/log/vdsm/* . Thanks.
Can you please set the agent in debug mode and share again its logs?
You have to edit /etc/ovirt-hosted-engine-ha/agent-log.conf changing from
[logger_root] level=INFO
to [logger_root] level=DEBUG
and then restart ovirt-ha-agent.
On Sun, Oct 9, 2016 at 3:19 PM, Doron Fediuck <dfediuck@redhat.com> wrote:
Can you please provide the HA agent logs?
On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Hi, i've been discussing this issue over at IRC (#ovirt on irc.oftc.net)) and was suggested to send an email with logs the ML.
The problem i have is that the HE is a loop getting up and down ( ( i assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a couple of VMs i use daily with SPICE and remoteviewer and i hope to get them up and running again. Due to this, i've put it in global maintenance mode. I'm attaching the server log in the hope of getting some help and can provide more details if needed to debug the issue further.
The log is too big for fpaste so it's to be found here: https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached here.
Thanks!
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
On Mon, Oct 10, 2016 at 10:13 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 10, 2016 at 10:56 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sun, Oct 9, 2016 at 5:13 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Sure, here it is: https://my.owndrive.com/index.php/s/MFoFyKJVLjzezey
The agent is periodically restarting the engine VM but from the logs I
don't
see why.
Also it keeps doing:
Yes, this is fine: by design ovirt-ha-agent periodically (about 30-40 seconds) reconnects the hosted-engine storage domain.
MainThread::INFO::2016-10-09 17:06:01,025::hosted_engine::612::ovirt_hosted_engine_ha. agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2016-10-09 17:06:05,118::hosted_engine::639::ovirt_hosted_engine_ha. agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2016-10-09 17:06:05,131::storage_server::218::ovirt_hosted_engine_ha. lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,459::storage_server::225::ovirt_hosted_engine_ha. lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,496::storage_server::232::ovirt_hosted_engine_ha. lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2016-10-09 17:06:13,737::hosted_engine::666::ovirt_hosted_engine_ha. agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2016-10-09 17:06:13,737::image::126::ovirt_hosted_engine_ha.lib. image.Image::(prepare_images) Preparing images
Does this make sense, Simone?
Please check/share also /var/log/vdsm/* . Thanks.
Can you please set the agent in debug mode and share again its logs?
You have to edit /etc/ovirt-hosted-engine-ha/agent-log.conf changing from
[logger_root] level=INFO
to [logger_root] level=DEBUG
and then restart ovirt-ha-agent.
On Sun, Oct 9, 2016 at 3:19 PM, Doron Fediuck <dfediuck@redhat.com>
wrote:
Can you please provide the HA agent logs?
On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Hi, i've been discussing this issue over at IRC (#ovirt on irc.oftc.net)) and was suggested to send an email with logs the ML.
The problem i have is that the HE is a loop getting up and down ( ( i assume it's the HA trying to get it up) after upgrade to oVirt 4. I
have a
couple of VMs i use daily with SPICE and remoteviewer and i hope to get them up and running again. Due to this, i've put it in global maintenance mode. I'm attaching the server log in the hope of getting some help and can provide more details if needed to debug the issue further.
The log is too big for fpaste so it's to be found here: https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached here.
Thanks!
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
Hi, all the logs are now at https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8 I did a quick debug in the VM and i think we are getting closer to the root cause: https://paste.fedoraproject.org/447579/14760912/ It seems the SSL/TLS certs are all missing. Now i wonder which RPM package does contain these so that i can try to reinstall it. Appreciate your help so far! On Mon, Oct 10, 2016 at 10:17 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 10, 2016 at 10:13 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 10, 2016 at 10:56 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sun, Oct 9, 2016 at 5:13 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Sure, here it is: https://my.owndrive.com/index.php/s/MFoFyKJVLjzezey
The agent is periodically restarting the engine VM but from the logs I
don't
see why.
Also it keeps doing:
Yes, this is fine: by design ovirt-ha-agent periodically (about 30-40 seconds) reconnects the hosted-engine storage domain.
MainThread::INFO::2016-10-09 17:06:01,025::hosted_engine::612::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2016-10-09 17:06:05,118::hosted_engine::639::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2016-10-09 17:06:05,131::storage_server::218::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,459::storage_server::225::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,496::storage_server::232::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2016-10-09 17:06:13,737::hosted_engine::666::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2016-10-09 17:06:13,737::image::126::ovirt_hosted_engine_ha.lib.image. Image::(prepare_images) Preparing images
Does this make sense, Simone?
Please check/share also /var/log/vdsm/* . Thanks.
Can you please set the agent in debug mode and share again its logs?
You have to edit /etc/ovirt-hosted-engine-ha/agent-log.conf changing from
[logger_root] level=INFO
to [logger_root] level=DEBUG
and then restart ovirt-ha-agent.
On Sun, Oct 9, 2016 at 3:19 PM, Doron Fediuck <dfediuck@redhat.com>
wrote:
Can you please provide the HA agent logs?
On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Hi, i've been discussing this issue over at IRC (#ovirt on irc.oftc.net
))
and was suggested to send an email with logs the ML.
The problem i have is that the HE is a loop getting up and down ( ( i assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a couple of VMs i use daily with SPICE and remoteviewer and i hope to get them up and running again. Due to this, i've put it in global maintenance mode. I'm attaching the server log in the hope of getting some help and can provide more details if needed to debug the issue further.
The log is too big for fpaste so it's to be found here: https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached here.
Thanks!
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
-- Susinthiran Sithamparanathan
On Mon, Oct 10, 2016 at 11:27 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, all the logs are now at https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8
I did a quick debug in the VM and i think we are getting closer to the root cause: https://paste.fedoraproject.org/447579/14760912/
It seems the SSL/TLS certs are all missing. Now i wonder which RPM package does contain these so that i can try to reinstall it.
Appreciate your help so far!
OK, the issue on the host is just here: MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::282::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Full response: success {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"} MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::255::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_checked_communicate) Successful response from socket MainThread::DEBUG::2016-10-10 11:18:34,170::brokerlink::151::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(get_monitor_status) Success, status {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"} the engine VM goes up but the engine no and so after a certain amount of time it tries again with a reboot. We should definitively add a more explicit log entry there! Now the point is just why your engine is not starting.
On Mon, Oct 10, 2016 at 10:17 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 10, 2016 at 10:13 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 10, 2016 at 10:56 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Sun, Oct 9, 2016 at 5:13 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Sure, here it is: https://my.owndrive.com/index.php/s/MFoFyKJVLjzezey
The agent is periodically restarting the engine VM but from the logs I
don't
see why.
Also it keeps doing:
Yes, this is fine: by design ovirt-ha-agent periodically (about 30-40 seconds) reconnects the hosted-engine storage domain.
MainThread::INFO::2016-10-09 17:06:01,025::hosted_engine::612::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2016-10-09 17:06:05,118::hosted_engine::639::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2016-10-09 17:06:05,131::storage_server::218::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,459::storage_server::225::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,496::storage_server::232::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2016-10-09 17:06:13,737::hosted_engine::666::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2016-10-09 17:06:13,737::image::126::ovirt_hosted_engine_ha.lib.image.I mage::(prepare_images) Preparing images
Does this make sense, Simone?
Please check/share also /var/log/vdsm/* . Thanks.
Can you please set the agent in debug mode and share again its logs?
You have to edit /etc/ovirt-hosted-engine-ha/agent-log.conf changing from
[logger_root] level=INFO
to [logger_root] level=DEBUG
and then restart ovirt-ha-agent.
On Sun, Oct 9, 2016 at 3:19 PM, Doron Fediuck <dfediuck@redhat.com>
Can you please provide the HA agent logs?
On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote: > > Hi, > i've been discussing this issue over at IRC (#ovirt on irc.oftc.net
))
> and was suggested to send an email with logs the ML. > > The problem i have is that the HE is a loop getting up and down ( ( i > assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a > couple of VMs i use daily with SPICE and remoteviewer and i hope to get them > up and running again. Due to this, i've put it in global
wrote: maintenance mode.
> I'm attaching the server log in the hope of getting some help and can > provide more details if needed to debug the issue further. > > The log is too big for fpaste so it's to be found here: > https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached > here. > > > Thanks! > > > -- > > Susinthiran Sithamparanathan > > _______________________________________________ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
--
Susinthiran Sithamparanathan
On Mon, Oct 10, 2016 at 11:40 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 10, 2016 at 11:27 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, all the logs are now at https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8
I did a quick debug in the VM and i think we are getting closer to the root cause: https://paste.fedoraproject.org/447579/14760912/
It seems the SSL/TLS certs are all missing. Now i wonder which RPM package does contain these so that i can try to reinstall it.
Appreciate your help so far!
OK, the issue on the host is just here: MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::282: :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Full response: success {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"} MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::255: :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_checked_communicate) Successful response from socket MainThread::DEBUG::2016-10-10 11:18:34,170::brokerlink::151: :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(get_monitor_status) Success, status {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"}
the engine VM goes up but the engine no and so after a certain amount of time it tries again with a reboot. We should definitively add a more explicit log entry there!
Now the point is just why your engine is not starting.
Can you please upload you engine-setup logs from the engine VM?
On Mon, Oct 10, 2016 at 10:17 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 10, 2016 at 10:13 AM, Yedidyah Bar David <didi@redhat.com> wrote:
On Mon, Oct 10, 2016 at 10:56 AM, Simone Tiraboschi < stirabos@redhat.com> wrote:
On Sun, Oct 9, 2016 at 5:13 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
Sure, here it is: https://my.owndrive.com/index.
php/s/MFoFyKJVLjzezey
The agent is periodically restarting the engine VM but from the logs I don't see why.
Also it keeps doing:
Yes, this is fine: by design ovirt-ha-agent periodically (about 30-40 seconds) reconnects the hosted-engine storage domain.
MainThread::INFO::2016-10-09 17:06:01,025::hosted_engine::612::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2016-10-09 17:06:05,118::hosted_engine::639::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2016-10-09 17:06:05,131::storage_server::218::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,459::storage_server::225::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2016-10-09 17:06:13,496::storage_server::232::ovirt_hosted_engine_ha.li b.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2016-10-09 17:06:13,737::hosted_engine::666::ovirt_hosted_engine_ha.age nt.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2016-10-09 17:06:13,737::image::126::ovirt_hosted_engine_ha.lib.image.I mage::(prepare_images) Preparing images
Does this make sense, Simone?
Please check/share also /var/log/vdsm/* . Thanks.
Can you please set the agent in debug mode and share again its logs?
You have to edit /etc/ovirt-hosted-engine-ha/agent-log.conf changing from
[logger_root] level=INFO
to [logger_root] level=DEBUG
and then restart ovirt-ha-agent.
On Sun, Oct 9, 2016 at 3:19 PM, Doron Fediuck <dfediuck@redhat.com>
> > Can you please provide the HA agent logs? > > On Sun, Oct 9, 2016 at 1:57 PM, Susinthiran Sithamparanathan > <chesusin@gmail.com> wrote: >> >> Hi, >> i've been discussing this issue over at IRC (#ovirt on irc.oftc.net)) >> and was suggested to send an email with logs the ML. >> >> The problem i have is that the HE is a loop getting up and down ( ( i >> assume it's the HA trying to get it up) after upgrade to oVirt 4. I have a >> couple of VMs i use daily with SPICE and remoteviewer and i hope to get them >> up and running again. Due to this, i've put it in global
wrote: maintenance mode.
>> I'm attaching the server log in the hope of getting some help and can >> provide more details if needed to debug the issue further. >> >> The log is too big for fpaste so it's to be found here: >> https://my.owndrive.com/index.php/s/Gk4De1jy9fWmFa8 as well as attached >> here. >> >> >> Thanks! >> >> >> -- >> >> Susinthiran Sithamparanathan >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
--
Susinthiran Sithamparanathan
On Mon, Oct 10, 2016 at 11:40 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
OK, the issue on the host is just here: MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::282: :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Full response: success {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"} MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::255: :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_checked_communicate) Successful response from socket MainThread::DEBUG::2016-10-10 11:18:34,170::brokerlink::151: :ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(get_monitor_status) Success, status {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"}
the engine VM goes up but the engine no and so after a certain amount of time it tries again with a reboot. We should definitively add a more explicit log entry there!
Now the point is just why your engine is not starting.
Perhaps you ignored https://paste.fedoraproject.org/447579/14760912/? It seems the reason it doesn't start is due to some missing SSL/TLS certs inside the VM. Or do you think the problem is still somewhere else? I've uploaded engine-setup logs to https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8 in the folder engine-setup and you'll see many. I did desperately try to get up the HE when it failed after upgrade by trying to manually setup a new HE which failed (couple of times IIRC). So that might have caused couple of additional logs files in there. -- Susinthiran Sithamparanathan
On Mon, Oct 10, 2016 at 9:22 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Mon, Oct 10, 2016 at 11:40 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
OK, the issue on the host is just here: MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::282:: ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Full response: success {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"} MainThread::DEBUG::2016-10-10 11:18:34,169::brokerlink::255:: ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_checked_communicate) Successful response from socket MainThread::DEBUG::2016-10-10 11:18:34,170::brokerlink::151:: ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(get_monitor_status) Success, status {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "up"}
the engine VM goes up but the engine no and so after a certain amount of time it tries again with a reboot. We should definitively add a more explicit log entry there!
Now the point is just why your engine is not starting.
Perhaps you ignored https://paste.fedoraproject.org/447579/14760912/? It seems the reason it doesn't start is due to some missing SSL/TLS certs inside the VM. Or do you think the problem is still somewhere else?
No, the issue is definitively there but I was trying to identify the root cause.
I've uploaded engine-setup logs to https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8 in the folder engine-setup and you'll see many.
You uploaded ovirt-hosted-engine-setup logs from the host while now we need engine-setup logs from the engine VM.
I did desperately try to get up the HE when it failed after upgrade by trying to manually setup a new HE which failed (couple of times IIRC). So that might have caused couple of additional logs files in there.
--
Susinthiran Sithamparanathan
On Tue, Oct 11, 2016 at 9:42 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
No, the issue is definitively there but I was trying to identify the root cause.
Yes, hopefully we'll pretty soon.
I've uploaded engine-setup logs to https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8 in the folder engine-setup and you'll see many.
You uploaded ovirt-hosted-engine-setup logs from the host while now we need engine-setup logs from the engine VM.
You can find the logs under engine-vm https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8. I've uploaded the other logs from the engine VM in case those could be useful in our debugging purpose. -- Susinthiran Sithamparanathan (who forgot to send to the list + Yedidyah hence this email)
On Tue, Oct 11, 2016 at 10:19 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Tue, Oct 11, 2016 at 9:42 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
No, the issue is definitively there but I was trying to identify the root cause.
Yes, hopefully we'll pretty soon.
I've uploaded engine-setup logs to https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8 in the folder engine-setup and you'll see many.
You uploaded ovirt-hosted-engine-setup logs from the host while now we need engine-setup logs from the engine VM.
You can find the logs under engine-vm https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8. I've uploaded the other logs from the engine VM in case those could be useful in our debugging purpose.
Thanks, I found the issue in your https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8/download?path=%2Fengine-vm%2Fsetup&files=ovirt-engine-setup-20160908144513-wfbxna.log due to this: 2016-09-08 14:45:14 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/core/advertise_dwh.py", line 65, in _init if os.path.isdir(self._engine_manual): File "/usr/lib64/python2.7/genericpath.py", line 41, in isdir st = os.stat(s) TypeError: coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Initializing': coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True' 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.TypeError'>, TypeError('coercing to Unicode: need string or buffer, NoneType found',), <traceback object at 0x181edd0>)]' engine-setup didn't completed during the upgrade process and you ended with a not working engine. I'll try to understand why it happened.
--
Susinthiran Sithamparanathan (who forgot to send to the list + Yedidyah hence this email)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Wed, Oct 12, 2016 at 11:47 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Tue, Oct 11, 2016 at 10:19 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Tue, Oct 11, 2016 at 9:42 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
No, the issue is definitively there but I was trying to identify the root cause.
Yes, hopefully we'll pretty soon.
I've uploaded engine-setup logs to https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8 in the folder engine-setup and you'll see many.
You uploaded ovirt-hosted-engine-setup logs from the host while now we need engine-setup logs from the engine VM.
You can find the logs under engine-vm https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8. I've uploaded the other logs from the engine VM in case those could be useful in our debugging purpose.
Thanks, I found the issue in your https://my.owndrive.com/ index.php/s/3Dcyho9bqo7oZs8/download?path=%2Fengine-vm% 2Fsetup&files=ovirt-engine-setup-20160908144513-wfbxna.log
due to this:
2016-09-08 14:45:14 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt- engine-setup/ovirt-engine/core/advertise_dwh.py", line 65, in _init if os.path.isdir(self._engine_manual): File "/usr/lib64/python2.7/genericpath.py", line 41, in isdir st = os.stat(s) TypeError: coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Initializing': coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True' 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.TypeError'>, TypeError('coercing to Unicode: need string or buffer, NoneType found',), <traceback object at 0x181edd0>)]'
engine-setup didn't completed during the upgrade process and you ended with a not working engine. I'll try to understand why it happened.
Can you please share /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf from that engine VM?
--
Susinthiran Sithamparanathan (who forgot to send to the list + Yedidyah hence this email)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I see. The logs in the root of https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8. Thanks for your time and looking forward to be able to use my VMs again :) On Wed, Oct 12, 2016 at 12:13 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Wed, Oct 12, 2016 at 11:47 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Tue, Oct 11, 2016 at 10:19 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Tue, Oct 11, 2016 at 9:42 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
No, the issue is definitively there but I was trying to identify the root cause.
Yes, hopefully we'll pretty soon.
I've uploaded engine-setup logs to https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8 in the folder engine-setup and you'll see many.
You uploaded ovirt-hosted-engine-setup logs from the host while now we need engine-setup logs from the engine VM.
You can find the logs under engine-vm https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8. I've uploaded the other logs from the engine VM in case those could be useful in our debugging purpose.
Thanks, I found the issue in your https://my.owndrive.com/i ndex.php/s/3Dcyho9bqo7oZs8/download?path=%2Fengine-vm%2Fsetu p&files=ovirt-engine-setup-20160908144513-wfbxna.log
due to this:
2016-09-08 14:45:14 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine- setup/ovirt-engine/core/advertise_dwh.py", line 65, in _init if os.path.isdir(self._engine_manual): File "/usr/lib64/python2.7/genericpath.py", line 41, in isdir st = os.stat(s) TypeError: coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Initializing': coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True' 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.TypeError'>, TypeError('coercing to Unicode: need string or buffer, NoneType found',), <traceback object at 0x181edd0>)]'
engine-setup didn't completed during the upgrade process and you ended with a not working engine. I'll try to understand why it happened.
Can you please share /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf from that engine VM?
--
Susinthiran Sithamparanathan (who forgot to send to the list + Yedidyah hence this email)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Susinthiran Sithamparanathan
On Wed, Oct 12, 2016 at 6:41 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
I see. The logs in the root of https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8. Thanks for your time and looking forward to be able to use my VMs again :)
On Wed, Oct 12, 2016 at 12:13 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Wed, Oct 12, 2016 at 11:47 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Tue, Oct 11, 2016 at 10:19 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Tue, Oct 11, 2016 at 9:42 AM, Simone Tiraboschi <stirabos@redhat.com
wrote:
No, the issue is definitively there but I was trying to identify the root cause.
Yes, hopefully we'll pretty soon.
I've uploaded engine-setup logs to https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8 in the folder engine-setup and you'll see many.
You uploaded ovirt-hosted-engine-setup logs from the host while now we need engine-setup logs from the engine VM.
You can find the logs under engine-vm https://my.owndrive.com/index. php/s/3Dcyho9bqo7oZs8. I've uploaded the other logs from the engine VM in case those could be useful in our debugging purpose.
Thanks, I found the issue in your https://my.owndrive.com/i ndex.php/s/3Dcyho9bqo7oZs8/download?path=%2Fengine-vm%2Fsetu p&files=ovirt-engine-setup-20160908144513-wfbxna.log
due to this:
2016-09-08 14:45:14 DEBUG otopi.context context._executeMethod:142 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-s etup/ovirt-engine/core/advertise_dwh.py", line 65, in _init if os.path.isdir(self._engine_manual): File "/usr/lib64/python2.7/genericpath.py", line 41, in isdir st = os.stat(s) TypeError: coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Initializing': coercing to Unicode: need string or buffer, NoneType found 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:760 ENVIRONMENT DUMP - BEGIN 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/error=bool:'True' 2016-09-08 14:45:14 DEBUG otopi.context context.dumpEnvironment:770 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.TypeError'>, TypeError('coercing to Unicode: need string or buffer, NoneType found',), <traceback object at 0x181edd0>)]'
engine-setup didn't completed during the upgrade process and you ended with a not working engine. I'll try to understand why it happened.
Can you please share /usr/share/ovirt-engine/servic es/ovirt-engine/ovirt-engine.conf from that engine VM?
Is this file copied to [1]? The file in [1] is empty - has a size of zero. Perhaps your engine vm suffered some serious corruption? Perhaps due to storage/network/power problems? Please check/post the output of: rpm -V ovirt-engine-backend [1] https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8/download?path=%2Fengine-vm&files=ovirt-engine.conf
--
Susinthiran Sithamparanathan (who forgot to send to the list + Yedidyah hence this email)
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--
Susinthiran Sithamparanathan
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Didi
On Thu, Oct 13, 2016 at 10:45 AM, Yedidyah Bar David <didi@redhat.com> wrote:
Is this file copied to [1]? The file in [1] is empty - has a size of zero. Perhaps your engine vm suffered some serious corruption? Perhaps due to storage/network/power problems? Please check/post the output of:
Yes, it was actually under engine-vm/ and not under the root, but something happened with the sync of the file, but is synced correctly now. Sorry for that!
rpm -V ovirt-engine-backend [root@engine ~]# rpm -V ovirt-engine-backend S.5....T. c /etc/logrotate.d/ovirt-engine S.5....T. c /etc/pki/ovirt-engine/cacert.template.in S.5....T. c /etc/pki/ovirt-engine/cert.template.in S.5....T. c /etc/pki/ovirt-engine/openssl.conf [root@engine ~]# [1] https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8/ download?path=%2Fengine-vm&files=ovirt-engine.conf --
Susinthiran Sithamparanathan
On Thu, Oct 13, 2016 at 2:42 PM, Susinthiran Sithamparanathan <chesusin@gmail.com> wrote:
On Thu, Oct 13, 2016 at 10:45 AM, Yedidyah Bar David <didi@redhat.com> wrote:
Is this file copied to [1]? The file in [1] is empty - has a size of zero. Perhaps your engine vm suffered some serious corruption? Perhaps due to storage/network/power problems? Please check/post the output of:
Yes, it was actually under engine-vm/ and not under the root, but something happened with the sync of the file, but is synced correctly now. Sorry for that!
rpm -V ovirt-engine-backend [root@engine ~]# rpm -V ovirt-engine-backend S.5....T. c /etc/logrotate.d/ovirt-engine S.5....T. c /etc/pki/ovirt-engine/cacert.template.in S.5....T. c /etc/pki/ovirt-engine/cert.template.in S.5....T. c /etc/pki/ovirt-engine/openssl.conf [root@engine ~]# [1] https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8/download?path=%2Fengine-vm&files=ovirt-engine.conf
OK. Can you please attach the output of: grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf Thanks. -- Didi
On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com> wrote:
OK. Can you please attach the output of:
grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf
[root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@ovirt01 ~]# ssh 192.168.0.101 root@192.168.0.101's password: Last login: Thu Oct 13 19:50:02 2016 from ovirt01 [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@engine ~]# I.e nothing found by grep for that search on the host and the engine-vm. -- Susinthiran Sithamparanathan
Hi guys, let me know if there anything else you need for further debugging purpose. Thanks! On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com> wrote:
OK. Can you please attach the output of:
grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf
[root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@ovirt01 ~]# ssh 192.168.0.101 root@192.168.0.101's password: Last login: Thu Oct 13 19:50:02 2016 from ovirt01 [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@engine ~]#
I.e nothing found by grep for that search on the host and the engine-vm.
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi guys, let me know if there anything else you need for further debugging purpose. Thanks!
Can you please try reinstalling all the oVirt rpms on the engine VM and re-executing engine-setup there?
On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com> wrote:
OK. Can you please attach the output of:
grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf
[root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@ovirt01 ~]# ssh 192.168.0.101 root@192.168.0.101's password: Last login: Thu Oct 13 19:50:02 2016 from ovirt01 [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@engine ~]#
I.e nothing found by grep for that search on the host and the engine-vm.
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
Hi, i tried that and ended up with : https://paste.fedoraproject.org/453892/71003314/ :( Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2Fengine-vm On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi guys, let me know if there anything else you need for further debugging purpose. Thanks!
Can you please try reinstalling all the oVirt rpms on the engine VM and re-executing engine-setup there?
On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com> wrote:
OK. Can you please attach the output of:
grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf
[root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@ovirt01 ~]# ssh 192.168.0.101 root@192.168.0.101's password: Last login: Thu Oct 13 19:50:02 2016 from ovirt01 [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@engine ~]#
I.e nothing found by grep for that search on the host and the engine-vm.
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
Analyzed the log to find out that the problem was in the creation of the certs with openssl (missing distinguished name in config). And /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! That lead me to: yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf yum remove ovirt-engine-backend yum install ovirt-engine-backend ovirt-engine ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi ovirt-engine-dashboard Now i was able to successfully run engine-setup and exit maintenance mode on the host. Let's see how things unfold within a 30 min. Will keep you updated! On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i tried that and ended up with : https://paste.fedoraproject. org/453892/71003314/ :( Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2Fengine-vm
On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi guys, let me know if there anything else you need for further debugging purpose. Thanks!
Can you please try reinstalling all the oVirt rpms on the engine VM and re-executing engine-setup there?
On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com> wrote:
OK. Can you please attach the output of:
grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf
[root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@ovirt01 ~]# ssh 192.168.0.101 root@192.168.0.101's password: Last login: Thu Oct 13 19:50:02 2016 from ovirt01 [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@engine ~]#
I.e nothing found by grep for that search on the host and the engine-vm.
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
Got the engine up finally :) But now met with The client is not authorized to request an authorization. It's required to access the system using FQDN. It worked fine prior to upgrade! Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not sure if i have FQDN case issues. Any idea how to fix this? On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Analyzed the log to find out that the problem was in the creation of the certs with openssl (missing distinguished name in config). And /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! That lead me to: yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf yum remove ovirt-engine-backend yum install ovirt-engine-backend ovirt-engine ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi ovirt-engine-dashboard
Now i was able to successfully run engine-setup and exit maintenance mode on the host. Let's see how things unfold within a 30 min. Will keep you updated!
On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i tried that and ended up with : https://paste.fedoraproject.or g/453892/71003314/ :( Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2Fengine-vm
On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi guys, let me know if there anything else you need for further debugging purpose. Thanks!
Can you please try reinstalling all the oVirt rpms on the engine VM and re-executing engine-setup there?
On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com> wrote:
OK. Can you please attach the output of:
grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf
[root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@ovirt01 ~]# ssh 192.168.0.101 root@192.168.0.101's password: Last login: Thu Oct 13 19:50:02 2016 from ovirt01 [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@engine ~]#
I.e nothing found by grep for that search on the host and the engine-vm.
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
On Mon, Oct 17, 2016 at 3:48 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Got the engine up finally :) But now met with The client is not authorized to request an authorization. It's required to access the system using FQDN. It worked fine prior to upgrade!
This is a new feature of 4.0; you cannot login anymore with the IP address if the cert has been signed for an fqdn.
Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not sure if i have FQDN case issues. Any idea how to fix this?
On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Analyzed the log to find out that the problem was in the creation of the certs with openssl (missing distinguished name in config). And /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! That lead me to: yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf yum remove ovirt-engine-backend yum install ovirt-engine-backend ovirt-engine ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi ovirt-engine-dashboard
Now i was able to successfully run engine-setup and exit maintenance mode on the host. Let's see how things unfold within a 30 min. Will keep you updated!
On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i tried that and ended up with : https://paste.fedoraproject.or g/453892/71003314/ :( Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2Fengine-vm
On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi <stirabos@redhat.com
wrote:
On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi guys, let me know if there anything else you need for further debugging purpose. Thanks!
Can you please try reinstalling all the oVirt rpms on the engine VM and re-executing engine-setup there?
On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com> wrote:
> > OK. Can you please attach the output of: > > grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >
[root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@ovirt01 ~]# ssh 192.168.0.101 root@192.168.0.101's password: Last login: Thu Oct 13 19:50:02 2016 from ovirt01 [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf [root@engine ~]#
I.e nothing found by grep for that search on the host and the engine-vm.
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
Thanks.Savior at https://www.mail-archive.com/users@ovirt.org/msg33874.html. When i logged into the web UI, i couldn't bring up storage, datacenter, cluster, everything was down. I restarted the host and now when i enter admin portal, it spins for ever. Seems to be some SSL communciation issues: https://paste.fedoraproject.org/453944/76715737/ Any hints are appreciated! On Mon, Oct 17, 2016 at 4:15 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 3:48 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Got the engine up finally :) But now met with The client is not authorized to request an authorization. It's required to access the system using FQDN. It worked fine prior to upgrade!
This is a new feature of 4.0; you cannot login anymore with the IP address if the cert has been signed for an fqdn.
Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not sure if i have FQDN case issues. Any idea how to fix this?
On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Analyzed the log to find out that the problem was in the creation of the certs with openssl (missing distinguished name in config). And /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! That lead me to: yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf yum remove ovirt-engine-backend yum install ovirt-engine-backend ovirt-engine ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi ovirt-engine-dashboard
Now i was able to successfully run engine-setup and exit maintenance mode on the host. Let's see how things unfold within a 30 min. Will keep you updated!
On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i tried that and ended up with : https://paste.fedoraproject.or g/453892/71003314/ :( Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2Fengine-vm
On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi < stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi guys, let me know if there anything else you need for further debugging purpose. Thanks!
Can you please try reinstalling all the oVirt rpms on the engine VM and re-executing engine-setup there?
On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
> On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David <didi@redhat.com > > wrote: > >> >> OK. Can you please attach the output of: >> >> grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >> > > [root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf. > d/*.conf > [root@ovirt01 ~]# ssh 192.168.0.101 > root@192.168.0.101's password: > Last login: Thu Oct 13 19:50:02 2016 from ovirt01 > [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf > [root@engine ~]# > > I.e nothing found by grep for that search on the host and the > engine-vm. > > > -- > > Susinthiran Sithamparanathan >
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
Now after a long time i got the prompt to login. What i see is that things are still down and unable to activate anything. I see [image: Inline image 1] This host is in non responding state. Try to Activate it; If the problem persists, switch Host to Maintenance mode and try to reinstall it. On Mon, Oct 17, 2016 at 4:51 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Thanks.Savior at https://www.mail-archive.com/u sers@ovirt.org/msg33874.html. When i logged into the web UI, i couldn't bring up storage, datacenter, cluster, everything was down. I restarted the host and now when i enter admin portal, it spins for ever. Seems to be some SSL communciation issues: https://paste.fedoraproject. org/453944/76715737/ Any hints are appreciated!
On Mon, Oct 17, 2016 at 4:15 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 3:48 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Got the engine up finally :) But now met with The client is not authorized to request an authorization. It's required to access the system using FQDN. It worked fine prior to upgrade!
This is a new feature of 4.0; you cannot login anymore with the IP address if the cert has been signed for an fqdn.
Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not sure if i have FQDN case issues. Any idea how to fix this?
On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Analyzed the log to find out that the problem was in the creation of the certs with openssl (missing distinguished name in config). And /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! That lead me to: yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf yum remove ovirt-engine-backend yum install ovirt-engine-backend ovirt-engine ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi ovirt-engine-dashboard
Now i was able to successfully run engine-setup and exit maintenance mode on the host. Let's see how things unfold within a 30 min. Will keep you updated!
On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i tried that and ended up with : https://paste.fedoraproject.or g/453892/71003314/ :( Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2Fengine-vm
On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi < stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
> Hi guys, > let me know if there anything else you need for further debugging > purpose. > Thanks! >
Can you please try reinstalling all the oVirt rpms on the engine VM and re-executing engine-setup there?
> > On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < > chesusin@gmail.com> wrote: > >> On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David < >> didi@redhat.com> wrote: >> >>> >>> OK. Can you please attach the output of: >>> >>> grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >>> >> >> [root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >> d/*.conf >> [root@ovirt01 ~]# ssh 192.168.0.101 >> root@192.168.0.101's password: >> Last login: Thu Oct 13 19:50:02 2016 from ovirt01 >> [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >> [root@engine ~]# >> >> I.e nothing found by grep for that search on the host and the >> engine-vm. >> >> >> -- >> >> Susinthiran Sithamparanathan >> > > > > -- > > Susinthiran Sithamparanathan >
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
Hi, still unable to get my system up with my VMs. Inside the web UI i can see a warning at the bottom: Host hosted_engine_1 is non responsive. When i try o activate master data domain NFS01, i get :Error while executing action: Cannot activate Storage. There is no active Host in the Data Center. The Data Center tab shows "VDSM hosted_engine_1 command failed: Message timeout which can be caused by communication issues" at the bottom. I can't see why the host isn't active in the engine-vm. Any help appreciated.Thanks. On Mon, Oct 17, 2016 at 5:00 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Now after a long time i got the prompt to login. What i see is that things are still down and unable to activate anything. I see [image: Inline image 1] This host is in non responding state. Try to Activate it; If the problem persists, switch Host to Maintenance mode and try to reinstall it.
On Mon, Oct 17, 2016 at 4:51 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Thanks.Savior at https://www.mail-archive.com/u sers@ovirt.org/msg33874.html. When i logged into the web UI, i couldn't bring up storage, datacenter, cluster, everything was down. I restarted the host and now when i enter admin portal, it spins for ever. Seems to be some SSL communciation issues: https://paste.fedoraproject.org/453944/76715737/ Any hints are appreciated!
On Mon, Oct 17, 2016 at 4:15 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 3:48 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Got the engine up finally :) But now met with The client is not authorized to request an authorization. It's required to access the system using FQDN. It worked fine prior to upgrade!
This is a new feature of 4.0; you cannot login anymore with the IP address if the cert has been signed for an fqdn.
Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not sure if i have FQDN case issues. Any idea how to fix this?
On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Analyzed the log to find out that the problem was in the creation of the certs with openssl (missing distinguished name in config). And /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! That lead me to: yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf yum remove ovirt-engine-backend yum install ovirt-engine-backend ovirt-engine ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi ovirt-engine-dashboard
Now i was able to successfully run engine-setup and exit maintenance mode on the host. Let's see how things unfold within a 30 min. Will keep you updated!
On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, i tried that and ended up with : https://paste.fedoraproject.or g/453892/71003314/ :( Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2Fengine-vm
On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi < stirabos@redhat.com> wrote:
> > > On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < > chesusin@gmail.com> wrote: > >> Hi guys, >> let me know if there anything else you need for further debugging >> purpose. >> Thanks! >> > > Can you please try reinstalling all the oVirt rpms on the engine VM > and re-executing engine-setup there? > > >> >> On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < >> chesusin@gmail.com> wrote: >> >>> On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David < >>> didi@redhat.com> wrote: >>> >>>> >>>> OK. Can you please attach the output of: >>>> >>>> grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >>>> >>> >>> [root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>> d/*.conf >>> [root@ovirt01 ~]# ssh 192.168.0.101 >>> root@192.168.0.101's password: >>> Last login: Thu Oct 13 19:50:02 2016 from ovirt01 >>> [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>> d/*.conf >>> [root@engine ~]# >>> >>> I.e nothing found by grep for that search on the host and the >>> engine-vm. >>> >>> >>> -- >>> >>> Susinthiran Sithamparanathan >>> >> >> >> >> -- >> >> Susinthiran Sithamparanathan >> > >
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
On Thu, Oct 20, 2016 at 6:16 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, still unable to get my system up with my VMs. Inside the web UI i can see a warning at the bottom: Host hosted_engine_1 is non responsive. When i try o activate master data domain NFS01, i get :Error while executing action: Cannot activate Storage. There is no active Host in the Data Center. The Data Center tab shows "VDSM hosted_engine_1 command failed: Message timeout which can be caused by communication issues" at the bottom. I can't see why the host isn't active in the engine-vm. Any help appreciated.Thanks.
Can you please try cat < /dev/tcp/<yourhostaddress>/54321 from the engine VM? If it's not able to connect, please check name resolution, addressing and so on.
On Mon, Oct 17, 2016 at 5:00 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Now after a long time i got the prompt to login. What i see is that things are still down and unable to activate anything. I see [image: Inline image 1] This host is in non responding state. Try to Activate it; If the problem persists, switch Host to Maintenance mode and try to reinstall it.
On Mon, Oct 17, 2016 at 4:51 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Thanks.Savior at https://www.mail-archive.com/u sers@ovirt.org/msg33874.html. When i logged into the web UI, i couldn't bring up storage, datacenter, cluster, everything was down. I restarted the host and now when i enter admin portal, it spins for ever. Seems to be some SSL communciation issues: https://paste.fedoraproject.org/453944/76715737/ Any hints are appreciated!
On Mon, Oct 17, 2016 at 4:15 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 3:48 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Got the engine up finally :) But now met with The client is not authorized to request an authorization. It's required to access the system using FQDN. It worked fine prior to upgrade!
This is a new feature of 4.0; you cannot login anymore with the IP address if the cert has been signed for an fqdn.
Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not sure if i have FQDN case issues. Any idea how to fix this?
On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Analyzed the log to find out that the problem was in the creation of the certs with openssl (missing distinguished name in config). And /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! That lead me to: yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf yum remove ovirt-engine-backend yum install ovirt-engine-backend ovirt-engine ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi ovirt-engine-dashboard
Now i was able to successfully run engine-setup and exit maintenance mode on the host. Let's see how things unfold within a 30 min. Will keep you updated!
On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
> Hi, > i tried that and ended up with : https://paste.fedoraproject.or > g/453892/71003314/ :( > Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to > https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2F > engine-vm > > > > On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi < > stirabos@redhat.com> wrote: > >> >> >> On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < >> chesusin@gmail.com> wrote: >> >>> Hi guys, >>> let me know if there anything else you need for further debugging >>> purpose. >>> Thanks! >>> >> >> Can you please try reinstalling all the oVirt rpms on the engine VM >> and re-executing engine-setup there? >> >> >>> >>> On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < >>> chesusin@gmail.com> wrote: >>> >>>> On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David < >>>> didi@redhat.com> wrote: >>>> >>>>> >>>>> OK. Can you please attach the output of: >>>>> >>>>> grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >>>>> >>>> >>>> [root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>>> d/*.conf >>>> [root@ovirt01 ~]# ssh 192.168.0.101 >>>> root@192.168.0.101's password: >>>> Last login: Thu Oct 13 19:50:02 2016 from ovirt01 >>>> [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>>> d/*.conf >>>> [root@engine ~]# >>>> >>>> I.e nothing found by grep for that search on the host and the >>>> engine-vm. >>>> >>>> >>>> -- >>>> >>>> Susinthiran Sithamparanathan >>>> >>> >>> >>> >>> -- >>> >>> Susinthiran Sithamparanathan >>> >> >> > > > -- > > Susinthiran Sithamparanathan >
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
Hi, i did run that command from the engine (now the hostname is changed to susin.myftp.org -> 192.168.0.101 ) and got: [root@susin ~]# cat < /dev/tcp/susin/54321 -bash: connect: Connection refused -bash: /dev/tcp/susin/54321: Connection refused [root@susin ~]# cat < /dev/tcp/susin.myftp.org/54321 -bash: connect: Connection refused -bash: /dev/tcp/susin.myftp.org/54321: Connection refused [root@susin ~]# cat < /dev/tcp/192.168.0.101/54321 -bash: connect: Connection refused -bash: /dev/tcp/192.168.0.101/54321: Connection refused [root@susin ~]# Both host and engine is behing a NAT and I've configured /etc/hosts correctly so the hosts ping by name from the engine and host. The hostname of the engine is susin.myftp.org so using dig or host, it will resolve to my public IP, and pinging will resolve correctly. But now i came over http://www.ovirt.org/documentation/how-to/networking/changing-engine-hostnam... since i actually changed the hostname for the engine to be able to login through the web UI. Especially the following "The bigger concern is with the engine's certificate. Currently, to the best of our knowledge, there is no component that actually checks this trust. But it's possible, that in some future version of one of the relevant tools - vdsm, libvirt, etc. - such a check will actually be made, and even prevent connections. If this happens, the engine might not be able to connect to the hosts, and the worst case is that they will have to be reinstalled, thus loosing all the configuration and data accumulated by then." tail -f /var/log/ovirt-engine/engine.log 2016-10-21 11:05:16,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [] Command 'GetAllVmStatsVDSCommand(HostName = hosted_engine_1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='826a8da5-74c1-4002-ab7b-e6e32be94fe6', vds='Host[hosted_engine_1,826a8da5-74c1-4002-ab7b-e6e32be94fe6]'})' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured 2016-10-21 11:05:16,888 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler1) [] Failed to fetch vms info for host 'hosted_engine_1' - skipping VMs monitoring. 2016-10-21 11:05:16,918 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM hosted_engine_1 command failed: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1f2e4065' 2016-10-21 11:05:16,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] HostName = hosted_engine_1 2016-10-21 11:05:16,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(HostName = hosted_engine_1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='826a8da5-74c1-4002-ab7b-e6e32be94fe6', vds='Host[hosted_engine_1,826a8da5-74c1-4002-ab7b-e6e32be94fe6]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,919 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh Vds runtime info: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,919 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand.executeVdsBrokerCommand(GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:451) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:653) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:121) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:85) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238) [vdsbroker.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_102] at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102] 2016-10-21 11:05:16,921 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler4) [] Failed to refresh VDS, network error, continuing, vds='hosted_engine_1'(826a8da5-74c1-4002-ab7b-e6e32be94fe6): VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,921 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-8-thread-1) [] Host 'hosted_engine_1' is not responding. 2016-10-21 11:05:16,993 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_1 is not responding. Host cannot be fenced automatically because power management for the host is disabled. 2016-10-21 11:05:19,943 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt01/192.168.0.100 2016-10-21 11:05:19,960 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: General SSLEngine problem .. 2016-10-21 11:08:40,871 WARN [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-6) [6d681722] Validation of action 'VdsNotRespondingTreatment' failed for user SYSTEM. Reasons: VAR__ACTION__RESTART,POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS 2016-10-21 11:08:41,023 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 826a8da5-74c1-4002-ab7b-e6e32be94fe6 Type: VDS 2016-10-21 11:08:41,088 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 826a8da5-74c1-4002-ab7b-e6e32be94fe6 Type: VDS 2016-10-21 11:08:41,116 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Opening SSH Soft Fencing session on host 'ovirt01' 2016-10-21 11:08:41,470 ERROR [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] SSH Soft Fencing command failed on host 'ovirt01': SSH authentication to 'root@ovirt01' failed. Please verify provided credentials. Make sure key is authorized at host Stdout: Stderr: 2016-10-21 11:08:41,483 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Lock freed to object 'EngineLock:{exclusiveLocks='[826a8da5-74c1-4002-ab7b-e6e32be94fe6=<VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', sharedLocks='null'}' 2016-10-21 11:08:41,545 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Trying to release exclusive lock which does not exist, lock key: '826a8da5-74c1-4002-ab7b-e6e32be94fe6VDS_FENCE' 2016-10-21 11:08:41,547 INFO [org.ovirt.engine.core.bll.pm.VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Lock freed to object 'EngineLock:{exclusiveLocks='[826a8da5-74c1-4002-ab7b-e6e32be94fe6=<VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', sharedLocks='null'}' 2016-10-21 11:08:43,503 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt01/192.168.0.100 2016-10-21 11:08:43,517 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: General SSLEngine problem 2016-10-21 11:08:55,446 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt01/192.168.0.100 2016-10-21 11:08:55,461 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: General SSLEngine problem Since the engine is complaining about SSL communication error, i suspect the problem is there. Is there still any ways to save my VMs or do i have to reinstall it? On Thu, Oct 20, 2016 at 6:27 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Thu, Oct 20, 2016 at 6:16 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, still unable to get my system up with my VMs. Inside the web UI i can see a warning at the bottom: Host hosted_engine_1 is non responsive. When i try o activate master data domain NFS01, i get :Error while executing action: Cannot activate Storage. There is no active Host in the Data Center. The Data Center tab shows "VDSM hosted_engine_1 command failed: Message timeout which can be caused by communication issues" at the bottom. I can't see why the host isn't active in the engine-vm. Any help appreciated.Thanks.
Can you please try cat < /dev/tcp/<yourhostaddress>/54321 from the engine VM? If it's not able to connect, please check name resolution, addressing and so on.
On Mon, Oct 17, 2016 at 5:00 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Now after a long time i got the prompt to login. What i see is that things are still down and unable to activate anything. I see [image: Inline image 1] This host is in non responding state. Try to Activate it; If the problem persists, switch Host to Maintenance mode and try to reinstall it.
On Mon, Oct 17, 2016 at 4:51 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Thanks.Savior at https://www.mail-archive.com/u sers@ovirt.org/msg33874.html. When i logged into the web UI, i couldn't bring up storage, datacenter, cluster, everything was down. I restarted the host and now when i enter admin portal, it spins for ever. Seems to be some SSL communciation issues: https://paste.fedoraproject.org/453944/76715737/ Any hints are appreciated!
On Mon, Oct 17, 2016 at 4:15 PM, Simone Tiraboschi <stirabos@redhat.com
wrote:
On Mon, Oct 17, 2016 at 3:48 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Got the engine up finally :) But now met with The client is not authorized to request an authorization. It's required to access the system using FQDN. It worked fine prior to upgrade!
This is a new feature of 4.0; you cannot login anymore with the IP address if the cert has been signed for an fqdn.
Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not sure if i have FQDN case issues. Any idea how to fix this?
On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
> Analyzed the log to find out that the problem was in the creation of > the certs with openssl (missing distinguished name in config). And > /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! > That lead me to: > yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf > yum remove ovirt-engine-backend > yum install ovirt-engine-backend ovirt-engine > ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools > ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi > ovirt-engine-dashboard > > Now i was able to successfully run engine-setup and exit maintenance > mode on the host. Let's see how things unfold within a 30 min. Will keep > you updated! > > > > > > > > > > On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < > chesusin@gmail.com> wrote: > >> Hi, >> i tried that and ended up with : https://paste.fedoraproject.or >> g/453892/71003314/ :( >> Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to >> https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2F >> engine-vm >> >> >> >> On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi < >> stirabos@redhat.com> wrote: >> >>> >>> >>> On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < >>> chesusin@gmail.com> wrote: >>> >>>> Hi guys, >>>> let me know if there anything else you need for further debugging >>>> purpose. >>>> Thanks! >>>> >>> >>> Can you please try reinstalling all the oVirt rpms on the engine >>> VM and re-executing engine-setup there? >>> >>> >>>> >>>> On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < >>>> chesusin@gmail.com> wrote: >>>> >>>>> On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David < >>>>> didi@redhat.com> wrote: >>>>> >>>>>> >>>>>> OK. Can you please attach the output of: >>>>>> >>>>>> grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >>>>>> >>>>> >>>>> [root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>>>> d/*.conf >>>>> [root@ovirt01 ~]# ssh 192.168.0.101 >>>>> root@192.168.0.101's password: >>>>> Last login: Thu Oct 13 19:50:02 2016 from ovirt01 >>>>> [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>>>> d/*.conf >>>>> [root@engine ~]# >>>>> >>>>> I.e nothing found by grep for that search on the host and the >>>>> engine-vm. >>>>> >>>>> >>>>> -- >>>>> >>>>> Susinthiran Sithamparanathan >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Susinthiran Sithamparanathan >>>> >>> >>> >> >> >> -- >> >> Susinthiran Sithamparanathan >> > > > > -- > > Susinthiran Sithamparanathan >
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
-- Susinthiran Sithamparanathan
On Fri, Oct 21, 2016 at 11:10 AM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi,
i did run that command from the engine (now the hostname is changed to susin.myftp.org -> 192.168.0.101 ) and got: [root@susin ~]# cat < /dev/tcp/susin/54321 -bash: connect: Connection refused -bash: /dev/tcp/susin/54321: Connection refused [root@susin ~]# cat < /dev/tcp/susin.myftp.org/54321 -bash: connect: Connection refused -bash: /dev/tcp/susin.myftp.org/54321: Connection refused [root@susin ~]# cat < /dev/tcp/192.168.0.101/54321 -bash: connect: Connection refused -bash: /dev/tcp/192.168.0.101/54321: Connection refused [root@susin ~]#
The engine should be able to reach vdsm: please fix this before touching anything else.
Both host and engine is behing a NAT and I've configured /etc/hosts correctly so the hosts ping by name from the engine and host. The hostname of the engine is susin.myftp.org so using dig or host, it will resolve to my public IP, and pinging will resolve correctly.
But now i came over http://www.ovirt.org/documentation/how-to/ networking/changing-engine-hostname/ since i actually changed the hostname for the engine to be able to login through the web UI. Especially the following "The bigger concern is with the engine's certificate. Currently, to the best of our knowledge, there is no component that actually checks this trust. But it's possible, that in some future version of one of the relevant tools - vdsm, libvirt, etc. - such a check will actually be made, and even prevent connections. If this happens, the engine might not be able to connect to the hosts, and the worst case is that they will have to be reinstalled, thus loosing all the configuration and data accumulated by then."
tail -f /var/log/ovirt-engine/engine.log 2016-10-21 11:05:16,888 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler1) [] Command 'GetAllVmStatsVDSCommand(HostName = hosted_engine_1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='826a8da5-74c1-4002-ab7b-e6e32be94fe6', vds='Host[hosted_engine_1, 826a8da5-74c1-4002-ab7b-e6e32be94fe6]'})' execution failed: VDSGenericException: VDSNetworkException: Vds timeout occured 2016-10-21 11:05:16,888 INFO [org.ovirt.engine.core.vdsbroker.monitoring.PollVmStatsRefresher] (DefaultQuartzScheduler1) [] Failed to fetch vms info for host 'hosted_engine_1' - skipping VMs monitoring. 2016-10-21 11:05:16,918 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler4) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM hosted_engine_1 command failed: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'org.ovirt.engine.core. vdsbroker.vdsbroker.GetCapabilitiesVDSCommand' return value 'org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@ 1f2e4065' 2016-10-21 11:05:16,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] HostName = hosted_engine_1 2016-10-21 11:05:16,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler4) [] Command 'GetCapabilitiesVDSCommand(HostName = hosted_engine_1, VdsIdAndVdsVDSCommandParametersBase:{runAsync='true', hostId='826a8da5-74c1-4002-ab7b-e6e32be94fe6', vds='Host[hosted_engine_1, 826a8da5-74c1-4002-ab7b-e6e32be94fe6]'})' execution failed: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,919 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Failure to refresh Vds runtime info: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,919 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (DefaultQuartzScheduler4) [] Exception: org.ovirt.engine.core. vdsbroker.vdsbroker.VDSNetworkException: VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase. proceedProxyReturnValue(BrokerCommandBase.java:188) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker. GetCapabilitiesVDSCommand.executeVdsBrokerCommand( GetCapabilitiesVDSCommand.java:16) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand. executeVDSCommand(VdsBrokerCommand.java:110) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase. executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.vdsbroker.ResourceManager. runVdsCommand(ResourceManager.java:451) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:653) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring. refreshVdsRunTimeInfo(HostMonitoring.java:121) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:85) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:238) [vdsbroker.jar:] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_102] at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_102] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_102] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_102] at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_102] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102]
2016-10-21 11:05:16,921 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler4) [] Failed to refresh VDS, network error, continuing, vds='hosted_engine_1'(826a8da5-74c1-4002-ab7b-e6e32be94fe6): VDSGenericException: VDSNetworkException: Message timeout which can be caused by communication issues 2016-10-21 11:05:16,921 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (org.ovirt.thread.pool-8-thread-1) [] Host 'hosted_engine_1' is not responding. 2016-10-21 11:05:16,993 WARN [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-1) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host hosted_engine_1 is not responding. Host cannot be fenced automatically because power management for the host is disabled. 2016-10-21 11:05:19,943 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt01/192.168.0.100 2016-10-21 11:05:19,960 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: General SSLEngine problem .. 2016-10-21 11:08:40,871 WARN [org.ovirt.engine.core.bll.pm. VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-6) [6d681722] Validation of action 'VdsNotRespondingTreatment' failed for user SYSTEM. Reasons: VAR__ACTION__RESTART,POWER_MANAGEMENT_ACTION_ON_ENTITY_ ALREADY_IN_PROGRESS 2016-10-21 11:08:41,023 INFO [org.ovirt.engine.core.bll.pm. VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Running command: VdsNotRespondingTreatmentCommand internal: true. Entities affected : ID: 826a8da5-74c1-4002-ab7b-e6e32be94fe6 Type: VDS 2016-10-21 11:08:41,088 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 826a8da5-74c1-4002-ab7b-e6e32be94fe6 Type: VDS 2016-10-21 11:08:41,116 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Opening SSH Soft Fencing session on host 'ovirt01' 2016-10-21 11:08:41,470 ERROR [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] SSH Soft Fencing command failed on host 'ovirt01': SSH authentication to 'root@ovirt01' failed. Please verify provided credentials. Make sure key is authorized at host Stdout: Stderr: 2016-10-21 11:08:41,483 INFO [org.ovirt.engine.core.bll.pm.SshSoftFencingCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Lock freed to object 'EngineLock:{exclusiveLocks='[826a8da5-74c1-4002-ab7b-e6e32be94fe6=<VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', sharedLocks='null'}' 2016-10-21 11:08:41,545 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Trying to release exclusive lock which does not exist, lock key: '826a8da5-74c1-4002-ab7b- e6e32be94fe6VDS_FENCE' 2016-10-21 11:08:41,547 INFO [org.ovirt.engine.core.bll.pm. VdsNotRespondingTreatmentCommand] (org.ovirt.thread.pool-8-thread-5) [193e2e22] Lock freed to object 'EngineLock:{exclusiveLocks='[ 826a8da5-74c1-4002-ab7b-e6e32be94fe6=<VDS_FENCE, POWER_MANAGEMENT_ACTION_ON_ENTITY_ALREADY_IN_PROGRESS>]', sharedLocks='null'}' 2016-10-21 11:08:43,503 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt01/192.168.0.100 2016-10-21 11:08:43,517 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: General SSLEngine problem 2016-10-21 11:08:55,446 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [] Connecting to ovirt01/192.168.0.100 2016-10-21 11:08:55,461 ERROR [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [] Unable to process messages: General SSLEngine problem
Since the engine is complaining about SSL communication error, i suspect the problem is there.
Is there still any ways to save my VMs or do i have to reinstall it?
On Thu, Oct 20, 2016 at 6:27 PM, Simone Tiraboschi <stirabos@redhat.com> wrote:
On Thu, Oct 20, 2016 at 6:16 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Hi, still unable to get my system up with my VMs. Inside the web UI i can see a warning at the bottom: Host hosted_engine_1 is non responsive. When i try o activate master data domain NFS01, i get :Error while executing action: Cannot activate Storage. There is no active Host in the Data Center. The Data Center tab shows "VDSM hosted_engine_1 command failed: Message timeout which can be caused by communication issues" at the bottom. I can't see why the host isn't active in the engine-vm. Any help appreciated.Thanks.
Can you please try cat < /dev/tcp/<yourhostaddress>/54321 from the engine VM? If it's not able to connect, please check name resolution, addressing and so on.
On Mon, Oct 17, 2016 at 5:00 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Now after a long time i got the prompt to login. What i see is that things are still down and unable to activate anything. I see [image: Inline image 1] This host is in non responding state. Try to Activate it; If the problem persists, switch Host to Maintenance mode and try to reinstall it.
On Mon, Oct 17, 2016 at 4:51 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
Thanks.Savior at https://www.mail-archive.com/u sers@ovirt.org/msg33874.html. When i logged into the web UI, i couldn't bring up storage, datacenter, cluster, everything was down. I restarted the host and now when i enter admin portal, it spins for ever. Seems to be some SSL communciation issues: https://paste.fedoraproject.org/453944/76715737/ Any hints are appreciated!
On Mon, Oct 17, 2016 at 4:15 PM, Simone Tiraboschi < stirabos@redhat.com> wrote:
On Mon, Oct 17, 2016 at 3:48 PM, Susinthiran Sithamparanathan < chesusin@gmail.com> wrote:
> Got the engine up finally :) > But now met with The client is not authorized to request an > authorization. It's required to access the system using FQDN. It worked > fine prior to upgrade! >
This is a new feature of 4.0; you cannot login anymore with the IP address if the cert has been signed for an fqdn.
> Found https://bugzilla.redhat.com/show_bug.cgi?id=1351217, but not > sure if i have FQDN case issues. > Any idea how to fix this? > > On Mon, Oct 17, 2016 at 3:33 PM, Susinthiran Sithamparanathan < > chesusin@gmail.com> wrote: > >> Analyzed the log to find out that the problem was in the creation >> of the certs with openssl (missing distinguished name in config). And >> /etc/pki/ovirt-engine/{cacert,openssl}.conf were empty! >> That lead me to: >> yum provides /etc/pki/ovirt-engine/{cacert,openssl}.conf >> yum remove ovirt-engine-backend >> yum install ovirt-engine-backend ovirt-engine >> ovirt-engine-dashboard ovirt-engine-setup ovirt-engine-tools >> ovirt-engine-userportal ovirt-engine-webadmin-portal ovirt-engine-restapi >> ovirt-engine-dashboard >> >> Now i was able to successfully run engine-setup and exit >> maintenance mode on the host. Let's see how things unfold within a 30 min. >> Will keep you updated! >> >> >> >> >> >> >> >> >> >> On Mon, Oct 17, 2016 at 3:14 PM, Susinthiran Sithamparanathan < >> chesusin@gmail.com> wrote: >> >>> Hi, >>> i tried that and ended up with : https://paste.fedoraproject.or >>> g/453892/71003314/ :( >>> Log ovirt-engine-setup-20161017150800-drmayj.log uploaded to >>> https://my.owndrive.com/index.php/s/3Dcyho9bqo7oZs8?path=%2F >>> engine-vm >>> >>> >>> >>> On Mon, Oct 17, 2016 at 11:54 AM, Simone Tiraboschi < >>> stirabos@redhat.com> wrote: >>> >>>> >>>> >>>> On Mon, Oct 17, 2016 at 9:55 AM, Susinthiran Sithamparanathan < >>>> chesusin@gmail.com> wrote: >>>> >>>>> Hi guys, >>>>> let me know if there anything else you need for further >>>>> debugging purpose. >>>>> Thanks! >>>>> >>>> >>>> Can you please try reinstalling all the oVirt rpms on the engine >>>> VM and re-executing engine-setup there? >>>> >>>> >>>>> >>>>> On Thu, Oct 13, 2016 at 7:53 PM, Susinthiran Sithamparanathan < >>>>> chesusin@gmail.com> wrote: >>>>> >>>>>> On Thu, Oct 13, 2016 at 3:23 PM, Yedidyah Bar David < >>>>>> didi@redhat.com> wrote: >>>>>> >>>>>>> >>>>>>> OK. Can you please attach the output of: >>>>>>> >>>>>>> grep MANUAL /etc/ovirt-engine/engine.conf.d/*.conf >>>>>>> >>>>>> >>>>>> [root@ovirt01 ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>>>>> d/*.conf >>>>>> [root@ovirt01 ~]# ssh 192.168.0.101 >>>>>> root@192.168.0.101's password: >>>>>> Last login: Thu Oct 13 19:50:02 2016 from ovirt01 >>>>>> [root@engine ~]# grep MANUAL /etc/ovirt-engine/engine.conf. >>>>>> d/*.conf >>>>>> [root@engine ~]# >>>>>> >>>>>> I.e nothing found by grep for that search on the host and the >>>>>> engine-vm. >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Susinthiran Sithamparanathan >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Susinthiran Sithamparanathan >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> Susinthiran Sithamparanathan >>> >> >> >> >> -- >> >> Susinthiran Sithamparanathan >> > > > > -- > > Susinthiran Sithamparanathan >
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
--
Susinthiran Sithamparanathan
participants (4)
-
Doron Fediuck -
Simone Tiraboschi -
Susinthiran Sithamparanathan -
Yedidyah Bar David