Hi Everyone, I'm having an issue with one of my hosts. I was performing a yum update when the box crashed and rebooted, which left a lot of packages broken. I resolved the issue by running "yum reinstall" on all the packages that were in the update, but since then VDSM won't start. I can't find any error message as to why it won't start, just that it won't. IE, I see this in /var/log/messages: Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: WARNING:ovirt_hosted_engine_ha.agent.agent.Agent:Restarting agent, attempt '5' Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Found certificate common name: ovirt-01.local.domain Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing VDSM Apr 13 07:57:09 ovirt-01 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Service vdsmd is not running and the admin is responsible for starting it. Waiting... Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Service vdsmd is not running and the admin is responsible for starting it. Waiting... And this in /var/log/vdsmd/mom.log (vdsmd.log is a 0 byte file) 2016-04-13 07:58:07,958 - mom - INFO - MOM starting 2016-04-13 07:58:07,992 - mom.HostMonitor - INFO - Host Monitor starting 2016-04-13 07:58:07,992 - mom - INFO - hypervisor interface vdsmxmlrpc 2016-04-13 07:58:08,089 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2016-04-13 07:58:08,090 - mom.GuestManager - INFO - Guest Manager starting: multi-thread 2016-04-13 07:58:08,105 - mom.Policy - INFO - Loaded policy '00-defines' 2016-04-13 07:58:08,109 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2016-04-13 07:58:08,169 - mom.Policy - INFO - Loaded policy '02-balloon' 2016-04-13 07:58:08,263 - mom.Policy - INFO - Loaded policy '03-ksm' 2016-04-13 07:58:08,339 - mom.HostMonitor - INFO - HostMonitor is ready 2016-04-13 07:58:08,380 - mom.Policy - INFO - Loaded policy '04-cputune' 2016-04-13 07:58:08,381 - mom.PolicyEngine - INFO - Policy Engine starting 2016-04-13 07:58:08,383 - mom.RPCServer - INFO - Using unix socket /var/run/vdsm/mom-vdsm.sock 2016-04-13 07:58:08,383 - mom.RPCServer - INFO - RPC Server starting 2016-04-13 07:58:08,884 - mom.RPCServer - INFO - RPC Server ending Can anyone give me some ideas of how to debug this? Or even some docs on how to properly remove the host from the manager app and re-register the guest to see if that fixes things up.
On Wed, Apr 13, 2016 at 1:06 PM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Everyone,
I'm having an issue with one of my hosts. I was performing a yum update when the box crashed and rebooted, which left a lot of packages broken. I resolved the issue by running "yum reinstall" on all the packages that were in the update, but since then VDSM won't start. I can't find any error message as to why it won't start, just that it won't. IE, I see this in /var/log/messages:
Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: WARNING:ovirt_hosted_engine_ha.agent.agent.Agent:Restarting agent, attempt '5' Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Found certificate common name: ovirt-01.local.domain Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing VDSM Apr 13 07:57:09 ovirt-01 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Service vdsmd is not running and the admin is responsible for starting it. Waiting... Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Service vdsmd is not running and the admin is responsible for starting it. Waiting...
And this in /var/log/vdsmd/mom.log (vdsmd.log is a 0 byte file)
2016-04-13 07:58:07,958 - mom - INFO - MOM starting 2016-04-13 07:58:07,992 - mom.HostMonitor - INFO - Host Monitor starting 2016-04-13 07:58:07,992 - mom - INFO - hypervisor interface vdsmxmlrpc 2016-04-13 07:58:08,089 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2016-04-13 07:58:08,090 - mom.GuestManager - INFO - Guest Manager starting: multi-thread 2016-04-13 07:58:08,105 - mom.Policy - INFO - Loaded policy '00-defines' 2016-04-13 07:58:08,109 - mom.vdsmInterface - ERROR - Cannot connect to VDSM! [Errno 111] Connection refused 2016-04-13 07:58:08,169 - mom.Policy - INFO - Loaded policy '02-balloon' 2016-04-13 07:58:08,263 - mom.Policy - INFO - Loaded policy '03-ksm' 2016-04-13 07:58:08,339 - mom.HostMonitor - INFO - HostMonitor is ready 2016-04-13 07:58:08,380 - mom.Policy - INFO - Loaded policy '04-cputune' 2016-04-13 07:58:08,381 - mom.PolicyEngine - INFO - Policy Engine starting 2016-04-13 07:58:08,383 - mom.RPCServer - INFO - Using unix socket /var/run/vdsm/mom-vdsm.sock 2016-04-13 07:58:08,383 - mom.RPCServer - INFO - RPC Server starting 2016-04-13 07:58:08,884 - mom.RPCServer - INFO - RPC Server ending
Can anyone give me some ideas of how to debug this? Or even some docs on how to properly remove the host from the manager app and re-register the guest to see if that fixes things up.
You can run hosted-engine --deploy again and choose to redeploy using the same host-id: it will reconfigure VDSM for you.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Simon, I tried that, but the same problem: vdsm won't start, so the deploy can't continue. There is some sort of bug with running VDSMd, but I can't find any error logging to say what that bug is so I'm at a loss as to how to fix it. On 16-04-13 08:47 AM, Simone Tiraboschi wrote:
[snip] You can run hosted-engine --deploy again and choose to redeploy using the same host-id: it will reconfigure VDSM for you.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Charles, can you please make sure you are not in local maintenance mode and the vdsmd systemd service is enabled? The line
Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Service vdsmd is not running and the admin is responsible for starting it. Waiting...
seems to indicate that hosted engine agent refused to start vdsm on its own and that usually happens when the host is in maintenance. Regards -- Martin Sivak SLA / oVirt On Wed, Apr 13, 2016 at 2:25 PM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Simon,
I tried that, but the same problem: vdsm won't start, so the deploy can't continue. There is some sort of bug with running VDSMd, but I can't find any error logging to say what that bug is so I'm at a loss as to how to fix it.
On 16-04-13 08:47 AM, Simone Tiraboschi wrote:
[snip] You can run hosted-engine --deploy again and choose to redeploy using the same host-id: it will reconfigure VDSM for you.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Guys, Thanks, I found the problem. I had edited the /etc/vdsm logger config files to cut down on all the debug logging and the process failed (silently in any system log I could find) due to a parsing error. Restored the logger files to defaults and it worked. For anyone going through the list logs in the future, try running: sudo -s /bin/bash vdsm python /usr/share/vdsm/vdsm to get an error message. On 16-04-13 09:46 AM, Martin Sivak wrote:
Hi Charles,
can you please make sure you are not in local maintenance mode and the vdsmd systemd service is enabled?
The line
Apr 13 07:57:09 ovirt-01 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Service vdsmd is not running and the admin is responsible for starting it. Waiting... seems to indicate that hosted engine agent refused to start vdsm on its own and that usually happens when the host is in maintenance.
Regards
-- Martin Sivak SLA / oVirt
On Wed, Apr 13, 2016 at 2:25 PM, Charles Tassell <ctassell@gmail.com> wrote:
Hi Simon,
I tried that, but the same problem: vdsm won't start, so the deploy can't continue. There is some sort of bug with running VDSMd, but I can't find any error logging to say what that bug is so I'm at a loss as to how to fix it.
On 16-04-13 08:47 AM, Simone Tiraboschi wrote:
[snip] You can run hosted-engine --deploy again and choose to redeploy using the same host-id: it will reconfigure VDSM for you.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (3)
-
Charles Tassell -
Martin Sivak -
Simone Tiraboschi