<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">[boxname ~]# systemctl status -l vdsm-network<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">● vdsm-network.service - Virtual Desktop Server Manager network restoration<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> Loaded: loaded (/usr/lib/systemd/system/vdsm-network.service; enabled; vendor preset: enabled)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> Active: activating (start) since Tue 2017-07-18 10:42:57 CDT; 1h 29min ago<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> Process: 8216 ExecStartPre=/usr/bin/vdsm-tool --vvverbose --append --logfile=/var/log/vdsm/upgrade.log upgrade-unified-persistence (code=exited, status=0/SUCCESS)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Main PID: 8231 (vdsm-tool)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> CGroup: /system.slice/vdsm-network.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="font-family:"Arial",sans-serif;color:#1F497D">├</span><span style="color:#1F497D">─</span><span style="color:#1F497D">8231 /usr/bin/python /usr/bin/vdsm-tool restore-nets<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> └─8240 /usr/bin/python /usr/share/vdsm/vdsm-restore-net-config<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Jul 18 10:42:57 t0894bmh1001.stores.target.com systemd[1]: Starting Virtual Desktop Server Manager network restoration...<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Tony<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Pavel Gashev [mailto:Pax@acronis.com] <br>
<b>Sent:</b> Tuesday, July 18, 2017 11:17 AM<br>
<b>To:</b> Anthony.Fillmore <Anthony.Fillmore@target.com>; users@ovirt.org<br>
<b>Cc:</b> Brandon.Markgraf <Brandon.Markgraf@target.com>; Sandeep.Mendiratta <Sandeep.Mendiratta@target.com><br>
<b>Subject:</b> [EXTERNAL] Re: [ovirt-users] Host stuck unresponsive after Network Outage<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-GB">Anthony,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Output of “systemctl status -l vdsm-network” would help.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-GB" style="color:black">From: </span></b><span lang="EN-GB" style="color:black"><<a href="mailto:users-bounces@ovirt.org">users-bounces@ovirt.org</a>> on behalf of "Anthony.Fillmore" <<a href="mailto:Anthony.Fillmore@target.com">Anthony.Fillmore@target.com</a>><br>
<b>Date: </b>Tuesday, 18 July 2017 at 18:13<br>
<b>To: </b>"<a href="mailto:users@ovirt.org">users@ovirt.org</a>" <<a href="mailto:users@ovirt.org">users@ovirt.org</a>><br>
<b>Cc: </b>"Brandon.Markgraf" <<a href="mailto:Brandon.Markgraf@target.com">Brandon.Markgraf@target.com</a>>, "Sandeep.Mendiratta" <<a href="mailto:Sandeep.Mendiratta@target.com">Sandeep.Mendiratta@target.com</a>><br>
<b>Subject: </b>[ovirt-users] Host stuck unresponsive after Network Outage</span><span lang="EN-GB" style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB" style="font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-GB">Hey Ovirt Users and Team,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">I have a host that I am unable to recover post a network outage. The host is stuck in unresponsive mode, even though the host is on the network, able to SSH and seems to be healthy. I’ve tried several things to recover
the host in Ovirt, but have had no success so far. I’d like to reach out to the community before blowing away and rebuilding the host.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-GB">Environment</span></b><span lang="EN-GB">: I have an Ovengine server with about 26 Datacenters, with 2 to 3 hosts per Datacenter. My Ovengine server is hosted centrally, with my hosts being bare-metal and distributed
throughout my environment. Ovengine is version 4.0.6. <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-GB">What I’ve tried: </span></b><span lang="EN-GB">put into maintenance mode, rebooted the host. Confirmed host was rebooted and tried to active, goes back to unresponsive. Attempted a reinstall, which fails.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-GB">Checking from the host perspective, I can see the following problems:
</span></b><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[boxname~]# systemctl status vdsmd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">● vdsmd.service - Virtual Desktop Server Manager<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> Active: inactive (dead)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 14 12:34:28 boxname systemd[1]: Dependency failed for Virtual Desktop Server Manager.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 14 12:34:28 boxname systemd[1]: Job vdsmd.service/start failed with result 'dependency'.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-GB">Going a bit deeper, the results of journalctl –xe:
</span></b><span lang="EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">[root@boxname ~]# journalctl -xe<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has begun shutting down.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Stopped Virtualization daemon.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit libvirtd.service has finished shutting down<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has finished shutting down.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Reloading.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Binding to IPv6 address not available since kernel does not support IPv6.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: [/usr/lib/systemd/system/rpcbind.socket:6] Failed to parse address value, ignoring: [::<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Started Auxiliary vdsm service for running helper functions as root.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit supervdsmd.service has finished start-up<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit supervdsmd.service has finished starting up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- The start-up result is done.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Starting Auxiliary vdsm service for running helper functions as root...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit supervdsmd.service has begun start-up<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit supervdsmd.service has begun starting up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Starting Virtualization daemon...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit libvirtd.service has begun start-up<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has begun starting up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:32 boxname systemd[1]: Started Virtualization daemon.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit libvirtd.service has finished start-up<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has finished starting up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- The start-up result is done.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:32 boxname systemd[1]: Starting Virtual Desktop Server Manager network restoration...<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit vdsm-network.service has begun start-up<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support: <a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel">
http://lists.freedesktop.org/mailman/listinfo/systemd-devel</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit vdsm-network.service has begun starting up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">lines 2751-2797/2797 (END)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Does the community have suggestions on what can be done next to recover this host within Ovirt? I can provide additional log dumps as needed, please inform with what you need to assist further.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Thank you,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Tony<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"> <o:p></o:p></span></p>
</div>
</body>
</html>