<div dir="ltr">Looking at vdsmd.service on one of my 4.0 hosts.<div><br></div><div><div>Requires=multipathd.service libvirtd.service time-sync.target \</div><div>         iscsid.service rpcbind.service supervdsmd.service sanlock.service \</div><div>         vdsm-network.service</div></div><div><br></div><div>Are all these services present and running?</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 19 July 2017 at 16:05, Anthony.Fillmore <span dir="ltr">&lt;<a href="mailto:Anthony.Fillmore@target.com" target="_blank">Anthony.Fillmore@target.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





<div lang="EN-US" link="blue" vlink="purple">
<div class="m_6225255169149210825WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">Are the vdsm.conf or mom.conf file in /etc/vdsm of note in this situation? 
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Anthony.Fillmore
<br>
<b>Sent:</b> Wednesday, July 19, 2017 9:57 AM<br>
<b>To:</b> &#39;Alan Griffiths&#39; &lt;<a href="mailto:apgriffiths79@gmail.com" target="_blank">apgriffiths79@gmail.com</a>&gt;<span class=""><br>
<b>Cc:</b> Pavel Gashev &lt;<a href="mailto:Pax@acronis.com" target="_blank">Pax@acronis.com</a>&gt;; <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>; Brandon.Markgraf &lt;<a href="mailto:Brandon.Markgraf@target.com" target="_blank">Brandon.Markgraf@target.com</a>&gt;; Sandeep.Mendiratta &lt;<a href="mailto:Sandeep.Mendiratta@target.com" target="_blank">Sandeep.Mendiratta@target.com</a><wbr>&gt;<br>
</span><b>Subject:</b> RE: [ovirt-users] [EXTERNAL] Re: Host stuck unresponsive after Network Outage<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">[boxname ~]# systemctl | grep -i dead<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">mom-vdsm.service              <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                        
 start MOM instance configured for VDSM purposes<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">vdsmd.service                 <wbr>                                 <wbr>                              <wbr>                              <wbr>                              <wbr>                      start
 Virtual Desktop Server Manager<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">[ boxname ~]# systemctl | grep -i exited<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">blk-availability.service                                  <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>   Availability
 of block devices<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">iptables.service              <wbr>                              <wbr>                                           <wbr>                              <wbr>                              <wbr>                  IPv4
 firewall with iptables<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">kdump.service                 <wbr>                              <wbr>                              <wbr>                                                            <wbr>                              <wbr> Crash
 recovery kernel arming<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">kmod-static-nodes.service     <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>               
                Create list of required static device nodes for the current kernel<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">lvm2-monitor.service          <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                       
        Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"><a href="mailto:lvm2-pvscan@253:3.service" target="_blank">lvm2-pvscan@253:3.service</a>     <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                    
           LVM2 PV scan on device 253:3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"><a href="mailto:lvm2-pvscan@253:4.service" target="_blank">lvm2-pvscan@253:4.service</a>     <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                              
 LVM2 PV scan on device 253:4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"><a href="mailto:lvm2-pvscan@8:3.service" target="_blank">lvm2-pvscan@8:3.service</a>       <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                              
 LVM2 PV scan on device 8:3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">network.service               <wbr>                                                <wbr>                              <wbr>                              <wbr>                              <wbr>             LSB:
 Bring up/down networking<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">openvswitch-nonetwork.service <wbr>                              <wbr>                              <wbr>                                 <wbr>                              <wbr>                            <wbr>Open
 vSwitch Internal Unit<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">openvswitch.service           <wbr>                              <wbr>                              <wbr>                              <wbr>                                                   <wbr>          Open
 vSwitch<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">rhel-dmesg.service            <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                              
 Dump dmesg to /var/log/dmesg<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">rhel-import-state.service     <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                              
 Import network configuration from initramfs<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">rhel-readonly.service                                 <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>       Configure
 read-only root support<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-journal-flush.service <wbr>                              <wbr>                                    <wbr>                              <wbr>                              <wbr>                         Flush
 Journal to Persistent Storage<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-modules-load.service  <wbr>                              <wbr>                              <wbr>                                             <wbr>                              <wbr>                Load
 Kernel Modules<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-random-seed.service   <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>         
                      Load/Save Random Seed<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-readahead-collect.<wbr>service                       <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>    
 Collect Read-Ahead Data<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-readahead-replay.<wbr>service                       <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>     
 Replay Read-Ahead Data<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-remount-fs.service    <wbr>                                                            <wbr>                              <wbr>                              <wbr>                              <wbr> Remount
 Root and Kernel File Systems<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-sysctl.service        <wbr>                              <wbr>                              <wbr>                                      <wbr>                              <wbr>                       Apply
 Kernel Variables<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-tmpfiles-setup-dev.<wbr>service                       <wbr>                              <wbr>                              <wbr>                              <wbr>                                 <wbr> Create
 Static Device Nodes in /dev<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-tmpfiles-setup.<wbr>service                       <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>       
 Create Volatile Files and Directories<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-udev-trigger.service  <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                              
 udev Coldplug all Devices<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-update-utmp.service   <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>                              
 Update UTMP about System Boot/Shutdown<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-user-sessions.service <wbr>                                                        <wbr>                              <wbr>                              <wbr>                              <wbr>     Permit
 User Sessions<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">systemd-vconsole-setup.<wbr>service                       <wbr>                              <wbr>                                                         <wbr>                              <wbr>           Setup
 Virtual Console<u></u><u></u></span></p>
</div></div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">vdsm-network-init.service     <wbr>                              <wbr>                              <wbr>                              <wbr>                              <wbr>            
                   Virtual Desktop Server Manager network IP+link restoration<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Alan Griffiths [<a href="mailto:apgriffiths79@gmail.com" target="_blank">mailto:apgriffiths79@gmail.<wbr>com</a>]
<br>
<b>Sent:</b> Wednesday, July 19, 2017 9:47 AM</span></p><div><div class="h5"><br>
<b>To:</b> Anthony.Fillmore &lt;<a href="mailto:Anthony.Fillmore@target.com" target="_blank">Anthony.Fillmore@target.com</a>&gt;<br>
<b>Cc:</b> Pavel Gashev &lt;<a href="mailto:Pax@acronis.com" target="_blank">Pax@acronis.com</a>&gt;; <a href="mailto:users@ovirt.org" target="_blank">
users@ovirt.org</a>; Brandon.Markgraf &lt;<a href="mailto:Brandon.Markgraf@target.com" target="_blank">Brandon.Markgraf@target.com</a>&gt;; Sandeep.Mendiratta &lt;<a href="mailto:Sandeep.Mendiratta@target.com" target="_blank">Sandeep.Mendiratta@target.com</a><wbr>&gt;<br>
<b>Subject:</b> Re: [ovirt-users] [EXTERNAL] Re: Host stuck unresponsive after Network Outage<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Are there other failed services?<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">systemctl --state=failed<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 19 July 2017 at 15:40, Anthony.Fillmore &lt;<a href="mailto:Anthony.Fillmore@target.com" target="_blank">Anthony.Fillmore@target.com</a>&gt; wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">Hey Alan,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">Rpcbind is running on my box, looks like no issue there.  Any other ideas on what could be keeping
 vdsmd dead?  I even uninstalled all Ovirt related components from the host and went for a reinstall of the host through Ovirt (just short of actually fully removing the host from ovirt and re-adding, which I want to avoid) and the reinstall ends up timing
 out when it attempts to start VDSM (checking logs can see the service is dead when it gets here).</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d">Tony</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Alan Griffiths [mailto:<a href="mailto:apgriffiths79@gmail.com" target="_blank">apgriffiths79@gmail.<wbr>com</a>]
<br>
<b>Sent:</b> Wednesday, July 19, 2017 4:14 AM<br>
<b>To:</b> Anthony.Fillmore &lt;<a href="mailto:Anthony.Fillmore@target.com" target="_blank">Anthony.Fillmore@target.com</a>&gt;<br>
<b>Cc:</b> Pavel Gashev &lt;<a href="mailto:Pax@acronis.com" target="_blank">Pax@acronis.com</a>&gt;;
<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>; Brandon.Markgraf &lt;<a href="mailto:Brandon.Markgraf@target.com" target="_blank">Brandon.Markgraf@target.com</a>&gt;; Sandeep.Mendiratta &lt;<a href="mailto:Sandeep.Mendiratta@target.com" target="_blank">Sandeep.Mendiratta@target.com</a><wbr>&gt;<br>
<b>Subject:</b> Re: [ovirt-users] [EXTERNAL] Re: Host stuck unresponsive after Network Outage</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Is rpcbind running? This is a dependency for vdsmd.<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I&#39;ve seen issues where rpcbind will not start on boot if IPv6 is disabled. The solution for me was to rebuild the initramfs, aka &quot;dracut -f&quot;<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On 18 July 2017 at 18:13, Anthony.Fillmore &lt;<a href="mailto:Anthony.Fillmore@target.com" target="_blank">Anthony.Fillmore@target.com</a>&gt; wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="color:#1f497d">[boxname ~]# systemctl status -l vdsm-network</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">● vdsm-network.service - Virtual Desktop Server Manager network restoration</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">   Loaded: loaded (/usr/lib/systemd/system/vdsm-<wbr>network.service; enabled; vendor preset: enabled)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">   Active: activating (start) since Tue 2017-07-18 10:42:57 CDT; 1h 29min ago</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">  Process: 8216 ExecStartPre=/usr/bin/vdsm-<wbr>tool --vvverbose --append --logfile=/var/log/vdsm/<wbr>upgrade.log upgrade-unified-persistence (code=exited, status=0/SUCCESS)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Main PID: 8231 (vdsm-tool)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">   CGroup: /system.slice/vdsm-network.<wbr>service</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">          
</span><span style="font-family:&quot;Arial&quot;,sans-serif;color:#1f497d">├</span><span style="color:#1f497d">─8231 /usr/bin/python /usr/bin/vdsm-tool restore-nets</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">           └─8240 /usr/bin/python /usr/share/vdsm/vdsm-restore-<wbr>net-config</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Jul 18 10:42:57
<a href="http://t0894bmh1001.stores.target.com" target="_blank">t0894bmh1001.stores.target.com</a> systemd[1]: Starting Virtual Desktop Server Manager network restoration...</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#1f497d">Tony</span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Pavel Gashev [mailto:<a href="mailto:Pax@acronis.com" target="_blank">Pax@acronis.com</a>]
<br>
<b>Sent:</b> Tuesday, July 18, 2017 11:17 AM<br>
<b>To:</b> Anthony.Fillmore &lt;<a href="mailto:Anthony.Fillmore@target.com" target="_blank">Anthony.Fillmore@target.com</a>&gt;;
<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
<b>Cc:</b> Brandon.Markgraf &lt;<a href="mailto:Brandon.Markgraf@target.com" target="_blank">Brandon.Markgraf@target.com</a>&gt;; Sandeep.Mendiratta &lt;<a href="mailto:Sandeep.Mendiratta@target.com" target="_blank">Sandeep.Mendiratta@target.com</a><wbr>&gt;<br>
<b>Subject:</b> [EXTERNAL] Re: [ovirt-users] Host stuck unresponsive after Network Outage<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Anthony,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Output of “systemctl status -l vdsm-network” would help.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span lang="EN-GB" style="color:black">From:
</span></b><span lang="EN-GB" style="color:black">&lt;<a href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.org</a>&gt; on behalf of &quot;Anthony.Fillmore&quot; &lt;<a href="mailto:Anthony.Fillmore@target.com" target="_blank">Anthony.Fillmore@target.com</a>&gt;<br>
<b>Date: </b>Tuesday, 18 July 2017 at 18:13<br>
<b>To: </b>&quot;<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>&quot; &lt;<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>&gt;<br>
<b>Cc: </b>&quot;Brandon.Markgraf&quot; &lt;<a href="mailto:Brandon.Markgraf@target.com" target="_blank">Brandon.Markgraf@target.com</a>&gt;, &quot;Sandeep.Mendiratta&quot; &lt;<a href="mailto:Sandeep.Mendiratta@target.com" target="_blank">Sandeep.Mendiratta@target.com</a><wbr>&gt;<br>
<b>Subject: </b>[ovirt-users] Host stuck unresponsive after Network Outage</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span lang="EN-GB">Hey Ovirt Users and Team,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I have a host that I am unable to recover post a network outage.  The host is stuck in unresponsive mode, even though the host is on the network, able to SSH
 and seems to be healthy.  I’ve tried several things to recover the host in Ovirt, but have had no success so far.  I’d like to reach out to the community before blowing away and rebuilding the host.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-GB">Environment</span></b><span lang="EN-GB">: I have an Ovengine server with about 26 Datacenters, with 2 to 3 hosts per Datacenter.  My Ovengine server is hosted
 centrally, with my hosts being bare-metal and distributed throughout my environment.    Ovengine is version 4.0.6. 
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-GB">What I’ve tried:
</span></b><span lang="EN-GB">put into maintenance mode, rebooted the host.  Confirmed host was rebooted and tried to active, goes back to unresponsive.   Attempted a reinstall, which fails. 
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-GB">Checking from the host perspective, I can see the following problems:
</span></b><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">[boxname~]# systemctl status vdsmd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">● vdsmd.service - Virtual Desktop Server Manager</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">   Loaded: loaded (/usr/lib/systemd/system/<wbr>vdsmd.service; enabled; vendor preset: enabled)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">   Active: inactive (dead)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 14 12:34:28 boxname systemd[1]: Dependency failed for Virtual Desktop Server Manager.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 14 12:34:28 boxname systemd[1]: Job vdsmd.service/start failed with result &#39;dependency&#39;.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-GB">Going a bit deeper, the results of journalctl –xe:
</span></b><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">[root@boxname ~]# journalctl -xe</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support:
<a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">
http://lists.freedesktop.org/<wbr>mailman/listinfo/systemd-devel</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has begun shutting down.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Stopped Virtualization daemon.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit libvirtd.service has finished shutting down</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support:
<a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">
http://lists.freedesktop.org/<wbr>mailman/listinfo/systemd-devel</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has finished shutting down.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Reloading.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Binding to IPv6 address not available since kernel does not support IPv6.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: [/usr/lib/systemd/system/<wbr>rpcbind.socket:6] Failed to parse address value, ignoring: [::</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Started Auxiliary vdsm service for running helper functions as root.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit supervdsmd.service has finished start-up</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support:
<a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">
http://lists.freedesktop.org/<wbr>mailman/listinfo/systemd-devel</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit supervdsmd.service has finished starting up.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- The start-up result is done.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Starting Auxiliary vdsm service for running helper functions as root...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit supervdsmd.service has begun start-up</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support:
<a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">
http://lists.freedesktop.org/<wbr>mailman/listinfo/systemd-devel</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit supervdsmd.service has begun starting up.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:31 boxname systemd[1]: Starting Virtualization daemon...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit libvirtd.service has begun start-up</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support:
<a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">
http://lists.freedesktop.org/<wbr>mailman/listinfo/systemd-devel</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has begun starting up.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:32 boxname systemd[1]: Started Virtualization daemon.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit libvirtd.service has finished start-up</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support:
<a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">
http://lists.freedesktop.org/<wbr>mailman/listinfo/systemd-devel</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit libvirtd.service has finished starting up.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- The start-up result is done.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Jul 18 09:07:32 boxname systemd[1]: Starting Virtual Desktop Server Manager network restoration...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Subject: Unit vdsm-network.service has begun start-up</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Defined-By: systemd</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Support:
<a href="http://lists.freedesktop.org/mailman/listinfo/systemd-devel" target="_blank">
http://lists.freedesktop.org/<wbr>mailman/listinfo/systemd-devel</a></span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">--</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-- Unit vdsm-network.service has begun starting up.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">lines 2751-2797/2797 (END)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Does the community have suggestions on what can be done next to recover this host within Ovirt?  I can provide additional log dumps as needed, please inform with
 what you need to assist further.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Thank you,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Tony</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>

</blockquote></div><br></div>