<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 22, 2015 at 4:03 PM, Will Dennis <span dir="ltr"><<a href="mailto:wdennis@nec-labs.com" target="_blank">wdennis@nec-labs.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I believe IPtables may be the culprit...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Host 1:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[root@ovirt-node-01 ~]# iptables -L<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain INPUT (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT icmp -- anywhere anywhere<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT all -- anywhere anywhere<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:54321<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT udp -- anywhere anywhere udp dpt:sunrpc<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:ssh<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT udp -- anywhere anywhere udp dpt:snmp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:16514<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere multiport dports rockwell-csp2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere multiport dports rfb:6923<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere multiport dports 49152:49216<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">REJECT all -- anywhere anywhere reject-with icmp-host-prohibited<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain FORWARD (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">REJECT all -- anywhere anywhere PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain OUTPUT (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Host 2:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[root@ovirt-node-02 ~]# iptables -L<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain INPUT (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT all -- anywhere anywhere state RELATED,ESTABLISHED<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT icmp -- anywhere anywhere<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT all -- anywhere anywhere<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:54321<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:sunrpc<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT udp -- anywhere anywhere udp dpt:sunrpc<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:ssh<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT udp -- anywhere anywhere udp dpt:snmp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere tcp dpt:16514<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere multiport dports rockwell-csp2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere multiport dports rfb:6923<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">ACCEPT tcp -- anywhere anywhere multiport dports 49152:49216<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">REJECT all -- anywhere anywhere reject-with icmp-host-prohibited<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain FORWARD (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">REJECT all -- anywhere anywhere PHYSDEV match ! --physdev-is-bridged reject-with icmp-host-prohibited<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain OUTPUT (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Host 3:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[root@ovirt-node-03 ~]# iptables -L<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain INPUT (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain FORWARD (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Chain OUTPUT (policy ACCEPT)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">target prot opt source destination<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">An example of my Gluster engine volume status (off host #2):<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[root@ovirt-node-02 ~]# gluster volume status<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Status of volume: engine<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Gluster process TCP Port RDMA Port Online Pid<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">------------------------------------------------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Brick ovirt-node-02:/gluster_brick2/engine_<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">brick 49217 0 Y 2973<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Brick ovirt-node-03:/gluster_brick3/engine_<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">brick N/A N/A N N/A<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Brick ovirt-node-02:/gluster_brick4/engine_<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">brick 49218 0 Y 2988<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Brick ovirt-node-03:/gluster_brick5/engine_<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">brick N/A N/A N N/A<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">NFS Server on localhost 2049 0 Y 3007<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Self-heal Daemon on localhost N/A N/A Y 3012<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">NFS Server on ovirt-node-03 2049 0 Y 1671<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Self-heal Daemon on ovirt-node-03 N/A N/A Y 1707<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I had changed the base port # per instructions found at</span>
<span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a href="http://www.ovirt.org/Features/Self_Hosted_Engine_Hyper_Converged_Gluster_Support" target="_blank">http://www.ovirt.org/Features/Self_Hosted_Engine_Hyper_Converged_Gluster_Support</a> :<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">“By default gluster uses a port that vdsm also wants, so we need to change base-port setting avoiding the clash between the two daemons. We need to add<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">option base-port 49217<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">to /etc/glusterfs/glusterd.vol<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">and ensure glusterd service is enabled and started before proceeding.”<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">So I did that on all the hosts:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">[root@ovirt-node-02 ~]# cat /etc/glusterfs/glusterd.vol<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">volume management<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> type mgmt/glusterd<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option working-directory /var/lib/glusterd<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option transport-type socket,rdma<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option transport.socket.keepalive-time 10<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option transport.socket.keepalive-interval 2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option transport.socket.read-fail-log off<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option ping-timeout 30<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"># option base-port 49152<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option base-port 49217<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> option rpc-auth-allow-insecure on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">end-volume<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Question: does oVirt really need IPtables to be enforcing rules, or can I just set everything wide open? If I can, how to specify that in setup?</span></p></div></div></blockquote><div><br></div>hosted-engine-setup asks:<br> iptables was detected on your computer, do you wish setup to configure it? (Yes, No)[Yes]: <br><br>You have just to say no here.<br><br>If you say no it's completely up to you to configure it opening the required ports or everything disabling it if you don't care.<br><br>The issue with gluster ports is that hosted-engine-setup simply configure iptables for what it knows you'll need and on 3.6 it's always assuming that the gluster volume is served by external hosts.<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div bgcolor="white" lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">W.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext"> Sahina Bose [mailto:<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, December 22, 2015 9:19 AM<br>
<b>To:</b> Will Dennis; Simone Tiraboschi; Dan Kenigsberg</span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [ovirt-users] Hosted Engine crash - state = EngineUp-EngineUpBadHealth<u></u><u></u></div></div><p></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 12/22/2015 07:47 PM, Sahina Bose wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal" style="margin-bottom:12pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 12/22/2015 07:28 PM, Will Dennis wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">See attached for requested log files</span><u></u><u></u></p>
</blockquote>
<p class="MsoNormal"><br>
>From gluster logs <br>
<br>
[2015-12-22 00:40:53.501341] W [MSGID: 108001] [afr-common.c:3924:afr_notify] 0-engine-replicate-1: Client-quorum is not met<br>
[2015-12-22 00:40:53.502288] W [socket.c:588:__socket_rwv] 0-engine-client-2: readv on <a href="http://138.15.200.93:49217" target="_blank">138.15.200.93:49217</a> failed (No data available)<br>
<br>
[2015-12-22 00:41:17.667302] W [fuse-bridge.c:2292:fuse_writev_cbk] 0-glusterfs-fuse: 3875597: WRITE => -1 (Read-only file system)<br>
<br>
Could you check if the gluster ports are open on all nodes?<u></u><u></u></p>
</blockquote>
<p class="MsoNormal"><br>
It's possible you ran into this ? - <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1288979" target="_blank">
https://bugzilla.redhat.com/show_bug.cgi?id=1288979</a><br>
<br>
<br>
<u></u><u></u></p>
<p class="MsoNormal"><br>
<br>
<br>
<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<div>
<div style="border-style:solid none none;border-top-color:rgb(181,196,223);border-top-width:1pt;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext">From:</span></b><span style="font-size:10pt;font-family:Tahoma,sans-serif;color:windowtext"> Sahina Bose [<a href="mailto:sabose@redhat.com" target="_blank">mailto:sabose@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, December 22, 2015 4:59 AM<br>
<b>To:</b> Simone Tiraboschi; Will Dennis; Dan Kenigsberg<br>
<b>Cc:</b> users<br>
<b>Subject:</b> Re: [ovirt-users] Hosted Engine crash - state = EngineUp-EngineUpBadHealth</span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12pt"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On 12/22/2015 02:38 PM, Simone Tiraboschi wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Dec 22, 2015 at 2:31 AM, Will Dennis <<a href="mailto:wdennis@nec-labs.com" target="_blank">wdennis@nec-labs.com</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12pt">OK, another problem :(<br>
<br>
I was having the same problem with my second oVirt host that I had with my first one, where when I ran “hosted-engine —deploy” on it, after it completed successfully, then I was experiencing a ~50sec lag when SSH’ing into the node…<br>
<br>
vpnp71:~ will$ time ssh root@ovirt-node-02 uptime<br>
19:36:06 up 4 days, 8:31, 0 users, load average: 0.68, 0.70, 0.67<br>
<br>
real 0m50.540s<br>
user 0m0.025s<br>
sys 0m0.008s<br>
<br>
<br>
So, in the oVirt web admin console, I put the "ovirt-node-02” node into Maintenance mode, then SSH’d to the server and rebooted it. Sure enough, after the server came back up, SSH was fine (no delay), which again was the same experience I had had with the first
oVirt host. So, I went back to the web console, and choose the “Confirm host has been rebooted” option, which I thought would be the right action to take after a reboot. The system opened a dialog box with a spinner, which never stopped spinning… So finally,
I closed the dialog box with the upper right (X) symbol, and then for this same host choose “Activate” from the menu. It was then I noticed I had recieved a state transition email notifying me that "EngineUp-EngineUpBadHealth” and sure enough, the web UI was
then unresponsive. I checked on the first oVirt host, the VM with the name “HostedEngine” is still running, but obviously isn’t working…<br>
<br>
So, looks like I need to restart the HostedEngine VM or take whatever action is needed to return oVirt to operation… Hate to keep asking this question, but what’s the correct action at this point?<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ovirt-ha-agent should always restart it for you after a few minutes but the point is that the network configuration seams to be not that stable.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I know from another thread that you are trying to deploy hosted-engine over GlusterFS in an hyperconverged way and this, as I said, is currently not supported.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I think that it can also requires some specific configuration on network side.<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
<p class="MsoNormal"><br>
For hyperconverged gluster+engine , it should work without any specific configuration on network side. However if the network is flaky, it is possible that there are errors with gluster volume access. Could you provide the ovirt-ha-agent logs as well as gluster
mount logs?<br>
<br>
<br>
<br>
<br>
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Adding Sahina and Dan here.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<p class="MsoNormal">Thanks, again,<br>
Will<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div></div>
</div>
</blockquote></div><br></div></div>