<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body>
<div>I did not have a good experience putting both gluster and virt on the same node. I was doing hosted engine with replicate across two nodes, and one day it went into split brain hell... i was never able to track down why. However i do have a gluster with distribute and replica setup on its own with a couple nodes, and it has given me zero problems in the last 60 days. It seems to me that gluster and virt need to stay seperate for now. Both are great products and both work as described, just not on the same node at the same time.</div><div><br></div><div><br></div><div><br></div><div id="composer_signature"><div style="font-size:85%;color:#575757">Happy Connecting. Sent from my Sprint Samsung Galaxy S® 5</div></div><br><br>-------- Original message --------<br>From: George Skorup <george@mwcomm.com> <br>Date: 02/18/2015 12:50 PM (GMT-07:00) <br>To: users@ovirt.org <br>Subject: Re: [ovirt-users] ovirt and glusterfs setup <br><br>
Bill,<br>
<br>
I have done exactly what you're looking to do. I was trying to mimic
vSAN as well. They had VSA for a while which was acceptable
licensing costs, but that was replaced with vSAN which is
ridiculously expensive for an extra-small business.<br>
<br>
I have a four node cluster with 1TB of storage each. Gluster is
configured with replica 4. So basically I have 1TB of usable
storage, which is fine for my needs. VM migration, Gluster
replication, hosted engine and all that works fine. Performance is
generally fine, even with only dual LACP bonded 1GbE NICs in each
node. I can do what I want with networking to fit our NOC and office
network environment.<br>
<br>
I have been playing with this for about three weeks. Over the
weekend, I had a handful of VMs running, including the hosted
engine. They were pretty much sitting idle doing nothing. I came in
Monday and found everything offline. No power outages,
network/switch didn't fail or reboot, none of the hardware reset.
What I found happened was that glusterd went nuts on two of the
nodes (I have no idea why). Gluster was spitting out logs like
crazy, /var filled up, then RAM and swap was depleted. Two gluster
processes/hosts offline out of the four meant quorum broke,
everything came to a halt. I was unable to recover the gluster logs
since they had to be deleted to free up space in /var.<br>
<br>
I was able to get everything fixed and back online in about 2-1/2
hours. So this is impossible to put into production. Storage is the
weakest link and mostly likely to fail, which in this case it did.<br>
<br>
I do not recommend this configuration at all. A dedicated machine
for the engine, dedicated hypervisors and dedicated storage nodes or
a SAN is needed for anything beyond experimentation.<br>
<br>
I have zero knowledge of VMWare's vSAN, other than basic concepts,
so I cannot say the results would be similar.<br>
<br>
<br>
<div class="moz-cite-prefix">On 2/18/2015 5:32 AM, Bill Dossett
wrote:<br>
</div>
<blockquote cite="mid:B47F00825D68444ABC037330782753603771A364@013-BR1MPN1-012.MGDPBI.global.pvt" type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m in at the deep end…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Looking for some advice on if this is
possible – and what version I should try I with if so.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are a heavily VMware oriented company…
but I am trying to get RHEV/oVirt in the door. Honestly I
would prefer oVirt, but if they insist on enterprise support I
go to Redhat.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So I have had a play with VMware VSAN and
know how it works. I am trying to more or less replicate this
setup using open source.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So VSAN uses 3 or more ESXi nodes, with
local harddisks as the storage medium as a virtual SAN. The
SAN performance is quite high as you put at least 1 SSD in
each node.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What is nice is you don’t have the NAS
element here. VMs are on the local storage, which is
partially SSD, so performance is quite good.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I went to a Redhat presentation on Redhat
storage and GlusterFS and basically this acts as a big
software defined NAS which does some pretty cool things, but
that’s not exactly what I need.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I would like to build oVirt on top of
CentOS Oss… that have local storage in them that is
distributed and redundant in the event of a node failure.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And I probably need to try and build this
in a lab under Fusion on my Mac Book pro to begin with anyway
(that bit I’m farily confident with) and if I get that working
can probably get some older kit to try it out for real as a
PoC to few people.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So, I’ve set up oVirt before that should be
ok, I haven’t setup Gluster is there any documents that would
help me down this road and make sure I start out using the
best version.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any advice or pointers would be gratefully
received.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" style="width:480.0pt" border="0" cellpadding="0" cellspacing="0" width="640">
<tbody>
<tr>
<td style="width:480.0pt;padding:0cm 0cm 0cm 0cm" width="640">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0082D5;mso-fareast-language:EN-GB">Bill
Dossett</span></b><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">Systems
Architect</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="width:480.0pt;padding:0cm 0cm 0cm 0cm" width="640">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">Tech
Central – Global Engineering Services</span></b><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-GB"> </span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">T</span></b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB"> +44
(0)1923 279353</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">M</span></b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB"> +44
(0)777 590 8612</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-GB"><a moz-do-not-send="true" href="mailto:jeff.gonzalez@pb.com"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:purple">b</span></a></span><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0082D5;mso-fareast-language:EN-GB"><a moz-do-not-send="true" href="mailto:ill.dossett@pb.com"><span style="color:blue">ill.dossett@pb.com</span></a></span><span style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0082D5;mso-fareast-language:EN-GB"><a moz-do-not-send="true" href="pitneybowes.com"><span style="color:#0082D5">pitneybowes.com</span></a></span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-GB"><br>
</span><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">Pitney
Bowes</span></b><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">6
Hercules Way | Leavesden | WD25 7GS | UK</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB"> </span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="width:480.0pt;padding:0cm 0cm 0cm 0cm" width="640"><br>
</td>
</tr>
<tr>
<td style="width:480.0pt;padding:0cm 0cm 0cm 0cm" width="640">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#0082D5;mso-fareast-language:EN-GB"><a moz-do-not-send="true" href="http://teamsite.inside.pb.com/sites/TCPPM/GES/SitePages/Home.aspx"><span style="color:#0082D5">Learn more about Global
Engineering Services</span></a></span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="page-break-after:avoid"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB"> </span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal" style="page-break-after:avoid"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">In
Engineering?</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">Raise
a ticket via Remedy Anywhere [<span style="color:#0082D5"><a moz-do-not-send="true" href="teamsite.inside.pb.com/sites/TCPPM/GES/Lists/Create%20Incident/newIncident.aspx"><span style="color:#0082D5">HERE</span></a></span>]<span style="color:blue"> </span>takes less than a
minute</span></b><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB"> </span></b><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB">CloudForms
User Guide available</span></b><span style="font-size:10.0pt;font-family:"Arial","sans-serif";mso-fareast-language:EN-GB"> [<b><span style="color:#0082D5"><a moz-do-not-send="true" href="https://dl.dropboxusercontent.com/u/6219441/ManageIQ%20-%20User%20Manual%20PB%20v5.pdf"><span style="color:#0082D5">HERE</span></a></span></b>]</span><span style="mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New
Roman","serif";mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<hr>
<font color="Black" face="Arial" size="2"><br>
</font>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</body></html>