<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 17, 2017 at 9:26 AM, Pavel Gashev <span dir="ltr"><<a href="mailto:Pax@acronis.com" target="_blank">Pax@acronis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" link="blue" vlink="purple" lang="EN-GB">
<div class="m_-3566270498995188994WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Nir,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri">Isn’t SPM managed via Sanlock? I believe there is no need to fence SPM host. Especially if there are no SPM tasks running.</span></p></div></div></blockquote><div><br></div><div>It's true that the exclusivity of the SPM role is enforced by Sanlock but you always need to fence a non-responsive SPM because there is no way to guarantee that the host is not still manipulating storage (eg. LV extensions) and we must ensure that only one host has the masterfs on the master storage domain mounted.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="white" link="blue" vlink="purple" lang="EN-GB"><div class="m_-3566270498995188994WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black"><<a href="mailto:users-bounces@ovirt.org" target="_blank">users-bounces@ovirt.org</a>> on behalf of Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>><br>
<b>Date: </b>Monday, 17 April 2017 at 16:06<br>
<b>To: </b>Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>>, Dan Yasny <<a href="mailto:dyasny@gmail.com" target="_blank">dyasny@gmail.com</a>><br>
<b>Cc: </b>users <<a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a>>, FERNANDO FREDIANI <<a href="mailto:fernando.frediani@upx.com" target="_blank">fernando.frediani@upx.com</a>><br>
<b>Subject: </b>Re: [ovirt-users] storage redundancy in Ovirt<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div><span class="">
<div>
<p class="MsoNormal">On Mon, Apr 17, 2017 at 8:24 AM Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
</span><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div><span class="">
<p class="MsoNormal">But actually, it didn't work well. After main SPM host went down I see this
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</span><div>
<p class="MsoNormal"><br></p><div><div class="h5">
2017-04-17 05:23:15,554Z ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [4dcc033d-26bf-49bb-bfaa-<wbr>03a970dbbec1] SPM Init: could not find reported vds or not up - pool: 'STG' vds_spm_id: '1'<u></u><u></u></div></div><p></p>
</div><div><div class="h5">
<div>
<p class="MsoNormal">2017-04-17 05:23:15,567Z INFO [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [4dcc033d-26bf-49bb-bfaa-<wbr>03a970dbbec1] SPM selection - vds seems as spm 'tank5'<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">2017-04-17 05:23:15,567Z WARN [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.IrsProxy] (DefaultQuartzScheduler5) [4dcc033d-26bf-49bb-bfaa-<wbr>03a970dbbec1] spm vds is non responsive, stopping spm selection.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So that means only if BMC is up it's possible to automatically switch SPM host?<u></u><u></u></p>
</div>
</div></div></div>
</blockquote><div><div class="h5">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">BMC?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If your SPM is no responsive, the system will try to fence it. Did you <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">configure power management for all hosts? did you check that it <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">work? How did you simulate non-responsive host? <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If power management is not configured or fail, the system cannot<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">move the spm to another host, unless you manually confirm that the<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">SPM host was rebooted.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Nir<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 8:29 PM, Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Oh, fence agent works fine if I select ilo4,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thank you for your help!<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 8:22 PM Dan Yasny <<a href="mailto:dyasny@gmail.com" target="_blank">dyasny@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 11:19 PM, Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Makes sense. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I was trying to set it up, but doesn't work with our staging hardware. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">We have old ilo100, I'll try again. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks! <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">It is absolutely necessary for any HA to work properly. There's of course the "confirm host has been shutdown" option, which serves as an override for the fence command, but it's manual<u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 8:18 PM Dan Yasny <<a href="mailto:dyasny@gmail.com" target="_blank">dyasny@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 11:15 PM, Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Fence agent under each node?<u></u><u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">When you configure a host, there's the power management tab, where you need to enter the bmc details for the host. If you don't have fencing enabled, how do you expect the system to make sure a host running a service is actually down (and
it is safe to start HA services elsewhere), and not, for example, just unreachable by the engine? How do you avoid a splitbraid -> SBA ?<u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 8:14 PM Dan Yasny <<a href="mailto:dyasny@gmail.com" target="_blank">dyasny@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 11:13 PM, Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">"Corner cases"? <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I tried to simulate crash of SPM server and ovirt kept trying to reistablished connection to the failed node. <u></u><u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Did you configure fencing?<u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 8:10 PM Dan Yasny <<a href="mailto:dyasny@gmail.com" target="_blank">dyasny@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 7:29 AM, Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 2:05 PM Dan Yasny <<a href="mailto:dyasny@redhat.com" target="_blank">dyasny@redhat.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Apr 16, 2017 7:01 AM, "Nir Soffer" <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<p class="MsoNormal">On Sun, Apr 16, 2017 at 4:17 AM Dan Yasny <<a href="mailto:dyasny@gmail.com" target="_blank">dyasny@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">When you set up a storage domain, you need to specify a host to perform the initial storage operations, but once the SD is defined, it's details are in the engine database, and all the hosts get connected to it directly. If the first host
you used to define the SD goes down, all other hosts will still remain connected and work. SPM is an HA service, and if the current SPM host goes down, SPM gets started on another host in the DC. In short, unless your actual NFS exporting host goes down, there
is no outage. <u></u><u></u></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">There is no storage outage, but if you shutdown the spm host, the spm host<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">will not move to a new host until the spm host is online again, or you confirm<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">manually that the spm host was rebooted.<u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">In a properly configured setup the SBA should take care of that. That's the whole point of HA services <u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In some cases like power loss or hardware failure, there is no way to start<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">the spm host, and the system cannot recover automatically.<u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">There are always corner cases, no doubt. But in a normal situation. where an SPM host goes down because of a hardware failure, it gets fenced, other hosts contend for SPM and start it. No surprises there. <u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Nir<u></u><u></u></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><span style="color:#888888"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Nir<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Sat, Apr 15, 2017 at 1:53 PM, Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Hi Fernando,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I see each host has direct connection nfs mount, but yes, if main host to which I connected nfs storage going down the storage becomes unavailable and all vms are down<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sat, Apr 15, 2017 at 10:37 AM FERNANDO FREDIANI <<a href="mailto:fernando.frediani@upx.com" target="_blank">fernando.frediani@upx.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello Konstantin.<u></u><u></u></p>
</div>
<p class="MsoNormal">That doesn`t make much sense make a whole cluster depend on a single host. From what I know any host talk directly to NFS Storage Array or whatever other Shared Storage you have.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Have you tested that host going down if that affects the other with the NFS mounted directlly in a NFS Storage array ?<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">Fernando<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2017-04-15 12:42 GMT-03:00 Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>>:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">In ovirt you have to attach storage through specific host. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If host goes down storage is not available. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sat, Apr 15, 2017 at 7:31 AM FERNANDO FREDIANI <<a href="mailto:fernando.frediani@upx.com" target="_blank">fernando.frediani@upx.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal">Well, make it not go through host1 and dedicate a storage server for running NFS and make both hosts connect to it.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">In my view NFS is much easier to manage than any other type of storage, specially FC and iSCSI and performance is pretty much the same, so you won`t get better results other than management going to other type.<u></u><u></u></p>
</div>
<p class="MsoNormal">Fernando<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">2017-04-15 5:25 GMT-03:00 Konstantin Raskoshnyi <<a href="mailto:konrasko@gmail.com" target="_blank">konrasko@gmail.com</a>>:<u></u><u></u></p>
</div>
</div>
<div>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Hi guys, <u></u><u></u></p>
<div>
<p class="MsoNormal">I have one nfs storage,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">it's connected through host1.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">host2 also has access to it, I can easily migrate vms between them.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The question is - if host1 is down - all infrastructure is down, since all traffic goes through host1,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">is there any way in oVirt to use redundant storage?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Only glusterfs?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</blockquote>
</div>
</div>
<div>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt">______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</blockquote>
</div></div></div>
</div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Adam Litke</div></div>
</div></div>