<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Can you post the gluster mount logs from the node where paused VM
was running (under
/var/log/glusterfs/rhev-datarhev-data-center-mnt-glusterSD<mount-path>.log)
? <br>
Which version of glusterfs are you running?<br>
<br>
<div class="moz-cite-prefix">On 06/24/2016 07:49 AM, Bill Bill
wrote:<br>
</div>
<blockquote
cite="mid:CO2PR0801MB0743180FC142ED11D7D5D3E0A62E0@CO2PR0801MB0743.namprd08.prod.outlook.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal">Hello,</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Have 3 nodes running both oVirt and Gluster
on 4 SSD’s each. At the moment, there are two physical nics,
one has public internet access and the other is a non-routable
network used for ovirtmgmt & gluster.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In the logical networks, I have selected
gluster for the nonroutable network running ovirtmgmt and
gluster however, two VM’s randomly pause for what seems like
no reason. They can both be resumed without issue.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One test VM has 4GB of memory and a small
disk – no problems with this one. Two others have 800GB disks
and 32GB of RAM – both vm’s exhibit the same issue.</p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">I also see these in the oVirt dashboard:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">
<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">Failed to update OVF disks
9e60328d-29af-4533-84f9-633d87f548a7, OVF data isn't updated
on those OVF stores (Data Center xxxxx, Storage Domain
sr-volume01).<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">Jun 23, 2016 9:54:03 PM<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">VDSM command failed: Could not acquire
resource. Probably resource factory threw an exception.: ()<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">///////////////<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">VM xxxxx has been paused due to unknown
storage error.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">///////////////<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">In the error log on the engine, I see
these:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-7) [10caf93e] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx
has been paused due to unknown storage error.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-11) [10caf93e] Correlation ID: null,
Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx
has recovered from paused back to up.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">///////////////<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">Hostnames are all local to /etc/hosts on
all servers – they also resolve without issue from each
host.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">//////////////<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,611 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3435:/mnt/data/sr-volume01' of volume
'93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,614 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3637:/mnt/data/sr-volume01' of volume
'93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,616 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3839:/mnt/data/sr-volume01' of volume
'93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,618 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3435:/mnt/data/distributed' of volume
'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,620 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3637:/mnt/data/distributed' of volume
'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,622 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3839:/mnt/data/distributed' of volume
'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,624 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3435:/mnt/data/iso' of volume
'89f32457-c8c3-490e-b491-16dd27de0073' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,626 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3637:/mnt/data/iso' of volume
'89f32457-c8c3-490e-b491-16dd27de0073' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,628 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not
associate brick 'ovirt3839:/mnt/data/iso' of volume
'89f32457-c8c3-490e-b491-16dd27de0073' with correct network
as no gluster network found in cluster
'75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:08:59,629 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-76) [1c1cf4f] FINISH,
GlusterVolumesListVDSCommand, return:
{b887b05e-2ea6-496e-9552-155d658eeaa6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d8c9039b,
93e36cdc-ab1b-41ec-ac7f-966cf3856b59=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe5ab019,
89f32457-c8c3-490e-b491-16dd27de0073=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9a56d633},
log id: 485a0611<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:12.0pt;font-family:"Times New
Roman",serif">2016-06-23 22:09:04,645 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler_Worker-89) [] START,
GlusterVolumesListVDSCommand(HostName = ovirt3839,
GlusterVolumesListVDSParameters:{runAsync='true',
hostId='32c500e5-268d-426a-9a4a-108535e67722'}), log id:
41b6479d<o:p></o:p></span></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
<br>
</body>
</html>