This is a multi-part message in MIME format. --------------070800090401040005080909 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Can you post the gluster mount logs from the node where paused VM was running (under /var/log/glusterfs/rhev-datarhev-data-center-mnt-glusterSD<mount-path>.log) ? Which version of glusterfs are you running? On 06/24/2016 07:49 AM, Bill Bill wrote:
Hello,
Have 3 nodes running both oVirt and Gluster on 4 SSDs each. At the moment, there are two physical nics, one has public internet access and the other is a non-routable network used for ovirtmgmt & gluster.
In the logical networks, I have selected gluster for the nonroutable network running ovirtmgmt and gluster however, two VMs randomly pause for what seems like no reason. They can both be resumed without issue.
One test VM has 4GB of memory and a small disk no problems with this one. Two others have 800GB disks and 32GB of RAM both vms exhibit the same issue.
I also see these in the oVirt dashboard:
Failed to update OVF disks 9e60328d-29af-4533-84f9-633d87f548a7, OVF data isn't updated on those OVF stores (Data Center xxxxx, Storage Domain sr-volume01).
Jun 23, 2016 9:54:03 PM
VDSM command failed: Could not acquire resource. Probably resource factory threw an exception.: ()
///////////////
VM xxxxx has been paused due to unknown storage error.
///////////////
In the error log on the engine, I see these:
ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has been paused due to unknown storage error.
INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has recovered from paused back to up.
///////////////
Hostnames are all local to /etc/hosts on all servers they also resolve without issue from each host.
//////////////
2016-06-23 22:08:59,611 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,614 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,616 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,618 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,620 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,622 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,624 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,626 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,628 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'
2016-06-23 22:08:59,629 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] FINISH, GlusterVolumesListVDSCommand, return: {b887b05e-2ea6-496e-9552-155d658eeaa6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d8c9039b, 93e36cdc-ab1b-41ec-ac7f-966cf3856b59=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe5ab019, 89f32457-c8c3-490e-b491-16dd27de0073=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9a56d633}, log id: 485a0611
2016-06-23 22:09:04,645 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [] START, GlusterVolumesListVDSCommand(HostName = ovirt3839, GlusterVolumesListVDSParameters:{runAsync='true', hostId='32c500e5-268d-426a-9a4a-108535e67722'}), log id: 41b6479d
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------070800090401040005080909 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> Can you post the gluster mount logs from the node where paused VM was running (under /var/log/glusterfs/rhev-datarhev-data-center-mnt-glusterSD<mount-path>.log) ? <br> Which version of glusterfs are you running?<br> <br> <div class="moz-cite-prefix">On 06/24/2016 07:49 AM, Bill Bill wrote:<br> </div> <blockquote cite="mid:CO2PR0801MB0743180FC142ED11D7D5D3E0A62E0@CO2PR0801MB0743.namprd08.prod.outlook.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} .MsoChpDefault {mso-style-type:export-only;} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style> <div class="WordSection1"> <p class="MsoNormal">Hello,</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">Have 3 nodes running both oVirt and Gluster on 4 SSDs each. At the moment, there are two physical nics, one has public internet access and the other is a non-routable network used for ovirtmgmt & gluster.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">In the logical networks, I have selected gluster for the nonroutable network running ovirtmgmt and gluster however, two VMs randomly pause for what seems like no reason. They can both be resumed without issue.</p> <p class="MsoNormal"><o:p> </o:p></p> <p class="MsoNormal">One test VM has 4GB of memory and a small disk no problems with this one. Two others have 800GB disks and 32GB of RAM both vms exhibit the same issue.</p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">I also see these in the oVirt dashboard:<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"> <o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Failed to update OVF disks 9e60328d-29af-4533-84f9-633d87f548a7, OVF data isn't updated on those OVF stores (Data Center xxxxx, Storage Domain sr-volume01).<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Jun 23, 2016 9:54:03 PM<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">VDSM command failed: Could not acquire resource. Probably resource factory threw an exception.: ()<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">///////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">VM xxxxx has been paused due to unknown storage error.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">///////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">In the error log on the engine, I see these:<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-7) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has been paused due to unknown storage error.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-11) [10caf93e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM xxxxxx has recovered from paused back to up.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">///////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">Hostnames are all local to /etc/hosts on all servers they also resolve without issue from each host.<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">//////////////<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,611 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,614 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,616 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/sr-volume01' of volume '93e36cdc-ab1b-41ec-ac7f-966cf3856b59' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,618 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,620 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,622 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/distributed' of volume 'b887b05e-2ea6-496e-9552-155d658eeaa6' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,624 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3435:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,626 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3637:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,628 WARN [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturnForXmlRpc] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] Could not associate brick 'ovirt3839:/mnt/data/iso' of volume '89f32457-c8c3-490e-b491-16dd27de0073' with correct network as no gluster network found in cluster '75bd64de-04b2-4a99-9cd0-b63e919b9aca'<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:08:59,629 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-76) [1c1cf4f] FINISH, GlusterVolumesListVDSCommand, return: {b887b05e-2ea6-496e-9552-155d658eeaa6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@d8c9039b, 93e36cdc-ab1b-41ec-ac7f-966cf3856b59=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@fe5ab019, 89f32457-c8c3-490e-b491-16dd27de0073=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@9a56d633}, log id: 485a0611<o:p></o:p></span></p> <p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif">2016-06-23 22:09:04,645 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-89) [] START, GlusterVolumesListVDSCommand(HostName = ovirt3839, GlusterVolumesListVDSParameters:{runAsync='true', hostId='32c500e5-268d-426a-9a4a-108535e67722'}), log id: 41b6479d<o:p></o:p></span></p> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------070800090401040005080909--