<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
trying again with smaller attachments...<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 04/28/2016 10:40 AM, Bill James
wrote:<br>
</div>
<blockquote cite="mid:57224B1F.6070808@j2.com" type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
thank you for response.<br>
I bold-ed the ones that are listed as "paused".<br>
<br>
<br>
[root@ovirt1 test vdsm]# virsh -r list --all<br>
Id Name State<br>
----------------------------------------------------<br>
<b> 2 puppet.test.j2noc.com running</b><br>
<b> 4 sftp2.test.j2noc.com running</b><br>
<b> 5 oct.test.j2noc.com running</b><br>
<b> 6 sftp2.dev.j2noc.com running</b><br>
<b> 10 darmaster1.test.j2noc.com running</b><br>
<b> 14 api1.test.j2noc.com running</b><br>
25 ftp1.frb.test.j2noc.com running<br>
26 auto7.test.j2noc.com running<br>
<b> 32 epaymv02.j2noc.com running</b><br>
34 media2.frb.test.j2noc.com running<br>
36 auto2.j2noc.com running<br>
44 nfs.testhvy2.colo.j2noc.com running<br>
<b> 53 billapp-zuma1.dev.j2noc.com running</b><br>
<b> 54 billing-ci.dev.j2noc.com running</b><br>
60 log2.test.j2noc.com running<br>
63 log1.test.j2noc.com running<br>
<b> 69 sonar.dev.j2noc.com running</b><br>
<b> 73 billapp-ui1.dev.j2noc.com running</b><br>
<b> 74 billappvm01.dev.j2noc.com running</b><br>
75 db2.frb.test.j2noc.com running<br>
83 billapp-ui1.test.j2noc.com running<br>
<b> 84 epayvm01.test.j2noc.com running</b><br>
<b> 87 billappvm01.test.j2noc.com running</b><br>
<b> 89 etapi1.test.j2noc.com running</b><br>
<b> 93 billapp-zuma2.test.j2noc.com running</b><br>
<b> 94 git.dev.j2noc.com running</b><br>
<br>
Yes I did "systemctl restart libvirtd" which apparently also
restart vdsm?<br>
<br>
<br>
Looks like problem started around 2016-04-17 20:19:34,822, based
on engine.log attached.<br>
There's a lot of vdsm logs!<br>
<br>
fyi, the storage domain for these Vms is a "local" nfs share,
7e566f55-e060-47b7-bfa4-ac3c48d70dda.<br>
<br>
attached more logs.<br>
<br>
<br>
<div class="moz-cite-prefix">On 04/28/2016 12:53 AM, Michal
Skrivanek wrote:<br>
</div>
<blockquote
cite="mid:28BF55E6-3A90-4BB7-90B9-1EE0A82FC460@redhat.com"
type="cite">
<blockquote type="cite">
<pre wrap="">On 27 Apr 2016, at 19:16, Bill James <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:bill.james@j2.com"><bill.james@j2.com></a> wrote:
virsh # list --all
error: failed to connect to the hypervisor
error: no valid connection
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
</pre>
</blockquote>
<pre wrap="">you need to run virsh in read-only mode
virsh -r list —all
</pre>
<blockquote type="cite">
<pre wrap="">[root@ovirt1 test vdsm]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
└─unlimited-core.conf
Active: active (running) since Thu 2016-04-21 16:00:03 PDT; 5 days ago
tried systemctl restart libvirtd.
No change.
Attached vdsm.log and supervdsm.log.
[root@ovirt1 test vdsm]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-04-27 10:09:14 PDT; 3min 46s ago
vdsm-4.17.18-0.el7.centos.noarch
</pre>
</blockquote>
<pre wrap="">the vdsm.log attach is good, but it’s too short interval, it only shows recovery(vdsm restart) phase when the VMs are identified as paused….can you add earlier logs? Did you restart vdsm yourself or did it crash?
</pre>
<blockquote type="cite">
<pre wrap="">libvirt-daemon-1.2.17-13.el7_2.4.x86_64
Thanks.
On 04/26/2016 11:35 PM, Michal Skrivanek wrote:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">On 27 Apr 2016, at 02:04, Nir Soffer <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:nsoffer@redhat.com"><nsoffer@redhat.com></a> wrote:
jjOn Wed, Apr 27, 2016 at 2:03 AM, Bill James <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:bill.james@j2.com"><bill.james@j2.com></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I have a hardware node that has 26 VMs.
9 are listed as "running", 17 are listed as "paused".
In truth all VMs are up and running fine.
I tried telling the db they are up:
engine=> update vm_dynamic set status = 1 where vm_guid =(select
vm_guid from vm_static where vm_name = 'api1.test.j2noc.com');
GUI then shows it up for a short while,
then puts it back in paused state.
2016-04-26 15:16:46,095 INFO [org.ovirt.engine.core.vdsbroker.VmAnalyzer]
(DefaultQuartzScheduler_Worker-16) [157cc21e] VM '242ca0af-4ab2-4dd6-b515-5
d435e6452c4'(api1.test.j2noc.com) moved from 'Up' --> 'Paused'
2016-04-26 15:16:46,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditlogh
andling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [157cc21e] Cor
relation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM api1.
test.j2noc.com has been paused.
Why does the engine think the VMs are paused?
Attached engine.log.
I can fix the problem by powering off the VM then starting it back up.
But the VM is working fine! How do I get ovirt to realize that?
</pre>
</blockquote>
<pre wrap="">If this is an issue in engine, restarting engine may fix this.
but having this problem only with one node, I don't think this is the issue.
If this is an issue in vdsm, restarting vdsm may fix this.
If this does not help, maybe this is libvirt issue? did you try to check vm
status using virsh?
</pre>
</blockquote>
<pre wrap="">this looks more likely as it seems such status is being reported
logs would help, vdsm.log at the very least.
</pre>
<blockquote type="cite">
<pre wrap="">If virsh thinks that the vms are paused, you can try to restart libvirtd.
Please file a bug about this in any case with engine and vdsm logs.
Adding Michal in case he has better idea how to proceed.
Nir
</pre>
</blockquote>
</blockquote>
<pre wrap="">
Cloud Services for Business <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="http://www.j2.com">www.j2.com</a>
j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
This email, its contents and attachments contain information from j2 Global, Inc. and/or its affiliates which may be privileged, confidential or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is prohibited. If you have received this email in error please notify the sender by reply e-mail and delete the original message and any copies. (c) 2015 j2 Global, Inc. All rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox are registered trademarks of j2 Global, Inc. and its affiliates.
<supervdsm.log.gz><vdsm.log.gz>_______________________________________________
Users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a>
</pre>
</blockquote>
</blockquote>
<br>
</blockquote>
<br>
</body>
</html>