4:56 a.m.
A large number of my virtual machines are reporting in a paused state but
they are actually up and running. Unfortunately I don't look at this
cluster very often so I don't know exactly when it got into this state, I
believe it may have happened a couple months back when I deployed a change
to /etc/vdsm/vdsm.conf to throttle virtual machines migrations. I deployed
this change via puppet and had puppet cycle the vdsm daemon. All of the
vdsm daemons in the cluster were restarted in a short period of time. Since
this happened some time back all the logs have rotated off.
I have tried restarting vdsmd and libvirtd, but that doesn't appear to
return the virtual machines to an up state. The output of "vdsClient -s 0
getAllVmStats" reports them as paused, "virsh -r list --all" reports them
as running. From the GUI I can click run and the VM will show “Up” in the
GUI and vdsClient. But if I restart vdsm, the VM's revert back to
displaying a paused state. I did find that the *.recovery files in
/var/run/vdsm contain a status of Paused even when the GUI and vdsClient
are reporting it as up. I did try and change the status in this file while
vdsm was down and then start vdsm, but it had no effect and the entry in
the *.recovery file which reverted to paused.
p335
sasS'status'
p336
S'Paused'
Any idea what might have caused the VM's to report as paused and how to
recover from it?
I am running
ovirt-3.3.4-1 and vdsm-4.13.3-4
2:44 p.m.
----- Original Message -----
From: "Thomas Scofield" <tscofield(a)gmail.com>
To: "users" <users(a)ovirt.org>
Sent: Wednesday, September 17, 2014 4:56:15 AM
Subject: [ovirt-users] Virtual Machines reporting as paused
A large number of my virtual machines are reporting in a paused state but
they are actually up and running. Unfortunately I don't look at this cluster
very often so I don't know exactly when it got into this state, I believe it
may have happened a couple months back when I deployed a change to
/etc/vdsm/vdsm.conf to throttle virtual machines migrations. I deployed this
change via puppet and had puppet cycle the vdsm daemon. All of the vdsm
daemons in the cluster were restarted in a short period of time. Since this
happened some time back all the logs have rotated off.
I have tried restarting vdsmd and libvirtd, but that doesn't appear to return
the virtual machines to an up state. The output of "vdsClient -s 0
getAllVmStats" reports them as paused, "virsh -r list --all" reports them
as
running. From the GUI I can click run and the VM will show “Up” in the GUI
and vdsClient. But if I restart vdsm, the VM's revert back to displaying a
paused state. I did find that the *.recovery files in /var/run/vdsm contain
a status of Paused even when the GUI and vdsClient are reporting it as up. I
did try and change the status in this file while vdsm was down and then
start vdsm, but it had no effect and the entry in the *.recovery file which
reverted to paused.
p335
sasS'status'
p336
S'Paused'
Any idea what might have caused the VM's to report as paused and how to
recover from it?
I am running
ovirt-3.3.4-1 and vdsm-4.13.3-4
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
what do you see in the 'events' subtab for the paused vms?
it should state the time and reason.
usually vms move to pause due to storage errors
if you have vm in up that moves to paused, you can attach the vdsm.log for the host that
runs the vm so we could have more info
5:08 p.m.
I don't see any events on the individual VM's that show them going into a
paused state. In the past I have seen the VM's paused due to storage
issues, but the difference here is that they only indicate they are paused.
I did find one event log entry from around the time we did the vdsm
restarts that indicated one host has an issue with accessing a storage
domain
Host vm7 cannot access one of the Storage Domains attached to the Data
Center OVIRT-PODB. Setting Host state to Non-Operational.
Unfortunately all the logs have been rotated off at this point so I can't
provide any details of the event. I have not seen any newly started VM's
transition into this state so recent logs won't provide insight either.
For now my plan is to stop and start the VM's to get them back into a
proper state.
On Wed, Sep 17, 2014 at 7:44 AM, Omer Frenkel <ofrenkel(a)redhat.com> wrote:
----- Original Message -----
> From: "Thomas Scofield" <tscofield(a)gmail.com>
> To: "users" <users(a)ovirt.org>
> Sent: Wednesday, September 17, 2014 4:56:15 AM
> Subject: [ovirt-users] Virtual Machines reporting as paused
>
>
>
> A large number of my virtual machines are reporting in a paused state but
> they are actually up and running. Unfortunately I don't look at this
cluster
> very often so I don't know exactly when it got into this state, I
believe it
> may have happened a couple months back when I deployed a change to
> /etc/vdsm/vdsm.conf to throttle virtual machines migrations. I deployed
this
> change via puppet and had puppet cycle the vdsm daemon. All of the vdsm
> daemons in the cluster were restarted in a short period of time. Since
this
> happened some time back all the logs have rotated off.
>
>
>
> I have tried restarting vdsmd and libvirtd, but that doesn't appear to
return
> the virtual machines to an up state. The output of "vdsClient -s 0
> getAllVmStats" reports them as paused, "virsh -r list --all" reports
them as
> running. From the GUI I can click run and the VM will show “Up” in the
GUI
> and vdsClient. But if I restart vdsm, the VM's revert back to displaying
a
> paused state. I did find that the *.recovery files in /var/run/vdsm
contain
> a status of Paused even when the GUI and vdsClient are reporting it as
up. I
> did try and change the status in this file while vdsm was down and then
> start vdsm, but it had no effect and the entry in the *.recovery file
which
> reverted to paused.
>
>
>
> p335
>
> sasS'status'
>
> p336
>
> S'Paused'
>
>
>
> Any idea what might have caused the VM's to report as paused and how to
> recover from it?
>
>
>
> I am running
>
> ovirt-3.3.4-1 and vdsm-4.13.3-4
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
what do you see in the 'events' subtab for the paused vms?
it should state the time and reason.
usually vms move to pause due to storage errors
if you have vm in up that moves to paused, you can attach the vdsm.log for
the host that runs the vm so we could have more info
3719
days inactive
3719
days old
2 comments
2 participants
participants (2)
-
Omer Frenkel -
Thomas Scofield