[Users] Vm's being paused

Tue Jan 21 09:51:59 UTC 2014

Hi Dan,

Sorry, attached is engine.log I've taken out the two sections where
each of the VM's were paused.

Does the error "VM babbage has paused due to no Storage space error"
mean the main storage domain has run out of storage, or that the VM
has run out?

Both VM's appear to have been running on node01 when they were paused.
My vdsm versions are all...

vdsm-cli-4.13.0-11.el6.noarch
vdsm-python-cpopen-4.13.0-11.el6.x86_64
vdsm-xmlrpc-4.13.0-11.el6.noarch
vdsm-4.13.0-11.el6.x86_64
vdsm-python-4.13.0-11.el6.x86_64

I currently have a 61% over allocation ratio on my primary storage
domain, with 1948GB available.

Thank you.

Regards.

Neil Wilson.

On Tue, Jan 21, 2014 at 11:24 AM, Neil <nwilson123 at gmail.com> wrote:
> Hi Dan,
>
> Sorry for only coming back to you now.
> The VM's are thin provisioned. The Server 2003 VM hasn't run out of
> disk space there is about 20Gigs free, and the usage barely grows as
> the VM only shares printers. The other VM that paused is also on thin
> provisioned disks and also has plenty space, this guest is running
> Centos 6.3 64bit and only runs basic reporting.
>
> After the 2003 guest was rebooted, the network card showed up as
> unplugged in ovirt, and we had to remove it, and re-add it again in
> order to correct the issue. The Centos VM did not have the same issue.
>
> I'm concerned that this might happen to a VM that's quite critical,
> any thoughts or ideas?
>
> The only recent changes have been updating from Dreyou 3.2 to the
> official Centos repo and updating to 3.3.1-2. Prior to updating I
> haven't had this issue.
>
> Any assistance is greatly appreciated.
>
> Thank you.
>
> Regards.
>
> Neil Wilson.
>
>
>
>
>
>
>
>
>
>
>
> On Sun, Jan 19, 2014 at 8:20 PM, Dan Yasny <dyasny at gmail.com> wrote:
>> Do you have the VMs on thin provisioned storage or sparse disks?
>>
>> Pausing happens when the VM has an IO error or runs out of space on the
>> storage domain, and it is done intentionally, so that the VM will not
>> experience a disk corruption. If you have thin provisioned disks, and the VM
>> writes to it's disks faster than the disks can grow, this is exactly what
>> you will see
>>
>>
>> On Sun, Jan 19, 2014 at 10:04 AM, Neil <nwilson123 at gmail.com> wrote:
>>>
>>> Hi guys,
>>>
>>> I've had two different Vm's randomly pause this past week and inside ovirt
>>> the error received is something like 'vm ran out of storage and was paused'.
>>> Resuming the vm's didn't work and I had to force them off and then on which
>>> resolved the issue.
>>>
>>> Has anyone had this issue before?
>>>
>>> I realise this is very vague so if you could please let me know which logs
>>> to send in.
>>>
>>> Thank you
>>>
>>> Regards.
>>>
>>> Neil Wilson
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
-------------- next part --------------
2014-01-17 17:04:46,092 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) VM babbage 502170aa-0fc6-4287-bb08-5844be6e0352 moved from Up --> RebootInProgress
2014-01-17 17:05:46,738 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) VM babbage 502170aa-0fc6-4287-bb08-5844be6e0352 moved from RebootInProgress --> Up
2014-01-18 16:20:51,921 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) VM babbage 502170aa-0fc6-4287-bb08-5844be6e0352 moved from Up --> RebootInProgress
2014-01-18 16:21:50,915 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) VM babbage 502170aa-0fc6-4287-bb08-5844be6e0352 moved from RebootInProgress --> Up
2014-01-19 16:01:00,551 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) VM babbage 502170aa-0fc6-4287-bb08-5844be6e0352 moved from Up --> Paused
2014-01-19 16:01:01,870 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM babbage has paused due to no Storage space error.
2014-01-19 17:12:41,146 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-6-thread-48) [34bfecb1] Correlation ID: 34bfecb1, Job ID: 90aec901-2da1-4d71-b1f5-ac9cc16427fa, Call Stack: null, Custom Event ID: -1, Message: VM babbage was resumed by admin at internal (Host: node01.blabla.com).
2014-01-19 17:13:22,653 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-5) Correlation ID: 80ef243, Call Stack: null, Custom Event ID: -1, Message: user admin at internal initiated console session for VM babbage
2014-01-19 17:14:30,222 INFO  [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-6-thread-48) [6a6d702a] Entered (VM babbage).
2014-01-19 17:14:30,223 INFO  [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-6-thread-48) [6a6d702a] Cannot shutdown VM babbage, status is not up. Stopping instead.
2014-01-19 17:14:31,786 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-6-thread-48) Correlation ID: 4c8a7e16, Call Stack: null, Custom Event ID: -1, Message: VM babbage was powered off ungracefully by admin at internal (Host: node01.blabla.com).

2014-01-07 12:05:25,963 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) Handing over VM reports ff9036fb-1499-45e4-8cde-e350eee3c489 to Host 06c7ee7d-2f1a-46c8-b942-9052d7cfb871. Setting VM to status MigratingTo
2014-01-07 12:05:25,964 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) vm reports running in db and not running in vds - add to rerun treatment. vds node01.blabla.com
2014-01-07 12:05:29,280 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) VM reports ff9036fb-1499-45e4-8cde-e350eee3c489 moved from MigratingTo --> Up
2014-01-12 23:01:08,774 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) VM reports ff9036fb-1499-45e4-8cde-e350eee3c489 moved from Up --> Paused
2014-01-12 23:01:08,928 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM reports has paused due to no Storage space error.
2014-01-13 10:37:38,737 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-6-thread-50) [59a4355f] Correlation ID: 59a4355f, Job ID: e793e365-5750-483b-999a-b692c4a4b5ea, Call Stack: null, Custom Event ID: -1, Message: VM reports was resumed by admin at internal (Host: node02.blabla.com).
2014-01-13 10:39:14,383 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-15) Correlation ID: 275da3a3, Call Stack: null, Custom Event ID: -1, Message: user admin at internal initiated console session for VM reports
2014-01-13 10:39:59,002 INFO  [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-6-thread-47) [4eafc68a] Entered (VM reports).
2014-01-13 10:39:59,003 INFO  [org.ovirt.engine.core.bll.ShutdownVmCommand] (pool-6-thread-47) [4eafc68a] Cannot shutdown VM reports, status is not up. Stopping instead.
2014-01-13 10:40:01,851 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-6-thread-47) Correlation ID: 6077a98f, Call Stack: null, Custom Event ID: -1, Message: VM reports was powered off ungracefully by admin at internal (Host: node02.blabla.com).
2014-01-13 10:40:12,930 INFO  [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-6-thread-47) [4159698f] START, CreateVmVDSCommand(HostName = node01..com, HostId = 0567d583-1aa8-49b4-bb62-16e426963318, vmId=ff9036fb-1499-45e4-8cde-e350eee3c489, vm=VM [reports]), log id: ec54b4d
2014-01-13 10:40:13,094 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-6-thread-47) [4159698f] START, CreateVDSCommand(HostName = node01.blabla.com, HostId = 0567d583-1aa8-49b4-bb62-16e426963318, vmId=ff9036fb-1499-45e4-8cde-e350eee3c489, vm=VM [reports]), log id: 463bd4b4
2014-01-13 10:40:13,202 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-6-thread-47) [4159698f] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=4096,kvmEnable=true,smp=2,vmType=kvm,emulatedMachine=rhel6.4.0,keyboardLayout=en-us,memGuaranteedSize=3064,pitReinjection=false,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=2,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=6,transparentHugePages=true,vmId=ff9036fb-1499-45e4-8cde-e350eee3c489,devices=[Ljava.util.HashMap;@5806d61d,acpiEnable=true,vmName=reports,cpuType=Westmere,custom={}
2014-01-13 10:40:13,355 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-6-thread-47) [4159698f] Correlation ID: 4159698f, Job ID: 4401b262-6060-4b0e-a97c-b37f9a9d80b1, Call Stack: null, Custom Event ID: -1, Message: VM reports was started by admin at internal (Host: node01.blabla.com).
2014-01-13 10:40:15,123 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) VM reports ff9036fb-1499-45e4-8cde-e350eee3c489 moved from WaitForLaunch --> PoweringUp
2014-01-13 10:40:23,417 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-4) Correlation ID: 7989187, Call Stack: null, Custom Event ID: -1, Message: user admin at internal initiated console session for VM reports
2014-01-13 10:41:14,675 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) VM reports ff9036fb-1499-45e4-8cde-e350eee3c489 moved from PoweringUp --> Up
2014-01-13 10:41:14,701 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) Correlation ID: 4159698f, Job ID: 4401b262-6060-4b0e-a97c-b37f9a9d80b1, Call Stack: null, Custom Event ID: -1, Message: VM reports started on Host node01.blabla.com