[ovirt-users] Some VMs in status "not responding" in oVirt interface
Christian Hailer
christian at hailer.eu
Sun Aug 30 14:32:04 UTC 2015
Hello Yaniv,
here are the engine logs around the time it happened today, I just rebooted the server and started up the VMs, 15 minutes later the VM “Management” didn’t respond anymore…
Do you want to see the VM’s logs, meaning the logs of the OS which the VM is running? Or are there any oVirt logs for each VM?
Best regards, Christian
2015-08-30 10:58:12,913 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) VM kube-minion3 69d30e11-4fdc-4129-9e17-36ff37e32bfa moved from WaitForLaunch --> PoweringUp
2015-08-30 10:58:12,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-94) START, FullListVdsCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vds=Host[ovirt-en
gine,5af1efa7-5ddd-4df3-ac64-faa101bb505b], vmIds=[69d30e11-4fdc-4129-9e17-36ff37e32bfa]), log id: 70ac29e4
2015-08-30 10:58:12,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-94) FINISH, FullListVdsCommand, return: [{acpiEnable=true, emulatedMachine=rhel6.5.0, vmId=69d30e11-4fdc-4129-9e17-36ff
37e32bfa, memGuaranteedSize=2048, transparentHugePages=true, displaySecurePort=5917, spiceSslCipherSuite=DEFAULT, cpuType=SandyBridge, smp=2, pauseCode=NOERR, smartcardEnable=false, custom={device_82b2d838-86c3-45fa-a5bb-52294d7d35c3device_a6c
48967-806c-4052-bc24-80d098d9330edevice_b92a9519-b2e0-438c-9969-e376edfad414=VmDevice {vmId=69d30e11-4fdc-4129-9e17-36ff37e32bfa, deviceId=b92a9519-b2e0-438c-9969-e376edfad414, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bu
s=0, controller=0, type=virtio-serial, port=2}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null, logicalName=null}, device_82b2d838-86c3-45fa-a5bb-52294d7d35c3device_a6c48967-806c-4052-bc
24-80d098d9330e=VmDevice {vmId=69d30e11-4fdc-4129-9e17-36ff37e32bfa, deviceId=a6c48967-806c-4052-bc24-80d098d9330e, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=1}, managed=false
, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null, logicalName=null}, device_82b2d838-86c3-45fa-a5bb-52294d7d35c3=VmDevice {vmId=69d30e11-4fdc-4129-9e17-36ff37e32bfa, deviceId=82b2d838-86c3-45fa-a5bb-52
294d7d35c3, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null, log
icalName=null}, device_82b2d838-86c3-45fa-a5bb-52294d7d35c3device_a6c48967-806c-4052-bc24-80d098d9330edevice_b92a9519-b2e0-438c-9969-e376edfad414device_ad58a63f-0aa4-4c3e-8232-c9e0f8d4ec4b=VmDevice {vmId=69d30e11-4fdc-4129-9e17-36ff37e32bfa, d
eviceId=ad58a63f-0aa4-4c3e-8232-c9e0f8d4ec4b, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=3}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProper
ties={}, snapshotId=null, logicalName=null}}, vmType=kvm, memSize=2048, smpCoresPerSocket=2, vmName=kube-minion3, nice=0, status=Up, bootMenuEnable=true, pid=11142, copyPasteEnable=true, displayIp=172.20.1.254, displayPort=-1, guestDiskMapping
={}, clientIp=, fileTransferEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, kvmEnable=true, pitReinjection=false, displayNetwork=zw2001, devices=[Ljava.lang.Object;@36f27186, timeOffset=7200, maxVCpus=32, spiceSecureChannels=smain,sinp
uts,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, display=qxl}], log id: 70ac29e4
2015-08-30 10:58:12,923 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) Received a spice Device without an address when processing VM 69d30e11-4fdc-4129-9e17-36ff37e32bfa devices, skipping device
: {device=spice, specParams={displayNetwork=zw2001, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayIp=172.20.1.254, copyPasteEnable=true}, deviceType=graphics, type=graphics, tls
Port=5917}
2015-08-30 10:58:19,143 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) VM kube-minion1 91454162-2bc6-4119-ab5e-2ca77740bb41 moved from WaitForLaunch --> PoweringUp
2015-08-30 10:58:19,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-13) START, FullListVdsCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vds=Host[ovirt-en
gine,5af1efa7-5ddd-4df3-ac64-faa101bb505b], vmIds=[91454162-2bc6-4119-ab5e-2ca77740bb41]), log id: 30059eb4
2015-08-30 10:58:19,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-13) FINISH, FullListVdsCommand, return: [{acpiEnable=true, emulatedMachine=rhel6.5.0, vmId=91454162-2bc6-4119-ab5e-2ca7
7740bb41, memGuaranteedSize=2048, transparentHugePages=true, displaySecurePort=5915, spiceSslCipherSuite=DEFAULT, cpuType=SandyBridge, smp=2, pauseCode=NOERR, smartcardEnable=false, custom={}, vmType=kvm, memSize=2048, smpCoresPerSocket=2, vmN
ame=kube-minion1, nice=0, status=Up, bootMenuEnable=true, pid=10982, copyPasteEnable=true, displayIp=172.20.1.254, displayPort=-1, guestDiskMapping={}, clientIp=, fileTransferEnable=true, nicModel=rtl8139,pv, keyboardLayout=en-us, kvmEnable=tr
ue, pitReinjection=false, displayNetwork=zw2001, devices=[Ljava.lang.Object;@40f6540d, timeOffset=7200, maxVCpus=32, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, display=qxl}], log id: 30059eb4
2015-08-30 10:58:19,151 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) Received a spice Device without an address when processing VM 91454162-2bc6-4119-ab5e-2ca77740bb41 devices, skipping device
: {device=spice, specParams={displayNetwork=zw2001, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, keyMap=en-us, displayIp=172.20.1.254, copyPasteEnable=true}, deviceType=graphics, type=graphics, tls
Port=5915}
2015-08-30 10:58:31,230 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp--127.0.0.1-8702-7) [1a127cd6] Running command: SetVmTicketCommand internal: false. Entities affected : ID: 8f39a435-c73e-40f0-997f-06a5354903df Type: VMAction g
roup CONNECT_TO_VM with role type USER
2015-08-30 10:58:31,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-7) [1a127cd6] START, SetVmTicketVDSCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vmId=8f39a4
35-c73e-40f0-997f-06a5354903df, ticket=5UrH5zlzwfkL, validTime=120,m userName=admin, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 53f23b6f
2015-08-30 10:58:31,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp--127.0.0.1-8702-7) [1a127cd6] FINISH, SetVmTicketVDSCommand, log id: 53f23b6f
2015-08-30 10:58:31,297 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp--127.0.0.1-8702-7) [1a127cd6] Correlation ID: 1a127cd6, Call Stack: null, Custom Event ID: -1, Message: user admin at internal initiated con
sole session for VM Management
2015-08-30 10:58:34,912 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin at internal is connected to V
M Management.
2015-08-30 10:59:06,431 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) VM kube-minion3 69d30e11-4fdc-4129-9e17-36ff37e32bfa moved from PoweringUp --> Up
2015-08-30 10:59:06,431 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) VM kube-master ab5c9d80-6600-4a97-b802-9d2ebdefb328 moved from PoweringUp --> Up
2015-08-30 10:59:06,432 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) VM kube-minion2 0fd6b83f-4a1a-475d-ac4b-70cd9bc3fd04 moved from PoweringUp --> Up
2015-08-30 10:59:06,432 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) VM kube-minion1 91454162-2bc6-4119-ab5e-2ca77740bb41 moved from PoweringUp --> Up
2015-08-30 10:59:06,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: 7c86e5aa, Job ID: a9522464-7da0-4d9a-be36-8b1be7936ad2, Call Stack: null, Custom Event ID:
-1, Message: VM kube-minion3 started on Host ovirt-engine
2015-08-30 10:59:06,446 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: 5027cf37, Job ID: 96394536-0ce9-4867-b1a1-f4b77ff312a9, Call Stack: null, Custom Event ID:
-1, Message: VM kube-master started on Host ovirt-engine
2015-08-30 10:59:06,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: 28fe9ed, Job ID: abf17a63-bf01-45f9-8190-0dc227da1102, Call Stack: null, Custom Event ID: -
1, Message: VM kube-minion2 started on Host ovirt-engine
2015-08-30 10:59:06,451 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) Correlation ID: 3437099e, Job ID: fa75e50c-1e88-46f9-8534-d928528bb591, Call Stack: null, Custom Event ID:
-1, Message: VM kube-minion1 started on Host ovirt-engine
2015-08-30 11:03:41,933 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] Running command: StopVmCommand internal: false. Entities affected : ID: 8f39a435-c73e-40f0-997f-06a5354903df Type: VMAction
group VM_BASIC_OPERATIONS with role type USER
2015-08-30 11:03:41,939 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] START, DestroyVmVDSCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vmId=8f39a435
-c73e-40f0-997f-06a5354903df, force=false, secondsToWait=0, gracefully=false, reason=), log id: 4e24f013
2015-08-30 11:03:41,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] START, DestroyVDSCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vmId=8f
39a435-c73e-40f0-997f-06a5354903df, force=false, secondsToWait=0, gracefully=false, reason=), log id: 3490545d
2015-08-30 11:03:57,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] Failed in DestroyVDS method
2015-08-30 11:03:57,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=42, mMessage=Virtual machine destroy error]]
2015-08-30 11:03:57,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] HostName = ovirt-engine
2015-08-30 11:03:57,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] Command DestroyVDSCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vmId=8
f39a435-c73e-40f0-997f-06a5354903df, force=false, secondsToWait=0, gracefully=false, reason=) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine destroy error,
code = 42
2015-08-30 11:03:57,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] FINISH, DestroyVDSCommand, log id: 3490545d
2015-08-30 11:03:57,089 ERROR [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] VDS::destroy Failed destroying vm 8f39a435-c73e-40f0-997f-06a5354903df in vds = 5af1efa7-5ddd-4df3-ac64-faa101bb
505b : ovirt-engine, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine destroy error, code = 42
2015-08-30 11:03:57,090 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] FINISH, DestroyVmVDSCommand, log id: 4e24f013
2015-08-30 11:03:57,090 ERROR [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] Command org.ovirt.engine.core.bll.StopVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engin
e.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine destroy error, code = 42 (Failed with error destroyErr and code 42)
2015-08-30 11:03:57,100 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-38) [15c6a5e8] Correlation ID: 15c6a5e8, Job ID: 6a9739a1-ba29-4e96-b585-dd85078adef7, Call Stack: null, Custo
m Event ID: -1, Message: Failed to power off VM Management (Host: ovirt-engine, User: admin at internal).
2015-08-30 11:03:57,175 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) VM Management 8f39a435-c73e-40f0-997f-06a5354903df moved from Up --> PoweringDown
2015-08-30 11:03:57,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-40) START, FullListVdsCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vds=Host[ovirt-engine,5af1efa7-5ddd-4df3-ac64-faa101bb505b], vmIds=[8f39a435-c73e-40f0-997f-06a5354903df]), log id: 24a7a72
2015-08-30 11:03:57,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-40) FINISH, FullListVdsCommand, return: [{acpiEnable=true, emulatedMachine=rhel6.5.0, tabletEnable=true, pid=6689, memGuaranteedSize=2048, transparentHugePages=true, displaySecurePort=-1, spiceSslCipherSuite=DEFAULT, cpuType=SandyBridge, smp=2, pauseCode=NOERR, smartcardEnable=false, custom={device_1320a122-342d-4b23-86b3-871279efb8d1device_8d325ee9-0d4b-43eb-8145-7deff09d046d=VmDevice {vmId=8f39a435-c73e-40f0-997f-06a5354903df, deviceId=8d325ee9-0d4b-43eb-8145-7deff09d046d, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=1}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null, logicalName=null}, device_1320a122-342d-4b23-86b3-871279efb8d1device_8d325ee9-0d4b-43eb-8145-7deff09d046ddevice_922bdf56-e17e-418b-8691-8e10295678ae=VmDevice {vmId=8f39a435-c73e-40f0-997f-06a5354903df, deviceId=922bdf56-e17e-418b-8691-8e10295678ae, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={bus=0, controller=0, type=virtio-serial, port=2}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null, logicalName=null}, device_1320a122-342d-4b23-86b3-871279efb8d1=VmDevice {vmId=8f39a435-c73e-40f0-997f-06a5354903df, deviceId=1320a122-342d-4b23-86b3-871279efb8d1, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={slot=0x01, bus=0x00, domain=0x0000, type=pci, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null, logicalName=null}}, vmType=kvm, memSize=2048, smpCoresPerSocket=1, vmName=Management, nice=0, status=Powering down, bootMenuEnable=false, vmId=8f39a435-c73e-40f0-997f-06a5354903df, displayIp=172.20.1.254, displayPort=5903, guestDiskMapping={}, clientIp=172.20.1.100, nicModel=rtl8139,pv, keyboardLayout=de, kvmEnable=true, pitReinjection=false, displayNetwork=zw2001, devices=[Ljava.lang.Object;@67acd38a, timeOffset=7200, maxVCpus=16, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, display=vnc}], log id: 24a7a72
2015-08-30 11:03:57,184 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) Received a vnc Device without an address when processing VM 8f39a435-c73e-40f0-997f-06a5354903df devices, skipping device: {device=vnc, specParams={displayNetwork=zw2001, keyMap=de, displayIp=172.20.1.254, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard}, deviceType=graphics, type=graphics, port=5903}
2015-08-30 11:04:44,047 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) VM Management 8f39a435-c73e-40f0-997f-06a5354903df moved from PoweringDown --> NotResponding
2015-08-30 11:04:44,075 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Management is not responding.
2015-08-30 11:06:58,341 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) VM Management 8f39a435-c73e-40f0-997f-06a5354903df moved from Down --> NotResponding
2015-08-30 11:06:58,367 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Management is not responding.
2015-08-30 11:07:52,016 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-44) [d68985c] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 8f39a435-c73e-40f0-997f-06a5354903df Type: VMAction group VM_BASIC_OPERATIONS with role type USER
2015-08-30 11:07:52,018 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-44) [d68985c] Entered (VM Management).
2015-08-30 11:07:52,018 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-44) [d68985c] Cannot shutdown VM Management, status is not up. Stopping instead.
2015-08-30 11:07:52,042 INFO [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] Running command: StopVmCommand internal: true. Entities affected : ID: 8f39a435-c73e-40f0-997f-06a5354903df Type: VMAction group VM_BASIC_OPERATIONS with role type USER
2015-08-30 11:07:52,044 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] START, DestroyVmVDSCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vmId=8f39a435-c73e-40f0-997f-06a5354903df, force=false, secondsToWait=0, gracefully=false, reason=), log id: 2499bd2d
2015-08-30 11:07:52,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] START, DestroyVDSCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vmId=8f39a435-c73e-40f0-997f-06a5354903df, force=false, secondsToWait=0, gracefully=false, reason=), log id: 32cc331c
2015-08-30 11:08:07,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] Failed in DestroyVDS method
2015-08-30 11:08:07,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=42, mMessage=Virtual machine destroy error]]
2015-08-30 11:08:07,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] HostName = ovirt-engine
2015-08-30 11:08:07,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] Command DestroyVDSCommand(HostName = ovirt-engine, HostId = 5af1efa7-5ddd-4df3-ac64-faa101bb505b, vmId=8f39a435-c73e-40f0-997f-06a5354903df, force=false, secondsToWait=0, gracefully=false, reason=) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine destroy error, code = 42
2015-08-30 11:08:07,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] FINISH, DestroyVDSCommand, log id: 32cc331c
2015-08-30 11:08:07,069 ERROR [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] VDS::destroy Failed destroying vm 8f39a435-c73e-40f0-997f-06a5354903df in vds = 5af1efa7-5ddd-4df3-ac64-faa101bb505b : ovirt-engine, error = org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine destroy error, code = 42
2015-08-30 11:08:07,069 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] FINISH, DestroyVmVDSCommand, log id: 2499bd2d
2015-08-30 11:08:07,069 ERROR [org.ovirt.engine.core.bll.StopVmCommand] (org.ovirt.thread.pool-8-thread-44) [4e84240b] Command org.ovirt.engine.core.bll.StopVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to DestroyVDS, error = Virtual machine destroy error, code = 42 (Failed with error destroyErr and code 42)
2015-08-30 11:08:07,077 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-44) [4e84240b] Correlation ID: 4e84240b, Job ID: fe465716-fba8-4510-beb5-a1f8f1859c76, Call Stack: null, Custom Event ID: -1, Message: Failed to power off VM Management (Host: ovirt-engine, User: admin at internal).
2015-08-30 11:08:28,992 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) VM Management 8f39a435-c73e-40f0-997f-06a5354903df moved from Down --> NotResponding
2015-08-30 11:08:29,018 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Management is not responding.
2015-08-30 11:08:51,830 INFO [org.ovirt.engine.core.bll.ShutdownVmCommand] (org.ovirt.thread.pool-8-thread-16) [57cf4a46] Running command: ShutdownVmCommand internal: false. Entities affected : ID: 8f39a435-c73e-40f0-997f-06a5354903df Type: VMAction group VM_BASIC_OPERATIONS with role type USER
2015-08-30 11:08:51,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-8-thread-16) [57cf4a46] Correlation ID: 57cf4a46, Job ID: 5f3ecbf6-4a84-41ac-afce-2268f4eec8f4, Call Stack: null, Custom Event ID: -1, Message: VM Management is already Down, Shutdown VM was skipped. User: admin at internal.
2015-08-30 11:09:00,212 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) VM Management 8f39a435-c73e-40f0-997f-06a5354903df moved from Down --> NotResponding
2015-08-30 11:09:00,217 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Management is not responding.
2015-08-30 11:09:15,787 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) VM Management 8f39a435-c73e-40f0-997f-06a5354903df moved from Down --> NotResponding
2015-08-30 11:09:15,792 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM Management is not responding.
Von: Yaniv Dary [mailto:ydary at redhat.com]
Gesendet: Sonntag, 30. August 2015 13:20
An: Christian Hailer <christian at hailer.eu>
Cc: users <users at ovirt.org>
Betreff: Re: [ovirt-users] Some VMs in status "not responding" in oVirt interface
First thing is to attach the affected hosts logs and the engine logs.
Yaniv Dary
Technical Product Manager
Red Hat Israel Ltd.
34 Jerusalem Road
Building A, 4th floor
Ra'anana, Israel 4350109
Tel : +972 (9) 7692306
8272306
Email: <mailto:ydary at redhat.com> ydary at redhat.com
IRC : ydary
On Sat, Aug 29, 2015 at 11:47 PM, Christian Hailer < <mailto:christian at hailer.eu> christian at hailer.eu> wrote:
Hello,
last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a single Centos 7 server, so I started by suspending the VMs in order to set the oVirt engine host to maintenance mode. During the process of suspending the VMs the server crashed, kernel panic…
After restarting the server I installed the updates via yum an restarted the server again. Afterwards, all the VMs could be started again. Some hours later my monitoring system registered some unresponsive hosts, I had a look in the oVirt interface, 3 of the VMs were in the state “not responding”, marked by a question mark.
I tried to shut down the VMs, but oVirt wasn’t able to do so. I tried to reset the status in the database with the sql statement
update vm_dynamic set status = 0 where vm_guid = (select vm_guid from vm_static where vm_name = 'MYVMNAME');
but that didn’t help, either. Only rebooting the whole hypervisor helped… afterwards everything worked again. But only for a few hours, then one of the VMs entered the “not responding” state again… again only a reboot helped. Yesterday it happened again:
2015-08-28 17:44:22,664 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding
2015-08-28 17:44:22,692 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM DC is not responding.
Does anybody know what I can do? Where should I have a look? Hints are greatly appreciated!
Thanks,
Christian
_______________________________________________
Users mailing list
<mailto:Users at ovirt.org> Users at ovirt.org
<http://lists.ovirt.org/mailman/listinfo/users> http://lists.ovirt.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20150830/ba882626/attachment-0001.html>
More information about the Users
mailing list