8:59 a.m.
I screwed up massively today when accidentally assigned a new virtual
network (on my host) the same IP-address as on my iscsi SAN subnet and
as a result the whole DC collapsed (I only have one host). Recovering
was not quite as easy since the host network configuration only can be
managed while the host is either in up, down or maintenance mode, and
mine seem to be somewhere in between.
I finally however managed to remove the network from the cluster tab
marking the net on the hosts nic with a "?" and when the host came back
online and I could later also be remove the faulty network from the nic.
All VM:s now where either in mode pause or down, but I could not get
any started to I just powered all of them down.
Now everything DC-wise it up and running again still several VM:s can
not be started. Out of about 30 vm;s I have only managed to get one up
and running.
The following it written to the engine.log when trying to fire up a
non-working VM.
2014-08-15 15:26:23,638 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-53) START, DestroyVDSCommand(HostName =
ft.elementary.se, HostId = 97c43383-3770-4b0c-810c-a597bb187898,
vmId=1b13210a-bf05-41f7-b42e-c4b34ca2214d, force=false, secondsToWait=0,
gracefully=false), log id: 1aaa6470
2014-08-15 15:26:34,110 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-53) FINISH, DestroyVDSCommand, log id:
1aaa6470
2014-08-15 15:26:34,117 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack:
null, Custom Event ID: -1, Message: VM moleman is down. Exit message:
Requested operation is not valid: domain 'moleman' is already active.
2014-08-15 15:26:34,118 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-53) Running on vds during rerun failed
vm: null
2014-08-15 15:26:34,119 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-53) VM moleman
(1b13210a-bf05-41f7-b42e-c4b34ca2214d) is running in db and not running
in VDS ft.elementary.se
2014-08-15 15:26:34,136 ERROR
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-53) Rerun vm
1b13210a-bf05-41f7-b42e-c4b34ca2214d. Called from vds ft.elementary.se
2014-08-15 15:26:34,142 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-44) Correlation ID: 3335521e, Job ID:
f334270e-2576-46c7-8c69-99c54d868d57, Call Stack: null, Custom Event ID:
-1, Message: Failed to run VM moleman on Host ft.elementary.se.
2014-08-15 15:26:34,150 INFO [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-44) Lock Acquired to object EngineLock
[exclusiveLocks= key: 1b13210a-bf05-41f7-b42e-c4b34ca2214d value: VM
, sharedLocks= ]
2014-08-15 15:26:34,155 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(org.ovirt.thread.pool-6-thread-44) START,
IsVmDuringInitiatingVDSCommand( vmId =
1b13210a-bf05-41f7-b42e-c4b34ca2214d), log id: ee3a648
2014-08-15 15:26:34,156 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(org.ovirt.thread.pool-6-thread-44) FINISH,
IsVmDuringInitiatingVDSCommand, return: false, log id: ee3a648
2014-08-15 15:26:34,160 WARN [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-44) CanDoAction of action RunVm failed.
Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT
2014-08-15 15:26:34,161 INFO [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-44) Lock freed to object EngineLock
[exclusiveLocks= key: 1b13210a-bf05-41f7-b42e-c4b34ca2214d value: VM
, sharedLocks= ]
2014-08-15 15:26:34,168 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-44) Correlation ID: 3335521e, Job ID:
f334270e-2576-46c7-8c69-99c54d868d57, Call Stack: null, Custom Event ID:
-1, Message: Failed to run VM moleman (User: isrjo).
I was on Ovirt 3.4.0-0.13.rc.el6 and right or wrong I decided to
upgrade the engine to a released version and I'm now on 3.4.0-1.el6, it
however made no difference
Any help would be greatly appreciated.
Rgds Jonas