On August 15, 2014 3:59:46 PM CEST, Jonas Israelsson <jonas(a)israelsson.com> wrote:
I screwed up massively today when accidentally assigned a new virtual
network (on my host) the same IP-address as on my iscsi SAN subnet and
as a result the whole DC collapsed (I only have one host). Recovering
was not quite as easy since the host network configuration only can be
managed while the host is either in up, down or maintenance mode, and
mine seem to be somewhere in between.
I finally however managed to remove the network from the cluster tab
marking the net on the hosts nic with a "?" and when the host came back
online and I could later also be remove the faulty network from the
nic.
All VM:s now where either in mode pause or down, but I could not get
any started to I just powered all of them down.
Now everything DC-wise it up and running again still several VM:s can
not be started. Out of about 30 vm;s I have only managed to get one up
and running.
The following it written to the engine.log when trying to fire up a
non-working VM.
2014-08-15 15:26:23,638 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-53) START, DestroyVDSCommand(HostName =
ft.elementary.se, HostId = 97c43383-3770-4b0c-810c-a597bb187898,
vmId=1b13210a-bf05-41f7-b42e-c4b34ca2214d, force=false,
secondsToWait=0,
gracefully=false), log id: 1aaa6470
2014-08-15 15:26:34,110 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand]
(DefaultQuartzScheduler_Worker-53) FINISH, DestroyVDSCommand, log id:
1aaa6470
2014-08-15 15:26:34,117 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack:
null, Custom Event ID: -1, Message: VM moleman is down. Exit message:
Requested operation is not valid: domain 'moleman' is already active.
2014-08-15 15:26:34,118 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-53) Running on vds during rerun failed
vm: null
2014-08-15 15:26:34,119 INFO
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-53) VM moleman
(1b13210a-bf05-41f7-b42e-c4b34ca2214d) is running in db and not running
in VDS ft.elementary.se
2014-08-15 15:26:34,136 ERROR
[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-53) Rerun vm
1b13210a-bf05-41f7-b42e-c4b34ca2214d. Called from vds ft.elementary.se
2014-08-15 15:26:34,142 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-44) Correlation ID: 3335521e, Job ID:
f334270e-2576-46c7-8c69-99c54d868d57, Call Stack: null, Custom Event
ID:
-1, Message: Failed to run VM moleman on Host ft.elementary.se.
2014-08-15 15:26:34,150 INFO [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-44) Lock Acquired to object EngineLock
[exclusiveLocks= key: 1b13210a-bf05-41f7-b42e-c4b34ca2214d value: VM
, sharedLocks= ]
2014-08-15 15:26:34,155 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(org.ovirt.thread.pool-6-thread-44) START,
IsVmDuringInitiatingVDSCommand( vmId =
1b13210a-bf05-41f7-b42e-c4b34ca2214d), log id: ee3a648
2014-08-15 15:26:34,156 INFO
[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand]
(org.ovirt.thread.pool-6-thread-44) FINISH,
IsVmDuringInitiatingVDSCommand, return: false, log id: ee3a648
2014-08-15 15:26:34,160 WARN [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-44) CanDoAction of action RunVm failed.
Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT
2014-08-15 15:26:34,161 INFO [org.ovirt.engine.core.bll.RunVmCommand]
(org.ovirt.thread.pool-6-thread-44) Lock freed to object EngineLock
[exclusiveLocks= key: 1b13210a-bf05-41f7-b42e-c4b34ca2214d value: VM
, sharedLocks= ]
2014-08-15 15:26:34,168 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(org.ovirt.thread.pool-6-thread-44) Correlation ID: 3335521e, Job ID:
f334270e-2576-46c7-8c69-99c54d868d57, Call Stack: null, Custom Event
ID:
-1, Message: Failed to run VM moleman (User: isrjo).
I was on Ovirt 3.4.0-0.13.rc.el6 and right or wrong I decided to
upgrade the engine to a released version and I'm now on 3.4.0-1.el6, it
however made no difference
Any help would be greatly appreciated.
Rgds Jonas
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
Sorry for top posting.
Seems that engine thinks the vms are still running, database says so but that vdsm says
not. One option might be to edit the database to set all vms to off. Make sure to have a
backup and that you know how to restore!
Maybe someone else might have a better idea.
Did you try irc?
I know its quiet but you might have a faster reponse there than on this list
Joop
--
Sent from my Android tablet with K-9 Mail. Please excuse my brevity.