[ovirt-users] Vm:s can not be started

[Joop]

On August 15, 2014 3:59:46 PM CEST, Jonas Israelsson <jonas at israelsson.com> wrote:
>I screwed up massively today when accidentally assigned a new virtual 
>network (on my host) the same IP-address as on my iscsi SAN subnet and 
>as a result the whole DC collapsed (I only have one host). Recovering 
>was not quite as easy since the host network configuration only can be 
>managed while the host is either in up, down or maintenance mode, and 
>mine seem to be somewhere in between.
>
>I finally however managed to remove the network from the cluster tab 
>marking the net on the hosts nic with a "?" and when the host came back
>
>online and I could later also be remove the faulty network from the
>nic.
>
>All VM:s now where either in mode pause or down, but I could not get 
>any started to I just powered all of them down.
>
>Now everything DC-wise it up and running again still several VM:s can 
>not be started. Out of about 30 vm;s I have only managed to get one up 
>and running.
>
>The following it written to the engine.log when trying to fire up a 
>non-working VM.
>
>2014-08-15 15:26:23,638 INFO  
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] 
>(DefaultQuartzScheduler_Worker-53) START, DestroyVDSCommand(HostName = 
>ft.elementary.se, HostId = 97c43383-3770-4b0c-810c-a597bb187898, 
>vmId=1b13210a-bf05-41f7-b42e-c4b34ca2214d, force=false,
>secondsToWait=0, 
>gracefully=false), log id: 1aaa6470
>2014-08-15 15:26:34,110 INFO  
>[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] 
>(DefaultQuartzScheduler_Worker-53) FINISH, DestroyVDSCommand, log id: 
>1aaa6470
>2014-08-15 15:26:34,117 INFO  
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
>(DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: 
>null, Custom Event ID: -1, Message: VM moleman is down. Exit message: 
>Requested operation is not valid: domain 'moleman' is already active.
>2014-08-15 15:26:34,118 INFO  
>[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] 
>(DefaultQuartzScheduler_Worker-53) Running on vds during rerun failed 
>vm: null
>2014-08-15 15:26:34,119 INFO  
>[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] 
>(DefaultQuartzScheduler_Worker-53) VM moleman 
>(1b13210a-bf05-41f7-b42e-c4b34ca2214d) is running in db and not running
>
>in VDS ft.elementary.se
>2014-08-15 15:26:34,136 ERROR 
>[org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] 
>(DefaultQuartzScheduler_Worker-53) Rerun vm 
>1b13210a-bf05-41f7-b42e-c4b34ca2214d. Called from vds ft.elementary.se
>2014-08-15 15:26:34,142 INFO  
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
>(org.ovirt.thread.pool-6-thread-44) Correlation ID: 3335521e, Job ID: 
>f334270e-2576-46c7-8c69-99c54d868d57, Call Stack: null, Custom Event
>ID: 
>-1, Message: Failed to run VM moleman on Host ft.elementary.se.
>2014-08-15 15:26:34,150 INFO  [org.ovirt.engine.core.bll.RunVmCommand] 
>(org.ovirt.thread.pool-6-thread-44) Lock Acquired to object EngineLock 
>[exclusiveLocks= key: 1b13210a-bf05-41f7-b42e-c4b34ca2214d value: VM
>, sharedLocks= ]
>2014-08-15 15:26:34,155 INFO  
>[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] 
>(org.ovirt.thread.pool-6-thread-44) START, 
>IsVmDuringInitiatingVDSCommand( vmId = 
>1b13210a-bf05-41f7-b42e-c4b34ca2214d), log id: ee3a648
>2014-08-15 15:26:34,156 INFO  
>[org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] 
>(org.ovirt.thread.pool-6-thread-44) FINISH, 
>IsVmDuringInitiatingVDSCommand, return: false, log id: ee3a648
>2014-08-15 15:26:34,160 WARN  [org.ovirt.engine.core.bll.RunVmCommand] 
>(org.ovirt.thread.pool-6-thread-44) CanDoAction of action RunVm failed.
>
>Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_ALL_HOSTS_FILTERED_OUT
>2014-08-15 15:26:34,161 INFO  [org.ovirt.engine.core.bll.RunVmCommand] 
>(org.ovirt.thread.pool-6-thread-44) Lock freed to object EngineLock 
>[exclusiveLocks= key: 1b13210a-bf05-41f7-b42e-c4b34ca2214d value: VM
>, sharedLocks= ]
>2014-08-15 15:26:34,168 INFO  
>[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] 
>(org.ovirt.thread.pool-6-thread-44) Correlation ID: 3335521e, Job ID: 
>f334270e-2576-46c7-8c69-99c54d868d57, Call Stack: null, Custom Event
>ID: 
>-1, Message: Failed to run VM moleman (User: isrjo).
>
>I was on Ovirt 3.4.0-0.13.rc.el6 and right or wrong I decided to 
>upgrade the engine to a released version and I'm now on 3.4.0-1.el6, it
>
>however made no difference
>
>Any help would be greatly appreciated.
>
>Rgds Jonas
>_______________________________________________
>Users mailing list
>Users at ovirt.org
>http://lists.ovirt.org/mailman/listinfo/users

Sorry for top posting.

Seems that engine thinks the vms are still running, database says so but that vdsm says not. One option might be to edit the database to set all vms to off. Make sure to have a backup and that you know how to restore! 
Maybe someone else might have a better idea.
Did you try irc? 
I know its quiet but you might have a faster reponse there than on this list

Joop
-- 
Sent from my Android tablet with K-9 Mail. Please excuse my brevity.