- Users - oVirt List Archives

[Users] ovirt deploy host reports unique id which already registered
by Mohsen Saeedi 28 Apr '13

28 Apr '13

2 1

[Users] Redundant management node
by Mohsen Saeedi 28 Apr '13

28 Apr '13

2 3

[Users] FreeIPA
by Ryan Wilkinson 28 Apr '13

28 Apr '13

Getting this error when I try to configure ldap authentication for Ovirt with FreeIPA server: Error: exception message: freeipa.wilk.local. Failure while testing domain wilk.local. Details: Kerberos error. Please check log for further details. Engine-manage-domains.log gives no further details. When I run "engine-manage-domains -action=add -domain='wilk.local' -user='admin' -provider=IPA -interactive" it is connecting and asking for the password but then giving the error. Any input would be appreciated.

2 3

[Users] Fwd: Failed to install Host Unsupported network manager configuration.
by Jairo Llopis 28 Apr '13

28 Apr '13

Hi everyone. After installing ovirt-engine with the all-in-one plugin, the local_host is not correctly working. Clicking on it raises this error: "Host installation failed. Fix installation issues and try to Re-Install". If I click on Re-Install, i get the following error: "Failed to install Host local_host. Failed to execute stage 'Misc configuration': Unsupported network manager configuration." Additional info: The host is a CentOS 6.4 machine, using the repo http://resources.ovirt.org/releases/ovirt-release-el.noarch.rpm. I tried everything I can, and I searched everywhere I know, and I found no solution. Can anyone help me? Thanks.

2 1

[Users] how to use ovirt-shell?
by bigclouds 28 Apr '13

28 Apr '13

------=_Part_276862_142504796.1366767170373 Content-Type: text/plain; charset=GBK Content-Transfer-Encoding: 7bit hi,all ovirt-shell -l https://127.0.0.1/ -u admin@internal -A /etc/pki/vdsm/certs/cacert.pem -C /etc/pki/vdsm/libvirt-spice/server-cert.pem -K /etc/pki/vdsm/libvirt-spice/server-key.pem -c i never successfully connect. thanks ------=_Part_276862_142504796.1366767170373 Content-Type: text/html; charset=GBK Content-Transfer-Encoding: 7bit <div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial"><div>hi,all</div><div> </div><div>ovirt-shell  -l  <a href="https://127.0.0.1/">https://127.0.0.1/</a> -u <a href="mailto:admin@internal">admin@internal</a> -A /etc/pki/vdsm/certs/cacert.pem -C /etc/pki/vdsm/libvirt-spice/server-cert.pem -K /etc/pki/vdsm/libvirt-spice/server-key.pem -c</div><div> </div><div>i never successfully connect.</div><div> </div><div>thanks<br></div></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span> ------=_Part_276862_142504796.1366767170373--

3 5

[Users] oVirt support for backup/restore
by Ricky Schneberger 28 Apr '13

28 Apr '13

Hi, How can I do if I want to schedule a daily live-snapshot of a VM for backup purpose? I have backup software inside my VMs (Netvault), but in some cases I just want to do daily snapshots and use them as backups. -- Ricky Schneberger

6 12

[Users] oVirt 3.2.1 - VM in unknown status
by Piotr Szubiakowski 28 Apr '13

28 Apr '13

Hi, I had a nfs storage problems and one of my VM's is now in unknown state. The VM is not attached to any host. The virsh tool shows that VM is not running now. Is there any way to bring that kind of VM back to life? Best regards, Piotr

2 4

[Users] oVirt 3.2.1 and clone from snapshot.... argh...!!!
by Gianluca Cecchi 28 Apr '13

28 Apr '13

Hello, in my opinion 3.2.1 for fedora 18 doesn't contain this: http://gerrit.ovirt.org/#/c/11254/ that was one of the biggest problems of 3.2 see also this thread of mine: http://lists.ovirt.org/pipermail/users/2013-February/012653.html I had to recompile 3.2 in the mean time and after update to 3.2.1 I have again the same problems: - clone from snapshot blocked white window - snapshot details pane no information for network and disks So this update was totally worthless for me and I have to recompile again 3.2.1.... bad! Gianluca

5 13

Re: [Users] Unable to activate data center after host reboot.
by Steven Wiltshire 28 Apr '13

28 Apr '13

Hi, I am using NFS storage, although it is local. Looking in to the NFS problems further, I read the NFS troubleshooting section of the oVirt wiki and made a few changes. nfs-check.py gives me the following: ------------------------------------------------------------------ Current hostname: bigserver - IP addr 10.0.0.154 Trying to /bin/mount -t nfs 10.0.0.154:/home/vm/data/data... Executing NFS tests.. Removing vdsmTest file.. Status of tests [OK] Disconnecting from NFS Server.. Done! ------------------------------------------------------------------ Success! Or so it seems, because when the machine boots there are no NFS shares mounted. When I attempt to activate the data centre in the oVirt web admin tool, the activation fails but an NFS share is mounted: ------------------------------------------------------------------ 10.0.0.154:/home/vm/data/data on /rhev/data-center/mnt/10.0.0.154:_home_vm_data_data type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,mountaddr=10.0.0.154,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.0.0.154) ------------------------------------------------------------------ Here is the engine.log associated with trying to activate the data centre: ------------------------------------------------------------------ 2013-04-27 22:45:01,229 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-44) Lock Acquired to object EngineLock [exclusiveLocks= key: org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand val ue: 025062dd-d69e-40ac-9031-c080b51647ae , sharedLocks= ] 2013-04-27 22:45:01,231 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-44) Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 025062dd-d69e-40ac-9031-c080b51647a e Type: Storage 2013-04-27 22:45:01,243 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-44) Lock freed to object EngineLock [exclusiveLocks= key: org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand value: 025062dd-d69e-40ac-9031-c080b51647ae , sharedLocks= ] 2013-04-27 22:45:01,244 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-44) ActivateStorage Domain. Before Connect all hosts to pool. Time:4/27/13 10:45 PM 2013-04-27 22:45:01,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thread-50) START, ValidateStorageServerConnectionVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storage PoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 60e73855-daa2-45f5-9c59-5973ca89e830, connection: 10.0.0.154:/home/vm/data/data };]), log id: 27174693 2013-04-27 22:45:01,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thread-50) FINISH, ValidateStorageServerConnectionVDSCommand, return: {60e73855-daa2-45f5-9c59-5973ca89e830=0}, l og id: 27174693 2013-04-27 22:45:01,402 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-50) Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Typ e: System 2013-04-27 22:45:01,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-50) START, ConnectStorageServerVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 00000000-0000 -0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 60e73855-daa2-45f5-9c59-5973ca89e830, connection: 10.0.0.154:/home/vm/data/data };]), log id: 397edfdf 2013-04-27 22:45:01,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-50) FINISH, ConnectStorageServerVDSCommand, return: {60e73855-daa2-45f5-9c59-5973ca89e830=0}, log id: 397edfdf 2013-04-27 22:45:01,450 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-44) ActivateStorage Domain. After Connect all hosts to pool. Time:4/27/13 10:45 PM 2013-04-27 22:45:01,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-5-thread-44) START, ActivateStorageDomainVDSCommand(storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 025062dd-d69e-40ac-9031-c080b51647ae), log id: 4a58fee6 2013-04-27 22:45:30,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 304 mMessage Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 22:45:30,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-95) Vds: BigServer 2013-04-27 22:45:30,062 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-95) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 22:45:30,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-95) FINISH, ConnectStoragePoolVDSCommand, log id: 1891eb24 2013-04-27 22:45:30,064 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (QuartzScheduler_Worker-95) transaction rolled back 2013-04-27 22:45:30,064 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) IrsBroker::Failed::GetStoragePoolInfoVDS 2013-04-27 22:45:30,065 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-95) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 22:45:30,086 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-95) CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2 2013-04-27 22:45:30,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-44) hostFromVds::selectedVds - BigServer, spmStatus Unknown_Pool, storage pool Default 2013-04-27 22:45:30,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-5-thread-44) START, ConnectStoragePoolVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 460d5550 2013-04-27 22:45:43,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) starting ProcessDomainRecovery for domain 503e6677-2d51-4668-ab17-f914bdb57c19 2013-04-27 22:45:43,327 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-7) domain 503e6677-2d51-4668-ab17-f914bdb57c19 was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. 2013-04-27 22:45:43,335 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-7) Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: 503e6677-2d51-4668-ab17-f914bdb57c19 Type: Storage 2013-04-27 22:45:43,344 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-7) CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL 2013-04-27 22:45:43,349 ERROR [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-7) Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand. 2013-04-27 22:45:43,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-7) No string for UNASSIGNED type. Use default Log 2013-04-27 22:45:43,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) starting ProcessDomainRecovery for domain d30604e3-3ca8-470a-8da4-18c9402f7504 2013-04-27 22:45:43,360 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-8) domain d30604e3-3ca8-470a-8da4-18c9402f7504 was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. 2013-04-27 22:45:43,368 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-8) Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: d30604e3-3ca8-470a-8da4-18c9402f7504 Type: Storage 2013-04-27 22:45:43,376 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-8) CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL 2013-04-27 22:45:43,381 ERROR [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-8) Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand. 2013-04-27 22:45:43,381 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-8) No string for UNASSIGNED type. Use default Log 2013-04-27 22:45:48,445 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-23) domain 503e6677-2d51-4668-ab17-f914bdb57c19 in problem. vds: BigServer 2013-04-27 22:46:34,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-44) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 304 mMessage Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 22:46:34,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-44) Vds: BigServer 2013-04-27 22:46:34,040 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-44) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 22:46:34,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-5-thread-44) FINISH, ConnectStoragePoolVDSCommand, log id: 460d5550 2013-04-27 22:46:34,042 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (pool-5-thread-44) transaction rolled back 2013-04-27 22:46:34,042 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-44) IrsBroker::Failed::ActivateStorageDomainVDS 2013-04-27 22:46:34,043 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-44) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 22:46:34,058 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-5-thread-44) CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2 2013-04-27 22:46:34,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-5-thread-44) FINISH, ActivateStorageDomainVDSCommand, log id: 4a58fee6 2013-04-27 22:46:34,059 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-44) Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 22:46:34,067 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-44) Command [id=7fe48e9f-b3c3-455e-a869-7cde583084cf]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.storage_pool_iso_map; snapshot: EntityStatusSnapshot [id=storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, storageId = 025062dd-d69e-40ac-9031-c080b51647ae, status=InActive]. 2013-04-27 22:46:34,094 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-93) hostFromVds::selectedVds - BigServer, spmStatus Unknown_Pool, storage pool Default 2013-04-27 22:46:34,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-93) START, ConnectStoragePoolVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 9632afe ------------------------------------------------------------------ Here is vdsm.log for the same time period: ------------------------------------------------------------------ Thread-309::DEBUG::2013-04-27 22:45:01,297::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-309::DEBUG::2013-04-27 22:45:01,298::task::588::TaskManager.Task::(_updateState) Task=`da5d1ac1-8b0e-4a9a-bd28-24c30125654a`::moving from state init -> state preparing Thread-309::INFO::2013-04-27 22:45:01,299::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.0.0.154:/home/vm/data/d ata', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '60e73855-daa2-45f5-9c59-5973ca89e830', 'port': ''}], options=None) Thread-309::INFO::2013-04-27 22:45:01,300::storage_connection::188::Storage.ServerConnection::(validate) Request to validate NFS storage server Thread-309::DEBUG::2013-04-27 22:45:01,301::mount::111::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 10.0.0.154:/home/vm/data/data /tmp/tmpXShfR0' (cwd None) Thread-309::DEBUG::2013-04-27 22:45:01,348::mount::111::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount /tmp/tmpXShfR0' (cwd None) Thread-309::INFO::2013-04-27 22:45:01,394::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-309::DEBUG::2013-04-27 22:45:01,395::task::1174::TaskManager.Task::(prepare) Task=`da5d1ac1-8b0e-4a9a-bd28-24c30125654a`::finished: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-309::DEBUG::2013-04-27 22:45:01,396::task::588::TaskManager.Task::(_updateState) Task=`da5d1ac1-8b0e-4a9a-bd28-24c30125654a`::moving from state preparing -> state finished Thread-309::DEBUG::2013-04-27 22:45:01,397::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-309::DEBUG::2013-04-27 22:45:01,397::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-309::DEBUG::2013-04-27 22:45:01,398::task::980::TaskManager.Task::(_decref) Task=`da5d1ac1-8b0e-4a9a-bd28-24c30125654a`::ref 0 aborting False Thread-310::DEBUG::2013-04-27 22:45:01,424::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-310::DEBUG::2013-04-27 22:45:01,425::task::588::TaskManager.Task::(_updateState) Task=`6e1d398a-02dc-423d-b500-6f7f8ccb20e7`::moving from state init -> state preparing Thread-310::INFO::2013-04-27 22:45:01,425::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.0.0.154:/home/vm/data/data', 'iqn' : '', 'portal': '', 'user': '', 'password': '******', 'id': '60e73855-daa2-45f5-9c59-5973ca89e830', 'port': ''}], options=None) Thread-310::INFO::2013-04-27 22:45:01,426::storage_connection::146::Storage.ServerConnection::(connect) Request to connect NFS storage server Thread-310::DEBUG::2013-04-27 22:45:01,438::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-310::DEBUG::2013-04-27 22:45:01,438::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-310::DEBUG::2013-04-27 22:45:01,439::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-310::DEBUG::2013-04-27 22:45:01,440::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-310::DEBUG::2013-04-27 22:45:01,440::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-310::DEBUG::2013-04-27 22:45:01,441::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-310::INFO::2013-04-27 22:45:01,442::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-310::DEBUG::2013-04-27 22:45:01,442::task::1174::TaskManager.Task::(prepare) Task=`6e1d398a-02dc-423d-b500-6f7f8ccb20e7`::finished: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-310::DEBUG::2013-04-27 22:45:01,443::task::588::TaskManager.Task::(_updateState) Task=`6e1d398a-02dc-423d-b500-6f7f8ccb20e7`::moving from state preparing -> state finished Thread-310::DEBUG::2013-04-27 22:45:01,443::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-310::DEBUG::2013-04-27 22:45:01,444::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-310::DEBUG::2013-04-27 22:45:01,444::task::980::TaskManager.Task::(_decref) Task=`6e1d398a-02dc-423d-b500-6f7f8ccb20e7`::ref 0 aborting False Thread-314::DEBUG::2013-04-27 22:45:07,704::task::588::TaskManager.Task::(_updateState) Task=`f43264f4-0160-4737-8e1a-1322b35cfe3f`::moving from state init -> state preparing Thread-314::INFO::2013-04-27 22:45:07,705::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-314::INFO::2013-04-27 22:45:07,705::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-314::DEBUG::2013-04-27 22:45:07,706::task::1174::TaskManager.Task::(prepare) Task=`f43264f4-0160-4737-8e1a-1322b35cfe3f`::finished: {} Thread-314::DEBUG::2013-04-27 22:45:07,706::task::588::TaskManager.Task::(_updateState) Task=`f43264f4-0160-4737-8e1a-1322b35cfe3f`::moving from state preparing -> state finished Thread-314::DEBUG::2013-04-27 22:45:07,707::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-314::DEBUG::2013-04-27 22:45:07,708::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-314::DEBUG::2013-04-27 22:45:07,708::task::980::TaskManager.Task::(_decref) Task=`f43264f4-0160-4737-8e1a-1322b35cfe3f`::ref 0 aborting False Thread-320::DEBUG::2013-04-27 22:45:17,899::task::588::TaskManager.Task::(_updateState) Task=`b3c57791-d573-4dc5-99ef-9f17578a54d2`::moving from state init -> state preparing Thread-320::INFO::2013-04-27 22:45:17,900::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-320::INFO::2013-04-27 22:45:17,901::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-320::DEBUG::2013-04-27 22:45:17,901::task::1174::TaskManager.Task::(prepare) Task=`b3c57791-d573-4dc5-99ef-9f17578a54d2`::finished: {} Thread-320::DEBUG::2013-04-27 22:45:17,902::task::588::TaskManager.Task::(_updateState) Task=`b3c57791-d573-4dc5-99ef-9f17578a54d2`::moving from state preparing -> state finished Thread-320::DEBUG::2013-04-27 22:45:17,902::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-320::DEBUG::2013-04-27 22:45:17,903::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-320::DEBUG::2013-04-27 22:45:17,904::task::980::TaskManager.Task::(_decref) Task=`b3c57791-d573-4dc5-99ef-9f17578a54d2`::ref 0 aborting False Thread-326::DEBUG::2013-04-27 22:45:28,062::task::588::TaskManager.Task::(_updateState) Task=`4cec58af-afca-4003-a3d0-789ed881116f`::moving from state init -> state preparing Thread-326::INFO::2013-04-27 22:45:28,063::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-326::INFO::2013-04-27 22:45:28,063::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-326::DEBUG::2013-04-27 22:45:28,064::task::1174::TaskManager.Task::(prepare) Task=`4cec58af-afca-4003-a3d0-789ed881116f`::finished: {} Thread-326::DEBUG::2013-04-27 22:45:28,065::task::588::TaskManager.Task::(_updateState) Task=`4cec58af-afca-4003-a3d0-789ed881116f`::moving from state preparing -> state finished Thread-326::DEBUG::2013-04-27 22:45:28,065::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-326::DEBUG::2013-04-27 22:45:28,066::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-326::DEBUG::2013-04-27 22:45:28,066::task::980::TaskManager.Task::(_decref) Task=`4cec58af-afca-4003-a3d0-789ed881116f`::ref 0 aborting False Thread-286::ERROR::2013-04-27 22:45:30,046::sdc::113::Storage.StorageDomainCache::(_findDomain) Error while looking for domain `025062dd-d69e-40ac-9031-c080b51647ae` Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 109, in _findDomain return mod.findDomain(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 149, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 104, in __init__ sdUUID = metadata[sd.DMDK_SDUUID] File "/usr/share/vdsm/storage/persistentDict.py", line 75, in __getitem__ return dec(self._dict[key]) File "/usr/share/vdsm/storage/persistentDict.py", line 183, in __getitem__ with self._accessWrapper(): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/share/vdsm/storage/persistentDict.py", line 137, in _accessWrapper self.refresh() File "/usr/share/vdsm/storage/persistentDict.py", line 214, in refresh lines = self._metaRW.readlines() File "/usr/share/vdsm/storage/fileSD.py", line 71, in readlines return misc.stripNewLines(self._oop.directReadLines(self._metafile)) File "/usr/share/vdsm/storage/processPool.py", line 53, in wrapper return self.runExternally(func, *args, **kwds) File "/usr/share/vdsm/storage/processPool.py", line 64, in runExternally return self._procPool.runExternally(*args, **kwargs) File "/usr/share/vdsm/storage/processPool.py", line 145, in runExternally raise Timeout("Operation Stuck") Timeout: Operation Stuck Thread-286::DEBUG::2013-04-27 22:45:30,048::resourceManager::535::ResourceManager::(releaseResource) Trying to release resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' Thread-286::DEBUG::2013-04-27 22:45:30,049::resourceManager::550::ResourceManager::(releaseResource) Released resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' (0 active users) Thread-286::DEBUG::2013-04-27 22:45:30,049::resourceManager::555::ResourceManager::(releaseResource) Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free, finding out if anyone is waiting for it. Thread-286::DEBUG::2013-04-27 22:45:30,050::resourceManager::562::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84', Clearing records. Thread-286::ERROR::2013-04-27 22:45:30,051::task::855::TaskManager.Task::(_setError) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 863, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 721, in connectStoragePool return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) File "/usr/share/vdsm/storage/hsm.py", line 763, in _connectStoragePool res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 624, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1097, in __rebuild self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1435, in getMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' Thread-286::DEBUG::2013-04-27 22:45:30,052::task::874::TaskManager.Task::(_run) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::Task._run: a7d4f658-136b-46f5-9458-f86f059498b1 ('2b633f9a-8840-11e1-9ed7-17e6c491db84', 1, '2b633f9a-8840-11e1-9ed7-17e6c491db84', '025062dd-d69e-40ac-9031-c080b51647ae', 1) {} failed - stopping task Thread-286::DEBUG::2013-04-27 22:45:30,052::task::1201::TaskManager.Task::(stop) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::stopping in state preparing (force False) Thread-286::DEBUG::2013-04-27 22:45:30,053::task::980::TaskManager.Task::(_decref) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::ref 1 aborting True Thread-286::INFO::2013-04-27 22:45:30,053::task::1159::TaskManager.Task::(prepare) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::aborting: Task is aborted: 'Cannot find master domain' - code 304 Thread-286::DEBUG::2013-04-27 22:45:30,054::task::1164::TaskManager.Task::(prepare) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::Prepare: aborted: Cannot find master domain Thread-286::DEBUG::2013-04-27 22:45:30,054::task::980::TaskManager.Task::(_decref) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::ref 0 aborting True Thread-286::DEBUG::2013-04-27 22:45:30,055::task::915::TaskManager.Task::(_doAbort) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::Task._doAbort: force False Thread-286::DEBUG::2013-04-27 22:45:30,055::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-286::DEBUG::2013-04-27 22:45:30,056::task::588::TaskManager.Task::(_updateState) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::moving from state preparing -> state aborting Thread-286::DEBUG::2013-04-27 22:45:30,056::task::537::TaskManager.Task::(__state_aborting) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::_aborting: recover policy none Thread-286::DEBUG::2013-04-27 22:45:30,057::task::588::TaskManager.Task::(_updateState) Task=`a7d4f658-136b-46f5-9458-f86f059498b1`::moving from state aborting -> state failed Thread-286::DEBUG::2013-04-27 22:45:30,057::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-286::DEBUG::2013-04-27 22:45:30,058::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-286::ERROR::2013-04-27 22:45:30,059::dispatcher::90::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'", 'code': 304}} Thread-329::DEBUG::2013-04-27 22:45:30,107::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-329::DEBUG::2013-04-27 22:45:30,108::task::588::TaskManager.Task::(_updateState) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::moving from state init -> state preparing Thread-329::INFO::2013-04-27 22:45:30,109::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', options=None) Thread-329::ERROR::2013-04-27 22:45:30,110::task::855::TaskManager.Task::(_setError) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 863, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 438, in getSpmStatus pool = self.getPool(spUUID) File "/usr/share/vdsm/storage/hsm.py", line 186, in getPool raise se.StoragePoolUnknown(spUUID) StoragePoolUnknown: Unknown pool id, pool not connected: ('2b633f9a-8840-11e1-9ed7-17e6c491db84',) Thread-329::DEBUG::2013-04-27 22:45:30,110::task::874::TaskManager.Task::(_run) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::Task._run: 162609d1-b21e-49ac-b4e6-061283cfa375 ('2b633f9a-8840-11e1-9ed7-17e6c491db84',) {} failed - stopping task Thread-329::DEBUG::2013-04-27 22:45:30,111::task::1201::TaskManager.Task::(stop) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::stopping in state preparing (force False) Thread-329::DEBUG::2013-04-27 22:45:30,111::task::980::TaskManager.Task::(_decref) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::ref 1 aborting True Thread-329::INFO::2013-04-27 22:45:30,112::task::1159::TaskManager.Task::(prepare) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 Thread-329::DEBUG::2013-04-27 22:45:30,112::task::1164::TaskManager.Task::(prepare) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::Prepare: aborted: Unknown pool id, pool not connected Thread-329::DEBUG::2013-04-27 22:45:30,113::task::980::TaskManager.Task::(_decref) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::ref 0 aborting True Thread-329::DEBUG::2013-04-27 22:45:30,114::task::915::TaskManager.Task::(_doAbort) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::Task._doAbort: force False Thread-329::DEBUG::2013-04-27 22:45:30,114::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-329::DEBUG::2013-04-27 22:45:30,115::task::588::TaskManager.Task::(_updateState) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::moving from state preparing -> state aborting Thread-329::DEBUG::2013-04-27 22:45:30,115::task::537::TaskManager.Task::(__state_aborting) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::_aborting: recover policy none Thread-329::DEBUG::2013-04-27 22:45:30,116::task::588::TaskManager.Task::(_updateState) Task=`162609d1-b21e-49ac-b4e6-061283cfa375`::moving from state aborting -> state failed Thread-329::DEBUG::2013-04-27 22:45:30,116::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-329::DEBUG::2013-04-27 22:45:30,117::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-329::ERROR::2013-04-27 22:45:30,117::dispatcher::90::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('2b633f9a-8840-11e1-9ed7-17e6c491db84',)", 'code': 309}} Thread-331::DEBUG::2013-04-27 22:45:30,159::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-331::DEBUG::2013-04-27 22:45:30,160::task::588::TaskManager.Task::(_updateState) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::moving from state init -> state preparing Thread-331::INFO::2013-04-27 22:45:30,161::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', hostID=1, scsiKey='2b633f9a-8840-11e1-9ed7-17e6c491db84', msdUUID='025062dd-d69e-40ac-9031-c080b51647ae', masterVersion=1, options=None) Thread-331::DEBUG::2013-04-27 22:45:30,162::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`f6f7913c-4cb2-4811-baa8-431b4e4eac91`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '747' at '_connectStoragePool' Thread-331::DEBUG::2013-04-27 22:45:30,162::resourceManager::483::ResourceManager::(registerResource) Trying to register resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' for lock type 'exclusive' Thread-331::DEBUG::2013-04-27 22:45:30,163::resourceManager::525::ResourceManager::(registerResource) Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free. Now locking as 'exclusive' (1 active user) Thread-331::DEBUG::2013-04-27 22:45:30,164::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`f6f7913c-4cb2-4811-baa8-431b4e4eac91`::Granted request Thread-331::INFO::2013-04-27 22:45:30,165::sp::608::Storage.StoragePool::(connect) Connect host #1 to the storage pool 2b633f9a-8840-11e1-9ed7-17e6c491db84 with master domain: 025062dd-d69e-40ac-9031-c080b51647ae (ver = 1) Thread-331::DEBUG::2013-04-27 22:45:30,166::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-331::DEBUG::2013-04-27 22:45:30,166::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-331::DEBUG::2013-04-27 22:45:30,167::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-331::DEBUG::2013-04-27 22:45:30,167::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-331::DEBUG::2013-04-27 22:45:30,168::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-331::DEBUG::2013-04-27 22:45:30,169::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-331::DEBUG::2013-04-27 22:45:30,169::misc::1017::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-331::DEBUG::2013-04-27 22:45:30,170::misc::1019::SamplingMethod::(__call__) Got in to sampling method Thread-331::DEBUG::2013-04-27 22:45:30,170::misc::1017::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-331::DEBUG::2013-04-27 22:45:30,171::misc::1019::SamplingMethod::(__call__) Got in to sampling method Thread-331::DEBUG::2013-04-27 22:45:30,172::iscsi::389::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) Thread-331::DEBUG::2013-04-27 22:45:30,194::iscsi::389::Storage.Misc.excCmd::(rescan) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 Thread-331::DEBUG::2013-04-27 22:45:30,195::misc::1027::SamplingMethod::(__call__) Returning last result Thread-331::DEBUG::2013-04-27 22:45:31,778::multipath::71::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None) Thread-331::DEBUG::2013-04-27 22:45:31,807::multipath::71::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-331::DEBUG::2013-04-27 22:45:31,808::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,808::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,809::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,810::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,810::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,811::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,811::misc::1027::SamplingMethod::(__call__) Returning last result Thread-331::DEBUG::2013-04-27 22:45:31,812::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,815::lvm::287::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36782bcb063f75600162a25c119f14819%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 025062dd-d69e-40ac-9031-c080b51647ae' (cwd None) Thread-331::DEBUG::2013-04-27 22:45:31,934::lvm::287::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found\n'; <rc> = 5 Thread-331::WARNING::2013-04-27 22:45:31,937::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found'] Thread-331::DEBUG::2013-04-27 22:45:31,938::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-331::DEBUG::2013-04-27 22:45:31,958::fileSD::98::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.0.0.154:_home_vm_data_data/025062dd-d69e-40ac-9031-c080b51647ae Thread-331::DEBUG::2013-04-27 22:45:31,960::persistentDict::175::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend Thread-336::DEBUG::2013-04-27 22:45:38,237::task::588::TaskManager.Task::(_updateState) Task=`1a7e26c3-d715-4cac-bce5-d0ddd21ab373`::moving from state init -> state preparing Thread-336::INFO::2013-04-27 22:45:38,238::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-336::INFO::2013-04-27 22:45:38,238::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-336::DEBUG::2013-04-27 22:45:38,239::task::1174::TaskManager.Task::(prepare) Task=`1a7e26c3-d715-4cac-bce5-d0ddd21ab373`::finished: {} Thread-336::DEBUG::2013-04-27 22:45:38,240::task::588::TaskManager.Task::(_updateState) Task=`1a7e26c3-d715-4cac-bce5-d0ddd21ab373`::moving from state preparing -> state finished Thread-336::DEBUG::2013-04-27 22:45:38,240::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-336::DEBUG::2013-04-27 22:45:38,241::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-336::DEBUG::2013-04-27 22:45:38,241::task::980::TaskManager.Task::(_decref) Task=`1a7e26c3-d715-4cac-bce5-d0ddd21ab373`::ref 0 aborting False Thread-342::DEBUG::2013-04-27 22:45:48,395::task::588::TaskManager.Task::(_updateState) Task=`cf4c41c0-e453-4493-9a68-b4044d0f0124`::moving from state init -> state preparing Thread-342::INFO::2013-04-27 22:45:48,396::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-342::INFO::2013-04-27 22:45:48,396::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-342::DEBUG::2013-04-27 22:45:48,397::task::1174::TaskManager.Task::(prepare) Task=`cf4c41c0-e453-4493-9a68-b4044d0f0124`::finished: {} Thread-342::DEBUG::2013-04-27 22:45:48,397::task::588::TaskManager.Task::(_updateState) Task=`cf4c41c0-e453-4493-9a68-b4044d0f0124`::moving from state preparing -> state finished Thread-342::DEBUG::2013-04-27 22:45:48,398::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-342::DEBUG::2013-04-27 22:45:48,399::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-342::DEBUG::2013-04-27 22:45:48,399::task::980::TaskManager.Task::(_decref) Task=`cf4c41c0-e453-4493-9a68-b4044d0f0124`::ref 0 aborting False Thread-348::DEBUG::2013-04-27 22:45:58,557::task::588::TaskManager.Task::(_updateState) Task=`b536de2b-6e78-43a9-b2b1-3667b51828df`::moving from state init -> state preparing Thread-348::INFO::2013-04-27 22:45:58,558::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-348::INFO::2013-04-27 22:45:58,558::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-348::DEBUG::2013-04-27 22:45:58,559::task::1174::TaskManager.Task::(prepare) Task=`b536de2b-6e78-43a9-b2b1-3667b51828df`::finished: {} Thread-348::DEBUG::2013-04-27 22:45:58,560::task::588::TaskManager.Task::(_updateState) Task=`b536de2b-6e78-43a9-b2b1-3667b51828df`::moving from state preparing -> state finished Thread-348::DEBUG::2013-04-27 22:45:58,560::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-348::DEBUG::2013-04-27 22:45:58,561::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-348::DEBUG::2013-04-27 22:45:58,561::task::980::TaskManager.Task::(_decref) Task=`b536de2b-6e78-43a9-b2b1-3667b51828df`::ref 0 aborting False Thread-354::DEBUG::2013-04-27 22:46:08,718::task::588::TaskManager.Task::(_updateState) Task=`a8e24fe3-5490-480c-a497-ee3f291698b0`::moving from state init -> state preparing Thread-354::INFO::2013-04-27 22:46:08,719::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-354::INFO::2013-04-27 22:46:08,720::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-354::DEBUG::2013-04-27 22:46:08,720::task::1174::TaskManager.Task::(prepare) Task=`a8e24fe3-5490-480c-a497-ee3f291698b0`::finished: {} Thread-354::DEBUG::2013-04-27 22:46:08,721::task::588::TaskManager.Task::(_updateState) Task=`a8e24fe3-5490-480c-a497-ee3f291698b0`::moving from state preparing -> state finished Thread-354::DEBUG::2013-04-27 22:46:08,721::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-354::DEBUG::2013-04-27 22:46:08,722::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-354::DEBUG::2013-04-27 22:46:08,722::task::980::TaskManager.Task::(_decref) Task=`a8e24fe3-5490-480c-a497-ee3f291698b0`::ref 0 aborting False Thread-360::DEBUG::2013-04-27 22:46:18,879::task::588::TaskManager.Task::(_updateState) Task=`9febbab9-7bae-4df5-9d16-5e89397ae101`::moving from state init -> state preparing Thread-360::INFO::2013-04-27 22:46:18,880::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-360::INFO::2013-04-27 22:46:18,880::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-360::DEBUG::2013-04-27 22:46:18,881::task::1174::TaskManager.Task::(prepare) Task=`9febbab9-7bae-4df5-9d16-5e89397ae101`::finished: {} Thread-360::DEBUG::2013-04-27 22:46:18,882::task::588::TaskManager.Task::(_updateState) Task=`9febbab9-7bae-4df5-9d16-5e89397ae101`::moving from state preparing -> state finished Thread-360::DEBUG::2013-04-27 22:46:18,882::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-360::DEBUG::2013-04-27 22:46:18,883::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-360::DEBUG::2013-04-27 22:46:18,883::task::980::TaskManager.Task::(_decref) Task=`9febbab9-7bae-4df5-9d16-5e89397ae101`::ref 0 aborting False Thread-366::DEBUG::2013-04-27 22:46:29,039::task::588::TaskManager.Task::(_updateState) Task=`f0468880-10a0-429b-8fce-547289a14fd1`::moving from state init -> state preparing Thread-366::INFO::2013-04-27 22:46:29,040::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-366::INFO::2013-04-27 22:46:29,041::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-366::DEBUG::2013-04-27 22:46:29,041::task::1174::TaskManager.Task::(prepare) Task=`f0468880-10a0-429b-8fce-547289a14fd1`::finished: {} Thread-366::DEBUG::2013-04-27 22:46:29,042::task::588::TaskManager.Task::(_updateState) Task=`f0468880-10a0-429b-8fce-547289a14fd1`::moving from state preparing -> state finished Thread-366::DEBUG::2013-04-27 22:46:29,042::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-366::DEBUG::2013-04-27 22:46:29,043::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-366::DEBUG::2013-04-27 22:46:29,043::task::980::TaskManager.Task::(_decref) Task=`f0468880-10a0-429b-8fce-547289a14fd1`::ref 0 aborting False Thread-331::ERROR::2013-04-27 22:46:34,024::sdc::113::Storage.StorageDomainCache::(_findDomain) Error while looking for domain `025062dd-d69e-40ac-9031-c080b51647ae` Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 109, in _findDomain return mod.findDomain(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 149, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 104, in __init__ sdUUID = metadata[sd.DMDK_SDUUID] File "/usr/share/vdsm/storage/persistentDict.py", line 75, in __getitem__ return dec(self._dict[key]) File "/usr/share/vdsm/storage/persistentDict.py", line 183, in __getitem__ with self._accessWrapper(): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/share/vdsm/storage/persistentDict.py", line 137, in _accessWrapper self.refresh() File "/usr/share/vdsm/storage/persistentDict.py", line 214, in refresh lines = self._metaRW.readlines() File "/usr/share/vdsm/storage/fileSD.py", line 71, in readlines return misc.stripNewLines(self._oop.directReadLines(self._metafile)) File "/usr/share/vdsm/storage/processPool.py", line 53, in wrapper return self.runExternally(func, *args, **kwds) File "/usr/share/vdsm/storage/processPool.py", line 64, in runExternally return self._procPool.runExternally(*args, **kwargs) File "/usr/share/vdsm/storage/processPool.py", line 145, in runExternally raise Timeout("Operation Stuck") Timeout: Operation Stuck Thread-331::DEBUG::2013-04-27 22:46:34,026::resourceManager::535::ResourceManager::(releaseResource) Trying to release resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' Thread-331::DEBUG::2013-04-27 22:46:34,027::resourceManager::550::ResourceManager::(releaseResource) Released resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' (0 active users) Thread-331::DEBUG::2013-04-27 22:46:34,028::resourceManager::555::ResourceManager::(releaseResource) Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free, finding out if anyone is waiting for it. Thread-331::DEBUG::2013-04-27 22:46:34,028::resourceManager::562::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84', Clearing records. Thread-331::ERROR::2013-04-27 22:46:34,029::task::855::TaskManager.Task::(_setError) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 863, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 721, in connectStoragePool return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) File "/usr/share/vdsm/storage/hsm.py", line 763, in _connectStoragePool res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 624, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1097, in __rebuild self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1435, in getMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' Thread-331::DEBUG::2013-04-27 22:46:34,030::task::874::TaskManager.Task::(_run) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::Task._run: 4a7f87fa-cbbc-41de-a457-bfe6a87e00a0 ('2b633f9a-8840-11e1-9ed7-17e6c491db84', 1, '2b633f9a-8840-11e1-9ed7-17e6c491db84', '025062dd-d69e-40ac-9031-c080b51647ae', 1) {} failed - stopping task Thread-331::DEBUG::2013-04-27 22:46:34,030::task::1201::TaskManager.Task::(stop) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::stopping in state preparing (force False) Thread-331::DEBUG::2013-04-27 22:46:34,031::task::980::TaskManager.Task::(_decref) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::ref 1 aborting True Thread-331::INFO::2013-04-27 22:46:34,032::task::1159::TaskManager.Task::(prepare) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::aborting: Task is aborted: 'Cannot find master domain' - code 304 Thread-331::DEBUG::2013-04-27 22:46:34,032::task::1164::TaskManager.Task::(prepare) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::Prepare: aborted: Cannot find master domain Thread-331::DEBUG::2013-04-27 22:46:34,033::task::980::TaskManager.Task::(_decref) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::ref 0 aborting True Thread-331::DEBUG::2013-04-27 22:46:34,033::task::915::TaskManager.Task::(_doAbort) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::Task._doAbort: force False Thread-331::DEBUG::2013-04-27 22:46:34,034::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-331::DEBUG::2013-04-27 22:46:34,034::task::588::TaskManager.Task::(_updateState) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::moving from state preparing -> state aborting Thread-331::DEBUG::2013-04-27 22:46:34,035::task::537::TaskManager.Task::(__state_aborting) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::_aborting: recover policy none Thread-331::DEBUG::2013-04-27 22:46:34,035::task::588::TaskManager.Task::(_updateState) Task=`4a7f87fa-cbbc-41de-a457-bfe6a87e00a0`::moving from state aborting -> state failed Thread-331::DEBUG::2013-04-27 22:46:34,036::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-331::DEBUG::2013-04-27 22:46:34,036::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-331::ERROR::2013-04-27 22:46:34,037::dispatcher::90::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'", 'code': 304}} Thread-371::DEBUG::2013-04-27 22:46:34,082::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-371::DEBUG::2013-04-27 22:46:34,083::task::588::TaskManager.Task::(_updateState) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::moving from state init -> state preparing Thread-371::INFO::2013-04-27 22:46:34,084::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', options=None) Thread-371::ERROR::2013-04-27 22:46:34,084::task::855::TaskManager.Task::(_setError) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 863, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 438, in getSpmStatus pool = self.getPool(spUUID) File "/usr/share/vdsm/storage/hsm.py", line 186, in getPool raise se.StoragePoolUnknown(spUUID) StoragePoolUnknown: Unknown pool id, pool not connected: ('2b633f9a-8840-11e1-9ed7-17e6c491db84',) Thread-371::DEBUG::2013-04-27 22:46:34,085::task::874::TaskManager.Task::(_run) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::Task._run: e8ba0700-30eb-4467-b1d0-fbbd7636441b ('2b633f9a-8840-11e1-9ed7-17e6c491db84',) {} failed - stopping task Thread-371::DEBUG::2013-04-27 22:46:34,086::task::1201::TaskManager.Task::(stop) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::stopping in state preparing (force False) Thread-371::DEBUG::2013-04-27 22:46:34,086::task::980::TaskManager.Task::(_decref) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::ref 1 aborting True Thread-371::INFO::2013-04-27 22:46:34,087::task::1159::TaskManager.Task::(prepare) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 Thread-371::DEBUG::2013-04-27 22:46:34,087::task::1164::TaskManager.Task::(prepare) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::Prepare: aborted: Unknown pool id, pool not connected Thread-371::DEBUG::2013-04-27 22:46:34,088::task::980::TaskManager.Task::(_decref) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::ref 0 aborting True Thread-371::DEBUG::2013-04-27 22:46:34,088::task::915::TaskManager.Task::(_doAbort) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::Task._doAbort: force False Thread-371::DEBUG::2013-04-27 22:46:34,089::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-371::DEBUG::2013-04-27 22:46:34,089::task::588::TaskManager.Task::(_updateState) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::moving from state preparing -> state aborting Thread-371::DEBUG::2013-04-27 22:46:34,090::task::537::TaskManager.Task::(__state_aborting) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::_aborting: recover policy none Thread-371::DEBUG::2013-04-27 22:46:34,090::task::588::TaskManager.Task::(_updateState) Task=`e8ba0700-30eb-4467-b1d0-fbbd7636441b`::moving from state aborting -> state failed Thread-371::DEBUG::2013-04-27 22:46:34,091::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-371::DEBUG::2013-04-27 22:46:34,092::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-371::ERROR::2013-04-27 22:46:34,092::dispatcher::90::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('2b633f9a-8840-11e1-9ed7-17e6c491db84',)", 'code': 309}} Thread-372::DEBUG::2013-04-27 22:46:34,116::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-372::DEBUG::2013-04-27 22:46:34,117::task::588::TaskManager.Task::(_updateState) Task=`4a764a93-93ca-432b-ac1d-ca18c36f4b47`::moving from state init -> state preparing Thread-372::INFO::2013-04-27 22:46:34,118::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', hostID=1, scsiKey='2b633f9a-8840-11e1-9ed7-17e6c491db84', msdUUID='025062dd-d69e-40ac-9031-c080b51647ae', masterVersion=1, options=None) Thread-372::DEBUG::2013-04-27 22:46:34,119::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`da6e89ed-9d5c-46d7-831a-53f401443358`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '747' at '_connectStoragePool' Thread-372::DEBUG::2013-04-27 22:46:34,119::resourceManager::483::ResourceManager::(registerResource) Trying to register resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' for lock type 'exclusive' Thread-372::DEBUG::2013-04-27 22:46:34,120::resourceManager::525::ResourceManager::(registerResource) Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free. Now locking as 'exclusive' (1 active user) Thread-372::DEBUG::2013-04-27 22:46:34,120::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`da6e89ed-9d5c-46d7-831a-53f401443358`::Granted request Thread-372::INFO::2013-04-27 22:46:34,121::sp::608::Storage.StoragePool::(connect) Connect host #1 to the storage pool 2b633f9a-8840-11e1-9ed7-17e6c491db84 with master domain: 025062dd-d69e-40ac-9031-c080b51647ae (ver = 1) Thread-372::DEBUG::2013-04-27 22:46:34,122::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-372::DEBUG::2013-04-27 22:46:34,123::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-372::DEBUG::2013-04-27 22:46:34,124::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-372::DEBUG::2013-04-27 22:46:34,124::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-372::DEBUG::2013-04-27 22:46:34,125::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-372::DEBUG::2013-04-27 22:46:34,126::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-372::DEBUG::2013-04-27 22:46:34,126::misc::1017::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-372::DEBUG::2013-04-27 22:46:34,127::misc::1019::SamplingMethod::(__call__) Got in to sampling method Thread-372::DEBUG::2013-04-27 22:46:34,127::misc::1017::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-372::DEBUG::2013-04-27 22:46:34,128::misc::1019::SamplingMethod::(__call__) Got in to sampling method Thread-372::DEBUG::2013-04-27 22:46:34,128::iscsi::389::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) Thread-372::DEBUG::2013-04-27 22:46:34,151::iscsi::389::Storage.Misc.excCmd::(rescan) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 Thread-372::DEBUG::2013-04-27 22:46:34,152::misc::1027::SamplingMethod::(__call__) Returning last result Thread-372::DEBUG::2013-04-27 22:46:35,735::multipath::71::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None) Thread-372::DEBUG::2013-04-27 22:46:35,764::multipath::71::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-372::DEBUG::2013-04-27 22:46:35,765::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,766::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,766::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,767::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,768::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,768::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,769::misc::1027::SamplingMethod::(__call__) Returning last result Thread-372::DEBUG::2013-04-27 22:46:35,770::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,772::lvm::287::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36782bcb063f75600162a25c119f14819%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 025062dd-d69e-40ac-9031-c080b51647ae' (cwd None) Thread-372::DEBUG::2013-04-27 22:46:35,874::lvm::287::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found\n'; <rc> = 5 Thread-372::WARNING::2013-04-27 22:46:35,877::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found'] Thread-372::DEBUG::2013-04-27 22:46:35,878::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-372::DEBUG::2013-04-27 22:46:35,898::fileSD::98::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.0.0.154:_home_vm_data_data/025062dd-d69e-40ac-9031-c080b51647ae ------------------------------------------------------------------ Now, one interesting aspect I've noticed is that when I reboot the server, the mounted NFS shares change. As I mentioned earlier, in this instance there were no NFS shares mounted when the machine booted and none were mounted until I tried to activate the data centre. I subsequently rebooted the server after making some changes to nfsmount.conf, and upon booting the ISO share was mounted. Intrigued, I rebooted again and both the ISO and Export shares were mounted, but not the data share. Do you - or does anybody - have any further insight? Thanks for your help so far - looking in to the NFS side of things is giving some interesting results, and I'm sure that we're on the path to great success. --Steven.

3 4

[Users] what is ovirt's latest roadmap?
by bigclouds 27 Apr '13

27 Apr '13

------=_Part_58079_321434948.1367033361704 Content-Type: text/plain; charset=GBK Content-Transfer-Encoding: 7bit what is ovirt's latest roadmap? thanks ------=_Part_58079_321434948.1367033361704 Content-Type: text/html; charset=GBK Content-Transfer-Encoding: 7bit <div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial"><div><span>what is ovirt's latest roadmap?</span></div><div><span></span> </div><div><span>thanks</span></div></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span> ------=_Part_58079_321434948.1367033361704--

2 2

[Users] How to rescue storage domain structure
by Chris Smith 27 Apr '13

27 Apr '13

List, I have lost the ability to manage the hosts or VM's using ovirt engine web interface. The data center is offline, and I can't actually perform any operations with the hosts or VM's. I don't think that there are any actions I can perform in the web interface at all. What's odd is that I can tell the host to go into maintenance mode using the ovirt-engine web interface and it seems to go into maintenance mode. It even shows the wrench icon next to the host. I can also try and activate it after it susposedly goes into maintenance mode, and It states that the host was activated, but the host never actually comes up or contends for SPM status, and the data center never comes online. >From the logs it seems that at least PKI is broken between the engine and the hosts as I see numerous certificate errors on both the ovirt-engine and clients. vdsm.log shows: Traceback (most recent call last): File "/usr/lib64/python2.7/SocketServer.py", line 582, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py", line 66, in finish_request request.do_handshake() File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake self._sslobj.do_handshake() SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate unknown and engine.log shows: 2013-04-18 18:42:43,632 ERROR [org.ovirt.engine.core. engineencryptutils.EncryptionUtils] (QuartzScheduler_Worker-68) Failed to decryptData must start with zero 2013-04-18 18:42:43,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand] (QuartzScheduler_Worker-68) XML RPC error in command Alon Bar-Lev was able to offer several good pointers in another thread titled "Certificates and PKI seem to be broken after yum update" and eventually concluded that the installation seems to be corrupted more than just the certificates, truststore, and keystore, and suggested that I start a new thread to ask about how to rescue the storage domain structure. The storage used for the data center is ISCSI, which is intact and working. In fact 2 of the VM's are still online and running on one of the original FC17 hosts systems. I'm not able to reinstall any of the existing hosts from the ovirt-engine web interface. I attempted to reinstall one of the hosts (not the SPM) which failed. I also tried to bring up a new, third host and add it to the cluster. I setup another Fedora 17 box up and tried to add it to the cluster, but it states that there are no available servers in the cluster to probe the new host. This is a test environment that I would like to fix, but I'm also willing to just run engine cleanup and start over. That said, there are 3 VM's that I would like to keep. Two are online and running, and I'm able to see them with virsh on that host. I was wondering about using virsh to backup these vm's. The third VM exists in the database, and was set to run on the host that I attempted to reinstall, but that VM isn't running, and when I use virsh on it's host, virsh can't seem to find it, when I perform the list commands, and I can't start it with virsh <vm-name> What is the best way to proceed? It seems like it would be easier to export the VM's using virsh from the host that they run on if possible, then update ovirt to the latest version, recreate everything and then import the VM's back in to the new environment. Will this work? Is there a procedure I can follow to do this? Here's some additional information about the installed ovirt packages on the ovirt-engine [root@reliant yum.repos.d]# yum list installed | grep ovirt ovirt-engine.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-backend.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-cli.noarch 3.2.0.5-1.fc17 @updates ovirt-engine-config.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-dbscripts.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-genericapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-notification- service.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-restapi.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-sdk.noarch 3.2.0.2-1.fc17 @updates ovirt-engine-setup.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-tools-common.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-userportal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-engine-webadmin-portal.noarch 3.1.0-4.fc17 @ovirt-stable ovirt-image-uploader.noarch 3.1.0-0.git9c42c8.fc17 @ovirt-stable ovirt-iso-uploader.noarch 3.1.0-0.git1841d9.fc17 @ovirt-stable ovirt-log-collector.noarch 3.1.0-0.git10d719.fc17 @ovirt-stable ovirt-release-fedora.noarch 4-2 @/ovirt-release-fedora.noarch

4 4

[Users] Unable to activate data center after host reboot.
by Steven Wiltshire 27 Apr '13

27 Apr '13

Hi, I have a single machine with an older oVirt installation that was configured along the lines of this document: http://blog.jebpages.com/archives/how-to-get-up-and-running-with-ovirt/ Installed packages are: ovirt-engine-3.0.0_0001-1.6.fc16.x86_64 vdsm-4.9.3.3-0.fc16.x86_64 It has been largely untouched since configured, and has been trouble free. I've had to physically move the server, which involved powering it down Until then, it had been up for around 250 days. Upon restarting it, the oVirt web admin page tells me: No valid Data Storage Domains are available in Data Center Default (please check your storage infrastructure). ... and I can't activate my "Default" data center, meaning I can't activate any storage domains and thus am unable to start my VM's. Below is the output for both engine.log and vdsm.log, taken from the moment I attempt to activate the data center. Does anybody have any insights as to what's going wrong? Best regards, --Steven Wiltshire. Storage is by NFS on the single host. Here is the relevant output of "mount" ------------------------------------------------------------------ 10.0.0.154:/home/vm/data/iso on /rhev/data-center/mnt/10.0.0.154:_home_vm_data_iso type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.154) 10.0.0.154:/home/vm/data/export on /rhev/data-center/mnt/10.0.0.154:_home_vm_data_export type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.154) 10.0.0.154:/home/vm/data/data on /rhev/data-center/mnt/10.0.0.154:_home_vm_data_data type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,port=0,timeo=600,retrans=6,sec=sys,clientaddr=10.0.0.154,local_lock=none,addr=10.0.0.154) ------------------------------------------------------------------ Here is the contents of the metadata file in /home/vm/data/data/025062dd-d69e-40ac-9031-c080b51647ae/dom_md: ------------------------------------------------------------------ CLASS=Data DESCRIPTION=Data IOOPTIMEOUTSEC=10 LEASERETRIES=3 LEASETIMESEC=60 LOCKPOLICY= LOCKRENEWALINTERVALSEC=5 MASTER_VERSION=1 POOL_DESCRIPTION=Default POOL_DOMAINS=025062dd-d69e-40ac-9031-c080b51647ae:Active,d30604e3-3ca8-470a-8da4-18c9402f7504:Active,503e6677-2d51-4668-ab17-f914bdb57c19:Active POOL_SPM_ID=1 POOL_SPM_LVER=30 POOL_UUID=2b633f9a-8840-11e1-9ed7-17e6c491db84 REMOTE_PATH=10.0.0.154:/home/vm/data/data ROLE=Master SDUUID=025062dd-d69e-40ac-9031-c080b51647ae TYPE=NFS VERSION=0 _SHA_CKSUM=c838aeb3b1f5180fd519ee3461e154d2bd79568d ------------------------------------------------------------------ From engine.log: ------------------------------------------------------------------ 2013-04-27 14:14:50,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 304 mMessage Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:14:50,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-100) Vds: BigServer 2013-04-27 14:14:50,026 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-100) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:14:50,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-100) FINISH, ConnectStoragePoolVDSCommand, log id: 7fec8068 2013-04-27 14:14:50,027 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (QuartzScheduler_Worker-100) transaction rolled back 2013-04-27 14:14:50,028 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) IrsBroker::Failed::GetStoragePoolInfoVDS 2013-04-27 14:14:50,028 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:14:50,063 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-100) Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 025062dd-d69e-40ac-9031-c080b51647ae Type: Storage 2013-04-27 14:14:50,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) START, MarkPoolInReconstructModeVDSCommand(storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 31dad08a 2013-04-27 14:14:50,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-100) clearing cache for problematic entities in pool 2b633f9a-8840-11e1-9ed7-17e6c491db84 2013-04-27 14:14:50,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-100) FINISH, MarkPoolInReconstructModeVDSCommand, log id: 31dad08a 2013-04-27 14:14:57,494 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) domain 503e6677-2d51-4668-ab17-f914bdb57c19 in problem. vds: BigServer 2013-04-27 14:14:57,495 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-27) domain d30604e3-3ca8-470a-8da4-18c9402f7504 in problem. vds: BigServer 2013-04-27 14:15:00,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) hostFromVds::selectedVds - BigServer, spmStatus Unknown_Pool, storage pool Default 2013-04-27 14:15:00,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) START, ConnectStoragePoolVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 5b22a385 2013-04-27 14:15:23,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) starting ProcessDomainRecovery for domain d30604e3-3ca8-470a-8da4-18c9402f7504 2013-04-27 14:15:23,617 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) domain d30604e3-3ca8-470a-8da4-18c9402f7504 was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. 2013-04-27 14:15:23,625 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-13) Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: d30604e3-3ca8-470a-8da4-18c9402f7504 Type: Storage 2013-04-27 14:15:23,633 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-13) CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL 2013-04-27 14:15:23,637 ERROR [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-13) Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand. 2013-04-27 14:15:23,637 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) No string for UNASSIGNED type. Use default Log 2013-04-27 14:15:23,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) starting ProcessDomainRecovery for domain 503e6677-2d51-4668-ab17-f914bdb57c19 2013-04-27 14:15:23,646 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-12) domain 503e6677-2d51-4668-ab17-f914bdb57c19 was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. 2013-04-27 14:15:23,653 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-12) Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: 503e6677-2d51-4668-ab17-f914bdb57c19 Type: Storage 2013-04-27 14:15:23,661 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-12) CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL 2013-04-27 14:15:23,665 ERROR [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-12) Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand. 2013-04-27 14:15:23,665 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-12) No string for UNASSIGNED type. Use default Log 2013-04-27 14:15:27,944 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) domain 503e6677-2d51-4668-ab17-f914bdb57c19 in problem. vds: BigServer 2013-04-27 14:15:27,944 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-24) domain d30604e3-3ca8-470a-8da4-18c9402f7504 in problem. vds: BigServer 2013-04-27 14:15:34,953 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-33) Lock Acquired to object EngineLock [exclusiveLocks= key: org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand value: 025062dd-d69e-40ac-9031-c080b51647ae , sharedLocks= ] 2013-04-27 14:15:34,954 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-33) Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: 025062dd-d69e-40ac-9031-c080b51647ae Type: Storage 2013-04-27 14:15:34,965 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-33) Lock freed to object EngineLock [exclusiveLocks= key: org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand value: 025062dd-d69e-40ac-9031-c080b51647ae , sharedLocks= ] 2013-04-27 14:15:34,965 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-33) ActivateStorage Domain. Before Connect all hosts to pool. Time:4/27/13 2:15 PM 2013-04-27 14:15:35,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thread-48) START, ValidateStorageServerConnectionVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 60e73855-daa2-45f5-9c59-5973ca89e830, connection: 10.0.0.154:/home/vm/data/data };]), log id: 3f3f3015 2013-04-27 14:15:35,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ValidateStorageServerConnectionVDSCommand] (pool-5-thread-48) FINISH, ValidateStorageServerConnectionVDSCommand, return: {60e73855-daa2-45f5-9c59-5973ca89e830=0}, log id: 3f3f3015 2013-04-27 14:15:35,140 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-5-thread-48) Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System 2013-04-27 14:15:35,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-48) START, ConnectStorageServerVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 60e73855-daa2-45f5-9c59-5973ca89e830, connection: 10.0.0.154:/home/vm/data/data };]), log id: 32a6a7fd 2013-04-27 14:15:35,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-5-thread-48) FINISH, ConnectStorageServerVDSCommand, return: {60e73855-daa2-45f5-9c59-5973ca89e830=0}, log id: 32a6a7fd 2013-04-27 14:15:35,186 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-33) ActivateStorage Domain. After Connect all hosts to pool. Time:4/27/13 2:15 PM 2013-04-27 14:15:35,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-5-thread-33) START, ActivateStorageDomainVDSCommand(storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, ignoreFailoverLimit = false, compatabilityVersion = null, storageDomainId = 025062dd-d69e-40ac-9031-c080b51647ae), log id: 4e53941a 2013-04-27 14:16:04,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 304 mMessage Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:16:04,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-30) Vds: BigServer 2013-04-27 14:16:04,043 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-30) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:16:04,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-30) FINISH, ConnectStoragePoolVDSCommand, log id: 5b22a385 2013-04-27 14:16:04,045 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (QuartzScheduler_Worker-30) transaction rolled back 2013-04-27 14:16:04,045 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) IrsBroker::Failed::GetStoragePoolInfoVDS 2013-04-27 14:16:04,046 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-30) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:16:04,060 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-30) CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2 2013-04-27 14:16:04,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-33) hostFromVds::selectedVds - BigServer, spmStatus Unknown_Pool, storage pool Default 2013-04-27 14:16:04,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-5-thread-33) START, ConnectStoragePoolVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 674d0c6d 2013-04-27 14:16:41,155 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (http--0.0.0.0-8080-5) calling GetConfigurationValueQuery with null version, using default general for version 2013-04-27 14:16:41,155 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (http--0.0.0.0-8080-7) calling GetConfigurationValueQuery with null version, using default general for version 2013-04-27 14:16:41,156 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (http--0.0.0.0-8080-8) calling GetConfigurationValueQuery with null version, using default general for version 2013-04-27 14:16:41,157 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (http--0.0.0.0-8080-12) calling GetConfigurationValueQuery with null version, using default general for version 2013-04-27 14:16:41,256 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (http--0.0.0.0-8080-12) calling GetConfigurationValueQuery with null version, using default general for version 2013-04-27 14:16:41,285 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (http--0.0.0.0-8080-6) calling GetConfigurationValueQuery with null version, using default general for version 2013-04-27 14:17:08,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-33) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 304 mMessage Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:17:08,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-33) Vds: BigServer 2013-04-27 14:17:08,015 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-33) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:17:08,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-5-thread-33) FINISH, ConnectStoragePoolVDSCommand, log id: 674d0c6d 2013-04-27 14:17:08,016 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (pool-5-thread-33) transaction rolled back 2013-04-27 14:17:08,016 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-33) IrsBroker::Failed::ActivateStorageDomainVDS 2013-04-27 14:17:08,017 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-5-thread-33) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:17:08,030 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-5-thread-33) CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2 2013-04-27 14:17:08,030 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-5-thread-33) FINISH, ActivateStorageDomainVDSCommand, log id: 4e53941a 2013-04-27 14:17:08,031 ERROR [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-33) Command org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:17:08,039 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-5-thread-33) Command [id=e01d2f93-6052-40bb-847f-1ca7fedbdb0c]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.storage_pool_iso_map; snapshot: EntityStatusSnapshot [id=storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, storageId = 025062dd-d69e-40ac-9031-c080b51647ae, status=InActive]. 2013-04-27 14:17:08,065 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) hostFromVds::selectedVds - BigServer, spmStatus Unknown_Pool, storage pool Default 2013-04-27 14:17:08,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-32) START, ConnectStoragePoolVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 2792d3ec 2013-04-27 14:18:11,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 304 mMessage Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:18:11,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (QuartzScheduler_Worker-32) Vds: BigServer 2013-04-27 14:18:11,976 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (QuartzScheduler_Worker-32) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:18:11,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-32) FINISH, ConnectStoragePoolVDSCommand, log id: 2792d3ec 2013-04-27 14:18:11,978 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (QuartzScheduler_Worker-32) transaction rolled back 2013-04-27 14:18:11,978 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) IrsBroker::Failed::GetStoragePoolInfoVDS 2013-04-27 14:18:11,978 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) Exception: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' 2013-04-27 14:18:12,017 INFO [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (QuartzScheduler_Worker-32) Running command: ReconstructMasterDomainCommand internal: true. Entities affected : ID: 025062dd-d69e-40ac-9031-c080b51647ae Type: Storage 2013-04-27 14:18:12,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) START, MarkPoolInReconstructModeVDSCommand(storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, ignoreFailoverLimit = false, compatabilityVersion = null, reconstructMarkAction = ClearCache), log id: 59cf4ee8 2013-04-27 14:18:12,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-32) clearing cache for problematic entities in pool 2b633f9a-8840-11e1-9ed7-17e6c491db84 2013-04-27 14:18:12,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.MarkPoolInReconstructModeVDSCommand] (QuartzScheduler_Worker-32) FINISH, MarkPoolInReconstructModeVDSCommand, log id: 59cf4ee8 2013-04-27 14:18:20,632 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) domain 503e6677-2d51-4668-ab17-f914bdb57c19 in problem. vds: BigServer 2013-04-27 14:18:20,633 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-83) domain d30604e3-3ca8-470a-8da4-18c9402f7504 in problem. vds: BigServer 2013-04-27 14:18:22,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-85) hostFromVds::selectedVds - BigServer, spmStatus Unknown_Pool, storage pool Default 2013-04-27 14:18:22,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (QuartzScheduler_Worker-85) START, ConnectStoragePoolVDSCommand(vdsId = aeac7aa6-8840-11e1-99aa-7fbd3e56e26e, storagePoolId = 2b633f9a-8840-11e1-9ed7-17e6c491db84, vds_spm_id = 1, masterDomainId = 025062dd-d69e-40ac-9031-c080b51647ae, masterVersion = 1), log id: 73e64219 2013-04-27 14:19:06,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) starting ProcessDomainRecovery for domain 503e6677-2d51-4668-ab17-f914bdb57c19 2013-04-27 14:19:06,778 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-6) domain 503e6677-2d51-4668-ab17-f914bdb57c19 was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. 2013-04-27 14:19:06,785 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-6) Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: 503e6677-2d51-4668-ab17-f914bdb57c19 Type: Storage 2013-04-27 14:19:06,794 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-6) CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL 2013-04-27 14:19:06,799 ERROR [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-6) Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand. 2013-04-27 14:19:06,800 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-6) No string for UNASSIGNED type. Use default Log 2013-04-27 14:19:06,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) starting ProcessDomainRecovery for domain d30604e3-3ca8-470a-8da4-18c9402f7504 2013-04-27 14:19:06,809 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-13) domain d30604e3-3ca8-470a-8da4-18c9402f7504 was reported by all hosts in status UP as problematic. Moving the Domain to NonOperational. 2013-04-27 14:19:06,816 INFO [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-13) Running command: HandleFailedStorageDomainCommand internal: true. Entities affected : ID: d30604e3-3ca8-470a-8da4-18c9402f7504 Type: Storage 2013-04-27 14:19:06,824 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (QuartzScheduler_Worker-13) CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL 2013-04-27 14:19:06,828 ERROR [org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand] (QuartzScheduler_Worker-13) Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.HandleFailedStorageDomainCommand. 2013-04-27 14:19:06,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (QuartzScheduler_Worker-13) No string for UNASSIGNED type. Use default Log 2013-04-27 14:19:11,388 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) domain 503e6677-2d51-4668-ab17-f914bdb57c19 in problem. vds: BigServer 2013-04-27 14:19:11,389 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (QuartzScheduler_Worker-17) domain d30604e3-3ca8-470a-8da4-18c9402f7504 in problem. vds: BigServer ------------------------------------------------------------------ And from vdsm.log: ------------------------------------------------------------------ Thread-1643::ERROR::2013-04-27 14:23:08,038::sdc::113::Storage.StorageDomainCache::(_findDomain) Error while looking for domain `025062dd-d69e-40ac-9031-c080b51647ae` Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 109, in _findDomain return mod.findDomain(sdUUID) File "/usr/share/vdsm/storage/nfsSD.py", line 149, in findDomain return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) File "/usr/share/vdsm/storage/fileSD.py", line 104, in __init__ sdUUID = metadata[sd.DMDK_SDUUID] File "/usr/share/vdsm/storage/persistentDict.py", line 75, in __getitem__ return dec(self._dict[key]) File "/usr/share/vdsm/storage/persistentDict.py", line 183, in __getitem__ with self._accessWrapper(): File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/share/vdsm/storage/persistentDict.py", line 137, in _accessWrapper self.refresh() File "/usr/share/vdsm/storage/persistentDict.py", line 214, in refresh lines = self._metaRW.readlines() File "/usr/share/vdsm/storage/fileSD.py", line 71, in readlines return misc.stripNewLines(self._oop.directReadLines(self._metafile)) File "/usr/share/vdsm/storage/processPool.py", line 53, in wrapper return self.runExternally(func, *args, **kwds) File "/usr/share/vdsm/storage/processPool.py", line 64, in runExternally return self._procPool.runExternally(*args, **kwargs) File "/usr/share/vdsm/storage/processPool.py", line 145, in runExternally raise Timeout("Operation Stuck") Timeout: Operation Stuck Thread-1643::DEBUG::2013-04-27 14:23:08,039::resourceManager::535::ResourceManager::(releaseResource) Trying to release resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' Thread-1643::DEBUG::2013-04-27 14:23:08,040::resourceManager::550::ResourceManager::(releaseResource) Released resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' (0 active users) Thread-1643::DEBUG::2013-04-27 14:23:08,041::resourceManager::555::ResourceManager::(releaseResource) Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free, finding out if anyone is waiting for it. Thread-1643::DEBUG::2013-04-27 14:23:08,041::resourceManager::562::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84', Clearing records. Thread-1643::ERROR::2013-04-27 14:23:08,042::task::855::TaskManager.Task::(_setError) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 863, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 721, in connectStoragePool return self._connectStoragePool(spUUID, hostID, scsiKey, msdUUID, masterVersion, options) File "/usr/share/vdsm/storage/hsm.py", line 763, in _connectStoragePool res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 624, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1097, in __rebuild self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1435, in getMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae' Thread-1643::DEBUG::2013-04-27 14:23:08,043::task::874::TaskManager.Task::(_run) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Task._run: 049a3ccf-e6d5-4363-b05b-90796a99fefb ('2b633f9a-8840-11e1-9ed7-17e6c491db84', 1, '2b633f9a-8840-11e1 -9ed7-17e6c491db84', '025062dd-d69e-40ac-9031-c080b51647ae', 1) {} failed - stopping task Thread-1643::DEBUG::2013-04-27 14:23:08,044::task::1201::TaskManager.Task::(stop) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::stopping in state preparing (force False) Thread-1643::DEBUG::2013-04-27 14:23:08,044::task::980::TaskManager.Task::(_decref) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::ref 1 aborting True Thread-1643::INFO::2013-04-27 14:23:08,045::task::1159::TaskManager.Task::(prepare) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::aborting: Task is aborted: 'Cannot find master domain' - code 304 Thread-1643::DEBUG::2013-04-27 14:23:08,045::task::1164::TaskManager.Task::(prepare) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Prepare: aborted: Cannot find master domain Thread-1643::DEBUG::2013-04-27 14:23:08,046::task::980::TaskManager.Task::(_decref) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::ref 0 aborting True Thread-1643::DEBUG::2013-04-27 14:23:08,046::task::915::TaskManager.Task::(_doAbort) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::Task._doAbort: force False Thread-1643::DEBUG::2013-04-27 14:23:08,047::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1643::DEBUG::2013-04-27 14:23:08,047::task::588::TaskManager.Task::(_updateState) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::moving from state preparing -> state aborting Thread-1643::DEBUG::2013-04-27 14:23:08,048::task::537::TaskManager.Task::(__state_aborting) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::_aborting: recover policy none Thread-1643::DEBUG::2013-04-27 14:23:08,048::task::588::TaskManager.Task::(_updateState) Task=`049a3ccf-e6d5-4363-b05b-90796a99fefb`::moving from state aborting -> state failed Thread-1643::DEBUG::2013-04-27 14:23:08,049::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-1643::DEBUG::2013-04-27 14:23:08,049::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1643::ERROR::2013-04-27 14:23:08,050::dispatcher::90::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=2b633f9a-8840-11e1-9ed7-17e6c491db84, msdUUID=025062dd-d69e-40ac-9031-c080b51647ae'", 'code': 304}} Thread-1688::DEBUG::2013-04-27 14:23:14,955::task::588::TaskManager.Task::(_updateState) Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::moving from state init -> state preparing Thread-1688::INFO::2013-04-27 14:23:14,956::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-1688::INFO::2013-04-27 14:23:14,956::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-1688::DEBUG::2013-04-27 14:23:14,957::task::1174::TaskManager.Task::(prepare) Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::finished: {} Thread-1688::DEBUG::2013-04-27 14:23:14,957::task::588::TaskManager.Task::(_updateState) Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::moving from state preparing -> state finished Thread-1688::DEBUG::2013-04-27 14:23:14,958::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-1688::DEBUG::2013-04-27 14:23:14,958::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1688::DEBUG::2013-04-27 14:23:14,959::task::980::TaskManager.Task::(_decref) Task=`5a33d6bc-3dba-4dea-8423-e242fa3809d3`::ref 0 aborting False Thread-1691::DEBUG::2013-04-27 14:23:18,148::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-1691::DEBUG::2013-04-27 14:23:18,149::task::588::TaskManager.Task::(_updateState) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::moving from state init -> state preparing Thread-1691::INFO::2013-04-27 14:23:18,149::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', options=None) Thread-1691::ERROR::2013-04-27 14:23:18,150::task::855::TaskManager.Task::(_setError) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 863, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 38, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 438, in getSpmStatus pool = self.getPool(spUUID) File "/usr/share/vdsm/storage/hsm.py", line 186, in getPool raise se.StoragePoolUnknown(spUUID) StoragePoolUnknown: Unknown pool id, pool not connected: ('2b633f9a-8840-11e1-9ed7-17e6c491db84',) Thread-1691::DEBUG::2013-04-27 14:23:18,151::task::874::TaskManager.Task::(_run) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Task._run: a0eebdaf-3a1c-4c6e-acc2-c41def4ec963 ('2b633f9a-8840-11e1-9ed7-17e6c491db84',) {} failed - stopping task Thread-1691::DEBUG::2013-04-27 14:23:18,151::task::1201::TaskManager.Task::(stop) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::stopping in state preparing (force False) Thread-1691::DEBUG::2013-04-27 14:23:18,152::task::980::TaskManager.Task::(_decref) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::ref 1 aborting True Thread-1691::INFO::2013-04-27 14:23:18,152::task::1159::TaskManager.Task::(prepare) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::aborting: Task is aborted: 'Unknown pool id, pool not connected' - code 309 Thread-1691::DEBUG::2013-04-27 14:23:18,153::task::1164::TaskManager.Task::(prepare) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Prepare: aborted: Unknown pool id, pool not connected Thread-1691::DEBUG::2013-04-27 14:23:18,153::task::980::TaskManager.Task::(_decref) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::ref 0 aborting True Thread-1691::DEBUG::2013-04-27 14:23:18,154::task::915::TaskManager.Task::(_doAbort) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::Task._doAbort: force False Thread-1691::DEBUG::2013-04-27 14:23:18,154::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1691::DEBUG::2013-04-27 14:23:18,155::task::588::TaskManager.Task::(_updateState) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::moving from state preparing -> state aborting Thread-1691::DEBUG::2013-04-27 14:23:18,155::task::537::TaskManager.Task::(__state_aborting) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::_aborting: recover policy none Thread-1691::DEBUG::2013-04-27 14:23:18,156::task::588::TaskManager.Task::(_updateState) Task=`a0eebdaf-3a1c-4c6e-acc2-c41def4ec963`::moving from state aborting -> state failed Thread-1691::DEBUG::2013-04-27 14:23:18,156::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-1691::DEBUG::2013-04-27 14:23:18,157::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1691::ERROR::2013-04-27 14:23:18,158::dispatcher::90::Storage.Dispatcher.Protect::(run) {'status': {'message': "Unknown pool id, pool not connected: ('2b633f9a-8840-11e1-9ed7-17e6c491db84',)", 'code': 309}} Thread-1692::DEBUG::2013-04-27 14:23:18,180::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-1692::DEBUG::2013-04-27 14:23:18,181::task::588::TaskManager.Task::(_updateState) Task=`e144930f-11a4-4be3-b7ba-5ff22981e7e0`::moving from state init -> state preparing Thread-1692::INFO::2013-04-27 14:23:18,182::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='2b633f9a-8840-11e1-9ed7-17e6c491db84', hostID=1, scsiKey='2b633f9a-8840-11e1-9ed7-17e6c491db84', msdUUID='025062dd-d69e-40ac-9031-c080b51647ae', masterVersion=1, options=None) Thread-1692::DEBUG::2013-04-27 14:23:18,183::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`1e46c60e-7d09-4bcd-bcd6-4f20118a3f84`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '747' at '_connectStoragePool' Thread-1692::DEBUG::2013-04-27 14:23:18,183::resourceManager::483::ResourceManager::(registerResource) Trying to register resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' for lock type 'exclusive' Thread-1692::DEBUG::2013-04-27 14:23:18,184::resourceManager::525::ResourceManager::(registerResource) Resource 'Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84' is free. Now locking as 'exclusive' (1 active user) Thread-1692::DEBUG::2013-04-27 14:23:18,184::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.2b633f9a-8840-11e1-9ed7-17e6c491db84`ReqID=`1e46c60e-7d09-4bcd-bcd6-4f20118a3f84`::Granted request Thread-1692::INFO::2013-04-27 14:23:18,185::sp::608::Storage.StoragePool::(connect) Connect host #1 to the storage pool 2b633f9a-8840-11e1-9ed7-17e6c491db84 with master domain: 025062dd-d69e-40ac-9031-c080b51647ae (ver = 1) Thread-1692::DEBUG::2013-04-27 14:23:18,186::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:18,187::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:18,188::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:18,188::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:18,189::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:18,190::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:18,190::misc::1017::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-1692::DEBUG::2013-04-27 14:23:18,191::misc::1019::SamplingMethod::(__call__) Got in to sampling method Thread-1692::DEBUG::2013-04-27 14:23:18,191::misc::1017::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-1692::DEBUG::2013-04-27 14:23:18,192::misc::1019::SamplingMethod::(__call__) Got in to sampling method Thread-1692::DEBUG::2013-04-27 14:23:18,192::iscsi::389::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) Thread-1692::DEBUG::2013-04-27 14:23:18,215::iscsi::389::Storage.Misc.excCmd::(rescan) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 Thread-1692::DEBUG::2013-04-27 14:23:18,216::misc::1027::SamplingMethod::(__call__) Returning last result Thread-1692::DEBUG::2013-04-27 14:23:19,820::multipath::71::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath' (cwd None) Thread-1692::DEBUG::2013-04-27 14:23:19,849::multipath::71::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-1692::DEBUG::2013-04-27 14:23:19,850::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,850::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,851::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,852::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,852::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,853::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,853::misc::1027::SamplingMethod::(__call__) Returning last result Thread-1692::DEBUG::2013-04-27 14:23:19,854::lvm::352::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,857::lvm::287::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%36782bcb063f75600162a25c119f14819%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 025062dd-d69e-40ac-9031-c080b51647ae' (cwd None) Thread-1692::DEBUG::2013-04-27 14:23:19,973::lvm::287::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found\n'; <rc> = 5 Thread-1692::WARNING::2013-04-27 14:23:19,975::lvm::356::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "025062dd-d69e-40ac-9031-c080b51647ae" not found'] Thread-1692::DEBUG::2013-04-27 14:23:19,976::lvm::379::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-1692::DEBUG::2013-04-27 14:23:19,996::fileSD::98::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.0.0.154:_home_vm_data_data/025062dd-d69e-40ac-9031-c080b51647ae Thread-1692::DEBUG::2013-04-27 14:23:19,998::persistentDict::175::Storage.PersistentDict::(__init__) Created a persistant dict with FileMetadataRW backend Thread-1699::DEBUG::2013-04-27 14:23:25,106::task::588::TaskManager.Task::(_updateState) Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::moving from state init -> state preparing Thread-1699::INFO::2013-04-27 14:23:25,107::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-1699::INFO::2013-04-27 14:23:25,107::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-1699::DEBUG::2013-04-27 14:23:25,108::task::1174::TaskManager.Task::(prepare) Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::finished: {} Thread-1699::DEBUG::2013-04-27 14:23:25,108::task::588::TaskManager.Task::(_updateState) Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::moving from state preparing -> state finished Thread-1699::DEBUG::2013-04-27 14:23:25,109::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-1699::DEBUG::2013-04-27 14:23:25,109::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1699::DEBUG::2013-04-27 14:23:25,110::task::980::TaskManager.Task::(_decref) Task=`3289175f-e99f-4625-9ee7-90b3f6cce1df`::ref 0 aborting False Thread-1705::DEBUG::2013-04-27 14:23:33,970::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-1705::DEBUG::2013-04-27 14:23:33,971::task::588::TaskManager.Task::(_updateState) Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::moving from state init -> state preparing Thread-1705::INFO::2013-04-27 14:23:33,972::logUtils::37::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.0.0.154:/home/vm/data/data', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '60e73855-daa2-45f5-9c59-5973ca89e830', 'port': ''}], options=None) Thread-1705::INFO::2013-04-27 14:23:33,972::storage_connection::188::Storage.ServerConnection::(validate) Request to validate NFS storage server Thread-1705::DEBUG::2013-04-27 14:23:33,973::mount::111::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6 10.0.0.154:/home/vm/data/data /tmp/tmp6s__VT' (cwd None) Thread-1705::DEBUG::2013-04-27 14:23:34,034::mount::111::Storage.Misc.excCmd::(_runcmd) '/usr/bin/sudo -n /bin/umount /tmp/tmp6s__VT' (cwd None) Thread-1705::INFO::2013-04-27 14:23:34,084::logUtils::39::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-1705::DEBUG::2013-04-27 14:23:34,085::task::1174::TaskManager.Task::(prepare) Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::finished: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-1705::DEBUG::2013-04-27 14:23:34,085::task::588::TaskManager.Task::(_updateState) Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::moving from state preparing -> state finished Thread-1705::DEBUG::2013-04-27 14:23:34,086::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-1705::DEBUG::2013-04-27 14:23:34,086::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1705::DEBUG::2013-04-27 14:23:34,087::task::980::TaskManager.Task::(_decref) Task=`a421e0e3-3c83-43c0-a867-5f76f34e7619`::ref 0 aborting False Thread-1706::DEBUG::2013-04-27 14:23:34,110::clientIF::261::Storage.Dispatcher.Protect::(wrapper) [10.0.0.154] Thread-1706::DEBUG::2013-04-27 14:23:34,111::task::588::TaskManager.Task::(_updateState) Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::moving from state init -> state preparing Thread-1706::INFO::2013-04-27 14:23:34,111::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '10.0.0.154:/home/vm/data/data', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '60e73855-daa2-45f5-9c59-5973ca89e830', 'port': ''}], options=None) Thread-1706::INFO::2013-04-27 14:23:34,112::storage_connection::146::Storage.ServerConnection::(connect) Request to connect NFS storage server Thread-1706::DEBUG::2013-04-27 14:23:34,123::lvm::460::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1706::DEBUG::2013-04-27 14:23:34,124::lvm::462::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1706::DEBUG::2013-04-27 14:23:34,125::lvm::472::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-1706::DEBUG::2013-04-27 14:23:34,125::lvm::474::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-1706::DEBUG::2013-04-27 14:23:34,126::lvm::493::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-1706::DEBUG::2013-04-27 14:23:34,127::lvm::495::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-1706::INFO::2013-04-27 14:23:34,127::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-1706::DEBUG::2013-04-27 14:23:34,128::task::1174::TaskManager.Task::(prepare) Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::finished: {'statuslist': [{'status': 0, 'id': '60e73855-daa2-45f5-9c59-5973ca89e830'}]} Thread-1706::DEBUG::2013-04-27 14:23:34,129::task::588::TaskManager.Task::(_updateState) Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::moving from state preparing -> state finished Thread-1706::DEBUG::2013-04-27 14:23:34,129::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-1706::DEBUG::2013-04-27 14:23:34,130::resourceManager::841::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-1706::DEBUG::2013-04-27 14:23:34,130::task::980::TaskManager.Task::(_decref) Task=`abbdddbc-3fa2-4799-b5d4-f902cd3a7ac7`::ref 0 aborting False Thread-1707::DEBUG::2013-04-27 14:23:35,251::task::588::TaskManager.Task::(_updateState) Task=`f6026160-a4cf-48a1-9f08-2c9da1cd6184`::moving from state init -> state preparing Thread-1707::INFO::2013-04-27 14:23:35,252::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-1707::INFO::2013-04-27 14:23:35,252::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-1707::DEBUG::2013-04-27 14:23:35,253::task::1174::TaskManager.Task::(prepare) Task=`f6026160-a4cf-48a1-9f08-2c9da1cd6184`::finished: {} Thread-1707::DEBUG::2013-04-27 14:23:35,253::task::588::TaskManager.Task::(_updateState) Task=`f6026160-a4cf-48a1-9f08-2c9da1cd6184`::moving from state preparing -> state finished Thread-1707::DEBUG::2013-04-27 14:23:35,254::resourceManager::806::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} ------------------------------------------------------------------

1 0

[Users] the procedure of storage-related
by bigclouds 27 Apr '13

27 Apr '13

------=_Part_344354_1639059946.1366972698511 Content-Type: text/plain; charset=GBK Content-Transfer-Encoding: 7bit hi,all 1.i am now not very familar with the right procedure of storage, please give me a simply introduction of their concept and relation. help me out StoragePool,StorageDomain image,volume 2.please help me confirm. i find only one place call mount command, which is poolConnectStorageServer. if poolConnectStorageServer is before create and active StoragePool,StorageDomain? 3.if attach is just modify metadata, not anything else? 4.sp.py, deactivateSD if i deactivate ISO storage,it finally umount masterDir, why ? if masterDir is not mastersd? thanks ------=_Part_344354_1639059946.1366972698511 Content-Type: text/html; charset=GBK Content-Transfer-Encoding: 7bit <div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial"><div>hi,all</div><div>1.i am now not very familar with the right procedure of storage, </div><div>    please give me a simply introduction of their concept and relation. help me out</div><div> </div><div>StoragePool,StorageDomain</div><div>image,volume</div><div> </div><div> </div><div>2.please help me confirm.</div><div>i find only one place call mount command, which is poolConnectStorageServer.</div><div>if poolConnectStorageServer is before create and active StoragePool,StorageDomain?</div><div> </div><div>3.if attach is just modify metadata, not anything else?</div><div> </div><div>4.sp.py, deactivateSD</div><div>  if i deactivate ISO storage,it finally umount masterDir, why  ?</div><div>  if masterDir is not  mastersd?</div><div>thanks</div></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span> ------=_Part_344354_1639059946.1366972698511--

1 0

[Users] VM Pool
by Marc Seward 26 Apr '13

26 Apr '13

In RHEV 3.1,when a VM from a pool is started on the User Portal,any changes made to it are discarded after it is shutdown.However,if it is started on the Admin Portal,any change made to it is persistent.Hence,the VM is stateless only when it is started on the User Portal. Is this behavior expected? -Marc

2 1

[Users] Master domain locked, error code 304
by Tommy McNeely 25 Apr '13

25 Apr '13

We had a hard crash (network, then power) on our 2 node Ovirt Cluster. We have NFS datastore on CentOS 6 (3.2.0-1.39.el6). We can no longer get the hosts to activate. They are unable to activate the "master" domain. The master storage domain show "locked" while the other storage domains show Unknown (disks) and inactive (ISO) All the domains are on the same NFS server, we are able to mount it, the permissions are good. We believe we might be getting bit by https://bugzilla.redhat.com/show_bug.cgi?id=920694or http://gerrit.ovirt.org/#/c/13709/ which says to cease working on it: Michael KublinApr 10 Patch Set 5: Do not submit Liron, please abondon this work. This interacts with host life cycle which will be changed, during a change a following problem will be solved as well. So, We were wondering what we can do to get our oVirt back online, or rather what the correct way is to solve this. We have a few VMs that are down which we are looking for ways to recover as quickly as possible. Thanks in advance, Tommy Here are the ovirt-engine logs: 2013-04-23 21:30:04,041 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-3-thread-49) Command ConnectStoragePoolVDS execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: 'spUUID=0f63de0e-7d98-48ce-99ec-add109f83c4f, msdUUID=774e3604-f449-4b3e-8c06-7cd16f98720c' 2013-04-23 21:30:04,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-3-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 50524b34 2013-04-23 21:30:04,049 WARN [org.ovirt.engine.core.bll.storage.ReconstructMasterDomainCommand] (pool-3-thread-49) [7c5867d6] CanDoAction of action ReconstructMasterDomain failed. Reasons:VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status Locked Here are the logs from vdsm: Thread-29::DEBUG::2013-04-23 21:36:05,906::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.101.0.148:/c/vpt1-vmdisks1 /rhev/data-center/mnt/10.101.0.148:_c_vpt1-vmdisks1' (cwd None) Thread-29::DEBUG::2013-04-23 21:36:06,008::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 10.101.0.148:/c/vpool-iso /rhev/data-center/mnt/10.101.0.148:_c_vpool-iso' (cwd None) Thread-29::INFO::2013-04-23 21:36:06,065::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '7c19bd42-c3dc-41b9-b81b-d9b75214b8dc'}, {'status': 0, 'id': 'eff2ef61-0b12-4429-b087-8742be17ae90'}]} Thread-29::DEBUG::2013-04-23 21:36:06,071::task::1151::TaskManager.Task::(prepare) Task=`48337e40-2446-4357-b6dc-2c86f4da67e2`::finished: {'statuslist': [{'status': 0, 'id': '7c19bd42-c3dc-41b9-b81b-d9b75214b8dc'}, {'status': 0, 'id': 'eff2ef61-0b12-4429-b087-8742be17ae90'}]} Thread-29::DEBUG::2013-04-23 21:36:06,071::task::568::TaskManager.Task::(_updateState) Task=`48337e40-2446-4357-b6dc-2c86f4da67e2`::moving from state preparing -> state finished Thread-29::DEBUG::2013-04-23 21:36:06,071::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-29::DEBUG::2013-04-23 21:36:06,072::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-29::DEBUG::2013-04-23 21:36:06,072::task::957::TaskManager.Task::(_decref) Task=`48337e40-2446-4357-b6dc-2c86f4da67e2`::ref 0 aborting False Thread-30::DEBUG::2013-04-23 21:36:06,112::BindingXMLRPC::161::vds::(wrapper) [10.101.0.197] Thread-30::DEBUG::2013-04-23 21:36:06,112::task::568::TaskManager.Task::(_updateState) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::moving from state init -> state preparing Thread-30::INFO::2013-04-23 21:36:06,113::logUtils::41::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='0f63de0e-7d98-48ce-99ec-add109f83c4f', hostID=1, scsiKey='0f63de0e-7d98-48ce-99ec-add109f83c4f', msdUUID='774e3604-f449-4b3e-8c06-7cd16f98720c', masterVersion=73, options=None) Thread-30::DEBUG::2013-04-23 21:36:06,113::resourceManager::190::ResourceManager.Request::(__init__) ResName=`Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f`ReqID=`ee74329a-0a92-465a-be50-b8acc6d7246a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '189' at '__init__' Thread-30::DEBUG::2013-04-23 21:36:06,114::resourceManager::504::ResourceManager::(registerResource) Trying to register resource 'Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f' for lock type 'exclusive' Thread-30::DEBUG::2013-04-23 21:36:06,114::resourceManager::547::ResourceManager::(registerResource) Resource 'Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f' is free. Now locking as 'exclusive' (1 active user) Thread-30::DEBUG::2013-04-23 21:36:06,114::resourceManager::227::ResourceManager.Request::(grant) ResName=`Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f`ReqID=`ee74329a-0a92-465a-be50-b8acc6d7246a`::Granted request Thread-30::INFO::2013-04-23 21:36:06,115::sp::625::Storage.StoragePool::(connect) Connect host #1 to the storage pool 0f63de0e-7d98-48ce-99ec-add109f83c4f with master domain: 774e3604-f449-4b3e-8c06-7cd16f98720c (ver = 73) Thread-30::DEBUG::2013-04-23 21:36:06,116::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-30::DEBUG::2013-04-23 21:36:06,116::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-30::DEBUG::2013-04-23 21:36:06,117::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-30::DEBUG::2013-04-23 21:36:06,117::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-30::DEBUG::2013-04-23 21:36:06,117::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-30::DEBUG::2013-04-23 21:36:06,118::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-30::DEBUG::2013-04-23 21:36:06,118::misc::1054::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-30::DEBUG::2013-04-23 21:36:06,118::misc::1056::SamplingMethod::(__call__) Got in to sampling method Thread-30::DEBUG::2013-04-23 21:36:06,119::misc::1054::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-30::DEBUG::2013-04-23 21:36:06,119::misc::1056::SamplingMethod::(__call__) Got in to sampling method Thread-30::DEBUG::2013-04-23 21:36:06,119::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) Thread-30::DEBUG::2013-04-23 21:36:06,136::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 Thread-30::DEBUG::2013-04-23 21:36:06,136::misc::1064::SamplingMethod::(__call__) Returning last result MainProcess|Thread-30::DEBUG::2013-04-23 21:36:06,139::misc::84::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host0/scan' (cwd None) MainProcess|Thread-30::DEBUG::2013-04-23 21:36:06,142::misc::84::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host1/scan' (cwd None) MainProcess|Thread-30::DEBUG::2013-04-23 21:36:06,146::misc::84::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host2/scan' (cwd None) MainProcess|Thread-30::DEBUG::2013-04-23 21:36:06,149::iscsi::402::Storage.ISCSI::(forceIScsiScan) Performing SCSI scan, this will take up to 30 seconds Thread-30::DEBUG::2013-04-23 21:36:08,152::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/multipath' (cwd None) Thread-30::DEBUG::2013-04-23 21:36:08,254::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-30::DEBUG::2013-04-23 21:36:08,256::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,256::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,257::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,257::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,258::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,258::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,258::misc::1064::SamplingMethod::(__call__) Returning last result Thread-30::DEBUG::2013-04-23 21:36:08,259::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,261::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 774e3604-f449-4b3e-8c06-7cd16f98720c' (cwd None) Thread-30::DEBUG::2013-04-23 21:36:08,514::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ' Volume group "774e3604-f449-4b3e-8c06-7cd16f98720c" not found\n'; <rc> = 5 Thread-30::WARNING::2013-04-23 21:36:08,516::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' Volume group "774e3604-f449-4b3e-8c06-7cd16f98720c" not found'] Thread-30::DEBUG::2013-04-23 21:36:08,518::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-30::DEBUG::2013-04-23 21:36:08,524::resourceManager::557::ResourceManager::(releaseResource) Trying to release resource 'Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f' Thread-30::DEBUG::2013-04-23 21:36:08,525::resourceManager::573::ResourceManager::(releaseResource) Released resource 'Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f' (0 active users) Thread-30::DEBUG::2013-04-23 21:36:08,525::resourceManager::578::ResourceManager::(releaseResource) Resource 'Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f' is free, finding out if anyone is waiting for it. Thread-30::DEBUG::2013-04-23 21:36:08,525::resourceManager::585::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.0f63de0e-7d98-48ce-99ec-add109f83c4f', Clearing records. Thread-30::ERROR::2013-04-23 21:36:08,526::task::833::TaskManager.Task::(_setError) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 42, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 926, in connectStoragePool masterVersion, options) File "/usr/share/vdsm/storage/hsm.py", line 973, in _connectStoragePool res = pool.connect(hostID, scsiKey, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 642, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1166, in __rebuild self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1505, in getMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=0f63de0e-7d98-48ce-99ec-add109f83c4f, msdUUID=774e3604-f449-4b3e-8c06-7cd16f98720c' Thread-30::DEBUG::2013-04-23 21:36:08,527::task::852::TaskManager.Task::(_run) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::Task._run: f551fa3f-9d8c-4de3-895a-964c821060d4 ('0f63de0e-7d98-48ce-99ec-add109f83c4f', 1, '0f63de0e-7d98-48ce-99ec-add109f83c4f', '774e3604-f449-4b3e-8c06-7cd16f98720c', 73) {} failed - stopping task Thread-30::DEBUG::2013-04-23 21:36:08,528::task::1177::TaskManager.Task::(stop) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::stopping in state preparing (force False) Thread-30::DEBUG::2013-04-23 21:36:08,528::task::957::TaskManager.Task::(_decref) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::ref 1 aborting True Thread-30::INFO::2013-04-23 21:36:08,528::task::1134::TaskManager.Task::(prepare) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::aborting: Task is aborted: 'Cannot find master domain' - code 304 Thread-30::DEBUG::2013-04-23 21:36:08,529::task::1139::TaskManager.Task::(prepare) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::Prepare: aborted: Cannot find master domain Thread-30::DEBUG::2013-04-23 21:36:08,529::task::957::TaskManager.Task::(_decref) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::ref 0 aborting True Thread-30::DEBUG::2013-04-23 21:36:08,529::task::892::TaskManager.Task::(_doAbort) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::Task._doAbort: force False Thread-30::DEBUG::2013-04-23 21:36:08,530::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-30::DEBUG::2013-04-23 21:36:08,530::task::568::TaskManager.Task::(_updateState) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::moving from state preparing -> state aborting Thread-30::DEBUG::2013-04-23 21:36:08,530::task::523::TaskManager.Task::(__state_aborting) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::_aborting: recover policy none Thread-30::DEBUG::2013-04-23 21:36:08,531::task::568::TaskManager.Task::(_updateState) Task=`f551fa3f-9d8c-4de3-895a-964c821060d4`::moving from state aborting -> state failed Thread-30::DEBUG::2013-04-23 21:36:08,531::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-30::DEBUG::2013-04-23 21:36:08,531::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-30::ERROR::2013-04-23 21:36:08,532::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Cannot find master domain: 'spUUID=0f63de0e-7d98-48ce-99ec-add109f83c4f, msdUUID=774e3604-f449-4b3e-8c06-7cd16f98720c'", 'code': 304}} [root@vmserver3 vdsm]#

4 5

[Users] 102 VDSM processes ?
by Alex Leonhardt 24 Apr '13

24 Apr '13

This is a multi-part message in MIME format. --------------080207050600010608050805 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, I'm wondering why my Hyper-Visor is running 102 vdsm processes, each using ~20mb = 2GB used ?? Is this expected and normal ?? Thanks, Alex -- www.vcore.co | www.vsearchcloud.com | Follow @www_vcore_co <https://twitter.com/www_vcore_co> --------------080207050600010608050805 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body bgcolor="#FFFFFF" text="#000000"> <font size="-1"><font face="Tahoma">Hi,<br> <br> I'm wondering why my Hyper-Visor is running 102 vdsm processes, each using ~20mb = 2GB used ?? Is this expected and normal ?? <br> <br> Thanks,<br> Alex<br> <br> </font></font> <div class="moz-signature">-- <br> <a class="moz-txt-link-abbreviated" href="http://www.vcore.co">www.vcore.co</a> | <a class="moz-txt-link-abbreviated" href="http://www.vsearchcloud.com">www.vsearchcloud.com</a> | <a href="https://twitter.com/www_vcore_co" class="twitter-follow-button" data-show-count="false">Follow @www_vcore_co</a> <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0];if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src="//platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script> <br> </div> </body> </html> --------------080207050600010608050805--

3 2

[Users] oVirt Weekly Meeting Minutes -- 2013-04-24
by Ofer Schreiber 24 Apr '13

24 Apr '13

Minutes: http://ovirt.org/meetings/ovirt/2013/ovirt.2013-04-24-14.04.html Minutes (text): http://ovirt.org/meetings/ovirt/2013/ovirt.2013-04-24-14.04.txt Log: http://ovirt.org/meetings/ovirt/2013/ovirt.2013-04-24-14.04.log.html ============================ #ovirt: oVirt Weekly Meeting ============================ Meeting started by oschreib at 14:04:11 UTC. The full logs are available at http://ovirt.org/meetings/ovirt/2013/ovirt.2013-04-24-14.04.log.html . Meeting summary --------------- * agenda and roll call (oschreib, 14:04:36) * Release Updates (oschreib, 14:04:46) * Conferences and Workshops (oschreib, 14:04:51) * Infra team update (oschreib, 14:04:56) * Other Topics (oschreib, 14:05:02) * oVirt 3.2.2 Update (oschreib, 14:07:31) * : Update should be out next week (oschreib, 14:08:15) * ACTION: oschreib to coordinate 3.2.2 for next week (2013-05-01) (oschreib, 14:08:55) * oVirt 3.3 Release (oschreib, 14:10:21) * http link://www.ovirt.org/OVirt_3.3_release-management (oschreib, 14:10:36) * Next milestone (oschreib, 14:10:47) * Feature freeze and beta release: 2013-05-31 (oschreib, 14:10:50) * scheduling: major re-factoring in 3.3. new API won't be in. (oschreib, 14:16:31) * Conferences and Workshops (oschreib, 14:18:15) * Shanghai workshop coming up in a few weeks, contact theron if you have questions (oschreib, 14:19:01) * Infra team update (oschreib, 14:22:09) * infra team is working on reducing SPOF's (oschreib, 14:24:22) * a bit more space on ovirt web site for builds (oschreib, 14:25:05) * Additional topics (oschreib, 14:27:37) Meeting ended at 14:35:03 UTC. Action Items ------------ * oschreib to coordinate 3.2.2 for next week (2013-05-01) Action Items, by person ----------------------- * oschreib * oschreib to coordinate 3.2.2 for next week (2013-05-01) People Present (lines said) --------------------------- * oschreib (54) * evilissimo (10) * ofri_ (9) * Rydekull (9) * loreste (7) * verwilst (4) * ovirtbot (2) * dustins (1) Generated by `MeetBot`_ 0.1.4 .. _`MeetBot`: http://wiki.debian.org/MeetBot

1 0

[Users] Unable to save host networking: Network address must be specified when using static ip
by Ernest Beinrohr 24 Apr '13

24 Apr '13

This is a multi-part message in MIME format. --------------020605030005030009010403 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hi, I have a couple of hypervisors running 3.2 on centos6 and have set up the bridges and VLANs by myself. Everything works, my VMs run, but for some reason i cant save my "Setup Host Networks". Every vlan and bridge interface has a network specified in /etc/sysconfig/networks-scripts/ifcfg-* and some do not have an IP address. This is the error I get: Error: finwe: * Network address must be specified when using static ip -- Ernest Beinrohr, AXON PRO Ing <http://www.beinrohr.sk/ing.php>, RHCE <http://www.beinrohr.sk/rhce.php>, RHCVA <http://www.beinrohr.sk/rhce.php>, LPIC <http://www.beinrohr.sk/lpic.php>, +421-2--6241-0360 <callto://+421-2--6241-0360>, +421-903--482-603 <callto://+421-903--482-603> icq:28153343, skype:oernii-work <callto://oernii-work>, jabber:oernii@jabber.org ------------------------------------------------------------------------ “The bureaucracy is expanding to meet the needs of the expanding bureaucracy.” ― Oscar Wilde --------------020605030005030009010403 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hi, I have a couple of hypervisors running 3.2 on centos6 and have set up the bridges and VLANs by myself. Everything works, my VMs run, but for some reason i cant save my "Setup Host Networks". Every vlan and bridge interface has a network specified in /etc/sysconfig/networks-scripts/ifcfg-* and some do not have an IP address.<br> <br> <br> This is the error I get:<br> <br> <br> <div class="GE2Q1TJFSB"> <div class="GE2Q1TJJHC GE2Q1TJGSB"> <div class="gwt-HTML">Error: <br> <br> finwe: <ul style="margin-top:0"> <li>Network address must be specified when using static ip</li> </ul> </div> </div> </div> <br> <br> <div class="moz-signature">-- <br> <div id="oernii_footer" style="color: gray;"> <span style="font-family: Lucida Console, Luxi Mono, Courier, monospace; font-size: 90%;"> Ernest Beinrohr, AXON PRO<br> <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/ing.php">Ing</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCE</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/rhce.php">RHCVA</a>, <a style="text-decoration: none; color: gray;" href="http://www.beinrohr.sk/lpic.php">LPIC</a>, <a style="text-decoration: none; color: gray;" href="callto://+421-2--6241-0360">+421-2--6241-0360</a>, <a style="text-decoration: none; color: gray;" href="callto://+421-903--482-603">+421-903--482-603</a><br> icq:28153343, <a style="text-decoration: none; color: gray;" href="callto://oernii-work">skype:oernii-work</a>, <a class="moz-txt-link-abbreviated" href="mailto:jabber:oernii@jabber.org">jabber:oernii@jabber.org</a> <br> </span> <hr style="height: 1px; width: 95%"> “The bureaucracy is expanding to meet the needs of the expanding bureaucracy.” ― Oscar Wilde </div> </div> </body> </html> --------------020605030005030009010403--

2 1

[Users] Server 2k8
by Ryan Wilkinson 24 Apr '13

24 Apr '13

Are spice guest tools that will work with Server 2k8 available - especially video drivers?

5 7

[Users] Unplanned Outage :: Mailman and resources :: 2013-04-24 09:00 UTC
by Karsten 'quaid' Wade 24 Apr '13

24 Apr '13

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2WJIFUSSGTDNRKHTENMGS Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable There was an unplanned outage of lists.ovirt.org/resources.ovirt.org for 30 minutes. =3D=3D Details =3D=3D We began receiving disk full alerts from Postfix on linode01.ovirt.org, that houses lists.ovirt.org and resources.ovirt.org. I looked around the server to find more disk space, but there was nothing obvious that I could drop. Regardless, it's clear from existing needs that more disk space was needed. I shutdown the server, added 12 GB more to the drive image, and restarted the host. =3D=3D Affected services =3D=3D * Mailman (lists.ovirt.org) * Package repo (resources.ovirt.org) * ovirtbot =3D=3D=3D Not-affected services =3D=3D * www.ovirt.org * wiki.ovirt.org * jenkins.ovirt.org * gerrit.ovirt.org =3D=3D Future plans =3D=3D Migrate the entire host to a set of VMs running in one of the new hosting environments. ASAP. --=20 Karsten 'quaid' Wade, Sr. Analyst - Community Growth http://TheOpenSourceWay.org .^\ http://community.redhat.com @quaid (identi.ca/twitter/IRC) \v' gpg: AD0E0C41 ------enig2WJIFUSSGTDNRKHTENMGS Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iD8DBQFRd7PX2ZIOBq0ODEERAgdQAKDXjB4JbDFEKhA+wDcd472DitW1UwCg0aw7 4W0c7Bci4/QqnXpwrOHwx08= =FcL7 -----END PGP SIGNATURE----- ------enig2WJIFUSSGTDNRKHTENMGS--

1 0

[Users] oVirt 3.2.1 - virt-v2v import problem
by Piotr Szubiakowski 24 Apr '13

24 Apr '13

Hi, I have problem with virt-v2v tool. I'm trying to import XEN image. On oVirt 3.0 the import works well. On oVirt 3.2.1 there is an error: virt-v2v -i libvirtxml -o rhev -os localhost:/var/ovirt/export api.xml arch-dev.img: 100% [==============================================================================================================================================================================================================]D 0h02m53s Please enter your authentication name: vdsm@rhevh Please enter your password: could not connect to libvirt (URI = NULL): authentication failed: authentication failed [code=45 domain=7] at /usr/share/perl5/vendor_perl/Sys/VirtConvert/GuestfsHandle.pm line 96. It looks like after image upload, virt-v2v ties to connect to libvirtd. The vdsm@rhevh login with shibboleth password doesn't work. The oVirt is installed on Fedora 18. Engine and vdsmd runs on the same host. Is there any difference between authentication to libvirt on oVirt 3.0 and oVirt 3.2? Could I ask you for help with this problem? Best regards, Piotr

3 4

Re: [Users] Wrong Master domain or its version
by Michael Kublin 24 Apr '13

24 Apr '13

----- Original Message ----- > From: "Jonas Israelsson" <jonas.israelsson(a)elementary.se> > To: "Michael Kublin" <mkublin(a)redhat.com> > Cc: users(a)ovirt.org > Sent: Tuesday, April 23, 2013 3:04:17 PM > Subject: Re: [Users] Wrong Master domain or its version > > > Hi, I suppose you have one host, if you will take a look to > > rhev-engine logs you will see > > kind of the following error: Cannot connect to Pool wrong master > > domain or it version. > > One host (only), yes. > > > The only advice is to access to DB and update msater domain version > > inside storage_pool table. > > The query should be "update storage_pool set > > master_domain_version=110" > > after that please try to activate a host again > > Indeed, that seem to do the trick (thanks!). Setting the version in the > database to match the one from the log does make me able to bring the > host back online > > While the host came back online as well as all other storage domain the > master domain did not. I remains in status locked. Part from not being > able to access the vm-data from that domain it also gives some funny > side effects. For instance I could quite happily put the export-domain > to maintenance mode but not re-attach the same since the master domain > is not active. > If domains are marked as Locked in GUI they be also unlocked in DB. My advice is to put host in maintainence, after that please run the following query : update storage_pool_iso_map set status = 0 (The following table also contains storage_pool_id and storage_id, so you can make query more specific, info about domains is located inside storage_domain_static table) Activate a host, after that host should try to connect to all storages and to pool again. > Not sure if related, but when trying to also take the host into > maintenance mode it refuses due to ongoing tasks. > > [root@fat-tony admin]# vdsClient -s 0 getAllTasks > 4ca8e2f9-b8a6-425d-baba-b3cb9a6a154f : > verb = createVolume > code = 0 > state = failed > tag = spm > result = > message = Task is initializing > id = 4ca8e2f9-b8a6-425d-baba-b3cb9a6a154f > > I did manage to get rid of that with 'vdsClient clearTask' but it makes > no difference putting the host in and out of maintenance mode and or > rebooting it. The lock remains. > Task are left in host and not located in engine it is a known issue, and I think that engine should handle such problem if domains was not in status Locked > I fail to find any information where this lock is set. Is is done by the > vdsm-layer and therefore to be controlled by the... > > 'vdsClient aquire|releaseDomainLock' .. ? > > That however seem to fail as well > > vdsClient -s 0 releaseDomainLock dae1ba5e-548a-473e-94b6-7d08d3a6229a > ea5643f7-f41f-4c44-a27f-f31240cd0f19 > Traceback (most recent call last): > File "/usr/share/vdsm/vdsClient.py", line 2411, in <module> > File "/usr/share/vdsm/vdsClient.py", line 1381, in releaseDomainLock > File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ > File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request > File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request > File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request > File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response > File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close > Fault: <Fault 1: '<type \'exceptions.Exception\'>:method > "releaseDomainLock" is not supported'> > > > Could someone please point me in the right direction ? > > Rgds Jonas >

1 1

[Users] AD authentication for ovirt manager
by Jonathan Horne 24 Apr '13

24 Apr '13

--_000_9BE6F493F83A594DA60C45E6A09DC5AC2EFD3FF5AUSP01DAG0201_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Is there a write up out there for setting up ovirt users and adminstrators = to authenticate into the portal via AD? Thanks, Jonathan ________________________________ This is a PRIVATE message. If you are not the intended recipient, please de= lete without copying and kindly advise us by e-mail of the mistake in deliv= ery. NOTE: Regardless of content, this e-mail shall not operate to bind SKO= POS to any order or other contract unless pursuant to explicit written agre= ement or government initiative expressly permitting the use of e-mail for s= uch purpose. --_000_9BE6F493F83A594DA60C45E6A09DC5AC2EFD3FF5AUSP01DAG0201_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <style>  </style> </head> <body lang=3D"EN-US" link=3D"blue" vlink=3D"purple"> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Is there a write up out there for setting up ovirt u= sers and adminstrators to authenticate into the portal via AD?</p> <p class=3D"MsoNormal"> </p> <p class=3D"MsoNormal">Thanks,</p> <p class=3D"MsoNormal">Jonathan </p> <p class=3D"MsoNormal"> </p> </div> <br> <hr> <font color=3D"Gray" face=3D"Arial" size=3D"1">This is a PRIVATE message. I= f you are not the intended recipient, please delete without copying and kin= dly advise us by e-mail of the mistake in delivery. NOTE: Regardless of con= tent, this e-mail shall not operate to bind SKOPOS to any order or other contract unless pursuant to explicit wri= tten agreement or government initiative expressly permitting the use of e-m= ail for such purpose.</font> </body> </html> --_000_9BE6F493F83A594DA60C45E6A09DC5AC2EFD3FF5AUSP01DAG0201_--

6 6

[Users] NFS ISO DOmain
by suporte＠logicworks.pt 23 Apr '13

23 Apr '13

------=_Part_1372_26712946.1365527645345 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, I just installed the last version of engine and host using fedora18. I notice that in the engine the nfs server is not up and running, so is not possible to attach the iso domain. How can I start the nfs server on engine? Is this the correct thing to do? vi /etc/idmapd.conf # line 5: uncomment and change to your domain name Domain =server.world [root@dlp ~]# systemctl start rpcbind.service [root@dlp ~]# systemctl start nfs-server.service [root@dlp ~]# systemctl start nfs-lock.service [root@dlp ~]# systemctl start nfs-idmap.service [root@dlp ~]# systemctl enable rpcbind.service [root@dlp ~]# systemctl enable nfs-server.service [root@dlp ~]# systemctl enable nfs-lock.service [root@dlp ~]# systemctl enable nfs-idmap.service -- Jose Ferradeira http://www.logicworks.pt ------=_Part_1372_26712946.1365527645345 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'>Hi,<br><br>I just installed the last version of engine and host using fedora18. I notice that in the engine the nfs server is not up and running, so is not possible to attach the iso domain.<br>How can I start the nfs server on engine?<br>Is this the correct thing to do?<br><br> <div class="color1"><a href="http://www.server-world.info/en/command/html/vi.html" style="color: rgb(255, 255, 0);">vi</a> /etc/idmapd.conf </div> <div class="block"> <div class="color2"># line 5: uncomment and change to your domain name</div><br> Domain =server.world </div><br>[root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> start rpcbind.service <br> [root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> start nfs-server.service <br> [root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> start nfs-lock.service <br> [root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> start nfs-idmap.service <br> [root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> enable rpcbind.service <br> [root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> enable nfs-server.service <br> [root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> enable nfs-lock.service <br> [root@dlp ~]#<a href="http://www.server-world.info/en/command/html/systemctl.html" style="color: rgb(255, 255, 0);"> systemctl</a> enable nfs-idmap.service <br><br>-- <br><div><span name="x"></span><hr style="width: 100%; height: 2px;">Jose Ferradeira<br>http://www.logicworks.pt<br><span name="x"></span><br></div></div></body></html> ------=_Part_1372_26712946.1365527645345--

4 7

[Users] Wrong Master domain or its version
by Jonas Israelsson 23 Apr '13

23 Apr '13

Greetings. This post is related to my previous post (http://lists.ovirt.org/pipermail/users/2013-April/013912.html) but with some more investigation and findings. When trying to activate my host/node it goes back to non-operational since it's unable to attach the master storage domain. A few snippet from the log (whole log attached): Thread-12399::ERROR::2013-04-22 11:02:17,627::sp::1519::Storage.StoragePool::(getMasterDomain) Requested master domain ea5643f7-f41f-4c44-a27f-f31240cd0f19 does not have expected version 110 it is version 1 Thread-12399::INFO::2013-04-22 11:02:17,630::task::1134::TaskManager.Task::(prepare) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::aborting: Task is aborted: 'Wrong Master domain or its version' - code 324 Thread-12399::DEBUG::2013-04-22 11:02:17,630::task::1139::TaskManager.Task::(prepare) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::Prepare: aborted: Wrong Master domain or its version Not sure if "does not have expected version 110 it is version 1" is a red herring or if this is actual problem. I get the following output from vdsClient, where no version seem to match versions from the log. vdsClient -s 0 getStorageDomainInfo ea5643f7-f41f-4c44-a27f-f31240cd0f19 uuid = ea5643f7-f41f-4c44-a27f-f31240cd0f19 vguuid = 5Qox85-NQCz-pyVM-aFRN-KJ2P-jB1T-IiETQr lver = -1 state = OK version = 3 role = Master pool = ['dae1ba5e-548a-473e-94b6-7d08d3a6229a'] spm_id = -1 type = ISCSI class = Data master_ver = 0 name = hinken Found these that seem somewhat similar: https://bugzilla.redhat.com/show_bug.cgi?id=784038 http://lists.ovirt.org/pipermail/users/2012-December/010982.html My whole site is dead now due to this, so any help to the resolve the issue would be greatly appreciated. With kind regards Jonas vdsm.log Thread-12388::DEBUG::2013-04-22 11:02:09,292::BindingXMLRPC::913::vds::(wrapper) client [46.22.124.43]::call getCapabilities with () {} Thread-12388::DEBUG::2013-04-22 11:02:09,349::BindingXMLRPC::920::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:a07da5fa539'}], 'FC': []}, 'packages2': {'kernel': {'release': '205.fc18.x86_64', 'buildtime': 1361736602.0, 'version': '3.7.9'}, 'spice-server': {'release': '3.fc18', 'buildtime': 1358784016L, 'version': '0.12.2'}, 'vdsm': {'release': '9.fc18', 'buildtime': 1362046220L, 'version': '4.10.3'}, 'qemu-kvm': {'release': '6.fc18', 'buildtime': 1359843954L, 'version': '1.2.2'}, 'libvirt': {'release': '1.fc18', 'buildtime': 1359405439L, 'version': '0.10.2.3'}, 'qemu-img': {'release': '6.fc18', 'buildtime': 1359843954L, 'version': '1.2.2'}, 'mom': {'release': '1.fc18', 'buildtime': 1349470214L, 'version': '0.3.0'}}, 'cpuModel': 'AMD Opteron(TM) Processor 6276', 'hooks': {}, 'cpuSockets': '4', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '46.22.124.37', 'cfg': {'IPADDR': '46.22.124.37', 'PEERDNS': 'no', 'GATEWAY': '46.22.124.36', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.224', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.224', 'stp': 'off', 'bridged': True, 'gateway': '46.22.124.36', 'ports': ['em1']}, 'san': {'iface': 'san', 'addr': '192.168.43.11', 'cfg': {'IPADDR': '192.168.43.11', 'PEERDNS': 'no', 'MTU': '9000', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'san', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '9000', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '0.0.0.0', 'ports': ['em2']}, 'vminternet': {'iface': 'vminternet', 'addr': '', 'cfg': {'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vminternet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'gateway': '0.0.0.0', 'ports': ['em4']}}, 'bridges': {'ovirtmgmt': {'addr': '46.22.124.37', 'cfg': {'IPADDR': '46.22.124.37', 'PEERDNS': 'no', 'GATEWAY': '46.22.124.36', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.224', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.224', 'stp': 'off', 'ports': ['em1']}, 'san': {'addr': '192.168.43.11', 'cfg': {'IPADDR': '192.168.43.11', 'PEERDNS': 'no', 'MTU': '9000', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'san', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '9000', 'netmask': '255.255.255.0', 'stp': 'off', 'ports': ['em2']}, 'vminternet': {'addr': '', 'cfg': {'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vminternet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': ['em4']}}, 'uuid': '4C4C4544-0042-4710-8030-B3C04F37354A', 'lastClientIface': 'ovirtmgmt', 'nics': {'em4': {'addr': '', 'cfg': {'BRIDGE': 'vminternet', 'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'HWADDR': 'd0:67:e5:f9:2e:20', 'STP': 'no', 'DEVICE': 'em4', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:20', 'speed': 100}, 'em1': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'HWADDR': 'd0:67:e5:f9:2e:1a', 'STP': 'no', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:1a', 'speed': 1000}, 'em3': {'addr': '', 'cfg': {'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'HWADDR': 'd0:67:e5:f9:2e:1e', 'STP': 'no', 'DEVICE': 'em3', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:1e', 'speed': 0}, 'em2': {'addr': '', 'cfg': {'BRIDGE': 'san', 'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'MTU': '9000', 'HWADDR': 'd0:67:e5:f9:2e:1c', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'mtu': '9000', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:1c', 'speed': 1000}}, 'software_revision': '9', 'clusterLevels': ['3.0', '3.1', '3.2'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,ht,syscall,nx,mmxext,fxsr_opt,pdpe1gb,rdtscp,lm,constant_tsc,rep_good,nopl,nonstop_tsc,extd_apicid,amd_dcm,aperfmperf,pni,pclmulqdq,monitor,ssse3,cx16,sse4_1,sse4_2,popcnt,aes,xsave,avx,lahf_lm,cmp_legacy,svm,extapic,cr8_legacy,abm,sse4a,misalignsse,3dnowprefetch,osvw,ibs,xop,skinit,wdt,lwp,fma4,nodeid_msr,topoext,perfctr_core,arat,cpb,hw_pstate,npt,lbrv,svm_lock,nrip_save,tsc_scale,vmcb_clean,flushbyasid,decodeassists,pausefilter,pfthreshold,model_Opteron_G3,model_Opteron_G1,model_Opteron_G4,model_Opteron_G2', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:a07da5fa539', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1'], 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'memSize': '64210', 'cpuSpeed': '2299.953', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '32', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '', 'cpuThreads': '32', 'emulatedMachines': [u'pc-1.2', u'none', u'pc', u'pc-1.1', u'pc-1.0', u'pc-0.15', u'pc-0.14', u'pc-0.13', u'pc-0.12', u'pc-0.11', u'pc-0.10', u'isapc', u'pc-1.2', u'none', u'pc', u'pc-1.1', u'pc-1.0', u'pc-0.15', u'pc-0.14', u'pc-0.13', u'pc-0.12', u'pc-0.11', u'pc-0.10', u'isapc'], 'operatingSystem': {'release': '20120228.fc18', 'version': '2.6.1', 'name': 'oVirt Node'}, 'lastClient': '46.22.124.43'}} Thread-12389::DEBUG::2013-04-22 11:02:09,364::BindingXMLRPC::913::vds::(wrapper) client [46.22.124.43]::call getHardwareInfo with () {} Thread-12389::DEBUG::2013-04-22 11:02:09,365::BindingXMLRPC::920::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R715', 'systemSerialNumber': '3BG075J', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0042-4710-8030-B3C04F37354A', 'systemManufacturer': 'Dell Inc.'}} Thread-12390::DEBUG::2013-04-22 11:02:11,238::BindingXMLRPC::913::vds::(wrapper) client [46.22.124.43]::call getCapabilities with () {} Thread-12390::DEBUG::2013-04-22 11:02:11,296::BindingXMLRPC::920::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:a07da5fa539'}], 'FC': []}, 'packages2': {'kernel': {'release': '205.fc18.x86_64', 'buildtime': 1361736602.0, 'version': '3.7.9'}, 'spice-server': {'release': '3.fc18', 'buildtime': 1358784016L, 'version': '0.12.2'}, 'vdsm': {'release': '9.fc18', 'buildtime': 1362046220L, 'version': '4.10.3'}, 'qemu-kvm': {'release': '6.fc18', 'buildtime': 1359843954L, 'version': '1.2.2'}, 'libvirt': {'release': '1.fc18', 'buildtime': 1359405439L, 'version': '0.10.2.3'}, 'qemu-img': {'release': '6.fc18', 'buildtime': 1359843954L, 'version': '1.2.2'}, 'mom': {'release': '1.fc18', 'buildtime': 1349470214L, 'version': '0.3.0'}}, 'cpuModel': 'AMD Opteron(TM) Processor 6276', 'hooks': {}, 'cpuSockets': '4', 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'ovirtmgmt': {'iface': 'ovirtmgmt', 'addr': '46.22.124.37', 'cfg': {'IPADDR': '46.22.124.37', 'PEERDNS': 'no', 'GATEWAY': '46.22.124.36', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.224', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.224', 'stp': 'off', 'bridged': True, 'gateway': '46.22.124.36', 'ports': ['em1']}, 'san': {'iface': 'san', 'addr': '192.168.43.11', 'cfg': {'IPADDR': '192.168.43.11', 'PEERDNS': 'no', 'MTU': '9000', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'san', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '9000', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '0.0.0.0', 'ports': ['em2']}, 'vminternet': {'iface': 'vminternet', 'addr': '', 'cfg': {'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vminternet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'bridged': True, 'gateway': '0.0.0.0', 'ports': ['em4']}}, 'bridges': {'ovirtmgmt': {'addr': '46.22.124.37', 'cfg': {'IPADDR': '46.22.124.37', 'PEERDNS': 'no', 'GATEWAY': '46.22.124.36', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.224', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'ovirtmgmt', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.224', 'stp': 'off', 'ports': ['em1']}, 'san': {'addr': '192.168.43.11', 'cfg': {'IPADDR': '192.168.43.11', 'PEERDNS': 'no', 'MTU': '9000', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'NETMASK': '255.255.255.0', 'BOOTPROTO': 'none', 'STP': 'no', 'DEVICE': 'san', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '9000', 'netmask': '255.255.255.0', 'stp': 'off', 'ports': ['em2']}, 'vminternet': {'addr': '', 'cfg': {'PEERDNS': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'STP': 'no', 'DEVICE': 'vminternet', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'stp': 'off', 'ports': ['em4']}}, 'uuid': '4C4C4544-0042-4710-8030-B3C04F37354A', 'lastClientIface': 'ovirtmgmt', 'nics': {'em4': {'addr': '', 'cfg': {'BRIDGE': 'vminternet', 'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'HWADDR': 'd0:67:e5:f9:2e:20', 'STP': 'no', 'DEVICE': 'em4', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:20', 'speed': 100}, 'em1': {'addr': '', 'cfg': {'BRIDGE': 'ovirtmgmt', 'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'HWADDR': 'd0:67:e5:f9:2e:1a', 'STP': 'no', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:1a', 'speed': 1000}, 'em3': {'addr': '', 'cfg': {'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'HWADDR': 'd0:67:e5:f9:2e:1e', 'STP': 'no', 'DEVICE': 'em3', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:1e', 'speed': 0}, 'em2': {'addr': '', 'cfg': {'BRIDGE': 'san', 'NM_CONTROLLED': 'no', 'PEERDNS': 'no', 'MTU': '9000', 'HWADDR': 'd0:67:e5:f9:2e:1c', 'STP': 'no', 'DEVICE': 'em2', 'ONBOOT': 'yes'}, 'mtu': '9000', 'netmask': '', 'hwaddr': 'd0:67:e5:f9:2e:1c', 'speed': 1000}}, 'software_revision': '9', 'clusterLevels': ['3.0', '3.1', '3.2'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,ht,syscall,nx,mmxext,fxsr_opt,pdpe1gb,rdtscp,lm,constant_tsc,rep_good,nopl,nonstop_tsc,extd_apicid,amd_dcm,aperfmperf,pni,pclmulqdq,monitor,ssse3,cx16,sse4_1,sse4_2,popcnt,aes,xsave,avx,lahf_lm,cmp_legacy,svm,extapic,cr8_legacy,abm,sse4a,misalignsse,3dnowprefetch,osvw,ibs,xop,skinit,wdt,lwp,fma4,nodeid_msr,topoext,perfctr_core,arat,cpb,hw_pstate,npt,lbrv,svm_lock,nrip_save,tsc_scale,vmcb_clean,flushbyasid,decodeassists,pausefilter,pfthreshold,model_Opteron_G3,model_Opteron_G1,model_Opteron_G4,model_Opteron_G2', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:a07da5fa539', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1'], 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'memSize': '64210', 'cpuSpeed': '2299.953', 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '32', 'kvmEnabled': 'true', 'guestOverhead': '65', 'management_ip': '', 'cpuThreads': '32', 'emulatedMachines': [u'pc-1.2', u'none', u'pc', u'pc-1.1', u'pc-1.0', u'pc-0.15', u'pc-0.14', u'pc-0.13', u'pc-0.12', u'pc-0.11', u'pc-0.10', u'isapc', u'pc-1.2', u'none', u'pc', u'pc-1.1', u'pc-1.0', u'pc-0.15', u'pc-0.14', u'pc-0.13', u'pc-0.12', u'pc-0.11', u'pc-0.10', u'isapc'], 'operatingSystem': {'release': '20120228.fc18', 'version': '2.6.1', 'name': 'oVirt Node'}, 'lastClient': '46.22.124.43'}} Thread-12391::DEBUG::2013-04-22 11:02:11,306::BindingXMLRPC::913::vds::(wrapper) client [46.22.124.43]::call getHardwareInfo with () {} Thread-12391::DEBUG::2013-04-22 11:02:11,307::BindingXMLRPC::920::vds::(wrapper) return getHardwareInfo with {'status': {'message': 'Done', 'code': 0}, 'info': {'systemProductName': 'PowerEdge R715', 'systemSerialNumber': '3BG075J', 'systemFamily': 'Not Specified', 'systemVersion': 'Not Specified', 'systemUUID': '4C4C4544-0042-4710-8030-B3C04F37354A', 'systemManufacturer': 'Dell Inc.'}} Thread-12393::DEBUG::2013-04-22 11:02:11,366::BindingXMLRPC::161::vds::(wrapper) [46.22.124.43] Thread-12393::DEBUG::2013-04-22 11:02:11,367::task::568::TaskManager.Task::(_updateState) Task=`bf358e1a-079c-4d29-8a9b-99cd5267d4f8`::moving from state init -> state preparing Thread-12393::INFO::2013-04-22 11:02:11,367::logUtils::41::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=3, spUUID='dae1ba5e-548a-473e-94b6-7d08d3a6229a', conList=[{'connection': '192.168.43.10', 'iqn': 'iqn.2012-02.se.elementary:ovirt', 'portal': '1', 'user': '', 'password': '******', 'id': '906d9ac3-4ea0-4c5c-bb74-207f768412c4', 'port': '3260'}], options=None) Thread-12393::INFO::2013-04-22 11:02:11,368::logUtils::44::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '906d9ac3-4ea0-4c5c-bb74-207f768412c4'}]} Thread-12393::DEBUG::2013-04-22 11:02:11,368::task::1151::TaskManager.Task::(prepare) Task=`bf358e1a-079c-4d29-8a9b-99cd5267d4f8`::finished: {'statuslist': [{'status': 0, 'id': '906d9ac3-4ea0-4c5c-bb74-207f768412c4'}]} Thread-12393::DEBUG::2013-04-22 11:02:11,368::task::568::TaskManager.Task::(_updateState) Task=`bf358e1a-079c-4d29-8a9b-99cd5267d4f8`::moving from state preparing -> state finished Thread-12393::DEBUG::2013-04-22 11:02:11,368::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-12393::DEBUG::2013-04-22 11:02:11,368::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12393::DEBUG::2013-04-22 11:02:11,368::task::957::TaskManager.Task::(_decref) Task=`bf358e1a-079c-4d29-8a9b-99cd5267d4f8`::ref 0 aborting False Thread-12394::DEBUG::2013-04-22 11:02:11,378::BindingXMLRPC::161::vds::(wrapper) [46.22.124.43] Thread-12394::DEBUG::2013-04-22 11:02:11,379::task::568::TaskManager.Task::(_updateState) Task=`e6d93206-9c45-4326-b1f5-bc74775605a1`::moving from state init -> state preparing Thread-12394::INFO::2013-04-22 11:02:11,379::logUtils::41::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='dae1ba5e-548a-473e-94b6-7d08d3a6229a', conList=[{'connection': '192.168.43.10:/storage/ovirt/nfs/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '480ce9bb-4299-43f8-9fe0-8fbac71e54be', 'port': ''}], options=None) Thread-12394::INFO::2013-04-22 11:02:11,379::logUtils::44::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': '480ce9bb-4299-43f8-9fe0-8fbac71e54be'}]} Thread-12394::DEBUG::2013-04-22 11:02:11,379::task::1151::TaskManager.Task::(prepare) Task=`e6d93206-9c45-4326-b1f5-bc74775605a1`::finished: {'statuslist': [{'status': 0, 'id': '480ce9bb-4299-43f8-9fe0-8fbac71e54be'}]} Thread-12394::DEBUG::2013-04-22 11:02:11,379::task::568::TaskManager.Task::(_updateState) Task=`e6d93206-9c45-4326-b1f5-bc74775605a1`::moving from state preparing -> state finished Thread-12394::DEBUG::2013-04-22 11:02:11,380::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-12394::DEBUG::2013-04-22 11:02:11,380::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12394::DEBUG::2013-04-22 11:02:11,380::task::957::TaskManager.Task::(_decref) Task=`e6d93206-9c45-4326-b1f5-bc74775605a1`::ref 0 aborting False Thread-12395::DEBUG::2013-04-22 11:02:11,387::BindingXMLRPC::161::vds::(wrapper) [46.22.124.43] Thread-12395::DEBUG::2013-04-22 11:02:11,387::task::568::TaskManager.Task::(_updateState) Task=`5348f964-3322-4250-a78a-d35e682664dd`::moving from state init -> state preparing Thread-12395::INFO::2013-04-22 11:02:11,388::logUtils::41::dispatcher::(wrapper) Run and protect: validateStorageServerConnection(domType=1, spUUID='dae1ba5e-548a-473e-94b6-7d08d3a6229a', conList=[{'connection': '192.168.43.10:/storage/ovirt/nfs/export_domain', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'cbec69e2-884b-4904-89ec-4b947e1f33ba', 'port': ''}], options=None) Thread-12395::INFO::2013-04-22 11:02:11,388::logUtils::44::dispatcher::(wrapper) Run and protect: validateStorageServerConnection, Return response: {'statuslist': [{'status': 0, 'id': 'cbec69e2-884b-4904-89ec-4b947e1f33ba'}]} Thread-12395::DEBUG::2013-04-22 11:02:11,388::task::1151::TaskManager.Task::(prepare) Task=`5348f964-3322-4250-a78a-d35e682664dd`::finished: {'statuslist': [{'status': 0, 'id': 'cbec69e2-884b-4904-89ec-4b947e1f33ba'}]} Thread-12395::DEBUG::2013-04-22 11:02:11,388::task::568::TaskManager.Task::(_updateState) Task=`5348f964-3322-4250-a78a-d35e682664dd`::moving from state preparing -> state finished Thread-12395::DEBUG::2013-04-22 11:02:11,388::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-12395::DEBUG::2013-04-22 11:02:11,388::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12395::DEBUG::2013-04-22 11:02:11,389::task::957::TaskManager.Task::(_decref) Task=`5348f964-3322-4250-a78a-d35e682664dd`::ref 0 aborting False Thread-12396::DEBUG::2013-04-22 11:02:11,396::BindingXMLRPC::161::vds::(wrapper) [46.22.124.43] Thread-12396::DEBUG::2013-04-22 11:02:11,396::task::568::TaskManager.Task::(_updateState) Task=`f5d1e7c3-a014-4d63-983b-6dc088a56f49`::moving from state init -> state preparing Thread-12396::INFO::2013-04-22 11:02:11,397::logUtils::41::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=3, spUUID='dae1ba5e-548a-473e-94b6-7d08d3a6229a', conList=[{'connection': '192.168.43.10', 'iqn': 'iqn.2012-02.se.elementary:ovirt', 'portal': '1', 'user': '', 'password': '******', 'id': '906d9ac3-4ea0-4c5c-bb74-207f768412c4', 'port': '3260'}], options=None) Thread-12396::DEBUG::2013-04-22 11:02:11,397::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2012-02.se.elementary:ovirt -I default -p 192.168.43.10:3260 --op=new' (cwd None) Thread-12396::DEBUG::2013-04-22 11:02:11,420::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12396::DEBUG::2013-04-22 11:02:11,421::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2012-02.se.elementary:ovirt -I default -p 192.168.43.10:3260 -l' (cwd None) Thread-12396::DEBUG::2013-04-22 11:02:11,441::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12396::DEBUG::2013-04-22 11:02:11,442::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2012-02.se.elementary:ovirt -I default -p 192.168.43.10:3260 -n node.startup -v manual --op=update' (cwd None) Thread-12396::DEBUG::2013-04-22 11:02:11,460::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12396::INFO::2013-04-22 11:02:11,460::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '906d9ac3-4ea0-4c5c-bb74-207f768412c4'}]} Thread-12396::DEBUG::2013-04-22 11:02:11,461::task::1151::TaskManager.Task::(prepare) Task=`f5d1e7c3-a014-4d63-983b-6dc088a56f49`::finished: {'statuslist': [{'status': 0, 'id': '906d9ac3-4ea0-4c5c-bb74-207f768412c4'}]} Thread-12396::DEBUG::2013-04-22 11:02:11,461::task::568::TaskManager.Task::(_updateState) Task=`f5d1e7c3-a014-4d63-983b-6dc088a56f49`::moving from state preparing -> state finished Thread-12396::DEBUG::2013-04-22 11:02:11,461::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-12396::DEBUG::2013-04-22 11:02:11,461::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12396::DEBUG::2013-04-22 11:02:11,462::task::957::TaskManager.Task::(_decref) Task=`f5d1e7c3-a014-4d63-983b-6dc088a56f49`::ref 0 aborting False Thread-12397::DEBUG::2013-04-22 11:02:11,470::BindingXMLRPC::161::vds::(wrapper) [46.22.124.43] Thread-12397::DEBUG::2013-04-22 11:02:11,470::task::568::TaskManager.Task::(_updateState) Task=`768743d8-d35d-4a1d-b861-b8ef0d74000c`::moving from state init -> state preparing Thread-12397::INFO::2013-04-22 11:02:11,471::logUtils::41::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='dae1ba5e-548a-473e-94b6-7d08d3a6229a', conList=[{'connection': '192.168.43.10:/storage/ovirt/nfs/iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '480ce9bb-4299-43f8-9fe0-8fbac71e54be', 'port': ''}], options=None) Thread-12397::INFO::2013-04-22 11:02:12,695::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '480ce9bb-4299-43f8-9fe0-8fbac71e54be'}]} Thread-12397::DEBUG::2013-04-22 11:02:12,696::task::1151::TaskManager.Task::(prepare) Task=`768743d8-d35d-4a1d-b861-b8ef0d74000c`::finished: {'statuslist': [{'status': 0, 'id': '480ce9bb-4299-43f8-9fe0-8fbac71e54be'}]} Thread-12397::DEBUG::2013-04-22 11:02:12,696::task::568::TaskManager.Task::(_updateState) Task=`768743d8-d35d-4a1d-b861-b8ef0d74000c`::moving from state preparing -> state finished Thread-12397::DEBUG::2013-04-22 11:02:12,696::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-12397::DEBUG::2013-04-22 11:02:12,696::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12397::DEBUG::2013-04-22 11:02:12,696::task::957::TaskManager.Task::(_decref) Task=`768743d8-d35d-4a1d-b861-b8ef0d74000c`::ref 0 aborting False Thread-12398::DEBUG::2013-04-22 11:02:12,705::BindingXMLRPC::161::vds::(wrapper) [46.22.124.43] Thread-12398::DEBUG::2013-04-22 11:02:12,705::task::568::TaskManager.Task::(_updateState) Task=`cd8323a6-b1fe-4fc3-a457-c352bb0712f7`::moving from state init -> state preparing Thread-12398::INFO::2013-04-22 11:02:12,706::logUtils::41::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='dae1ba5e-548a-473e-94b6-7d08d3a6229a', conList=[{'connection': '192.168.43.10:/storage/ovirt/nfs/export_domain', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'cbec69e2-884b-4904-89ec-4b947e1f33ba', 'port': ''}], options=None) Thread-12398::INFO::2013-04-22 11:02:13,902::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': 'cbec69e2-884b-4904-89ec-4b947e1f33ba'}]} Thread-12398::DEBUG::2013-04-22 11:02:13,903::task::1151::TaskManager.Task::(prepare) Task=`cd8323a6-b1fe-4fc3-a457-c352bb0712f7`::finished: {'statuslist': [{'status': 0, 'id': 'cbec69e2-884b-4904-89ec-4b947e1f33ba'}]} Thread-12398::DEBUG::2013-04-22 11:02:13,903::task::568::TaskManager.Task::(_updateState) Task=`cd8323a6-b1fe-4fc3-a457-c352bb0712f7`::moving from state preparing -> state finished Thread-12398::DEBUG::2013-04-22 11:02:13,903::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-12398::DEBUG::2013-04-22 11:02:13,903::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12398::DEBUG::2013-04-22 11:02:13,903::task::957::TaskManager.Task::(_decref) Task=`cd8323a6-b1fe-4fc3-a457-c352bb0712f7`::ref 0 aborting False Thread-12399::DEBUG::2013-04-22 11:02:13,917::BindingXMLRPC::161::vds::(wrapper) [46.22.124.43] Thread-12399::DEBUG::2013-04-22 11:02:13,917::task::568::TaskManager.Task::(_updateState) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::moving from state init -> state preparing Thread-12399::INFO::2013-04-22 11:02:13,918::logUtils::41::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='dae1ba5e-548a-473e-94b6-7d08d3a6229a', hostID=1, scsiKey='dae1ba5e-548a-473e-94b6-7d08d3a6229a', msdUUID='ea5643f7-f41f-4c44-a27f-f31240cd0f19', masterVersion=110, options=None) Thread-12399::DEBUG::2013-04-22 11:02:13,918::resourceManager::190::ResourceManager.Request::(__init__) ResName=`Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a`ReqID=`15473924-edce-43f6-a4dc-04b7fa1aff61`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '189' at '__init__' Thread-12399::DEBUG::2013-04-22 11:02:13,918::resourceManager::504::ResourceManager::(registerResource) Trying to register resource 'Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a' for lock type 'exclusive' Thread-12399::DEBUG::2013-04-22 11:02:13,919::resourceManager::547::ResourceManager::(registerResource) Resource 'Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a' is free. Now locking as 'exclusive' (1 active user) Thread-12399::DEBUG::2013-04-22 11:02:13,919::resourceManager::227::ResourceManager.Request::(grant) ResName=`Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a`ReqID=`15473924-edce-43f6-a4dc-04b7fa1aff61`::Granted request Thread-12399::INFO::2013-04-22 11:02:13,920::sp::625::Storage.StoragePool::(connect) Connect host #1 to the storage pool dae1ba5e-548a-473e-94b6-7d08d3a6229a with master domain: ea5643f7-f41f-4c44-a27f-f31240cd0f19 (ver = 110) Thread-12399::DEBUG::2013-04-22 11:02:13,920::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:13,921::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:13,921::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:13,921::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:13,921::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:13,922::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:13,922::misc::1054::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-12399::DEBUG::2013-04-22 11:02:13,922::misc::1056::SamplingMethod::(__call__) Got in to sampling method Thread-12399::DEBUG::2013-04-22 11:02:13,922::misc::1054::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-12399::DEBUG::2013-04-22 11:02:13,923::misc::1056::SamplingMethod::(__call__) Got in to sampling method Thread-12399::DEBUG::2013-04-22 11:02:13,923::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) Thread-12399::DEBUG::2013-04-22 11:02:13,947::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12399::DEBUG::2013-04-22 11:02:13,948::misc::1064::SamplingMethod::(__call__) Returning last result MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,950::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host0/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,955::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host1/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,959::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host2/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,964::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host3/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,968::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host4/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,972::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host5/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,976::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host6/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,980::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host7/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,985::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host8/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,989::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd of=/sys/class/scsi_host/host10/scan' (cwd None) MainProcess|Thread-12399::DEBUG::2013-04-22 11:02:13,993::iscsi::402::Storage.ISCSI::(forceIScsiScan) Performing SCSI scan, this will take up to 30 seconds Thread-12399::DEBUG::2013-04-22 11:02:15,997::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/multipath' (cwd None) Thread-12399::DEBUG::2013-04-22 11:02:16,048::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12399::DEBUG::2013-04-22 11:02:16,048::lvm::477::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,049::lvm::479::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,049::lvm::488::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,049::lvm::490::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,049::lvm::508::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,050::lvm::510::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,050::misc::1064::SamplingMethod::(__call__) Returning last result Thread-12399::DEBUG::2013-04-22 11:02:16,050::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,057::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%1IET_00020001|1IET_00020002|1IET_00020003|3500000e019ee26e0|3500000e019ee27f0|3500000e019ee2830|3500000e019efff80|3600508e000000000fdc993aef946ac0f%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free ea5643f7-f41f-4c44-a27f-f31240cd0f19' (cwd None) Thread-12399::DEBUG::2013-04-22 11:02:16,243::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12399::DEBUG::2013-04-22 11:02:16,245::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:16,245::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend Thread-12399::DEBUG::2013-04-22 11:02:17,380::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd iflag=direct skip=0 bs=2048 if=/dev/ea5643f7-f41f-4c44-a27f-f31240cd0f19/metadata count=1' (cwd None) Thread-12399::DEBUG::2013-04-22 11:02:17,390::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000642052 s, 3.2 MB/s\n'; <rc> = 0 Thread-12399::DEBUG::2013-04-22 11:02:17,390::misc::325::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000642052 s, 3.2 MB/s'], size: 2048 Thread-12399::DEBUG::2013-04-22 11:02:17,390::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] Thread-12399::WARNING::2013-04-22 11:02:17,391::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is Thread-12399::DEBUG::2013-04-22 11:02:17,392::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend Thread-12399::DEBUG::2013-04-22 11:02:17,392::lvm::483::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:17,393::lvm::485::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:17,393::lvm::493::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:17,393::lvm::505::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:17,394::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:17,394::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%1IET_00020001|1IET_00020002|1IET_00020003|3500000e019ee26e0|3500000e019ee27f0|3500000e019ee2830|3500000e019efff80|3600508e000000000fdc993aef946ac0f%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free ea5643f7-f41f-4c44-a27f-f31240cd0f19' (cwd None) Thread-12399::DEBUG::2013-04-22 11:02:17,504::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12399::DEBUG::2013-04-22 11:02:17,506::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:17,507::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['PHYBLKSIZE=4096', 'SDUUID=ea5643f7-f41f-4c44-a27f-f31240cd0f19', 'VERSION=3', 'TYPE=ISCSI', u'PV0=pv:1IET_00020001,uuid:0RBLsc-FB0t-RA6I-d12o-jtUx-lIx3-eQwv8Z,pestart:0,pecount:1951,mapoffset:0', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'DESCRIPTION=hinken', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'VGUUID=5Qox85-NQCz-pyVM-aFRN-KJ2P-jB1T-IiETQr', 'CLASS=Data', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_DESCRIPTION=Elementary', 'POOL_UUID=dae1ba5e-548a-473e-94b6-7d08d3a6229a', 'POOL_SPM_ID=1', u'POOL_DOMAINS=ea5643f7-f41f-4c44-a27f-f31240cd0f19:Active,c5c68e94-14c6-4b29-8f83-2daa680dc6c0:Active,fecb861a-5cf6-4f1e-8bfe-b7a71e122c2f:Active,9f296bdb-2903-4e8c-a019-f10bc3ae4d17:Active,c7452b13-e72e-4ce8-83e1-79c22f76dabf:Active', 'POOL_SPM_LVER=12', '_SHA_CKSUM=0924d492075d543bc008063efa0cd0239f60920e'] Thread-12399::DEBUG::2013-04-22 11:02:17,508::lvm::334::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex Thread-12399::DEBUG::2013-04-22 11:02:17,508::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%1IET_00020001|1IET_00020002|1IET_00020003|3500000e019ee26e0|3500000e019ee27f0|3500000e019ee2830|3500000e019efff80|3600508e000000000fdc993aef946ac0f%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None) Thread-12399::DEBUG::2013-04-22 11:02:17,624::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0 Thread-12399::DEBUG::2013-04-22 11:02:17,625::lvm::359::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex Thread-12399::WARNING::2013-04-22 11:02:17,626::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace ea5643f7-f41f-4c44-a27f-f31240cd0f19_imageNS already registered Thread-12399::WARNING::2013-04-22 11:02:17,626::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace ea5643f7-f41f-4c44-a27f-f31240cd0f19_volumeNS already registered Thread-12399::WARNING::2013-04-22 11:02:17,626::blockSD::432::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace ea5643f7-f41f-4c44-a27f-f31240cd0f19_lvmActivationNS already registered Thread-12399::ERROR::2013-04-22 11:02:17,627::sp::1519::Storage.StoragePool::(getMasterDomain) Requested master domain ea5643f7-f41f-4c44-a27f-f31240cd0f19 does not have expected version 110 it is version 1 Thread-12399::DEBUG::2013-04-22 11:02:17,627::resourceManager::557::ResourceManager::(releaseResource) Trying to release resource 'Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a' Thread-12399::DEBUG::2013-04-22 11:02:17,628::resourceManager::573::ResourceManager::(releaseResource) Released resource 'Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a' (0 active users) Thread-12399::DEBUG::2013-04-22 11:02:17,628::resourceManager::578::ResourceManager::(releaseResource) Resource 'Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a' is free, finding out if anyone is waiting for it. Thread-12399::DEBUG::2013-04-22 11:02:17,628::resourceManager::585::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.dae1ba5e-548a-473e-94b6-7d08d3a6229a', Clearing records. Thread-12399::ERROR::2013-04-22 11:02:17,628::task::833::TaskManager.Task::(_setError) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 840, in _run File "/usr/share/vdsm/logUtils.py", line 42, in wrapper File "/usr/share/vdsm/storage/hsm.py", line 926, in connectStoragePool File "/usr/share/vdsm/storage/hsm.py", line 973, in _connectStoragePool File "/usr/share/vdsm/storage/sp.py", line 642, in connect File "/usr/share/vdsm/storage/sp.py", line 1166, in __rebuild File "/usr/share/vdsm/storage/sp.py", line 1520, in getMasterDomain StoragePoolWrongMaster: Wrong Master domain or its version: 'SD=ea5643f7-f41f-4c44-a27f-f31240cd0f19, pool=dae1ba5e-548a-473e-94b6-7d08d3a6229a' Thread-12399::DEBUG::2013-04-22 11:02:17,629::task::852::TaskManager.Task::(_run) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::Task._run: 086e8dc5-7bbd-499e-a12d-736cb14fe79b ('dae1ba5e-548a-473e-94b6-7d08d3a6229a', 1, 'dae1ba5e-548a-473e-94b6-7d08d3a6229a', 'ea5643f7-f41f-4c44-a27f-f31240cd0f19', 110) {} failed - stopping task Thread-12399::DEBUG::2013-04-22 11:02:17,629::task::1177::TaskManager.Task::(stop) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::stopping in state preparing (force False) Thread-12399::DEBUG::2013-04-22 11:02:17,630::task::957::TaskManager.Task::(_decref) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::ref 1 aborting True Thread-12399::INFO::2013-04-22 11:02:17,630::task::1134::TaskManager.Task::(prepare) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::aborting: Task is aborted: 'Wrong Master domain or its version' - code 324 Thread-12399::DEBUG::2013-04-22 11:02:17,630::task::1139::TaskManager.Task::(prepare) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::Prepare: aborted: Wrong Master domain or its version Thread-12399::DEBUG::2013-04-22 11:02:17,630::task::957::TaskManager.Task::(_decref) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::ref 0 aborting True Thread-12399::DEBUG::2013-04-22 11:02:17,630::task::892::TaskManager.Task::(_doAbort) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::Task._doAbort: force False Thread-12399::DEBUG::2013-04-22 11:02:17,631::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12399::DEBUG::2013-04-22 11:02:17,631::task::568::TaskManager.Task::(_updateState) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::moving from state preparing -> state aborting Thread-12399::DEBUG::2013-04-22 11:02:17,631::task::523::TaskManager.Task::(__state_aborting) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::_aborting: recover policy none Thread-12399::DEBUG::2013-04-22 11:02:17,631::task::568::TaskManager.Task::(_updateState) Task=`086e8dc5-7bbd-499e-a12d-736cb14fe79b`::moving from state aborting -> state failed Thread-12399::DEBUG::2013-04-22 11:02:17,631::resourceManager::830::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-12399::DEBUG::2013-04-22 11:02:17,632::resourceManager::864::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-12399::ERROR::2013-04-22 11:02:17,632::dispatcher::67::Storage.Dispatcher.Protect::(run) {'status': {'message': "Wrong Master domain or its version: 'SD=ea5643f7-f41f-4c44-a27f-f31240cd0f19, pool=dae1ba5e-548a-473e-94b6-7d08d3a6229a'", 'code': 324}}

2 2