Indeed reverting back to an older vdsm clears up the above issue. However now I the issue is see is:
Thread-18::ERROR::2013-01-23 15:50:42,885::task::833::TaskManager.Task::(_setError) Task=`08709e68-bcbc-40d8-843a-d69d4df40ac6`::Unexpected error

Traceback (most recent call last):

  File "/usr/share/vdsm/storage/task.py", line 840, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 42, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 923, in connectStoragePool
    masterVersion, options)
  File "/usr/share/vdsm/storage/hsm.py", line 970, in _connectStoragePool
    res = pool.connect(hostID, scsiKey, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 643, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1167, in __rebuild
    self.masterDomain = self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1506, in getMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: 'spUUID=f90a0d1c-06ca-11e2-a05b-00151712f280, msdUUID=67534cca-1327-462a-b455-a04464084b31'
Thread-18::DEBUG::2013-01-23 15:50:42,887::task::852::TaskManager.Task::(_run) Task=`08709e68-bcbc-40d8-843a-d69d4df40ac6`::Task._run: 08709e68-bcbc-40d8-843a-d69d4df40ac6 ('f90a0d1c-06ca-11e2-a05b-00151712f280', 2, 'f90a0d1c-06ca-11e2-a05b-00151712f280', '67534cca-1327-462a-b455-a04464084b31', 433) {} failed - stopping task

This is with vdsm built from

commit

25a2d8572ad32352227c98a86631300fbd6523c1

- DHC

On Wed, Jan 23, 2013 at 10:44 AM, Dead Horse <deadhorseconsulting@gmail.com> wrote:

VDSM was built from:
commit 166138e37e75767b32227746bb671b1dab9cdd5e

Attached is the full vdsm log

I should also note that from engine perspective it sees the master storage domain as locked and the others as unknown.

On Wed, Jan 23, 2013 at 2:49 AM, Dan Kenigsberg <danken@redhat.com> wrote:

On Tue, Jan 22, 2013 at 04:02:24PM -0600, Dead Horse wrote:
> Any ideas on this one? (from VDSM log):
> Thread-25::DEBUG::2013-01-22
> 15:35:29,065::BindingXMLRPC::914::vds::(wrapper) client [3.57.111.30]::call
> getCapabilities with () {}
> Thread-25::ERROR::2013-01-22 15:35:29,113::netinfo::159::root::(speed)
> cannot read ib0 speed
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/site-packages/vdsm/netinfo.py", line 155, in
> speed
>     s = int(file('/sys/class/net/%s/speed' % dev).read())
> IOError: [Errno 22] Invalid argument
>
> Causes VDSM to fail to attach storage

I doubt that this is the cause of the failure, as vdsm has always
reported "0" for ib devices, and still is.

Does a former version works with your Engine?
Could you share more of your vdsm.log? I suppose the culprit lies in one
one of the storage-related commands, not in statistics retrieval.

>
> Engine side sees:
> ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper]
> (QuartzScheduler_Worker-96) [553ef26e] The connection with details
> 192.168.0.1:/ovirt/ds failed because of error code 100 and error message
> is: general exception
> 2013-01-22 15:35:30,160 INFO
> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
> (QuartzScheduler_Worker-96) [1ab78378] Running command:
> SetNonOperationalVdsCommand internal: true. Entities affected :  ID:
> 8970b3fe-1faf-11e2-bc1f-00151712f280 Type: VDS
> 2013-01-22 15:35:30,200 INFO
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (QuartzScheduler_Worker-96) [1ab78378] START,
> SetVdsStatusVDSCommand(HostName = kezan, HostId =
> 8970b3fe-1faf-11e2-bc1f-00151712f280, status=NonOperational,
> nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE), log id: 4af5c4cd
> 2013-01-22 15:35:30,211 INFO
> [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (QuartzScheduler_Worker-96) [1ab78378] FINISH, SetVdsStatusVDSCommand, log
> id: 4af5c4cd
> 2013-01-22 15:35:30,242 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (QuartzScheduler_Worker-96) [1ab78378] Try to add duplicate audit log
> values with the same name. Type: VDS_SET_NONOPERATIONAL_DOMAIN. Value:
> storagepoolname
>
> Engine = latest master
> VDSM = latest master

Since "latest master" is an unstable reference by definition, I'm sure
that History would thank you if you post the exact version (git hash?)
of the code.

> node = el6