Dear all,

We have updated our hypervisors with yum. This included an update ov vdsm also. We now are with these version:
vdsm-4.16.7-1.gitdb83943.el6.x86_64
vdsm-python-4.16.7-1.gitdb83943.el6.noarch
vdsm-python-zombiereaper-4.16.7-1.gitdb83943.el6.noarch
vdsm-xmlrpc-4.16.7-1.gitdb83943.el6.noarch
vdsm-yajsonrpc-4.16.7-1.gitdb83943.el6.noarch
vdsm-jsonrpc-4.16.7-1.gitdb83943.el6.noarch
vdsm-cli-4.16.7-1.gitdb83943.el6.noarch

And ever since these updates we experience BIG troubles with our fibre connections. I've already update the brocade cards to the latest version. This seemed to help, they already came back up and saw the storage domains (before the brocade update, they didn't even see their storage domains). But after a day or so, one of the hypersisors began to freak out again. Coming up and going back down... Below you can find the errors:


Thread-821::ERROR::2014-12-08 07:10:33,190::task::866::Storage.TaskManager.Task::(_setError) Task=`27cb9779-a8e9-4080-988d-9772c922710b`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-821::ERROR::2014-12-08 07:10:33,194::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-822::ERROR::2014-12-08 07:11:03,878::task::866::Storage.TaskManager.Task::(_setError) Task=`30177931-68c0-420f-950f-da5b770fe35c`::Unexpected error
Thread-822::ERROR::2014-12-08 07:11:03,882::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Unknown pool id, pool not connected: ('1d03dc05-008b-4d14-97ce-b17bd714183d',)", 'code': 309}}
Thread-813::ERROR::2014-12-08 07:11:07,634::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-813::ERROR::2014-12-08 07:11:07,634::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-813::DEBUG::2014-12-08 07:11:07,638::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/36005076802810d489000000000000062|/dev/mapper/36005076802810d48e0000000000000ae|/dev/mapper/36005076802810d48e0000000000000de|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 78d84adf-7274-4efe-a711-fbec31196ece (cwd None)
Thread-813::DEBUG::2014-12-08 07:11:07,835::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/36005076802810d489000000000000062|/dev/mapper/36005076802810d48e0000000000000ae|/dev/mapper/36005076802810d48e0000000000000de|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 78d84adf-7274-4efe-a711-fbec31196ece (cwd None)
Thread-813::ERROR::2014-12-08 07:11:07,896::spbackends::271::Storage.StoragePoolDiskBackend::(validateMasterDomainVersion) Requested master domain 78d84adf-7274-4efe-a711-fbec31196ece does not have expected version 42 it is version 17
Thread-813::ERROR::2014-12-08 07:11:07,903::task::866::Storage.TaskManager.Task::(_setError) Task=`c434f325-5193-4236-a04d-2fee9ac095bc`::Unexpected error
Thread-813::ERROR::2014-12-08 07:11:07,946::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Wrong Master domain or its version: 'SD=78d84adf-7274-4efe-a711-fbec31196ece, pool=1d03dc05-008b-4d14-97ce-b17bd714183d'", 'code': 324}}
Thread-823::ERROR::2014-12-08 07:11:43,993::task::866::Storage.TaskManager.Task::(_setError) Task=`9abbccd9-88a7-4632-b350-f9af1f65bebd`::Unexpected error
Thread-823::ERROR::2014-12-08 07:11:43,998::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Unknown pool id, pool not connected: ('1d03dc05-008b-4d14-97ce-b17bd714183d',)", 'code': 309}}
Thread-823::ERROR::2014-12-08 07:11:44,003::task::866::Storage.TaskManager.Task::(_setError) Task=`7ef1ac39-e7c2-4538-b30b-ab2fcefac01d`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-823::ERROR::2014-12-08 07:11:44,007::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-823::ERROR::2014-12-08 07:11:44,133::task::866::Storage.TaskManager.Task::(_setError) Task=`cc1ae82c-f3c4-4efa-9cd2-c62a27801e76`::Unexpected error
Thread-823::ERROR::2014-12-08 07:11:44,137::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Unknown pool id, pool not connected: ('1d03dc05-008b-4d14-97ce-b17bd714183d',)", 'code': 309}}
Thread-823::ERROR::2014-12-08 07:12:24,580::task::866::Storage.TaskManager.Task::(_setError) Task=`9bcbb87d-3093-4894-879b-3fe2b09ef351`::Unexpected error
Thread-823::ERROR::2014-12-08 07:12:24,585::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Unknown pool id, pool not connected: ('1d03dc05-008b-4d14-97ce-b17bd714183d',)", 'code': 309}}
Thread-823::ERROR::2014-12-08 07:13:04,926::task::866::Storage.TaskManager.Task::(_setError) Task=`8bdd0c1f-e681-4a8e-ad55-296c021389ed`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-823::ERROR::2014-12-08 07:13:04,931::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-823::ERROR::2014-12-08 07:13:45,342::task::866::Storage.TaskManager.Task::(_setError) Task=`160ea2a7-b6cb-4102-9df4-71ba87fd863e`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-823::ERROR::2014-12-08 07:13:45,346::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-823::ERROR::2014-12-08 07:14:25,879::task::866::Storage.TaskManager.Task::(_setError) Task=`985628db-8f48-44b5-8f61-631a922f7f71`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-823::ERROR::2014-12-08 07:14:25,883::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-823::ERROR::2014-12-08 07:15:06,175::task::866::Storage.TaskManager.Task::(_setError) Task=`ddca1c88-0565-41e8-bf0c-22eadcc75918`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-823::ERROR::2014-12-08 07:15:06,179::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-823::ERROR::2014-12-08 07:15:46,585::task::866::Storage.TaskManager.Task::(_setError) Task=`12bbded5-59ce-46d8-9e67-f48862a03606`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-823::ERROR::2014-12-08 07:15:46,589::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}
Thread-814::ERROR::2014-12-08 07:16:08,619::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-814::ERROR::2014-12-08 07:16:08,619::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-814::DEBUG::2014-12-08 07:16:08,624::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/36005076802810d489000000000000062|/dev/mapper/36005076802810d48e0000000000000ae|/dev/mapper/36005076802810d48e0000000000000de|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 78d84adf-7274-4efe-a711-fbec31196ece (cwd None)
Thread-814::DEBUG::2014-12-08 07:16:08,740::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/36005076802810d489000000000000062|/dev/mapper/36005076802810d48e0000000000000ae|/dev/mapper/36005076802810d48e0000000000000de|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 78d84adf-7274-4efe-a711-fbec31196ece (cwd None)
Thread-814::ERROR::2014-12-08 07:16:08,812::spbackends::271::Storage.StoragePoolDiskBackend::(validateMasterDomainVersion) Requested master domain 78d84adf-7274-4efe-a711-fbec31196ece does not have expected version 42 it is version 17
Thread-814::ERROR::2014-12-08 07:16:08,820::task::866::Storage.TaskManager.Task::(_setError) Task=`5cdce5cd-6e6d-421e-bc2a-f999d8cbb056`::Unexpected error
Thread-814::ERROR::2014-12-08 07:16:08,865::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Wrong Master domain or its version: 'SD=78d84adf-7274-4efe-a711-fbec31196ece, pool=1d03dc05-008b-4d14-97ce-b17bd714183d'", 'code': 324}}
Thread-815::ERROR::2014-12-08 07:16:09,471::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-815::ERROR::2014-12-08 07:16:09,472::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-815::DEBUG::2014-12-08 07:16:09,476::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/36005076802810d489000000000000062|/dev/mapper/36005076802810d48e0000000000000ae|/dev/mapper/36005076802810d48e0000000000000de|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 78d84adf-7274-4efe-a711-fbec31196ece (cwd None)
Thread-815::DEBUG::2014-12-08 07:16:09,564::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/36005076802810d489000000000000062|/dev/mapper/36005076802810d48e0000000000000ae|/dev/mapper/36005076802810d48e0000000000000de|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 78d84adf-7274-4efe-a711-fbec31196ece (cwd None)
Thread-815::ERROR::2014-12-08 07:16:09,627::spbackends::271::Storage.StoragePoolDiskBackend::(validateMasterDomainVersion) Requested master domain 78d84adf-7274-4efe-a711-fbec31196ece does not have expected version 42 it is version 17
Thread-815::ERROR::2014-12-08 07:16:09,635::task::866::Storage.TaskManager.Task::(_setError) Task=`abfa0fd0-04b3-4c65-b3d0-be18b085a65d`::Unexpected error
Thread-815::ERROR::2014-12-08 07:16:09,681::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Wrong Master domain or its version: 'SD=78d84adf-7274-4efe-a711-fbec31196ece, pool=1d03dc05-008b-4d14-97ce-b17bd714183d'", 'code': 324}}
Thread-816::ERROR::2014-12-08 07:16:10,182::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-816::ERROR::2014-12-08 07:16:10,183::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain 78d84adf-7274-4efe-a711-fbec31196ece
Thread-816::DEBUG::2014-12-08 07:16:10,187::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''a|/dev/mapper/36005076802810d489000000000000062|/dev/mapper/36005076802810d48e0000000000000ae|/dev/mapper/36005076802810d48e0000000000000de|'\'', '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 78d84adf-7274-4efe-a711-fbec31196ece (cwd None)
Thread-823::ERROR::2014-12-08 07:16:27,163::task::866::Storage.TaskManager.Task::(_setError) Task=`9b0fd676-7941-40a7-a71e-0f1dee48a107`::Unexpected error
    raise se.SpmStatusError()
SpmStatusError: Not SPM: ()
Thread-823::ERROR::2014-12-08 07:16:27,168::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': 'Not SPM: ()', 'code': 654}}