Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

I’ve just reinstall the host and have the same issue, here is the ERROR messages from the vdsm logs : Thread-43::ERROR::2016-09-06 16:02:54,399::hsm::2551::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer Thread-43::ERROR::2016-09-06 16:02:54,453::hsm::2551::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer Thread-43::ERROR::2016-09-06 16:02:54,475::hsm::2551::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer Thread-51::ERROR::2016-09-06 16:05:38,319::hsm::2453::Storage.HSM::(connectStorageServer) Could not connect to storageServer Thread-52::ERROR::2016-09-06 16:05:38,636::sdc::137:: Storage.StorageDomainCache::(_findDomain) looking for unfetched domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc Thread-52::ERROR::2016-09-06 16:05:38,637::sdc::154:: Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc Thread-52::ERROR::2016-09-06 16:05:38,756::sdc::143:: Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc not found Thread-52::ERROR::2016-09-06 16:05:38,769::task::866:: Storage.TaskManager.Task::(_setError) Task=`1cfc5c30-fc82-44d6-a296-223fa9426417`::Unexpected error Thread-52::ERROR::2016-09-06 16:05:38,788::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'", 'code': 304}} This SD is present on the host1 (CentOS 6.8) : [root@unc-srv-hyp1 ~]$ ll /rhev/data-center/ 00000002-0002-0002-0002-000000000193/ mnt/ Not in the host2 (CentOS 7.2) : [root@unc-srv-hyp2 ~]# ll /rhev/data-center/ total 4 drwxr-xr-x. 5 vdsm kvm 4096 6 sept. 16:03 mnt The vdsm.log more complete : Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsi::424::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) Thread-88::DEBUG::2016-09-06 16:22:03,078::misc::751:: Storage.SamplingMethod::(__call__) Returning last result Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::741:: Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.hba.rescan) Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::743:: Storage.SamplingMethod::(__call__) Got in to sampling method Thread-88::DEBUG::2016-09-06 16:22:03,080::hba::53::Storage.HBA::(rescan) Starting scan Thread-88::DEBUG::2016-09-06 16:22:03,080::utils::755::Storage.HBA::(execCmd) /usr/bin/sudo -n /usr/libexec/vdsm/fc-scan (cwd None) Thread-88::DEBUG::2016-09-06 16:22:03,134::hba::66::Storage.HBA::(rescan) Scan finished Thread-88::DEBUG::2016-09-06 16:22:03,134::misc::751:: Storage.SamplingMethod::(__call__) Returning last result Thread-88::DEBUG::2016-09-06 16:22:03,135::multipath::131::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None) Thread-88::DEBUG::2016-09-06 16:22:03,201::multipath::131::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-88::DEBUG::2016-09-06 16:22:03,202::utils::755::root::(execCmd) /sbin/udevadm settle --timeout=5 (cwd None) Thread-88::DEBUG::2016-09-06 16:22:03,227::utils::775::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::498:: Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::500:: Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::509:: Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::511:: Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::529:: Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-88::DEBUG::2016-09-06 16:22:03,230::lvm::531:: Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-88::DEBUG::2016-09-06 16:22:03,230::misc::751:: Storage.SamplingMethod::(__call__) Returning last result Thread-88::ERROR::2016-09-06 16:22:03,230::sdc::137:: Storage.StorageDomainCache::(_findDomain) looking for unfetched domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc Thread-88::ERROR::2016-09-06 16:22:03,231::sdc::154:: Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc Thread-88::DEBUG::2016-09-06 16:22:03,231::lvm::371:: Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-88::DEBUG::2016-09-06 16:22:03,233::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free, extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name cc9ab4b2-9880-427b-8f3b-61f03e520cbc (cwd None) Thread-88::DEBUG::2016-09-06 16:22:03,275::lvm::291::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "cc9ab4b2-9880-427b-8f3b-61f03e520cbc" not found\n Cannot process volume group cc9ab4b2-9880-427b-8f3b-61f03e520cbc\n'; <rc> = 5 Thread-88::WARNING::2016-09-06 16:22:03,277::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "cc9ab4b2-9880-427b-8f3b-61f03e520cbc" not found', ' Cannot process volume group cc9ab4b2-9880-427b-8f3b- 61f03e520cbc'] Thread-88::DEBUG::2016-09-06 16:22:03,277::lvm::416:: Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex Thread-90::DEBUG::2016-09-06 16:22:03,293::__init__::318::IOProcessClient::(_run) Starting IOProcess... Thread-91::DEBUG::2016-09-06 16:22:03,314::__init__::318::IOProcessClient::(_run) Starting IOProcess... Thread-88::ERROR::2016-09-06 16:22:03,334::sdc::143:: Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain dom = findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc',) Thread-88::DEBUG::2016-09-06 16:22:03,335::resourceManager: :616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-000000000193' Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager: :635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-000000000193' (0 active users) Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager: :641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-000000000193' is free, finding out if anyone is waiting for it. Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager: :649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-000000000193', Clearing records. Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager: :616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager: :635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager: :641::Storage.ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager: :649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. Thread-88::ERROR::2016-09-06 16:22:03,338::task::866:: Storage.TaskManager.Task::(_setError) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/logUtils.py", line 45, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 1039, in connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File "/usr/share/vdsm/storage/hsm.py", line 1104, in _connectStoragePool res = pool.connect(hostID, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 637, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1179, in __rebuild self.setMasterDomain(msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1390, in setMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID) StoragePoolMasterNotFound: Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b- 8f3b-61f03e520cbc' Thread-88::DEBUG::2016-09-06 16:22:03,339::task::885:: Storage.TaskManager.Task::(_run) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._run: 4898233c-f6c6-45fa-a2ee-42e63e189063 (u'00000002-0002-0002-0002-000000000193', 2, u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc', 7, {u'015799ac-ec66-4a9d-a8d6-6e9ec980972d': u'active', u'2fcd37ce-cb88-4026-88df-d4d472b41ecf': u'active', u'76a1fed9-2e60-4b3e-9f00-efca8acd133d': u'active', u'7e40772a-fe94-4fb2-94c4-6198bed04a6a': u'active', u'cb4c84c1-489a-433f-999a-f1aeec9d62cf': u'active', u'ea05d014-f8f0-4f1d-906e-2e93c8907d7d': u'active', u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc': u'active'}) {} failed - stopping task Thread-88::DEBUG::2016-09-06 16:22:03,339::task::1217:: Storage.TaskManager.Task::(stop) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::stopping in state preparing (force False) Thread-88::DEBUG::2016-09-06 16:22:03,340::task::993:: Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref 1 aborting True Thread-88::INFO::2016-09-06 16:22:03,340::task::1171:: Storage.TaskManager.Task::(prepare) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::aborting: Task is aborted: 'Cannot find master domain' - code 304 Thread-88::DEBUG::2016-09-06 16:22:03,340::task::1176:: Storage.TaskManager.Task::(prepare) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Prepare: aborted: Cannot find master domain Thread-88::DEBUG::2016-09-06 16:22:03,341::task::993:: Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref 0 aborting True Thread-88::DEBUG::2016-09-06 16:22:03,341::task::928:: Storage.TaskManager.Task::(_doAbort) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._doAbort: force False Thread-88::DEBUG::2016-09-06 16:22:03,341::resourceManager: :977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-88::DEBUG::2016-09-06 16:22:03,341::task::595:: Storage.TaskManager.Task::(_updateState) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving from state preparing -> state aborting Thread-88::DEBUG::2016-09-06 16:22:03,342::task::550:: Storage.TaskManager.Task::(__state_aborting) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::_aborting: recover policy none Thread-88::DEBUG::2016-09-06 16:22:03,342::task::595:: Storage.TaskManager.Task::(_updateState) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving from state aborting -> state failed Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager: :940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager: :977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-88::ERROR::2016-09-06 16:22:03,343::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'", 'code': 304}} I ‘ve restart the service lvm2-lvmetad.service and enable it as adviced in the log without success The engine.log : 2016-09-06 16:20:02,971 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-26) [6f596728] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000193 Type: StoragePool 2016-09-06 16:20:02,982 INFO [org.ovirt.engine.core.bll.storage. ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000193 Type: StoragePool 2016-09-06 16:20:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker. ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, storageType = NFS, connectionList = [{ id: 2ceca65a-90a0-4daf-82a3-366de490a71e, connection: unc-srv-oman.cfu.local:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6, connection: unc-srv-hyp1.cfu.local:/exports/import_domain, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea46418b-68b3-4f9b-9316-b1d57f17ecbc, connection: unc-srv-oman.cfu.local:/data/Master, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 315222a9 2016-09-06 16:20:03,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker. ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] FINISH, ConnectStorageServerVDSCommand, return: {ea46418b-68b3-4f9b-9316-b1d57f17ecbc=0, 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6=0, 2ceca65a-90a0-4daf-82a3-366de490a71e=0}, log id: 315222a9 2016-09-06 16:20:03,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker. ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, storageType = ISCSI, connectionList = [{ id: 3efad56e-a86d-4682-b799-51f42713cda6, connection: 192.168.4.1, iqn: iqn.1984-05.com.dell:powervault.md3600i. 690b11c0005592a90000000051c3efc7, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f9c9ff3 2016-09-06 16:22:03,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker. ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] FINISH, ConnectStorageServerVDSCommand, return: {3efad56e-a86d-4682-b799-51f42713cda6=465}, log id: f9c9ff3 2016-09-06 16:22:03,976 INFO [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper] (DefaultQuartzScheduler_Worker-26) [7d7c3300] The lun with id xu5AAG-1FHh-Qxbx-ZXIe-55J0-vscw-kb6fdp was reported as problematic ! 2016-09-06 16:22:03,992 ERROR [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 192.168.4.1 iqn.1984-05.com.dell: powervault.md3600i.690b11c0005592a90000000051c3efc7 (LUN mpathc) returned by VDSM was: Failed to setup iSCSI subsystem 2016-09-06 16:22:03,997 ERROR [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper] (DefaultQuartzScheduler_Worker-26) [7d7c3300] The connection with details 192.168.4.1 iqn.1984-05.com.dell:powervault.md3600i. 690b11c0005592a90000000051c3efc7 (LUN mpathc) failed because of error code 465 and error message is: failed to setup iscsi subsystem

2016-09-06 16:22:03,998 INFO [org.ovirt.engine.core.bll.storage. ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Host unc-srv-hyp2 storage connection was failed 2016-09-06 16:22:04,004 WARN [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Correlation ID: 7d7c3300, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host unc-srv-hyp2 to Storage Servers 2016-09-06 16:22:04,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, masterVersion = 7), log id: 321780b7 2016-09-06 16:22:04,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) Command org.ovirt.engine.core. vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc']] 2016-09-06 16:22:04,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) HostName = unc-srv-hyp2 2016-09-06 16:22:04,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) Command ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, masterVersion = 7) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b- 8f3b-61f03e520cbc' 2016-09-06 16:22:04,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, ConnectStoragePoolVDSCommand, log id: 321780b7 2016-09-06 16:22:04,351 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-20) Could not connect host unc-srv-hyp2 to pool UNICEF with the message: null 2016-09-06 16:22:04,399 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-26) [60403257] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS 2016-09-06 16:22:04,426 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [60403257] START, SetVdsStatusVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7f47d8b 2016-09-06 16:22:04,435 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [60403257] FINISH, SetVdsStatusVDSCommand, log id: 7f47d8b 2016-09-06 16:22:04,448 WARN [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [60403257] Correlation ID: 60403257, Job ID: 4203a573-20f2-45e5-b251-e4e09f1ed951, Call Stack: null, Custom Event ID: -1, Message: Host unc-srv-hyp2 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center UNICEF. Setting Host state to Non-Operational. 2016-09-06 16:22:04,484 WARN [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [60403257] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host unc-srv-hyp2.There is no other host in the data center that can be used to test the power management settings. 2016-09-06 16:22:04,488 WARN [org.ovirt.engine.core.dal. dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [60403257] Correlation ID: 6f596728, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host unc-srv-hyp2 to Storage Pool UNICEF 2016-09-06 16:22:04,528 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [709cb459] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS 2016-09-06 16:22:04,530 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [709cb459] Host 5bf45c09-41d4-4125-a4bd-81af2a100db8 : unc-srv-hyp2 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. ------------------------------ *Alain VONDRA * *Chargé d'exploitation des Systèmes d'Information * *Direction Administrative et Financière* * +33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> * *UNICEF France 3 rue Duguay Trouin 75006 PARIS* * www.unicef.fr <http://www.unicef.fr/> * <http://www.unicef.fr> <http://www.unicef.fr/> <http://www.unicef.fr/> <http://www.unicef.fr/> <https://www.unicef.fr/> ------------------------------ <http://www.unicef.fr> *De :* Simone Tiraboschi [mailto:stirabos@redhat.com] *Envoyé :* lundi 5 septembre 2016 17:21 *À :* VONDRA Alain <AVONDRA(a)unicef.fr&gt; *Cc :* Yedidyah Bar David <didi(a)redhat.com>; users <users(a)ovirt.org&gt; *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7 On Mon, Sep 5, 2016 at 5:09 PM, VONDRA Alain <AVONDRA(a)unicef.fr&gt; wrote: To resume the situation and be the most clear I can. I have : - 2 hypervisors (physical) with CentOS 6.8 : HYP1 and HYP2 - 1 oVirt manager 3.5 (physical and not hosted-engine) with CentOS 7.2 And to be able to upgrade to oVirt 3.6 or more I need to upgrade the 2 hypervisors to CentOS 7.2. So I began with HYP2, I’ve removed it from the Cluster, and installed it with CentOS 7.2, but when I wanted to join it to a new cluster as shown in the doc, HYP2 stayed always unresponsive. We need vdsm logs to understand the issue here. Could you please reproduce it? So I’ve restored the HYP2 to Centos 6.8, reinstalled it on the previous cluster and everything works well, but I really want to upgrade to CentOS 7.2 to be up-to-date, because I will have soonly more than two versions late. ------------------------------ *Alain VONDRA * *Chargé d'exploitation des Systèmes d'Information * *Direction Administrative et Financière* *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> * *UNICEF France* *3 rue Duguay Trouin 75006 PARIS* *www.unicef.fr <http://www.unicef.fr/>* <https://www.unicef.fr/> ------------------------------ *De :* Yedidyah Bar David [mailto:didi@redhat.com] *Envoyé :* lundi 5 septembre 2016 15:53 *À :* VONDRA Alain <AVONDRA(a)unicef.fr&gt; *Cc :* Simone Tiraboschi <stirabos(a)redhat.com>; users <users(a)ovirt.org&gt; *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7 On Mon, Sep 5, 2016 at 4:34 PM, VONDRA Alain <AVONDRA(a)unicef.fr&gt; wrote: As you say, “This should”, but is there anybody there who tried this operation ? When I wrote [1], it worked for me. If something does not work for you, I think it's best to open a bug and attach all relevant logs, including hosted-engine-setup and vdsm logs from the host and all engine logs from the engine vm. I can't tell from your log snippet why your new host failed to attach to the storage domain. If it's reproducible, please check/post vdsm logs. [1] https://www.ovirt.org/documentation/how-to/hosted- engine-host-OS-upgrade/ Is there any other option to upgrade the hypervisors ? Even if there is another option, the fact that you can't add a host is probably problematic in itself, no? What if you actually need to add a host? Did you try adding a el6 host and it did work? Best, ------------------------------ *Alain VONDRA * *Chargé d'exploitation des Systèmes d'Information * *Direction Administrative et Financière* *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> * *UNICEF France* *3 rue Duguay Trouin 75006 PARIS* *www.unicef.fr <http://www.unicef.fr/>* <https://www.unicef.fr/> ------------------------------ *De :* Yedidyah Bar David [mailto:didi@redhat.com] *Envoyé :* lundi 5 septembre 2016 12:22 *À :* VONDRA Alain <AVONDRA(a)unicef.fr&gt; *Cc :* Simone Tiraboschi <stirabos(a)redhat.com>; users <users(a)ovirt.org&gt; *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7 On Mon, Sep 5, 2016 at 1:11 PM, VONDRA Alain <AVONDRA(a)unicef.fr&gt; wrote: Unlikely, I didn’t save them, I had to rollback the host quickly before the Week-end. All that I can tell, that is all seemed to work well during the installation of the host, all the networks were connected to the SAN, but the host didn’t want to go UP, still staying unresponsive with the message below from the oVirt engine. Can you assure me that the installation of a physical hypervisor with CentOS 7, is possible if I put it on a different cluster than the other host using CentOS 6.8 ? Yes, this should work. Thanks ------------------------------ *Alain VONDRA * *Chargé d'exploitation des Systèmes d'Information * *Direction Administrative et Financière* *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> * *UNICEF France* *3 rue Duguay Trouin 75006 PARIS* *www.unicef.fr <http://www.unicef.fr/>* <https://www.unicef.fr/> ------------------------------ *De :* Yedidyah Bar David [mailto:didi@redhat.com] *Envoyé :* dimanche 4 septembre 2016 08:50 *À :* VONDRA Alain <AVONDRA(a)unicef.fr&gt; *Cc :* Simone Tiraboschi <stirabos(a)redhat.com>; users <users(a)ovirt.org&gt; *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7 On Fri, Sep 2, 2016 at 7:31 PM, VONDRA Alain <AVONDRA(a)unicef.fr&gt; wrote: Hi, I’ve followed this doc, and hoped to find a solution with it, so I did’nt use the hosted-engine –deploy command, I’ve added a new host and at the end of the installation, the host stays unresponsive because unable to be attached to the same Storage volume . Engine log : 2016-09-02 16:57:01,780 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-28) Could not connect host unc-srv-hyp2 to pool UNICEF with the message: null 2016-09-02 17:00:01,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) Command ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = ee1c57ce-1c77-47b1-b466-7bf99382dd77, vdsId = ee1c57ce-1c77-47b1-b466-7bf99382dd77, storagePoolId = 00000002-0002-0002-0002-000000000193, masterVersion = 7) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b- 8f3b-61f03e520cbc' Please check/post vdsm logs from the host. Thanks. ------------------------------ *Alain VONDRA * *Chargé d'exploitation des Systèmes d'Information * *Direction Administrative et Financière* *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> * *UNICEF France* *3 rue Duguay Trouin 75006 PARIS* *www.unicef.fr <http://www.unicef.fr/>* <https://www.unicef.fr/> ------------------------------ *De :* Simone Tiraboschi [mailto:stirabos@redhat.com] *Envoyé :* vendredi 2 septembre 2016 18:20 *À :* VONDRA Alain <AVONDRA(a)unicef.fr&gt; *Cc :* Nir Soffer <nsoffer(a)redhat.com>; users <users(a)ovirt.org&gt; *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7 On Fri, Sep 2, 2016 at 5:21 PM, VONDRA Alain <AVONDRA(a)unicef.fr&gt; wrote: Hi, I'd like to upgrade my oVirt environment from 3.5 to 3.6 and maybe 4, actually the manager oVirt is un 3.5 version, installed on a CentOS 7.2 and the two hypervisors are installed on CentOS 6.8. I need anyway to upgrade the hosts to be able to move to 3.6. I've tried to upgrade the first host, but I had of course issues telling that it can't be possible to mix different OS in the same cluster, I've also tried to create another cluster to put this host without success. What is the best way to upgrade cleanly and safely ? Thank you in advance for your advices Follow this: https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/ Simply ignore the steps that refers to the engine VM if you are with the engine on a physical system. Alain VONDRA Chargé d'exploitation des Systèmes d'Information Direction Administrative et Financière +33 1 44 39 77 76 UNICEF France 3 rue Duguay Trouin 75006 PARIS www.unicef.fr _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Didi -- Didi -- Didi