[ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

Tue Sep 6 16:33:29 UTC 2016

Great, I found it !
It was an initiator issue due to the reinstalling of the host. After a re-configuration of the SAN, evrything works fine and I could Up the brand new host with CentOS 7.2. I will reproduce the same plan on the other host.
Thank so much for your help

________________________________

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr<http://www.unicef.fr/>

<http://www.unicef.fr>

<http://www.unicef.fr/>

<http://www.unicef.fr/><http://www.unicef.fr/>[cid:signature_Unicef-France_a0c5ef98-a0b5-4b96-9e15-f37a03a75cb1.gif]<https://www.unicef.fr/>

________________________________
<http://www.unicef.fr>

De : Simone Tiraboschi [mailto:stirabos at redhat.com]
Envoyé : mardi 6 septembre 2016 16:48
À : VONDRA Alain <AVONDRA at unicef.fr>
Cc : Yedidyah Bar David <didi at redhat.com>; users <users at ovirt.org>
Objet : Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

On Tue, Sep 6, 2016 at 4:25 PM, VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>> wrote:
I’ve just reinstall the host and have the same issue, here is the ERROR messages from the vdsm logs :

Thread-43::ERROR::2016-09-06 16:02:54,399::hsm::2551::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Thread-43::ERROR::2016-09-06 16:02:54,453::hsm::2551::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Thread-43::ERROR::2016-09-06 16:02:54,475::hsm::2551::Storage.HSM::(disconnectStorageServer) Could not disconnect from storageServer
Thread-51::ERROR::2016-09-06 16:05:38,319::hsm::2453::Storage.HSM::(connectStorageServer) Could not connect to storageServer
Thread-52::ERROR::2016-09-06 16:05:38,636::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-52::ERROR::2016-09-06 16:05:38,637::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-52::ERROR::2016-09-06 16:05:38,756::sdc::143::Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc not found
Thread-52::ERROR::2016-09-06 16:05:38,769::task::866::Storage.TaskManager.Task::(_setError) Task=`1cfc5c30-fc82-44d6-a296-223fa9426417`::Unexpected error
Thread-52::ERROR::2016-09-06 16:05:38,788::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'", 'code': 304}}

This SD is present on the host1 (CentOS 6.8) :
[root at unc-srv-hyp1  ~]$ ll /rhev/data-center/
00000002-0002-0002-0002-000000000193/ mnt/

Not in the host2 (CentOS 7.2) :
[root at unc-srv-hyp2 ~]# ll /rhev/data-center/
total 4
drwxr-xr-x. 5 vdsm kvm 4096  6 sept. 16:03 mnt

The vdsm.log more complete :
Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsi::424::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds
Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsiadm::97::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,078::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.hba.rescan)
Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-88::DEBUG::2016-09-06 16:22:03,080::hba::53::Storage.HBA::(rescan) Starting scan
Thread-88::DEBUG::2016-09-06 16:22:03,080::utils::755::Storage.HBA::(execCmd) /usr/bin/sudo -n /usr/libexec/vdsm/fc-scan (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,134::hba::66::Storage.HBA::(rescan) Scan finished
Thread-88::DEBUG::2016-09-06 16:22:03,134::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-88::DEBUG::2016-09-06 16:22:03,135::multipath::131::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,201::multipath::131::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0
Thread-88::DEBUG::2016-09-06 16:22:03,202::utils::755::root::(execCmd) /sbin/udevadm settle --timeout=5 (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,227::utils::775::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::498::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::500::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::509::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::511::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::529::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,230::lvm::531::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,230::misc::751::Storage.SamplingMethod::(__call__) Returning last result
Thread-88::ERROR::2016-09-06 16:22:03,230::sdc::137::Storage.StorageDomainCache::(_findDomain) looking for unfetched domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-88::ERROR::2016-09-06 16:22:03,231::sdc::154::Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-88::DEBUG::2016-09-06 16:22:03,231::lvm::371::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,233::lvm::291::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''r|.*|'\'' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1  use_lvmetad=0 }  backup {  retain_min = 50  retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name cc9ab4b2-9880-427b-8f3b-61f03e520cbc (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,275::lvm::291::Storage.Misc.excCmd::(cmd) FAILED: <err> = '  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n  Volume group "cc9ab4b2-9880-427b-8f3b-61f03e520cbc" not found\n  Cannot process volume group cc9ab4b2-9880-427b-8f3b-61f03e520cbc\n'; <rc> = 5
Thread-88::WARNING::2016-09-06 16:22:03,277::lvm::376::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', '  Volume group "cc9ab4b2-9880-427b-8f3b-61f03e520cbc" not found', '  Cannot process volume group cc9ab4b2-9880-427b-8f3b-61f03e520cbc']
Thread-88::DEBUG::2016-09-06 16:22:03,277::lvm::416::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
Thread-90::DEBUG::2016-09-06 16:22:03,293::__init__::318::IOProcessClient::(_run) Starting IOProcess...
Thread-91::DEBUG::2016-09-06 16:22:03,314::__init__::318::IOProcessClient::(_run) Starting IOProcess...
Thread-88::ERROR::2016-09-06 16:22:03,334::sdc::143::Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc not found
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
    dom = findMethod(sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
    raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc',)
Thread-88::DEBUG::2016-09-06 16:22:03,335::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-000000000193'
Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-000000000193' (0 active users)
Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-000000000193' is free, finding out if anyone is waiting for it.
Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-000000000193', Clearing records.
Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock'
Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users)
Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it.
Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-88::ERROR::2016-09-06 16:22:03,338::task::866::Storage.TaskManager.Task::(_setError) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 1039, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1104, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 637, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1179, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1390, in setMasterDomain
    raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'
Thread-88::DEBUG::2016-09-06 16:22:03,339::task::885::Storage.TaskManager.Task::(_run) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._run: 4898233c-f6c6-45fa-a2ee-42e63e189063 (u'00000002-0002-0002-0002-000000000193', 2, u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc', 7, {u'015799ac-ec66-4a9d-a8d6-6e9ec980972d': u'active', u'2fcd37ce-cb88-4026-88df-d4d472b41ecf': u'active', u'76a1fed9-2e60-4b3e-9f00-efca8acd133d': u'active', u'7e40772a-fe94-4fb2-94c4-6198bed04a6a': u'active', u'cb4c84c1-489a-433f-999a-f1aeec9d62cf': u'active', u'ea05d014-f8f0-4f1d-906e-2e93c8907d7d': u'active', u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc': u'active'}) {} failed - stopping task
Thread-88::DEBUG::2016-09-06 16:22:03,339::task::1217::Storage.TaskManager.Task::(stop) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::stopping in state preparing (force False)
Thread-88::DEBUG::2016-09-06 16:22:03,340::task::993::Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref 1 aborting True
Thread-88::INFO::2016-09-06 16:22:03,340::task::1171::Storage.TaskManager.Task::(prepare) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::aborting: Task is aborted: 'Cannot find master domain' - code 304
Thread-88::DEBUG::2016-09-06 16:22:03,340::task::1176::Storage.TaskManager.Task::(prepare) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Prepare: aborted: Cannot find master domain
Thread-88::DEBUG::2016-09-06 16:22:03,341::task::993::Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref 0 aborting True
Thread-88::DEBUG::2016-09-06 16:22:03,341::task::928::Storage.TaskManager.Task::(_doAbort) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._doAbort: force False
Thread-88::DEBUG::2016-09-06 16:22:03,341::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-88::DEBUG::2016-09-06 16:22:03,341::task::595::Storage.TaskManager.Task::(_updateState) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving from state preparing -> state aborting
Thread-88::DEBUG::2016-09-06 16:22:03,342::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::_aborting: recover policy none
Thread-88::DEBUG::2016-09-06 16:22:03,342::task::595::Storage.TaskManager.Task::(_updateState) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving from state aborting -> state failed
Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
Thread-88::ERROR::2016-09-06 16:22:03,343::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'", 'code': 304}}

I ‘ve restart the service lvm2-lvmetad.service and enable it as adviced in the log without success

The engine.log :
2016-09-06 16:20:02,971 INFO  [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-26) [6f596728] Running command: InitVdsOnUpCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-000000000193 Type: StoragePool
2016-09-06 16:20:02,982 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected :  ID: 00000002-0002-0002-0002-000000000193 Type: StoragePool
2016-09-06 16:20:03,035 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, storageType = NFS, connectionList = [{ id: 2ceca65a-90a0-4daf-82a3-366de490a71e, connection: unc-srv-oman.cfu.local:/var/lib/exports/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6, connection: unc-srv-hyp1.cfu.local:/exports/import_domain, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: ea46418b-68b3-4f9b-9316-b1d57f17ecbc, connection: unc-srv-oman.cfu.local:/data/Master, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 315222a9
2016-09-06 16:20:03,080 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] FINISH, ConnectStorageServerVDSCommand, return: {ea46418b-68b3-4f9b-9316-b1d57f17ecbc=0, 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6=0, 2ceca65a-90a0-4daf-82a3-366de490a71e=0}, log id: 315222a9
2016-09-06 16:20:03,115 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, storageType = ISCSI, connectionList = [{ id: 3efad56e-a86d-4682-b799-51f42713cda6, connection: 192.168.4.1, iqn: iqn.1984-05.com.dell:powervault.md3600i.690b11c0005592a90000000051c3efc7, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f9c9ff3
2016-09-06 16:22:03,952 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] FINISH, ConnectStorageServerVDSCommand, return: {3efad56e-a86d-4682-b799-51f42713cda6=465}, log id: f9c9ff3
2016-09-06 16:22:03,976 INFO  [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper] (DefaultQuartzScheduler_Worker-26) [7d7c3300] The lun with id xu5AAG-1FHh-Qxbx-ZXIe-55J0-vscw-kb6fdp was reported as problematic !
2016-09-06 16:22:03,992 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 192.168.4.1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0005592a90000000051c3efc7 (LUN mpathc) returned by VDSM was: Failed to setup iSCSI subsystem
2016-09-06 16:22:03,997 ERROR [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper] (DefaultQuartzScheduler_Worker-26) [7d7c3300] The connection with details 192.168.4.1 iqn.1984-05.com.dell:powervault.md3600i.690b11c0005592a90000000051c3efc7 (LUN mpathc) failed because of error code 465 and error message is: failed to setup iscsi subsystem

^^^
Thanks, the issue is here.
Can you please ensure that your host could properly access that iSCSI LUN?

Can you please check VDSM logs for that time frame?

2016-09-06 16:22:03,998 INFO  [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Host unc-srv-hyp2 storage connection was failed
2016-09-06 16:22:04,004 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [7d7c3300] Correlation ID: 7d7c3300, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host unc-srv-hyp2 to Storage Servers
2016-09-06 16:22:04,040 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) START, ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, masterVersion = 7), log id: 321780b7
2016-09-06 16:22:04,346 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) Command org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304, mMessage=Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc']]
2016-09-06 16:22:04,347 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) HostName = unc-srv-hyp2
2016-09-06 16:22:04,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) Command ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId = 00000002-0002-0002-0002-000000000193, masterVersion = 7) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'
2016-09-06 16:22:04,350 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-20) FINISH, ConnectStoragePoolVDSCommand, log id: 321780b7
2016-09-06 16:22:04,351 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-20) Could not connect host unc-srv-hyp2 to pool UNICEF with the message: null
2016-09-06 16:22:04,399 INFO  [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-26) [60403257] Running command: SetNonOperationalVdsCommand internal: true. Entities affected :  ID: 5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS
2016-09-06 16:22:04,426 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [60403257] START, SetVdsStatusVDSCommand(HostName = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7f47d8b
2016-09-06 16:22:04,435 INFO  [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [60403257] FINISH, SetVdsStatusVDSCommand, log id: 7f47d8b
2016-09-06 16:22:04,448 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [60403257] Correlation ID: 60403257, Job ID: 4203a573-20f2-45e5-b251-e4e09f1ed951, Call Stack: null, Custom Event ID: -1, Message: Host unc-srv-hyp2 cannot access the Storage Domain(s) <UNKNOWN> attached to the Data Center UNICEF. Setting Host state to Non-Operational.
2016-09-06 16:22:04,484 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [60403257] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Power Management test failed for Host unc-srv-hyp2.There is no other host in the data center that can be used to test the power management settings.
2016-09-06 16:22:04,488 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [60403257] Correlation ID: 6f596728, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host unc-srv-hyp2 to Storage Pool UNICEF
2016-09-06 16:22:04,528 INFO  [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [709cb459] Running command: HandleVdsVersionCommand internal: true. Entities affected :  ID: 5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS
2016-09-06 16:22:04,530 INFO  [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [709cb459] Host 5bf45c09-41d4-4125-a4bd-81af2a100db8 : unc-srv-hyp2 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped.

________________________________

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76<tel:%2B33%201%2044%2039%2077%2076>
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr<http://www.unicef.fr/>

[cid:image001.gif at 01D2086D.29412B80]<https://www.unicef.fr/>

________________________________

De : Simone Tiraboschi [mailto:stirabos at redhat.com<mailto:stirabos at redhat.com>]
Envoyé : lundi 5 septembre 2016 17:21
À : VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>>
Cc : Yedidyah Bar David <didi at redhat.com<mailto:didi at redhat.com>>; users <users at ovirt.org<mailto:users at ovirt.org>>

Objet : Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

On Mon, Sep 5, 2016 at 5:09 PM, VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>> wrote:
To resume the situation and be the most clear I can.
I have :

-          2 hypervisors (physical) with CentOS 6.8 : HYP1 and HYP2

-          1 oVirt manager 3.5  (physical and not hosted-engine) with CentOS 7.2
And to be able to upgrade to oVirt 3.6 or more I need to upgrade the 2 hypervisors to CentOS 7.2.
So I began with HYP2, I’ve removed it from the Cluster, and installed it with CentOS 7.2, but when I wanted to join it to a new cluster as shown in the doc, HYP2 stayed always unresponsive.

We need vdsm logs to understand the issue here.
Could you please reproduce it?

So I’ve restored the HYP2 to Centos 6.8, reinstalled it on the previous cluster and everything works well, but I really want to upgrade to CentOS 7.2 to be up-to-date, because I will have soonly more than two versions late.

________________________________

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76<tel:%2B33%201%2044%2039%2077%2076>
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr<http://www.unicef.fr/>

[cid:image001.gif at 01D2086D.29412B80]<https://www.unicef.fr/>

________________________________

De : Yedidyah Bar David [mailto:didi at redhat.com<mailto:didi at redhat.com>]
Envoyé : lundi 5 septembre 2016 15:53

À : VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>>
Cc : Simone Tiraboschi <stirabos at redhat.com<mailto:stirabos at redhat.com>>; users <users at ovirt.org<mailto:users at ovirt.org>>
Objet : Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

On Mon, Sep 5, 2016 at 4:34 PM, VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>> wrote:
As you say, “This should”, but is there anybody there who tried this operation ?

When I wrote [1], it worked for me.
If something does not work for you, I think it's best to open a bug
and attach all relevant logs, including hosted-engine-setup and vdsm
logs from the host and all engine logs from the engine vm.
I can't tell from your log snippet why your new host failed to attach
to the storage domain. If it's reproducible, please check/post vdsm logs.

[1] https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/

Is there any other option to upgrade the hypervisors ?

Even if there is another option, the fact that you can't add a host
is probably problematic in itself, no? What if you actually need to
add a host?
Did you try adding a el6 host and it did work?
Best,

________________________________

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76<tel:%2B33%201%2044%2039%2077%2076>
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr<http://www.unicef.fr/>

[cid:image001.gif at 01D2086D.29412B80]<https://www.unicef.fr/>

________________________________

De : Yedidyah Bar David [mailto:didi at redhat.com<mailto:didi at redhat.com>]
Envoyé : lundi 5 septembre 2016 12:22

À : VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>>
Cc : Simone Tiraboschi <stirabos at redhat.com<mailto:stirabos at redhat.com>>; users <users at ovirt.org<mailto:users at ovirt.org>>
Objet : Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

On Mon, Sep 5, 2016 at 1:11 PM, VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>> wrote:
Unlikely, I didn’t save them, I had to rollback the host quickly before the Week-end.
All that I can tell, that is all seemed to work well during the installation of the host, all the networks were connected to the SAN, but the host didn’t want to go UP, still staying unresponsive with the message below from the oVirt engine.
Can you assure me that the installation of a physical hypervisor with CentOS 7, is possible if I put it on a different cluster than the other host using CentOS 6.8 ?

Yes, this should work.

Thanks

________________________________

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76<tel:%2B33%201%2044%2039%2077%2076>
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr<http://www.unicef.fr/>

[cid:image001.gif at 01D2086D.29412B80]<https://www.unicef.fr/>

________________________________

De : Yedidyah Bar David [mailto:didi at redhat.com<mailto:didi at redhat.com>]
Envoyé : dimanche 4 septembre 2016 08:50
À : VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>>
Cc : Simone Tiraboschi <stirabos at redhat.com<mailto:stirabos at redhat.com>>; users <users at ovirt.org<mailto:users at ovirt.org>>

Objet : Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

On Fri, Sep 2, 2016 at 7:31 PM, VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>> wrote:
Hi,
I’ve followed this doc, and hoped to find a solution with it, so I did’nt use the hosted-engine –deploy command, I’ve added a new host and at the end of the installation, the host stays unresponsive because unable to be attached to the same Storage volume .
Engine log :
2016-09-02 16:57:01,780 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (org.ovirt.thread.pool-8-thread-28) Could not connect host unc-srv-hyp2 to pool UNICEF with the message: null
2016-09-02 17:00:01,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (org.ovirt.thread.pool-8-thread-3) Command ConnectStoragePoolVDSCommand(HostName = unc-srv-hyp2, HostId = ee1c57ce-1c77-47b1-b466-7bf99382dd77, vdsId = ee1c57ce-1c77-47b1-b466-7bf99382dd77, storagePoolId = 00000002-0002-0002-0002-000000000193, masterVersion = 7) execution failed. Exception: IRSNoMasterDomainException: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'

Please check/post vdsm logs from the host. Thanks.

________________________________

Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76<tel:%2B33%201%2044%2039%2077%2076>
UNICEF France
3 rue Duguay Trouin  75006 PARIS
www.unicef.fr<http://www.unicef.fr/>

[cid:image001.gif at 01D2086D.29412B80]<https://www.unicef.fr/>

________________________________

De : Simone Tiraboschi [mailto:stirabos at redhat.com<mailto:stirabos at redhat.com>]
Envoyé : vendredi 2 septembre 2016 18:20
À : VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>>
Cc : Nir Soffer <nsoffer at redhat.com<mailto:nsoffer at redhat.com>>; users <users at ovirt.org<mailto:users at ovirt.org>>
Objet : Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7

On Fri, Sep 2, 2016 at 5:21 PM, VONDRA Alain <AVONDRA at unicef.fr<mailto:AVONDRA at unicef.fr>> wrote:
Hi,
I'd like to upgrade my oVirt environment from 3.5 to 3.6 and maybe 4, actually the manager oVirt is un 3.5 version, installed on a CentOS 7.2 and the two hypervisors are installed on CentOS 6.8.
I need anyway to upgrade the hosts to be able to move to 3.6.
I've tried to upgrade the first host, but I had of course issues telling that it can't be possible to mix different OS in the same cluster, I've also tried to create another cluster to put this host without success.
What is the best way to upgrade cleanly and safely ?
Thank you in advance for your advices

Follow this:
https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/

Simply ignore the steps that refers to the engine VM if you are with the engine on a physical system.

Alain VONDRA

Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76<tel:%2B33%201%2044%2039%2077%2076>

UNICEF France
3 rue Duguay Trouin  75006
PARIS
www.unicef.fr<http://www.unicef.fr>

_______________________________________________
Users mailing list
Users at ovirt.org<mailto:Users at ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users

--
Didi

--
Didi

--
Didi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160906/53b96a07/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 3115 bytes
Desc: image001.gif
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160906/53b96a07/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature_Unicef-France_a0c5ef98-a0b5-4b96-9e15-f37a03a75cb1.gif
Type: image/gif
Size: 3115 bytes
Desc: signature_Unicef-France_a0c5ef98-a0b5-4b96-9e15-f37a03a75cb1.gif
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160906/53b96a07/attachment-0003.gif>