[ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to CentOS 7
Nir Soffer
nsoffer at redhat.com
Tue Sep 6 17:06:04 UTC 2016
On Tue, Sep 6, 2016 at 7:33 PM, VONDRA Alain <AVONDRA at unicef.fr> wrote:
> Great, I found it !
>
> It was an initiator issue due to the reinstalling of the host. After a
> re-configuration of the SAN, evrything works fine and I could Up the brand
> new host with CentOS 7.2. I will reproduce the same plan on the other host.
>
> Thank so much for your help
>
I think the root cause is that ovirt do not manage the iscsi initiator on
the hosts.
If we had an option to set the initiator when adding a host, like the way
you
set the host password, it would be hard to have such bugs.
Can you file a bug about this?
Nir
>
>
>
>
>
>
> ------------------------------
>
> *Alain VONDRA *
> *Chargé d'exploitation des Systèmes d'Information *
> *Direction Administrative et Financière*
> * +33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
>
> *UNICEF France 3 rue Duguay Trouin 75006 PARIS*
> * www.unicef.fr <http://www.unicef.fr/> *
> <http://www.unicef.fr>
>
> <http://www.unicef.fr/>
>
> <http://www.unicef.fr/> <http://www.unicef.fr/> <https://www.unicef.fr/>
>
>
>
> ------------------------------
> <http://www.unicef.fr>
>
> *De :* Simone Tiraboschi [mailto:stirabos at redhat.com]
> *Envoyé :* mardi 6 septembre 2016 16:48
>
> *À :* VONDRA Alain <AVONDRA at unicef.fr>
> *Cc :* Yedidyah Bar David <didi at redhat.com>; users <users at ovirt.org>
> *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
> CentOS 7
>
>
>
>
>
>
>
> On Tue, Sep 6, 2016 at 4:25 PM, VONDRA Alain <AVONDRA at unicef.fr> wrote:
>
> I’ve just reinstall the host and have the same issue, here is the ERROR
> messages from the vdsm logs :
>
>
>
> Thread-43::ERROR::2016-09-06 16:02:54,399::hsm::2551::Storage.HSM::(disconnectStorageServer)
> Could not disconnect from storageServer
>
> Thread-43::ERROR::2016-09-06 16:02:54,453::hsm::2551::Storage.HSM::(disconnectStorageServer)
> Could not disconnect from storageServer
>
> Thread-43::ERROR::2016-09-06 16:02:54,475::hsm::2551::Storage.HSM::(disconnectStorageServer)
> Could not disconnect from storageServer
>
> Thread-51::ERROR::2016-09-06 16:05:38,319::hsm::2453::Storage.HSM::(connectStorageServer)
> Could not connect to storageServer
>
> Thread-52::ERROR::2016-09-06 16:05:38,636::sdc::137::
> Storage.StorageDomainCache::(_findDomain) looking for unfetched domain
> cc9ab4b2-9880-427b-8f3b-61f03e520cbc
>
> Thread-52::ERROR::2016-09-06 16:05:38,637::sdc::154::
> Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain
> cc9ab4b2-9880-427b-8f3b-61f03e520cbc
>
> Thread-52::ERROR::2016-09-06 16:05:38,756::sdc::143::
> Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
> not found
>
> Thread-52::ERROR::2016-09-06 16:05:38,769::task::866::
> Storage.TaskManager.Task::(_setError) Task=`1cfc5c30-fc82-44d6-a296-223fa9426417`::Unexpected
> error
>
> Thread-52::ERROR::2016-09-06 16:05:38,788::dispatcher::76::Storage.Dispatcher::(wrapper)
> {'status': {'message': "Cannot find master domain:
> u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'",
> 'code': 304}}
>
>
>
> This SD is present on the host1 (CentOS 6.8) :
>
> [root at unc-srv-hyp1 ~]$ ll /rhev/data-center/
>
> 00000002-0002-0002-0002-000000000193/ mnt/
>
>
>
> Not in the host2 (CentOS 7.2) :
>
> [root at unc-srv-hyp2 ~]# ll /rhev/data-center/
>
> total 4
>
> drwxr-xr-x. 5 vdsm kvm 4096 6 sept. 16:03 mnt
>
>
>
> The vdsm.log more complete :
>
> Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsi::424::Storage.ISCSI::(rescan)
> Performing SCSI scan, this will take up to 30 seconds
>
> Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsiadm::97::Storage.Misc.excCmd::(_runCmd)
> /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,078::misc::751::
> Storage.SamplingMethod::(__call__) Returning last result
>
> Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::741::
> Storage.SamplingMethod::(__call__) Trying to enter sampling method
> (storage.hba.rescan)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::743::
> Storage.SamplingMethod::(__call__) Got in to sampling method
>
> Thread-88::DEBUG::2016-09-06 16:22:03,080::hba::53::Storage.HBA::(rescan)
> Starting scan
>
> Thread-88::DEBUG::2016-09-06 16:22:03,080::utils::755::Storage.HBA::(execCmd)
> /usr/bin/sudo -n /usr/libexec/vdsm/fc-scan (cwd None)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,134::hba::66::Storage.HBA::(rescan)
> Scan finished
>
> Thread-88::DEBUG::2016-09-06 16:22:03,134::misc::751::
> Storage.SamplingMethod::(__call__) Returning last result
>
> Thread-88::DEBUG::2016-09-06 16:22:03,135::multipath::131::Storage.Misc.excCmd::(rescan)
> /usr/bin/sudo -n /sbin/multipath (cwd None)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,201::multipath::131::Storage.Misc.excCmd::(rescan)
> SUCCESS: <err> = ''; <rc> = 0
>
> Thread-88::DEBUG::2016-09-06 16:22:03,202::utils::755::root::(execCmd)
> /sbin/udevadm settle --timeout=5 (cwd None)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,227::utils::775::root::(execCmd)
> SUCCESS: <err> = ''; <rc> = 0
>
> Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::498::
> Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate
> operation' got the operation mutex
>
> Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::500::
> Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate
> operation' released the operation mutex
>
> Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::509::
> Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate
> operation' got the operation mutex
>
> Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::511::
> Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate
> operation' released the operation mutex
>
> Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::529::
> Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate
> operation' got the operation mutex
>
> Thread-88::DEBUG::2016-09-06 16:22:03,230::lvm::531::
> Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate
> operation' released the operation mutex
>
> Thread-88::DEBUG::2016-09-06 16:22:03,230::misc::751::
> Storage.SamplingMethod::(__call__) Returning last result
>
> Thread-88::ERROR::2016-09-06 16:22:03,230::sdc::137::
> Storage.StorageDomainCache::(_findDomain) looking for unfetched domain
> cc9ab4b2-9880-427b-8f3b-61f03e520cbc
>
> Thread-88::ERROR::2016-09-06 16:22:03,231::sdc::154::
> Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain
> cc9ab4b2-9880-427b-8f3b-61f03e520cbc
>
> Thread-88::DEBUG::2016-09-06 16:22:03,231::lvm::371::
> Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got
> the operation mutex
>
> Thread-88::DEBUG::2016-09-06 16:22:03,233::lvm::291::Storage.Misc.excCmd::(cmd)
> /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names =
> ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0
> disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
> '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1
> wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days =
> 0 } ' --noheadings --units b --nosuffix --separator '|'
> --ignoreskippedcluster -o uuid,name,attr,size,free,
> extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
> cc9ab4b2-9880-427b-8f3b-61f03e520cbc (cwd None)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,275::lvm::291::Storage.Misc.excCmd::(cmd)
> FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart
> lvmetad before enabling it!\n Volume group "cc9ab4b2-9880-427b-8f3b-61f03e520cbc"
> not found\n Cannot process volume group cc9ab4b2-9880-427b-8f3b-61f03e520cbc\n';
> <rc> = 5
>
> Thread-88::WARNING::2016-09-06 16:22:03,277::lvm::376::Storage.LVM::(_reloadvgs)
> lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart
> lvmetad before enabling it!', ' Volume group "cc9ab4b2-9880-427b-8f3b-61f03e520cbc"
> not found', ' Cannot process volume group cc9ab4b2-9880-427b-8f3b-
> 61f03e520cbc']
>
> Thread-88::DEBUG::2016-09-06 16:22:03,277::lvm::416::
> Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation'
> released the operation mutex
>
> Thread-90::DEBUG::2016-09-06 16:22:03,293::__init__::318::IOProcessClient::(_run)
> Starting IOProcess...
>
> Thread-91::DEBUG::2016-09-06 16:22:03,314::__init__::318::IOProcessClient::(_run)
> Starting IOProcess...
>
> Thread-88::ERROR::2016-09-06 16:22:03,334::sdc::143::
> Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
> not found
>
> Traceback (most recent call last):
>
> File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
>
> dom = findMethod(sdUUID)
>
> File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
>
> raise se.StorageDomainDoesNotExist(sdUUID)
>
> StorageDomainDoesNotExist: Storage domain does not exist:
> (u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc',)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,335::resourceManager:
> :616::Storage.ResourceManager::(releaseResource) Trying to release
> resource 'Storage.00000002-0002-0002-0002-000000000193'
>
> Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager:
> :635::Storage.ResourceManager::(releaseResource) Released resource
> 'Storage.00000002-0002-0002-0002-000000000193' (0 active users)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager:
> :641::Storage.ResourceManager::(releaseResource) Resource
> 'Storage.00000002-0002-0002-0002-000000000193' is free, finding out if
> anyone is waiting for it.
>
> Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager:
> :649::Storage.ResourceManager::(releaseResource) No one is waiting for
> resource 'Storage.00000002-0002-0002-0002-000000000193', Clearing records.
>
> Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager:
> :616::Storage.ResourceManager::(releaseResource) Trying to release
> resource 'Storage.HsmDomainMonitorLock'
>
> Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager:
> :635::Storage.ResourceManager::(releaseResource) Released resource
> 'Storage.HsmDomainMonitorLock' (0 active users)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager:
> :641::Storage.ResourceManager::(releaseResource) Resource
> 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting
> for it.
>
> Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager:
> :649::Storage.ResourceManager::(releaseResource) No one is waiting for
> resource 'Storage.HsmDomainMonitorLock', Clearing records.
>
> Thread-88::ERROR::2016-09-06 16:22:03,338::task::866::
> Storage.TaskManager.Task::(_setError) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Unexpected
> error
>
> Traceback (most recent call last):
>
> File "/usr/share/vdsm/storage/task.py", line 873, in _run
>
> return fn(*args, **kargs)
>
> File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
>
> res = f(*args, **kwargs)
>
> File "/usr/share/vdsm/storage/hsm.py", line 1039, in connectStoragePool
>
> spUUID, hostID, msdUUID, masterVersion, domainsMap)
>
> File "/usr/share/vdsm/storage/hsm.py", line 1104, in _connectStoragePool
>
> res = pool.connect(hostID, msdUUID, masterVersion)
>
> File "/usr/share/vdsm/storage/sp.py", line 637, in connect
>
> self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
>
> File "/usr/share/vdsm/storage/sp.py", line 1179, in __rebuild
>
> self.setMasterDomain(msdUUID, masterVersion)
>
> File "/usr/share/vdsm/storage/sp.py", line 1390, in setMasterDomain
>
> raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
>
> StoragePoolMasterNotFound: Cannot find master domain:
> u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-
> 8f3b-61f03e520cbc'
>
> Thread-88::DEBUG::2016-09-06 16:22:03,339::task::885::
> Storage.TaskManager.Task::(_run) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._run:
> 4898233c-f6c6-45fa-a2ee-42e63e189063 (u'00000002-0002-0002-0002-000000000193',
> 2, u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc', 7, {u'015799ac-ec66-4a9d-a8d6-6e9ec980972d':
> u'active', u'2fcd37ce-cb88-4026-88df-d4d472b41ecf': u'active',
> u'76a1fed9-2e60-4b3e-9f00-efca8acd133d': u'active',
> u'7e40772a-fe94-4fb2-94c4-6198bed04a6a': u'active',
> u'cb4c84c1-489a-433f-999a-f1aeec9d62cf': u'active',
> u'ea05d014-f8f0-4f1d-906e-2e93c8907d7d': u'active',
> u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc': u'active'}) {} failed - stopping
> task
>
> Thread-88::DEBUG::2016-09-06 16:22:03,339::task::1217::
> Storage.TaskManager.Task::(stop) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::stopping
> in state preparing (force False)
>
> Thread-88::DEBUG::2016-09-06 16:22:03,340::task::993::
> Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref
> 1 aborting True
>
> Thread-88::INFO::2016-09-06 16:22:03,340::task::1171::
> Storage.TaskManager.Task::(prepare) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::aborting:
> Task is aborted: 'Cannot find master domain' - code 304
>
> Thread-88::DEBUG::2016-09-06 16:22:03,340::task::1176::
> Storage.TaskManager.Task::(prepare) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Prepare:
> aborted: Cannot find master domain
>
> Thread-88::DEBUG::2016-09-06 16:22:03,341::task::993::
> Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref
> 0 aborting True
>
> Thread-88::DEBUG::2016-09-06 16:22:03,341::task::928::
> Storage.TaskManager.Task::(_doAbort) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._doAbort:
> force False
>
> Thread-88::DEBUG::2016-09-06 16:22:03,341::resourceManager:
> :977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests
> {}
>
> Thread-88::DEBUG::2016-09-06 16:22:03,341::task::595::
> Storage.TaskManager.Task::(_updateState) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving
> from state preparing -> state aborting
>
> Thread-88::DEBUG::2016-09-06 16:22:03,342::task::550::
> Storage.TaskManager.Task::(__state_aborting)
> Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::_aborting: recover policy
> none
>
> Thread-88::DEBUG::2016-09-06 16:22:03,342::task::595::
> Storage.TaskManager.Task::(_updateState) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving
> from state aborting -> state failed
>
> Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager:
> :940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll
> requests {} resources {}
>
> Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager:
> :977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests
> {}
>
> Thread-88::ERROR::2016-09-06 16:22:03,343::dispatcher::76::Storage.Dispatcher::(wrapper)
> {'status': {'message': "Cannot find master domain:
> u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'",
> 'code': 304}}
>
>
>
> I ‘ve restart the service lvm2-lvmetad.service and enable it as adviced in
> the log without success
>
>
>
>
>
> The engine.log :
>
> 2016-09-06 16:20:02,971 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
> (DefaultQuartzScheduler_Worker-26) [6f596728] Running command:
> InitVdsOnUpCommand internal: true. Entities affected : ID:
> 00000002-0002-0002-0002-000000000193 Type: StoragePool
>
> 2016-09-06 16:20:02,982 INFO [org.ovirt.engine.core.bll.storage.
> ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] Running command: ConnectHostToStoragePoolServersCommand
> internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000193
> Type: StoragePool
>
> 2016-09-06 16:20:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
> ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2,
> HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
> 00000002-0002-0002-0002-000000000193, storageType = NFS, connectionList =
> [{ id: 2ceca65a-90a0-4daf-82a3-366de490a71e, connection:
> unc-srv-oman.cfu.local:/var/lib/exports/iso, iqn: null, vfsType: null,
> mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{
> id: 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6, connection:
> unc-srv-hyp1.cfu.local:/exports/import_domain, iqn: null, vfsType: null,
> mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{
> id: ea46418b-68b3-4f9b-9316-b1d57f17ecbc, connection:
> unc-srv-oman.cfu.local:/data/Master, iqn: null, vfsType: null,
> mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null
> };]), log id: 315222a9
>
> 2016-09-06 16:20:03,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
> ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] FINISH, ConnectStorageServerVDSCommand, return:
> {ea46418b-68b3-4f9b-9316-b1d57f17ecbc=0, 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6=0,
> 2ceca65a-90a0-4daf-82a3-366de490a71e=0}, log id: 315222a9
>
> 2016-09-06 16:20:03,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
> ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2,
> HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
> 00000002-0002-0002-0002-000000000193, storageType = ISCSI, connectionList
> = [{ id: 3efad56e-a86d-4682-b799-51f42713cda6, connection: 192.168.4.1,
> iqn: iqn.1984-05.com.dell:powervault.md3600i.
> 690b11c0005592a90000000051c3efc7, vfsType: null, mountOptions: null,
> nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f9c9ff3
>
> 2016-09-06 16:22:03,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
> ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] FINISH, ConnectStorageServerVDSCommand, return:
> {3efad56e-a86d-4682-b799-51f42713cda6=465}, log id: f9c9ff3
>
> 2016-09-06 16:22:03,976 INFO [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper]
> (DefaultQuartzScheduler_Worker-26) [7d7c3300] The lun with id
> xu5AAG-1FHh-Qxbx-ZXIe-55J0-vscw-kb6fdp was reported as problematic !
>
> 2016-09-06 16:22:03,992 ERROR [org.ovirt.engine.core.dal.
> dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] Correlation ID: null, Call Stack: null, Custom Event ID: -1,
> Message: The error message for connection 192.168.4.1 iqn.1984-05.com.dell:
> powervault.md3600i.690b11c0005592a90000000051c3efc7 (LUN mpathc) returned
> by VDSM was: Failed to setup iSCSI subsystem
>
> 2016-09-06 16:22:03,997 ERROR [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper]
> (DefaultQuartzScheduler_Worker-26) [7d7c3300] The connection with details
> 192.168.4.1 iqn.1984-05.com.dell:powervault.md3600i.
> 690b11c0005592a90000000051c3efc7 (LUN mpathc) failed because of error
> code 465 and error message is: failed to setup iscsi subsystem
>
>
>
> ^^^
>
> Thanks, the issue is here.
>
> Can you please ensure that your host could properly access that iSCSI LUN?
>
>
>
> Can you please check VDSM logs for that time frame?
>
>
>
> 2016-09-06 16:22:03,998 INFO [org.ovirt.engine.core.bll.storage.
> ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] Host unc-srv-hyp2 storage connection was failed
>
> 2016-09-06 16:22:04,004 WARN [org.ovirt.engine.core.dal.
> dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
> [7d7c3300] Correlation ID: 7d7c3300, Call Stack: null, Custom Event ID: -1,
> Message: Failed to connect Host unc-srv-hyp2 to Storage Servers
>
> 2016-09-06 16:22:04,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) START, ConnectStoragePoolVDSCommand(HostName
> = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId =
> 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
> 00000002-0002-0002-0002-000000000193, masterVersion = 7), log id: 321780b7
>
> 2016-09-06 16:22:04,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) Command org.ovirt.engine.core.
> vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value
>
> StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304,
> mMessage=Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193,
> msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc']]
>
> 2016-09-06 16:22:04,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) HostName = unc-srv-hyp2
>
> 2016-09-06 16:22:04,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) Command ConnectStoragePoolVDSCommand(HostName
> = unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId =
> 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
> 00000002-0002-0002-0002-000000000193, masterVersion = 7) execution
> failed. Exception: IRSNoMasterDomainException: IRSGenericException:
> IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
> u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-
> 8f3b-61f03e520cbc'
>
> 2016-09-06 16:22:04,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) FINISH, ConnectStoragePoolVDSCommand,
> log id: 321780b7
>
> 2016-09-06 16:22:04,351 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
> (org.ovirt.thread.pool-8-thread-20) Could not connect host unc-srv-hyp2
> to pool UNICEF with the message: null
>
> 2016-09-06 16:22:04,399 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
> (DefaultQuartzScheduler_Worker-26) [60403257] Running command:
> SetNonOperationalVdsCommand internal: true. Entities affected : ID:
> 5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS
>
> 2016-09-06 16:22:04,426 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (DefaultQuartzScheduler_Worker-26) [60403257] START,
> SetVdsStatusVDSCommand(HostName = unc-srv-hyp2, HostId =
> 5bf45c09-41d4-4125-a4bd-81af2a100db8, status=NonOperational,
> nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE,
> stopSpmFailureLogged=false), log id: 7f47d8b
>
> 2016-09-06 16:22:04,435 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
> (DefaultQuartzScheduler_Worker-26) [60403257] FINISH,
> SetVdsStatusVDSCommand, log id: 7f47d8b
>
> 2016-09-06 16:22:04,448 WARN [org.ovirt.engine.core.dal.
> dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
> [60403257] Correlation ID: 60403257, Job ID: 4203a573-20f2-45e5-b251-e4e09f1ed951,
> Call Stack: null, Custom Event ID: -1, Message: Host unc-srv-hyp2 cannot
> access the Storage Domain(s) <UNKNOWN> attached to the Data Center UNICEF.
> Setting Host state to Non-Operational.
>
> 2016-09-06 16:22:04,484 WARN [org.ovirt.engine.core.dal.
> dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
> [60403257] Correlation ID: null, Call Stack: null, Custom Event ID: -1,
> Message: Power Management test failed for Host unc-srv-hyp2.There is no
> other host in the data center that can be used to test the power management
> settings.
>
> 2016-09-06 16:22:04,488 WARN [org.ovirt.engine.core.dal.
> dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
> [60403257] Correlation ID: 6f596728, Call Stack: null, Custom Event ID: -1,
> Message: Failed to connect Host unc-srv-hyp2 to Storage Pool UNICEF
>
> 2016-09-06 16:22:04,528 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
> (DefaultQuartzScheduler_Worker-26) [709cb459] Running command:
> HandleVdsVersionCommand internal: true. Entities affected : ID:
> 5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS
>
> 2016-09-06 16:22:04,530 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-26) [709cb459] Host
> 5bf45c09-41d4-4125-a4bd-81af2a100db8 : unc-srv-hyp2 is already in
> NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE.
> SetNonOperationalVds command is skipped.
>
>
>
>
> ------------------------------
>
> *Alain VONDRA *
> *Chargé d'exploitation des Systèmes d'Information *
> *Direction Administrative et Financière*
> *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
> *UNICEF France*
> *3 rue Duguay Trouin 75006 PARIS*
> *www.unicef.fr <http://www.unicef.fr/>*
>
> <https://www.unicef.fr/>
>
>
> ------------------------------
>
> *De :* Simone Tiraboschi [mailto:stirabos at redhat.com]
> *Envoyé :* lundi 5 septembre 2016 17:21
> *À :* VONDRA Alain <AVONDRA at unicef.fr>
> *Cc :* Yedidyah Bar David <didi at redhat.com>; users <users at ovirt.org>
>
>
> *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
> CentOS 7
>
>
>
>
>
>
>
> On Mon, Sep 5, 2016 at 5:09 PM, VONDRA Alain <AVONDRA at unicef.fr> wrote:
>
> To resume the situation and be the most clear I can.
>
> I have :
>
> - 2 hypervisors (physical) with CentOS 6.8 : HYP1 and HYP2
>
> - 1 oVirt manager 3.5 (physical and not hosted-engine) with
> CentOS 7.2
>
> And to be able to upgrade to oVirt 3.6 or more I need to upgrade the 2
> hypervisors to CentOS 7.2.
>
> So I began with HYP2, I’ve removed it from the Cluster, and installed it
> with CentOS 7.2, but when I wanted to join it to a new cluster as shown in
> the doc, HYP2 stayed always unresponsive.
>
>
>
> We need vdsm logs to understand the issue here.
>
> Could you please reproduce it?
>
>
>
> So I’ve restored the HYP2 to Centos 6.8, reinstalled it on the previous
> cluster and everything works well, but I really want to upgrade to CentOS
> 7.2 to be up-to-date, because I will have soonly more than two versions
> late.
>
>
>
>
> ------------------------------
>
> *Alain VONDRA *
> *Chargé d'exploitation des Systèmes d'Information *
> *Direction Administrative et Financière*
> *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
> *UNICEF France*
> *3 rue Duguay Trouin 75006 PARIS*
> *www.unicef.fr <http://www.unicef.fr/>*
>
> <https://www.unicef.fr/>
>
>
> ------------------------------
>
> *De :* Yedidyah Bar David [mailto:didi at redhat.com]
> *Envoyé :* lundi 5 septembre 2016 15:53
>
>
> *À :* VONDRA Alain <AVONDRA at unicef.fr>
> *Cc :* Simone Tiraboschi <stirabos at redhat.com>; users <users at ovirt.org>
> *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
> CentOS 7
>
>
>
> On Mon, Sep 5, 2016 at 4:34 PM, VONDRA Alain <AVONDRA at unicef.fr> wrote:
>
> As you say, “This should”, but is there anybody there who tried this
> operation ?
>
>
>
> When I wrote [1], it worked for me.
>
> If something does not work for you, I think it's best to open a bug
>
> and attach all relevant logs, including hosted-engine-setup and vdsm
>
> logs from the host and all engine logs from the engine vm.
>
> I can't tell from your log snippet why your new host failed to attach
>
> to the storage domain. If it's reproducible, please check/post vdsm logs.
>
> [1] https://www.ovirt.org/documentation/how-to/hosted-
> engine-host-OS-upgrade/
>
>
>
> Is there any other option to upgrade the hypervisors ?
>
>
>
> Even if there is another option, the fact that you can't add a host
>
> is probably problematic in itself, no? What if you actually need to
>
> add a host?
>
> Did you try adding a el6 host and it did work?
>
> Best,
>
>
>
>
>
>
> ------------------------------
>
> *Alain VONDRA *
> *Chargé d'exploitation des Systèmes d'Information *
> *Direction Administrative et Financière*
> *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
> *UNICEF France*
> *3 rue Duguay Trouin 75006 PARIS*
> *www.unicef.fr <http://www.unicef.fr/>*
>
> <https://www.unicef.fr/>
>
>
> ------------------------------
>
> *De :* Yedidyah Bar David [mailto:didi at redhat.com]
> *Envoyé :* lundi 5 septembre 2016 12:22
>
>
> *À :* VONDRA Alain <AVONDRA at unicef.fr>
> *Cc :* Simone Tiraboschi <stirabos at redhat.com>; users <users at ovirt.org>
> *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
> CentOS 7
>
>
>
> On Mon, Sep 5, 2016 at 1:11 PM, VONDRA Alain <AVONDRA at unicef.fr> wrote:
>
> Unlikely, I didn’t save them, I had to rollback the host quickly before
> the Week-end.
>
> All that I can tell, that is all seemed to work well during the
> installation of the host, all the networks were connected to the SAN, but
> the host didn’t want to go UP, still staying unresponsive with the message
> below from the oVirt engine.
>
> Can you assure me that the installation of a physical hypervisor with
> CentOS 7, is possible if I put it on a different cluster than the other
> host using CentOS 6.8 ?
>
>
>
> Yes, this should work.
>
>
>
> Thanks
>
>
> ------------------------------
>
> *Alain VONDRA *
> *Chargé d'exploitation des Systèmes d'Information *
> *Direction Administrative et Financière*
> *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
> *UNICEF France*
> *3 rue Duguay Trouin 75006 PARIS*
> *www.unicef.fr <http://www.unicef.fr/>*
>
> <https://www.unicef.fr/>
>
>
> ------------------------------
>
> *De :* Yedidyah Bar David [mailto:didi at redhat.com]
> *Envoyé :* dimanche 4 septembre 2016 08:50
> *À :* VONDRA Alain <AVONDRA at unicef.fr>
> *Cc :* Simone Tiraboschi <stirabos at redhat.com>; users <users at ovirt.org>
>
>
> *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
> CentOS 7
>
>
>
> On Fri, Sep 2, 2016 at 7:31 PM, VONDRA Alain <AVONDRA at unicef.fr> wrote:
>
> Hi,
>
> I’ve followed this doc, and hoped to find a solution with it, so I did’nt
> use the hosted-engine –deploy command, I’ve added a new host and at the end
> of the installation, the host stays unresponsive because unable to be
> attached to the same Storage volume .
>
> Engine log :
>
> 2016-09-02 16:57:01,780 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
> (org.ovirt.thread.pool-8-thread-28) Could not connect host unc-srv-hyp2
> to pool UNICEF with the message: null
>
> 2016-09-02 17:00:01,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
> (org.ovirt.thread.pool-8-thread-3) Command ConnectStoragePoolVDSCommand(HostName
> = unc-srv-hyp2, HostId = ee1c57ce-1c77-47b1-b466-7bf99382dd77, vdsId =
> ee1c57ce-1c77-47b1-b466-7bf99382dd77, storagePoolId =
> 00000002-0002-0002-0002-000000000193, masterVersion = 7) execution
> failed. Exception: IRSNoMasterDomainException: IRSGenericException:
> IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
> u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-
> 8f3b-61f03e520cbc'
>
>
>
> Please check/post vdsm logs from the host. Thanks.
>
>
>
>
>
>
>
>
> ------------------------------
>
> *Alain VONDRA *
> *Chargé d'exploitation des Systèmes d'Information *
> *Direction Administrative et Financière*
> *+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
> *UNICEF France*
> *3 rue Duguay Trouin 75006 PARIS*
> *www.unicef.fr <http://www.unicef.fr/>*
>
> <https://www.unicef.fr/>
>
>
> ------------------------------
>
> *De :* Simone Tiraboschi [mailto:stirabos at redhat.com]
> *Envoyé :* vendredi 2 septembre 2016 18:20
> *À :* VONDRA Alain <AVONDRA at unicef.fr>
> *Cc :* Nir Soffer <nsoffer at redhat.com>; users <users at ovirt.org>
> *Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
> CentOS 7
>
>
>
>
>
>
>
> On Fri, Sep 2, 2016 at 5:21 PM, VONDRA Alain <AVONDRA at unicef.fr> wrote:
>
> Hi,
> I'd like to upgrade my oVirt environment from 3.5 to 3.6 and maybe 4,
> actually the manager oVirt is un 3.5 version, installed on a CentOS 7.2 and
> the two hypervisors are installed on CentOS 6.8.
> I need anyway to upgrade the hosts to be able to move to 3.6.
> I've tried to upgrade the first host, but I had of course issues telling
> that it can't be possible to mix different OS in the same cluster, I've
> also tried to create another cluster to put this host without success.
> What is the best way to upgrade cleanly and safely ?
> Thank you in advance for your advices
>
>
>
> Follow this:
>
> https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/
>
>
>
> Simply ignore the steps that refers to the engine VM if you are with the
> engine on a physical system.
>
>
>
>
> Alain VONDRA
>
> Chargé d'exploitation des Systèmes d'Information
> Direction Administrative et Financière
> +33 1 44 39 77 76
>
> UNICEF France
> 3 rue Duguay Trouin 75006
> PARIS
> www.unicef.fr
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
>
> --
>
> Didi
>
>
>
>
> --
>
> Didi
>
>
>
>
> --
>
> Didi
>
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160906/a984279d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature_Unicef-France_a0c5ef98-a0b5-4b96-9e15-f37a03a75cb1.gif
Type: image/gif
Size: 3115 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160906/a984279d/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 3115 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160906/a984279d/attachment-0003.gif>
More information about the Users
mailing list