Great, I found it !
It was an initiator issue due to the reinstalling of the host. After a
re-configuration of the SAN, evrything works fine and I could Up the brand
new host with CentOS 7.2. I will reproduce the same plan on the other host.
Thank so much for your help
I think the root cause is that ovirt do not manage the iscsi initiator on
the hosts.
If we had an option to set the initiator when adding a host, like the way
you
set the host password, it would be hard to have such bugs.
Can you file a bug about this?
Nir
------------------------------
*Alain VONDRA *
*Chargé d'exploitation des Systèmes d'Information *
*Direction Administrative et Financière*
* +33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
*UNICEF France 3 rue Duguay Trouin 75006 PARIS*
*
www.unicef.fr <
http://www.unicef.fr/> *
<
http://www.unicef.fr>
<
http://www.unicef.fr/>
<
http://www.unicef.fr/> <
http://www.unicef.fr/>
<
https://www.unicef.fr/>
------------------------------
<
http://www.unicef.fr>
*De :* Simone Tiraboschi [mailto:stirabos@redhat.com]
*Envoyé :* mardi 6 septembre 2016 16:48
*À :* VONDRA Alain <AVONDRA(a)unicef.fr>
*Cc :* Yedidyah Bar David <didi(a)redhat.com>; users <users(a)ovirt.org>
*Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
CentOS 7
On Tue, Sep 6, 2016 at 4:25 PM, VONDRA Alain <AVONDRA(a)unicef.fr> wrote:
I’ve just reinstall the host and have the same issue, here is the ERROR
messages from the vdsm logs :
Thread-43::ERROR::2016-09-06
16:02:54,399::hsm::2551::Storage.HSM::(disconnectStorageServer)
Could not disconnect from storageServer
Thread-43::ERROR::2016-09-06
16:02:54,453::hsm::2551::Storage.HSM::(disconnectStorageServer)
Could not disconnect from storageServer
Thread-43::ERROR::2016-09-06
16:02:54,475::hsm::2551::Storage.HSM::(disconnectStorageServer)
Could not disconnect from storageServer
Thread-51::ERROR::2016-09-06
16:05:38,319::hsm::2453::Storage.HSM::(connectStorageServer)
Could not connect to storageServer
Thread-52::ERROR::2016-09-06 16:05:38,636::sdc::137::
Storage.StorageDomainCache::(_findDomain) looking for unfetched domain
cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-52::ERROR::2016-09-06 16:05:38,637::sdc::154::
Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain
cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-52::ERROR::2016-09-06 16:05:38,756::sdc::143::
Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
not found
Thread-52::ERROR::2016-09-06 16:05:38,769::task::866::
Storage.TaskManager.Task::(_setError)
Task=`1cfc5c30-fc82-44d6-a296-223fa9426417`::Unexpected
error
Thread-52::ERROR::2016-09-06 16:05:38,788::dispatcher::76::Storage.Dispatcher::(wrapper)
{'status': {'message': "Cannot find master domain:
u'spUUID=00000002-0002-0002-0002-000000000193,
msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'",
'code': 304}}
This SD is present on the host1 (CentOS 6.8) :
[root@unc-srv-hyp1 ~]$ ll /rhev/data-center/
00000002-0002-0002-0002-000000000193/ mnt/
Not in the host2 (CentOS 7.2) :
[root@unc-srv-hyp2 ~]# ll /rhev/data-center/
total 4
drwxr-xr-x. 5 vdsm kvm 4096 6 sept. 16:03 mnt
The vdsm.log more complete :
Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsi::424::Storage.ISCSI::(rescan)
Performing SCSI scan, this will take up to 30 seconds
Thread-88::DEBUG::2016-09-06 16:22:03,057::iscsiadm::97::Storage.Misc.excCmd::(_runCmd)
/usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,078::misc::751::
Storage.SamplingMethod::(__call__) Returning last result
Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::741::
Storage.SamplingMethod::(__call__) Trying to enter sampling method
(storage.hba.rescan)
Thread-88::DEBUG::2016-09-06 16:22:03,079::misc::743::
Storage.SamplingMethod::(__call__) Got in to sampling method
Thread-88::DEBUG::2016-09-06 16:22:03,080::hba::53::Storage.HBA::(rescan)
Starting scan
Thread-88::DEBUG::2016-09-06 16:22:03,080::utils::755::Storage.HBA::(execCmd)
/usr/bin/sudo -n /usr/libexec/vdsm/fc-scan (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,134::hba::66::Storage.HBA::(rescan)
Scan finished
Thread-88::DEBUG::2016-09-06 16:22:03,134::misc::751::
Storage.SamplingMethod::(__call__) Returning last result
Thread-88::DEBUG::2016-09-06 16:22:03,135::multipath::131::Storage.Misc.excCmd::(rescan)
/usr/bin/sudo -n /sbin/multipath (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,201::multipath::131::Storage.Misc.excCmd::(rescan)
SUCCESS: <err> = ''; <rc> = 0
Thread-88::DEBUG::2016-09-06 16:22:03,202::utils::755::root::(execCmd)
/sbin/udevadm settle --timeout=5 (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,227::utils::775::root::(execCmd)
SUCCESS: <err> = ''; <rc> = 0
Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::498::
Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate
operation' got the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,228::lvm::500::
Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate
operation' released the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::509::
Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate
operation' got the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::511::
Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate
operation' released the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,229::lvm::529::
Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate
operation' got the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,230::lvm::531::
Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate
operation' released the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,230::misc::751::
Storage.SamplingMethod::(__call__) Returning last result
Thread-88::ERROR::2016-09-06 16:22:03,230::sdc::137::
Storage.StorageDomainCache::(_findDomain) looking for unfetched domain
cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-88::ERROR::2016-09-06 16:22:03,231::sdc::154::
Storage.StorageDomainCache::(_findUnfetchedDomain) looking for domain
cc9ab4b2-9880-427b-8f3b-61f03e520cbc
Thread-88::DEBUG::2016-09-06 16:22:03,231::lvm::371::
Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got
the operation mutex
Thread-88::DEBUG::2016-09-06 16:22:03,233::lvm::291::Storage.Misc.excCmd::(cmd)
/usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names =
["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [
'\''r|.*|'\'' ] } global { locking_type=1
prioritise_write_locks=1
wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days =
0 } ' --noheadings --units b --nosuffix --separator '|'
--ignoreskippedcluster -o uuid,name,attr,size,free,
extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name
cc9ab4b2-9880-427b-8f3b-61f03e520cbc (cwd None)
Thread-88::DEBUG::2016-09-06 16:22:03,275::lvm::291::Storage.Misc.excCmd::(cmd)
FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart
lvmetad before enabling it!\n Volume group
"cc9ab4b2-9880-427b-8f3b-61f03e520cbc"
not found\n Cannot process volume group cc9ab4b2-9880-427b-8f3b-61f03e520cbc\n';
<rc> = 5
Thread-88::WARNING::2016-09-06 16:22:03,277::lvm::376::Storage.LVM::(_reloadvgs)
lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart
lvmetad before enabling it!', ' Volume group
"cc9ab4b2-9880-427b-8f3b-61f03e520cbc"
not found', ' Cannot process volume group cc9ab4b2-9880-427b-8f3b-
61f03e520cbc']
Thread-88::DEBUG::2016-09-06 16:22:03,277::lvm::416::
Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation'
released the operation mutex
Thread-90::DEBUG::2016-09-06 16:22:03,293::__init__::318::IOProcessClient::(_run)
Starting IOProcess...
Thread-91::DEBUG::2016-09-06 16:22:03,314::__init__::318::IOProcessClient::(_run)
Starting IOProcess...
Thread-88::ERROR::2016-09-06 16:22:03,334::sdc::143::
Storage.StorageDomainCache::(_findDomain) domain cc9ab4b2-9880-427b-8f3b-61f03e520cbc
not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain
dom = findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 171, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc',)
Thread-88::DEBUG::2016-09-06 16:22:03,335::resourceManager:
:616::Storage.ResourceManager::(releaseResource) Trying to release
resource 'Storage.00000002-0002-0002-0002-000000000193'
Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager:
:635::Storage.ResourceManager::(releaseResource) Released resource
'Storage.00000002-0002-0002-0002-000000000193' (0 active users)
Thread-88::DEBUG::2016-09-06 16:22:03,336::resourceManager:
:641::Storage.ResourceManager::(releaseResource) Resource
'Storage.00000002-0002-0002-0002-000000000193' is free, finding out if
anyone is waiting for it.
Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager:
:649::Storage.ResourceManager::(releaseResource) No one is waiting for
resource 'Storage.00000002-0002-0002-0002-000000000193', Clearing records.
Thread-88::DEBUG::2016-09-06 16:22:03,337::resourceManager:
:616::Storage.ResourceManager::(releaseResource) Trying to release
resource 'Storage.HsmDomainMonitorLock'
Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager:
:635::Storage.ResourceManager::(releaseResource) Released resource
'Storage.HsmDomainMonitorLock' (0 active users)
Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager:
:641::Storage.ResourceManager::(releaseResource) Resource
'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting
for it.
Thread-88::DEBUG::2016-09-06 16:22:03,338::resourceManager:
:649::Storage.ResourceManager::(releaseResource) No one is waiting for
resource 'Storage.HsmDomainMonitorLock', Clearing records.
Thread-88::ERROR::2016-09-06 16:22:03,338::task::866::
Storage.TaskManager.Task::(_setError)
Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Unexpected
error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 45, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 1039, in connectStoragePool
spUUID, hostID, msdUUID, masterVersion, domainsMap)
File "/usr/share/vdsm/storage/hsm.py", line 1104, in _connectStoragePool
res = pool.connect(hostID, msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 637, in connect
self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1179, in __rebuild
self.setMasterDomain(msdUUID, masterVersion)
File "/usr/share/vdsm/storage/sp.py", line 1390, in setMasterDomain
raise se.StoragePoolMasterNotFound(self.spUUID, msdUUID)
StoragePoolMasterNotFound: Cannot find master domain:
u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-
8f3b-61f03e520cbc'
Thread-88::DEBUG::2016-09-06 16:22:03,339::task::885::
Storage.TaskManager.Task::(_run) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._run:
4898233c-f6c6-45fa-a2ee-42e63e189063 (u'00000002-0002-0002-0002-000000000193',
2, u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc', 7,
{u'015799ac-ec66-4a9d-a8d6-6e9ec980972d':
u'active', u'2fcd37ce-cb88-4026-88df-d4d472b41ecf': u'active',
u'76a1fed9-2e60-4b3e-9f00-efca8acd133d': u'active',
u'7e40772a-fe94-4fb2-94c4-6198bed04a6a': u'active',
u'cb4c84c1-489a-433f-999a-f1aeec9d62cf': u'active',
u'ea05d014-f8f0-4f1d-906e-2e93c8907d7d': u'active',
u'cc9ab4b2-9880-427b-8f3b-61f03e520cbc': u'active'}) {} failed -
stopping
task
Thread-88::DEBUG::2016-09-06 16:22:03,339::task::1217::
Storage.TaskManager.Task::(stop) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::stopping
in state preparing (force False)
Thread-88::DEBUG::2016-09-06 16:22:03,340::task::993::
Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref
1 aborting True
Thread-88::INFO::2016-09-06 16:22:03,340::task::1171::
Storage.TaskManager.Task::(prepare)
Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::aborting:
Task is aborted: 'Cannot find master domain' - code 304
Thread-88::DEBUG::2016-09-06 16:22:03,340::task::1176::
Storage.TaskManager.Task::(prepare)
Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Prepare:
aborted: Cannot find master domain
Thread-88::DEBUG::2016-09-06 16:22:03,341::task::993::
Storage.TaskManager.Task::(_decref) Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::ref
0 aborting True
Thread-88::DEBUG::2016-09-06 16:22:03,341::task::928::
Storage.TaskManager.Task::(_doAbort)
Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::Task._doAbort:
force False
Thread-88::DEBUG::2016-09-06 16:22:03,341::resourceManager:
:977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests
{}
Thread-88::DEBUG::2016-09-06 16:22:03,341::task::595::
Storage.TaskManager.Task::(_updateState)
Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving
from state preparing -> state aborting
Thread-88::DEBUG::2016-09-06 16:22:03,342::task::550::
Storage.TaskManager.Task::(__state_aborting)
Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::_aborting: recover policy
none
Thread-88::DEBUG::2016-09-06 16:22:03,342::task::595::
Storage.TaskManager.Task::(_updateState)
Task=`4898233c-f6c6-45fa-a2ee-42e63e189063`::moving
from state aborting -> state failed
Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager:
:940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll
requests {} resources {}
Thread-88::DEBUG::2016-09-06 16:22:03,342::resourceManager:
:977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests
{}
Thread-88::ERROR::2016-09-06 16:22:03,343::dispatcher::76::Storage.Dispatcher::(wrapper)
{'status': {'message': "Cannot find master domain:
u'spUUID=00000002-0002-0002-0002-000000000193,
msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc'",
'code': 304}}
I ‘ve restart the service lvm2-lvmetad.service and enable it as adviced in
the log without success
The engine.log :
2016-09-06 16:20:02,971 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(DefaultQuartzScheduler_Worker-26) [6f596728] Running command:
InitVdsOnUpCommand internal: true. Entities affected : ID:
00000002-0002-0002-0002-000000000193 Type: StoragePool
2016-09-06 16:20:02,982 INFO [org.ovirt.engine.core.bll.storage.
ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] Running command: ConnectHostToStoragePoolServersCommand
internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000193
Type: StoragePool
2016-09-06 16:20:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2,
HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
00000002-0002-0002-0002-000000000193, storageType = NFS, connectionList =
[{ id: 2ceca65a-90a0-4daf-82a3-366de490a71e, connection:
unc-srv-oman.cfu.local:/var/lib/exports/iso, iqn: null, vfsType: null,
mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{
id: 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6, connection:
unc-srv-hyp1.cfu.local:/exports/import_domain, iqn: null, vfsType: null,
mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{
id: ea46418b-68b3-4f9b-9316-b1d57f17ecbc, connection:
unc-srv-oman.cfu.local:/data/Master, iqn: null, vfsType: null,
mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null
};]), log id: 315222a9
2016-09-06 16:20:03,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] FINISH, ConnectStorageServerVDSCommand, return:
{ea46418b-68b3-4f9b-9316-b1d57f17ecbc=0, 76a3f706-e7d4-49cd-9e0a-da6061b6b2d6=0,
2ceca65a-90a0-4daf-82a3-366de490a71e=0}, log id: 315222a9
2016-09-06 16:20:03,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] START, ConnectStorageServerVDSCommand(HostName = unc-srv-hyp2,
HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
00000002-0002-0002-0002-000000000193, storageType = ISCSI, connectionList
= [{ id: 3efad56e-a86d-4682-b799-51f42713cda6, connection: 192.168.4.1,
iqn: iqn.1984-05.com.dell:powervault.md3600i.
690b11c0005592a90000000051c3efc7, vfsType: null, mountOptions: null,
nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f9c9ff3
2016-09-06 16:22:03,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.
ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] FINISH, ConnectStorageServerVDSCommand, return:
{3efad56e-a86d-4682-b799-51f42713cda6=465}, log id: f9c9ff3
2016-09-06 16:22:03,976 INFO [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper]
(DefaultQuartzScheduler_Worker-26) [7d7c3300] The lun with id
xu5AAG-1FHh-Qxbx-ZXIe-55J0-vscw-kb6fdp was reported as problematic !
2016-09-06 16:22:03,992 ERROR [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] Correlation ID: null, Call Stack: null, Custom Event ID: -1,
Message: The error message for connection 192.168.4.1 iqn.1984-05.com.dell:
powervault.md3600i.690b11c0005592a90000000051c3efc7 (LUN mpathc) returned
by VDSM was: Failed to setup iSCSI subsystem
2016-09-06 16:22:03,997 ERROR [org.ovirt.engine.core.bll.storage.ISCSIStorageHelper]
(DefaultQuartzScheduler_Worker-26) [7d7c3300] The connection with details
192.168.4.1 iqn.1984-05.com.dell:powervault.md3600i.
690b11c0005592a90000000051c3efc7 (LUN mpathc) failed because of error
code 465 and error message is: failed to setup iscsi subsystem
^^^
Thanks, the issue is here.
Can you please ensure that your host could properly access that iSCSI LUN?
Can you please check VDSM logs for that time frame?
2016-09-06 16:22:03,998 INFO [org.ovirt.engine.core.bll.storage.
ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] Host unc-srv-hyp2 storage connection was failed
2016-09-06 16:22:04,004 WARN [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
[7d7c3300] Correlation ID: 7d7c3300, Call Stack: null, Custom Event ID: -1,
Message: Failed to connect Host unc-srv-hyp2 to Storage Servers
2016-09-06 16:22:04,040 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-20) START, ConnectStoragePoolVDSCommand(HostName
= unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId =
5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
00000002-0002-0002-0002-000000000193, masterVersion = 7), log id: 321780b7
2016-09-06 16:22:04,346 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-20) Command org.ovirt.engine.core.
vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand return value
StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=304,
mMessage=Cannot find master domain: u'spUUID=00000002-0002-0002-0002-000000000193,
msdUUID=cc9ab4b2-9880-427b-8f3b-61f03e520cbc']]
2016-09-06 16:22:04,347 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-20) HostName = unc-srv-hyp2
2016-09-06 16:22:04,348 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-20) Command ConnectStoragePoolVDSCommand(HostName
= unc-srv-hyp2, HostId = 5bf45c09-41d4-4125-a4bd-81af2a100db8, vdsId =
5bf45c09-41d4-4125-a4bd-81af2a100db8, storagePoolId =
00000002-0002-0002-0002-000000000193, masterVersion = 7) execution
failed. Exception: IRSNoMasterDomainException: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-
8f3b-61f03e520cbc'
2016-09-06 16:22:04,350 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-20) FINISH, ConnectStoragePoolVDSCommand,
log id: 321780b7
2016-09-06 16:22:04,351 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(org.ovirt.thread.pool-8-thread-20) Could not connect host unc-srv-hyp2
to pool UNICEF with the message: null
2016-09-06 16:22:04,399 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(DefaultQuartzScheduler_Worker-26) [60403257] Running command:
SetNonOperationalVdsCommand internal: true. Entities affected : ID:
5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS
2016-09-06 16:22:04,426 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(DefaultQuartzScheduler_Worker-26) [60403257] START,
SetVdsStatusVDSCommand(HostName = unc-srv-hyp2, HostId =
5bf45c09-41d4-4125-a4bd-81af2a100db8, status=NonOperational,
nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE,
stopSpmFailureLogged=false), log id: 7f47d8b
2016-09-06 16:22:04,435 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand]
(DefaultQuartzScheduler_Worker-26) [60403257] FINISH,
SetVdsStatusVDSCommand, log id: 7f47d8b
2016-09-06 16:22:04,448 WARN [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
[60403257] Correlation ID: 60403257, Job ID: 4203a573-20f2-45e5-b251-e4e09f1ed951,
Call Stack: null, Custom Event ID: -1, Message: Host unc-srv-hyp2 cannot
access the Storage Domain(s) <UNKNOWN> attached to the Data Center UNICEF.
Setting Host state to Non-Operational.
2016-09-06 16:22:04,484 WARN [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
[60403257] Correlation ID: null, Call Stack: null, Custom Event ID: -1,
Message: Power Management test failed for Host unc-srv-hyp2.There is no
other host in the data center that can be used to test the power management
settings.
2016-09-06 16:22:04,488 WARN [org.ovirt.engine.core.dal.
dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26)
[60403257] Correlation ID: 6f596728, Call Stack: null, Custom Event ID: -1,
Message: Failed to connect Host unc-srv-hyp2 to Storage Pool UNICEF
2016-09-06 16:22:04,528 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand]
(DefaultQuartzScheduler_Worker-26) [709cb459] Running command:
HandleVdsVersionCommand internal: true. Entities affected : ID:
5bf45c09-41d4-4125-a4bd-81af2a100db8 Type: VDS
2016-09-06 16:22:04,530 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
(DefaultQuartzScheduler_Worker-26) [709cb459] Host
5bf45c09-41d4-4125-a4bd-81af2a100db8 : unc-srv-hyp2 is already in
NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE.
SetNonOperationalVds command is skipped.
------------------------------
*Alain VONDRA *
*Chargé d'exploitation des Systèmes d'Information *
*Direction Administrative et Financière*
*+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
*UNICEF France*
*3 rue Duguay Trouin 75006 PARIS*
*www.unicef.fr <
http://www.unicef.fr/>*
<
https://www.unicef.fr/>
------------------------------
*De :* Simone Tiraboschi [mailto:stirabos@redhat.com]
*Envoyé :* lundi 5 septembre 2016 17:21
*À :* VONDRA Alain <AVONDRA(a)unicef.fr>
*Cc :* Yedidyah Bar David <didi(a)redhat.com>; users <users(a)ovirt.org>
*Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
CentOS 7
On Mon, Sep 5, 2016 at 5:09 PM, VONDRA Alain <AVONDRA(a)unicef.fr> wrote:
To resume the situation and be the most clear I can.
I have :
- 2 hypervisors (physical) with CentOS 6.8 : HYP1 and HYP2
- 1 oVirt manager 3.5 (physical and not hosted-engine) with
CentOS 7.2
And to be able to upgrade to oVirt 3.6 or more I need to upgrade the 2
hypervisors to CentOS 7.2.
So I began with HYP2, I’ve removed it from the Cluster, and installed it
with CentOS 7.2, but when I wanted to join it to a new cluster as shown in
the doc, HYP2 stayed always unresponsive.
We need vdsm logs to understand the issue here.
Could you please reproduce it?
So I’ve restored the HYP2 to Centos 6.8, reinstalled it on the previous
cluster and everything works well, but I really want to upgrade to CentOS
7.2 to be up-to-date, because I will have soonly more than two versions
late.
------------------------------
*Alain VONDRA *
*Chargé d'exploitation des Systèmes d'Information *
*Direction Administrative et Financière*
*+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
*UNICEF France*
*3 rue Duguay Trouin 75006 PARIS*
*www.unicef.fr <
http://www.unicef.fr/>*
<
https://www.unicef.fr/>
------------------------------
*De :* Yedidyah Bar David [mailto:didi@redhat.com]
*Envoyé :* lundi 5 septembre 2016 15:53
*À :* VONDRA Alain <AVONDRA(a)unicef.fr>
*Cc :* Simone Tiraboschi <stirabos(a)redhat.com>; users <users(a)ovirt.org>
*Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
CentOS 7
On Mon, Sep 5, 2016 at 4:34 PM, VONDRA Alain <AVONDRA(a)unicef.fr> wrote:
As you say, “This should”, but is there anybody there who tried this
operation ?
When I wrote [1], it worked for me.
If something does not work for you, I think it's best to open a bug
and attach all relevant logs, including hosted-engine-setup and vdsm
logs from the host and all engine logs from the engine vm.
I can't tell from your log snippet why your new host failed to attach
to the storage domain. If it's reproducible, please check/post vdsm logs.
[1]
https://www.ovirt.org/documentation/how-to/hosted-
engine-host-OS-upgrade/
Is there any other option to upgrade the hypervisors ?
Even if there is another option, the fact that you can't add a host
is probably problematic in itself, no? What if you actually need to
add a host?
Did you try adding a el6 host and it did work?
Best,
------------------------------
*Alain VONDRA *
*Chargé d'exploitation des Systèmes d'Information *
*Direction Administrative et Financière*
*+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
*UNICEF France*
*3 rue Duguay Trouin 75006 PARIS*
*www.unicef.fr <
http://www.unicef.fr/>*
<
https://www.unicef.fr/>
------------------------------
*De :* Yedidyah Bar David [mailto:didi@redhat.com]
*Envoyé :* lundi 5 septembre 2016 12:22
*À :* VONDRA Alain <AVONDRA(a)unicef.fr>
*Cc :* Simone Tiraboschi <stirabos(a)redhat.com>; users <users(a)ovirt.org>
*Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
CentOS 7
On Mon, Sep 5, 2016 at 1:11 PM, VONDRA Alain <AVONDRA(a)unicef.fr> wrote:
Unlikely, I didn’t save them, I had to rollback the host quickly before
the Week-end.
All that I can tell, that is all seemed to work well during the
installation of the host, all the networks were connected to the SAN, but
the host didn’t want to go UP, still staying unresponsive with the message
below from the oVirt engine.
Can you assure me that the installation of a physical hypervisor with
CentOS 7, is possible if I put it on a different cluster than the other
host using CentOS 6.8 ?
Yes, this should work.
Thanks
------------------------------
*Alain VONDRA *
*Chargé d'exploitation des Systèmes d'Information *
*Direction Administrative et Financière*
*+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
*UNICEF France*
*3 rue Duguay Trouin 75006 PARIS*
*www.unicef.fr <
http://www.unicef.fr/>*
<
https://www.unicef.fr/>
------------------------------
*De :* Yedidyah Bar David [mailto:didi@redhat.com]
*Envoyé :* dimanche 4 septembre 2016 08:50
*À :* VONDRA Alain <AVONDRA(a)unicef.fr>
*Cc :* Simone Tiraboschi <stirabos(a)redhat.com>; users <users(a)ovirt.org>
*Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
CentOS 7
On Fri, Sep 2, 2016 at 7:31 PM, VONDRA Alain <AVONDRA(a)unicef.fr> wrote:
Hi,
I’ve followed this doc, and hoped to find a solution with it, so I did’nt
use the hosted-engine –deploy command, I’ve added a new host and at the end
of the installation, the host stays unresponsive because unable to be
attached to the same Storage volume .
Engine log :
2016-09-02 16:57:01,780 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand]
(org.ovirt.thread.pool-8-thread-28) Could not connect host unc-srv-hyp2
to pool UNICEF with the message: null
2016-09-02 17:00:01,634 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(org.ovirt.thread.pool-8-thread-3) Command ConnectStoragePoolVDSCommand(HostName
= unc-srv-hyp2, HostId = ee1c57ce-1c77-47b1-b466-7bf99382dd77, vdsId =
ee1c57ce-1c77-47b1-b466-7bf99382dd77, storagePoolId =
00000002-0002-0002-0002-000000000193, masterVersion = 7) execution
failed. Exception: IRSNoMasterDomainException: IRSGenericException:
IRSErrorException: IRSNoMasterDomainException: Cannot find master domain:
u'spUUID=00000002-0002-0002-0002-000000000193, msdUUID=cc9ab4b2-9880-427b-
8f3b-61f03e520cbc'
Please check/post vdsm logs from the host. Thanks.
------------------------------
*Alain VONDRA *
*Chargé d'exploitation des Systèmes d'Information *
*Direction Administrative et Financière*
*+33 1 44 39 77 76 <%2B33%201%2044%2039%2077%2076> *
*UNICEF France*
*3 rue Duguay Trouin 75006 PARIS*
*www.unicef.fr <
http://www.unicef.fr/>*
<
https://www.unicef.fr/>
------------------------------
*De :* Simone Tiraboschi [mailto:stirabos@redhat.com]
*Envoyé :* vendredi 2 septembre 2016 18:20
*À :* VONDRA Alain <AVONDRA(a)unicef.fr>
*Cc :* Nir Soffer <nsoffer(a)redhat.com>; users <users(a)ovirt.org>
*Objet :* Re: [ovirt-users] HELP Upgrade hypervisors from CentOS 6.8 to
CentOS 7
On Fri, Sep 2, 2016 at 5:21 PM, VONDRA Alain <AVONDRA(a)unicef.fr> wrote:
Hi,
I'd like to upgrade my oVirt environment from 3.5 to 3.6 and maybe 4,
actually the manager oVirt is un 3.5 version, installed on a CentOS 7.2 and
the two hypervisors are installed on CentOS 6.8.
I need anyway to upgrade the hosts to be able to move to 3.6.
I've tried to upgrade the first host, but I had of course issues telling
that it can't be possible to mix different OS in the same cluster, I've
also tried to create another cluster to put this host without success.
What is the best way to upgrade cleanly and safely ?
Thank you in advance for your advices
Follow this:
https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/
Simply ignore the steps that refers to the engine VM if you are with the
engine on a physical system.
Alain VONDRA
Chargé d'exploitation des Systèmes d'Information
Direction Administrative et Financière
+33 1 44 39 77 76
UNICEF France
3 rue Duguay Trouin 75006
PARIS
www.unicef.fr
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Didi
--
Didi
--
Didi
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users