This is a multi-part message in MIME format. --------------060401020402070903050401 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi, I'm adding two twin nodes to a cluster and I was able to add the first while the second fails as it removes the iscsi sessions and don't bring them back and thus fails with the message : Host Dell4 cannot access one of the Storage Domains attached to the Data Center Default. Setting Host state to Non-Operational. Here is the vdsm.log entries for one activationintent: Thread-20::DEBUG::2013-12-04 09:02:41,760::task::579::TaskManager.Task::(_updateState) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::moving from state init -> state preparing Thread-20::INFO::2013-12-04 09:02:41,760::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) Thread-20::INFO::2013-12-04 09:02:41,760::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {} Thread-20::DEBUG::2013-12-04 09:02:41,760::task::1168::TaskManager.Task::(prepare) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::finished: {} Thread-20::DEBUG::2013-12-04 09:02:41,761::task::579::TaskManager.Task::(_updateState) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::moving from state preparing -> state finish ed Thread-20::DEBUG::2013-12-04 09:02:41,761::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-20::DEBUG::2013-12-04 09:02:41,761::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-20::DEBUG::2013-12-04 09:02:41,761::task::974::TaskManager.Task::(_decref) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::ref 0 aborting False Thread-22::DEBUG::2013-12-04 09:02:41,814::BindingXMLRPC::177::vds::(wrapper) client [192.168.128.79] Thread-22::DEBUG::2013-12-04 09:02:41,815::task::579::TaskManager.Task::(_updateState) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::moving from state init -> state preparing Thread-22::INFO::2013-12-04 09:02:41,815::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', hostID=5, s csiKey='5849b030-626e-47cb-ad90-3ce782d831b3', remove=False, options=None) Thread-22::WARNING::2013-12-04 09:02:41,815::hsm::1094::Storage.HSM::(disconnectStoragePool) disconnect sp: 5849b030-626e-47cb-ad90-3ce782d831b3 failed. Known pools {} Thread-22::INFO::2013-12-04 09:02:41,815::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None Thread-22::DEBUG::2013-12-04 09:02:41,815::task::1168::TaskManager.Task::(prepare) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::finished: None Thread-22::DEBUG::2013-12-04 09:02:41,816::task::579::TaskManager.Task::(_updateState) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::moving from state preparing -> state finish ed Thread-22::DEBUG::2013-12-04 09:02:41,816::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-22::DEBUG::2013-12-04 09:02:41,816::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-22::DEBUG::2013-12-04 09:02:41,816::task::974::TaskManager.Task::(_decref) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::ref 0 aborting False Thread-23::DEBUG::2013-12-04 09:02:41,872::BindingXMLRPC::177::vds::(wrapper) client [192.168.128.79] Thread-23::DEBUG::2013-12-04 09:02:41,872::task::579::TaskManager.Task::(_updateState) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::moving from state init -> state preparing Thread-23::INFO::2013-12-04 09:02:41,873::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=3, spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', conList=[{'connection': '192.168.130.10', 'iqn': 'iqn.1992-08.com.netapp:sn.135056614', 'portal': '1000', 'user': '', 'password': '******', 'id': '478b35df-da9a-4cb0-847f-2f61f17a02f7', 'port': '3260'}, {'connection': '192.168.131.10', 'iqn': 'iqn.1992-08.com.netapp:sn.135056614', 'portal': '1001', 'user': '', 'password': '******', 'id': '717e3505-3290-421b-bb00-25e046f63362', 'port': '3260'}], options=None) Thread-23::DEBUG::2013-12-04 09:02:41,876::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) Thread-23::DEBUG::2013-12-04 09:02:41,876::misc::811::SamplingMethod::(__call__) Got in to sampling method Thread-23::DEBUG::2013-12-04 09:02:41,877::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) Thread-23::DEBUG::2013-12-04 09:02:41,877::misc::811::SamplingMethod::(__call__) Got in to sampling method Thread-23::DEBUG::2013-12-04 09:02:41,877::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None) Thread-23::DEBUG::2013-12-04 09:02:41,896::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21 Thread-23::DEBUG::2013-12-04 09:02:41,896::misc::819::SamplingMethod::(__call__) Returning last result Thread-23::DEBUG::2013-12-04 09:02:44,915::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None) Thread-23::DEBUG::2013-12-04 09:02:44,966::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 Thread-23::DEBUG::2013-12-04 09:02:44,967::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex Thread-23::DEBUG::2013-12-04 09:02:44,967::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex Thread-23::DEBUG::2013-12-04 09:02:44,968::misc::819::SamplingMethod::(__call__) Returning last result Thread-23::INFO::2013-12-04 09:02:44,969::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '478b35df-da9a-4cb0-847f-2f61f17a02f7'}, {'status': 0, 'id': '717e3505-3290-421b-bb00-25e046f63362'}]} Thread-23::DEBUG::2013-12-04 09:02:44,969::task::1168::TaskManager.Task::(prepare) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::finished: {'statuslist': [{'status': 0, 'id': '478b35df-da9a-4cb0-847f-2f61f17a02f7'}, {'status': 0, 'id': '717e3505-3290-421b-bb00-25e046f63362'}]} Thread-23::DEBUG::2013-12-04 09:02:44,969::task::579::TaskManager.Task::(_updateState) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::moving from state preparing -> state finished Thread-23::DEBUG::2013-12-04 09:02:44,969::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} Thread-23::DEBUG::2013-12-04 09:02:44,969::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} Thread-23::DEBUG::2013-12-04 09:02:44,970::task::974::TaskManager.Task::(_decref) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::ref 0 aborting False Regards, --------------060401020402070903050401 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hi,<br> <br> I'm adding two twin nodes to a cluster and I was able to add the first while the second fails as it removes the iscsi sessions and don't bring them back and thus fails with the message :<br> <br> <div title="" tabindex="0" style="outline-style: none;" __gwt_cell="cell-gwt-uid-303550"> <div id="gwt-uid-3368_col2_row2">Host Dell4 cannot access one of the Storage Domains attached to the Data Center Default. Setting Host state to Non-Operational.<br> <br> <small><big>Here is the vdsm.log entries for one activation<big> </big></big></small>intent:</div> </div> <br> <br> Thread-20::DEBUG::2013-12-04 09:02:41,760::task::579::TaskManager.Task::(_updateState) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::moving from state init -> state preparing<br> Thread-20::<a class="moz-txt-link-freetext" href="INFO::2013-12-04">INFO::2013-12-04</a> 09:02:41,760::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None)<br> Thread-20::<a class="moz-txt-link-freetext" href="INFO::2013-12-04">INFO::2013-12-04</a> 09:02:41,760::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {}<br> Thread-20::DEBUG::2013-12-04 09:02:41,760::task::1168::TaskManager.Task::(prepare) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::finished: {}<br> Thread-20::DEBUG::2013-12-04 09:02:41,761::task::579::TaskManager.Task::(_updateState) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::moving from state preparing -> state finish<br> ed<br> Thread-20::DEBUG::2013-12-04 09:02:41,761::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<br> Thread-20::DEBUG::2013-12-04 09:02:41,761::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<br> Thread-20::DEBUG::2013-12-04 09:02:41,761::task::974::TaskManager.Task::(_decref) Task=`e3529e49-3083-422e-b2e6-c7d8c47b886a`::ref 0 aborting False<br> Thread-22::DEBUG::2013-12-04 09:02:41,814::BindingXMLRPC::177::vds::(wrapper) client [192.168.128.79]<br> Thread-22::DEBUG::2013-12-04 09:02:41,815::task::579::TaskManager.Task::(_updateState) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::moving from state init -> state preparing<br> Thread-22::<a class="moz-txt-link-freetext" href="INFO::2013-12-04">INFO::2013-12-04</a> 09:02:41,815::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStoragePool(spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', hostID=5, s<br> csiKey='5849b030-626e-47cb-ad90-3ce782d831b3', remove=False, options=None)<br> Thread-22::WARNING::2013-12-04 09:02:41,815::hsm::1094::Storage.HSM::(disconnectStoragePool) disconnect sp: 5849b030-626e-47cb-ad90-3ce782d831b3 failed. Known pools {}<br> Thread-22::<a class="moz-txt-link-freetext" href="INFO::2013-12-04">INFO::2013-12-04</a> 09:02:41,815::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStoragePool, Return response: None<br> Thread-22::DEBUG::2013-12-04 09:02:41,815::task::1168::TaskManager.Task::(prepare) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::finished: None<br> Thread-22::DEBUG::2013-12-04 09:02:41,816::task::579::TaskManager.Task::(_updateState) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::moving from state preparing -> state finish<br> ed<br> Thread-22::DEBUG::2013-12-04 09:02:41,816::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<br> Thread-22::DEBUG::2013-12-04 09:02:41,816::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<br> Thread-22::DEBUG::2013-12-04 09:02:41,816::task::974::TaskManager.Task::(_decref) Task=`337b1ad3-2559-450c-9133-4fb733e6b5da`::ref 0 aborting False<br> Thread-23::DEBUG::2013-12-04 09:02:41,872::BindingXMLRPC::177::vds::(wrapper) client [192.168.128.79]<br> Thread-23::DEBUG::2013-12-04 09:02:41,872::task::579::TaskManager.Task::(_updateState) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::moving from state init -> state preparing<br> Thread-23::<a class="moz-txt-link-freetext" href="INFO::2013-12-04">INFO::2013-12-04</a> 09:02:41,873::logUtils::44::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=3, spUUID='5849b030-626e-47cb-ad90-3ce782d831b3', conList=[{'connection': '192.168.130.10', 'iqn': 'iqn.1992-08.com.netapp:sn.135056614', 'portal': '1000', 'user': '', 'password': '******', 'id': '478b35df-da9a-4cb0-847f-2f61f17a02f7', 'port': '3260'}, {'connection': '192.168.131.10', 'iqn': 'iqn.1992-08.com.netapp:sn.135056614', 'portal': '1001', 'user': '', 'password': '******', 'id': '717e3505-3290-421b-bb00-25e046f63362', 'port': '3260'}], options=None)<br> Thread-23::DEBUG::2013-12-04 09:02:41,876::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)<br> Thread-23::DEBUG::2013-12-04 09:02:41,876::misc::811::SamplingMethod::(__call__) Got in to sampling method<br> Thread-23::DEBUG::2013-12-04 09:02:41,877::misc::809::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)<br> Thread-23::DEBUG::2013-12-04 09:02:41,877::misc::811::SamplingMethod::(__call__) Got in to sampling method<br> Thread-23::DEBUG::2013-12-04 09:02:41,877::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)<br> Thread-23::DEBUG::2013-12-04 09:02:41,896::iscsiadm::91::Storage.Misc.excCmd::(_runCmd) FAILED: <err> = 'iscsiadm: No session found.\n'; <rc> = 21<br> Thread-23::DEBUG::2013-12-04 09:02:41,896::misc::819::SamplingMethod::(__call__) Returning last result<br> Thread-23::DEBUG::2013-12-04 09:02:44,915::multipath::112::Storage.Misc.excCmd::(rescan) '/usr/bin/sudo -n /sbin/multipath -r' (cwd None)<br> Thread-23::DEBUG::2013-12-04 09:02:44,966::multipath::112::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0<br> Thread-23::DEBUG::2013-12-04 09:02:44,967::lvm::510::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex<br> Thread-23::DEBUG::2013-12-04 09:02:44,967::lvm::512::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex<br> Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::521::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex<br> Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::523::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex<br> Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::541::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex<br> Thread-23::DEBUG::2013-12-04 09:02:44,968::lvm::543::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex<br> Thread-23::DEBUG::2013-12-04 09:02:44,968::misc::819::SamplingMethod::(__call__) Returning last result<br> Thread-23::<a class="moz-txt-link-freetext" href="INFO::2013-12-04">INFO::2013-12-04</a> 09:02:44,969::logUtils::47::dispatcher::(wrapper) Run and protect: disconnectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '478b35df-da9a-4cb0-847f-2f61f17a02f7'}, {'status': 0, 'id': '717e3505-3290-421b-bb00-25e046f63362'}]}<br> Thread-23::DEBUG::2013-12-04 09:02:44,969::task::1168::TaskManager.Task::(prepare) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::finished: {'statuslist': [{'status': 0, 'id': '478b35df-da9a-4cb0-847f-2f61f17a02f7'}, {'status': 0, 'id': '717e3505-3290-421b-bb00-25e046f63362'}]}<br> Thread-23::DEBUG::2013-12-04 09:02:44,969::task::579::TaskManager.Task::(_updateState) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::moving from state preparing -> state finished<br> Thread-23::DEBUG::2013-12-04 09:02:44,969::resourceManager::939::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}<br> Thread-23::DEBUG::2013-12-04 09:02:44,969::resourceManager::976::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}<br> Thread-23::DEBUG::2013-12-04 09:02:44,970::task::974::TaskManager.Task::(_decref) Task=`155824b5-48c3-4eb2-90fb-d90e84f2900f`::ref 0 aborting False<br> <br> Regards,<br> <br> <br> </body> </html> --------------060401020402070903050401--