[ovirt-users] Recovering iSCSI domain (Was: Changing iSCSI LUN host IP and changing master domain)

Wed Oct 22 03:34:08 EDT 2014

Trey,

I see you got answers offline and on similar topics.
Is this still relevant?

Regards,
Vered

----- Original Message -----
> From: "Trey Dockendorf" <treydock at gmail.com>
> To: "Sandra Taylor" <jtt77777 at gmail.com>, "users" <users at ovirt.org>
> Sent: Tuesday, October 21, 2014 11:59:23 PM
> Subject: [ovirt-users] Recovering iSCSI domain (Was: Changing iSCSI LUN host IP and changing master domain)
> 
> Somehow my NFS domain got to be master again. I went into the database and
> updated the connections for NFS and I noticed that once I updated the IP for
> the ISCSI in the "storage_server_connections" table that the interface kept
> moving "(master)" between the iSCSI and NFS domain...very odd.
> 
> I did these commands and now NFS is up.
> 
> update storage_server_connections set connection='10.0.0.10:/tank/ovirt/data'
> where id='a89fa66b-8737-4bb8-a089-d9067f61b58a';
> update storage_server_connections set
> connection='10.0.0.10:/tank/ovirt/import_export' where
> id='521a8477-9e88-4f2d-96e2-d3667ec407df';
> update storage_server_connections set
> connection='192.168.202.245:/tank/ovirt/iso' where
> id='fb55cfea-c7ef-49f2-b77f-16ddd2de0f7a';
> update storage_server_connections set connection='10.0.0.10' where
> id='d6da7fbf-5056-44a7-9fc8-e76a1ff9f525';
> 
> Once I activated the NFS master domain all my other domains went to active,
> including iSCSI.
> 
> My concern now is whether the iSCSI domain is usable. The API path at
> "/api/storagedomains/4eeb8415-c912-44bf-b482-2673849705c9/storageconnections"
> shows
> 
> <storage_connections/>
> 
> If I go to edit the iSCSI domain and check the LUN the warning I get is this:
> 
> This operation might be unrecoverable and destructive!
> The following LUNs are already in use:
> - 1IET_00010001 (Used by VG: 3nxXNr-bIHu-9YS5-Kfzc-A2Na-sMhb-jihwdt)
> 
> That alone makes me very hesitant to approve the operation. I could use some
> wisdom if this is safe or not.
> 
> Thanks,
> - Trey
> 
> On Tue, Oct 21, 2014 at 3:17 PM, Trey Dockendorf < treydock at gmail.com >
> wrote:
> 
> 
> 
> John,
> 
> Thanks for reply. The Discover function in GUI works...it's once I try and
> login (Click the array next to target) that things just hang indefinitely.
> 
> # iscsiadm -m session
> tcp: [2] 10.0.0.10:3260 ,1
> iqn.2014-04.edu.tamu.brazos.vmstore1:ovirt-data_iscsi
> 
> # iscsiadm -m node
> 10.0.0.10:3260 ,1 iqn.2014-04.edu.tamu.brazos.vmstore1:ovirt-data_iscsi
> 
> # multipath -ll
> 1IET_00010001 dm-3 IET,VIRTUAL-DISK
> size=500G features='0' hwhandler='0' wp=rw
> `-+- policy='round-robin 0' prio=1 status=active
> `- 8:0:0:1 sdd 8:48 active ready running
> 1ATA_WDC_WD5003ABYZ-011FA0_WD-WMAYP0DNSAEZ dm-2 ATA,WDC WD5003ABYZ-0
> size=466G features='0' hwhandler='0' wp=rw
> `-+- policy='round-robin 0' prio=1 status=active
> `- 3:0:0:0 sdc 8:32 active ready running
> 
> The first entry, 1IET_00010001 is the iSCSI LUN.
> 
> The log when I click the array in the interface for the target is this:
> 
> Thread-14::DEBUG::2014-10-21 15:12:49,900::BindingXMLRPC::251::vds::(wrapper)
> client [192.168.202.99] flowID [7177dafe]
> Thread-14::DEBUG::2014-10-21
> 15:12:49,901::task::595::TaskManager.Task::(_updateState)
> Task=`01d8d01e-8bfd-4764-890f-2026fdeb78d9`::moving from state init -> state
> preparing
> Thread-14::INFO::2014-10-21 15:12:49,901::logUtils::44::dispatcher::(wrapper)
> Run and protect: connectStorageServer(domType=3,
> spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection':
> '10.0.0.10', 'iqn': 'iqn.2014-04.edu.tamu.brazos.)
> Thread-14::DEBUG::2014-10-21
> 15:12:49,902::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n
> /sbin/iscsiadm -m node -T
> iqn.2014-04.edu.tamu.brazos.vmstore1:ovirt-data_iscsi -I default -p
> 10.0.0.10:3260 ,1 --op=new' (cwd None)
> Thread-14::DEBUG::2014-10-21
> 15:12:56,684::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> =
> ''; <rc> = 0
> Thread-14::DEBUG::2014-10-21
> 15:12:56,685::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n
> /sbin/iscsiadm -m node -T
> iqn.2014-04.edu.tamu.brazos.vmstore1:ovirt-data_iscsi -I default -p
> 10.0.0.10:3260 ,1 -l' (cwd None)
> Thread-14::DEBUG::2014-10-21
> 15:12:56,711::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> =
> ''; <rc> = 0
> Thread-14::DEBUG::2014-10-21
> 15:12:56,711::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) '/usr/bin/sudo -n
> /sbin/iscsiadm -m node -T
> iqn.2014-04.edu.tamu.brazos.vmstore1:ovirt-data_iscsi -I default -p
> 10.0.0.10:3260 ,1 -n node.startup -v manual --op)
> Thread-14::DEBUG::2014-10-21
> 15:12:56,767::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) SUCCESS: <err> =
> ''; <rc> = 0
> Thread-14::DEBUG::2014-10-21
> 15:12:56,767::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload
> operation' got the operation mutex
> Thread-14::DEBUG::2014-10-21
> 15:12:56,768::lvm::296::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n
> /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"]
> ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3)
> Thread-14::DEBUG::2014-10-21
> 15:12:56,968::lvm::296::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ' No
> volume groups found\n'; <rc> = 0
> Thread-14::DEBUG::2014-10-21
> 15:12:56,969::lvm::415::OperationMutex::(_reloadvgs) Operation 'lvm reload
> operation' released the operation mutex
> Thread-14::DEBUG::2014-10-21
> 15:12:56,974::hsm::2352::Storage.HSM::(__prefetchDomains) Found SD uuids: ()
> Thread-14::DEBUG::2014-10-21
> 15:12:56,974::hsm::2408::Storage.HSM::(connectStorageServer) knownSDs: {}
> Thread-14::INFO::2014-10-21 15:12:56,974::logUtils::47::dispatcher::(wrapper)
> Run and protect: connectStorageServer, Return response: {'statuslist':
> [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]}
> Thread-14::DEBUG::2014-10-21
> 15:12:56,974::task::1185::TaskManager.Task::(prepare)
> Task=`01d8d01e-8bfd-4764-890f-2026fdeb78d9`::finished: {'statuslist':
> [{'status': 0, 'id': '00000000-0000-0000-0000-000000000000'}]}
> Thread-14::DEBUG::2014-10-21
> 15:12:56,975::task::595::TaskManager.Task::(_updateState)
> Task=`01d8d01e-8bfd-4764-890f-2026fdeb78d9`::moving from state preparing ->
> state finished
> Thread-14::DEBUG::2014-10-21
> 15:12:56,975::resourceManager::940::ResourceManager.Owner::(releaseAll)
> Owner.releaseAll requests {} resources {}
> Thread-14::DEBUG::2014-10-21
> 15:12:56,975::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> Thread-14::DEBUG::2014-10-21
> 15:12:56,975::task::990::TaskManager.Task::(_decref)
> Task=`01d8d01e-8bfd-4764-890f-2026fdeb78d9`::ref 0 aborting False
> Thread-13::DEBUG::2014-10-21
> 15:13:18,281::task::595::TaskManager.Task::(_updateState)
> Task=`8674b6b0-5e4c-4f0c-8b6b-c5fa5fef6126`::moving from state init -> state
> preparing
> Thread-13::INFO::2014-10-21 15:13:18,281::logUtils::44::dispatcher::(wrapper)
> Run and protect: repoStats(options=None)
> Thread-13::INFO::2014-10-21 15:13:18,282::logUtils::47::dispatcher::(wrapper)
> Run and protect: repoStats, Return response: {}
> Thread-13::DEBUG::2014-10-21
> 15:13:18,282::task::1185::TaskManager.Task::(prepare)
> Task=`8674b6b0-5e4c-4f0c-8b6b-c5fa5fef6126`::finished: {}
> Thread-13::DEBUG::2014-10-21
> 15:13:18,282::task::595::TaskManager.Task::(_updateState)
> Task=`8674b6b0-5e4c-4f0c-8b6b-c5fa5fef6126`::moving from state preparing ->
> state finished
> Thread-13::DEBUG::2014-10-21
> 15:13:18,282::resourceManager::940::ResourceManager.Owner::(releaseAll)
> Owner.releaseAll requests {} resources {}
> Thread-13::DEBUG::2014-10-21
> 15:13:18,282::resourceManager::977::ResourceManager.Owner::(cancelAll)
> Owner.cancelAll requests {}
> Thread-13::DEBUG::2014-10-21
> 15:13:18,283::task::990::TaskManager.Task::(_decref)
> Task=`8674b6b0-5e4c-4f0c-8b6b-c5fa5fef6126`::ref 0 aborting False
> 
> The lines prefixed with "Thread-13" just repeat over and over only changing
> the Task value.
> 
> Unsure what could be done to restore things. The iscsi connection is good and
> I'm able to see the logical volumes:
> 
> # lvscan
> ACTIVE '/dev/4eeb8415-c912-44bf-b482-2673849705c9/metadata' [512.00 MiB]
> inherit
> ACTIVE '/dev/4eeb8415-c912-44bf-b482-2673849705c9/leases' [2.00 GiB] inherit
> ACTIVE '/dev/4eeb8415-c912-44bf-b482-2673849705c9/ids' [128.00 MiB] inherit
> ACTIVE '/dev/4eeb8415-c912-44bf-b482-2673849705c9/inbox' [128.00 MiB] inherit
> ACTIVE '/dev/4eeb8415-c912-44bf-b482-2673849705c9/outbox' [128.00 MiB]
> inherit
> ACTIVE '/dev/4eeb8415-c912-44bf-b482-2673849705c9/master' [1.00 GiB] inherit
> inactive
> '/dev/4eeb8415-c912-44bf-b482-2673849705c9/aced9726-5a28-4d52-96f5-89553ba770af'
> [100.00 GiB] inherit
> inactive
> '/dev/4eeb8415-c912-44bf-b482-2673849705c9/87bf28aa-be25-4a93-9b23-f70bfd8accc0'
> [1.00 GiB] inherit
> inactive
> '/dev/4eeb8415-c912-44bf-b482-2673849705c9/27256587-bf87-4519-89e7-260e13697de3'
> [20.00 GiB] inherit
> inactive
> '/dev/4eeb8415-c912-44bf-b482-2673849705c9/ac2cb7f9-1df9-43dc-9fda-8a9958ef970f'
> [20.00 GiB] inherit
> inactive
> '/dev/4eeb8415-c912-44bf-b482-2673849705c9/d8c41f05-006a-492b-8e5f-101c4e113b28'
> [100.00 GiB] inherit
> inactive
> '/dev/4eeb8415-c912-44bf-b482-2673849705c9/83f17e9b-183e-4bad-ada5-bcef1c5c8e6a'
> [20.00 GiB] inherit
> inactive
> '/dev/4eeb8415-c912-44bf-b482-2673849705c9/cf79052e-b4ef-4bda-96dc-c53b7c2acfb5'
> [20.00 GiB] inherit
> ACTIVE '/dev/vg_ovirtnode02/lv_swap' [46.59 GiB] inherit
> ACTIVE '/dev/vg_ovirtnode02/lv_root' [418.53 GiB] inherit
> 
> Thanks,
> - Trey
> 
> 
> 
> On Tue, Oct 21, 2014 at 2:49 PM, Sandra Taylor < jtt77777 at gmail.com > wrote:
> 
> 
> Hi Trey,
> Sorry for your trouble.
> Don't know if I can help but I run iscsi here as my primary domain so
> I've had some experience with it.
> I don't know the answer to the master domain question.
> 
> Does iscsi show connected using iscsiadm -m session and -m node ?
> in the vdsm log there should be the iscsiadm commands that were
> executed to connect.
> Does multipath -ll show anything?
> 
> -John
> 
> On Tue, Oct 21, 2014 at 3:18 PM, Trey Dockendorf < treydock at gmail.com >
> wrote:
> > I was able to get iSCSI over TCP working...but now the task of adding the
> > LUN to the GUI has been stuck at the "spinning" icon for about 20 minutes.
> > 
> > I see these entries in vdsm.log over and over with the Task value changing:
> > 
> > Thread-14::DEBUG::2014-10-21
> > 14:16:50,086::task::595::TaskManager.Task::(_updateState)
> > Task=`ebcd8e0a-54b1-43d2-92a2-ed9fd62d00fa`::moving from state init ->
> > state
> > preparing
> > Thread-14::INFO::2014-10-21
> > 14:16:50,086::logUtils::44::dispatcher::(wrapper) Run and protect:
> > repoStats(options=None)
> > Thread-14::INFO::2014-10-21
> > 14:16:50,086::logUtils::47::dispatcher::(wrapper) Run and protect:
> > repoStats, Return response: {}
> > Thread-14::DEBUG::2014-10-21
> > 14:16:50,087::task::1185::TaskManager.Task::(prepare)
> > Task=`ebcd8e0a-54b1-43d2-92a2-ed9fd62d00fa`::finished: {}
> > Thread-14::DEBUG::2014-10-21
> > 14:16:50,087::task::595::TaskManager.Task::(_updateState)
> > Task=`ebcd8e0a-54b1-43d2-92a2-ed9fd62d00fa`::moving from state preparing ->
> > state finished
> > Thread-14::DEBUG::2014-10-21
> > 14:16:50,087::resourceManager::940::ResourceManager.Owner::(releaseAll)
> > Owner.releaseAll requests {} resources {}
> > Thread-14::DEBUG::2014-10-21
> > 14:16:50,087::resourceManager::977::ResourceManager.Owner::(cancelAll)
> > Owner.cancelAll requests {}
> > Thread-14::DEBUG::2014-10-21
> > 14:16:50,087::task::990::TaskManager.Task::(_decref)
> > Task=`ebcd8e0a-54b1-43d2-92a2-ed9fd62d00fa`::ref 0 aborting False
> > 
> > What is there I can do to get my storage back online? Right now my iSCSI is
> > master (something I did not want) which is odd considering the NFS data
> > domain was added as master when I setup oVirt. Nothing will come back until
> > I get the master domain online and unsure what to do now.
> > 
> > Thanks,
> > - Trey
> > 
> > On Tue, Oct 21, 2014 at 12:58 PM, Trey Dockendorf < treydock at gmail.com >
> > wrote:
> >> 
> >> I had a catastrophic failure of the IB switch that was used by all my
> >> storage domains. I had one data domain that was NFS and one that was
> >> iSCSI.
> >> I managed to get the iSCSI LUN detached using the docs [1] but now I
> >> noticed
> >> that somehow my master domain went from the NFS domain to the iSCSI domain
> >> and I'm unable to switch them back.
> >> 
> >> How does one change the master? Right now I am having issues getting
> >> iSCSI over TCP to work, so am sort of stuck with 30 VMs down and an entire
> >> cluster inaccessible.
> >> 
> >> Thanks,
> >> - Trey
> >> 
> >> [1] http://www.ovirt.org/Features/Manage_Storage_Connections
> > 
> > 
> > 
> > _______________________________________________
> > Users mailing list
> > Users at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
> > 
> 
> 
> 
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 