[Users] Unable to attach to storage domain (Ovirt 3.2)
Dan Ferris
dferris at prometheusresearch.com
Sun Sep 22 13:48:29 UTC 2013
We actually got it working. Both of us were tired from working late, so
we didn't find that the missing storage domain was actually one of the
NFS exports for awhile. After removing our NFS ISO domain and NFS
export domain everything came up.
Dan
On 9/22/13 6:08 AM, Ayal Baron wrote:
> If I understand correctly you have a storage domain which is built of multiple (at least 2) LUNs.
> One of these LUNs seems to be missing (Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z is an LVM PV UUID).
> It looks like you are either not fully connected to the storage server (missing a connection) or the LUN mapping in LIO has been changed or that the chap password has changed or something.
>
> LVM is able to report the LVs since the PV which contains the metadata is still accessible (which is also why you see the VG and why LVM knows that the Wy3Ymi... device is missing).
>
> Can you compress and attach *all* of the vdsm.log* files?
>
> ----- Original Message -----
>> Hi Dan, it looks like one of the domains is missing:
>>
>> 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50
>>
>> Is there any target missing? (disconnected or somehow faulty or
>> unreachable)
>>
>> --
>> Federico
>>
>> ----- Original Message -----
>>> From: "Dan Ferris" <dferris at prometheusresearch.com>
>>> To: users at ovirt.org
>>> Sent: Friday, September 20, 2013 4:01:06 AM
>>> Subject: [Users] Unable to attach to storage domain (Ovirt 3.2)
>>>
>>> Hi,
>>>
>>> This is my first post to the list. I am happy to say that we have been
>>> using Ovirt for 6 months with a few bumps, but it's mostly been ok.
>>>
>>> Until tonight that is...
>>>
>>> I had to do a maintenance that required rebooting both of our Hypervisor
>>> nodes. Both of them run Fedora Core 18 and have been happy for months.
>>> After rebooting them tonight, they will not attach to the storage. If
>>> it matters, the storage is a server running LIO with a Fibre Channel
>>> target.
>>>
>>> Vdsm log:
>>>
>>> Thread-22::DEBUG::2013-09-19
>>> 21:57:09,392::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd
>>> iflag=direct if=/dev/b358e46b-635b-4c0e-8e73-0a494602e21d/metadata
>>> bs=4096 count=1' (cwd None)
>>> Thread-22::DEBUG::2013-09-19
>>> 21:57:09,400::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> =
>>> '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied,
>>> 0.000547161 s, 7.5 MB/s\n'; <rc> = 0
>>> Thread-23::DEBUG::2013-09-19
>>> 21:57:16,587::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm
>>> reload operation' got the operation mutex
>>> Thread-23::DEBUG::2013-09-19
>>> 21:57:16,587::misc::84::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo
>>> -n /sbin/lvm vgs --config " devices { preferred_names =
>>> [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
>>> disable_after_error_count=3 filter = [
>>> \\"a%360014055193f840cb3743f9befef7aa3%\\", \\"r%.*%\\" ] } global {
>>> locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup {
>>> retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix
>>> --separator | -o
>>> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free
>>> 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50' (cwd None)
>>> Thread-23::DEBUG::2013-09-19
>>> 21:57:16,643::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> =
>>> ' Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found\n';
>>> <rc> = 5
>>> Thread-23::WARNING::2013-09-19
>>> 21:57:16,649::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
>>> [' Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found']
>>> Thread-23::DEBUG::2013-09-19
>>> 21:57:16,649::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm
>>> reload operation' released the operation mutex
>>> Thread-23::ERROR::2013-09-19
>>> 21:57:16,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain)
>>> Error while collecting domain 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50
>>> monitoring information
>>> Traceback (most recent call last):
>>> File "/usr/share/vdsm/storage/domainMonitor.py", line 182, in
>>> _monitorDomain
>>> self.domain = sdCache.produce(self.sdUUID)
>>> File "/usr/share/vdsm/storage/sdc.py", line 97, in produce
>>> domain.getRealDomain()
>>> File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>>> return self._cache._realProduce(self._sdUUID)
>>> File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>>> domain = self._findDomain(sdUUID)
>>> File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>>> raise se.StorageDomainDoesNotExist(sdUUID)
>>> StorageDomainDoesNotExist: Storage domain does not exist:
>>> (u'6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50',)
>>>
>>> vgs output (Note that I don't know what the device
>>> (Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z is) :
>>>
>>> [root at node01 vdsm]# vgs
>>> Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
>>> VG #PV #LV #SN Attr VSize VFree
>>> b358e46b-635b-4c0e-8e73-0a494602e21d 1 39 0 wz--n- 8.19t 5.88t
>>> build 2 2 0 wz-pn- 299.75g 16.00m
>>> fedora 1 3 0 wz--n- 557.88g 0
>>>
>>> lvs output:
>>>
>>> [root at node01 vdsm]# lvs
>>> Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
>>> LV VG
>>> Attr LSize Pool Origin Data% Move Log Copy% Convert
>>> 0b8cca47-313f-48da-84f2-154810790d5a
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> 0f6f7572-8797-4d84-831b-87dbc4e1aa48
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> 19a1473f-c375-411f-9a02-c6054b9a28d2
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 50.00g
>>> 221144dc-51dc-46ae-9399-c0b8e030f38a
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> 2386932f-5f68-46e1-99a4-e96c944ac21b
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> 3e027010-931b-43d6-9c9f-eeeabbdcd47a
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 2.00g
>>> 4257ccc2-94d5-4d71-b21a-c188acbf7ca1
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 200.00g
>>> 4979b2a4-04aa-46a1-be0d-f10be0a1f587
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> 4e1b8a1a-1704-422b-9d79-60f15e165cb7
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> 70bce792-410f-479f-8e04-a2a4093d3dfb
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> 791f6bda-c7eb-4d90-84c1-d7e33e73de62
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> 818ad6bc-8da2-4099-b38a-8c5b52f69e32
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 120.00g
>>> 861c9c44-fdeb-43cd-8e5c-32c00ce3cd3d
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> 86b69521-14db-43d1-801f-9d21f0a0e00f
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> 8a578e50-683d-47c3-af41-c7e508d493e8
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> 90463a7a-ecd4-4838-bc91-adccf99d9997
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> 9170a33d-3bdf-4c15-8e6b-451622c8093b
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 80.00g
>>> 964b9e32-c1ee-4152-a05b-0c43815f5ea6
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> 975c0a26-f699-4351-bd27-dd7621eac6bd
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> 9ec24f39-8b32-4247-bfb4-4b7f2cf86d9d
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> a4f303bf-6e89-43c3-a801-046920cb24d6
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> a7115874-0f1c-4f43-ab3a-a6026ad99013
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> b1fb5597-a3bb-4e4b-b73f-d1752cc576cb
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> bc28d7c6-a14b-4398-8166-ac2f25b17312
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> bc72da88-f5fd-4f53-9c2c-af2fcd14d117
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> c2c1ba71-c938-4d71-876a-1bba89a5d8a9
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> c54eb342-b79b-45fe-8117-aab7137f5f9d
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> c892f8b5-fadc-4774-a355-32655512a462
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> c9f636ce-efed-495d-9a29-cfaac1f289d3
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> cb657c62-44c8-43dd-8ea2-cbf5927cff72
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> cdba05ac-5f68-4213-827b-3d3518c67251
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> d98f3bfc-55b0-44f7-8a39-cb0920762cba
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
>>> e0708bc4-19df-4d48-a0e7-682070634dea
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
>>> ids
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-ao--- 128.00m
>>> inbox
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 128.00m
>>> leases
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 2.00g
>>> master
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 1.00g
>>> metadata
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 512.00m
>>> outbox
>>> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 128.00m
>>> root build
>>> -wi-----p 298.74g
>>> swap_1 build
>>> -wi-----p 1020.00m
>>> home fedora
>>> -wi-ao--- 503.88g
>>> root fedora
>>> -wi-ao--- 50.00g
>>> swap fedora
>>> -wi-ao--- 4.00g
>>>
>>>
>>> The strange thing is that I can see all of the LVM volumes for the VMs.
>>> Both servers see the storage just fine. This error has me baffled and
>>> it's a total show stopper since the cluster will not come back up.
>>>
>>> If anyone can help, that would be very appreciated.
>>>
>>> Thanks,
>>>
>>> Dan
>>> _______________________________________________
>>> Users mailing list
>>> Users at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
More information about the Users
mailing list