5:01 a.m.
Hi,
This is my first post to the list. I am happy to say that we have been
using Ovirt for 6 months with a few bumps, but it's mostly been ok.
Until tonight that is...
I had to do a maintenance that required rebooting both of our Hypervisor
nodes. Both of them run Fedora Core 18 and have been happy for months.
After rebooting them tonight, they will not attach to the storage. If
it matters, the storage is a server running LIO with a Fibre Channel target.
Vdsm log:
Thread-22::DEBUG::2013-09-19
21:57:09,392::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd
iflag=direct if=/dev/b358e46b-635b-4c0e-8e73-0a494602e21d/metadata
bs=4096 count=1' (cwd None)
Thread-22::DEBUG::2013-09-19
21:57:09,400::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> =
'1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied,
0.000547161 s, 7.5 MB/s\n'; <rc> = 0
Thread-23::DEBUG::2013-09-19
21:57:16,587::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' got the operation mutex
Thread-23::DEBUG::2013-09-19
21:57:16,587::misc::84::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo
-n /sbin/lvm vgs --config " devices { preferred_names =
[\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
disable_after_error_count=3 filter = [
\\"a%360014055193f840cb3743f9befef7aa3%\\", \\"r%.*%\\" ] } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup {
retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix
--separator | -o
uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free
6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50' (cwd None)
Thread-23::DEBUG::2013-09-19
21:57:16,643::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> =
' Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found\n';
<rc> = 5
Thread-23::WARNING::2013-09-19
21:57:16,649::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
[' Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found']
Thread-23::DEBUG::2013-09-19
21:57:16,649::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm
reload operation' released the operation mutex
Thread-23::ERROR::2013-09-19
21:57:16,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain)
Error while collecting domain 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50
monitoring information
Traceback (most recent call last):
File "/usr/share/vdsm/storage/domainMonitor.py", line 182, in
_monitorDomain
self.domain = sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 97, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
domain = self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist:
(u'6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50',)
vgs output (Note that I don't know what the device
(Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z is) :
[root@node01 vdsm]# vgs
Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
VG #PV #LV #SN Attr VSize VFree
b358e46b-635b-4c0e-8e73-0a494602e21d 1 39 0 wz--n- 8.19t 5.88t
build 2 2 0 wz-pn- 299.75g 16.00m
fedora 1 3 0 wz--n- 557.88g 0
lvs output:
[root@node01 vdsm]# lvs
Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
LV VG
Attr LSize Pool Origin Data% Move Log Copy% Convert
0b8cca47-313f-48da-84f2-154810790d5a
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
0f6f7572-8797-4d84-831b-87dbc4e1aa48
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
19a1473f-c375-411f-9a02-c6054b9a28d2
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 50.00g
221144dc-51dc-46ae-9399-c0b8e030f38a
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
2386932f-5f68-46e1-99a4-e96c944ac21b
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
3e027010-931b-43d6-9c9f-eeeabbdcd47a
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 2.00g
4257ccc2-94d5-4d71-b21a-c188acbf7ca1
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 200.00g
4979b2a4-04aa-46a1-be0d-f10be0a1f587
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
4e1b8a1a-1704-422b-9d79-60f15e165cb7
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
70bce792-410f-479f-8e04-a2a4093d3dfb
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
791f6bda-c7eb-4d90-84c1-d7e33e73de62
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
818ad6bc-8da2-4099-b38a-8c5b52f69e32
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 120.00g
861c9c44-fdeb-43cd-8e5c-32c00ce3cd3d
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
86b69521-14db-43d1-801f-9d21f0a0e00f
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
8a578e50-683d-47c3-af41-c7e508d493e8
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
90463a7a-ecd4-4838-bc91-adccf99d9997
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
9170a33d-3bdf-4c15-8e6b-451622c8093b
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 80.00g
964b9e32-c1ee-4152-a05b-0c43815f5ea6
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
975c0a26-f699-4351-bd27-dd7621eac6bd
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
9ec24f39-8b32-4247-bfb4-4b7f2cf86d9d
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
a4f303bf-6e89-43c3-a801-046920cb24d6
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
a7115874-0f1c-4f43-ab3a-a6026ad99013
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
b1fb5597-a3bb-4e4b-b73f-d1752cc576cb
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
bc28d7c6-a14b-4398-8166-ac2f25b17312
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
bc72da88-f5fd-4f53-9c2c-af2fcd14d117
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
c2c1ba71-c938-4d71-876a-1bba89a5d8a9
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
c54eb342-b79b-45fe-8117-aab7137f5f9d
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
c892f8b5-fadc-4774-a355-32655512a462
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
c9f636ce-efed-495d-9a29-cfaac1f289d3
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
cb657c62-44c8-43dd-8ea2-cbf5927cff72
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
cdba05ac-5f68-4213-827b-3d3518c67251
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
d98f3bfc-55b0-44f7-8a39-cb0920762cba
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 100.00g
e0708bc4-19df-4d48-a0e7-682070634dea
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 40.00g
ids
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-ao--- 128.00m
inbox
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 128.00m
leases
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 2.00g
master
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 1.00g
metadata
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 512.00m
outbox
b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a---- 128.00m
root build
-wi-----p 298.74g
swap_1 build
-wi-----p 1020.00m
home fedora
-wi-ao--- 503.88g
root fedora
-wi-ao--- 50.00g
swap fedora
-wi-ao--- 4.00g
The strange thing is that I can see all of the LVM volumes for the VMs.
Both servers see the storage just fine. This error has me baffled and
it's a total show stopper since the cluster will not come back up.
If anyone can help, that would be very appreciated.
Thanks,
Dan