[Users] Unable to attach to storage domain (Ovirt 3.2)

Itamar Heim iheim at redhat.com
Wed Oct 2 19:27:20 UTC 2013


On 09/20/2013 05:01 AM, Dan Ferris wrote:
> Hi,
>
> This is my first post to the list.  I am happy to say that we have been
> using Ovirt for 6 months with a few bumps, but it's mostly been ok.
>
> Until tonight that is...
>
> I had to do a maintenance that required rebooting both of our Hypervisor
> nodes.  Both of them run Fedora Core 18 and have been happy for months.
>   After rebooting them tonight, they will not attach to the storage.  If
> it matters, the storage is a server running LIO with a Fibre Channel
> target.
>
> Vdsm log:
>
> Thread-22::DEBUG::2013-09-19
> 21:57:09,392::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/dd
> iflag=direct if=/dev/b358e46b-635b-4c0e-8e73-0a494602e21d/metadata
> bs=4096 count=1' (cwd None)
> Thread-22::DEBUG::2013-09-19
> 21:57:09,400::misc::84::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> =
> '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied,
> 0.000547161 s, 7.5 MB/s\n'; <rc> = 0
> Thread-23::DEBUG::2013-09-19
> 21:57:16,587::lvm::368::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' got the operation mutex
> Thread-23::DEBUG::2013-09-19
> 21:57:16,587::misc::84::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo
> -n /sbin/lvm vgs --config " devices { preferred_names =
> [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0
> disable_after_error_count=3 filter = [
> \\"a%360014055193f840cb3743f9befef7aa3%\\", \\"r%.*%\\" ] }  global {
> locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {
> retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix
> --separator | -o
> uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free
> 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50' (cwd None)
> Thread-23::DEBUG::2013-09-19
> 21:57:16,643::misc::84::Storage.Misc.excCmd::(<lambda>) FAILED: <err> =
> '  Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found\n';
> <rc> = 5
> Thread-23::WARNING::2013-09-19
> 21:57:16,649::lvm::373::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 []
> ['  Volume group "6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50" not found']
> Thread-23::DEBUG::2013-09-19
> 21:57:16,649::lvm::397::OperationMutex::(_reloadvgs) Operation 'lvm
> reload operation' released the operation mutex
> Thread-23::ERROR::2013-09-19
> 21:57:16,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain)
> Error while collecting domain 6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50
> monitoring information
> Traceback (most recent call last):
>    File "/usr/share/vdsm/storage/domainMonitor.py", line 182, in
> _monitorDomain
>      self.domain = sdCache.produce(self.sdUUID)
>    File "/usr/share/vdsm/storage/sdc.py", line 97, in produce
>      domain.getRealDomain()
>    File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>      return self._cache._realProduce(self._sdUUID)
>    File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>      domain = self._findDomain(sdUUID)
>    File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>      raise se.StorageDomainDoesNotExist(sdUUID)
> StorageDomainDoesNotExist: Storage domain does not exist:
> (u'6cf7e7e9-3ae5-4645-a29c-fb17ecb38a50',)
>
> vgs output (Note that I don't know what the device
> (Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z is) :
>
> [root at node01 vdsm]# vgs
>    Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
>    VG                                   #PV #LV #SN Attr   VSize   VFree
>    b358e46b-635b-4c0e-8e73-0a494602e21d   1  39   0 wz--n-   8.19t  5.88t
>    build                                  2   2   0 wz-pn- 299.75g 16.00m
>    fedora                                 1   3   0 wz--n- 557.88g     0
>
> lvs output:
>
> [root at node01 vdsm]# lvs
>    Couldn't find device with uuid Wy3Ymi-J7bJ-hVxg-sg3L-F5Gv-MQmz-Utwv7z.
>    LV                                   VG      Attr      LSize    Pool
> Origin Data%  Move Log Copy%  Convert
>    0b8cca47-313f-48da-84f2-154810790d5a
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    0f6f7572-8797-4d84-831b-87dbc4e1aa48
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    19a1473f-c375-411f-9a02-c6054b9a28d2
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   50.00g
>    221144dc-51dc-46ae-9399-c0b8e030f38a
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    2386932f-5f68-46e1-99a4-e96c944ac21b
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    3e027010-931b-43d6-9c9f-eeeabbdcd47a
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----    2.00g
>    4257ccc2-94d5-4d71-b21a-c188acbf7ca1
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  200.00g
>    4979b2a4-04aa-46a1-be0d-f10be0a1f587
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    4e1b8a1a-1704-422b-9d79-60f15e165cb7
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    70bce792-410f-479f-8e04-a2a4093d3dfb
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    791f6bda-c7eb-4d90-84c1-d7e33e73de62
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    818ad6bc-8da2-4099-b38a-8c5b52f69e32
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  120.00g
>    861c9c44-fdeb-43cd-8e5c-32c00ce3cd3d
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    86b69521-14db-43d1-801f-9d21f0a0e00f
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    8a578e50-683d-47c3-af41-c7e508d493e8
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    90463a7a-ecd4-4838-bc91-adccf99d9997
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    9170a33d-3bdf-4c15-8e6b-451622c8093b
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   80.00g
>    964b9e32-c1ee-4152-a05b-0c43815f5ea6
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    975c0a26-f699-4351-bd27-dd7621eac6bd
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    9ec24f39-8b32-4247-bfb4-4b7f2cf86d9d
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    a4f303bf-6e89-43c3-a801-046920cb24d6
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    a7115874-0f1c-4f43-ab3a-a6026ad99013
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    b1fb5597-a3bb-4e4b-b73f-d1752cc576cb
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    bc28d7c6-a14b-4398-8166-ac2f25b17312
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    bc72da88-f5fd-4f53-9c2c-af2fcd14d117
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    c2c1ba71-c938-4d71-876a-1bba89a5d8a9
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    c54eb342-b79b-45fe-8117-aab7137f5f9d
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    c892f8b5-fadc-4774-a355-32655512a462
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    c9f636ce-efed-495d-9a29-cfaac1f289d3
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    cb657c62-44c8-43dd-8ea2-cbf5927cff72
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    cdba05ac-5f68-4213-827b-3d3518c67251
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    d98f3bfc-55b0-44f7-8a39-cb0920762cba
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  100.00g
>    e0708bc4-19df-4d48-a0e7-682070634dea
> b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----   40.00g
>    ids b358e46b-635b-4c0e-8e73-0a494602e21d -wi-ao---  128.00m
>    inbox b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  128.00m
>    leases b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----    2.00g
>    master b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----    1.00g
>    metadata b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  512.00m
>    outbox b358e46b-635b-4c0e-8e73-0a494602e21d -wi-a----  128.00m
>    root                                 build      -wi-----p  298.74g
>    swap_1                               build      -wi-----p 1020.00m
>    home                                 fedora      -wi-ao---  503.88g
>    root                                 fedora      -wi-ao---   50.00g
>    swap                                 fedora      -wi-ao---    4.00g
>
>
> The strange thing is that I can see all of the LVM volumes for the VMs.
>   Both servers see the storage just fine.  This error has me baffled and
> it's a total show stopper since the cluster will not come back up.
>
> If anyone can help, that would be very appreciated.
>
> Thanks,
>
> Dan
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users

was this resolved?



More information about the Users mailing list