[Users] How to rescue storage domain structure
Chris Smith
whitehat237 at gmail.com
Thu Apr 25 01:35:09 UTC 2013
Does this work with volume groups? I have several virtual disks
presented to the VM which are part of a volume group.
[root at voyager media]# fsarchiver probe
[======DISK======] [=============NAME==============] [====SIZE====] [MAJ] [MIN]
[vda ] [ ] [ 40.00 GB] [252] [ 0]
[vdb ] [ ] [ 100.00 GB] [252] [ 16]
[vdc ] [ ] [ 20.00 GB] [252] [ 32]
[vdd ] [ ] [ 40.00 GB] [252] [ 48]
[=====DEVICE=====] [==FILESYS==] [======LABEL======] [====SIZE====] [MAJ] [MIN]
[vda1 ] [ext4 ] [<unknown> ] [ 500.00 MB] [252] [ 1]
[vda2 ] [LVM2_member] [<unknown> ] [ 39.51 GB] [252] [ 2]
[vdb1 ] [LVM2_member] [<unknown> ] [ 100.00 GB] [252] [ 17]
[vdc1 ] [LVM2_member] [<unknown> ] [ 20.00 GB] [252] [ 33]
[dm-0 ] [ext4 ] [<unknown> ] [ 4.00 GB] [253] [ 0]
[dm-1 ] [swap ] [<unknown> ] [ 3.94 GB] [253] [ 1]
[dm-2 ] [ext4 ] [<unknown> ] [ 119.99 GB] [253] [ 2]
[dm-3 ] [ext4 ] [<unknown> ] [ 15.00 GB] [253] [ 3]
[dm-4 ] [ext4 ] [<unknown> ] [ 8.00 GB] [253] [ 4]
[dm-5 ] [ext4 ] [<unknown> ] [ 8.00 GB] [253] [ 5]
I'm thinking that during restore, I can just re-create the volume
groups and logical volumes and then restore each file system backup to
that logical volume. Or better yet, since I know how much space I'm
actually using, just create one logical volume of the right size. I
kept adding virtual disks as needed to store repos in /var/satellite
I also want to verify the syntax I'm using:
fsarchiver savefs -Aa -e "/mnt/media/*" -j 2
/mnt/media/voyager/boot.fsa /dev/vda1
Seemed to work fine for backing up /boot
Are there any other recommended options I should be using for backing
up live file systems mounted read / write?
I've also stopped all of the spacewalk services and other services on
the VM in order to minimize open files being skipped, etc.
Volume group structure.
[root at voyager media]# pvdisplay
--- Physical volume ---
PV Name /dev/vdb1
VG Name satellite
PV Size 100.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 25599
Free PE 0
Allocated PE 25599
PV UUID g3uGGu-p0b3-eSIJ-Bwy7-YOTD-GKnd-prWP7a
--- Physical volume ---
PV Name /dev/vdc1
VG Name satellite
PV Size 20.00 GiB / not usable 3.89 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 5119
Free PE 0
Allocated PE 5119
PV UUID W35GYr-T6pg-3e0o-s8I7-aqtc-fxcD-Emh62K
--- Physical volume ---
PV Name /dev/vda2
VG Name vg_voyager
PV Size 39.51 GiB / not usable 3.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 10114
Free PE 146
Allocated PE 9968
PV UUID hJCdct-iR6Q-NPYi-eBZN-dZdP-x4YP-U1zyvE
[root at voyager media]# vgdisplay
--- Volume group ---
VG Name satellite
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 119.99 GiB
PE Size 4.00 MiB
Total PE 30718
Alloc PE / Size 30718 / 119.99 GiB
Free PE / Size 0 / 0
VG UUID fXvCp3-N0uG-rBRc-FWVJ-Kpv3-AH9L-1PnYUy
--- Volume group ---
VG Name vg_voyager
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 9
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 5
Open LV 5
Max PV 0
Cur PV 1
Act PV 1
VG Size 39.51 GiB
PE Size 4.00 MiB
Total PE 10114
Alloc PE / Size 9968 / 38.94 GiB
Free PE / Size 146 / 584.00 MiB
VG UUID 3txqia-eDtn-j5wn-iixS-gfpv-90b9-ButDqh
[root at voyager media]# lvdisplay
--- Logical volume ---
LV Path /dev/satellite/lv_packages
LV Name lv_packages
VG Name satellite
LV UUID 03VUWu-bxGf-hG2b-c3cx-m3lu-7Dlp-iaiWzu
LV Write Access read/write
LV Creation host, time voyager, 2012-11-11 12:53:54 -0500
LV Status available
# open 1
LV Size 119.99 GiB
Current LE 30718
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2
--- Logical volume ---
LV Path /dev/vg_voyager/lv_var
LV Name lv_var
VG Name vg_voyager
LV UUID serQHO-uSog-ci5m-Xx7B-AElf-GTqi-HYCRY6
LV Write Access read/write
LV Creation host, time voyager, 2012-11-11 01:55:01 -0500
LV Status available
# open 1
LV Size 15.00 GiB
Current LE 3840
Segments 2
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3
--- Logical volume ---
LV Path /dev/vg_voyager/lv_root
LV Name lv_root
VG Name vg_voyager
LV UUID Kc43IB-5EWZ-N05E-FrN5-NgcQ-kWTv-LxfSig
LV Write Access read/write
LV Creation host, time voyager, 2012-11-11 01:55:05 -0500
LV Status available
# open 1
LV Size 4.00 GiB
Current LE 1024
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
--- Logical volume ---
LV Path /dev/vg_voyager/lv_home
LV Name lv_home
VG Name vg_voyager
LV UUID F7aJrw-FqwN-yML2-7bbX-kcuQ-12pX-1QG8Gp
LV Write Access read/write
LV Creation host, time voyager, 2012-11-11 01:55:09 -0500
LV Status available
# open 1
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
--- Logical volume ---
LV Path /dev/vg_voyager/lv_swap
LV Name lv_swap
VG Name vg_voyager
LV UUID S5uYT4-Q3x4-3icm-SEFW-yZVW-DhLl-vSkcLc
LV Write Access read/write
LV Creation host, time voyager, 2012-11-11 01:55:14 -0500
LV Status available
# open 1
LV Size 3.94 GiB
Current LE 1008
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1
--- Logical volume ---
LV Path /dev/vg_voyager/lv_tmp
LV Name lv_tmp
VG Name vg_voyager
LV UUID 2QeEXe-7zpq-0yLV-NT0u-9ZgY-mk8w-30n2Nz
LV Write Access read/write
LV Creation host, time voyager, 2012-11-11 01:55:14 -0500
LV Status available
# open 1
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
On Mon, Apr 22, 2013 at 3:53 PM, Joop <jvdwege at xs4all.nl> wrote:
> Chris Smith wrote:
>>
>> List,
>>
>> I have lost the ability to manage the hosts or VM's using ovirt engine
>> web interface. The data center is offline, and I
>> can't actually perform any operations with the hosts or VM's. I don't
>> think that there
>> are any actions I can perform in the web interface at all.
>>
>> What's odd is that I can tell the host to go into maintenance mode
>> using the ovirt-engine web interface and it seems to go into
>> maintenance mode. It even shows the wrench icon next to the host. I
>> can also try and activate it after it susposedly goes into maintenance
>> mode, and It states that the host was activated, but the host never
>> actually comes up or contends for SPM status, and the data center
>> never comes online.
>>
>> >From the logs it seems that at least PKI is broken between the engine
>> and the hosts as I see numerous certificate errors on both the
>> ovirt-engine and clients.
>>
>> vdsm.log shows:
>>
>> Traceback (most recent call last):
>> File "/usr/lib64/python2.7/SocketServer.py", line 582, in
>> process_request_thread
>> self.finish_request(request, client_address)
>> File "/usr/lib/python2.7/site-packages/vdsm/SecureXMLRPCServer.py",
>> line 66, in finish_request
>> request.do_handshake()
>> File "/usr/lib64/python2.7/ssl.py", line 305, in do_handshake
>> self._sslobj.do_handshake()
>> SSLError: [Errno 1] _ssl.c:504: error:14094416:SSL
>> routines:SSL3_READ_BYTES:sslv3 alert certificate unknown
>>
>> and engine.log shows:
>>
>> 2013-04-18 18:42:43,632 ERROR
>> [org.ovirt.engine.core.
>> engineencryptutils.EncryptionUtils]
>> (QuartzScheduler_Worker-68) Failed to decryptData must start with zero
>> 2013-04-18 18:42:43,642 ERROR
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand]
>> (QuartzScheduler_Worker-68) XML RPC error in command
>>
>>
>> Alon Bar-Lev was able to offer several good pointers in another thread
>> titled "Certificates and PKI seem to be broken after yum update" and
>> eventually concluded that the installation seems to be corrupted more
>> than just the certificates, truststore, and keystore, and suggested
>> that I start a new thread to ask about how to rescue the storage
>> domain structure.
>>
>> The storage used for the data center is ISCSI, which is intact and
>> working. In fact 2 of the VM's are still online and running on one of
>> the original FC17 hosts systems.
>>
>> I'm not able to reinstall any of the existing hosts from the ovirt-engine
>> web
>> interface. I attempted to reinstall one of the hosts (not the SPM)
>> which failed.
>>
>> I also tried to bring up a new, third host and add it to the cluster.
>> I setup another Fedora 17 box up and tried to add it to the
>> cluster, but it states that there are no available servers in the
>> cluster to probe the new host.
>>
>> This is a test environment that I would like to fix, but I'm also
>> willing to just run engine cleanup and start over.
>>
>> That said, there are 3 VM's that I would like to keep. Two are online
>> and running, and I'm able to see them with virsh on that host. I was
>> wondering about using virsh to backup these vm's.
>>
>> The third VM exists in the database, and was set to run on the host
>> that I attempted to reinstall, but that VM isn't running, and when I
>> use virsh on it's host, virsh can't seem to find it, when I perform
>> the list commands, and I can't start it with virsh <vm-name>
>>
>> What is the best way to proceed? It seems like it would be easier to
>> export the VM's using virsh from the host that they run on if
>> possible, then update ovirt to the latest version, recreate everything
>> and then import the VM's back in to the new environment.
>>
>> Will this work? Is there a procedure I can follow to do this?
>>
>> Here's some additional information about the installed ovirt packages
>> on the ovirt-engine
>>
>> [
>
> If you want a backup of the currently running hosts you can use fsarchiver.
> There is a statically linked version consisting of one executable on the
> website of fsarchiver and you can use options to overrule the fact that
> you're backing up a live system.
> You can't shutdown the VM's, I think, and then do an export to an export
> domain since you don't have a master storage domain thats why the above
> workaround with fsarchiver. You can ofcourse use you're favourite backup
> programme.
>
>
> Joop
>
> --
> irc: jvandewege
>
More information about the Users
mailing list