[ovirt-users] Recommended setup for a FC based storage domain

Thu Jun 12 13:31:24 UTC 2014

CC'ing the devel list, maybe some VDSM and storage people can
explain this?

Am 10.06.2014 12:24, schrieb combuster:
> /etc/libvirt/libvirtd.conf and /etc/vdsm/logger.conf
> 
> , but unfortunately maybe I've jumped to conclusions, last weekend, that
> very same thin provisioned vm was running a simple export for 3hrs
> before I've killed the process. But I wondered:
> 
> 1. The process that runs behind the export is qemu-img convert (from raw
> to raw), and running iotop shows that every three or four seconds it
> reads 10-13 MBps and then idles for a few seconds. Run the numbers on
> 100GB (why is he covering the entire 100 of 15GB used on thin volume I
> still don't get it) and you get precisely 3-4 hrs estimated time remaining.
> 2. When I run export with SPM on a node that doesn't have any vm's
> running, export finishes for aprox. 30min (iotop shows 40-70MBps read
> speed constantly)
> 3. Renicing I/O priority of the qemu-img process as well as the CPU
> priority gave no results, it was still runing slow beyond any explanation.
> 
> Debug logs showed nothing of interest, so I disabled anything above
> warning and it suddenly accelerated the export, so I've connected the
> wrong dots.
> 
> On 06/10/2014 11:18 AM, Andrew Lau wrote:
>> Interesting, which files did you modify to lower the log levels?
>>
>> On Tue, Jun 3, 2014 at 12:38 AM,  <combuster at archlinux.us> wrote:
>>> One word of caution so far, when exporting any vm, the node that acts
>>> as SPM
>>> is stressed out to the max. I releived the stress by a certain margin
>>> with
>>> lowering libvirtd and vdsm log levels to WARNING. That shortened out the
>>> export procedure by at least five times. But vdsm process on the SPM
>>> node  is
>>> still with high cpu usage so it's best that the SPM node should be
>>> left with a
>>> decent CPU time amount to spare. Also, export of VM's with high vdisk
>>> capacity
>>> and thin provisioning enabled (let's say 14GB used of 100GB defined)
>>> took
>>> around 50min over a 10Gb ethernet interface to a 1Gb export NAS
>>> device that
>>> was not stressed out at all by other processes. When I did that
>>> export with
>>> debug log levels it took 5hrs :(
>>>
>>> So lowering log levels is a must in production enviroment. I've
>>> deleted the
>>> lun that I exported on the storage (removed it first from ovirt) and
>>> for the
>>> next weekend I am planing to add a new one, export it again on all
>>> the nodes
>>> and start a few fresh vm installations. Things I'm going to look for are
>>> partition alignment and running them from different nodes in the
>>> cluster at
>>> the same time. I just hope that not all I/O is going to pass through
>>> the SPM,
>>> this is the one thing that bothers me the most.
>>>
>>> I'll report back on these results next week, but if anyone has
>>> experience with
>>> this kind of things or can point  to some documentation would be great.
>>>
>>> On Monday, 2. June 2014. 18.51.52 you wrote:
>>>> I'm curious to hear what other comments arise, as we're analyzing a
>>>> production setup shortly.
>>>>
>>>> On Sun, Jun 1, 2014 at 10:11 PM,  <combuster at archlinux.us> wrote:
>>>>> I need to scratch gluster off because setup is based on CentOS 6.5, so
>>>>> essential prerequisites like qemu 1.3 and libvirt 1.0.1 are not met.
>>>> Gluster would still work with EL6, afaik it just won't use libgfapi and
>>>> instead use just a standard mount.
>>>>
>>>>> Any info regarding FC storage domain would be appreciated though.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Ivan
>>>>>
>>>>> On Sunday, 1. June 2014. 11.44.33 combuster at archlinux.us wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have a 4 node cluster setup and my storage options right now are
>>>>>> a FC
>>>>>> based storage, one partition per node on a local drive (~200GB
>>>>>> each) and
>>>>>> a
>>>>>> NFS based NAS device. I want to setup export and ISO domain on the
>>>>>> NAS
>>>>>> and
>>>>>> there are no issues or questions regarding those two. I wasn't
>>>>>> aware of
>>>>>> any
>>>>>> other options at the time for utilizing a local storage (since
>>>>>> this is a
>>>>>> shared based datacenter) so I exported a directory from each
>>>>>> partition
>>>>>> via
>>>>>> NFS and it works. But I am little in the dark with the following:
>>>>>>
>>>>>> 1. Are there any advantages for switching from NFS based local
>>>>>> storage to
>>>>>> a
>>>>>> Gluster based domain with blocks for each partition. I guess it
>>>>>> can be
>>>>>> only
>>>>>> performance wise but maybe I'm wrong. If there are advantages, are
>>>>>> there
>>>>>> any tips regarding xfs mount options etc ?
>>>>>>
>>>>>> 2. I've created a volume on the FC based storage and exported it
>>>>>> to all
>>>>>> of
>>>>>> the nodes in the cluster on the storage itself. I've configured
>>>>>> multipathing correctly and added an alias for the wwid of the LUN
>>>>>> so I
>>>>>> can
>>>>>> distinct this one and any other future volumes more easily. At
>>>>>> first I
>>>>>> created a partition on it but since oVirt saw only the whole LUN
>>>>>> as raw
>>>>>> device I erased it before adding it as the FC master storage
>>>>>> domain. I've
>>>>>> imported a few VM's and point them to the FC storage domain. This
>>>>>> setup
>>>>>> works, but:
>>>>>>
>>>>>> - All of the nodes see a device with the alias for the wwid of the
>>>>>> volume,
>>>>>> but only the node wich is currently the SPM for the cluster can see
>>>>>> logical
>>>>>> volumes inside. Also when I setup the high availability for VM's
>>>>>> residing
>>>>>> on the FC storage and select to start on any node on the cluster,
>>>>>> they
>>>>>> always start on the SPM. Can multiple nodes run different VM's on the
>>>>>> same
>>>>>> FC storage at the same time (logical thing would be that they can,
>>>>>> but I
>>>>>> wanted to be sure first). I am not familiar with the logic oVirt
>>>>>> utilizes
>>>>>> that locks the vm's logical volume to prevent corruption.
>>>>>>
>>>>>> - Fdisk shows that logical volumes on the LUN of the FC volume are
>>>>>> missaligned (partition doesn't end on cylindar boundary), so I
>>>>>> wonder if
>>>>>> this is becuase I imported the VM's with disks that were created
>>>>>> on local
>>>>>> storage before and that any _new_ VM's with disks on the fc
>>>>>> storage would
>>>>>> be propperly aligned.
>>>>>>
>>>>>> This is a new setup with oVirt 3.4 (did an export of all the VM's
>>>>>> on 3.3
>>>>>> and after a fresh installation of the 3.4 imported them back
>>>>>> again). I
>>>>>> have room to experiment a little with 2 of the 4 nodes because
>>>>>> currently
>>>>>> they are free from running any VM's, but I have limited room for
>>>>>> anything else that would cause an unplanned downtime for four virtual
>>>>>> machines running on the other two nodes on the cluster (currently
>>>>>> highly
>>>>>> available and their drives are on the FC storage domain). All in
>>>>>> all I
>>>>>> have 12 VM's running and I'm asking on the list for advice and
>>>>>> guidance
>>>>>> before I make any changes.
>>>>>>
>>>>>> Just trying to find as much info regarding all of this as possible
>>>>>> before
>>>>>> acting upon.
>>>>>>
>>>>>> Thank you in advance,
>>>>>>
>>>>>> Ivan

-- 
Mit freundlichen Grüßen / Regards

Sven Kieske

Systemadministrator
Mittwald CM Service GmbH & Co. KG
Königsberger Straße 6
32339 Espelkamp
T: +49-5772-293-100
F: +49-5772-293-333
https://www.mittwald.de
Geschäftsführer: Robert Meyer
St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen
Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen