[ovirt-users] Proper way to change and persist vdsm configuration options

Tue Aug 5 15:14:28 EDT 2014

Thank you for rapid responses!  I'll try to way a few minutes before
responding to keep from spamming the list :)

On Tue, Aug 5, 2014 at 1:54 PM, Alon Bar-Lev <alonbl at redhat.com> wrote:
>
>
> ----- Original Message -----
>> From: "Trey Dockendorf" <treydock at gmail.com>
>> To: "Alon Bar-Lev" <alonbl at redhat.com>
>> Cc: "users" <users at ovirt.org>, "Fabian Deutsch" <fabiand at redhat.com>, "Dan Kenigsberg" <danken at redhat.com>, "Itamar
>> Heim" <iheim at redhat.com>, "Douglas Landgraf" <dougsland at redhat.com>, "ybronhei" <ybronhei at redhat.com>
>> Sent: Tuesday, August 5, 2014 9:50:04 PM
>> Subject: Re: [ovirt-users] Proper way to change and persist vdsm configuration options
>>
>> On Tue, Aug 5, 2014 at 12:37 PM, Alon Bar-Lev <alonbl at redhat.com> wrote:
>> > Hi,
>> >
>> > Do you actually use puppet over ovirt-node?
>> > This is unsupported.
>> >
>> > Regards,
>> > Alon
>> >
>>
>> I use Puppet to configure everything on the system and some of those
>> things conflict with changes made by ovirt-engine when adding a node
>> so I've moved to managing those changes in an ovirt Puppet module.
>>
>> When you refer to "ovirt-node" your refering to the pre-built ISO
>> images?  AFAIK the ovirt-node is not an option for me as I use
>> Infiniband to do all storage connections.  Currently NFS is done using
>> IPoIB (NFS over RDMA was crashing so I am not pursuing at the moment)
>> and iSCSI is done using iSER.  At some point in time I'd like to use
>> IB interfaces in my guests utilizing SR-IOV which my Mellanox cards
>> support, but I'm postponing that project till after our HPC cluster's
>> upgraded :)
>>
>> Right now the Puppet module does the following:
>>
>> * Configures firewall rules just like what is done by ovirt (with
>> exception of a few not supported by puppetlabs-firewall, so I override
>> global directive that purges unknown firewall rules)
>
> this should be done at engine side, using engine-config, seek IPTablesConfig or IPTablesConfigSiteCustom in 3.5.
>

I will look into this, thank you!  I had seen other threads about this
but since I apply my "firewall" module globally I thought it be
"easier" to handle in Puppet, but if the oVirt support method is using
"engine-config" then I'll switch to that and ensure the firewall is
managed via oVirt for nodes.

>> ** This is necessary as I have to add other Firewall rules, such as Zabbix
>> * Exports /etc/hosts entry for the node that is collected by
>> ovirt-engine host so that if DNS goes down ovirt-engine does not lose
>> access to ovirtmgmt interfaces
>
> why not just have a backup for dns?
>

This is somewhat complicated.  My oVirt nodes have no IP address on my
campus' public networks.  They are physically connected to numerous
public networks (one being on science DMZ) and for security reasons
they only have IPs on my private LAN.  My private LAN's DNS is managed
by me as our campus DNS does not yet support a way for me to apply DNS
changes to RFC1918 spaces via any form of API (foreman's smart-proxy).
I do have two DNS VMs that can handle things.  In the past I had only
1 VM doing DNS and when it went offline for maintenance ovirt-engine
saw the ovirt-nodes as offline and tried to fence them due to name
resolution being down.  To remove DNS as a point-of-failure for oVirt
I've decided to use /etc/hosts.  It's a hack and one I'm comfortable
with until I'm more comfortable with my DNS deployment :)

>> * Install yum repos for ovirt
>
> this should not be done within ovirt-node.
>
>> * install vdsm
>
> this should be done by host-deploy.
>
>> * ensure vdsm.conf exists
>> * Populates /etc/vdsm/vdsm.id (IIRC a bug in previous ovirt required this)
>
> why? you machine should have valid bios uuid.
>

IIRC it's because my servers BIOS serial numbers are all the same.
They are all 0123456789.  I vaguely recall running into an issue when
EL6 was first supported where the vdsm.id file on my first two nodes
ended up having identical contents and I believe it was due to the
BIOS serial numbers being the same (it's something our vendor
neglected I think, but before my time).

>> * Ensures vdsmd is running and will start at boot
>> * Ensures vdsm sudo rules are present.
>> * Manages default vdsm.conf configurations as a Puppet type,
>> "vdsm_config", rather than managing file contents via template (allows
>> for purging unmanaged entries also)
>
> the above should be done via host-deploy.
>
>> A lot of the above is handled by ovirt already, but in the past
>> customizations were not possible (firewall rules, vdsm.conf entries,
>> etc) so if I was going to have to manage those separately I wanted
>> them in Puppet :)
>>
>> Now that I'm aware of "ovirt-host-deploy" and have seen the potential
>> of using vdsm-reg in Puppet, I'm curious what is the "right" way to
>> automate node deployment's in oVirt. Ideally I could still use Puppet
>> to configure the method or fill in the "gaps" for customizations that
>> are needed (ie enabling iSER).
>>
>> I'd be glad to know what the recommended method for automating ovirt
>> would be, and would be happy to refactor my module in hopes it would
>> offer other Puppet users a quick-start way to deploy oVirt while still
>> doing things the "ovirt way".
>>
>> If the right way is using the ovirt-node images then I'd like to know
>> what customizations are possible on those images and so on.  oVirt is
>> moving very rapidly and despite using ovirt for a long time I'm still
>> learning new things about it almost daily, so forgive any of my
>> assumptions above that may be wrong :).
>
> ovirt-node is not flexible as standard host, if you require sssd and misc packages I suggest you stick with generic host.
>

That was one of my assumptions.  The use of sssd is not entirely
required but for simplicity sake I apply it across the board to all
systems to maintain UID and GID consistency.  If ovirt-node supports
using InfiniBand and configuring an Infiniband with
"CONNECTED_MODE=yes" (MTU of 65520) then that's really the main
requirement.  To support SR-IOV using Mellanox cards I may have to use
the Mellanox OFED packages instead of upstream "Infiniband Support"
packages, and if that's the case then I will have to use a generic
host as something like Mellanox packages obviously can't be considered
for support in ovirt-node.

Thanks,
- Trey

>>
>> Thanks,
>> - Trey
>>
>> > ----- Original Message -----
>> >> From: "ybronhei" <ybronhei at redhat.com>
>> >> To: "Trey Dockendorf" <treydock at gmail.com>
>> >> Cc: "users" <users at ovirt.org>, "Fabian Deutsch" <fabiand at redhat.com>, "Dan
>> >> Kenigsberg" <danken at redhat.com>, "Itamar
>> >> Heim" <iheim at redhat.com>, "Douglas Landgraf" <dougsland at redhat.com>, "Alon
>> >> Bar-Lev" <alonbl at redhat.com>
>> >> Sent: Tuesday, August 5, 2014 8:32:04 PM
>> >> Subject: Re: [ovirt-users] Proper way to change and persist vdsm
>> >> configuration options
>> >>
>> >> Hey,
>> >>
>> >> Just noticed something that I forgot about..
>> >> before filing new BZ, see in ovirt-host-deploy README.environment [1]
>> >> the section:
>> >> VDSM/configOverride(bool) [True]
>> >>      Override vdsm configuration file.
>> >>
>> >> changing it to false will keep your vdsm.conf file as is after deploying
>> >> the host again (what happens after node upgrade)
>> >>
>> >> [1]
>> >> https://github.com/oVirt/ovirt-host-deploy/blob/master/README.environment
>> >>
>> >> please check if that what you meant..
>> >>
>> >> Thanks,
>> >> Yaniv Bronhaim.
>> >>
>> >> On 08/05/2014 08:12 AM, Trey Dockendorf wrote:
>> >> > I'll file BZ.  As far as I can recall this has been an issue since 3.3.x
>> >> > as
>> >> > I have been using Puppet to modify values and have had to rerun Puppet
>> >> > after installing a node via GUI and when performing update from GUI.
>> >> > Given
>> >> > that it has occurred when VDSM version didn't change on the node it
>> >> > seems
>> >> > likely to be something being done by Python code that bootstraps a node
>> >> > and
>> >> > performs the other tasks.  I won't have any systems available to test
>> >> > with
>> >> > for a few days.  New hardware specifically for our oVirt deployment is
>> >> > on
>> >> > order so should be able to more thoroughly debug and capture logs at
>> >> > that
>> >> > time.
>> >> >
>> >> > Would using vdsm-reg be a better solution for adding new nodes?  I only
>> >> > tried using vdsm-reg once and it went very poorly...lots of missing
>> >> > dependencies not pulled in from yum install I had to install manually
>> >> > via
>> >> > yum.  Then the node was auto added to newest cluster with no ability to
>> >> > change the cluster.  Be happy to debug that too if there's some docs
>> >> > that
>> >> > outline the expected behavior.
>> >> >
>> >> > Using vdsm-reg or something similar seems like a better fit for puppet
>> >> > deployed nodes, as opposed to requiring GUI steps to add the node.
>> >> >
>> >> > Thanks
>> >> > - Trey
>> >> > On Aug 4, 2014 5:53 AM, "ybronhei" <ybronhei at redhat.com> wrote:
>> >> >
>> >> >> On 07/31/2014 01:28 AM, Trey Dockendorf wrote:
>> >> >>
>> >> >>> I'm running ovirt nodes that are stock CentOS 6.5 systems with VDSM
>> >> >>> installed.  I am using iSER to do iSCSI over RDMA and to make that
>> >> >>> work I have to modify /etc/vdsm/vdsm.conf to include the following:
>> >> >>>
>> >> >>> [irs]
>> >> >>> iscsi_default_ifaces = iser,default
>> >> >>>
>> >> >>> I've noticed that any time I upgrade a node from the engine web
>> >> >>> interface that changes to vdsm.conf are wiped out.  I don't know if
>> >> >>> this is being done by the configuration code or by the vdsm package.
>> >> >>> Is there a more reliable way to ensure changes to vdsm.conf are NOT
>> >> >>> removed automatically?
>> >> >>>
>> >> >>
>> >> >> Hey,
>> >> >>
>> >> >> vdsm.conf shouldn't wiped out and shouldn't changed at all during
>> >> >> upgrade.
>> >> >> other related conf files (such as libvirtd.conf) might be overrided to
>> >> >> keep
>> >> >> defaults configurations for vdsm. but vdsm.conf should persist with
>> >> >> user's
>> >> >> modification. from my check, regular yum upgrade doesn't touch
>> >> >> vdsm.conf
>> >> >>
>> >> >> Douglas can you verify that with node upgrade? might be specific to
>> >> >> that
>> >> >> flow..
>> >> >>
>> >> >> Trey, can file a bugzilla on that and describe your steps there?
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> Yaniv Bronhaim,
>> >> >>
>> >> >>>
>> >> >>> Thanks,
>> >> >>> - Trey
>> >> >>> _______________________________________________
>> >> >>> Users mailing list
>> >> >>> Users at ovirt.org
>> >> >>> http://lists.ovirt.org/mailman/listinfo/users
>> >> >>>
>> >> >>>
>> >> >>
>> >> >> --
>> >> >> Yaniv Bronhaim.
>> >> >>
>> >> >
>> >>
>>