The ovirt node-ng installer iso creates imgbased systems with baseos and appstream repos disabled,
not something you would have with a regular base OS installed system with added oVirt repos...

so with node, 'dnf install foo' usually fails if not adding an extra --enablerepo flag, which seems to change with the OS version.

here's some old notes I had.
download latest mlnx ofed archive
tar xfvz *.tgz
cd *64

# mount -o loop MLNX*iso /mnt
# cd /mnt

#./mlnxinstall requires more RPMS to be installed
# note: some versions of CentOS use different case reponames, look at contents of /etc/yum.repos.d files
yum --enablerepo baseos install perl-Term-ANSIColor
yum --enablerepo baseos --enablerepo appstream install perl-Getopt-Long tcl gcc-gfortran tcsh tk make
./mlnxinstall

On Thu, Aug 5, 2021 at 3:32 PM Vinícius Ferrão <ferrao@versatushpc.com.br> wrote:
Hi Edward, it seems that running mlnxofedinstall would do the job. Although I've some questions.

You mentioned the --enable-repo option but I didn't find it. There's a disable one, so I'm assuming that it's enabled by default. Anyway there's no repos added after the script.

I've run the script with the arguments: ./mlnxofedinstall --with-nfsrdma -vvv; and everything went fine:

[root@rhvepyc2 mnt]# /etc/init.d/openibd status

  HCA driver loaded

Configured IPoIB devices:
ib0

Currently active IPoIB devices:
ib0
Configured Mellanox EN devices:

Currently active Mellanox devices:
ib0

The following OFED modules are loaded:

  rdma_ucm
  rdma_cm
  ib_ipoib
  mlx5_core
  mlx5_ib
  ib_uverbs
  ib_umad
  ib_cm
  ib_core
  mlxfw

[root@rhvepyc2 mnt]# rpm -qa | grep -i mlnx
libibverbs-54mlnx1-1.54103.x86_64
infiniband-diags-54mlnx1-1.54103.x86_64
mlnx-ethtool-5.10-1.54103.x86_64
rdma-core-54mlnx1-1.54103.x86_64
dapl-utils-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64
kmod-mlnx-nfsrdma-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64
dapl-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64
mlnx-tools-5.2.0-0.54103.x86_64
libibumad-54mlnx1-1.54103.x86_64
opensm-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64
kmod-kernel-mft-mlnx-4.17.0-1.rhel8u4.x86_64
ibacm-54mlnx1-1.54103.x86_64
dapl-devel-static-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64
ar_mgr-1.0-5.9.0.MLNX20210617.g5dd71ee.54103.x86_64
mlnx-ofa_kernel-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64
rdma-core-devel-54mlnx1-1.54103.x86_64
opensm-static-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64
srp_daemon-54mlnx1-1.54103.x86_64
sharp-2.5.0.MLNX20210613.83fe753-1.54103.x86_64
mlnx-iproute2-5.11.0-1.54103.x86_64
kmod-knem-1.1.4.90mlnx1-OFED.5.1.2.5.0.1.rhel8u4.x86_64
librdmacm-54mlnx1-1.54103.x86_64
opensm-libs-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64
mlnx-ofa_kernel-devel-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64
dapl-devel-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64
dump_pr-1.0-5.9.0.MLNX20210617.g5dd71ee.54103.x86_64
mlnxofed-docs-5.4-1.0.3.0.noarch
opensm-devel-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64
knem-1.1.4.90mlnx1-OFED.5.1.2.5.0.1.rhel8u4.x86_64
librdmacm-utils-54mlnx1-1.54103.x86_64
mlnx-fw-updater-5.4-1.0.3.0.x86_64
kmod-mlnx-ofa_kernel-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64
libibverbs-utils-54mlnx1-1.54103.x86_64
ibutils2-2.1.1-0.136.MLNX20210617.g4883fca.54103.x86_64

As a final question, did you selected the option: --add-kernel-support on the script? I couldn't find the difference between enabling it or not.

Thank you.

On 5 Aug 2021, at 15:20, Vinícius Ferrão <ferrao@versatushpc.com.br> wrote:

Hmmm. Running the mlnx_ofed_install.sh script is a pain. But I got your idea. I'll do this test right now and report back. Ideally using the repo would guarantee an easy upgrade path between release, but Mellanox is lacking on this part.

And yes Edward, I want to use the virtual Infiniband interfaces too.

Thank you.

On 5 Aug 2021, at 10:52, Edward Berger <edwberger@gmail.com> wrote:

I don't know if you can just remove the gluster-rdma rpm.

I'm using mlnx ofed on some 4.4 ovirt node hosts by installing it via the mellanox tar/iso and
running the mellanox install script after adding the required dependencies with --enable-repo,
which isn't the same as adding a repository and 'dnf install'.  So I would try that on a test host.

I use it for the 'virtual infiniband' interfaces that get attached to VMs as 'host device passthru'.

I'll note the node versions of gluster are 7.8 (node 4.4.4.0/CentOS8.3) and 7.9 (node 4.4.4.1/CentOS8.3).
unlike your glusterfs version 6.0.x

I'll be trying to install mellanox ofed on node 4.4.7.1 (CentOS 8 stream) soon to see how that works out.



On Wed, Aug 4, 2021 at 10:04 PM Vinícius Ferrão via Users <users@ovirt.org> wrote:
Hello,

Is there a way to keep Mellanox OFED and oVirt/RHV playing nice with each other?

The real issue is regarding GlusterFS. It seems to be a Mellanox issue, but I would like to know if there's something that we can do make both play nice on the same machine:

[root@rhvepyc2 ~]# dnf update --nobest
Updating Subscription Management repositories.
Last metadata expiration check: 0:14:25 ago on Wed 04 Aug 2021 02:01:11 AM -03.
Dependencies resolved.

 Problem: both package mlnx-ofed-all-user-only-5.4-1.0.3.0.rhel8.4.noarch and mlnx-ofed-all-5.4-1.0.3.0.rhel8.4.noarch obsolete glusterfs-rdma
  - cannot install the best update candidate for package glusterfs-rdma-6.0-49.1.el8.x86_64
  - package ovirt-host-4.4.7-1.el8ev.x86_64 requires glusterfs-rdma, but none of the providers can be installed
  - package mlnx-ofed-all-5.4-1.0.3.0.rhel8.4.noarch obsoletes glusterfs-rdma provided by glusterfs-rdma-6.0-49.1.el8.x86_64
  - package glusterfs-rdma-3.12.2-40.2.el8.x86_64 requires glusterfs(x86-64) = 3.12.2-40.2.el8, but none of the providers can be installed
  - package glusterfs-rdma-6.0-15.el8.x86_64 requires glusterfs(x86-64) = 6.0-15.el8, but none of the providers can be installed
  - package glusterfs-rdma-6.0-20.el8.x86_64 requires glusterfs(x86-64) = 6.0-20.el8, but none of the providers can be installed
  - package glusterfs-rdma-6.0-37.el8.x86_64 requires glusterfs(x86-64) = 6.0-37.el8, but none of the providers can be installed
  - package glusterfs-rdma-6.0-37.2.el8.x86_64 requires glusterfs(x86-64) = 6.0-37.2.el8, but none of the providers can be installed
  - cannot install both glusterfs-3.12.2-40.2.el8.x86_64 and glusterfs-6.0-49.1.el8.x86_64
  - cannot install both glusterfs-6.0-15.el8.x86_64 and glusterfs-6.0-49.1.el8.x86_64
  - cannot install both glusterfs-6.0-20.el8.x86_64 and glusterfs-6.0-49.1.el8.x86_64
  - cannot install both glusterfs-6.0-37.el8.x86_64 and glusterfs-6.0-49.1.el8.x86_64
  - cannot install both glusterfs-6.0-37.2.el8.x86_64 and glusterfs-6.0-49.1.el8.x86_64
  - cannot install the best update candidate for package ovirt-host-4.4.7-1.el8ev.x86_64
  - cannot install the best update candidate for package glusterfs-6.0-49.1.el8.x86_64
=============================================================================================================================================================
 Package                            Architecture            Version                           Repository                                                Size
=============================================================================================================================================================
Installing dependencies:
 openvswitch                        x86_64                  2.14.1-1.54103                    mlnx_ofed_5.4-1.0.3.0_base                                17 M
 ovirt-openvswitch                  noarch                  2.11-1.el8ev                      rhv-4-mgmt-agent-for-rhel-8-x86_64-rpms                  8.7 k
     replacing  rhv-openvswitch.noarch 1:2.11-7.el8ev
 unbound                            x86_64                  1.7.3-15.el8                      rhel-8-for-x86_64-appstream-rpms                         895 k
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 glusterfs                          x86_64                  3.12.2-40.2.el8                   rhel-8-for-x86_64-baseos-rpms                            558 k
 glusterfs                          x86_64                  6.0-15.el8                        rhel-8-for-x86_64-baseos-rpms                            658 k
 glusterfs                          x86_64                  6.0-20.el8                        rhel-8-for-x86_64-baseos-rpms                            659 k
 glusterfs                          x86_64                  6.0-37.el8                        rhel-8-for-x86_64-baseos-rpms                            663 k
 glusterfs                          x86_64                  6.0-37.2.el8                      rhel-8-for-x86_64-baseos-rpms                            662 k
Skipping packages with broken dependencies:
 glusterfs-rdma                     x86_64                  3.12.2-40.2.el8                   rhel-8-for-x86_64-baseos-rpms                             49 k
 glusterfs-rdma                     x86_64                  6.0-15.el8                        rhel-8-for-x86_64-baseos-rpms                             46 k
 glusterfs-rdma                     x86_64                  6.0-20.el8                        rhel-8-for-x86_64-baseos-rpms                             46 k
 glusterfs-rdma                     x86_64                  6.0-37.2.el8                      rhel-8-for-x86_64-baseos-rpms                             48 k
 glusterfs-rdma                     x86_64                  6.0-37.el8                        rhel-8-for-x86_64-baseos-rpms                             48 k

Transaction Summary
=============================================================================================================================================================
Install   3 Packages
Skip     10 Packages

Total size: 18 M
Is this ok [y/N]:

I really don't care for GlusterFS on this cluster, but Mellanox OFED is much more relevant do me.

Thank you all,
Vinícius.
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MGQBIBM4BCHBBMLCY2QDKAR3Q6OE5LCX/