[ovirt-users] Re: Is there a way to support Mellanox OFED with oVirt/RHV?

Hi Edward, it seems that running mlnxofedinstall would do the job. Although I've some questions. You mentioned the --enable-repo option but I didn't find it. There's a disable one, so I'm assuming that it's enabled by default. Anyway there's no repos added after the script. I've run the script with the arguments: ./mlnxofedinstall --with-nfsrdma -vvv; and everything went fine: [root@rhvepyc2 mnt]# /etc/init.d/openibd status HCA driver loaded Configured IPoIB devices: ib0 Currently active IPoIB devices: ib0 Configured Mellanox EN devices: Currently active Mellanox devices: ib0 The following OFED modules are loaded: rdma_ucm rdma_cm ib_ipoib mlx5_core mlx5_ib ib_uverbs ib_umad ib_cm ib_core mlxfw [root@rhvepyc2 mnt]# rpm -qa | grep -i mlnx libibverbs-54mlnx1-1.54103.x86_64 infiniband-diags-54mlnx1-1.54103.x86_64 mlnx-ethtool-5.10-1.54103.x86_64 rdma-core-54mlnx1-1.54103.x86_64 dapl-utils-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64 kmod-mlnx-nfsrdma-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64 dapl-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64 mlnx-tools-5.2.0-0.54103.x86_64 libibumad-54mlnx1-1.54103.x86_64 opensm-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64 kmod-kernel-mft-mlnx-4.17.0-1.rhel8u4.x86_64 ibacm-54mlnx1-1.54103.x86_64 dapl-devel-static-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64 ar_mgr-1.0-5.9.0.MLNX20210617.g5dd71ee.54103.x86_64 mlnx-ofa_kernel-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64 rdma-core-devel-54mlnx1-1.54103.x86_64 opensm-static-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64 srp_daemon-54mlnx1-1.54103.x86_64 sharp-2.5.0.MLNX20210613.83fe753-1.54103.x86_64 mlnx-iproute2-5.11.0-1.54103.x86_64 kmod-knem-1.1.4.90mlnx1-OFED.5.1.2.5.0.1.rhel8u4.x86_64 librdmacm-54mlnx1-1.54103.x86_64 opensm-libs-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64 mlnx-ofa_kernel-devel-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64 dapl-devel-2.1.10.1.mlnx-OFED.4.9.0.1.4.54103.x86_64 dump_pr-1.0-5.9.0.MLNX20210617.g5dd71ee.54103.x86_64 mlnxofed-docs-5.4-1.0.3.0.noarch opensm-devel-5.9.0.MLNX20210617.c9f2ade-0.1.54103.x86_64 knem-1.1.4.90mlnx1-OFED.5.1.2.5.0.1.rhel8u4.x86_64 librdmacm-utils-54mlnx1-1.54103.x86_64 mlnx-fw-updater-5.4-1.0.3.0.x86_64 kmod-mlnx-ofa_kernel-5.4-OFED.5.4.1.0.3.1.rhel8u4.x86_64 libibverbs-utils-54mlnx1-1.54103.x86_64 ibutils2-2.1.1-0.136.MLNX20210617.g4883fca.54103.x86_64 As a final question, did you selected the option: --add-kernel-support on the script? I couldn't find the difference between enabling it or not. Thank you. On 5 Aug 2021, at 15:20, Vinícius Ferrão <ferrao(a)versatushpc.com.br&gt; wrote: Hmmm. Running the mlnx_ofed_install.sh script is a pain. But I got your idea. I'll do this test right now and report back. Ideally using the repo would guarantee an easy upgrade path between release, but Mellanox is lacking on this part. And yes Edward, I want to use the virtual Infiniband interfaces too. Thank you. On 5 Aug 2021, at 10:52, Edward Berger <edwberger(a)gmail.com&gt; wrote: I don't know if you can just remove the gluster-rdma rpm. I'm using mlnx ofed on some 4.4 ovirt node hosts by installing it via the mellanox tar/iso and running the mellanox install script after adding the required dependencies with --enable-repo, which isn't the same as adding a repository and 'dnf install'. So I would try that on a test host. I use it for the 'virtual infiniband' interfaces that get attached to VMs as 'host device passthru'. I'll note the node versions of gluster are 7.8 (node 4.4.4.0/CentOS8.3) and 7.9 (node 4.4.4.1/CentOS8.3). unlike your glusterfs version 6.0.x I'll be trying to install mellanox ofed on node 4.4.7.1 (CentOS 8 stream) soon to see how that works out. On Wed, Aug 4, 2021 at 10:04 PM Vinícius Ferrão via Users <users(a)ovirt.org&gt; wrote: > Hello, > > Is there a way to keep Mellanox OFED and oVirt/RHV playing nice with each > other? > > The real issue is regarding GlusterFS. It seems to be a Mellanox issue, > but I would like to know if there's something that we can do make both play > nice on the same machine: > > [root@rhvepyc2 ~]# dnf update --nobest > Updating Subscription Management repositories. > Last metadata expiration check: 0:14:25 ago on Wed 04 Aug 2021 02:01:11 > AM -03. > Dependencies resolved. > > Problem: both package mlnx-ofed-all-user-only-5.4-1.0.3.0.rhel8.4.noarch > and mlnx-ofed-all-5.4-1.0.3.0.rhel8.4.noarch obsolete glusterfs-rdma > - cannot install the best update candidate for package > glusterfs-rdma-6.0-49.1.el8.x86_64 > - package ovirt-host-4.4.7-1.el8ev.x86_64 requires glusterfs-rdma, but > none of the providers can be installed > - package mlnx-ofed-all-5.4-1.0.3.0.rhel8.4.noarch obsoletes > glusterfs-rdma provided by glusterfs-rdma-6.0-49.1.el8.x86_64 > - package glusterfs-rdma-3.12.2-40.2.el8.x86_64 requires > glusterfs(x86-64) = 3.12.2-40.2.el8, but none of the providers can be > installed > - package glusterfs-rdma-6.0-15.el8.x86_64 requires glusterfs(x86-64) = > 6.0-15.el8, but none of the providers can be installed > - package glusterfs-rdma-6.0-20.el8.x86_64 requires glusterfs(x86-64) = > 6.0-20.el8, but none of the providers can be installed > - package glusterfs-rdma-6.0-37.el8.x86_64 requires glusterfs(x86-64) = > 6.0-37.el8, but none of the providers can be installed > - package glusterfs-rdma-6.0-37.2.el8.x86_64 requires glusterfs(x86-64) > = 6.0-37.2.el8, but none of the providers can be installed > - cannot install both glusterfs-3.12.2-40.2.el8.x86_64 and > glusterfs-6.0-49.1.el8.x86_64 > - cannot install both glusterfs-6.0-15.el8.x86_64 and > glusterfs-6.0-49.1.el8.x86_64 > - cannot install both glusterfs-6.0-20.el8.x86_64 and > glusterfs-6.0-49.1.el8.x86_64 > - cannot install both glusterfs-6.0-37.el8.x86_64 and > glusterfs-6.0-49.1.el8.x86_64 > - cannot install both glusterfs-6.0-37.2.el8.x86_64 and > glusterfs-6.0-49.1.el8.x86_64 > - cannot install the best update candidate for package > ovirt-host-4.4.7-1.el8ev.x86_64 > - cannot install the best update candidate for package > glusterfs-6.0-49.1.el8.x86_64 > > ============================================================================================================================================================= > Package Architecture Version > Repository > Size > > ============================================================================================================================================================= > Installing dependencies: > openvswitch x86_64 > 2.14.1-1.54103 mlnx_ofed_5.4-1.0.3.0_base > 17 M > ovirt-openvswitch noarch 2.11-1.el8ev > rhv-4-mgmt-agent-for-rhel-8-x86_64-rpms > 8.7 k > replacing rhv-openvswitch.noarch 1:2.11-7.el8ev > unbound x86_64 1.7.3-15.el8 > rhel-8-for-x86_64-appstream-rpms > 895 k > Skipping packages with conflicts: > (add '--best --allowerasing' to command line to force their upgrade): > glusterfs x86_64 > 3.12.2-40.2.el8 rhel-8-for-x86_64-baseos-rpms > 558 k > glusterfs x86_64 6.0-15.el8 > rhel-8-for-x86_64-baseos-rpms > 658 k > glusterfs x86_64 6.0-20.el8 > rhel-8-for-x86_64-baseos-rpms > 659 k > glusterfs x86_64 6.0-37.el8 > rhel-8-for-x86_64-baseos-rpms > 663 k > glusterfs x86_64 6.0-37.2.el8 > rhel-8-for-x86_64-baseos-rpms > 662 k > Skipping packages with broken dependencies: > glusterfs-rdma x86_64 > 3.12.2-40.2.el8 rhel-8-for-x86_64-baseos-rpms > 49 k > glusterfs-rdma x86_64 6.0-15.el8 > rhel-8-for-x86_64-baseos-rpms > 46 k > glusterfs-rdma x86_64 6.0-20.el8 > rhel-8-for-x86_64-baseos-rpms > 46 k > glusterfs-rdma x86_64 6.0-37.2.el8 > rhel-8-for-x86_64-baseos-rpms > 48 k > glusterfs-rdma x86_64 6.0-37.el8 > rhel-8-for-x86_64-baseos-rpms > 48 k > > Transaction Summary > > ============================================================================================================================================================= > Install 3 Packages > Skip 10 Packages > > Total size: 18 M > Is this ok [y/N]: > > I really don't care for GlusterFS on this cluster, but Mellanox OFED is > much more relevant do me. > > Thank you all, > Vinícius. > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/MGQBIBM4BCH... >