[ovirt-users] oVirt 3.2 - iSCSI offload (broadcom - bnx2i)

Wed Oct 8 20:28:49 UTC 2014

On 10/08/2014 08:29 AM, Ricardo Esteves wrote:
> Hi,
>
> lspci -nn
>
> 00:00.0 Host bridge [0600]: Intel Corporation 5500 I/O Hub to ESI Port 
> [8086:3403] (rev 13)
> 00:01.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI 
> Express Root Port 1 [8086:3408] (rev 13)
> 00:02.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI 
> Express Root Port 2 [8086:3409] (rev 13)
> 00:03.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI 
> Express Root Port 3 [8086:340a] (rev 13)
> 00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI 
> Express Root Port 7 [8086:340e] (rev 13)
> 00:08.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI 
> Express Root Port 8 [8086:340f] (rev 13)
> 00:09.0 PCI bridge [0604]: Intel Corporation 7500/5520/5500/X58 I/O 
> Hub PCI Express Root Port 9 [8086:3410] (rev 13)
> 00:0a.0 PCI bridge [0604]: Intel Corporation 7500/5520/5500/X58 I/O 
> Hub PCI Express Root Port 10 [8086:3411] (rev 13)
> 00:0d.0 Host bridge [0600]: Intel Corporation Device [8086:343a] (rev 13)
> 00:0d.1 Host bridge [0600]: Intel Corporation Device [8086:343b] (rev 13)
> 00:0d.2 Host bridge [0600]: Intel Corporation Device [8086:343c] (rev 13)
> 00:0d.3 Host bridge [0600]: Intel Corporation Device [8086:343d] (rev 13)
> 00:0d.4 Host bridge [0600]: Intel Corporation 7500/5520/5500/X58 
> Physical Layer Port 0 [8086:3418] (rev 13)
> 00:0d.5 Host bridge [0600]: Intel Corporation 7500/5520/5500 Physical 
> Layer Port 1 [8086:3419] (rev 13)
> 00:0d.6 Host bridge [0600]: Intel Corporation Device [8086:341a] (rev 13)
> 00:0e.0 Host bridge [0600]: Intel Corporation Device [8086:341c] (rev 13)
> 00:0e.1 Host bridge [0600]: Intel Corporation Device [8086:341d] (rev 13)
> 00:0e.2 Host bridge [0600]: Intel Corporation Device [8086:341e] (rev 13)
> 00:0e.3 Host bridge [0600]: Intel Corporation Device [8086:341f] (rev 13)
> 00:0e.4 Host bridge [0600]: Intel Corporation Device [8086:3439] (rev 13)
> 00:14.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub 
> System Management Registers [8086:342e] (rev 13)
> 00:14.1 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO 
> and Scratch Pad Registers [8086:3422] (rev 13)
> 00:14.2 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub 
> Control Status and RAS Registers [8086:3423] (rev 13)
> 00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) 
> PCI Express Root Port 1 [8086:3a40]
> 00:1d.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 
> Family) USB UHCI Controller #1 [8086:3a34]
> 00:1d.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 
> Family) USB UHCI Controller #2 [8086:3a35]
> 00:1d.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 
> Family) USB UHCI Controller #3 [8086:3a36]
> 00:1d.3 USB controller [0c03]: Intel Corporation 82801JI (ICH10 
> Family) USB UHCI Controller #6 [8086:3a39]
> 00:1d.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 
> Family) USB2 EHCI Controller #1 [8086:3a3a]
> 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge 
> [8086:244e] (rev 90)
> 00:1f.0 ISA bridge [0601]: Intel Corporation 82801JIB (ICH10) LPC 
> Interface Controller [8086:3a18]
> 01:03.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. 
> [AMD/ATI] ES1000 [1002:515e] (rev 02)
> 01:04.0 System peripheral [0880]: Compaq Computer Corporation 
> Integrated Lights Out Controller [0e11:b203] (rev 03)
> 01:04.2 System peripheral [0880]: Compaq Computer Corporation 
> Integrated Lights Out  Processor [0e11:b204] (rev 03)
> 01:04.4 USB controller [0c03]: Hewlett-Packard Company Integrated 
> Lights-Out Standard Virtual USB Controller [103c:3300]
> 01:04.6 IPMI SMIC interface [0c07]: Hewlett-Packard Company Integrated 
> Lights-Out Standard KCS Interface [103c:3302]
> 02:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II 
> BCM57711E 10-Gigabit PCIe [14e4:1650]
> 02:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II 
> BCM57711E 10-Gigabit PCIe [14e4:1650]
> 06:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II 
> BCM5709S Gigabit Ethernet [14e4:163a] (rev 20)
> 06:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme II 
> BCM5709S Gigabit Ethernet [14e4:163a] (rev 20)
> 0c:00.0 RAID bus controller [0104]: Hewlett-Packard Company Smart 
> Array G6 controllers [103c:323a] (rev 01)
>
> -----------
>
> lsmod | grep bnx2
>
> bnx2i                  52167  0
> libiscsi               49836  8 
> be2iscsi,cxgb4i,cxgb3i,libcxgbi,ib_iser,iscsi_tcp,libiscsi_tcp,bnx2i
> scsi_transport_iscsi    84241  8 
> be2iscsi,libcxgbi,ib_iser,iscsi_tcp,bnx2i,libiscsi
> cnic                   72983  1 bnx2i
> bnx2                  199384  0
> bnx2x                1445902  0
> libcrc32c               1246  1 bnx2x
> mdio                    4769  2 cxgb3,bnx2x
>
> --------
>
> dmesg | grep bnx2
>
> bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit Ethernet Driver 
> bnx2x 1.710.39 ($DateTime: 2014/06/02 04:31:55 $)
> bnx2x 0000:02:00.0: msix capability found
> bnx2x 0000:02:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
> bnx2x 0000:02:00.0: setting latency timer to 64
> bnx2x 0000:02:00.0: part number 412F4E-0-0-0
> bnx2x 0000:02:00.0: irq 53 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 54 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 55 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 56 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 57 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 58 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 59 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 60 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 61 for MSI/MSI-X
> bnx2x 0000:02:00.0: irq 62 for MSI/MSI-X
> bnx2x 0000:02:00.1: msix capability found
> bnx2x 0000:02:00.1: PCI INT B -> GSI 40 (level, low) -> IRQ 40
> bnx2x 0000:02:00.1: setting latency timer to 64
> bnx2x 0000:02:00.1: part number 412F4E-0-0-0
> bnx2x 0000:02:00.1: irq 63 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 64 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 65 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 66 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 67 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 68 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 69 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 70 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 71 for MSI/MSI-X
> bnx2x 0000:02:00.1: irq 72 for MSI/MSI-X
> bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.2.4g (Nov 
> 07, 2013)
> bnx2 0000:06:00.0: PCI INT A -> GSI 30 (level, low) -> IRQ 30
> bnx2 0000:06:00.0: setting latency timer to 64
> bnx2 0000:06:00.0: eth2: Broadcom NetXtreme II BCM5709 1000Base-SX 
> (C0) PCI Express found at mem f8000000, IRQ 30, node addr 
> 1c:c1:de:7b:71:30
> bnx2 0000:06:00.1: PCI INT B -> GSI 37 (level, low) -> IRQ 37
> bnx2 0000:06:00.1: setting latency timer to 64
> bnx2 0000:06:00.1: eth3: Broadcom NetXtreme II BCM5709 1000Base-SX 
> (C0) PCI Express found at mem f6000000, IRQ 37, node addr 
> 1c:c1:de:7b:71:32
> bnx2x 0000:02:00.0: eth0: using MSI-X  IRQs: sp 53  fp[0] 55 ... fp[7] 62
> bnx2 0000:06:00.0: irq 73 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 74 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 75 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 76 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 77 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 78 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 79 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 80 for MSI/MSI-X
> bnx2 0000:06:00.0: irq 81 for MSI/MSI-X
> bnx2 0000:06:00.0: eth2: using MSIX
> bnx2 0000:06:00.0: eth2: NIC SerDes Link is Up, 1000 Mbps full duplex, 
> receive & transmit flow control ON
> bnx2x 0000:02:00.0: eth0: NIC Link is Up, 1000 Mbps full duplex, Flow 
> control: ON - receive & transmit
> bnx2x 0000:02:00.1: eth1: using MSI-X  IRQs: sp 63  fp[0] 65 ... fp[7] 72
> bnx2 0000:06:00.0: eth2: NIC SerDes Link is Down
> bnx2 0000:06:00.0: eth2: NIC SerDes Link is Up, 1000 Mbps full duplex, 
> receive & transmit flow control ON
> bnx2x 0000:02:00.1: eth1: NIC Link is Up, 1000 Mbps full duplex, Flow 
> control: ON - receive & transmit
> bnx2 0000:06:00.1: irq 82 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 83 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 84 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 85 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 86 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 87 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 88 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 89 for MSI/MSI-X
> bnx2 0000:06:00.1: irq 90 for MSI/MSI-X
> bnx2 0000:06:00.1: eth3: using MSIX
> bnx2 0000:06:00.1: eth3: NIC SerDes Link is Up, 1000 Mbps full duplex, 
> receive & transmit flow control ON
> bnx2x 0000:02:00.0: eth0: Added CNIC device
> bnx2x 0000:02:00.1: eth1: Added CNIC device
> bnx2 0000:06:00.0: eth2: Added CNIC device
> bnx2 0000:06:00.1: eth3: Added CNIC device
> Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.10.31b (Mar 06, 2014)
> iscsi: registered transport (bnx2i)
> bnx2i [06:00.01]: ISCSI_INIT passed
> bnx2i [06:00.00]: ISCSI_INIT passed
> bnx2i: dev eth1 does not support iSCSI
> bnx2i init: (eth1) hba ffff880493155640 init failed
> bnx2i: eth1 free_hba done after 0 retries
> bnx2i: dev eth0 does not support iSCSI
> bnx2i init: (eth0) hba ffff880493155640 init failed
> bnx2i: eth0 free_hba done after 0 retries
> Loading kernel module for a network device with CAP_SYS_MODULE 
> (deprecated).  Use CAP_NET_ADMIN and alias netdev-bnx2x instead
>
> -------
>
> There is no option on bios to activate offloading, and i had it 
> working on vmware esxi.
>
> Anyway, i think i found the solution, the mac address of my eth3 nic 
> is 1c:c1:de:7b:71:32, so from what i read it seems that that 
> corresponding iscsi device mac address is 1c:c1:de:7b:71:33, so adding 
> the mac addres and ip to /var/lib/iscsi/ifaces/eth3 and change from 
> tcp to bnx2i and then
> remove and re add it in "iSCSI Mutipathing" tab works
>
> iface.hwaddress = 1c:c1:de:7b:71:33
> iface.ipaddress = 192.168.12.125
> iface.transport_name = bnx2i

Currently "iscsi multipath bond" feature is not associating the iscsi 
iface created to a system device. This is why you need to fill the iface 
file with the iface.hwaddress. The bz#1112861 is tracking this issue.

When using "iscsi multipath bond", vdsm will create the iscsi ifaces 
with iface.transport_name = 'tcp'. This is hard coded: 
http://gerrit.ovirt.org/gitweb?p=vdsm.git;a=blob;f=vdsm/storage/iscsi.py;hb=HEAD#l323. 
Please open a Bugzilla ticket if you think this should be improved.

You can overwrite the iscsi iface files, but if using ovirt-node, the 
files will not persist over system reboot.

> I see that already exists a 
> /var/lib/iscsi/ifaces/bnx2i.1c:c1:de:7b:71:33, so if i create 
> /etc/sysconfig/network-script/ifcfg-bnx2i.1c:c1:de:7b:71:33 i should 
> be able to add it in Ovirt GUI to a logical network and then use it in 
> "iSCSI Mutipathing" tab? I'm going to try this...
>
> Best regards,
> Ricardo Esteves.
>
> On 08-10-2014 09:24, Federico Simoncelli wrote:
>> ----- Original Message -----
>>> From: "Ricardo Esteves" <ricardo.m.esteves at gmail.com>
>>> To: "Federico Simoncelli" <fsimonce at redhat.com>
>>> Cc: users at ovirt.org
>>> Sent: Wednesday, October 8, 2014 1:32:51 AM
>>> Subject: Re: [ovirt-users] oVirt 3.2 - iSCSI offload (broadcom - bnx2i)
>>>
>>> Hi, here it goes:
>>>
>>> ethtool -i eth3
>>>
>>> driver: bnx2
>>> version: 2.2.4g
>>> firmware-version: bc 5.2.3
>>> bus-info: 0000:06:00.1
>>> supports-statistics: yes
>>> supports-test: yes
>>> supports-eeprom-access: yes
>>> supports-register-dump: yes
>>> supports-priv-flags: no
>> Thanks, can you also add the output of:
>>
>>   # lspci -nn
>>
>> I'd have expected the driver to be bnx2i (bnx2 is a regular ethernet 
>> driver,
>> no offloading).
>>
>> Can you also check if you have the bnx2i driver loaded?
>>
>>   # lsmod | grep bnx2
>>
>> and eventually if there's any bnx2 related message in /var/log/messages
>>
>> Check also the adapter bios (at boot time) if by any chance you have 
>> to enable
>> the offloading there first (check the specific manual too).
>>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users