[ovirt-users] IO errors when adding new disk on iSCSI storage

Fri May 16 02:45:19 UTC 2014

A colleague of mine has run into issues doing Linux (EL6.5) iSCSI over
bonded interfaces.  The solution he found was to abandon bonding and
instead use multiple interfaces and multipath on the iscsi initiators.  I
can't provide details yet as this is all second hand knowledge.  The errors
you posted are almost identical to what he faced.

- Trey
On May 13, 2014 3:10 AM, "Morten A. Middelthon" <morten at flipp.net> wrote:

> Hi,
>
> I have a new oVirt setup which is giving me some headache. I have one
> administration server running on CentOS 6.5, and two hosts also running
> CentOS 6.5. Storage is running on both NFS and iSCSI, but iSCSI is the
> preferred storage, running on a HP LeftHand server.
>
> oVirt Engine Version: 3.4.0-1.el6
>
> Administration server and hosts are all updated with latest packages both
> for CentOS and oVirt.
>
> Both hosts are attached to a dedicated storage network with bonded
> interfaces:
>
> # ifconfig bond1
> bond1     Link encap:Ethernet  HWaddr AC:16:2D:8B:90:5E
>           inet addr:192.168.40.160  Bcast:192.168.40.255 Mask:255.255.255.0
>           inet6 addr: fe80::ae16:2dff:fe8b:905e/64 Scope:Link
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:4149464 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5590278 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:5387975600 (5.0 GiB)  TX bytes:6981018034 (6.5 GiB)
>
> On the hosts the LUN is seen as:
>
> scsi3 : iSCSI Initiator over TCP/IP
>  connection1:0: detected conn error (1020)
>  scsi 3:0:0:0: Direct-Access     LEFTHAND iSCSIDisk        a500 PQ: 0
> ANSI: 5
>  sd 3:0:0:0: Attached scsi generic sg2 type 0
>  sd 3:0:0:0: [sdb] 8589934592 512-byte logical blocks: (4.39 TB/4.00 TiB)
>  sd 3:0:0:0: [sdb] Write Protect is off
>  sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
>  sd 3:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't
> support DPO or FUA
>   sdb: unknown partition table
>   sd 3:0:0:0: [sdb] Attached SCSI disk
>
> The iSCSI LUN I'm using is 4TB large, with 2.5TB free, according to the
> oVirt admin interface:
>
> Size:   4095 GB
> Available:      2551 GB
> Used:   1544 GB
> Allocated:      1540 GB
> Over Allocation Ratio:  0%
>
> Over to the problem:
>
> On the virtual machine I'm testing on, I'm adding a new disk on the iSCSI
> storage server. The virtual machine is Debian 7.4 amd64. In this case 20GB
> large with thin provisioning. Then on the virtual machine I create a new
> logical volume group and disk:
>
> Create a new partition with type 8e (Linux LVM):
> fdisk /dev/vdc
>
> Setup LVM:
> pvcreate /dev/vdc1
> vgcreate VGTEST /dev/vdc1
> lvcreate -n lv_test -l 100%FREE VGTEST
> mkfs.ext4 /dev/mapper/VGTEST-lv_test
> mkdir /mnt/test
> mount /dev/mapper/VGTEST-lv_test /mnt/test
>
> Then I try writing to the new disk with f.ex dd:
> dd if=/dev/zero of=/mnt/test/storfil bs=64k count=20480
>
> Almost immediately oVirt will pause the machine with the following error
> message:
>
> VM INT-KVM-SSM02 has paused due to no Storage space error.
>
> dmesg on the virtual machine gives to following errors:
>
> [353290.564507] Buffer I/O error on device dm-7, logical block 294774
> [353290.564509] Buffer I/O error on device dm-7, logical block 294775
> [353290.564511] EXT4-fs warning (device dm-7): ext4_end_bio:250: I/O error
> writing to inode 12 (offset 1039114240 size 520192 starting block 294650)
> [353290.564514] end_request: I/O error, dev vdc, sector 2362304
> [353290.564516] Buffer I/O error on device dm-7, logical block 294776
> [353290.564518] Buffer I/O error on device dm-7, logical block 294777
> [353290.564519] Buffer I/O error on device dm-7, logical block 294778
> [353290.564521] Buffer I/O error on device dm-7, logical block 294779
> [353290.564522] Buffer I/O error on device dm-7, logical block 294780
> [353290.564524] Buffer I/O error on device dm-7, logical block 294781
> [353290.564525] Buffer I/O error on device dm-7, logical block 294782
> [353290.564744] Buffer I/O error on device dm-7, logical block 294910
> [353290.564746] Buffer I/O error on device dm-7, logical block 294911
> [353290.564748] EXT4-fs warning (device dm-7): ext4_end_bio:250: I/O error
> writing to inode 12 (offset 1040154624 size 32768 starting block 294904)
> [353291.035703] EXT4-fs (dm-7): This should not happen!! Data will be lost
> [353291.035705]
> [353291.036280] JBD2: Detected IO errors while flushing file data on dm-7-8
> [353291.036506] end_request: I/O error, dev vdc, sector 17043640
> [353291.036509] end_request: I/O error, dev vdc, sector 17043640
> [353291.036518] Aborting journal on device dm-7-8.
> [353291.036734] end_request: I/O error, dev vdc, sector 17043456
> [353291.036737] Buffer I/O error on device dm-7, logical block 2129920
> [353291.036739] lost page write due to I/O error on dm-7
> etc etc
>
> If I try to do large writes on the existing disk on either the iSCSI or
> NFS storage there are no problems at all.
>
> I have attached the engine.log from the administration server, and also
> the vdsm.log from the host
>
> --
> Morten A. Middelthon
> Email: morten at flipp.net
> Phone: +47 907 83 708
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20140515/8ac1fbeb/attachment-0001.html>