4:37 a.m.
On Mon, Jan 16, 2017 at 7:19 PM, Mark Greenall
<m.greenall(a)iontrading.com> wrote:
Hi,
To try and get a baseline here I've reverted most of the changes we've made and
am running the host with just the following iSCSI related configuration settings. The
tweaks had been made over time to try and alleviate several storage related problems, but
it's possible that fixes in Ovirt (we've gradually gone from early 3.x to 4.0.6)
make them redundant now and they simply compound the problem. I'll start with these
configuration settings and then move onto trying the vdsm patch.
/etc/multipath.conf (note: polling_interval and max_fds would not get accepted in the
devices section. I think they are for default only):
Write, my error.
# VDSM REVISION 1.3
# VDSM PRIVATE
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^sda$"
}
defaults {
deferred_remove yes
dev_loss_tmo 30
fast_io_fail_tmo 5
flush_on_last_del yes
max_fds 4096
You can bump the number of fds here if this is needed.
no_path_retry fail
polling_interval 5
user_friendly_names no
}
devices {
device {
vendor "EQLOGIC"
product "100E-00"
# Ovirt defaults
deferred_remove yes
dev_loss_tmo 30
fast_io_fail_tmo 5
flush_on_last_del yes
# polling_interval 5
user_friendly_names no
# Local settings
# max_fds 8192
path_checker tur
path_grouping_policy multibus
path_selector "round-robin 0"
# Use 4 retries will provide additional 20 seconds gracetime when no
# path is available before the device is disabled. (assuming 5 seconds
# polling interval). This may prevent vms from pausing when there is
# short outage on the storage server or network.
no_path_retry 4
}
device {
# These settings overrides built-in devices settings. It does not apply
# to devices without built-in settings (these use the settings in the
# "defaults" section), or to devices defined in the "devices"
section.
all_devs yes
no_path_retry fail
}
}
/etc/iscsi/iscsid.conf default apart from:
node.session.initial_login_retry_max = 12
node.session.cmds_max = 1024
node.session.queue_depth = 128
node.startup = manual
node.session.iscsi.FastAbort = No
I don't know about these options, I would try to defaults first, unless
you can explain why they are needed.
The following settings have been commented out / removed:
/etc/sysctl.conf:
# For more information, see sysctl.conf(5) and sysctl.d(5).
# Prevent ARP Flux for multiple NICs on the same subnet:
#net.ipv4.conf.all.arp_ignore = 1
#net.ipv4.conf.all.arp_announce = 2
# Loosen RP Filter to alow multiple iSCSI connections
#net.ipv4.conf.all.rp_filter = 2
You need these if you are connecting to to addresses on the same subnet.
Vdsm will do this automatically if needed, if this is configured properly on
the engine side. Unfortunately I don't know to configure this on the
engine side,
but maybe other users using same configuration know.
/lib/udev/rules.d:
# Various Settings for Dell Equallogic disks based on Dell Optimizing SAN Environment for
Linux Guide
#
# Modify disk scheduler mode to noop
#ACTION=="add|change", SUBSYSTEM=="block",
ATTRS{vendor}=="EQLOGIC", RUN+="/bin/sh -c 'echo noop >
/sys/${DEVPATH}/queue/scheduler'"
# Modify disk timeout value to 60 seconds
#ACTION!="remove", SUBSYSTEM=="block",
ATTRS{vendor}=="EQLOGIC", RUN+="/bin/sh -c 'echo 60 >
/sys/%p/device/timeout'"
# Modify read ahead value to 1024
#ACTION!="remove", SUBSYSTEM=="block",
ATTRS{vendor}=="EQLOGIC", RUN+="/bin/sh -c 'echo 1024 >
/sys/${DEVPATH}/bdi/read_ahead_kb'"
I've also removed our defined iSCSI interfaces and have simply left the Ovirt
'default'
The default will probably use only single path for each device, unless you
configure engine to use both nics.
Rebooted and 'Activated' host:
16:09 - Host Activated
16:10 - Non Operational saying it can't access storage domain 'Unknown'
This means a pv is not accessible, smell like connectivity issue with storage.
16:12 - Host Activated again
16:12 - Host not responding goes 'Connecting'
16:15 - Can't access ALL the storage Domains. Host goes Non Operational again
Do you mean it cannot access any storage domain, or it can access only some?
16:17 - Host Activated again
16:18 - Can't access ALL the storage Domains. Host goes Non Operational again
16:20 - Host Autorecovers and goes Activating again
That cycle repeated until I started getting VDSM timeout messages and the constant LVM
processes and high CPU load. @16:30 I rebooted the host and set the status to
maintenance.
Second host Activation attempt just resulted in the same cycle as above. Host now
doesn't come online at all.
Do you have logs from this session?
Next step will be to try the vdsm patch.
Mark