On Tue, Jan 31, 2017 at 3:23 PM, Nathanaël Blanchet <blanchet(a)abes.fr&gt; wrote: I'm trying to mitigate inserting a timeout for my SAN devices but I'm not sure of its effectiveness as CentOS 7 behavior of "multipathd -k" and then "show config" seems different from CentOS 6.x In fact my attempt for multipath.conf is this # VDSM REVISION 1.3 # VDSM PRIVATE defaults { polling_interval 5 no_path_retry fail user_friendly_names no flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo 30 max_fds 4096 } # Remove devices entries when overrides section is available. devices { device { # These settings overrides built-in devices settings. It does not apply # to devices without built-in settings (these use the settings in the # "defaults" section), or to devices defined in the "devices" section. # Note: This is not available yet on Fedora 21. For more info see # https://bugzilla.redhat.com/1253799 all_devs yes no_path_retry fail } device { vendor "IBM" product "^1814" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "0" hardware_handler "1 rdac" prio "rdac" failback immediate rr_weight "uniform" no_path_retry "12" } } So I put exactly the default device config for my IBM/1814 device but no_path_retry set to 12. In CentOS 6.x when you do something like this, "show config" gives you the modified entry only for your device section. Instead in CentOS 7.3 it seems I get anyway the default one for IBM/1814 and also the customized one at the end of the output.... Two facts: - before I could reproduce the problem if I selected Maintenance Power Mgmt ---> Restart (tried 3 times with same behavior) Instead if I executed in separate steps Maintenance Power Mgmt --> Stop wait a moment Power Mgmt --> Start I didn't get problems (tried only one time...) With this "new" multipath config (to be confirmed if in effect, how?) I don't get the VM paused problem even with Restart option of Power Mgmt In active host messages I see these ones when the other reboots: Jan 31 16:50:01 ovmsrv06 systemd: Started Session 705 of user root. Jan 31 16:50:01 ovmsrv06 systemd: Starting Session 705 of user root. Jan 31 16:53:47 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sde - rdac checker reports path is up Jan 31 16:53:47 ovmsrv06 multipathd: 8:64: reinstated Jan 31 16:53:47 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: load table [0 41943040 multipath 1 queue_if_no_path 1 rdac 2 1 service-time 0 2 1 8:224 1 65:0 1 service-time 0 2 1 8:64 1 8:160 1] Jan 31 16:53:47 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sdo - rdac checker reports path is ghost Jan 31 16:53:47 ovmsrv06 multipathd: 8:224: reinstated Jan 31 16:53:47 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sdk - rdac checker reports path is up Jan 31 16:53:47 ovmsrv06 multipathd: 8:160: reinstated Jan 31 16:53:47 ovmsrv06 kernel: sd 0:0:1:4: rdac: array Z1_DS4700, ctlr 1, queueing MODE_SELECT command Jan 31 16:53:47 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sdq - rdac checker reports path is ghost Jan 31 16:53:47 ovmsrv06 multipathd: 65:0: reinstated Jan 31 16:53:48 ovmsrv06 kernel: sd 0:0:1:4: rdac: array Z1_DS4700, ctlr 1, MODE_SELECT returned with sense 05/91/36 Jan 31 16:53:48 ovmsrv06 kernel: sd 0:0:1:4: rdac: array Z1_DS4700, ctlr 1, queueing MODE_SELECT command Jan 31 16:53:49 ovmsrv06 kernel: sd 0:0:1:4: rdac: array Z1_DS4700, ctlr 1, MODE_SELECT returned with sense 05/91/36 Jan 31 16:53:49 ovmsrv06 kernel: sd 0:0:1:4: rdac: array Z1_DS4700, ctlr 1, queueing MODE_SELECT command Jan 31 16:53:49 ovmsrv06 kernel: sd 0:0:1:4: rdac: array Z1_DS4700, ctlr 1, MODE_SELECT completed Jan 31 16:53:49 ovmsrv06 kernel: sd 2:0:1:4: rdac: array Z1_DS4700, ctlr 1, queueing MODE_SELECT command Jan 31 16:53:49 ovmsrv06 kernel: sd 2:0:1:4: rdac: array Z1_DS4700, ctlr 1, MODE_SELECT completed Jan 31 16:53:52 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sde - rdac checker reports path is ghost Jan 31 16:53:52 ovmsrv06 multipathd: 8:64: reinstated Jan 31 16:53:52 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: load table [0 41943040 multipath 1 queue_if_no_path 1 rdac 2 1 service-time 0 2 1 8:224 1 65:0 1 service-time 0 2 1 8:64 1 8:160 1] Jan 31 16:53:52 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sdo - rdac checker reports path is up Jan 31 16:53:52 ovmsrv06 multipathd: 8:224: reinstated Jan 31 16:53:52 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sdk - rdac checker reports path is ghost Jan 31 16:53:52 ovmsrv06 multipathd: 8:160: reinstated Jan 31 16:53:52 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: sdq - rdac checker reports path is up Jan 31 16:53:52 ovmsrv06 multipathd: 65:0: reinstated But they are not related to the multipath device dedicated to oVirt storage domain in this case.... What lets me be optimistic seems the difference in these lines: before I got Jan 31 10:27:47 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: load table [0 41943040 multipath 0 1 rdac 2 1 service-time 0 2 1 8:224 1 65:0 1 service-time 0 2 1 8:64 1 8:160 1] now I get Jan 31 16:53:52 ovmsrv06 multipathd: 3600a0b8000299aa80000d08955014098: load table [0 41943040 multipath 1 queue_if_no_path 1 rdac 2 1 service-time 0 2 1 8:224 1 65:0 1 service-time 0 2 1 8:64 1 8:160 1] multipath 0 1 rdac vs multipath 1 queue_if_no_path 1 rdac Any confirmation? Thanks in advance

On Wed, Feb 1, 2017 at 8:39 AM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: I don't think so. Also because reading dm multipath guide at https://access.redhat.com/documentation/en-US/Red_Hat_Enterp rise_Linux/7/html/DM_Multipath/multipath_config_confirm.html It seems that in RH EL 7.3 the "show config" command has this behaviour: " For example, the following command sequence displays the multipath configuration, including the defaults, before exiting the console. # multipathd -k " So the output has to include the default too. Anyway I changed it, see below <<<<< BEGIN OF PARENTHESIS In theory it should be the same on RH EL 6.8 (see https://access.redhat.com/documentation/en-US/Red_Hat_Enterp rise_Linux/6/html/DM_Multipath/multipath_config_confirm.html ) but it is not so for me on a system that is on 6.5, with device-mapper-multipath-0.4.9-93.el6.x86_64 and connected to Netapp In /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf.defaults # vendor "NETAPP" # product "LUN.*" # path_grouping_policy group_by_prio # getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n" # path_selector "round-robin 0" # path_checker tur # features "3 queue_if_no_path pg_init_retries 50" # hardware_handler "0" # prio ontap # failback immediate # rr_weight uniform # rr_min_io 128 # rr_min_io_rq 1 # flush_on_last_del yes # fast_io_fail_tmo 5 # dev_loss_tmo infinity # retain_attached_hw_handler yes # detect_prio yes # reload_readwrite yes # } My customization in multipath.conf, based on Netapp guidelines and my Netapp storage array setup: devices { device { vendor "NETAPP" product "LUN.*" getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n" hardware_handler "1 alua" prio alua } If I run "multipathd show config" I see only 1 entry for NETAPP/LUN vendor/product and it is a merge of default and my custom. device { vendor "NETAPP" product "LUN.*" path_grouping_policy group_by_prio getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n" path_selector "round-robin 0" path_checker tur features "3 queue_if_no_path pg_init_retries 50" hardware_handler "1 alua" prio alua failback immediate rr_weight uniform rr_min_io 128 rr_min_io_rq 1 flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo infinity retain_attached_hw_handler yes detect_prio yes reload_readwrite yes } So this difference confused me when configuring multipath in CentOS 7.3. I have to see when I'm going to update from 6.5 to 6.8 if this changes. <<<<< END OF PARENTHESIS OK. In fact it could be also a ds4700 overall problem.... The LUNs are configured as LNX CLUSTER type, that should be ok in theory, even if this kind of storage was never so supported with Linux. Initially one had to use proprietary IBM kernel modules/drivers. I will see consistency and robustness through testing. I have to do a POC and this is the hw I have and I should at least try to have a working solution for it. I think it is the different behaviour as outlined above. I think you can confirm in another system where some customization has been done too... I think so too. Eventually when possible, if errors repeat with the new config, I'll manage to do stop/start instead of restart I have not understood what you mean exactly here... Can you elaborate? Suppose I have to power off one hypervisor (yum update, pathing, fw update or planned server room maintenance, ...), my workflow is this one all from inside web admin gui: Move running VMs in charge of the host (or delegate to the following step) Put host into maintenance Power Mgmt --> Stop When planned maintenance ha finished Power Mgmt --> Start I should see the host in maintenance Activate Or do you mean I should do anything from the host itself and not the GUI? OK. In the mean time I have applied your suggested config and restarted the 2 nodes. Let we test and see if I find any problems running also some I/O tests. Thanks in the mean time, Gianluca

On Thu, Feb 2, 2017 at 1:11 AM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: I don't think this test is related to the issues you reported earlier. What you test here is how fast ovirt thin provisioning can extend your disk when writing zeros. We don't handle that very well, each extend needs at couple 4-6 seconds to complete before we refresh the lv on the host running the vm, and this is the *best* case. In the worst case it can take much longer. Also what you tested here is how fast you can write to your vm buffer cache, since you are not using direct io. A better way to perform this test is: time dd if=/dev/zero bs=1024k count=1024 of=/home/g.cecchi/testfile oflag=direct This will give you the time to actually write data to storage. If you have a real issue with vms pausing during writes when vm disk has to be extended, you can enlarge the extend chunk, 1GiB by default. To use chunks of 2GiB, set: [irs] volume_utilization_percent = 50 volume_utilization_chunk_mb = 2048 This will extend the drive when free space is less than 1024MiB (volume_utilization_chunk_mb * (100 - volume_utilization_percent) / 100) If this is not enough, you can also use lower volume_utilization_percent, for example, this will extend the disk in 2GiB chunks when free space is bellow 1536MiB: [irs] volume_utilization_percent = 25 volume_utilization_chunk_mb = 2048 This is interesting, we have seen this messages before, but could never detect the flow causing them, are you sure you see this each time you extend your disk? If you can reproduce this, please file a bug. Qemu probably cannot handle disk size on block devices.

On Thu, Feb 2, 2017 at 12:04 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: A single 'dd' rarely saturates a high performance storage. There are better utilities to test ('fio' , 'vdbench' and 'ddpt' for example). It's also testing a very theoretical scenario - very rarely you write zeros and very rarely you write so much sequential IO, and with a fixed block size. So it's almost 'hero numbers'. Depends on your SAN. On thin provisioned one (with potentially inline dedup and compression, such as XtremIO, Pure, Nimble and others) I don't see a great value in thin provisioning. What is your expectation? Is it FC, iSCSI? How many paths? What is the IO scheduler in the VM? Is it using virtio-blk or virtio-SCSI? Y.

On Thu, Feb 2, 2017 at 1:34 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Yes but you should not modify them in vdsm.conf. No, the defaults did not change in the last 3.5 years. In 4.0 we introduced dropin support, and this is the recommend way to perform configuration changes. To change these values, you create a file at /etc/vdsm/vdsm.conf.d/50_my.conf The name of the file does not matter, vdsm will read all files in the vdsm.conf.d directory, sort them by name (this is way you should use 50_ prefix), and apply the changes to the configuration. In this file you put the sections and options you need, like: [irs] volume_utilization_percent = 25 volume_utilization_chunk_mb = 2048 Restart vdsm Using these method, you can provision the same file on all hosts using standard provisioning tools. It is not recommended to modify /etc/vdsm/vdsm.conf. If you do this you will have to manually merge changes from vdsm.conf.rpmnew after upgrading vdsm. Nir

On Thu, Feb 2, 2017 at 4:16 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Yes, this should be a commented out option in /etc/vdsm/vdsm.conf. If you don't have a vdsm.conf file, or the file is empty, you can generate a new file like this: python /usr/lib64/python2.7/site-packages/vdsm/config.py > vdsm.conf.examle

On Thu, Feb 2, 2017 at 4:42 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Indeed, my mistake, we move do lib several versions ago. Sure. This may abort a storage operation if one is running when you restart vdsm, but vdsm is designed so you can restart or kill it safely. For example, if you abort a disk copy in the middle, the operation will fail and the destination disk will be deleted. If you want to avoid such issue, you can put a host to maintenance, but this requires migration of vms to other hosts. Nir

On Thu, Feb 2, 2017 at 6:05 PM, Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Yes, engine will move spm to another host when spm fails, unless you disabled spm role for any other host (see host > spm tab). It is expected the restarting the spm is slower, but we need to see vdsm logs to understand why. It look like the engine discovered that the SPM was down, and reconnected. It is expected that changes in the spm status are detected early and engine is trying to recover the spm, the SPM role is critical in ovirt. Are you sure you did not get any message when restarting the other host? I would expect that engine detect and report a restart of all hosts. If you can reproduce this, restarting vdsm is not detected on engine and not reported in engine even log, please file a bug. Nir

On Thu, Feb 2, 2017 at 10:48 AM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: I got the same on 4.1.1 while testing on iSCSI so I opened a bugzilla now... let's see if we can find anything usefule https://bugzilla.redhat.com/show_bug.cgi?id=1438809 Gianluca

On Thu, Feb 2, 2017 at 10:53 PM, Benjamin Marzinski <bmarzins(a)redhat.com&gt; wrote: Hi Benjamin, thank you very much for the explanations, especially the long one ;-) I tried and confirmed that I has no_path_retry = 4 as expected The regex matching is only for merge, correct? So in your example if in RH EL 7 I put this device { vendor "IBM" product "^1814" no_path_retry 12 } It would not match for merging, but it would match for applying to my device (because it is put at the end of config read backwards). And it would apply only the no_path_retry setting, while all other ones would not be picked from builtin configuration for device, but from defaults in general. So for example it would set path_checker not this way: path_checker "rdac" but this way: path_checker "directio" that is default.. correct?

This is a multi-part message in MIME format. --------------06F287541BC9D8DB37199903 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit I'm also seeing this error using a Dell MD3800i array. The multipath errors shown in our logs are different however. Feb 1 15:11:58 ovirt-node-production2 kernel: dd: sending ioctl 80306d02 to a partition! Feb 1 15:21:01 ovirt-node-production2 multipathd: dm-31: remove map (uevent) Feb 1 15:21:01 ovirt-node-production2 multipathd: dm-31: devmap not registered, can't remove Feb 1 15:21:01 ovirt-node-production2 multipathd: dm-31: remove map (uevent) The dd error seems to happen every time that SPM runs a test. On 01/31/2017 09:23 AM, Nathanaël Blanchet wrote: --------------06F287541BC9D8DB37199903 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>I'm also seeing this error using a Dell MD3800i array.  The multipath errors shown in our logs are different however.</p> <p>Feb  1 15:11:58 ovirt-node-production2 kernel: dd: sending ioctl 80306d02 to a partition!<br> Feb  1 15:21:01 ovirt-node-production2 multipathd: dm-31: remove map (uevent)<br> Feb  1 15:21:01 ovirt-node-production2 multipathd: dm-31: devmap not registered, can't remove<br> Feb  1 15:21:01 ovirt-node-production2 multipathd: dm-31: remove map (uevent)<br> </p> <p>The dd error seems to happen every time that SPM runs a test.<br> </p> <br> <div class="moz-cite-prefix">On 01/31/2017 09:23 AM, Nathanaël Blanchet wrote:<br> </div> <blockquote cite="mid:304eaca8-f962-d994-1607-dbbdd265f0d5@abes.fr" type="cite"> <meta content="text/html; charset=windows-1252" http-equiv="Content-Type"> <p>exactly the same issue by there with FC EMC domain storage...</p> <br> <div class="moz-cite-prefix">Le 31/01/2017 à 15:20, Gianluca Cecchi a écrit :<br> </div> <blockquote cite="mid:CAG2kNCxWKBRLZ43OTXfAVJUPpLd3vNbN1ckw3sq2ZCr=tY+CtQ@mail.gmail.com" type="cite"> <div dir="ltr">Hello, <div>my test environment is composed by 2 old HP blades BL685c G1 (ovmsrv05 and ovmsrv06) and they are connected in a SAN with FC-switches to an old IBM DS4700 storage array.</div> <div>Apart from being old, they seem all ok from an hw point of view.</div> <div>I have configured oVirt 4.0.6 and an FCP storage domain.</div> <div>The hosts are plain CentOS 7.3 servers fully updated.</div> <div>It is not an hosted engine environment: the manager is a vm outside of the cluster.</div> <div>I have configured power mgmt on both and it works good.</div> <div><br> </div> <div>I have at the moment  only one VM for test and it is doing quite nothing.<br> </div> <div><br> </div> <div>Starting point: ovmsrv05 is in maintenance (since about 2 days) and the VM is running on ovmsrv06.</div> <div>I update qemu-kvm package on ovmsrv05 and then I restart it from web admin gui:</div> <div>Power Mgmt --&gt; Restart</div> <div><br> </div> <div>Sequence of events in pane and the problem in subject:</div> <div> <div>Jan 31, 2017 10:29:43 AM Host ovmsrv05 power management was verified successfully.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:29:43 AM Status of host ovmsrv05 was set to Up.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:29:38 AM Executing power management status on Host ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:29:29 AM Activation of host ovmsrv05 initiated by admin(a)internal-authz.&lt;/div&gt; <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:28:05 AM VM ol65 has recovered from paused back to up.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:27:55 AM VM ol65 has been paused due to storage I/O problem.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:27:55 AM VM ol65 has been paused.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:52 AM Host ovmsrv05 was restarted by admin(a)internal-authz.&lt;/div&gt; <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:52 AM Host ovmsrv05 was started by admin(a)internal-authz.&lt;/div&gt; <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:52 AM Power management start of Host ovmsrv05 succeeded.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:50 AM Executing power management status on Host ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:37 AM Executing power management start on Host ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:37 AM Power management start of Host ovmsrv05 initiated.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:37 AM Auto fence for host ovmsrv05 was started.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:37 AM All VMs' status on Non Responsive Host ovmsrv05 were changed to 'Down' by admin@internal-authz</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:36 AM Host ovmsrv05 was stopped by admin(a)internal-authz.&lt;/div&gt; <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:36 AM Power management stop of Host ovmsrv05 succeeded.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:34 AM Executing power management status on Host ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:15 AM Executing power management stop on Host ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:15 AM Power management stop of Host ovmsrv05 initiated.</div> <div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div> <div>Jan 31, 2017 10:25:12 AM Executing power management status on Host ovmsrv05 using Proxy Host ovmsrv06 and Fence Agent ilo:10.4.192.212.</div> </div> <div><br> </div> <div>Watching the timestamps, the culprit seems the reboot time of ovmsrv05 that detects some LUNs in owned state and other ones in unowned</div> <div>Full messages of both hosts here:</div> <div><a moz-do-not-send="true" href="https://drive.google.com/file/d/0BwoPbcrMv8mvekZQT1pjc0NMRlU/v... </div> <div>and</div> <div><a moz-do-not-send="true" href="https://drive.google.com/file/d/0BwoPbcrMv8mvcjBCYVdFZWdXTms/v... </div> <div><br> </div> <div>At this time there are 4 LUNs globally seen by the two hosts but only 1 of them is currently configured as the only storage domain in oVirt cluster.</div> <div><br> </div> <div> <div>[root@ovmsrv05 ~]# multipath -l | grep ^36</div> <div>3600a0b8000299aa80000d08b55014119 dm-5 IBM     ,1814      FAStT </div> <div>3600a0b80002999020000cd3c5501458f dm-3 IBM     ,1814      FAStT </div> <div>3600a0b80002999020000ccf855011198 dm-2 IBM     ,1814      FAStT </div> <div>3600a0b8000299aa80000d08955014098 dm-4 IBM     ,1814      FAStT </div> </div> <div><br> </div> <div>the configured one:</div> <div> <div>[root@ovmsrv05 ~]# multipath -l 3600a0b8000299aa80000d08b55014119</div> <div>3600a0b8000299aa80000d08b55014119 dm-5 IBM     ,1814      FAStT </div> <div>size=4.0T features='0' hwhandler='1 rdac' wp=rw</div> <div>|-+- policy='service-time 0' prio=0 status=active</div> <div>| |- 0:0:1:3 sdl 8:176 active undef running</div> <div>| `- 2:0:1:3 sdp 8:240 active undef running</div> <div>`-+- policy='service-time 0' prio=0 status=enabled</div> <div>  |- 0:0:0:3 sdd 8:48  active undef running</div> <div>  `- 2:0:0:3 sdi 8:128 active undef running</div> </div> <div><br> </div> <div>In mesages of booting node, arounf the problem registered by the storage:</div> <div> <div>[root@ovmsrv05 ~]# grep owned /var/log/messages</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:1: rdac: LUN 1 (RDAC) (owned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:2: rdac: LUN 2 (RDAC) (owned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:3: rdac: LUN 3 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:1: rdac: LUN 1 (RDAC) (owned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:0:4: rdac: LUN 4 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:2: rdac: LUN 2 (RDAC) (owned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:1: rdac: LUN 1 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:3: rdac: LUN 3 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:0:4: rdac: LUN 4 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:2: rdac: LUN 2 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:1:1: rdac: LUN 1 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:3: rdac: LUN 3 (RDAC) (owned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:1:2: rdac: LUN 2 (RDAC) (unowned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 0:0:1:4: rdac: LUN 4 (RDAC) (owned)</div> <div>Jan 31 10:27:38 ovmsrv05 kernel: scsi 2:0:1:3: rdac: LUN 3 (RDAC) (owned)</div> <div>Jan 31 10:27:39 ovmsrv05 kernel: scsi 2:0:1:4: rdac: LUN 4 (RDAC) (owned)</div> </div> <div><br> </div> <div>I don't know exactly the meaning of owned/unowned in the output above..</div> <div>Possibly it detects the 0:0:1:3 and 2:0:1:3 paths (those of the active group) as "owned" and this could have created problems with the active node?</div> <div><br> </div> <div>On active node strangely I don't loose all the paths, but the VM has been paused anyway</div> <div><br> </div> <div> <div>[root@ovmsrv06 log]# grep "remaining active path" /var/log/messages </div> <div>Jan 31 10:27:48 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 3</div> <div>Jan 31 10:27:49 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 2</div> <div>Jan 31 10:27:56 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 3</div> <div>Jan 31 10:27:56 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 2</div> <div>Jan 31 10:27:56 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 1</div> <div>Jan 31 10:27:57 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 2</div> <div>Jan 31 10:28:01 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 3</div> <div>Jan 31 10:28:01 ovmsrv06 multipathd: 3600a0b8000299aa80000d08b55014119: remaining active paths: 4</div> </div> <div><br> </div> <div>I'm not an expert of this storage array in particular, and of the rdac hardware handler in general.</div> <div><br> </div> <div>What I see is that multipath.conf on both nodes:</div> <div><br> </div> <div> <div># VDSM REVISION 1.3</div> <div><br> </div> <div>defaults {</div> <div>    polling_interval            5</div> <div>    no_path_retry               fail</div> <div>    user_friendly_names         no</div> <div>    flush_on_last_del           yes</div> <div>    fast_io_fail_tmo            5</div> <div>    dev_loss_tmo                30</div> <div>    max_fds                     4096</div> <div>}</div> <div><br> </div> <div><br> </div> <div>devices {</div> <div>    device {</div> <div>        # These settings overrides built-in devices settings. It does not apply</div> <div>        # to devices without built-in settings (these use the settings in the</div> <div>        # "defaults" section), or to devices defined in the "devices" section.</div> <div>        # Note: This is not available yet on Fedora 21. For more info see</div> <div>        # <a moz-do-not-send="true" href="https://bugzilla.redhat.com/1253799">https://bugzilla.... <div>        all_devs                yes</div> <div>        no_path_retry           fail</div> <div>    }</div> <div>}</div> </div> <div><br> </div> <div><br> </div> <div> <div>beginning of /proc/scsi/scsi</div> <div><br> </div> <div>[root@ovmsrv06 ~]# cat /proc/scsi/scsi </div> <div>Attached devices:</div> <div>Host: scsi1 Channel: 01 Id: 00 Lun: 00</div> <div>  Vendor: HP       Model: LOGICAL VOLUME   Rev: 1.86</div> <div>  Type:   Direct-Access                    ANSI  SCSI revision: 05</div> <div>Host: scsi0 Channel: 00 Id: 00 Lun: 01</div> <div>  Vendor: IBM      Model: 1814      FAStT  Rev: 0916</div> <div>  Type:   Direct-Access                    ANSI  SCSI revision: 05</div> </div> <div>...</div> <div><br> </div> <div>To get default acquired config for this storage:</div> <div> <div><br> </div> <div>multpathd -k</div> <div>&gt; show config</div> <div><br> </div> <div>I can see:</div> <div><br> </div> <div>        device {</div> <div>                vendor "IBM"</div> <div>                product "^1814"</div> <div>                product_blacklist "Universal Xport"</div> <div>                path_grouping_policy "group_by_prio"</div> <div>                path_checker "rdac"</div> <div>                features "0"</div> <div>                hardware_handler "1 rdac"</div> <div>                prio "rdac"</div> <div>                failback immediate</div> <div>                rr_weight "uniform"</div> <div>                no_path_retry "fail"</div> <div>        }</div> <div><br> </div> <div><br> </div> <div>and</div> <div><br> </div> <div>defaults {</div> <div>        verbosity 2</div> <div>        polling_interval 5</div> <div>        max_polling_interval 20</div> <div>        reassign_maps "yes"</div> <div>        multipath_dir "/lib64/multipath"</div> <div>        path_selector "service-time 0"</div> <div>        path_grouping_policy "failover"</div> <div>        uid_attribute "ID_SERIAL"</div> <div>        prio "const"</div> <div>        prio_args ""</div> <div>        features "0"</div> <div>        path_checker "directio"</div> <div>        alias_prefix "mpath"</div> <div>        failback "manual"</div> <div>        rr_min_io 1000</div> <div>        rr_min_io_rq 1</div> <div>        max_fds 4096</div> <div>        rr_weight "uniform"</div> <div>        no_path_retry "fail"</div> <div>        queue_without_daemon "no"</div> <div>        flush_on_last_del "yes"</div> <div>        user_friendly_names "no"</div> <div>        fast_io_fail_tmo 5</div> <div>        dev_loss_tmo 30</div> <div>        bindings_file "/etc/multipath/bindings"</div> <div>        wwids_file /etc/multipath/wwids</div> <div>        log_checker_err always</div> <div>        find_multipaths no</div> <div>        retain_attached_hw_handler no</div> <div>        detect_prio no</div> <div>        hw_str_match no</div> <div>        force_sync no</div> <div>        deferred_remove no</div> <div>        ignore_new_boot_devs no</div> <div>        skip_kpartx no</div> <div>        config_dir "/etc/multipath/conf.d"</div> <div>        delay_watch_checks no</div> <div>        delay_wait_checks no</div> <div>        retrigger_tries 3</div> <div>        retrigger_delay 10</div> <div>        missing_uev_wait_timeout 30</div> <div>        new_bindings_in_boot no</div> <div>}</div> <div><br> </div> </div> <div>Any hint on how to tune multipath.conf so that a powering on server doesn't create problems to running VMs?</div> <div><br> </div> <div>Thanks in advance,</div> <div>Gianluca</div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://... </pre> </blockquote> <br> <pre class="moz-signature" cols="72">-- Nathanaël Blanchet Supervision réseau Pôle Infrastrutures Informatiques 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:blanchet@abes.fr">blanchet@abes.fr</a> </pre> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://... </pre> </blockquote> <br> </body> </html> --------------06F287541BC9D8DB37199903--