From: "Dan Kenigsberg" <danken(a)redhat.com&gt; To: "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com&gt;, nsoffer(a)redhat.com Cc: "users" <users(a)ovirt.org&gt;, ykaplan(a)redhat.com Sent: Monday, January 26, 2015 2:09:23 PM Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf? On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote: > Hello, > on my all-in-one installation @home I had 3.5.0 with F20. > Today I updated to 3.5.1. > > it seems it modified /etc/multipath.conf preventing me from using my second > disk at all... > > My system has internal ssd disk (sda) for OS and one local storage domain > and another disk (sdb) with some partitions (on one of them there is also > another local storage domain). > > At reboot I was put in emergency boot because partitions at sdb disk could > not be mounted (they were busy). > it took me some time to understand that the problem was due to sdb gone > managed as multipath device and so busy for partitions to be mounted. > > Here you can find how multipath became after update and reboot > https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sha... > > No device-mapper-multipath update in yum.log > > Also it seems that after changing it, it was then reverted at boot again (I > don't know if the responsible was initrd/dracut or vdsmd) so in the mean > time the only thing I could do was to make the file immutable with > > chattr +i /etc/multipath.conf The "supported" method of achieving this is to place "# RHEV PRIVATE" in the second line of your hand-modified multipath.conf I do not understand why this has happened only after upgrade to 3.5.1 - 3.5.0's should have reverted you multipath.conf just as well during each vdsm startup. The good thing is that this annoying behavior has been dropped from the master branch, so that 3.6 is not going to have it. Vdsm is not to mess with other services config file while it is running. The logic moved to `vdsm-tool configure` > > and so I was able to reboot and verify that my partitions on sdb were ok > and I was able to mount them (for safe I also ran an fsck against them) > > Update ran around 19:20 and finished at 19:34 > here the log in gzip format > https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sha... > > Reboot was done around 21:10-21:14 > > Here my /var/log/messages in gzip format, where you can see latest days. > https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sha... > > > Any suggestion appreciated. > > Current multipath.conf (where I also commented out the getuid_callout that > is not used anymore): > > [root@tekkaman setup]# cat /etc/multipath.conf > # RHEV REVISION 1.1 > > blacklist { > devnode "^(sda|sdb)[0-9]*" > }

> > defaults { > polling_interval 5 > #getuid_callout "/usr/lib/udev/scsi_id --whitelisted > --replace-whitespace --device=/dev/%n" > no_path_retry fail > user_friendly_names no > flush_on_last_del yes > fast_io_fail_tmo 5 > dev_loss_tmo 30 > max_fds 4096 > }

----- Original Message ----- > ----- Original Message ----- > > From: "Dan Kenigsberg" <danken(a)redhat.com&gt; > > To: "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com&gt;, nsoffer(a)redhat.com > > Cc: "users" <users(a)ovirt.org&gt;, ykaplan(a)redhat.com > > Sent: Monday, January 26, 2015 2:09:23 PM > > Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf? > > > > On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote: > > > Hello, > > > on my all-in-one installation @home I had 3.5.0 with F20. > > > Today I updated to 3.5.1. > > > > > > it seems it modified /etc/multipath.conf preventing me from using my > > > second > > > disk at all... > > > > > > My system has internal ssd disk (sda) for OS and one local storage domain > > > and another disk (sdb) with some partitions (on one of them there is also > > > another local storage domain). > > > > > > At reboot I was put in emergency boot because partitions at sdb disk > > > could > > > not be mounted (they were busy). > > > it took me some time to understand that the problem was due to sdb gone > > > managed as multipath device and so busy for partitions to be mounted. > > > > > > Here you can find how multipath became after update and reboot > > > https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sha... > > > > > > No device-mapper-multipath update in yum.log > > > > > > Also it seems that after changing it, it was then reverted at boot again > > > (I > > > don't know if the responsible was initrd/dracut or vdsmd) so in the mean > > > time the only thing I could do was to make the file immutable with > > > > > > chattr +i /etc/multipath.conf > > > > The "supported" method of achieving this is to place "# RHEV PRIVATE" in > > the second line of your hand-modified multipath.conf > > > > I do not understand why this has happened only after upgrade to 3.5.1 - > > 3.5.0's should have reverted you multipath.conf just as well during each > > vdsm startup. > > > > The good thing is that this annoying behavior has been dropped from the > > master branch, so that 3.6 is not going to have it. Vdsm is not to mess > > with other services config file while it is running. The logic moved to > > `vdsm-tool configure` > > > > > > > > and so I was able to reboot and verify that my partitions on sdb were ok > > > and I was able to mount them (for safe I also ran an fsck against them) > > > > > > Update ran around 19:20 and finished at 19:34 > > > here the log in gzip format > > > https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sha... > > > > > > Reboot was done around 21:10-21:14 > > > > > > Here my /var/log/messages in gzip format, where you can see latest days. > > > https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sha... > > > > > > > > > Any suggestion appreciated. > > > > > > Current multipath.conf (where I also commented out the getuid_callout > > > that > > > is not used anymore): > > > > > > [root@tekkaman setup]# cat /etc/multipath.conf > > > # RHEV REVISION 1.1 > > > > > > blacklist { > > > devnode "^(sda|sdb)[0-9]*" > > > } > > > I think what happened is: > > 1. 3.5.1 had new multipath version > 2. So vdsm upgraded the local file > 3. blacklist above was removed > (it should exists in /etc/multipath.bak) > > To prevent local changes, you have to mark the file as private > as Dan suggests. > > Seems to be related to the find_multiapth = "yes" bug: > https://bugzilla.redhat.com/show_bug.cgi?id=1173290 The symptoms above sound exactly liek this issue. When find_multipahts is no (the default when the directive is not present) I sthat all non-blacklisted devices are tried to get claimed, and this happened above. Blacklisting the devices works, or adding "find_mutlipaths yes" should also work, because in that case only device which have more than one path (or are explicitly named) will be claimed by multipath.

My 2ct. - fabian > Ben, can you confirm that this is the same issue?

> > > > > > > defaults { > > > polling_interval 5 > > > #getuid_callout "/usr/lib/udev/scsi_id --whitelisted > > > --replace-whitespace --device=/dev/%n" > > > no_path_retry fail > > > user_friendly_names no > > > flush_on_last_del yes > > > fast_io_fail_tmo 5 > > > dev_loss_tmo 30 > > > max_fds 4096 > > > } > > > > Regards, > Nir > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users >

From: "Gianluca Cecchi" <gianluca.cecchi(a)gmail.com&gt; To: "Nir Soffer" <nsoffer(a)redhat.com&gt; Cc: "Dan Kenigsberg" <danken(a)redhat.com&gt;, "users" <users(a)ovirt.org&gt;, "Yeela Kaplan" <ykaplan(a)redhat.com&gt;, "Benjamin Marzinski" <bmarzins(a)redhat.com&gt; Sent: Monday, January 26, 2015 8:46:52 PM Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf? On Mon, Jan 26, 2015 at 1:37 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > > > > Any suggestion appreciated. > > > > > > Current multipath.conf (where I also commented out the getuid_callout > that > > > is not used anymore): > > > > > > [root@tekkaman setup]# cat /etc/multipath.conf > > > # RHEV REVISION 1.1 > > > > > > blacklist { > > > devnode "^(sda|sdb)[0-9]*" > > > } > > > I think what happened is: > > 1. 3.5.1 had new multipath version > what do you mean with "new multipath version"?

I currently have device-mapper-multipath-0.4.9-56.fc20.x86_64 The system came from f18 to f19 and then to f20 via fedup in both cases In my yum.log files I see this about January 2013 when I was in Fedora 18 Jan 07 00:11:44 Installed: device-mapper-multipath-0.4.9-36.fc18.x86_64 I then upgraded to f19 on 24th November 2013 and device-mapper-multipath was the one pushed during the fedup update: device-mapper-multipath-libs-0.4.9-51.fc19.x86_64 Then on 12th of November 2014 I passed form f19 to f20 and fedup pushed device-mapper-multipath-0.4.9-56.fc20.x86_64 that is my current version At that time I also passed from oVirt 3.4.4 to 3.5.0. And I didn't register any problem with my internal disk It was sufficient to keep inside blacklist { devnode "sd[a-b]" } At the head of the file I only had: # RHEV REVISION 1.0

No reference about # RHEV PRIVATE But right now that I'm writing I notice that my rule for blacklist after migration to 3.5.1 was devnode "^(sda|sdb)[0-9]*" probably blacklists only partitions and not the full disks.... ;-) So I'm going to check with the old blacklist option and/or the PRIVATE label as suggested by Dan... Probably passing from RHEV REVISION 1.0 to 1.1 the original blacklist part was thrown away... 2. So vdsm upgraded the local file > 3. blacklist above was removed > (it should exists in /etc/multipath.bak) > > It seems it didn't generate any multipath.conf.bak ...