Update to 3.5.1 scrambled multipath.conf?
Hello, on my all-in-one installation @home I had 3.5.0 with F20. Today I updated to 3.5.1. it seems it modified /etc/multipath.conf preventing me from using my second disk at all... My system has internal ssd disk (sda) for OS and one local storage domain and another disk (sdb) with some partitions (on one of them there is also another local storage domain). At reboot I was put in emergency boot because partitions at sdb disk could not be mounted (they were busy). it took me some time to understand that the problem was due to sdb gone managed as multipath device and so busy for partitions to be mounted. Here you can find how multipath became after update and reboot https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharin... No device-mapper-multipath update in yum.log Also it seems that after changing it, it was then reverted at boot again (I don't know if the responsible was initrd/dracut or vdsmd) so in the mean time the only thing I could do was to make the file immutable with chattr +i /etc/multipath.conf and so I was able to reboot and verify that my partitions on sdb were ok and I was able to mount them (for safe I also ran an fsck against them) Update ran around 19:20 and finished at 19:34 here the log in gzip format https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharin... Reboot was done around 21:10-21:14 Here my /var/log/messages in gzip format, where you can see latest days. https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharin... Any suggestion appreciated. Current multipath.conf (where I also commented out the getuid_callout that is not used anymore): [root@tekkaman setup]# cat /etc/multipath.conf # RHEV REVISION 1.1 blacklist { devnode "^(sda|sdb)[0-9]*" } defaults { polling_interval 5 #getuid_callout "/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n" no_path_retry fail user_friendly_names no flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo 30 max_fds 4096 } Gianluca
On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
Hello, on my all-in-one installation @home I had 3.5.0 with F20. Today I updated to 3.5.1.
it seems it modified /etc/multipath.conf preventing me from using my second disk at all...
My system has internal ssd disk (sda) for OS and one local storage domain and another disk (sdb) with some partitions (on one of them there is also another local storage domain).
At reboot I was put in emergency boot because partitions at sdb disk could not be mounted (they were busy). it took me some time to understand that the problem was due to sdb gone managed as multipath device and so busy for partitions to be mounted.
Here you can find how multipath became after update and reboot https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharin...
No device-mapper-multipath update in yum.log
Also it seems that after changing it, it was then reverted at boot again (I don't know if the responsible was initrd/dracut or vdsmd) so in the mean time the only thing I could do was to make the file immutable with
chattr +i /etc/multipath.conf
The "supported" method of achieving this is to place "# RHEV PRIVATE" in the second line of your hand-modified multipath.conf I do not understand why this has happened only after upgrade to 3.5.1 - 3.5.0's should have reverted you multipath.conf just as well during each vdsm startup. The good thing is that this annoying behavior has been dropped from the master branch, so that 3.6 is not going to have it. Vdsm is not to mess with other services config file while it is running. The logic moved to `vdsm-tool configure`
and so I was able to reboot and verify that my partitions on sdb were ok and I was able to mount them (for safe I also ran an fsck against them)
Update ran around 19:20 and finished at 19:34 here the log in gzip format https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharin...
Reboot was done around 21:10-21:14
Here my /var/log/messages in gzip format, where you can see latest days. https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharin...
Any suggestion appreciated.
Current multipath.conf (where I also commented out the getuid_callout that is not used anymore):
[root@tekkaman setup]# cat /etc/multipath.conf # RHEV REVISION 1.1
blacklist { devnode "^(sda|sdb)[0-9]*" }
defaults { polling_interval 5 #getuid_callout "/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n" no_path_retry fail user_friendly_names no flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo 30 max_fds 4096 }
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Gianluca Cecchi" <gianluca.cecchi@gmail.com>, nsoffer@redhat.com Cc: "users" <users@ovirt.org>, ykaplan@redhat.com Sent: Monday, January 26, 2015 2:09:23 PM Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
Hello, on my all-in-one installation @home I had 3.5.0 with F20. Today I updated to 3.5.1.
it seems it modified /etc/multipath.conf preventing me from using my second disk at all...
My system has internal ssd disk (sda) for OS and one local storage domain and another disk (sdb) with some partitions (on one of them there is also another local storage domain).
At reboot I was put in emergency boot because partitions at sdb disk could not be mounted (they were busy). it took me some time to understand that the problem was due to sdb gone managed as multipath device and so busy for partitions to be mounted.
Here you can find how multipath became after update and reboot https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharin...
No device-mapper-multipath update in yum.log
Also it seems that after changing it, it was then reverted at boot again (I don't know if the responsible was initrd/dracut or vdsmd) so in the mean time the only thing I could do was to make the file immutable with
chattr +i /etc/multipath.conf
The "supported" method of achieving this is to place "# RHEV PRIVATE" in the second line of your hand-modified multipath.conf
I do not understand why this has happened only after upgrade to 3.5.1 - 3.5.0's should have reverted you multipath.conf just as well during each vdsm startup.
The good thing is that this annoying behavior has been dropped from the master branch, so that 3.6 is not going to have it. Vdsm is not to mess with other services config file while it is running. The logic moved to `vdsm-tool configure`
and so I was able to reboot and verify that my partitions on sdb were ok and I was able to mount them (for safe I also ran an fsck against them)
Update ran around 19:20 and finished at 19:34 here the log in gzip format https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharin...
Reboot was done around 21:10-21:14
Here my /var/log/messages in gzip format, where you can see latest days. https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharin...
Any suggestion appreciated.
Current multipath.conf (where I also commented out the getuid_callout that is not used anymore):
[root@tekkaman setup]# cat /etc/multipath.conf # RHEV REVISION 1.1
blacklist { devnode "^(sda|sdb)[0-9]*" }
I think what happened is: 1. 3.5.1 had new multipath version 2. So vdsm upgraded the local file 3. blacklist above was removed (it should exists in /etc/multipath.bak) To prevent local changes, you have to mark the file as private as Dan suggests. Seems to be related to the find_multiapth = "yes" bug: https://bugzilla.redhat.com/show_bug.cgi?id=1173290 Ben, can you confirm that this is the same issue?
defaults { polling_interval 5 #getuid_callout "/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n" no_path_retry fail user_friendly_names no flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo 30 max_fds 4096 }
Regards, Nir
----- Original Message -----
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Gianluca Cecchi" <gianluca.cecchi@gmail.com>, nsoffer@redhat.com Cc: "users" <users@ovirt.org>, ykaplan@redhat.com Sent: Monday, January 26, 2015 2:09:23 PM Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
Hello, on my all-in-one installation @home I had 3.5.0 with F20. Today I updated to 3.5.1.
it seems it modified /etc/multipath.conf preventing me from using my second disk at all...
My system has internal ssd disk (sda) for OS and one local storage domain and another disk (sdb) with some partitions (on one of them there is also another local storage domain).
At reboot I was put in emergency boot because partitions at sdb disk could not be mounted (they were busy). it took me some time to understand that the problem was due to sdb gone managed as multipath device and so busy for partitions to be mounted.
Here you can find how multipath became after update and reboot https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharin...
No device-mapper-multipath update in yum.log
Also it seems that after changing it, it was then reverted at boot again (I don't know if the responsible was initrd/dracut or vdsmd) so in the mean time the only thing I could do was to make the file immutable with
chattr +i /etc/multipath.conf
The "supported" method of achieving this is to place "# RHEV PRIVATE" in the second line of your hand-modified multipath.conf
I do not understand why this has happened only after upgrade to 3.5.1 - 3.5.0's should have reverted you multipath.conf just as well during each vdsm startup.
The good thing is that this annoying behavior has been dropped from the master branch, so that 3.6 is not going to have it. Vdsm is not to mess with other services config file while it is running. The logic moved to `vdsm-tool configure`
and so I was able to reboot and verify that my partitions on sdb were ok and I was able to mount them (for safe I also ran an fsck against them)
Update ran around 19:20 and finished at 19:34 here the log in gzip format https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharin...
Reboot was done around 21:10-21:14
Here my /var/log/messages in gzip format, where you can see latest days. https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharin...
Any suggestion appreciated.
Current multipath.conf (where I also commented out the getuid_callout that is not used anymore):
[root@tekkaman setup]# cat /etc/multipath.conf # RHEV REVISION 1.1
blacklist { devnode "^(sda|sdb)[0-9]*" }
I think what happened is:
1. 3.5.1 had new multipath version 2. So vdsm upgraded the local file 3. blacklist above was removed (it should exists in /etc/multipath.bak)
To prevent local changes, you have to mark the file as private as Dan suggests.
Seems to be related to the find_multiapth = "yes" bug: https://bugzilla.redhat.com/show_bug.cgi?id=1173290
The symptoms above sound exactly liek this issue. When find_multipahts is no (the default when the directive is not present) I sthat all non-blacklisted devices are tried to get claimed, and this happened above. Blacklisting the devices works, or adding "find_mutlipaths yes" should also work, because in that case only device which have more than one path (or are explicitly named) will be claimed by multipath. My 2ct. - fabian
Ben, can you confirm that this is the same issue?
defaults { polling_interval 5 #getuid_callout "/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n" no_path_retry fail user_friendly_names no flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo 30 max_fds 4096 }
Regards, Nir _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Jan 26, 2015 at 10:27:23AM -0500, Fabian Deutsch wrote:
----- Original Message -----
----- Original Message -----
From: "Dan Kenigsberg" <danken@redhat.com> To: "Gianluca Cecchi" <gianluca.cecchi@gmail.com>, nsoffer@redhat.com Cc: "users" <users@ovirt.org>, ykaplan@redhat.com Sent: Monday, January 26, 2015 2:09:23 PM Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
On Sat, Jan 24, 2015 at 12:59:01AM +0100, Gianluca Cecchi wrote:
Hello, on my all-in-one installation @home I had 3.5.0 with F20. Today I updated to 3.5.1.
it seems it modified /etc/multipath.conf preventing me from using my second disk at all...
My system has internal ssd disk (sda) for OS and one local storage domain and another disk (sdb) with some partitions (on one of them there is also another local storage domain).
At reboot I was put in emergency boot because partitions at sdb disk could not be mounted (they were busy). it took me some time to understand that the problem was due to sdb gone managed as multipath device and so busy for partitions to be mounted.
Here you can find how multipath became after update and reboot https://drive.google.com/file/d/0BwoPbcrMv8mvS0FkMnNyMTdVTms/view?usp=sharin...
No device-mapper-multipath update in yum.log
Also it seems that after changing it, it was then reverted at boot again (I don't know if the responsible was initrd/dracut or vdsmd) so in the mean time the only thing I could do was to make the file immutable with
chattr +i /etc/multipath.conf
The "supported" method of achieving this is to place "# RHEV PRIVATE" in the second line of your hand-modified multipath.conf
I do not understand why this has happened only after upgrade to 3.5.1 - 3.5.0's should have reverted you multipath.conf just as well during each vdsm startup.
The good thing is that this annoying behavior has been dropped from the master branch, so that 3.6 is not going to have it. Vdsm is not to mess with other services config file while it is running. The logic moved to `vdsm-tool configure`
and so I was able to reboot and verify that my partitions on sdb were ok and I was able to mount them (for safe I also ran an fsck against them)
Update ran around 19:20 and finished at 19:34 here the log in gzip format https://drive.google.com/file/d/0BwoPbcrMv8mvWjJDTXU1YjRWOFk/view?usp=sharin...
Reboot was done around 21:10-21:14
Here my /var/log/messages in gzip format, where you can see latest days. https://drive.google.com/file/d/0BwoPbcrMv8mvMm1ldXljd3hZWnM/view?usp=sharin...
Any suggestion appreciated.
Current multipath.conf (where I also commented out the getuid_callout that is not used anymore):
[root@tekkaman setup]# cat /etc/multipath.conf # RHEV REVISION 1.1
blacklist { devnode "^(sda|sdb)[0-9]*" }
I think what happened is:
1. 3.5.1 had new multipath version 2. So vdsm upgraded the local file 3. blacklist above was removed (it should exists in /etc/multipath.bak)
To prevent local changes, you have to mark the file as private as Dan suggests.
Seems to be related to the find_multiapth = "yes" bug: https://bugzilla.redhat.com/show_bug.cgi?id=1173290
The symptoms above sound exactly liek this issue. When find_multipahts is no (the default when the directive is not present) I sthat all non-blacklisted devices are tried to get claimed, and this happened above.
Blacklisting the devices works, or adding "find_mutlipaths yes" should also work, because in that case only device which have more than one path (or are explicitly named) will be claimed by multipath.
I would like to point out one issue. Once a device is claimed (even if find_multipaths wasn't set when it was claimed) it will get added to /etc/multipath/wwids. This means that if you have previously claimed a single path device, adding "find_multipaths yes" won't stop that device from being claimed in the fulture (since it is in the wwids file). You would need to either run: # multipath -w <device> to remove the device's wwid from the wwids file, or run # multipath -W to reset the wwids file to only include the wwids of the current multipath devices (Obviously, you need to remove any devices that you don't want multipathed before your run this).
My 2ct.
- fabian
Ben, can you confirm that this is the same issue?
Yeah, I think so. -Ben
defaults { polling_interval 5 #getuid_callout "/usr/lib/udev/scsi_id --whitelisted --replace-whitespace --device=/dev/%n" no_path_retry fail user_friendly_names no flush_on_last_del yes fast_io_fail_tmo 5 dev_loss_tmo 30 max_fds 4096 }
Regards, Nir _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On Mon, Jan 26, 2015 at 1:37 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Any suggestion appreciated.
Current multipath.conf (where I also commented out the getuid_callout that is not used anymore):
[root@tekkaman setup]# cat /etc/multipath.conf # RHEV REVISION 1.1
blacklist { devnode "^(sda|sdb)[0-9]*" }
I think what happened is:
1. 3.5.1 had new multipath version
what do you mean with "new multipath version"? I currently have device-mapper-multipath-0.4.9-56.fc20.x86_64 The system came from f18 to f19 and then to f20 via fedup in both cases In my yum.log files I see this about January 2013 when I was in Fedora 18 Jan 07 00:11:44 Installed: device-mapper-multipath-0.4.9-36.fc18.x86_64 I then upgraded to f19 on 24th November 2013 and device-mapper-multipath was the one pushed during the fedup update: device-mapper-multipath-libs-0.4.9-51.fc19.x86_64 Then on 12th of November 2014 I passed form f19 to f20 and fedup pushed device-mapper-multipath-0.4.9-56.fc20.x86_64 that is my current version At that time I also passed from oVirt 3.4.4 to 3.5.0. And I didn't register any problem with my internal disk It was sufficient to keep inside blacklist { devnode "sd[a-b]" } At the head of the file I only had: # RHEV REVISION 1.0 No reference about # RHEV PRIVATE But right now that I'm writing I notice that my rule for blacklist after migration to 3.5.1 was devnode "^(sda|sdb)[0-9]*" probably blacklists only partitions and not the full disks.... ;-) So I'm going to check with the old blacklist option and/or the PRIVATE label as suggested by Dan... Probably passing from RHEV REVISION 1.0 to 1.1 the original blacklist part was thrown away... 2. So vdsm upgraded the local file
3. blacklist above was removed (it should exists in /etc/multipath.bak)
It seems it didn't generate any multipath.conf.bak ...
----- Original Message -----
From: "Gianluca Cecchi" <gianluca.cecchi@gmail.com> To: "Nir Soffer" <nsoffer@redhat.com> Cc: "Dan Kenigsberg" <danken@redhat.com>, "users" <users@ovirt.org>, "Yeela Kaplan" <ykaplan@redhat.com>, "Benjamin Marzinski" <bmarzins@redhat.com> Sent: Monday, January 26, 2015 8:46:52 PM Subject: Re: [ovirt-users] Update to 3.5.1 scrambled multipath.conf?
On Mon, Jan 26, 2015 at 1:37 PM, Nir Soffer <nsoffer@redhat.com> wrote:
Any suggestion appreciated.
Current multipath.conf (where I also commented out the getuid_callout that is not used anymore):
[root@tekkaman setup]# cat /etc/multipath.conf # RHEV REVISION 1.1
blacklist { devnode "^(sda|sdb)[0-9]*" }
I think what happened is:
1. 3.5.1 had new multipath version
what do you mean with "new multipath version"?
I mean new multipath.conf version: # RHEV REVISION 1.1 When vdsm finds that its current configuration version is different from current multipath.conf version, it replaces the current multipath.conf with a new version.
I currently have device-mapper-multipath-0.4.9-56.fc20.x86_64 The system came from f18 to f19 and then to f20 via fedup in both cases In my yum.log files I see this about January 2013 when I was in Fedora 18 Jan 07 00:11:44 Installed: device-mapper-multipath-0.4.9-36.fc18.x86_64
I then upgraded to f19 on 24th November 2013 and device-mapper-multipath was the one pushed during the fedup update: device-mapper-multipath-libs-0.4.9-51.fc19.x86_64
Then on 12th of November 2014 I passed form f19 to f20 and fedup pushed device-mapper-multipath-0.4.9-56.fc20.x86_64 that is my current version At that time I also passed from oVirt 3.4.4 to 3.5.0. And I didn't register any problem with my internal disk
It was sufficient to keep inside blacklist { devnode "sd[a-b]" }
At the head of the file I only had: # RHEV REVISION 1.0
This is why vdsm replaced the file.
No reference about # RHEV PRIVATE
But right now that I'm writing I notice that my rule for blacklist after migration to 3.5.1 was
devnode "^(sda|sdb)[0-9]*"
probably blacklists only partitions and not the full disks.... ;-) So I'm going to check with the old blacklist option and/or the PRIVATE label as suggested by Dan...
Probably passing from RHEV REVISION 1.0 to 1.1 the original blacklist part was thrown away...
2. So vdsm upgraded the local file
3. blacklist above was removed (it should exists in /etc/multipath.bak)
It seems it didn't generate any multipath.conf.bak ...
In 3.5.1 it should rotate multipath.conf, saving multipath.conf.1 ... If you don't find the backup file after multipath.conf was updated, this is a bug. Can you open a a bug about it? Nir
participants (5)
-
Benjamin Marzinski -
Dan Kenigsberg -
Fabian Deutsch -
Gianluca Cecchi -
Nir Soffer