Il giorno ven 25 set 2020 alle ore 15:32 Gianluca Cecchi < gianluca.cecchi(a)gmail.com&gt; ha scritto: I don't think we had someone testing 4.4.0 to 4.4.2 but above procedure should work for the same case. The problematic filter in /etc/lvm/lvm.conf looks like: # grep '^filter = ' /etc/lvm/lvm.conf filter = ["a|^/dev/mapper/mpatha2$|", "r|.*|"] -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>*

From the info it seems that startup panics because gluster bricks cannot be mounted. The filter that you do have in the 4.4.2 screenshot should correspond to your root pv, you can confirm that by doing (replace the pv-uuid with the one from your filter): #udevadm info /dev/disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ P: /devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sda/sda2 N: sda2 S: disk/by-id/ata-QEMU_HARDDISK_QM00003-part2 S: disk/by-id/lvm-pv-uuid-DXgufc-7riC-TqhU-f8yH-EfZt-ivvH-TVcnEQ In this case sda2 is the partition of the root-lv shown by lsblk. Can you give the output of lsblk on your node? Can you check that the same filter is in initramfs? # lsinitrd -f /etc/lvm/lvm.conf | grep filter We have the following tool on the hosts # vdsm-tool config-lvm-filter -y it only sets the filter for local lvm devices, this is run as part of deployment and upgrade when done from the engine. If you have other volumes which have to be mounted as part of your startup then you should add their uuids to the filter as well. On Sat, Oct 3, 2020 at 3:19 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote:

Sorry I see that there was an error in the lsinitrd command in 4.4.2, inerting the "-f" position. Here the screenshot that shows anyway no filter active: https://drive.google.com/file/d/19VmgvsHU2DhJCRzCbO9K_Xyr70x4BqXX/view?us... Gianluca On Sat, Oct 3, 2020 at 6:26 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote:

On Sat, Oct 3, 2020 at 7:26 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: What does "udevadm info" show for /dev/sdb3 on 4.4.2? The initramfs for 4.4.2 doesn't show any (wrong) filter, so i don't see what needs to be fixed in this case. Might work, probably not too tested. For the gluster bricks being filtered out in 4.4.2, this seems like [1]. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1883805

On Sat, Oct 3, 2020 at 9:42 PM Amit Bawer <abawer(a)redhat.com&gt; wrote: Indeed if i run, when in emergency shell in 4.4.2, the command: lvs --config 'devices { filter = [ "a|.*|" ] }' I see also all the gluster volumes, so I think the update injected the nasty filter. Possibly during update the command # vdsm-tool config-lvm-filter -y was executed and erroneously created the filter? Anyway remounting read write the root filesystem and removing the filter line from lvm.conf and rebooting worked and 4.4.2 booted ok and I was able to exit global maintenance and have the engine up. Thanks Amit for the help and all the insights. Right now only two problems: 1) a long running problem that from engine web admin all the volumes are seen as up and also the storage domains up, while only the hosted engine one is up, while "data" and vmstore" are down, as I can verify from the host, only one /rhev/data-center/ mount: [root@ovirt01 ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 16G 0 16G 0% /dev tmpfs 16G 16K 16G 1% /dev/shm tmpfs 16G 18M 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/mapper/onn-ovirt--node--ng--4.4.2--0.20200918.0+1 133G 3.9G 129G 3% / /dev/mapper/onn-tmp 1014M 40M 975M 4% /tmp /dev/mapper/gluster_vg_sda-gluster_lv_engine 100G 9.0G 91G 9% /gluster_bricks/engine /dev/mapper/gluster_vg_sda-gluster_lv_data 500G 126G 375G 26% /gluster_bricks/data /dev/mapper/gluster_vg_sda-gluster_lv_vmstore 90G 6.9G 84G 8% /gluster_bricks/vmstore /dev/mapper/onn-home 1014M 40M 975M 4% /home /dev/sdb2 976M 307M 603M 34% /boot /dev/sdb1 599M 6.8M 593M 2% /boot/efi /dev/mapper/onn-var 15G 263M 15G 2% /var /dev/mapper/onn-var_log 8.0G 541M 7.5G 7% /var/log /dev/mapper/onn-var_crash 10G 105M 9.9G 2% /var/crash /dev/mapper/onn-var_log_audit 2.0G 79M 2.0G 4% /var/log/audit ovirt01st.lutwyn.storage:/engine 100G 10G 90G 10% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_engine tmpfs 3.2G 0 3.2G 0% /run/user/1000 [root@ovirt01 ~]# I can also wait 10 minutes and no change. The way I use to exit from this stalled situation is power on a VM, so that obviously it fails VM f32 is down with error. Exit message: Unable to get volume size for domain d39ed9a3-3b10-46bf-b334-e8970f5deca1 volume 242d16c6-1fd9-4918-b9dd-0d477a86424c. 10/4/20 12:50:41 AM and suddenly all the data storage domains are deactivated (from engine point of view, because actually they were not active...): Storage Domain vmstore (Data Center Default) was deactivated by system because it's not visible by any of the hosts. 10/4/20 12:50:31 AM and I can go in Data Centers --> Default --> Storage and activate "vmstore" and "data" storage domains and suddenly I get them activated and filesystems mounted. [root@ovirt01 ~]# df -h | grep rhev ovirt01st.lutwyn.storage:/engine 100G 10G 90G 10% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_engine ovirt01st.lutwyn.storage:/data 500G 131G 370G 27% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_data ovirt01st.lutwyn.storage:/vmstore 90G 7.8G 83G 9% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_vmstore [root@ovirt01 ~]# and VM starts ok now. I already reported this, but I don't know if there is yet a bugzilla open for it. 2) I see that I cannot connect to cockpit console of node. In firefox (version 80) in my Fedora 31 I get: " Secure Connection Failed An error occurred during a connection to ovirt01.lutwyn.local:9090. PR_CONNECT_RESET_ERROR The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem. Learn more… " In Chrome (build 85.0.4183.121) " Your connection is not private Attackers might be trying to steal your information from ovirt01.lutwyn.local (for example, passwords, messages, or credit cards). Learn more NET::ERR_CERT_AUTHORITY_INVALID " Click Advanced and select to go to the site " This server could not prove that it is ovirt01.lutwyn.local; its security certificate is not trusted by your computer's operating system. This may be caused by a misconfiguration or an attacker intercepting your connection." If I select " This page isn’t working ovirt01.lutwyn.local didn’t send any data. ERR_EMPTY_RESPONSE " NOTE: the ost is not resolved by DNS but I put an entry in my hosts client. On host: [root@ovirt01 ~]# systemctl status cockpit.socket --no-pager ● cockpit.socket - Cockpit Web Service Socket Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: enabled) Active: active (listening) since Sun 2020-10-04 00:36:36 CEST; 25min ago Docs: man:cockpit-ws(8) Listen: [::]:9090 (Stream) Process: 1425 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS) Process: 1417 ExecStartPost=/usr/share/cockpit/motd/update-motd localhost (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 202981) Memory: 1.6M CGroup: /system.slice/cockpit.socket Oct 04 00:36:36 ovirt01.lutwyn.local systemd[1]: Starting Cockpit Web Service Socket. Oct 04 00:36:36 ovirt01.lutwyn.local systemd[1]: Listening on Cockpit Web Service Socket. [root@ovirt01 ~]# [root@ovirt01 ~]# systemctl status cockpit.service --no-pager ● cockpit.service - Cockpit Web Service Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled) Active: active (running) since Sun 2020-10-04 00:58:09 CEST; 3min 30s ago Docs: man:cockpit-ws(8) Process: 19260 ExecStartPre=/usr/sbin/remotectl certificate --ensure --user=root --group=cockpit-ws --selinux-type=etc_t (code=exited, status=0/SUCCESS) Main PID: 19263 (cockpit-tls) Tasks: 1 (limit: 202981) Memory: 1.4M CGroup: /system.slice/cockpit.service └─19263 /usr/libexec/cockpit-tls Oct 04 00:59:59 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(http-redirect.sock) failed: Permission denied Oct 04 00:59:59 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(http-redirect.sock) failed: Permission denied Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received. Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(https-factory.sock) failed: Permission denied Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received. Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(https-factory.sock) failed: Permission denied Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received. Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received. Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received. Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(https-factory.sock) failed: Permission denied [root@ovirt01 ~]# Gianluca

On Sun, Oct 4, 2020 at 10:21 AM Amit Bawer <abawer(a)redhat.com&gt; wrote: Do you mean that I will get the same problem upgrading from 4.4.2 to an upcoming 4.4.3, as also now I don't have any filter set? This would not be desirable.... I think it was this thread related to 4.4.0 released and question about auto-start of VMs. A script from Derek that tested if domains were active and got false positive, and my comments about the same registered behaviour: https://lists.ovirt.org/archives/list/users@ovirt.org/message/25KYZTFKX5Y... But I think there was no answer on that particular item/problem. Indeed I think you can easily reproduce, I don't know if only with Gluster or also with other storage domains. I don't know if it can have a part the fact that on the last host during a whole shutdown (and the only host in case of single host) you have to run the script /usr/share/glusterfs/scripts/stop-all-gluster-processes.sh otherwise you risk not to get a complete shutdown sometimes. And perhaps this stop can have an influence on the following startup. In any case the web admin gui (and the API access) should not show the domains active when they are not. I think there is a bug in the code that checks this. It would be the first time I see it. The access to web admin GUI works ok even without DNS resolution. I'm not sure if I had the same problem with the cockpit host console on 4.4.0. Gianluca

On Sun, Oct 4, 2020 at 6:09 PM Amit Bawer <abawer(a)redhat.com&gt; wrote: vdsm-tool is expected to add all the devices needed by the mounted logical volumes, so adding devices manually should not be needed. If this does not work please file a bug and include all the info to reproduce the issue.

Il giorno sab 3 ott 2020 alle ore 14:16 Gianluca Cecchi < gianluca.cecchi(a)gmail.com&gt; ha scritto: Please use the update command from the engine admin portal. The ansible code running from there also performs additional steps other than just yum update. +Dana Elfassy <delfassy(a)redhat.com&gt; can you elaborate on other steps performed during the upgrade? -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours. <https://mojo.redhat.com/docs/DOC-1199578>* * <https://www.redhat.com/it/forums/emea/italy-track>*

Yes. The additional main tasks that we execute during host upgrade besides updating packages are certificates related (check for certificates validity, enroll certificates) , configuring advanced virtualization and lvm filter Dana On Mon, Oct 5, 2020 at 9:31 AM Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote:

On Mon, Oct 5, 2020 at 9:06 AM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: Yes... This file will not change... Here we will have the second device... This file will not change, it was created by RHHI deploy, and vdsm knows nothing about it. RHHI should integrate with new vdsm-tool capabilities instead of duplicating the functionality. This file will not be changed by vdsm-tool. multipath manages this file. Sounds good.

On Mon, Oct 5, 2020 at 12:52 PM Dana Elfassy <delfassy(a)redhat.com&gt; wrote: Because when you upgrade a host you put it into maintenance before. And this implies no VMs in execution on it. But if you are in a single host composed environment you cannot.... Gianluca

On Mon, Oct 5, 2020 at 3:13 PM Dana Elfassy <delfassy(a)redhat.com&gt; wrote: we are talking about chicken-egg problem..... You say to drive a command form the engine that is a VM that runs inside the host, but ask to shutdown VMs running on host before... This is a self hosted engine composed by only one single host. Normally I would use the procedure from the engine web admin gui, one host at a time, but with single host it is not possible..... Gianluca

On Tue, Oct 6, 2020 at 11:25 AM Martin Perina <mperina(a)redhat.com&gt; wrote: We who???? In old times there was the all-in-one setup that was substituted from single host HCI ... developers also put extra efforts to setup the wizard comprising the single host scenario..... Obviously it is aimed at test bed / devel / home environments, not production ones. Do you want me to send you the list of bugzilla contributed by users using single host environments that helped Red Hat to have a better working RHV too? Please think more deeply next time, thanks Gianluca