On Sun, Oct 4, 2020 at 2:07 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Sat, Oct 3, 2020 at 9:42 PM Amit Bawer <abawer@redhat.com> wrote:


On Sat, Oct 3, 2020 at 10:24 PM Amit Bawer <abawer@redhat.com> wrote:


For the gluster bricks being filtered out in 4.4.2, this seems like [1].


Maybe remove the lvm filter from /etc/lvm/lvm.conf while in 4.4.2 maintenance mode
if the fs is mounted as read only, try
mount -o remount,rw /
sync and try to reboot 4.4.2.


Indeed if i run, when in emergency shell in 4.4.2, the command:

lvs --config 'devices { filter = [ "a|.*|" ] }'

I see also all the gluster volumes, so I think the update injected the nasty filter.
Possibly during update the command
# vdsm-tool config-lvm-filter -y
was executed and erroneously created the filter?
Since there wasn't a filter set on the node, the 4.4.2 update added the default filter for the root-lv pv
if there was some filter set before the upgrade, it would not have been added by the 4.4.2 update.


Anyway remounting read write the root filesystem and removing the filter line from lvm.conf and rebooting worked and 4.4.2 booted ok and I was able to exit global maintenance and have the engine up.

Thanks Amit for the help and all the insights.

Right now only two problems:

1) a long running problem that from engine web admin all the volumes are seen as up and also the storage domains up, while only the hosted engine one is up, while "data" and vmstore" are down, as I can verify from the host, only one /rhev/data-center/ mount:

[root@ovirt01 ~]# df -h
Filesystem                                              Size  Used Avail Use% Mounted on
devtmpfs                                                 16G     0   16G   0% /dev
tmpfs                                                    16G   16K   16G   1% /dev/shm
tmpfs                                                    16G   18M   16G   1% /run
tmpfs                                                    16G     0   16G   0% /sys/fs/cgroup
/dev/mapper/onn-ovirt--node--ng--4.4.2--0.20200918.0+1  133G  3.9G  129G   3% /
/dev/mapper/onn-tmp                                    1014M   40M  975M   4% /tmp
/dev/mapper/gluster_vg_sda-gluster_lv_engine            100G  9.0G   91G   9% /gluster_bricks/engine
/dev/mapper/gluster_vg_sda-gluster_lv_data              500G  126G  375G  26% /gluster_bricks/data
/dev/mapper/gluster_vg_sda-gluster_lv_vmstore            90G  6.9G   84G   8% /gluster_bricks/vmstore
/dev/mapper/onn-home                                   1014M   40M  975M   4% /home
/dev/sdb2                                               976M  307M  603M  34% /boot
/dev/sdb1                                               599M  6.8M  593M   2% /boot/efi
/dev/mapper/onn-var                                      15G  263M   15G   2% /var
/dev/mapper/onn-var_log                                 8.0G  541M  7.5G   7% /var/log
/dev/mapper/onn-var_crash                                10G  105M  9.9G   2% /var/crash
/dev/mapper/onn-var_log_audit                           2.0G   79M  2.0G   4% /var/log/audit
ovirt01st.lutwyn.storage:/engine                        100G   10G   90G  10% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_engine
tmpfs                                                   3.2G     0  3.2G   0% /run/user/1000
[root@ovirt01 ~]#

I can also wait 10 minutes and no change. The way I use to exit from this stalled situation is power on a VM, so that obviously it fails
VM f32 is down with error. Exit message: Unable to get volume size for domain d39ed9a3-3b10-46bf-b334-e8970f5deca1 volume 242d16c6-1fd9-4918-b9dd-0d477a86424c.
10/4/20 12:50:41 AM

and suddenly all the data storage domains are deactivated (from engine point of view, because actually they were not active...):
Storage Domain vmstore (Data Center Default) was deactivated by system because it's not visible by any of the hosts.
10/4/20 12:50:31 AM

and I can go in Data Centers --> Default --> Storage and activate "vmstore" and "data" storage domains and suddenly I get them activated and filesystems mounted.

[root@ovirt01 ~]# df -h | grep rhev
ovirt01st.lutwyn.storage:/engine                        100G   10G   90G  10% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_engine
ovirt01st.lutwyn.storage:/data                          500G  131G  370G  27% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_data
ovirt01st.lutwyn.storage:/vmstore                        90G  7.8G   83G   9% /rhev/data-center/mnt/glusterSD/ovirt01st.lutwyn.storage:_vmstore
[root@ovirt01 ~]#

and VM starts ok now.

I already reported this, but I don't know if there is yet a bugzilla open for it.
Did you get any response for the original mail? haven't seen it on the users-list.


2) I see that I cannot connect to cockpit console of node.

In firefox (version 80) in my Fedora 31 I get:
"
Secure Connection Failed

An error occurred during a connection to ovirt01.lutwyn.local:9090. PR_CONNECT_RESET_ERROR

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

Learn more…
"
In Chrome (build 85.0.4183.121)

"
Your connection is not private
Attackers might be trying to steal your information from ovirt01.lutwyn.local (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_AUTHORITY_INVALID
"
Click Advanced and select to go to the site

"
This server could not prove that it is ovirt01.lutwyn.local; its security certificate is not trusted by your computer's operating system. This may be caused by a misconfiguration or an attacker intercepting your connection."

If I select

"
This page isn’t working ovirt01.lutwyn.local didn’t send any data.
ERR_EMPTY_RESPONSE
"

NOTE: the ost is not resolved by DNS but I put an entry in my hosts client.
Might be required to set DNS for authenticity, maybe other members on the list could tell better.


On host:

[root@ovirt01 ~]# systemctl status cockpit.socket --no-pager
● cockpit.socket - Cockpit Web Service Socket
   Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; disabled; vendor preset: enabled)
   Active: active (listening) since Sun 2020-10-04 00:36:36 CEST; 25min ago
     Docs: man:cockpit-ws(8)
   Listen: [::]:9090 (Stream)
  Process: 1425 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
  Process: 1417 ExecStartPost=/usr/share/cockpit/motd/update-motd  localhost (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 202981)
   Memory: 1.6M
   CGroup: /system.slice/cockpit.socket

Oct 04 00:36:36 ovirt01.lutwyn.local systemd[1]: Starting Cockpit Web Service Socket.
Oct 04 00:36:36 ovirt01.lutwyn.local systemd[1]: Listening on Cockpit Web Service Socket.
[root@ovirt01 ~]#

[root@ovirt01 ~]# systemctl status cockpit.service --no-pager
● cockpit.service - Cockpit Web Service
   Loaded: loaded (/usr/lib/systemd/system/cockpit.service; static; vendor preset: disabled)
   Active: active (running) since Sun 2020-10-04 00:58:09 CEST; 3min 30s ago
     Docs: man:cockpit-ws(8)
  Process: 19260 ExecStartPre=/usr/sbin/remotectl certificate --ensure --user=root --group=cockpit-ws --selinux-type=etc_t (code=exited, status=0/SUCCESS)
 Main PID: 19263 (cockpit-tls)
    Tasks: 1 (limit: 202981)
   Memory: 1.4M
   CGroup: /system.slice/cockpit.service
           └─19263 /usr/libexec/cockpit-tls

Oct 04 00:59:59 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(http-redirect.sock) failed: Permission denied
Oct 04 00:59:59 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(http-redirect.sock) failed: Permission denied
Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(https-factory.sock) failed: Permission denied
Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
Oct 04 01:00:11 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(https-factory.sock) failed: Permission denied
Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
Oct 04 01:00:16 ovirt01.lutwyn.local cockpit-tls[19263]: cockpit-tls: connect(https-factory.sock) failed: Permission denied
[root@ovirt01 ~]#


Gianluca