On Tue, May 14, 2019 at 12:25 PM <axel.thimm@01lgc.com> wrote:
>
> > I'd personally first try to fix the engine<->vdsm comm issue.
>
> I fully agree. I might then take exports of all VMs and do a fresh start over. I'm not sure the upgrade instructions work, as the upgrade for 3.6->4.0 already activates much newer content. For example the quoted --upgrade-appliance options does not even exist, probably because a later package removed it. One would have to manually versionlock the 4.0 hypervisior packages.
>
> > How did you upgrade the host? yum update? Is is ovirt-node?
>
> Currently the state is as follows: engine is running on the 3.6 branch, all updates applied [1]. The hosting RHEL host is on the RHV4 branch with all updats in, but the gnutls mentioned [2].
>
> I managed to flag the host as in maintenance mode, so I could ask it to re-enroll the certs.

Did you? Please do.

> I can see from the hosts logs, that the ssh connection works, and the certs have been updated. But now I see in the host's vdsm log that the SSL connection is being dropped.

This can also be because a 3.6 engine defaults xmlrpc and a new 4.3
host talks only jsonrpc. Not sure, perhaps you can change the host (on
engine side) to use json rpc and it would work, although a 4.3 host is
definitely not supposed to support a 3.6 engine. If all you want is to
export the VMs, I'd downgrade the host to 3.6.

>
> > If you are ok debugging/fixing yourself and only need the occasional
> > tip, fine. Otherwise, please provide more details - exact versions of
> > engine/vdsm, last version that worked, whether you can reinstall the
> > host or need anything on it, etc.
>
> I'm infinitely grateful for any assistance! I didn't want to spam the list with my first mail. I hope I provided some better information now. Thanks!
>
> In the long run I will decommission this host. I need the VMs on it to migrate to a fresh 4.3 3 host hyperconverged setup which is yet to be built. Actually I am abusing one of the three nodes at the moment as a backup system for this failed upgrade, so I'm blocked with going forward with that setup as well. ... :(

Another option you might attempt is to import the data domain directly
from the new engine. Obviously, do this first on a test copy...

Re [2]: I'll use this opportunity to clarify something which is not
obvious, since you use RHV and not oVirt.

There is a significant difference between oVirt and RHV regarding
repos/channels. In oVirt, each minor version has its own repos, and
users can choose freely which one they want. In RHV, this the same for
the engine channels, but not for hosts. For hosts, there is only a
single channel per each major version. So if you use RHV 4 host
channel, you get latest. Older versions are still available there, but
you have to play with yum to choose them, and in RHV this is
considered not supported, other than using RHVH (RHV downstream of
oVirt node) which can be used for specific cases (perhaps like yours,
but not sure it would help).

This has the advantage, for RHV
customers, of not having to add a new repo to get updates, and the
disadvantage of not being able to stay on a specific minor version
when upgrading.

Good luck and best regards,


>
> [1]
> [root@engine ~]# engine-upgrade-check
> VERB: queue package rhevm-setup for update
> VERB: Downloading: repomdtDvMe7tmp.xml (0%)
> VERB: Downloading: repomdtDvMe7tmp.xml 3.4 k(100%)
> VERB: Downloading: jb-eap-6-for-rhel-6-server-rpms/primary_db (0%)
> VERB: Downloading: jb-eap-6-for-rhel-6-server-rpms/primary_db 1.2 M(100%)
> VERB: Downloading: repomd__rFIUtmp.xml (0%)
> VERB: Downloading: repomd__rFIUtmp.xml 3.4 k(100%)
> VERB: Downloading: repomdXw2WPVtmp.xml (0%)
> VERB: Downloading: repomdXw2WPVtmp.xml 4.0 k(100%)
> VERB: Downloading: repomdF3AGBOtmp.xml (0%)
> VERB: Downloading: repomdF3AGBOtmp.xml 3.5 k(100%)
> VERB: Downloading: repomd9DvU9vtmp.xml (0%)
> VERB: Downloading: repomd9DvU9vtmp.xml 3.4 k(100%)
> VERB: Downloading: rhel-6-server-supplementary-rpms/primary_db (0%)
> VERB: Downloading: rhel-6-server-supplementary-rpms/primary_db 851 k(100%)
> VERB: Building transaction
> VERB: Empty transaction
> VERB: Transaction Summary:
> No upgrade
>
> [2]
> # subscription-manager repos --list-enabled; yum check-update
> +----------------------------------------------------------+
>     Available Repositories in /etc/yum.repos.d/redhat.repo
> +----------------------------------------------------------+
> Repo ID:   rhel-7-server-ansible-2-rpms
> Repo Name: Red Hat Ansible Engine 2 RPMs for Red Hat Enterprise Linux 7 Server
> Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/7Server/$basearch/a
>            nsible/2/os
> Enabled:   1
>
> Repo ID:   rhel-7-server-rhev-mgmt-agent-rpms
> Repo Name: Red Hat Enterprise Virtualization Management Agents for RHEL 7 (RPMs)
> Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basear
>            ch/rhev-mgmt-agent/3/os
> Enabled:   1
>
> Repo ID:   rhel-7-server-rhv-4-mgmt-agent-rpms
> Repo Name: Red Hat Virtualization 4 Management Agents for RHEL 7 (RPMs)
> Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basear
>            ch/rhv-mgmt-agent/4/os
> Enabled:   1
>
> Repo ID:   rhel-7-server-rpms
> Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
> Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basear
>            ch/os
> Enabled:   1
>
> Repo ID:   rhel-7-server-optional-rpms
> Repo Name: Red Hat Enterprise Linux 7 Server - Optional (RPMs)
> Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/$releasever/$basear
>            ch/optional/os
> Enabled:   1
>
> Loaded plugins: enabled_repos_upload, package_upload, product-id, search-
>               : disabled-repos, subscription-manager, vdsmupgrade
>
> gnutls.x86_64                                                                                             3.3.29-9.el7_6                                                                                        rhel-7-server-rpms
> gnutls-dane.x86_64                                                                                        3.3.29-9.el7_6                                                                                        rhel-7-server-rpms
> gnutls-utils.x86_64                                                                                       3.3.29-9.el7_6                                                                                        rhel-7-server-rpms
> Uploading Enabled Repositories Report
> Loaded plugins: product-id, subscription-manager
> Unable to upload Enabled Repositories Report
>
> [3]
> May 14 10:57:18 hetzner-XXXXXX systemd[1]: Starting Virtual Desktop Server Manager...
> May 14 10:57:18 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running mkdirs
> May 14 10:57:18 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running configure_coredump
> May 14 10:57:18 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running configure_vdsm_logs
> May 14 10:57:18 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running wait_for_network
> May 14 10:57:18 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running run_init_hooks
> May 14 10:57:19 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running check_is_configured
> May 14 10:57:19 hetzner-XXXXXX sasldblistusers2[8203]: DIGEST-MD5 common mech free
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: abrt is already configured for vdsm
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: Managed volume database is already configured
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: lvm is configured for vdsm
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: libvirt is already configured for vdsm
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: Current revision of multipath.conf detected, preserving
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running validate_configuration
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: SUCCESS: ssl configured to true. No conflicts
> May 14 10:57:20 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running prepare_transient_repository
> May 14 10:57:21 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running syslog_available
> May 14 10:57:21 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running nwfilter
> May 14 10:57:21 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running dummybr
> May 14 10:57:21 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running tune_system
> May 14 10:57:21 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running test_space
> May 14 10:57:21 hetzner-XXXXXX vdsmd_init_common.sh[8177]: vdsm: Running test_lo
> May 14 10:57:21 hetzner-XXXXXX systemd[1]: Started Virtual Desktop Server Manager.
> May 14 10:57:22 hetzner-XXXXXX vdsm[8252]: WARN unhandled write event
> May 14 10:57:22 hetzner-XXXXXX vdsm[8252]: WARN Not ready yet, ignoring event '|virt|VM_status|4f28af23-dd7e-413e-a331-1875f4dd18b3' args={'4f28af23-dd7e-413e-a331-1875f4dd18b3': {'status': 'Down', 'displayInfo': [{'tlsPort': '-1', 'ipAddress': '0', 'type': 'vnc', 'port': '-1'}], 'hash': '-8231387692555228201', 'exitMessage': 'VM terminated with error', 'cpuUser': '0.00', 'monitorResponse': '0', 'vmId': '4f28af23-dd7e-413e-a331-1875f4dd18b3', 'exitReason': 1, 'cpuUsage': '0.00', 'elapsedTime': '8420', 'cpuSys': '0.00', 'timeOffset': '0', 'clientIp': '', 'exitCode': 1}}
> May 14 10:57:22 hetzner-XXXXXX vdsm[8252]: WARN MOM not available.
> May 14 10:57:22 hetzner-XXXXXX vdsm[8252]: WARN MOM not available, KSM stats will be missing.
> May 14 10:58:47 hetzner-XXXXXX vdsm[8252]: WARN File: /var/lib/libvirt/qemu/channels/4f28af23-dd7e-413e-a331-1875f4dd18b3.com.redhat.rhevm.vdsm already removed
> May 14 10:58:47 hetzner-XXXXXX vdsm[8252]: WARN File: /var/lib/libvirt/qemu/channels/4f28af23-dd7e-413e-a331-1875f4dd18b3.org.qemu.guest_agent.0 already removed
> May 14 11:00:52 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:01:54 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:02:06 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:05:18 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:05:29 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:08:41 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:08:53 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:12:04 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:12:16 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:15:28 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> May 14 11:15:40 hetzner-XXXXXX vdsm[8252]: ERROR ssl handshake: SSLError, address: ::ffff:192.168.111.10
> _______________________________________________
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/DOD7T7DL55TR5LTDCAHA64464WQBV5QX/



--
Didi
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VVT3XFYQPSBQUEK76NEQOMSZTARQ4KJR/