I need some help. We had a working ovirt cluster in the testing environment. We have just moved it to the production environment with the same network settings. The only thing we changed is the public VLAN. In production we're using a different subnet.
The problem is we can't get the HostedEngine Up. It does come up but it fails the LIVELINESS CHECK and its health status is bad. We can't ping even ping it. It is on the same subnet as host machines: 192.168.x.x/24:
[root@garlic qemu]# hosted-engine --vm-status
--== Host 1 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Host ID : 1
Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 57b2ece9
local_conf_timestamp : 8463
Host timestamp : 8463
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=8463 (Thu Feb 21 16:32:29 2019)
host-id=1
score=3400
vm_conf_refresh_time=8463 (Thu Feb 21 16:32:29 2019)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
--== Host 2 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Host ID : 2
Engine status : {"reason": "failed liveliness check", "health": "bad", "vm": "up", "detail": "Powering down"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 71dc3daf
local_conf_timestamp : 8540
Host timestamp : 8540
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=8540 (Thu Feb 21 16:32:31 2019)
host-id=2
score=3400
vm_conf_refresh_time=8540 (Thu Feb 21 16:32:31 2019)
conf_on_shared_storage=True
maintenance=False
state=EngineStop
stopped=False
timeout=Thu Jan 1 04:24:29 1970
--== Host 3 status ==--
conf_on_shared_storage : True
Status up-to-date : True
Host ID : 3
Engine status : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 49645620
local_conf_timestamp : 5480
Host timestamp : 5480
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=5480 (Thu Feb 21 16:32:22 2019)
host-id=3
score=3400
vm_conf_refresh_time=5480 (Thu Feb 21 16:32:22 2019)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
You have new mail in /var/spool/mail/root
[root@garlic qemu]# systemctl status vdsmd
● vdsmd.service - Virtual Desktop Server Manager
Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2019-02-21 16:12:12 SAST; 3min 31s ago
Process: 40117 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)
Process: 40121 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
Main PID: 40224 (vdsmd)
Tasks: 47
CGroup: /system.slice/vdsmd.service
├─40224 /usr/bin/python2 /usr/share/vdsm/vdsmd
└─40346 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd 64 --max-threads 10 --max-queued-requests 10
Feb 21 16:12:11 garlic.sanren.ac.za vdsmd_init_common.sh[40121]: libvirt: Network Filter Driver error : Requested operation is not valid: nwfilter is in use Feb 21 16:12:13 garlic.sanren.ac.za vdsm[40224]: WARN MOM not available, KSM stats will be missing. Feb 21 16:12:13 garlic.sanren.ac.za vdsm[40224]: WARN Not ready yet, ignoring event '|virt|VM_status|e2608f14-39fe-4ab6-b6be-9c60679e8c76' args={'e2608f14-39fe-4ab6-b6be-9c606..., 'type': ' Hint: Some lines were ellipsized, use -l to show in full.
libvirtd.service:
[root@garlic qemu]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/libvirtd.service.d
└─unlimited-core.conf
Active: active (running) since Thu 2019-02-21 16:06:50 SAST; 9min ago
Docs: man:libvirtd(8)
Main PID: 38485 (libvirtd)
Tasks: 17 (limit: 32768)
CGroup: /system.slice/libvirtd.service
└─38485 /usr/sbin/libvirtd --listen
Feb 21 16:07:43 garlic.sanren.ac.za libvirtd[38485]: 2019-02-21 14:07:43.033+0000: 38485: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Feb 21 16:12:08 garlic.sanren.ac.za libvirtd[38485]: 2019-02-21 14:12:08.791+0000: 38485: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error Hint: Some lines were ellipsized, use -l to show in full.