6:25 p.m.
Hi, I working on a CentOS7 based Ovirt 3.6 system (ovirt-engine/db on one machine, two
separate ovirt vm hosts) which has been running fine but mostly ignored for 2-3 of years.
Recently it was decided to update the OS as it was far behind on security updates, so one
host was put into maintenance mode, yum update'd, rebooted, and then it was attempted
to take out of maintenance mode but it's "non-responsive" now.
If I look in /var/log/ovirt-engine/engine.log on the engine machine I see for this host
(vmserver2):
"ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-36) [2bc0978d] Command
'GetCapabilitiesVDSCommand(HostName = vmserver2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='6725086f-42c0-40eb-91f1-0f2411ea9432',
vds='Host[vmserver2,6725086f-42c0-40eb-91f1-0f2411ea9432]'})' execution
failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed"
and thereafter more errors. This keep repeating in the log.
In the Ovirt GUI I see multiple occurrences of log entries for the problem host:
"vmserver2...command failed: Vds timeout occurred"
"vmserver2...command failed: Heartbeat exceeded"
"vmserver2...command failed: internal error: Unknown CPU model
Broadwell-noTSX-IBRS"
Firewall rules look identical to the host which is working normally but has not been
updated.
Any thoughts about how to fix or further troubleshoot this?
10:44 p.m.
On Tue, Jan 1, 2019 at 4:26 AM <jaherring(a)usa.net> wrote:
Hi, I working on a CentOS7 based Ovirt 3.6 system (ovirt-engine/db on one machine, two
separate ovirt vm hosts) which has been running fine but mostly ignored for 2-3 of years.
Recently it was decided to update the OS as it was far behind on security updates, so one
host was put into maintenance mode, yum update'd, rebooted, and then it was attempted
to take out of maintenance mode but it's "non-responsive" now.
If I look in /var/log/ovirt-engine/engine.log on the engine machine I see for this host
(vmserver2):
"ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-36) [2bc0978d] Command
'GetCapabilitiesVDSCommand(HostName = vmserver2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='6725086f-42c0-40eb-91f1-0f2411ea9432',
vds='Host[vmserver2,6725086f-42c0-40eb-91f1-0f2411ea9432]'})' execution
failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed"
and thereafter more errors. This keep repeating in the log.
In the Ovirt GUI I see multiple occurrences of log entries for the problem host:
"vmserver2...command failed: Vds timeout occurred"
"vmserver2...command failed: Heartbeat exceeded"
"vmserver2...command failed: internal error: Unknown CPU model
Broadwell-noTSX-IBRS"
Firewall rules look identical to the host which is working normally but has not been
updated.
Any thoughts about how to fix or further troubleshoot this?
Please note that 3.6 is old and was not tested with recent CentOS
(at least AFAIK).
Does vdsm manage to start? Please check/share its logs. Thanks.
Best regards,
--
Didi
10:32 a.m.
New subject: Host non-responsive after yum update CentOS7/Ovirt3.6
I assumed that if ovirt3.6 required older versions of CentOS7, etc, it
would not allow the yum update due to dependency. I guess that's a bad
assumption.
vdsm does not start in fact. Here's the journal when attempting to
start it.
# journalctl -xe
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: libvirt.libvirtError:
authentication failed: authentication failed
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: Traceback (most recent call
last):
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: File "/usr/bin/vdsm-tool",
line 219, in main
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: return
tool_command[cmd]["command"](*args)
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: File
"/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 41,
in restore_command
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: exec_restore(cmd)
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: File
"/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 54,
in exec_restore
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: raise
EnvironmentError('Failed to restore the persisted networks')
Jan 19 10:13:36 vmserver2 vdsm-tool[6794]: EnvironmentError: Failed to
restore the persisted networks
Jan 19 10:13:36 vmserver2 systemd[1]: vdsm-network.service: main process
exited, code=exited, status=1/FAILURE
Jan 19 10:13:36 vmserver2 systemd[1]: Failed to start Virtual Desktop
Server Manager network restoration.
-- Subject: Unit vdsm-network.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit vdsm-network.service has failed.
--
-- The result is failed.
Jan 19 10:13:36 vmserver2 systemd[1]: Dependency failed for Virtual
Desktop Server Manager.
-- Subject: Unit vdsmd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit vdsmd.service has failed.
--
-- The result is dependency.
Jan 19 10:13:36 vmserver2 systemd[1]: Dependency failed for MOM instance
configured for VDSM purposes.
-- Subject: Unit mom-vdsm.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mom-vdsm.service has failed.
--
-- The result is dependency.
Jan 19 10:13:36 vmserver2 systemd[1]: Job mom-vdsm.service/start failed
with result 'dependency'.
Jan 19 10:13:36 vmserver2 systemd[1]: Job vdsmd.service/start failed
with result 'dependency'.
Jan 19 10:13:36 vmserver2 systemd[1]: Unit vdsm-network.service entered
failed state.
Jan 19 10:13:36 vmserver2 systemd[1]: vdsm-network.service failed.
Jan 19 10:13:36 vmserver2 polkitd[4789]: Unregistered Authentication
Agent for unix-process:6781:161167654 (system bus name :1.953, object
path /org/freedesktop/Policy
lines 2643-2681/2681 (END)
10:37 a.m.
New subject: Host non-responsive after yum update CentOS7/Ovirt3.6
So, I try to manually start vdsm-network.service and see this,
suggesting I look in upgrade.log:
# systemctl status vdsm-network.service
● vdsm-network.service - Virtual Desktop Server Manager network restoration
Loaded: loaded (/usr/lib/systemd/system/vdsm-network.service;
enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2019-01-19 10:25:02
PST; 8s ago
Process: 6845 ExecStart=/usr/bin/vdsm-tool restore-nets (code=exited,
status=1/FAILURE)
Process: 6837 ExecStartPre=/usr/bin/vdsm-tool --vvverbose --append
--logfile=/var/log/vdsm/upgrade.log upgrade-unified-persistence
(code=exited, status=0/SUCCESS)
Main PID: 6845 (code=exited, status=1/FAILURE)
Jan 19 10:25:02 vmserver2 vdsm-tool[6845]: return
tool_command[cmd]["command"](*args)
Jan 19 10:25:02 vmserver2 vdsm-tool[6845]: File
"/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 41,
in restore_command
Jan 19 10:25:02 vmserver2 vdsm-tool[6845]: exec_restore(cmd)
Jan 19 10:25:02 vmserver2 vdsm-tool[6845]: File
"/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py", line 54,
in exec_restore
Jan 19 10:25:02 vmserver2 vdsm-tool[6845]: raise
EnvironmentError('Failed to restore the persisted networks')
Jan 19 10:25:02 vmserver2 vdsm-tool[6845]: EnvironmentError: Failed to
restore the persisted networks
Jan 19 10:25:02 vmserver2 systemd[1]: vdsm-network.service: main process
exited, code=exited, status=1/FAILURE
Jan 19 10:25:02 vmserver2 systemd[1]: Failed to start Virtual Desktop
Server Manager network restoration.
Jan 19 10:25:02 vmserver2 systemd[1]: Unit vdsm-network.service entered
failed state.
Jan 19 10:25:02 vmserver2 systemd[1]: vdsm-network.service failed.
In upgrade.log I see this:
# cat upgrade.log
MainThread::DEBUG::2016-01-10
10:54:47,615::upgrade::90::upgrade::(apply_upgrade) Running upgrade
upgrade-unified-persistence
MainThread::DEBUG::2016-01-10
10:54:47,623::libvirtconnection::160::root::(get) trying to connect libvirt
MainThread::DEBUG::2016-01-10
10:54:47,639::netinfo::714::root::(_get_gateway) The gateway 192.168.1.1
is duplicated for the device em1
MainThread::DEBUG::2016-01-10 10:54:47,647::utils::669::root::(execCmd)
/sbin/ip route show to 0.0.0.0/0 table main (cwd None)
MainThread::DEBUG::2016-01-10 10:54:47,650::utils::687::root::(execCmd)
SUCCESS: <err> = ''; <rc> = 0
MainThread::DEBUG::2016-01-10
10:54:47,651::unified_persistence::46::root::(run)
upgrade-unified-persistence upgrade persisting networks {} and bondings {}
MainThread::INFO::2016-01-10
10:54:47,651::netconfpersistence::179::root::(_clearDisk) Clearing
/var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/
MainThread::DEBUG::2016-01-10
10:54:47,651::netconfpersistence::187::root::(_clearDisk) No existent
config to clear.
MainThread::INFO::2016-01-10
10:54:47,652::netconfpersistence::179::root::(_clearDisk) Clearing
/var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/
MainThread::DEBUG::2016-01-10
10:54:47,652::netconfpersistence::187::root::(_clearDisk) No existent
config to clear.
MainThread::INFO::2016-01-10
10:54:47,652::netconfpersistence::129::root::(save) Saved new config
RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and
/var/run/vdsm/netconf/bonds/
MainThread::DEBUG::2016-01-10 10:54:47,652::utils::669::root::(execCmd)
/usr/share/vdsm/vdsm-store-net-config unified (cwd None)
MainThread::DEBUG::2016-01-10 10:54:47,672::utils::687::root::(execCmd)
SUCCESS: <err> = 'cp: cannot stat
\xe2\x80\x98/var/run/vdsm/netconf\xe2\x80\x99: No such file or
directory\n'; <rc> = 0
MainThread::DEBUG::2016-01-10
10:54:47,672::upgrade::51::upgrade::(_upgrade_seal) Upgrade
upgrade-unified-persistence successfully performed
I see some references to allowing duplicate gateways if identical on the
mailing list...
11:11 a.m.
New subject: Host non-responsive after yum update CentOS7/Ovirt3.6
If I try to run "vdsm-tool restore-nets", which is what starting the
vdsm-network.service seems to do first, I get the following:
( a large number of lines of the first error)
......
libvirt: XML-RPC error : authentication failed: authentication failed
libvirt: XML-RPC error : authentication failed: authentication failed
libvirt: XML-RPC error : authentication failed: authentication failed
libvirt: XML-RPC error : authentication failed: authentication failed
Traceback (most recent call last):
File "/usr/share/vdsm/vdsm-restore-net-config", line 476, in <module>
restore(args)
File "/usr/share/vdsm/vdsm-restore-net-config", line 434, in restore
_restore_sriov_numvfs()
File "/usr/share/vdsm/vdsm-restore-net-config", line 84, in
_restore_sriov_numvfs
sriov_devices = _get_sriov_devices()
File "/usr/share/vdsm/vdsm-restore-net-config", line 56, in
_get_sriov_devices
devices = hostdev.list_by_caps()
File "/usr/share/vdsm/hostdev.py", line 219, in list_by_caps
libvirt_devices = _get_devices_from_libvirt()
File "/usr/share/vdsm/hostdev.py", line 204, in _get_devices_from_libvirt
for device in libvirtconnection.get().listAllDevices(0))
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 164, in get
password)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 100, in open_connection
return utils.retry(libvirtOpen, timeout=10, sleep=0.2)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 959, in retry
return func()
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 104, in
openAuth
if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: authentication failed: authentication failed
Traceback (most recent call last):
File "/usr/bin/vdsm-tool", line 219, in main
return tool_command[cmd]["command"](*args)
File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py",
line 41, in restore_command
exec_restore(cmd)
File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py",
line 54, in exec_restore
raise EnvironmentError('Failed to restore the persisted networks')
EnvironmentError: Failed to restore the persisted networks
12:53 p.m.
On Sat, Jan 19, 2019 at 9:14 PM Jason Herring <jaherring(a)usa.net> wrote:
If I try to run "vdsm-tool restore-nets", which is what starting the
vdsm-network.service seems to do first, I get the following:
( a large number of lines of the first error)
......
libvirt: XML-RPC error : authentication failed: authentication failed
libvirt: XML-RPC error : authentication failed: authentication failed
libvirt: XML-RPC error : authentication failed: authentication failed
libvirt: XML-RPC error : authentication failed: authentication failed
Traceback (most recent call last):
File "/usr/share/vdsm/vdsm-restore-net-config", line 476, in <module>
restore(args)
File "/usr/share/vdsm/vdsm-restore-net-config", line 434, in restore
_restore_sriov_numvfs()
File "/usr/share/vdsm/vdsm-restore-net-config", line 84, in
_restore_sriov_numvfs
sriov_devices = _get_sriov_devices()
File "/usr/share/vdsm/vdsm-restore-net-config", line 56, in
_get_sriov_devices
devices = hostdev.list_by_caps()
File "/usr/share/vdsm/hostdev.py", line 219, in list_by_caps
libvirt_devices = _get_devices_from_libvirt()
File "/usr/share/vdsm/hostdev.py", line 204, in _get_devices_from_libvirt
for device in libvirtconnection.get().listAllDevices(0))
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 164, in get
password)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 100, in open_connection
return utils.retry(libvirtOpen, timeout=10, sleep=0.2)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 959, in retry
return func()
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 104, in
openAuth
if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirt.libvirtError: authentication failed: authentication failed
This is what happens when libvirt is not configured to work with vdsm.
Can you share your /etc/libvirt/libvirtd.conf?
Would you try `vdsm-tool configure --force` ? (bad for debugging root
cause, but can just fix your issue)
> Traceback (most recent call last):
> File "/usr/bin/vdsm-tool", line 219, in main
> return tool_command[cmd]["command"](*args)
> File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py",
> line 41, in restore_command
> exec_restore(cmd)
> File "/usr/lib/python2.7/site-packages/vdsm/tool/restore_nets.py",
> line 54, in exec_restore
> raise EnvironmentError('Failed to restore the persisted networks')
> EnvironmentError: Failed to restore the persisted networks
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/N5JHE4RECU5...
8:23 a.m.
Il giorno mar 1 gen 2019 alle ore 03:26 <jaherring(a)usa.net> ha scritto:
Hi, I working on a CentOS7 based Ovirt 3.6 system (ovirt-engine/db on
one
machine, two separate ovirt vm hosts) which has been running fine but
mostly ignored for 2-3 of years. Recently it was decided to update the OS
as it was far behind on security updates, so one host was put into
maintenance mode, yum update'd, rebooted, and then it was attempted to take
out of maintenance mode but it's "non-responsive" now.
Hi, please note oVirt 3.6 is no longer supported.
While waiting for 4.3 I would recommend to replace the non-responsive node
with an oVirt 4.2.8 + CentOS 7.6 or oVirt Node 4.2.8 and plan to upgrade
the whole data center.
If I look in /var/log/ovirt-engine/engine.log on the engine machine I see
for this host (vmserver2):
"ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand]
(DefaultQuartzScheduler_Worker-36) [2bc0978d] Command
'GetCapabilitiesVDSCommand(HostName = vmserver2,
VdsIdAndVdsVDSCommandParametersBase:{runAsync='true',
hostId='6725086f-42c0-40eb-91f1-0f2411ea9432',
vds='Host[vmserver2,6725086f-42c0-40eb-91f1-0f2411ea9432]'})' execution
failed: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection
failed" and thereafter more errors. This keep repeating in the log.
In the Ovirt GUI I see multiple occurrences of log entries for the problem
host:
"vmserver2...command failed: Vds timeout occurred"
"vmserver2...command failed: Heartbeat exceeded"
"vmserver2...command failed: internal error: Unknown CPU model
Broadwell-noTSX-IBRS"
Firewall rules look identical to the host which is working normally but
has not been updated.
Any thoughts about how to fix or further troubleshoot this?
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HVP2YCIYGSF...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://red.ht/sig>
1921
days inactive
1943
days old
7 comments
6 participants
participants (6)
-
Dan Kenigsberg -
jaherring＠usa.net -
Jason Herring
-
Sandro Bonazzola -
Yedidyah Bar David -
Николаев Алексей