failed to mount hosted engine gluster storage - how to debug?
by diego.ercolani＠ssis.sm
Hello, I have an issue probably related to my particular implementation but I think some controls are missing
Here is the story.
I have a cluster of two nodes 4.4.10.3 with an upgraded kernel as the cpu (Ryzen 5) suffer from an incompatibility issue with the kernel provided by 4.4.10.x series.
On each node there are three glusterfs "partitions" in replica mode, one for the hosted_engine, the other two are for user usage.
The third node was an old i3 workstation only used to provide the arbiter partition to the glusterfs cluster.
I installed a new server (ryzen processor) with 4.5.0 and successfully installed glusterfs 10.1 and inserted the arbiter bricks implemented on glusterfs 10.1 while the replica bricks are 8.6 after removing the old i3 provided bricks.
I successfully imported the new node in the ovirt engine (after updating the engine to 4.5)
The problem is that the ovirt-ha-broker doesn't start complaining that is not possible to connect the storage. (I suppose the hosted_engine storage) so I did some digs that I'm going to show here:
####
1. The node seem to be correctly configured:
[root@ovirt-node3 devices]# vdsm-tool validate-config
SUCCESS: ssl configured to true. No conflicts
[root@ovirt-node3 devices]# vdsm-tool configure
Checking configuration status...
libvirt is already configured for vdsm
SUCCESS: ssl configured to true. No conflicts
sanlock is configured for vdsm
Managed volume database is already configured
lvm is configured for vdsm
Current revision of multipath.conf detected, preserving
Running configure...
Done configuring modules to VDSM.
[root@ovirt-node3 devices]# vdsm-tool validate-config
SUCCESS: ssl configured to true. No conflicts
####
2. the node refuses to mount via hosted-engine (same error in broker.log)
[root@ovirt-node3 devices]# hosted-engine --connect-storage
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_setup/connect_storage_server.py", line 30, in <module>
timeout=ohostedcons.Const.STORAGE_SERVER_TIMEOUT,
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/client/client.py", line 312, in connect_storage_server
sserver.connect_storage_server(timeout=timeout)
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_server.py", line 451, in connect_storage_server
'Connection to storage server failed'
RuntimeError: Connection to storage server failed
#####
3. manually mount of glusterfs work correctly
[root@ovirt-node3 devices]# grep storage /etc/ovirt-hosted-engine/hosted-engine.conf
storage=ovirt-node2.ovirt:/gveng
# The following are used only for iSCSI storage
[root@ovirt-node3 devices]#
[root@ovirt-node3 devices]# mount -t glusterfs ovirt-node2.ovirt:/gveng /mnt/tmp/
[root@ovirt-node3 devices]# ls -l /mnt/tmp
total 0
drwxr-xr-x. 6 vdsm kvm 64 Dec 15 19:04 7b8f1cc9-e3de-401f-b97f-8c281ca30482
What else should I control? Thank you and sorry for the long message
Diego
3 years
After attaching the Storage domain, the VMs are disappeared from the VM import
by aminur.rahman＠iongroup.com
Hi,
We're noticing some weird issue while re-attaching the storage domain. After re-attach the storage domain, some VMs are completely missing from the VM Import. Before detaching the storage domain, all the VMs were shutdown gracefully.
I also noticed some disks are exists with no Alias under the disk import on the storage domain and I can't import those disks. Its failed to register the disk with <UNKONOWN> error.
We're using Ovirt 4.2 with multiple Dell hosts in the cluster and Compellent SAN with iSCSI volumes.
Please kindly advise if I am missing anything before detach the storage domain.
Thanks
3 years
Cannot log into oVirt Manager - certificate issue
by Diggy Mc
I cannot log into oVirt Manager. My browser gave me a warning that the site's certificate has expired. Then when I try to log in, I receive the following error message:
"PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed"
How can I fix this problem? In advance, thank you for your help.
hosted-engine: v4.4.8.6
hosts: oVirt Node v4.4.8.3
3 years
can't use vmconsole anymore
by Nathanaël Blanchet
Hi,
I was used to use the vmconsole proxy, but since a while, I'm getting
this issue (currently 4.4.5):
# ssh -t -p 2222 ovirt-vmconsole(a)air.v100.abes.fr connect
ovirt-vmconsole(a)air.v100.abes.fr: Permission denied (publickey).
I found following in the engine.log
2021-04-15 17:55:43,094+02 ERROR
[org.ovirt.engine.core.services.VMConsoleProxyServlet] (default task-4)
[] Error validating ticket: :
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target
at
java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at
java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at
java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at
org.ovirt.engine.core.uutils//org.ovirt.engine.core.uutils.crypto.CertificateChain.buildCertPath(CertificateChain.java:128)
at
org.ovirt.engine.core.uutils//org.ovirt.engine.core.uutils.crypto.ticket.TicketDecoder.decode(TicketDecoder.java:89)
at
deployment.engine.ear.services.war//org.ovirt.engine.core.services.VMConsoleProxyServlet.validateTicket(VMConsoleProxyServlet.java:175)
at
deployment.engine.ear.services.war//org.ovirt.engine.core.services.VMConsoleProxyServlet.doPost(VMConsoleProxyServlet.java:225)
The user key is the good one, I use the same with my other engines and I
can successfully connect to vm consoles.
Thank you for helping
--
Nathanaël Blanchet
Supervision réseau
SIRE
227 avenue Professeur-Jean-Louis-Viala
34193 MONTPELLIER CEDEX 5
Tél. 33 (0)4 67 54 84 55
Fax 33 (0)4 67 54 84 14
blanchet(a)abes.fr
3 years
VM HostedEngine is down with error
by souvaliotimaria＠mail.com
Hello everyone,
I have a replica 2 + arbiter installation and this morning the Hosted Engine gave the following error on the UI and resumed on a different node (node3) than the one it was originally running(node1). (The original node has more memory than the one it ended up, but it had a better memory usage percentage at the time). Also, the only way I discovered the migration had happened and there was an Error in Events, was because I logged in the web interface of ovirt for a routine inspection. Βesides that, everything was working properly and still is.
The error that popped is the following:
VM HostedEngine is down with error. Exit message: internal error: qemu unexpectedly closed the monitor:
2020-09-01T06:49:20.749126Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2020-09-01T06:49:20.927274Z qemu-kvm: -device virtio-blk-pci,iothread=iothread1,scsi=off,bus=pci.0,addr=0x7,drive=drive-ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,id=ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2,bootindex=1,write-cache=on: Failed to get "write" lock
Is another process using the image?.
Which from what I could gather concerns the following snippet from the HostedEngine.xml and it's the virtio disk of the Hosted Engine:
<disk type='file' device='disk' snapshot='no'>
<driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads' iothread='1'/>
<source file='/var/run/vdsm/storage/80f6e393-9718-4738-a14a-64cf43c3d8c2/d5de54b6-9f8e-4fba-819b-ebf6780757d2/a48555f4-be23-4467-8a54-400ae7baf9d7'>
<seclabel model='dac' relabel='no'/>
</source>
<target dev='vda' bus='virtio'/>
<serial>d5de54b6-9f8e-4fba-819b-ebf6780757d2</serial>
<alias name='ua-d5de54b6-9f8e-4fba-819b-ebf6780757d2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
I've tried looking into the logs and the sar command but I couldn't find anything to relate with the above errors and determining the reason for it to happen. Is this a Gluster or a QEMU problem?
The Hosted Engine was manually migrated five days before on node1.
Is there a standard practice I could follow to determine what happened and secure my system?
Thank you very much for your time,
Maria Souvalioti
3 years
LACP across multiple switches
by Jorge Visentini
Hi all.
Is it possible to configure oVirt for work with two NICs in bond/LACP
across two switches, according to the image below?
[image: LACP_Across_Two_Switchs.png]
Thank you all.
You guys do a wonderful job.
--
Att,
Jorge Visentini
+55 55 98432-9868
3 years
new host addition to OVN cluster fails with Connectivity check failed, rolling back
by ravi k
Hello,
I have a a oVirt 4.3 installation with two clusters. One of the cluster has switch type as OVS. I'm trying to add a second host to this cluster. I did a clean install of the OS, configured bond0 and bond0.2306 as the VLAN interface. I was able to add the host to the cluster.
When I go to setup networks and drag ovirtmgmt onto bond0, I notice in the host that it was able to create the vdsm and br_int bridges, create ovirtmgmt interface and assign the IP on top of it. However I also notice that the bond0.2306 interface also exists with the ip assigned. Then it rolls back the config removing the bridges. I checked the supervdsm log and see that it's rolling back because "connectivity::48::root::(check) Connectivity check failed, rolling back"
I'm pasting the relevant lines from supervdsm below
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:53,381::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return network_caps with {'bridges': {}, 'bondings': {'bond0': {'ipv4addrs': [], 'active_slave': '', 'ad_aggregator_id': '1', 'netmask': '', 'ad_partner_mac': '44:38:39:ff:01:33', 'hwaddr': '7c:d3:0a:60:e9:48', 'speed': 20000, 'gateway': '', 'ipv6autoconf': True, 'addr': '', 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '9000', 'dhcpv4': False, 'switch': 'legacy', 'ipv4defaultroute': False, 'slaves': ['eno1', 'eno2'], 'ipv6gateway': '::', 'opts': {'mode': '4'}}}, 'nameservers': ['10.222.0.6', '10.333.0.6'], 'nics': {'eno1': {'permhwaddr': '7c:d3:0a:60:e9:48', 'ipv6autoconf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:48', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': ''}, 'eno2': {'permhwaddr': '7c:d3:0a:60:e9:49', 'ipv6autoco
nf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'ad_aggregator_id': '1', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:48', 'mtu': '9000', 'ipv6gateway': '::', 'gateway': ''}, 'eno3': {'ipv6autoconf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:4a', 'ipv6gateway': '::', 'gateway': ''}, 'eno4': {'ipv6autoconf': True, 'addr': '', 'speed': 10000, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7c:d3:0a:60:e9:4b', 'ipv6gateway': '::', 'gateway': ''}, 'enp0s20f0u1u6': {'ipv6autoconf': True, 'addr': '', 'speed': 0, 'dhcpv6': False, 'ipv6addrs': [], 'mtu': '1500', 'dhcpv4': False, 'netmask': '', 'ipv4defaultroute': False, 'ipv4addrs': [], 'hwaddr': '7e:d3:0a:60:e9:4f', 'ipv6gateway': '
::', 'gateway': ''}}, 'supportsIPv6': True, 'vlans': {'bond0.2306': {'iface': 'bond0', 'ipv6autoconf': True, 'addr': '10.119.6.237', 'dhcpv6': False, 'ipv6addrs': [], 'vlanid': 2306, 'mtu': '9000', 'dhcpv4': False, 'netmask': '255.255.255.0', 'ipv4defaultroute': True, 'ipv4addrs': ['10.119.6.237/24'], 'ipv6gateway': '::', 'gateway': '10.119.6.1'}}, 'networks': {}}
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,243::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_pti with () {}
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,243::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_pti with -1
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,244::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_retp with () {}
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,244::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_retp with -1
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,244::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_ibrs with () {}
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,245::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_ibrs with 1
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,245::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call get_ssbd with () {}
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,245::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return get_ssbd with -1
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,246::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call check_qemu_conf_contains with ('vnc_tls', '1') {}
MainProcess|jsonrpc/1::DEBUG::2022-04-21 11:26:54,250::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return check_qemu_conf_contains with True
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,304::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call setupNetworks with ({u'ovirtmgmt': {u'ipv6autoconf': True, u'vlan': u'2306', u'ipaddr': u'10.119.6.237', u'switch': u'ovs', u'mtu': 9000, u'bonding': u'bond0', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True}}, {}, {u'connectivityCheck': u'true', u'connectivityTimeout': 120, u'commitOnSuccess': False}) {}
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,305::api::211::root::(setupNetworks) Setting up network according to configuration: networks:{u'ovirtmgmt': {u'ipv6autoconf': True, u'vlan': u'2306', u'ipaddr': u'10.119.6.237', u'bonding': u'bond0', u'mtu': 9000, u'switch': u'ovs', u'dhcpv6': False, u'STP': u'no', u'bridged': u'true', u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True}}, bondings:{}, options:{u'connectivityCheck': u'true', u'connectivityTimeout': 120, u'commitOnSuccess': False}
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,323::cmdutils::133::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,342::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,408::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,408::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,445::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,459::netconfpersistence::69::root::(setBonding) Adding bond0({'nics': ['eno1', 'eno2'], 'switch': 'legacy', 'options': 'mode=4'})
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,460::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({u'ipv6autoconf': True, 'nameservers': ['10.222.0.6', '10.333.0.6'], u'vlan': 2306, u'ipaddr': u'10.119.6.237', u'switch': u'ovs', u'mtu': 9000, u'bonding': u'bond0', u'dhcpv6': False, 'stp': False, u'bridged': True, u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True, 'bootproto': 'none'})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,462::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,909::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:55,909::hooks::114::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,911::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,911::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,947::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,953::configurator::265::root::(_remove_networks) Removing networks: []
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,955::setup::41::root::(remove_bonds) Removing bonds: []
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,958::ifacquire::70::root::(acquire) Acquiring ifaces: set([u'eno1', u'eno2', u'bond0'])
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:55,959::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno1 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:56,606::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:56,609::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno2 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:57,268::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:57,271::cmdutils::133::root::(exec_cmd) /sbin/ifdown bond0 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,940::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,940::setup::61::root::(edit_bonds) Editing bonds: [u'bond0']
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:58,946::netconfpersistence::69::root::(setBonding) Adding bond0({'nics': [], 'switch': u'ovs', 'options': {u'mode': '4'}})
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:58,949::sysfs_driver::104::root::(set_options) Bond bond0 options set: {u'mode': u'4'}.
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,950::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno1 down (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:58,956::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:59,000::sysfs_driver::78::root::(add_slaves) Slave eno1 has been added to bond bond0.
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,001::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno2 down (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,009::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:59,053::sysfs_driver::78::root::(add_slaves) Slave eno2 has been added to bond bond0.
MainProcess|jsonrpc/2::INFO::2022-04-21 11:26:59,053::netconfpersistence::69::root::(setBonding) Adding bond0({u'hwaddr': u'7c:d3:0a:60:e9:48', u'nics': [u'eno1', u'eno2'], u'switch': u'ovs', u'options': u'mode=4'})
netlink/events::DEBUG::2022-04-21 11:26:59,055::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d610>>, args=(), kwargs={})
netlink/events::DEBUG::2022-04-21 11:26:59,057::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363781850880)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe5e9550>>, args=(), kwargs={})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,058::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev bond0 up (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,067::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:26:59,071::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363781850880)>
netlink/events::DEBUG::2022-04-21 11:26:59,073::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363781850880)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe5e91d0>>, args=(), kwargs={})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,074::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno1 up (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,081::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:26:59,083::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363781850880)>
netlink/events::DEBUG::2022-04-21 11:26:59,084::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363781850880)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe5e91d0>>, args=(), kwargs={})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,085::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno2 up (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,090::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:26:59,092::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363781850880)>
netlink/events::DEBUG::2022-04-21 11:26:59,093::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)>
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,219::cmdutils::133::root::(exec_cmd) /sbin/ip addr flush dev eno1 scope global (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,225::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,340::cmdutils::133::root::(exec_cmd) /sbin/ip addr flush dev eno2 scope global (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,346::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,346::setup::92::root::(add_bonds) Creating bonds: []
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,346::configurator::274::root::(_add_networks) Adding networks: [u'ovirtmgmt']
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,347::ifacquire::70::root::(acquire) Acquiring ifaces: set([u'bond0'])
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:26:59,348::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno1 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,004::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,006::cmdutils::133::root::(exec_cmd) /sbin/ifdown eno2 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,657::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:00,660::cmdutils::133::root::(exec_cmd) /sbin/ifdown bond0 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,360::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,361::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- add-br vdsmbr_TtLDFxN1 -- set Bridge vdsmbr_TtLDFxN1 other-config:hwaddr="7a:b2:a6:5b:96:78" -- add-port vdsmbr_TtLDFxN1 bond0 -- set Port bond0 other_config:vdsm_level=southbound -- add-port vdsmbr_TtLDFxN1 ovirtmgmt -- set Port ovirtmgmt other_config:vdsm_level=northbound -- set Interface ovirtmgmt type=internal -- set Port ovirtmgmt tag=2306 -- set Interface ovirtmgmt mtu_request=9000 -- set Interface bond0 mtu_request=9000
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,361::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- add-br vdsmbr_TtLDFxN1 -- set Bridge vdsmbr_TtLDFxN1 'other-config:hwaddr="7a:b2:a6:5b:96:78"' -- add-port vdsmbr_TtLDFxN1 bond0 -- set Port bond0 other_config:vdsm_level=southbound -- add-port vdsmbr_TtLDFxN1 ovirtmgmt -- set Port ovirtmgmt other_config:vdsm_level=northbound -- set Interface ovirtmgmt type=internal -- set Port ovirtmgmt tag=2306 -- set Interface ovirtmgmt mtu_request=9000 -- set Interface bond0 mtu_request=9000 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,704::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:27:02,705::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({u'ipv6autoconf': True, u'nameservers': [u'10.129.0.60', u'10.229.0.60'], u'vlan': 2306, u'ipaddr': u'10.119.6.237', u'bonding': u'bond0', u'mtu': 9000, u'switch': u'ovs', u'dhcpv6': False, u'stp': False, u'bridged': True, u'netmask': u'255.255.255.0', u'gateway': u'10.119.6.1', u'defaultRoute': True, u'bootproto': u'none'})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,705::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,706::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,745::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,753::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . external-ids:ovn-bridge-mappings="ovirtmgmt:vdsmbr_TtLDFxN1"
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,753::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings="ovirtmgmt:vdsmbr_TtLDFxN1"' (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,794::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:27:02,796::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,797::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev ovirtmgmt up (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,805::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:27:02,807::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)>
netlink/events::DEBUG::2022-04-21 11:27:02,808::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,809::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev bond0 up (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,816::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:27:02,819::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)>
netlink/events::DEBUG::2022-04-21 11:27:02,820::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,821::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno1 up (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,867::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:27:02,870::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)>
netlink/events::DEBUG::2022-04-21 11:27:02,871::concurrent::258::root::(run) START thread <Thread(netlink/events, started daemon 140363790243584)> (func=<bound method Monitor._scan of <vdsm.network.netlink.monitor.Monitor object at 0x7fa8fe64d590>>, args=(), kwargs={})
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,872::cmdutils::133::root::(exec_cmd) /sbin/ip link set dev eno2 up (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,920::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
netlink/events::DEBUG::2022-04-21 11:27:02,922::concurrent::261::root::(run) FINISH thread <Thread(netlink/events, stopped daemon 140363790243584)>
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,923::cmdutils::133::root::(exec_cmd) /sbin/ip addr flush dev ovirtmgmt scope global (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,930::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,931::cmdutils::133::root::(exec_cmd) /sbin/ip -4 addr add dev ovirtmgmt 10.119.6.237/255.255.255.0 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,938::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,938::cmdutils::133::root::(exec_cmd) /sbin/ip -4 route add default via 10.119.6.1 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,945::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:27:02,947::connectivity::46::root::(check) Checking connectivity...
MainProcess|jsonrpc/3::DEBUG::2022-04-21 11:27:10,538::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call ksmTune with ({u'run': 0, u'merge_across_nodes': 1},) {}
MainProcess|jsonrpc/3::DEBUG::2022-04-21 11:27:10,538::supervdsm_server::106::SuperVdsm.ServerCallback::(wrapper) return ksmTune with None
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:04,075::connectivity::48::root::(check) Connectivity check failed, rolling back
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,075::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 136 seconds (max pending: 3)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,075::ifacquire::76::root::(_rollback) Acquiring transaction failed, reverting ifaces: [u'eno1', u'eno2', u'bond0']
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,077::cmdutils::133::root::(exec_cmd) /sbin/ifup eno1 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,485::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,488::cmdutils::133::root::(exec_cmd) /sbin/ifup eno2 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,897::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:04,900::cmdutils::133::root::(exec_cmd) /sbin/ifup bond0 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,045::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = './network-functions: line 119: .: /etc/sysconfig/network-scripts/: is a directory\n'; <rc> = 0
MainProcess|jsonrpc/2::WARNING::2022-04-21 11:29:06,046::netconfpersistence::294::root::(__exit__) Failed setup transaction,reverting to last known good network.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 249, in _setup_ovs
connectivity.check(options)
File "/usr/lib/python2.7/site-packages/vdsm/network/connectivity.py", line 50, in check
'connectivity check failed')
ConfigNetworkError: (10, 'connectivity check failed')
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,047::api::211::root::(setupNetworks) Setting up network according to configuration: networks:{u'ovirtmgmt': {'remove': True}}, bondings:{u'bond0': {'remove': True}}, options:{'connectivityCheck': 0, 'inRollback': True}
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,061::cmdutils::133::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,079::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,153::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,153::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,190::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,326::netconfpersistence::58::root::(setNetwork) Adding network ovirtmgmt({'bonding': u'bond0', 'ipv6autoconf': True, 'nameservers': [], 'vlan': 2306, 'ipaddr': '10.119.6.237', 'switch': 'ovs', 'mtu': 9000, 'netmask': '255.255.255.0', 'dhcpv6': False, 'stp': False, 'bridged': True, 'defaultRoute': False, 'bootproto': 'none'})
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,327::netconfpersistence::69::root::(setBonding) Adding bond0({'nics': ['eno1', 'eno2'], 'switch': 'legacy', 'options': 'mode=4'})
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,327::netconfpersistence::63::root::(removeNetwork) Removing network ovirtmgmt
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,334::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,911::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:06,911::hooks::114::root::(_runHooksDir) /usr/libexec/vdsm/hooks/before_network_setup/50_fcoe: rc=0 err=
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,918::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,918::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:06,956::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,091::configurator::265::root::(_remove_networks) Removing networks: [u'ovirtmgmt']
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,219::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set Interface bond0 mtu_request=1500
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,220::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set Interface bond0 mtu_request=1500 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,425::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,426::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- del-port ovirtmgmt -- del-port bond0 -- del-br vdsmbr_TtLDFxN1
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,426::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- del-port ovirtmgmt -- del-port bond0 -- del-br vdsmbr_TtLDFxN1 (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,949::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,950::netconfpersistence::65::root::(removeNetwork) Network ovirtmgmt not found for removal
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:07,951::setup::41::root::(remove_bonds) Removing bonds: [u'bond0']
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,165::sysfs_driver::69::root::(destroy) Bond bond0 has been destroyed.
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,166::netconfpersistence::76::root::(removeBonding) bond0 not found for removal
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::ifacquire::70::root::(acquire) Acquiring ifaces: set([])
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::setup::61::root::(edit_bonds) Editing bonds: []
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::setup::92::root::(add_bonds) Creating bonds: []
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::configurator::274::root::(_add_networks) Adding networks: []
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,172::ifacquire::70::root::(acquire) Acquiring ifaces: set([])
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,173::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,173::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- list Bridge -- list Port -- list Interface (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,211::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,217::vsctl::68::root::(commit) Executing commands: /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . external-ids:ovn-bridge-mappings=""
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,217::cmdutils::133::root::(exec_cmd) /usr/bin/ovs-vsctl --timeout=5 --oneline --format=json -- set open . 'external-ids:ovn-bridge-mappings=""' (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,247::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,249::netconfpersistence::231::root::(_clearDisk) Clearing netconf: /var/lib/vdsm/staging/netconf
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,253::netconfpersistence::181::root::(save) Saved new config RunningConfig({}, {}, {}) to [/var/lib/vdsm/staging/netconf/nets,/var/lib/vdsm/staging/netconf/bonds,/var/lib/vdsm/staging/netconf/devices]
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,256::commands::198::root::(execCmd) /usr/bin/taskset --cpu-list 0-95 /usr/libexec/vdsm/hooks/after_network_setup/30_ethtool_options (cwd None)
MainProcess|jsonrpc/2::DEBUG::2022-04-21 11:29:08,434::commands::219::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0
MainProcess|jsonrpc/2::INFO::2022-04-21 11:29:08,434::hooks::114::root::(_runHooksDir) /usr/libexec/vdsm/hooks/after_network_setup/30_ethtool_options: rc=0 err=
MainProcess|jsonrpc/2::ERROR::2022-04-21 11:29:08,436::supervdsm_server::103::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/supervdsm_server.py", line 101, in wrapper
res = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 228, in setupNetworks
_setup_networks(networks, bondings, options, net_info)
File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 249, in _setup_networks
networks, bondings, options, net_info, in_rollback)
File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 157, in _rollback
six.reraise(excType, value, tb)
File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 140, in _rollback
yield
File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 249, in _setup_networks
networks, bondings, options, net_info, in_rollback)
File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 139, in setup
_setup(networks, bondings, options, in_rollback, net_info)
File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 159, in _setup
_setup_ovs(ovs_nets, ovs_bonds, options, net_info, in_rollback)
File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch/configurator.py", line 249, in _setup_ovs
connectivity.check(options)
File "/usr/lib/python2.7/site-packages/vdsm/network/netconfpersistence.py", line 295, in __exit__
raise ne.RollbackIncomplete(config_diff, ex_type, ex_value)
ConfigNetworkError: (10, 'connectivity check failed')
MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,062::supervdsm_server::99::SuperVdsm.ServerCallback::(wrapper) call network_caps with () {}
MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,062::logutils::319::root::(_report_stats) ThreadedHandler is ok in the last 1935 seconds (max pending: 3)
MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,076::cmdutils::133::root::(exec_cmd) /sbin/tc qdisc show (cwd None)
MainProcess|jsonrpc/5::DEBUG::2022-04-21 12:01:20,098::cmdutils::141::root::(exec_cmd) SUCCESS: <err> = ''; <rc> = 0
Regards,
Ravi
3 years, 1 month
Deployment suddenly fails at engine check
by Harry O
Hi,
After the new update, my deployment fails at engine check.
What can I do to debug?
[ INFO ] skipping: [localhost]
[ INFO ] TASK [ovirt.ovirt.engine_setup : Check if Engine health page is up]
[ ERROR ] fatal: [localhost -> 192.168.222.12]: FAILED! => {"attempts": 30, "changed": false, "connection": "close", "content": "<html><head><title>Error</title></head><body>500 - Internal Server Error</body></html>", "content_encoding": "identity", "content_length": "86", "content_type": "text/html; charset=UTF-8", "date": "Fri, 22 Apr 2022 16:02:04 GMT", "elapsed": 0, "msg": "Status code was 500 and not [200]: HTTP Error 500: Internal Server Error", "redirected": false, "server": "Apache/2.4.37 (centos) OpenSSL/1.1.1k mod_auth_gssapi/1.6.1 mod_wsgi/4.6.4 Python/3.6", "status": 500, "url": "http://localhost/ovirt-engine/services/health"}
[ INFO ] TASK [ovirt.ovirt.engine_setup : Clean temporary files]
[ INFO ] changed: [localhost -> 192.168.222.12]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine]
[ INFO ] changed: [localhost -> 192.168.222.12]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set destination directory path]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Create destination directory]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Find the local appliance image]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set local_vm_disk_path]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Copy engine logs]
[ INFO ] changed: [localhost]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
3 years, 1 month
