CI system outage
by Evgheni Dereveanchin
Hello everyone,
We received several reports of Jenkins jobs not running properly. The
reason turned out to be the Jenkins Master which did not start jobs
properly. It has been restarted and is working properly now.
If you have any jobs that failed or did not start at all today - please
re-trigger them or reach out to me for assistance.
--
Regards,
Evgheni Dereveanchin
4 years, 4 months
published master engine is 13 days old
by Yedidyah Bar David
Hi all,
$subject. [1] has
ovirt-engine-4.4.0-0.0.master.20191204120550.git04d5d05.el7.noarch .
Tried to look around, and I have a few notes/questions:
1. Last successful run of [2] is 3 days old, but apparently it wasn't
published. Any idea why?
2. Failed runs of [2] are reported to infra, with emails such as:
[CQ]: 105472, 5 (ovirt-engine) failed "ovirt-master" system tests, but
isn't the failure root cause
Is anyone monitoring these?
Is this the only alerting that CI generates on such failures?
If first is No and second is Yes, then we need someone/something to
start monitoring. This was discussed a lot, but I do not see any
change. Ideally, such alerts should be To'ed or Cc'ed to the author
and reviewers of the patch that CI found to be guilty (which might be
wrong, that's not the point). Do we plan to have something like this?
Any idea when it will be ready?
3. I looked at a few recent failures of [2], specifically [3][4]. Both
seem to have been killed after a timeout, while running
'engine-config'. For [3] that's clear, see [5]:
2019-12-16 17:11:44,766::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:fb6611dc-55bb-4251-aeda-2578b2ec83a2:Get ssh client for
lago-basic-suite-master-engine:
2019-12-16 17:11:44,931::ssh.py::ssh::58::lago.ssh::DEBUG::Running
22e2b6b6 on lago-basic-suite-master-engine: engine-config --set
VdsmUseNmstate=true
2019-12-16 19:55:21,965::cmd.py::exit_handler::921::cli::DEBUG::signal
15 was caught
Can't find stdout/stderr of engine-config, so it's hard to tell if it
outputted anything helpful to understand why it was stuck.
It's hard to tell that about [4], because it has very few artifacts
collected, no idea why, notably no lago.log, but [6] does show:
[36m # initialize_engine: [32mSuccess [0m (in 0:04:00) [0m
[36m # engine_config: [0m [0m [0m
[36m * Collect artifacts: [0m [0m [0m
[36m - [Thread-34] lago-basic-suite-master-engine:
[31mERROR [0m (in 0:00:04) [0m
[36m * Collect artifacts: [31mERROR [0m (in 0:00:04) [0m
[36m # engine_config: [31mERROR [0m (in 2:42:57) [0m
/bin/bash: line 31: 5225 Killed
${_STDCI_TIMEOUT_CMD} "3h" "$script_path" < /dev/null
If I run 'engine-config --set VdsmUseNmstate=true' on my
20191204120550.git04d5d05 engine, it returns quickly.
Tried also adding a repo pointing at last successful run of [7], which
is currently [8], and it prompts me to input a version, probably as a
result of [9]. Ales/Martin, can you please have a look? Thanks.
[1] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el7/noarch/
[2] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/
[3] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/
[4] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/
[5] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17768/arti...
[6] https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/17761/arti...
[7] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/
[8] https://jenkins.ovirt.org/job/ovirt-engine_standard-on-merge/384/
[9] https://gerrit.ovirt.org/105440
--
Didi
4 years, 4 months
Failed to start Vagrant
by Ritesh Chikatwar
Hello,
[root@localhost vagrant]# vagrant reload
Error while connecting to libvirt: Error making a connection to libvirt URI
qemu:///system?no_verify=1&keyfile=/root/.ssh/id_rsa:
Call to virConnectOpen failed: authentication failed: Failed to start SASL
negotiation: -4 (SASL(-4): no mechanism available: No worthy mechs found)
Status Of libvirtd
https://pastebin.com/H7yNqXU7 (It is active)
I cannot see any logs in /var/log/libvirt/qemu/vagrant_default.log
My Vagrant file: https://pastebin.com/JXBd5ci1
Any help would be appreciated.
Ritesh CHikatwar
4 years, 4 months
oVirt 4.4.0 Alpha release is now available for testing
by Sandro Bonazzola
oVirt 4.4.0 Alpha release is now available for testing
The oVirt Project is excited to announce the availability of the alpha
release of oVirt 4.4.0 for testing, as of December 13th, 2019
This release unleashes an altogether more powerful and flexible open source
virtualization solution that encompasses hundreds of individual changes and
a wide range of enhancements across the engine, storage, network, user
interface, and analytics on top of oVirt 4.3.
Important notes before you try it
Please note this is an Alpha release.
The oVirt Project makes no guarantees as to its suitability or usefulness.
This pre-release must not to be used in production, and it is not feature
complete.
In particular, please note that upgrades from 4.3 and future upgrades from
this alpha to the final 4.4 release from this version are not supported.
Some of the features included in oVirt 4.4.0 Alpha require content that
will be available in CentOS Linux 8.1 which is currently being built from
Red Hat Enterprise Linux 8.1.
If you want to have a better experience you can test oVirt 4.4.0 Alpha on
Red Hat Enterprise Linux 8.1.
Known Issues
-
After installation open the Default cluster and hit “Save”, for any
other new Cluster using CPU autodetection the dialog needs to be explicitly
saved after the detection happens, after first host is added. (bug
https://bugzilla.redhat.com/1770697)
-
Migrate, Cluster Upgrade, Export VM buttons cannot be clicked. Fixed in
ovirt-engine-ui-extensions-1.0.11
Installation instructions
For the engine: either use appliance or:
- Install CentOS Linux 7.7 minimal from
http://centos.mirror.garr.it/centos/7.7.1908/isos/x86_64/CentOS-7-x86_64-...
- yum install
https://resources.ovirt.org/pub/yum-repo/ovirt-release44-pre.rpm
- yum update (reboot if needed)
- yum install ovirt-engine
- engine-setup
For the nodes:
Either use oVirt Node ISO or:
- Install CentOS Linux 8.0 from
http://centos.mirror.garr.it/centos/8.0.1905/isos/x86_64/CentOS-8-x86_64-...
; select minimal installation
- yum config-manager --set-enabled PowerTools
- yum install
https://resources.ovirt.org/pub/yum-repo/ovirt-release44-pre.rpm
- yum update (reboot if needed)
- Attach the host to engine and let it be deployed.
What’s new in oVirt 4.4.0 Alpha?
-
Hypervisors based on CentOS Linux 8 (rebuilt from award winning RHEL8),
for both oVirt Node and standalone CentOS Linux hosts
-
Easier network management and configuration flexibility with
NetworkManager
-
VMs based on a more modern Q35 chipset with legacy seabios and UEFI
firmware
-
Support for direct passthrough of local host disks to VMs
-
Live migration improvements for High Performance guests.
-
New Windows Guest tools installer based on WiX framework now moved to
VirtioWin project
-
Dropped support for cluster level prior to 4.2
-
Dropped SDK3 support
-
4K disks support
-
Exporting a VM to a data domain
-
Editing of floating disks
-
Integrating ansible-runner into engine, which allows a more detailed
monitoring of playbooks executed from engine
-
Adding/reinstalling hosts are now completely based on Ansible
-
The OpenStack Neutron Agent cannot be configured by oVirt anymore, it
should be configured by TripleO instead
This release is available now on x86_64 architecture for:
* Red Hat Enterprise Linux 7.7 or later (but < 8.0)
* CentOS Linux (or similar) 7.7 or later (but < 8.0)
This release supports Hypervisor Hosts on x86_64 and ppc64le architectures
for:
* Red Hat Enterprise Linux 8.0 or later
* CentOS Linux (or similar) 8.0 or later
* oVirt Node 4.4 based on CentOS Linux 8.0 (available for x86_64 only)
See the release notes [1] for installation instructions and a list of new
features and bugs fixed.
If you manage more than one oVirt instance, OKD or RDO we also recommend to
try ManageIQ <http://manageiq.org/>.
In such a case, please be sure to take the qc2 image and not the ova image.
Notes:
- oVirt Appliance is already available for CentOS Linux 8
- oVirt Node NG is already available for CentOS Linux 8
Additional Resources:
* Read more about the oVirt 4.4.0 release highlights:
http://www.ovirt.org/release/4.4.0/
* Get more oVirt project updates on Twitter: https://twitter.com/ovirt
* Check out the latest project news on the oVirt blog:
http://www.ovirt.org/blog/
[1] http://www.ovirt.org/release/4.4.0/
[2] http://resources.ovirt.org/pub/ovirt-4.4-pre/iso/
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.*
4 years, 4 months
Vdsm/CI: Failed to synchronize cache for repo 'epel-el8'
by Milan Zamazal
Hi, I've seen this error more than once in Jenkins runs on Vdsm patches
posted to gerrit:
+ python3 tests/profile debuginfo-install debuginfo-install -y python3
Error: Failed to synchronize cache for repo 'epel-el8'
PROFILE {"command": ["debuginfo-install", "-y", "python3"], "cpu": 25.8319700558958, "elapsed": 39.049031019210815, "idrss": 0, "inblock": 0, "isrss": 0, "ixrss": 0, "majflt": 0, "maxrss": 92900, "minflt": 35674, "msgrcv": 0, "msgsnd": 0, "name": "debuginfo-install", "nivcsw": 1096, "nsignals": 0, "nswap": 0, "nvcsw": 904, "oublock": 64848, "start": 1575653575.116399, "status": 1, "stime": 1.135238, "utime": 8.951896}
+ teardown
+ res=1
+ '[' 1 -ne 0 ']'
+ echo '*** err: 1'
*** err: 1
See e.g.
https://jenkins.ovirt.org/blue/organizations/jenkins/vdsm_standard-check-...
Does anybody know what's going on and how to remedy it?
Thanks,
Milan
4 years, 5 months
Vdsm/CI: TimeoutError in VdsmClientTests
by Milan Zamazal
Hi, this seems to be a frequent error in Jenkins run recently:
=================================== FAILURES ===================================
______________________ VdsmClientTests.test_failing_call _______________________
self = <stomprpcclient_test.VdsmClientTests testMethod=test_failing_call>
def test_failing_call(self):
with self._create_client() as client:
with self.assertRaises(ServerError) as ex:
> client.Test.failingCall()
lib/yajsonrpc/stomprpcclient_test.py:144:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def _call(self, namespace, method_name, **kwargs):
"""
Client call method, executes a given command
Args:
namespace (string): namespace name
method_name (string): method name
**kwargs: Arbitrary keyword arguments
Returns:
method result
Raises:
ClientError: in case of an error in the protocol.
TimeoutError: if there is no response after a pre configured time.
ServerError: in case of an error while executing the command
"""
method = namespace + "." + method_name
timeout = kwargs.pop("_timeout", self._default_timeout)
req = yajsonrpc.JsonRpcRequest(
method, kwargs, reqId=str(uuid.uuid4()))
try:
responses = self._client.call(
req, timeout=timeout, flow_id=self._flow_id)
except EnvironmentError as e:
raise ClientError(method, kwargs, e)
if not responses:
> raise TimeoutError(method, kwargs, timeout)
E vdsm.client.TimeoutError: Request Test.failingCall with args {} timed out after 3 seconds
../lib/vdsm/client.py:294: TimeoutError
------------------------------ Captured log call -------------------------------
ERROR vds.dispatcher:betterAsyncore.py:179 uncaptured python exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher ('::1', 47428, 0, 0) at 0x7f48ddc47d10> (<class 'ValueError'>:'b'ept-version:1.2'' contains illegal character ':' [/usr/lib64/python3.7/asyncore.py|readwrite|108] [/usr/lib64/python3.7/asyncore.py|handle_read_event|422] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/protocoldetector.py|handle_read|129] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|handle_socket|413] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|createListener|379] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|StompListener|345] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|init|363] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|set_message_handler|645] [/usr/lib64/python3.7/asyncore.py|handle_read_event|422] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|handle_read|421] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|parse|323] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|_parse_command|245] [/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|decode_value|167])
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled close event
ERROR root:concurrent.py:267 FINISH thread <Thread(JsonRpc (StompReactor), started daemon 139950158767872)> failed
Traceback (most recent call last):
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/common/concurrent.py", line 260, in run
ret = func(*args, **kwargs)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py", line 393, in process_requests
self._reactor.process_requests()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 238, in process_requests
timeout=self._get_timeout(self._map),
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 253, in _get_timeout
interval = disp.next_check_interval()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 99, in next_check_interval
return getattr(self.__impl, "next_check_interval", default_func)()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", line 486, in next_check_interval
self.handle_timeout()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", line 451, in handle_timeout
self._frame_handler.handle_timeout(self)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompclient.py", line 130, in handle_timeout
dispatcher._on_timeout)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py", line 630, in reconnect
AsyncDispatcher(self, self._async_client, count=count))
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 271, in reconnect
dispatcher.create_socket(address, sslctx)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py", line 112, in create_socket
sock = sslctx.wrapSocket(sock)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/sslutils.py", line 113, in wrapSocket
ca_certs=self.ca_certs)
File "/usr/lib64/python3.7/ssl.py", line 1230, in wrap_socket
context.load_verify_locations(ca_certs)
FileNotFoundError: [Errno 2] No such file or directory
_____________________ VdsmClientTests.test_missing_method ______________________
self = <stomprpcclient_test.VdsmClientTests testMethod=test_missing_method>
def test_missing_method(self):
with self._create_client() as client:
with self.assertRaises(ServerError) as ex:
> client.Test.missingMethod()
lib/yajsonrpc/stomprpcclient_test.py:155:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
def _call(self, namespace, method_name, **kwargs):
"""
Client call method, executes a given command
Args:
namespace (string): namespace name
method_name (string): method name
**kwargs: Arbitrary keyword arguments
Returns:
method result
Raises:
ClientError: in case of an error in the protocol.
TimeoutError: if there is no response after a pre configured time.
ServerError: in case of an error while executing the command
"""
method = namespace + "." + method_name
timeout = kwargs.pop("_timeout", self._default_timeout)
req = yajsonrpc.JsonRpcRequest(
method, kwargs, reqId=str(uuid.uuid4()))
try:
responses = self._client.call(
req, timeout=timeout, flow_id=self._flow_id)
except EnvironmentError as e:
raise ClientError(method, kwargs, e)
if not responses:
> raise TimeoutError(method, kwargs, timeout)
E vdsm.client.TimeoutError: Request Test.missingMethod with args {} timed out after 3 seconds
../lib/vdsm/client.py:294: TimeoutError
4 years, 5 months
