Unable to run create-brick.yml from ansible runner dev
by Kaustav Majumder
Hi all,
I am trying to run create-brick playbook from ansible-runner integrated
inside ovirt-engine but it throws errors as
```
Running command: CreateBrickCommand internal: false. Entities affected :
ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: VDSAction group
MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 ERROR
[org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default
task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed to
execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR
[org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29)
[5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible create
brick role. Please check logs for more details:
/home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log
2020-03-27 15:43:07,519+05 ERROR
[org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29)
[5ff56203-33ca-424b-9137-491cf1ef2cd3] Command
'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed:
EngineException: Failed to execute Ansible create brick role. Please check
logs for more details:
/home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log
(Failed with error GeneralException and code 100) 2020-03-27
15:43:07,526+05 ERROR
[org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29)
[5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back for command
'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27
15:43:07,530+05 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID:
CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host
10.70.35.17 of cluster Default. ```
*The corresponding log is not created*
kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± less
/home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log
/home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log:
No such file or directory
*checking ansible-runner status gives*
*[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s 130 ±
systemctl status ansible-runner-service.service ●
ansible-runner-service.service - Ansible Runner Service Loaded: loaded
(/usr/lib/systemd/system/ansible-runner-service.service; disabled; vendor
preset: disabled) Active: active (running) since Fri 2020-03-27 15:42:56
IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: 4915)
Memory: 62.1M CGroup: /system.slice/ansible-runner-service.service
├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b localhost:50001 -w 2
runner_service.wsgi:application ├─30112 /usr/bin/python3
/usr/bin/gunicorn-3 -b localhost:50001 -w 2
runner_service.wsgi:application └─30123 /usr/bin/python3
/usr/bin/gunicorn-3 -b localhost:50001 -w 2
runner_service.wsgi:applicationMar 27 15:42:58 localhost.localdomain
gunicorn-3[30097]: Request received, content-type :NoneMar 27 15:42:58
localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET
/api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01
localhost.localdomain gunicorn-3[30097]: Request received, content-type
:NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 -
GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03
localhost.localdomain gunicorn-3[30097]: Request received, content-type
:NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 -
GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07
localhost.localdomain gunicorn-3[30097]: Request received, content-type
:NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 -
GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07
localhost.localdomain gunicorn-3[30097]: Request received, content-type
:application/json; charset=UTF-8Mar 27 15:43:07 localhost.localdomain
gunicorn-3[30097]: 127.0.0.1 - POST /api/v1/playbooks/create-brick.yml*
*Is there a way to find the root cause for this?*
--
Thanks,
Kaustav Majumder
4 years, 7 months
[VDSM] VdsmClientTests.test_event_handler - flaky test?
by Nir Soffer
I had 2 of these unrelated failures today. Would be nice to mark this
test as broken on CI.
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/19353//artifact/c...
https://jenkins.ovirt.org/job/vdsm_standard-check-patch/19354//artifact/c...
=================================== FAILURES ===================================
______________________ VdsmClientTests.test_event_handler ______________________
self = <stomprpcclient_test.VdsmClientTests testMethod=test_event_handler>
def test_event_handler(self):
with self._create_client() as client:
event_queue = queue.Queue()
sub_id = client.subscribe(EVENT_TOPIC, event_queue)
> client.Test.sendEvent()
lib/yajsonrpc/stomprpcclient_test.py:215:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <stomprpcclient_test._MockedClient object at 0x7fe19f6db1d0>
namespace = 'Test', method_name = 'sendEvent', kwargs = {}
method = 'Test.sendEvent', timeout = 3
req = {'jsonrpc': '2.0', 'method': 'Test.sendEvent', 'params': {},
'id': 'c3985771-6f77-4431-9ed6-15082063a2d5'}
responses = None
def _call(self, namespace, method_name, **kwargs):
"""
Client call method, executes a given command
Args:
namespace (string): namespace name
method_name (string): method name
**kwargs: Arbitrary keyword arguments
Returns:
method result
Raises:
ClientError: in case of an error in the protocol.
TimeoutError: if there is no response after a pre configured time.
ServerError: in case of an error while executing the command
"""
method = namespace + "." + method_name
timeout = kwargs.pop("_timeout", self._default_timeout)
req = yajsonrpc.JsonRpcRequest(
method, kwargs, reqId=str(uuid.uuid4()))
try:
responses = self._client.call(
req, timeout=timeout, flow_id=self._flow_id)
except EnvironmentError as e:
raise ClientError(method, kwargs, e)
if not responses:
> raise TimeoutError(method, kwargs, timeout)
E vdsm.client.TimeoutError: Request Test.sendEvent with args
{} timed out after 3 seconds
../lib/vdsm/client.py:294: TimeoutError
------------------------------ Captured log call -------------------------------
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled read event
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled write event
ERROR vds.dispatcher:betterAsyncore.py:179 uncaptured python
exception, closing channel <yajsonrpc.betterAsyncore.Dispatcher
('::1', 36398, 0, 0) at 0x7fe19f6db208> (<class
'ValueError'>:'b'ept-version:1.2'' contains illegal character ':'
[/usr/lib64/python3.6/asyncore.py|readwrite|108]
[/usr/lib64/python3.6/asyncore.py|handle_read_event|423]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/protocoldetector.py|handle_read|129]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|handle_socket|413]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|add_socket|54]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|createListener|379]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|StompListener|345]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|__init__|47]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|switch_implementation|86]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py|init|363]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/rpc/bindingjsonrpc.py|_onAccept|57]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|set_message_handler|645]
[/usr/lib64/python3.6/asyncore.py|handle_read_event|423]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|handle_read|71]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py|_delegate_call|168]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|handle_read|421]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|parse|323]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|_parse_command|245]
[/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py|decode_value|167])
WARNING vds.dispatcher:betterAsyncore.py:179 unhandled close event
ERROR root:concurrent.py:267 FINISH thread <Thread(JsonRpc
(StompReactor), started daemon 140606575712000)> failed
Traceback (most recent call last):
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/common/concurrent.py",
line 260, in run
ret = func(*args, **kwargs)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompserver.py",
line 393, in process_requests
self._reactor.process_requests()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
line 238, in process_requests
timeout=self._get_timeout(self._map),
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
line 253, in _get_timeout
interval = disp.next_check_interval()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
line 99, in next_check_interval
return getattr(self.__impl, "next_check_interval", default_func)()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py",
line 486, in next_check_interval
self.handle_timeout()
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py",
line 451, in handle_timeout
self._frame_handler.handle_timeout(self)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stompclient.py",
line 130, in handle_timeout
dispatcher._on_timeout)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/stomp.py",
line 630, in reconnect
AsyncDispatcher(self, self._async_client, count=count))
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
line 271, in reconnect
dispatcher.create_socket(address, sslctx)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/yajsonrpc/betterAsyncore.py",
line 112, in create_socket
sock = sslctx.wrapSocket(sock)
File "/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/sslutils.py",
line 113, in wrapSocket
ca_certs=self.ca_certs)
File "/usr/lib64/python3.6/ssl.py", line 1114, in wrap_socket
ciphers=ciphers)
File "/usr/lib64/python3.6/ssl.py", line 704, in __init__
self._context.load_verify_locations(ca_certs)
FileNotFoundError: [Errno 2] No such file or directory
=============================== warnings summary ===============================
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334:
PytestUnknownMarkWarning: Unknown pytest.mark.stress - is this a typo?
You can register custom marks to avoid this warning - for details,
see https://docs.pytest.org/en/latest/mark.html
PytestUnknownMarkWarning,
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334:
PytestUnknownMarkWarning: Unknown pytest.mark.slow - is this a typo?
You can register custom marks to avoid this warning - for details, see
https://docs.pytest.org/en/latest/mark.html
PytestUnknownMarkWarning,
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:79
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:79:
DeprecationWarning: invalid escape sequence \d
_SSH_AUTH_RE = b'(SSH_AUTH_SOCK)=([^;]+).*;\nSSH_AGENT_PID=(\d+)'
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:1421
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/v2v.py:1421:
DeprecationWarning: invalid escape sequence \^
'(?P<m2_base>[0-9]+){sp}\^{sp}(?P<m2_exp>{exp}))'.format(
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/.tox/lib/lib/python3.6/site-packages/_pytest/mark/structures.py:334:
PytestUnknownMarkWarning: Unknown pytest.mark.xpass - is this a typo?
You can register custom marks to avoid this warning - for details, see
https://docs.pytest.org/en/latest/mark.html
PytestUnknownMarkWarning,
tests/lib/osinfo_test.py::test_package_versions
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/osinfo.py:284:
UnicodeWarning: decode() called on unicode string, see
https://bugzilla.redhat.com/show_bug.cgi?id=1693751
'version': mi['version'].decode('utf-8'),
tests/lib/osinfo_test.py::test_package_versions
/home/jenkins/workspace/vdsm_standard-check-patch/vdsm/lib/vdsm/osinfo.py:285:
UnicodeWarning: decode() called on unicode string, see
https://bugzilla.redhat.com/show_bug.cgi?id=1693751
'release': mi['release'].decode('utf-8'),
-- Docs: https://docs.pytest.org/en/latest/warnings.html
----------- coverage: platform linux, python 3.6.8-final-0 -----------
Coverage HTML written to dir htmlcov-lib
========================== slowest 10 test durations ===========================
18.29s call
tests/lib/yajsonrpc/stomprpcclient_test.py::VdsmClientTests::test_event_handler
2.96s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client_concurrency(True)
2.90s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client_concurrency(False)
2.75s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client(False)
2.63s call tests/lib/protocoldetector_test.py::AcceptorTests::test_reject_very_slow_client(True)
1.90s call tests/pywatch_test.py::TestPyWatch::test_timeout_backtrace
1.44s call tests/pywatch_test.py::TestPyWatch::test_timeout_output
1.19s call tests/pywatch_test.py::TestPyWatch::test_kill_grandkids
0.93s call tests/lib/protocoldetector_test.py::AcceptorTests::test_detect_slow_client(True)
0.90s call tests/lib/yajsonrpc/stomp_test.py::StompTests::test_echo(16384,
False)
=========================== short test summary info ============================
4 years, 7 months
packaging: setup: start ovn services on restore - broke make install-dev on fc30?
by Liran Rotenberg
Hi all,
Running on fc30 with engine, suddenly when rebasing or being on master
branch the command:
$ make install-dev SKIP_CHECKS=1 PREFIX=$HOME/ovirt-engine
DEV_EXTRA_BUILD_FLAGS_GWT_DEFAULTS="-Dgwt.userAgent=safari" -j8
Didn't pass, it showed to me no errors, after build success, I got:
touch "tmp.built"
make[2]: Leaving directory '/home/lrotenbe/git/ovirt-engine'
make[1]: Leaving directory '/home/lrotenbe/git/ovirt-engine'
make: *** [Makefile:584: all-dev] Error 2
$ make test
Did pass.
Burning time to figure out the situation, I found out that this[1] patch
caused it.
Any idea why did it happen? Or what possibly we can do?
[1] https://gerrit.ovirt.org/c/107847/
Regards,
Liran.
4 years, 7 months
AddClusterCommand is failing
by Piotr Kliczewski
Hi,
I posted revert [0] of commit [1] due to the issue with AddClusterCommand
which fails for Adding new Kubevirt provider with:
ERROR: null value in column "bios_type" violates not-null constraint
I am happy to see different solution than revet but at the moment add
cluster is not working.
Thanks,
Piotr
[0] https://gerrit.ovirt.org/#/c/107928/
[1] 6b1d1e3099879d6991401dd545338f7a856fc426
4 years, 7 months
Fwd: [CentOS-devel] Last call for video content: CentOS @ Red Hat Summit
by Sandro Bonazzola
FYI
---------- Forwarded message ---------
Da: Rich Bowen <rbowen(a)redhat.com>
Date: mar 24 mar 2020 alle ore 13:50
Subject: [CentOS-devel] Last call for video content: CentOS @ Red Hat Summit
To: The CentOS developers mailing list. <centos-devel(a)centos.org>
Next month, Red Hat Summit will be happening, as a virtual event.
They're expecting 30,000 "attendees", and we're delivering content via
videos.
If you'd like to present your work in the CentOS project to 30,000
interested people, PLEASE get in touch with me ASAP, and we can schedule
a short video interview for inclusion in that event. Any topic is fair
game, if it's related to the CentOS project.
The deadline for delivering these finished videos is Monday, March 30th,
so I really need to hear from you today or tomorrow if we want to make
this happen.
Thanks!
--
Rich Bowen - rbowen(a)redhat.com
@CentOSProject // @rbowen
859 351 9166
_______________________________________________
CentOS-devel mailing list
CentOS-devel(a)centos.org
https://lists.centos.org/mailman/listinfo/centos-devel
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>*
<https://www.redhat.com/en/summit?sc_cid=7013a000002D2QxAAK>*
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
4 years, 7 months
workaround for host deploy failure (get_events)
by Dana Elfassy
Hi,
Incase you face an error during host deploy, and see these messages in the
journal:
gunicorn-3[30567]: File
"/usr/lib/python3.6/site-packages/runner_service/services/jobs.py", line
181, in get_events
gunicorn-3[30567]: _events = os.listdir(event_dir) gunicorn-3[30567]:
FileNotFoundError: [Errno 2] No such file or directory:
'/.../share/ovirt-engine/ansible-runner-service-project/artifacts/175b5662-62b8-11ea-af39-144f8a9f8bed/job_events'
'Till Martin's fix is released, please execute the following steps:
1. sudo vi /usr/lib/python3.6/site-packages/ansible_runner/runner.py
2. edit the file with these changes:
https://github.com/ansible/ansible-runner/pull/425/files
3. sudo systemctl restart ansible-runner-service
Thanks,
Dana
4 years, 7 months
