Just update for others I and Kaustav were doing more investigation and found an issue which was introduced by https://gerrit.ovirt.org/#/c/107683/, because of removing over escaped parameters. Right now working on the fix. On Fri, Mar 27, 2020 at 4:54 PM Kaustav Majumder <kmajumde@redhat.com> wrote:
Unfortunately no artifact stdout files point to the playbook being run. There are others like when adding a new host and checking for host updates but none pertaining to create-brick . The latest one is this pointing to (i guess) host updates checker
[root@localhost artifacts] # less 83029f4c-702d-11ea-b308-68ecc5af4344/stdout PLAY [all] *********************************************************************
TASK [Gathering Facts] ********************************************************* ok: [10.70.35.88]
TASK [ovirt-host-check-upgrade : Check for system updates (remove empty lines)] *** this message. changed: [10.70.35.88]
TASK [ovirt-host-check-upgrade : Remove 'Obsoleting' title from file] ********** ok: [10.70.35.88]
TASK [ovirt-host-check-upgrade : Get yum updates file] ************************* ok: [10.70.35.88]
TASK [ovirt-host-check-upgrade : Get yum updates content] ********************** ok: [10.70.35.88]
TASK [ovirt-host-check-upgrade : Delete yum_updates file from host] ************ changed: [10.70.35.88]
PLAY RECAP ********************************************************************* 10.70.35.88 : ok=6 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
On Fri, Mar 27, 2020 at 9:07 PM Martin Necas <mnecas@redhat.com> wrote:
The journal looks okay to me... Maybe the playbook failed please go to ovirt-engine/share/ovirt-engine/ansible-runner-service-project/artifacts/ and find the newest folder and check stdout file.
On Fri, Mar 27, 2020 at 4:16 PM Martin Necas <mnecas@redhat.com> wrote:
Sorry did not notice that link to the journal.
On Fri, Mar 27, 2020 at 4:13 PM Martin Necas <mnecas@redhat.com> wrote:
Hi,
my bet is on the issue with ansible-runner please try to install new runner (https://cbs.centos.org/koji/packageinfo?packageID=7439) and restart service (`systemctl restart ansible-runner-service.service`). if it won't help please use `journalctl -u ansible-runner-service.service -f` to gather info from the runner. The log which is mentioned in engine.log is not created in dev env everything is logged to journal.
Martin Necas
On Fri, Mar 27, 2020 at 3:35 PM Martin Perina <mperina@redhat.com> wrote:
On Fri, Mar 27, 2020 at 2:07 PM Kaustav Majumder <kmajumde@redhat.com> wrote:
HI Martin, Below error after rebasing you patch 2020-03-27 18:34:24,996+05 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [] FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7fa278fe 2020-03-27 18:34:25,771+05 ERROR [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Exception: Failed to execute call to start playbook. 2020-03-27 18:34:25,771+05 ERROR [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Failed to execute Ansible create brick role. Please check logs for more details: /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log 2020-03-27 18:34:25,772+05 ERROR [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Command 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: EngineException: Failed to execute Ansible create brick role: 'Failed to execute call to start playbook. '. Please check logs for more details: /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log (Failed with error GeneralException and code 100) 2020-03-27 18:34:25,852+05 ERROR [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Transaction rolled-back for command 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 18:34:25,863+05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] EVENT_ID: CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host 10.70.35.17 of cluster Default.
Dana/Martin, any idea about the issue?
On Fri, Mar 27, 2020 at 6:08 PM Martin Perina <mperina@redhat.com> wrote:
> > > On Fri, Mar 27, 2020 at 12:45 PM Kaustav Majumder < > kmajumde@redhat.com> wrote: > >> Hi, >> Let me know if you require something else >> journalctl -u ansible-runner-service -> >> https://pastebin.com/e6tnrzSB >> engine.log - > https://pastebin.com/bVv7FxLB >> > > Hmm, could you please try to reproduce with below patch, maybe there > will be more information in stderr: > > https://gerrit.ovirt.org/108012 > > Thanks, > Martin > >> >> On Fri, Mar 27, 2020 at 5:06 PM Martin Perina <mperina@redhat.com> >> wrote: >> >>> Could you please share journal and engine.log? We need to >>> correlate and find out which error in journal is correlated to this issue? >>> >>> Thanks, >>> Martin >>> >>> >>> On Fri, Mar 27, 2020 at 12:33 PM Kaustav Majumder < >>> kmajumde@redhat.com> wrote: >>> >>>> Hi Artur, >>>> I changed the files mentioned in the mail >>>> >>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MXPFVUAVE6UYOY... >>>> >>>> but unfortunately it is not the same error as described in the >>>> mail and changing it gives me the same error. >>>> >>>> On Fri, Mar 27, 2020 at 4:45 PM Artur Socha <asocha@redhat.com> >>>> wrote: >>>> >>>>> Hi Kaustav, >>>>> You might be hitting a known issue. >>>>> Please check the solution posted by Dana on this list. Look for >>>>> title: '[ovirt-devel] workaround for host deploy failure (get_events)' >>>>> Artur >>>>> >>>>> On Fri, Mar 27, 2020 at 11:37 AM Kaustav Majumder < >>>>> kmajumde@redhat.com> wrote: >>>>> >>>>>> Hi all, >>>>>> I am trying to run create-brick playbook from ansible-runner >>>>>> integrated inside ovirt-engine but it throws errors as >>>>>> ``` >>>>>> Running command: CreateBrickCommand internal: false. Entities >>>>>> affected : ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: VDSAction group >>>>>> MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 ERROR >>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>>> task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed to >>>>>> execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR >>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible create >>>>>> brick role. Please check logs for more details: >>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>> 2020-03-27 15:43:07,519+05 ERROR >>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Command >>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>> EngineException: Failed to execute Ansible create brick role. Please check >>>>>> logs for more details: >>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>> (Failed with error GeneralException and code 100) 2020-03-27 >>>>>> 15:43:07,526+05 ERROR >>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back for command >>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 >>>>>> 15:43:07,530+05 ERROR >>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>> (default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID: >>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >>>>>> 10.70.35.17 of cluster Default. ``` >>>>>> *The corresponding log is not created* >>>>>> kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± less >>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log: >>>>>> No such file or directory >>>>>> *checking ansible-runner status gives* >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s >>>>>> 130 ± systemctl status ansible-runner-service.service ● >>>>>> ansible-runner-service.service - Ansible Runner Service Loaded: loaded >>>>>> (/usr/lib/systemd/system/ansible-runner-service.service; disabled; vendor >>>>>> preset: disabled) Active: active (running) since Fri 2020-03-27 15:42:56 >>>>>> IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: 4915) >>>>>> Memory: 62.1M CGroup: /system.slice/ansible-runner-service.service >>>>>> ├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>> runner_service.wsgi:application ├─30112 /usr/bin/python3 >>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>> runner_service.wsgi:application └─30123 /usr/bin/python3 >>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>> runner_service.wsgi:applicationMar 27 15:42:58 localhost.localdomain >>>>>> gunicorn-3[30097]: Request received, content-type :NoneMar 27 15:42:58 >>>>>> localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET >>>>>> /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01 >>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>> :NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03 >>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>> :NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07 >>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>> :NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>> GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07 >>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>> :application/json; charset=UTF-8Mar 27 15:43:07 localhost.localdomain >>>>>> gunicorn-3[30097]: 127.0.0.1 - POST /api/v1/playbooks/create-brick.yml* >>>>>> >>>>>> *Is there a way to find the root cause for this?* >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Kaustav Majumder >>>>>> _______________________________________________ >>>>>> Devel mailing list -- devel@ovirt.org >>>>>> To unsubscribe send an email to devel-leave@ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/P2GWVT2HBSTFAH... >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Artur Socha >>>>> >>>>> Senior Software Engineer, RHV >>>>> >>>>> Red Hat <https://www.redhat.com> >>>>> <https://www.redhat.com> >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks, >>>> >>>> Kaustav Majumder >>>> _______________________________________________ >>>> Devel mailing list -- devel@ovirt.org >>>> To unsubscribe send an email to devel-leave@ovirt.org >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ULE6X7ZF6NMB5M... >>>> >>> >>> >>> -- >>> Martin Perina >>> Manager, Software Engineering >>> Red Hat Czech s.r.o. >>> >> >> >> -- >> >> Thanks, >> >> Kaustav Majumder >> > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. >
--
Thanks,
Kaustav Majumder
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.
--
Thanks,
Kaustav Majumder