[ovirt-devel] Re: Unable to run create-brick.yml from ansible runner dev

Unfortunately no artifact stdout files point to the playbook being run. There are others like when adding a new host and checking for host updates but none pertaining to create-brick . The latest one is this pointing to (i guess) host updates checker [root@localhost artifacts] # less 83029f4c-702d-11ea-b308-68ecc5af4344/stdout PLAY [all] ********************************************************************* TASK [Gathering Facts] ********************************************************* ok: [10.70.35.88] TASK [ovirt-host-check-upgrade : Check for system updates (remove empty lines)] *** this message. changed: [10.70.35.88] TASK [ovirt-host-check-upgrade : Remove 'Obsoleting' title from file] ********** ok: [10.70.35.88] TASK [ovirt-host-check-upgrade : Get yum updates file] ************************* ok: [10.70.35.88] TASK [ovirt-host-check-upgrade : Get yum updates content] ********************** ok: [10.70.35.88] TASK [ovirt-host-check-upgrade : Delete yum_updates file from host] ************ changed: [10.70.35.88] PLAY RECAP ********************************************************************* 10.70.35.88 : ok=6 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 On Fri, Mar 27, 2020 at 9:07 PM Martin Necas <mnecas(a)redhat.com&gt; wrote: > The journal looks okay to me... > Maybe the playbook failed please go > to ovirt-engine/share/ovirt-engine/ansible-runner-service-project/artifacts/ > and find the newest folder and check stdout file. > > On Fri, Mar 27, 2020 at 4:16 PM Martin Necas <mnecas(a)redhat.com&gt; wrote: > >> Sorry did not notice that link to the journal. >> >> On Fri, Mar 27, 2020 at 4:13 PM Martin Necas <mnecas(a)redhat.com&gt; wrote: >> >>> Hi, >>> >>> my bet is on the issue with ansible-runner please try to install new >>> runner (https://cbs.centos.org/koji/packageinfo?packageID=7439) and >>> restart service (`systemctl restart ansible-runner-service.service`). >>> if it won't help please use `journalctl -u >>> ansible-runner-service.service -f` to gather info from the runner. >>> The log which is mentioned in engine.log is not created in dev env >>> everything is logged to journal. >>> >>> Martin Necas >>> >>> On Fri, Mar 27, 2020 at 3:35 PM Martin Perina <mperina(a)redhat.com&gt; >>> wrote: >>> >>>> >>>> >>>> On Fri, Mar 27, 2020 at 2:07 PM Kaustav Majumder <kmajumde(a)redhat.com&gt; >>>> wrote: >>>> >>>>> HI Martin, >>>>> Below error after rebasing you patch >>>>> 2020-03-27 18:34:24,996+05 INFO >>>>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [] >>>>> FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7fa278fe >>>>> 2020-03-27 18:34:25,771+05 ERROR >>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>> task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Exception: Failed to execute >>>>> call to start playbook. >>>>> 2020-03-27 18:34:25,771+05 ERROR >>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Failed to execute Ansible create >>>>> brick role. Please check logs for more details: >>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>>> 2020-03-27 18:34:25,772+05 ERROR >>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Command >>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>> EngineException: Failed to execute Ansible create brick role: 'Failed to >>>>> execute call to start playbook. '. Please check logs for more details: >>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>>> (Failed with error GeneralException and code 100) >>>>> 2020-03-27 18:34:25,852+05 ERROR >>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Transaction rolled-back for command >>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. >>>>> 2020-03-27 18:34:25,863+05 ERROR >>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>> (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] EVENT_ID: >>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >>>>> 10.70.35.17 of cluster Default. >>>>> >>>> >>>> Dana/Martin, any idea about the issue? >>>> >>>>> >>>>> On Fri, Mar 27, 2020 at 6:08 PM Martin Perina <mperina(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Mar 27, 2020 at 12:45 PM Kaustav Majumder < >>>>>> kmajumde(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Hi, >>>>>>> Let me know if you require something else >>>>>>> journalctl -u ansible-runner-service -> >>>>>>> https://pastebin.com/e6tnrzSB >>>>>>> engine.log - > https://pastebin.com/bVv7FxLB >>>>>>> >>>>>> >>>>>> Hmm, could you please try to reproduce with below patch, maybe there >>>>>> will be more information in stderr: >>>>>> >>>>>> https://gerrit.ovirt.org/108012 >>>>>> >>>>>> Thanks, >>>>>> Martin >>>>>> >>>>>>> >>>>>>> On Fri, Mar 27, 2020 at 5:06 PM Martin Perina <mperina(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> Could you please share journal and engine.log? We need to >>>>>>>> correlate and find out which error in journal is correlated to this issue? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Martin >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Mar 27, 2020 at 12:33 PM Kaustav Majumder < >>>>>>>> kmajumde(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi Artur, >>>>>>>>> I changed the files mentioned in the mail >>>>>>>>> >>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MXPFVUAVE6U... >>>>>>>>> >>>>>>>>> but unfortunately it is not the same error as described in the >>>>>>>>> mail and changing it gives me the same error. >>>>>>>>> >>>>>>>>> On Fri, Mar 27, 2020 at 4:45 PM Artur Socha <asocha(a)redhat.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Kaustav, >>>>>>>>>> You might be hitting a known issue. >>>>>>>>>> Please check the solution posted by Dana on this list. Look for >>>>>>>>>> title: '[ovirt-devel] workaround for host deploy failure (get_events)' >>>>>>>>>> Artur >>>>>>>>>> >>>>>>>>>> On Fri, Mar 27, 2020 at 11:37 AM Kaustav Majumder < >>>>>>>>>> kmajumde(a)redhat.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> I am trying to run create-brick playbook from ansible-runner >>>>>>>>>>> integrated inside ovirt-engine but it throws errors as >>>>>>>>>>> ``` >>>>>>>>>>> Running command: CreateBrickCommand internal: false. Entities >>>>>>>>>>> affected : ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: VDSAction group >>>>>>>>>>> MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 ERROR >>>>>>>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>>>>>>>> task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed to >>>>>>>>>>> execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible create >>>>>>>>>>> brick role. Please check logs for more details: >>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>> 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Command >>>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>>>>>>> EngineException: Failed to execute Ansible create brick role. Please check >>>>>>>>>>> logs for more details: >>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>> (Failed with error GeneralException and code 100) 2020-03-27 >>>>>>>>>>> 15:43:07,526+05 ERROR >>>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back for command >>>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 >>>>>>>>>>> 15:43:07,530+05 ERROR >>>>>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>>>>> (default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID: >>>>>>>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >>>>>>>>>>> 10.70.35.17 of cluster Default. ``` >>>>>>>>>>> *The corresponding log is not created* >>>>>>>>>>> kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± less >>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log: >>>>>>>>>>> No such file or directory >>>>>>>>>>> *checking ansible-runner status gives* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s >>>>>>>>>>> 130 ± systemctl status ansible-runner-service.service ● >>>>>>>>>>> ansible-runner-service.service - Ansible Runner Service Loaded: loaded >>>>>>>>>>> (/usr/lib/systemd/system/ansible-runner-service.service; disabled; vendor >>>>>>>>>>> preset: disabled) Active: active (running) since Fri 2020-03-27 15:42:56 >>>>>>>>>>> IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: 4915) >>>>>>>>>>> Memory: 62.1M CGroup: /system.slice/ansible-runner-service.service >>>>>>>>>>> ├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>>> runner_service.wsgi:application ├─30112 /usr/bin/python3 >>>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>>> runner_service.wsgi:application └─30123 /usr/bin/python3 >>>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>>> runner_service.wsgi:applicationMar 27 15:42:58 localhost.localdomain >>>>>>>>>>> gunicorn-3[30097]: Request received, content-type :NoneMar 27 15:42:58 >>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET >>>>>>>>>>> /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01 >>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>>> :NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03 >>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>>> :NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07 >>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>>> :NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>>>>> GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07 >>>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>>> :application/json; charset=UTF-8Mar 27 15:43:07 localhost.localdomain >>>>>>>>>>> gunicorn-3[30097]: 127.0.0.1 - POST /api/v1/playbooks/create-brick.yml* >>>>>>>>>>> >>>>>>>>>>> *Is there a way to find the root cause for this?* >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Kaustav Majumder >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>>> List Archives: >>>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/P2GWVT2HBST... >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Artur Socha >>>>>>>>>> >>>>>>>>>> Senior Software Engineer, RHV >>>>>>>>>> >>>>>>>>>> Red Hat <https://www.redhat.com> >>>>>>>>>> <https://www.redhat.com> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Kaustav Majumder >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>> oVirt Code of Conduct: >>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>> List Archives: >>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ULE6X7ZF6NM... >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Martin Perina >>>>>>>> Manager, Software Engineering >>>>>>>> Red Hat Czech s.r.o. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Kaustav Majumder >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Martin Perina >>>>>> Manager, Software Engineering >>>>>> Red Hat Czech s.r.o. >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks, >>>>> >>>>> Kaustav Majumder >>>>> >>>> >>>> >>>> -- >>>> Martin Perina >>>> Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>>> >>> -- Thanks, Kaustav Majumder