[ovirt-devel] Re: Unable to run create-brick.yml from ansible runner dev

The journal looks okay to me... Maybe the playbook failed please go to ovirt-engine/share/ovirt-engine/ansible-runner-service-project/artifacts/ and find the newest folder and check stdout file. On Fri, Mar 27, 2020 at 4:16 PM Martin Necas <mnecas(a)redhat.com&gt; wrote: > Sorry did not notice that link to the journal. > > On Fri, Mar 27, 2020 at 4:13 PM Martin Necas <mnecas(a)redhat.com&gt; wrote: > >> Hi, >> >> my bet is on the issue with ansible-runner please try to install new >> runner (https://cbs.centos.org/koji/packageinfo?packageID=7439) and >> restart service (`systemctl restart ansible-runner-service.service`). >> if it won't help please use `journalctl -u >> ansible-runner-service.service -f` to gather info from the runner. >> The log which is mentioned in engine.log is not created in dev env >> everything is logged to journal. >> >> Martin Necas >> >> On Fri, Mar 27, 2020 at 3:35 PM Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Fri, Mar 27, 2020 at 2:07 PM Kaustav Majumder <kmajumde(a)redhat.com&gt; >>> wrote: >>> >>>> HI Martin, >>>> Below error after rebasing you patch >>>> 2020-03-27 18:34:24,996+05 INFO >>>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >>>> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [] >>>> FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7fa278fe >>>> 2020-03-27 18:34:25,771+05 ERROR >>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>> task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Exception: Failed to execute >>>> call to start playbook. >>>> 2020-03-27 18:34:25,771+05 ERROR >>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Failed to execute Ansible create >>>> brick role. Please check logs for more details: >>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>> 2020-03-27 18:34:25,772+05 ERROR >>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Command >>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>> EngineException: Failed to execute Ansible create brick role: 'Failed to >>>> execute call to start playbook. '. Please check logs for more details: >>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >>>> (Failed with error GeneralException and code 100) >>>> 2020-03-27 18:34:25,852+05 ERROR >>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >>>> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Transaction rolled-back for command >>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. >>>> 2020-03-27 18:34:25,863+05 ERROR >>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>> (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] EVENT_ID: >>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >>>> 10.70.35.17 of cluster Default. >>>> >>> >>> Dana/Martin, any idea about the issue? >>> >>>> >>>> On Fri, Mar 27, 2020 at 6:08 PM Martin Perina <mperina(a)redhat.com&gt; >>>> wrote: >>>> >>>>> >>>>> >>>>> On Fri, Mar 27, 2020 at 12:45 PM Kaustav Majumder < >>>>> kmajumde(a)redhat.com&gt; wrote: >>>>> >>>>>> Hi, >>>>>> Let me know if you require something else >>>>>> journalctl -u ansible-runner-service -> >>>>>> https://pastebin.com/e6tnrzSB >>>>>> engine.log - > https://pastebin.com/bVv7FxLB >>>>>> >>>>> >>>>> Hmm, could you please try to reproduce with below patch, maybe there >>>>> will be more information in stderr: >>>>> >>>>> https://gerrit.ovirt.org/108012 >>>>> >>>>> Thanks, >>>>> Martin >>>>> >>>>>> >>>>>> On Fri, Mar 27, 2020 at 5:06 PM Martin Perina <mperina(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Could you please share journal and engine.log? We need to correlate >>>>>>> and find out which error in journal is correlated to this issue? >>>>>>> >>>>>>> Thanks, >>>>>>> Martin >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 27, 2020 at 12:33 PM Kaustav Majumder < >>>>>>> kmajumde(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> Hi Artur, >>>>>>>> I changed the files mentioned in the mail >>>>>>>> >>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MXPFVUAVE6U... >>>>>>>> >>>>>>>> but unfortunately it is not the same error as described in the >>>>>>>> mail and changing it gives me the same error. >>>>>>>> >>>>>>>> On Fri, Mar 27, 2020 at 4:45 PM Artur Socha <asocha(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Kaustav, >>>>>>>>> You might be hitting a known issue. >>>>>>>>> Please check the solution posted by Dana on this list. Look for >>>>>>>>> title: '[ovirt-devel] workaround for host deploy failure (get_events)' >>>>>>>>> Artur >>>>>>>>> >>>>>>>>> On Fri, Mar 27, 2020 at 11:37 AM Kaustav Majumder < >>>>>>>>> kmajumde(a)redhat.com&gt; wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> I am trying to run create-brick playbook from ansible-runner >>>>>>>>>> integrated inside ovirt-engine but it throws errors as >>>>>>>>>> ``` >>>>>>>>>> Running command: CreateBrickCommand internal: false. Entities >>>>>>>>>> affected : ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: VDSAction group >>>>>>>>>> MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 ERROR >>>>>>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>>>>>>> task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed to >>>>>>>>>> execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible create >>>>>>>>>> brick role. Please check logs for more details: >>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>> 2020-03-27 15:43:07,519+05 ERROR >>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Command >>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>>>>>> EngineException: Failed to execute Ansible create brick role. Please check >>>>>>>>>> logs for more details: >>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>> (Failed with error GeneralException and code 100) 2020-03-27 >>>>>>>>>> 15:43:07,526+05 ERROR >>>>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back for command >>>>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 >>>>>>>>>> 15:43:07,530+05 ERROR >>>>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>>>> (default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID: >>>>>>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >>>>>>>>>> 10.70.35.17 of cluster Default. ``` >>>>>>>>>> *The corresponding log is not created* >>>>>>>>>> kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± less >>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log: >>>>>>>>>> No such file or directory >>>>>>>>>> *checking ansible-runner status gives* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s 130 >>>>>>>>>> ± systemctl status ansible-runner-service.service ● >>>>>>>>>> ansible-runner-service.service - Ansible Runner Service Loaded: loaded >>>>>>>>>> (/usr/lib/systemd/system/ansible-runner-service.service; disabled; vendor >>>>>>>>>> preset: disabled) Active: active (running) since Fri 2020-03-27 15:42:56 >>>>>>>>>> IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: 4915) >>>>>>>>>> Memory: 62.1M CGroup: /system.slice/ansible-runner-service.service >>>>>>>>>> ├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>> runner_service.wsgi:application ├─30112 /usr/bin/python3 >>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>> runner_service.wsgi:application └─30123 /usr/bin/python3 >>>>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>>>> runner_service.wsgi:applicationMar 27 15:42:58 localhost.localdomain >>>>>>>>>> gunicorn-3[30097]: Request received, content-type :NoneMar 27 15:42:58 >>>>>>>>>> localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET >>>>>>>>>> /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01 >>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>> :NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03 >>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>> :NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07 >>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>> :NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>>>> GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07 >>>>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>>>> :application/json; charset=UTF-8Mar 27 15:43:07 localhost.localdomain >>>>>>>>>> gunicorn-3[30097]: 127.0.0.1 - POST /api/v1/playbooks/create-brick.yml* >>>>>>>>>> >>>>>>>>>> *Is there a way to find the root cause for this?* >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Kaustav Majumder >>>>>>>>>> _______________________________________________ >>>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>>>> oVirt Code of Conduct: >>>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>>> List Archives: >>>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/P2GWVT2HBST... >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Artur Socha >>>>>>>>> >>>>>>>>> Senior Software Engineer, RHV >>>>>>>>> >>>>>>>>> Red Hat <https://www.redhat.com> >>>>>>>>> <https://www.redhat.com> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Kaustav Majumder >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ULE6X7ZF6NM... >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Martin Perina >>>>>>> Manager, Software Engineering >>>>>>> Red Hat Czech s.r.o. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Kaustav Majumder >>>>>> >>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks, >>>> >>>> Kaustav Majumder >>>> >>> >>> >>> -- >>> Martin Perina >>> Manager, Software Engineering >>> Red Hat Czech s.r.o. >>> >>