[ovirt-devel] Re: Unable to run create-brick.yml from ansible runner dev

Hi, my bet is on the issue with ansible-runner please try to install new runner (https://cbs.centos.org/koji/packageinfo?packageID=7439) and restart service (`systemctl restart ansible-runner-service.service`). if it won't help please use `journalctl -u ansible-runner-service.service -f` to gather info from the runner. The log which is mentioned in engine.log is not created in dev env everything is logged to journal. Martin Necas On Fri, Mar 27, 2020 at 3:35 PM Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Fri, Mar 27, 2020 at 2:07 PM Kaustav Majumder <kmajumde(a)redhat.com&gt; > wrote: > >> HI Martin, >> Below error after rebasing you patch >> 2020-03-27 18:34:24,996+05 INFO >> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] >> (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-98) [] >> FINISH, GlusterVolumesListVDSCommand, return: {}, log id: 7fa278fe >> 2020-03-27 18:34:25,771+05 ERROR >> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >> task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] Exception: Failed to execute >> call to start playbook. >> 2020-03-27 18:34:25,771+05 ERROR >> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Failed to execute Ansible create >> brick role. Please check logs for more details: >> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >> 2020-03-27 18:34:25,772+05 ERROR >> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Command >> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >> EngineException: Failed to execute Ansible create brick role: 'Failed to >> execute call to start playbook. '. Please check logs for more details: >> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327183424-10.70.35.17-49b9e523-bdc1-4065-98f2-f8581a1119b3.log >> (Failed with error GeneralException and code 100) >> 2020-03-27 18:34:25,852+05 ERROR >> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-1) >> [49b9e523-bdc1-4065-98f2-f8581a1119b3] Transaction rolled-back for command >> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. >> 2020-03-27 18:34:25,863+05 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-1) [49b9e523-bdc1-4065-98f2-f8581a1119b3] EVENT_ID: >> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >> 10.70.35.17 of cluster Default. >> > > Dana/Martin, any idea about the issue? > >> >> On Fri, Mar 27, 2020 at 6:08 PM Martin Perina <mperina(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Fri, Mar 27, 2020 at 12:45 PM Kaustav Majumder <kmajumde(a)redhat.com&gt; >>> wrote: >>> >>>> Hi, >>>> Let me know if you require something else >>>> journalctl -u ansible-runner-service -> https://pastebin.com/e6tnrzSB >>>> engine.log - > https://pastebin.com/bVv7FxLB >>>> >>> >>> Hmm, could you please try to reproduce with below patch, maybe there >>> will be more information in stderr: >>> >>> https://gerrit.ovirt.org/108012 >>> >>> Thanks, >>> Martin >>> >>>> >>>> On Fri, Mar 27, 2020 at 5:06 PM Martin Perina <mperina(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Could you please share journal and engine.log? We need to correlate >>>>> and find out which error in journal is correlated to this issue? >>>>> >>>>> Thanks, >>>>> Martin >>>>> >>>>> >>>>> On Fri, Mar 27, 2020 at 12:33 PM Kaustav Majumder < >>>>> kmajumde(a)redhat.com&gt; wrote: >>>>> >>>>>> Hi Artur, >>>>>> I changed the files mentioned in the mail >>>>>> >>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/MXPFVUAVE6U... >>>>>> >>>>>> but unfortunately it is not the same error as described in the mail >>>>>> and changing it gives me the same error. >>>>>> >>>>>> On Fri, Mar 27, 2020 at 4:45 PM Artur Socha <asocha(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hi Kaustav, >>>>>>> You might be hitting a known issue. >>>>>>> Please check the solution posted by Dana on this list. Look for >>>>>>> title: '[ovirt-devel] workaround for host deploy failure (get_events)' >>>>>>> Artur >>>>>>> >>>>>>> On Fri, Mar 27, 2020 at 11:37 AM Kaustav Majumder < >>>>>>> kmajumde(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> I am trying to run create-brick playbook from ansible-runner >>>>>>>> integrated inside ovirt-engine but it throws errors as >>>>>>>> ``` >>>>>>>> Running command: CreateBrickCommand internal: false. Entities >>>>>>>> affected : ID: 45b7ce67-9939-4d5f-a1ba-92411356b7e6 Type: VDSAction group >>>>>>>> MANIPULATE_HOST with role type ADMIN 2020-03-27 15:43:07,518+05 ERROR >>>>>>>> [org.ovirt.engine.core.common.utils.ansible.AnsibleExecutor] (default >>>>>>>> task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] Exception: Failed to >>>>>>>> execute call to start playbook. 2020-03-27 15:43:07,519+05 ERROR >>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Failed to execute Ansible create >>>>>>>> brick role. Please check logs for more details: >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>> 2020-03-27 15:43:07,519+05 ERROR >>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Command >>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand' failed: >>>>>>>> EngineException: Failed to execute Ansible create brick role. Please check >>>>>>>> logs for more details: >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>> (Failed with error GeneralException and code 100) 2020-03-27 >>>>>>>> 15:43:07,526+05 ERROR >>>>>>>> [org.ovirt.engine.core.bll.gluster.CreateBrickCommand] (default task-29) >>>>>>>> [5ff56203-33ca-424b-9137-491cf1ef2cd3] Transaction rolled-back for command >>>>>>>> 'org.ovirt.engine.core.bll.gluster.CreateBrickCommand'. 2020-03-27 >>>>>>>> 15:43:07,530+05 ERROR >>>>>>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>>>>>> (default task-29) [5ff56203-33ca-424b-9137-491cf1ef2cd3] EVENT_ID: >>>>>>>> CREATE_GLUSTER_BRICK_FAILED(4,139), Failed to create brick brick-1 on host >>>>>>>> 10.70.35.17 of cluster Default. ``` >>>>>>>> *The corresponding log is not created* >>>>>>>> kmajumde:~/work/git/ovirt-engine] global_vol_options+ 7s ± less >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log >>>>>>>> /home/kmajumde/work/ovirt-engine-builds/27-03-global-vol-options/var/log/ovirt-engine/brick-setup/ovirt-gluster-brick-ansible-20200327154307-10.70.35.17-5ff56203-33ca-424b-9137-491cf1ef2cd3.log: >>>>>>>> No such file or directory >>>>>>>> *checking ansible-runner status gives* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *[kmajumde:~/work/git/ovirt-engine] global_vol_options+ 1m1s 130 ± >>>>>>>> systemctl status ansible-runner-service.service ● >>>>>>>> ansible-runner-service.service - Ansible Runner Service Loaded: loaded >>>>>>>> (/usr/lib/systemd/system/ansible-runner-service.service; disabled; vendor >>>>>>>> preset: disabled) Active: active (running) since Fri 2020-03-27 15:42:56 >>>>>>>> IST; 8min ago Main PID: 30097 (gunicorn-3) Tasks: 3 (limit: 4915) >>>>>>>> Memory: 62.1M CGroup: /system.slice/ansible-runner-service.service >>>>>>>> ├─30097 /usr/bin/python3 /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>> runner_service.wsgi:application ├─30112 /usr/bin/python3 >>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>> runner_service.wsgi:application └─30123 /usr/bin/python3 >>>>>>>> /usr/bin/gunicorn-3 -b localhost:50001 -w 2 >>>>>>>> runner_service.wsgi:applicationMar 27 15:42:58 localhost.localdomain >>>>>>>> gunicorn-3[30097]: Request received, content-type :NoneMar 27 15:42:58 >>>>>>>> localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - GET >>>>>>>> /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:01 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :NoneMar 27 15:43:01 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:03 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :NoneMar 27 15:43:03 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>> GET /api/v1/hosts/10.70.35.1 <http://10.70.35.1>Mar 27 15:43:07 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :NoneMar 27 15:43:07 localhost.localdomain gunicorn-3[30097]: 127.0.0.1 - >>>>>>>> GET /api/v1/hosts/10.70.35.17 <http://10.70.35.17>Mar 27 15:43:07 >>>>>>>> localhost.localdomain gunicorn-3[30097]: Request received, content-type >>>>>>>> :application/json; charset=UTF-8Mar 27 15:43:07 localhost.localdomain >>>>>>>> gunicorn-3[30097]: 127.0.0.1 - POST /api/v1/playbooks/create-brick.yml* >>>>>>>> >>>>>>>> *Is there a way to find the root cause for this?* >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Kaustav Majumder >>>>>>>> _______________________________________________ >>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>>>> oVirt Code of Conduct: >>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>> List Archives: >>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/P2GWVT2HBST... >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Artur Socha >>>>>>> >>>>>>> Senior Software Engineer, RHV >>>>>>> >>>>>>> Red Hat <https://www.redhat.com> >>>>>>> <https://www.redhat.com> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Kaustav Majumder >>>>>> _______________________________________________ >>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ULE6X7ZF6NM... >>>>>> >>>>> >>>>> >>>>> -- >>>>> Martin Perina >>>>> Manager, Software Engineering >>>>> Red Hat Czech s.r.o. >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks, >>>> >>>> Kaustav Majumder >>>> >>> >>> >>> -- >>> Martin Perina >>> Manager, Software Engineering >>> Red Hat Czech s.r.o. >>> >> >> >> -- >> >> Thanks, >> >> Kaustav Majumder >> > > > -- > Martin Perina > Manager, Software Engineering > Red Hat Czech s.r.o. >