<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 9:46 PM, RabidCicada <span dir="ltr"><<a href="mailto:rabidcicada@gmail.com" target="_blank">rabidcicada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Indeed that is the problem. I am sshing as root <div><br></div><div>The only possible explanations I have for this working in the ovirt autmated testing setup are:</div><div><ul><li>Either the ovirt automated tests must be using a host inventory file with become: true?</li><li>The inventory file ssh credentials happen to be to vdsm user? (And become is just not doing anything)<br></li><li>Some other janky thing that allows the access</li></ul></div></div></blockquote>Yes, you are absolutely right:<br>we are running the CI suite with no_root_squash<br><a href="https://github.com/oVirt/ovirt-system-tests/blob/master/common/deploy-scripts/setup_storage_nfs_el7.sh#L17">https://github.com/oVirt/ovirt-system-tests/blob/master/common/deploy-scripts/setup_storage_nfs_el7.sh#L17</a><br>while we recommend anonuid=36,anongid=36,all_squash</div><div class="gmail_quote"><a href="https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/">https://www.ovirt.org/documentation/how-to/troubleshooting/troubleshooting-nfs-storage-issues/</a><br></div><div class="gmail_quote">and this explains why it can write as root in the CI env.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Thanks for the report!<br><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div><div>It also happens for the immediately following task `Initialize metadata volume` and many more following that task.</div><div><br></div><div>I will correct and then open a pull request to fix with your input that you believe it's a problem.</div><div><br></div><div>~Kyle</div><div><br></div></div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 3:08 PM, RabidCicada <span dir="ltr"><<a href="mailto:rabidcicada@gmail.com" target="_blank">rabidcicada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Alright. By differential comparing to successfull postgres-command sudo-becomes. It looks like the salient difference is that `Copy configuration archive to storage` is missing become:true. Testing now.<div><br></div><div>The similar postgres commands have become:true while this one does not.</div><div><br></div><div>~Kyle</div></div><div class="gmail-m_-807229225562045180HOEnZb"><div class="gmail-m_-807229225562045180h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 1:19 PM, Simone Tiraboschi <span dir="ltr"><<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_-807229225562045180m_7831304234596663950h5">On Tue, Apr 3, 2018 at 5:21 PM, RabidCicada <span dir="ltr"><<a href="mailto:rabidcicada@gmail.com" target="_blank">rabidcicada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I've attached a full debug packet below. I include the log file from ovirt-hosted-engine-setup. I include relevant cmd line info. I include info from the command line where epdb has a breakpoint in playbook.py from ansible itself. I also include info from commands I ran after it failed. I also include attached the ferried over script in /root/.ansible/tmp that is run.</div><div><br></div><div><b><br></b></div><div><b>Output on cmd line:</b></div><div>[ INFO ] TASK [Copy configuration archive to storage]<br></div><div><div>[ ERROR ] fatal: [localhost]: FAILED! => {</div><div> "changed": true,</div><div> "cmd": [</div><div> "dd",</div><div> "bs=20480",</div><div> "count=1",</div><div> "oflag=direct",</div><div> "if=/var/tmp/localvmbCDQIR/5e<wbr>f881f5-c992-48d2-b969-a0b6156b<wbr>df7c",</div><div> "of=/rhev/data-center/mnt/nod<wbr>e.local:_srv_data/81292f3f-11d<wbr>3-4e38-9afa-62e133aa8017/image<wbr>s/c5510e77-1ee0-479c-b6cf-24c1<wbr>79313a45/5ef881f5-c992-48d2-b9<wbr>69-a0b6156bdf7c"</div><div> ],</div><div> "delta": "0:00:00.004336",</div><div> "end": "2018-04-03 15:01:55.581823",</div><div> "invocation": {</div><div> "module_args": {</div><div> "_raw_params": "dd bs=20480 count=1 oflag=direct if=\"/var/tmp/localvmbCDQIR/5e<wbr>f881f5-c992-48d2-b969-a0b6156b<wbr>df7c\" of=\"/rhev/data-center/mnt/nod<wbr>e.local:_srv_data/81292f3f-11d<wbr>3-4e38-9afa-62e133aa8017/image<wbr>s/c5510e77-1ee0-479c-b6cf-24c1<wbr>79313a45/5ef881f5-c992-48d2-b9<wbr>69-a0b6156bdf7c\"",</div><div> "_uses_shell": false,</div><div> "chdir": null,</div><div> "creates": null,</div><div> "executable": null,</div><div> "removes": null,</div><div> "stdin": null,</div><div> "warn": true</div><div> }</div><div> },</div><div> "msg": "non-zero return code",</div><div> "rc": 1,</div><div> "start": "2018-04-03 15:01:55.577487",</div><div> "stderr": "dd: failed to open ‘/rhev/data-center/mnt/node.lo<wbr>cal:_srv_data/81292f3f-11d3-4e<wbr>38-9afa-62e133aa8017/images/c5<wbr>510e77-1ee0-479c-b6cf-24c17931<wbr>3a45/5ef881f5-c992-48d2-b969-a<wbr>0b6156bdf7c’: Permission denied",</div><div> "stderr_lines": [</div><div> "dd: failed to open ‘/rhev/data-center/mnt/node.lo<wbr>cal:_srv_data/81292f3f-11d3-4e<wbr>38-9afa-62e133aa8017/images/c5<wbr>510e77-1ee0-479c-b6cf-24c17931<wbr>3a45/5ef881f5-c992-48d2-b969-a<wbr>0b6156bdf7c’: Permission denied"</div><div> ],</div><div> "stdout": "",</div><div> "stdout_lines": []</div><div> }</div><div>[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook</div></div></div></blockquote><div><br></div></div></div><div>In the playbook we have on that task:</div><div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> become_user: vdsm</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"> become_method: sudo</div><br></div><div>but I fear it got somehow ignored.</div><div>I'll investigate it.</div><div><div class="gmail-m_-807229225562045180m_7831304234596663950h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div><b>Output from ansible epdb tracepoint:</b></div><div><b><br></b></div><div><div>Using module file /usr/lib/python2.7/site-packag<wbr>es/ansible/modules/commands/co<wbr>mmand.py</div><div><localhost> ESTABLISH LOCAL CONNECTION FOR USER: root</div><div><localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'</div><div><localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp<wbr>-1522767715.36-81496549401055 `" && echo ansible-tmp-1522767715.36-8149<wbr>6549401055="` echo /root/.ansible/tmp/ansible-tmp<wbr>-1522767715.36-81496549401055 `" ) && sleep 0'</div><div><localhost> PUT /tmp/tmpGMGdjh TO /root/.ansible/tmp/ansible-tmp<wbr>-1522767715.36-81496549401055/<wbr>command.py</div><div><localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp<wbr>-1522767715.36-81496549401055/ /root/.ansible/tmp/ansible-tmp<wbr>-1522767715.36-81496549401055/<wbr>command.py && sleep 0'</div><div><localhost> EXEC /bin/sh -c '/usr/bin/python /root/.ansible/tmp/ansible-tmp<wbr>-1522767715.36-81496549401055/<wbr>command.py && sleep 0'</div><div> to retry, use: --limit @/usr/share/ovirt-hosted-engin<wbr>e-setup/ansible/create_target_<wbr>vm.retry</div></div><div><b><br></b></div><div>The above command.py is the on I have attached as problematic_command.py</div><div><br></div><div><b>Investigation After Failure:</b></div><div>[root@node ~]# ls -al '/rhev/data-center/mnt/node.lo<wbr>cal:_srv_data/81292f3f-11d3-4e<wbr>38-9afa-62e133aa8017/images/c5<wbr>510e77-1ee0-479c-b6cf-24c17931<wbr>3a45/5ef881f5-c992-48d2-b969-a<wbr>0b6156bdf7c'</div><div>-rw-rw----. 1 vdsm kvm 20480 Apr 3 15:01 /rhev/data-center/mnt/node.loc<wbr>al:_srv_data/81292f3f-11d3-4e3<wbr>8-9afa-62e133aa8017/images/c55<wbr>10e77-1ee0-479c-b6cf-24c179313<wbr>a45/5ef881f5-c992-48d2-b969-a0<wbr>b6156bdf7c</div><div><br></div><div>sudo -u vdsm dd bs=20480 count=1 oflag=direct if="/var/tmp/localvmbCDQIR/5ef<wbr>881f5-c992-48d2-b969-a0b6156bd<wbr>f7c" of="/rhev/data-center/mnt/node<wbr>.local:_srv_data/81292f3f-11d3<wbr>-4e38-9afa-62e133aa8017/images<wbr>/c5510e77-1ee0-479c-b6cf-24c17<wbr>9313a45/5ef881f5-c992-48d2-b96<wbr>9-a0b6156bdf7c</div><div><div>1+0 records in</div><div>1+0 records out</div></div><div><br></div><div><br></div><div>It seems to me that somehow it is not getting the right permissions even though the playbook has:</div><span><div><div>- name: Copy configuration archive to storage</div><div> command: dd bs=20480 count=1 oflag=direct if="{{ LOCAL_VM_DIR }}/{{ he_conf_disk_details.disk.imag<wbr>e_id }}" of="{{ he_conf_disk_path }}"</div><div> become_user: vdsm</div><div> become_method: sudo</div><div> changed_when: True</div><div> tags: [ 'skip_ansible_lint' ]</div></div><div><br></div></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628HOEnZb"><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 8:51 AM, RabidCicada <span dir="ltr"><<a href="mailto:rabidcicada@gmail.com" target="_blank">rabidcicada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I am now also running with:<div>
<span></span>
<p class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(51,187,200);background-color:rgb(255,255,255)"><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">export</span><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s2" style="font-variant-ligatures:no-common-ligatures"> ANSIBLE_VERBOSITY=</span><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">5</span></p>
<p class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(51,187,200);background-color:rgb(255,255,255)"><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">export</span><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s2" style="font-variant-ligatures:no-common-ligatures"> ANSIBLE_FORKS=</span><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span></p>
<p class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-p1" style="margin:0px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(51,187,200);background-color:rgb(255,255,255)"><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">export</span><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s2" style="font-variant-ligatures:no-common-ligatures"> ANSIBLE_KEEP_REMOTE_FILES=</span><span class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(195,55,32)">1</span></p>
<br></div></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429HOEnZb"><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 8:49 AM, RabidCicada <span dir="ltr"><<a href="mailto:rabidcicada@gmail.com" target="_blank">rabidcicada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Here's the log.<div><br></div><div>So the command that it says it ran is:</div><div>dd bs=20480 count=1 oflag=direct if=/var/tmp/localvmHaWb6G/1cce<wbr>8df2-1810-4063-b4e2-e19a2c5b19<wbr>09 of=/rhev/data-center/mnt/node.<wbr>local:_srv_data/3c7485ea-14e3-<wbr>40c1-b627-f89a819ed1d6/images/<wbr>2c1f7c2f-b8f7-46d4-ac66-8ff1e9<wbr>649e29/1cce8df2-1810-4063-b4e2<wbr>-e19a2c5b1909<br></div><div><br></div><div>But we all know that was done with:</div><div><br></div><div><div> - name: Copy configuration archive to storage</div><div> command: dd bs=20480 count=1 oflag=direct if="{{ LOCAL_VM_DIR }}/{{ he_conf_disk_details.disk.imag<wbr>e_id }}" of="{{ he_conf_disk_path }}"</div><div> become_user: vdsm</div><div> become_method: sudo</div><div> changed_when: True</div><div> tags: [ 'skip_ansible_lint' ]</div></div><div><br></div><div>So I manually replicated with `sudo vdsm <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">dd bs=20480 count=1 if=/var/tmp/localvmHaWb6G/1cce<wbr>8df2-1810-4063-b4e2-e19a2c5b19<wbr>09 of=/rhev/data-center/mnt/node.<wbr>local:_srv_data/3c7485ea-14e3-<wbr>40c1-b627-f89a819ed1d6/images/<wbr>2c1f7c2f-b8f7-46d4-ac66-8ff1e9<wbr>649e29/1cce8df2-1810-4063-b4e2<wbr>-e19a2c5b1909`</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">And it works when I manually do it. Though I think I didn't use the oflag=direct (Just realised this)</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I eventually put a pause task directly preceeding it with debug output that showed the file paths. I manually ran the command and it worked. Then let it do it....failed. I checked all the permissions e.g. vdsm:kvm. All looks good from a filesystem point of view. I'm beginning (naively) to suspect a race condition for the file access problem...but have come nowhere close to solving it.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><b><u>Can you suggest a good way to continue executing the install process from the create_target_vm.yml playbook (with proper variables and context from otopi etc)?</u></b></span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> I currently have to restart the entire process over again and wait quite a while for it to circle back around. </span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I have since discovered epdb and I've set breakpoints directly in playbook.py of ansible just to see better log output. I insert epdb.serve and use netcat to connect since epdb on python 2.7.5 and up seems to have problems using epdb.connect() itself.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">~Kyle</span></div></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035HOEnZb"><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 4:06 AM, Simone Tiraboschi <span dir="ltr"><<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Apr 2, 2018 at 4:52 PM, RabidCicada <span dir="ltr"><<a href="mailto:rabidcicada@gmail.com" target="_blank">rabidcicada@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span>Heyo everyone.<span> </span>I'm trying to debug hosted-engine --deploy. It is failing in <span><span style="color:rgb(0,0,0);font-family:Consolas,"Lucida Console",monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">`Copy configuration archive to storage` in `create_target_vm.yml` from `<span style="color:rgb(0,0,0);font-family:Consolas,"Lucida Console",monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">hosted-engine --deploy`. My general and most important query here is how to get good debug output from ansible through hosted-engine. I'm running hosted-engine through an ssh session.</span></span></span></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span><br></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span>I can't figure out how to get good debug output from ansible within that workflow.<span> </span>I see it's running through otopi, I tried setting typical `debugger: on_failed` hooks etc and tried many incantations on the command </span><span style="font-size:12.8px">line and config files to get ansible to help me out. The debugger: directive and other debugger related ansible config file stuff wouldn't result in any debugger popping up. I also can't seem to pass normal -vvvv flags to hosted-engine either and get it to ansible. Ultimately I tried to use a `pause` directive and it complained that it was in a non-interactive shell. I figured it might be the result of my ssh session so I enabled tty allocation with -t -t. It did not resolve the issue.</span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span><br></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span>I eventually wrote-my-own/stole a callback_plugin that checks an environmental variable and enables `display.verbosity = int(v)` since I can't seem to pass typical -vvvv stuff to ansible through `hosted-engine --deploy`. It give me the best info that I have so far. But it wont give me enough to debug issues around Gathering Facts or what looks like a sudo/permission problem in <span style="color:rgb(0,0,0);font-family:Consolas,"Lucida Console",monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">`Copy configuration archive to storage` in `create_target_vm.yml`. I took and used the exact command that they use manually and it works when I run it manually (But I can't get debug output to show me the exact sudo command being executed), hence my interest in passing -vvvv or equivalent to ansible through `hosted-engine`. I intentionally disabled the VM_directory cleanup so that I could execute the same stuff.</span></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span><span style="color:rgb(0,0,0);font-family:Consolas,"Lucida Console",monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">So....after all that...what is a good way to get deep debug info from hosted-engine ansible stuff?</div></div></blockquote><div><br></div></span><div>You should already find all the relevant log entries in a file called /var/log/ovirt-hosted-e<wbr>ngine-setup/ovirt-hosted-engin<wbr>e-setup-ansible-create_target_<wbr>vm-{timestamp}-{hash}.log</div><div><br></div><div>Can you please share it?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><div dir="ltr"><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Or does anyone have intuition for the possible sudo problem?</div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span><span style="color:rgb(0,0,0);font-family:Consolas,"Lucida Console",monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">~Kyle</span></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span><span style="color:rgb(0,0,0);font-family:Consolas,"Lucida Console",monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle1 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><span><span style="color:rgb(0,0,0);font-family:Consolas,"Lucida Console",monospace;font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div class="gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-linestyle2 gmail-m_-807229225562045180m_7831304234596663950m_766559802820559628m_5635528246031702429m_5983775323000025035m_278240774770278449m_6346744230857222759gmail-m_-5418010615936734164gmail-colourline" style="font-family:Consolas,"Lucida Console",monospace;padding-left:7px;word-wrap:break-word;color:rgb(0,0,0);font-size:12.8px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br></div></div>
<br></span>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>