[ovirt-devel] execCmd() and storing stdout and stderr in log file

[Nir Soffer]

On Thu, Jul 14, 2016 at 3:40 PM, Tomáš Golembiovský <tgolembi at redhat.com> wrote:
> On Thu, 14 Jul 2016 15:07:29 +0300
> Nir Soffer <nsoffer at redhat.com> wrote:
>
>> On Mon, Jul 11, 2016 at 12:53 PM, Tomáš Golembiovský
>> <tgolembi at redhat.com> wrote:
>> > On Wed, 6 Jul 2016 18:37:54 +0300
>> > Yaniv Bronheim <ybronhei at redhat.com> wrote:
>> >
>> >> On Wed, Jul 6, 2016 at 5:07 PM, Tomáš Golembiovský <tgolembi at redhat.com>
>> >> wrote:
>> >>
>> >> >
>> >> > Merging stdout and stderr to one can POpen do for us, I belive. Any
>> >> > logging can indeed be done as a wrapper around execCmd.
>> >>
>> >> saving stdout and err to log while the process is running is useful only
>> >> for your purpose currently. using asyncproc as you do now in v2v allows you
>> >> to to run a process and monitor it.. can you use overriding of aysncProc
>> >> wrapper for your needs instead of changing cpopen or execcmd code?
>> >
>> > I am not talking about CPOpen. I meant that when calling
>> > `subprocess.Popen`, you can pass it `stderr=subprocess.STDOUT` argument
>> > and it will handle the FD redirection (stream merging). To me it seems
>> > like a proper way of doing this.
>>
>> This should work with execCmd, since the special subprocess.STDOUT
>> parameter is handled in subprocess.Popen, and cpopen.CPopen inherit
>> this code. However this is not tested with cpopen, so it may be broken.
>>
>> But merging stdout and stderr is likely to break v2v output parser, and vdsm
>> log is not the place for virt-v2v debug logs.
>
> This is something that we can deal with.
>
>>
>> I understand that the issue is keeping virt-v2v debug logs (using --verbose?),
>> and the logs are spread in stdout and stderr. Did you discuss this issue
>> with Richard?
>
> Not yet.
>
>>
>> I would use tee to write stdout and stderr to an import log file, without
>> changing the code checking import progress.
>
> This is something I have tried as first, as you can see here:
>
> https://gerrit.ovirt.org/#/c/59834/4/lib/vdsm/v2v.py@404
>
> The problem with this approach is that we don't get proper exit code from virt-v2v
> because of the pipe. Fixing this in POSIX shell is not trivial and would lead
> to more complex shell code here. (Would be fairly easy to fix if we could
> resort to using bash here.)

After https://gerrit.ovirt.org/#/c/46733/ you should be able to create
the pipeline in python like this:

    v2v = Popen(["virt-v2v", ...], stdout=PIPE, stderr=STDOUT)
    tee = Popen(["tee", "-a", logfile], stdin=v2v.stdout, stdout=PIPE,
stderr=PIPE)

Now we can read output from tee.stdout, and when tee is finished, we can wait
for v2v to get the exit code.

Since all output would go to tee stdout and stderr may only contain tee usage
errors, we don't need to use AsyncProc, making this code python 3 compatible.

Nir