Bad setup code in vdsm_master_storage_functional_tests_localfs_gerrit

[Dan Kenigsberg]

On Tue, Jun 17, 2014 at 10:33:24AM +0200, David Caro wrote:
> On Tue 17 Jun 2014 10:24:49 AM CEST, Dan Kenigsberg wrote:
> > On Tue, Jun 17, 2014 at 03:40:51AM -0400, Vered Volansky wrote:
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Dan Kenigsberg" <danken at redhat.com>
> >>> To: "Vered Volansky" <vered at redhat.com>
> >>> Cc: "infra" <infra at ovirt.org>
> >>> Sent: Monday, June 16, 2014 11:29:42 AM
> >>> Subject: Re: Bad setup code in	vdsm_master_storage_functional_tests_localfs_gerrit
> >>>
> >>> On Sun, Jun 15, 2014 at 04:11:53AM -0400, Vered Volansky wrote:
> >>>> The job with this issue is gone, let me know if it's risen again.
> >>>
> >>> The fragile code is still in
> >>> http://jenkins.ovirt.org/view/All/job/vdsm_master_storage_functional_tests_localfs_gerrit/configure
> >>> why not make it more robust before /var/log/vdsm disappears and make it
> >>> break again?
> >>
> >> because I don't understand the issue. The file is only created if
> >> missing. The directory should be there.
> 
> It was fixed by me some time ago (I added the mkdir -p before the 
> touch, just in case)
> 
>     sudo mkdir -p /var/log/vdsm
>     sudo chown vdsm:kvm
>     sudo sh -c 'echo "" > /var/log/vdsm/vdsm.log'
>     sudo sh -c 'echo "" > /var/log/vdsm/supervdsm.log'
> 
> >
> > However, apparently it was not there, which made the echo fail, which
> > led to the job failing. We should understand why it disappeared.
> >
> > dcaro, eedri - do you have any idea?
> 
> Totally agree, and, if it was meant to be there, I'll remove the mkdir 
> to make the test fail if it's not there.
> But for what I see on the job, there's nothing that ensures you that 
> the directory will be there, vdsm might never have been installed on 
> that machine, or might have been properly cleaned at some point 
> (removing logs and leftovers).
> So, In my opinion the issue is that we are not cleaning up properly 
> after the vdsm jobs and leaving the logs behind.

We can never be sure that the former job running on a particular thread
has finished properly. It might have aborted before it cleaned the log.
Thus, it's prudent to clear the log before a new job starts.

But my original question begins - what could possibly remove the
directory between the mkdir and the echo? Are you aware of anything that
removes the directory (removal of vdsm.rpm does not)

Whatever it was, it must not be running while a functional test takes
place!