8:46 a.m.
Hi,
Should it be like that? Is there a way to clean this leftover in the
check-patch script?
Thanks
Gil
9:25 a.m.
IIRC, this is the default location libguestfs caches files.
It can be changed with LIBGUESTFS_TMPDIR env parameter, but whether
the default behaviour should be changed is a different question I
guess.
On Tue, May 23, 2017 at 4:46 PM, Gil Shinar <gshinar(a)redhat.com> wrote:
Hi,
Should it be like that? Is there a way to clean this leftover in the
check-patch script?
Thanks
Gil
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
9:34 a.m.
On 23 May 2017 at 17:25, Nadav Goldin <ngoldin(a)redhat.com> wrote:
IIRC, this is the default location libguestfs caches files.
It can be changed with LIBGUESTFS_TMPDIR env parameter, but whether
the default behaviour should be changed is a different question I
guess.
Question is why does it stay there once the job is done
Also I'm guessing its creating it via libvirt somehow b/c imageio does
not have any bind-mounts into mock.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
10:34 a.m.
On Tue, May 23, 2017 at 5:34 PM, Barak Korren <bkorren(a)redhat.com> wrote:
On 23 May 2017 at 17:25, Nadav Goldin <ngoldin(a)redhat.com>
wrote:
> IIRC, this is the default location libguestfs caches files.
> It can be changed with LIBGUESTFS_TMPDIR env parameter, but whether
> the default behaviour should be changed is a different question I
> guess.
Question is why does it stay there once the job is done
It's in /var/tmp and it's supposed to be cached. Is there an issues with
this? It's 400MB image, AFAIR.
Y.
Also I'm guessing its creating it via libvirt somehow b/c imageio
does
not have any bind-mounts into mock.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
11:22 p.m.
This may be vdsm, imageio does nor use libguestfs.
בתאריך יום ג׳, 23 במאי 2017, 18:35, מאת Yaniv Kaul <ykaul(a)redhat.com>:
On Tue, May 23, 2017 at 5:34 PM, Barak Korren
<bkorren(a)redhat.com> wrote:
> On 23 May 2017 at 17:25, Nadav Goldin <ngoldin(a)redhat.com> wrote:
> > IIRC, this is the default location libguestfs caches files.
> > It can be changed with LIBGUESTFS_TMPDIR env parameter, but whether
> > the default behaviour should be changed is a different question I
> > guess.
>
> Question is why does it stay there once the job is done
>
It's in /var/tmp and it's supposed to be cached. Is there an issues with
this? It's 400MB image, AFAIR.
Y.
> Also I'm guessing its creating it via libvirt somehow b/c imageio does
> not have any bind-mounts into mock.
>
>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
1:45 a.m.
On 23 May 2017 at 18:34, Yaniv Kaul <ykaul(a)redhat.com> wrote:
It's in /var/tmp and it's supposed to be cached. Is there an issues with
this? It's 400MB image, AFAIR.
We currently have /var/tmp wiped out after each and every job run. We
are looking into stopping that to allow it to be used for persistent
caches, but we don't want the slaves to fill up as a result.
We we need to understand how fast may this accumulate.
400MB accumulation per run is a lot. Unless this is stable and also
gets recycled automatically.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
2:30 a.m.
On Wed, May 24, 2017 at 9:46 AM Barak Korren <bkorren(a)redhat.com> wrote:
On 23 May 2017 at 18:34, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
> It's in /var/tmp and it's supposed to be cached. Is there an issues with
> this? It's 400MB image, AFAIR.
We currently have /var/tmp wiped out after each and every job run.
Please keep this behavior.
We
are looking into stopping that to allow it to be used for persistent
caches,
Use /var/cache?
but we don't want the slaves to fill up as a result.
We we need to understand how fast may this accumulate.
400MB accumulation per run is a lot. Unless this is stable and also
gets recycled automatically.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
3:17 a.m.
On Wed, May 24, 2017 at 10:30 AM, Nir Soffer <nsoffer(a)redhat.com> wrote:
On Wed, May 24, 2017 at 9:46 AM Barak Korren <bkorren(a)redhat.com> wrote:
> On 23 May 2017 at 18:34, Yaniv Kaul <ykaul(a)redhat.com> wrote:
> >
> > It's in /var/tmp and it's supposed to be cached. Is there an issues
with
> > this? It's 400MB image, AFAIR.
>
> We currently have /var/tmp wiped out after each and every job run.
Please keep this behavior.
> We
> are looking into stopping that to allow it to be used for persistent
> caches,
Use /var/cache?
/dev/shm is just as good. It's only 400MB.
Y.
but we don't want the slaves to fill up as a result.
>
> We we need to understand how fast may this accumulate.
>
> 400MB accumulation per run is a lot. Unless this is stable and also
> gets recycled automatically.
>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
3:35 a.m.
On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
/dev/shm is just as good. It's only 400MB.
Y.
Forgive my language but, hell no. This is not the gigantic Lago
bare metals you are used to. We don't want GWT builds to start
failing on running out of RAM.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
4:38 a.m.
On Wed, May 24, 2017 at 11:35 AM, Barak Korren <bkorren(a)redhat.com> wrote:
On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
> /dev/shm is just as good. It's only 400MB.
> Y.
>
Forgive my language but, hell no. This is not the gigantic Lago
bare metals you are used to. We don't want GWT builds to start
failing on running out of RAM.
Buy more RAM.
Y.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
5:34 a.m.
On Wed, May 24, 2017 at 12:38 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Wed, May 24, 2017 at 11:35 AM, Barak Korren <bkorren(a)redhat.com> wrote:
> On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
> >
> > /dev/shm is just as good. It's only 400MB.
> > Y.
> >
> Forgive my language but, hell no. This is not the gigantic Lago
> bare metals you are used to. We don't want GWT builds to start
> failing on running out of RAM.
>
Buy more RAM.
This is the best solution as having the cache on the ram will shorten the
time of engine jobs.
Y.
>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
10:28 a.m.
To get back to the original point - I do not see a connection with imageio
anywhere. It's libguestfs's temp dir. Now to decide what to do with it I
think we should first understand which test uses/invokes libguestfs and for
what purpose?
On 24 May 2017, at 12:35, Gil Shinar <gshinar(a)redhat.com> wrote:
On Wed, May 24, 2017 at 12:38 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Wed, May 24, 2017 at 11:35 AM, Barak Korren <bkorren(a)redhat.com> wrote:
> On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
> >
> > /dev/shm is just as good. It's only 400MB.
> > Y.
> >
> Forgive my language but, hell no. This is not the gigantic Lago
> bare metals you are used to. We don't want GWT builds to start
> failing on running out of RAM.
>
Buy more RAM.
This is the best solution as having the cache on the ram will shorten the
time of engine jobs.
Y.
>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2:58 a.m.
I wrote that it was imageio because I have disabled deletion of /var/tmp on
one job only (jenkins check-patch) and saw that on the same Jenkins slave
only imageio check-patch and Jenkins check-patch run. Jenkins check-patch
has nothing to do with libguestfs so I assumed that imageio did.
Here is a list of running jobs on the slave I have checked /var/tmp on. The
imageio job cleans /var/tmp and jenkins job doesn't.
[image: Inline image 1]
Anyhow, I'll take your word on that and assume that the Jenkins build
history has bugs and a VDSM or some other job run on that slave.
Now lets go back to the main interest of this thread. If we'll know, that
whatever is being written to /var/tmp, can be considered as cache and can
be used by the next run of the job that uses it, it might be a good idea
not to clean /var/tmp. Jenkins is helping us with that by trying to run
jobs on the same slave as much as possible.
We will start by monitoring our disks constantly to see how fast, if at
all, they are getting full.
On Wed, May 24, 2017 at 6:28 PM, Michal Skrivanek <mskrivan(a)redhat.com>
wrote:
To get back to the original point - I do not see a connection with
imageio
anywhere. It's libguestfs's temp dir. Now to decide what to do with it I
think we should first understand which test uses/invokes libguestfs and for
what purpose?
On 24 May 2017, at 12:35, Gil Shinar <gshinar(a)redhat.com> wrote:
On Wed, May 24, 2017 at 12:38 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
>
> On Wed, May 24, 2017 at 11:35 AM, Barak Korren <bkorren(a)redhat.com>
> wrote:
>
>> On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>> >
>> > /dev/shm is just as good. It's only 400MB.
>> > Y.
>> >
>> Forgive my language but, hell no. This is not the gigantic Lago
>> bare metals you are used to. We don't want GWT builds to start
>> failing on running out of RAM.
>>
>
> Buy more RAM.
>
This is the best solution as having the cache on the ram will shorten the
time of engine jobs.
> Y.
>
>
>>
>> --
>> Barak Korren
>> RHV DevOps team , RHCE, RHCi
>> Red Hat EMEA
>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>
>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
3:22 a.m.
בתאריך יום ה׳, 25 במאי 2017, 10:59, מאת Gil Shinar <gshinar(a)redhat.com>:
I wrote that it was imageio because I have disabled deletion of
/var/tmp
on one job only (jenkins check-patch) and saw that on the same Jenkins
slave only imageio check-patch and Jenkins check-patch run. Jenkins
check-patch has nothing to do with libguestfs so I assumed that imageio did.
Here is a list of running jobs on the slave I have checked /var/tmp on.
The imageio job cleans /var/tmp and jenkins job doesn't.
[image: Inline image 1]
Anyhow, I'll take your word on that and assume that the Jenkins build
history has bugs and a VDSM or some other job run on that slave.
Now lets go back to the main interest of this thread. If we'll know, that
whatever is being written to /var/tmp, can be considered as cache and can
be used by the next run of the job that uses it, it might be a good idea
not to clean /var/tmp. Jenkins is helping us with that by trying to run
jobs on the same slave as much as possible.
Vdsm and ovirt-imageio use /var/tmp because we need file system supporting
direct I/O. /tmp is using tmpfs which does not support it.
We have no need for "cached" data kept after a test run, and we cannot
promise that test will never leave junk in /var/tmp since tests run before
they are reviewed. Even correct tests can leave junk if the test runner is
killed (for example, on timeout).
The only way to keep slaves clean is to clean /tmp and /var/tmp after each
run. Treating /var/tmp as cache is very wrong.
We will start by monitoring our disks constantly to see how fast, if at
all, they are getting full.
On Wed, May 24, 2017 at 6:28 PM, Michal Skrivanek <mskrivan(a)redhat.com>
wrote:
> To get back to the original point - I do not see a connection with
> imageio anywhere. It's libguestfs's temp dir. Now to decide what to do with
> it I think we should first understand which test uses/invokes libguestfs
> and for what purpose?
>
> On 24 May 2017, at 12:35, Gil Shinar <gshinar(a)redhat.com> wrote:
>
> On Wed, May 24, 2017 at 12:38 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
>>
>>
>> On Wed, May 24, 2017 at 11:35 AM, Barak Korren <bkorren(a)redhat.com>
>> wrote:
>>
>>> On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>>> >
>>> > /dev/shm is just as good. It's only 400MB.
>>> > Y.
>>> >
>>> Forgive my language but, hell no. This is not the gigantic Lago
>>> bare metals you are used to. We don't want GWT builds to start
>>> failing on running out of RAM.
>>>
>>
>> Buy more RAM.
>>
>
> This is the best solution as having the cache on the ram will shorten the
> time of engine jobs.
>
>> Y.
>>
>>
>>>
>>> --
>>> Barak Korren
>>> RHV DevOps team , RHCE, RHCi
>>> Red Hat EMEA
>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
4:11 a.m.
On 25 May 2017 at 11:22, Nir Soffer <nsoffer(a)redhat.com> wrote:
Vdsm and ovirt-imageio use /var/tmp because we need file system
supporting
direct I/O. /tmp is using tmpfs which does not support it.
We have no need for "cached" data kept after a test run, and we cannot
promise that test will never leave junk in /var/tmp since tests run before
they are reviewed. Even correct tests can leave junk if the test runner is
killed (for example, on timeout).
The only way to keep slaves clean is to clean /tmp and /var/tmp after each
run. Treating /var/tmp as cache is very wrong.
You need to differentiate between the '/var/tmp' you see from your
scripts to the one we are talking about here.
- When you use /var/tmp in your script you use the one inside the mock
environment. It is specific to yore script run time environment and
will always be wiped out when its done.
- We are talking about "/var/tmp" _of_the_execution_slave_, the only
way you can get to it is either specifically bind-mount it from the
"*.mounts" file, or have some daemon like libvirtd or dockerd write
to it.
BTW if you want any guarantees about the FS you are using, you better
bind-mount something to the point you are writing to, otherwise things
will break when we make infrastructure changes like for example moving
the chroots to RAM or onto layered file-systems.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
7:42 a.m.
On Thu, May 25, 2017 at 12:11 PM Barak Korren <bkorren(a)redhat.com> wrote:
On 25 May 2017 at 11:22, Nir Soffer <nsoffer(a)redhat.com>
wrote:
> Vdsm and ovirt-imageio use /var/tmp because we need file system
> supporting direct I/O. /tmp is using tmpfs which does not support it.
>
> We have no need for "cached" data kept after a test run, and we cannot
> promise that test will never leave junk in /var/tmp since tests run before
> they are reviewed. Even correct tests can leave junk if the test runner is
> killed (for example, on timeout).
>
> The only way to keep slaves clean is to clean /tmp and /var/tmp after
> each run. Treating /var/tmp as cache is very wrong.
>
You need to differentiate between the '/var/tmp' you see from your
scripts to the one we are talking about here.
- When you use /var/tmp in your script you use the one inside the mock
environment. It is specific to yore script run time environment and
will always be wiped out when its done.
Great, this is what we need.
- We are talking about "/var/tmp" _of_the_execution_slave_, the only
way you can get to it is either specifically bind-mount it from the
"*.mounts" file, or have some daemon like libvirtd or dockerd write
to it.
In this case, I don't see how vdsm tests can pollute the host /var/tmp.
Vdsm run 2 tests running virt-alignment-scan, one with --help, and one
with non existing images, so the temporary directory cannot be created
by these tests.
BTW if you want any guarantees about the FS you are using, you better
bind-mount something to the point you are writing to, otherwise things
will break when we make infrastructure changes like for example moving
the chroots to RAM or onto layered file-systems.
We need a location which exists on developer laptop, developer hosts,
and CI environments, and /var/tmp proved to be a good choice so far.
We expect that ovirt CI will not break this assumption in the future.
However, writing test data to storage is a waste of time, and having
a memory based file system supporting direct I/O would speed up lot
of tests.
So we can do this:
truncate -s 5g /tmp/backing
mkfs.ext4 /tmp/backing
mount -o loop /tmp/backing /tmp/mnt
And now we have direct I/O support and great performance:
$ dd if=/dev/zero of=/tmp/mnt/direct-io-test bs=8M count=128 oflag=direct
128+0 records in
128+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.536528 s, 2.0 GB/s
This greatly speeds up some tests which are marked a slow tests, and
never run unless using --enable-slow-tests.
Without slow tests using /var/tmp:
$ ./run_tests_local.sh storage_qcow2_test.py -s
...
Ran 31 tests in 0.709s
With slow tests using loop device based temporary directory:
$ ./run_tests_local.sh storage_qcow2_test.py --enable-slow-tests -s
...
Ran 31 tests in 7.019s
With slow tests, using /var/tmp:
$ ./run_tests_local.sh storage_qcow2_test.py --enable-slow-tests -s
...
Ran 31 tests in 90.491s
This requires root to mounting and unmounting the backing file, so
it is not a good solution for developers, when you need to run certain
tests all the time, but it can be a good solution for the CI.
Barak, do you think ovirt CI can provide this functionality?
Nir
8 a.m.
On 25 May 2017 at 15:42, Nir Soffer <nsoffer(a)redhat.com> wrote:
In this case, I don't see how vdsm tests can pollute the host /var/tmp.
Vdsm run 2 tests running virt-alignment-scan, one with --help, and one
with non existing images, so the temporary directory cannot be created
by these tests.
As you see in $subject, we are looking for something that is invoking
libgustfs, could vdsm or imageio be doing that?
We need a location which exists on developer laptop, developer
hosts,
and CI environments, and /var/tmp proved to be a good choice so far.
We expect that ovirt CI will not break this assumption in the future.
One thing you can do is use something like "/var/tmp/vdst_test" and
also include that string in an "automation/*.mounts" file. This will
cause 'mock_runner.sh' to mkdir and bind-mount it into the environment
so you are using the hosts real '/var/tmp/vdsm_test' which will
probably remain a real file-system for the time being.
But if you do this, please clean up when you're done...
However, writing test data to storage is a waste of time, and having
a memory based file system supporting direct I/O would speed up lot
of tests.
So we can do this:
truncate -s 5g /tmp/backing
mkfs.ext4 /tmp/backing
mount -o loop /tmp/backing /tmp/mnt
And now we have direct I/O support and great performance:
This requires root to mounting and unmounting the backing file, so
it is not a good solution for developers, when you need to run certain
tests all the time, but it can be a good solution for the CI.
Have you tried doing this from inside mock? You are root there so it might work.
But note that /tmp is not in RAM in EL7. Better use /dev/shm for compatibility.
Barak, do you think ovirt CI can provide this functionality?
I guess we could come up with some special syntax in the '*.mounts'
file to support something like this, but I'd rather not add any big
features to mock_runner before we rewrite it to use containers.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
8:31 a.m.
On Thu, May 25, 2017 at 4:00 PM Barak Korren <bkorren(a)redhat.com> wrote:
On 25 May 2017 at 15:42, Nir Soffer <nsoffer(a)redhat.com>
wrote:
>
> In this case, I don't see how vdsm tests can pollute the host /var/tmp.
>
> Vdsm run 2 tests running virt-alignment-scan, one with --help, and one
> with non existing images, so the temporary directory cannot be created
> by these tests.
As you see in $subject, we are looking for something that is invoking
libgustfs, could vdsm or imageio be doing that?
ovirt-imagio does not invoke any external program.
Vdsm tests invoke virt-alignment-scan but not in a way that can leave
anything around:
LIBGUESTFS_BACKEND=direct virt-alignment-scan --help
LIBGUESTFS_BACKEND=direct virt-alignment-scan --add no-such-file
I don't know about other tests using libguestfs.
But how a program running inside mock chroot can access host's /var/tmp?
8:40 a.m.
But how a program running inside mock chroot can access host's /var/tmp?
By talking to libvirtd that runs outside.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
8:46 a.m.
On Thu, May 25, 2017 at 4:41 PM Barak Korren <bkorren(a)redhat.com> wrote:
>
> But how a program running inside mock chroot can access host's /var/tmp?
By talking to libvirtd that runs outside.
We don't require or use running libvirtd in the tests.
Adding Francesco to confirm.
9:51 a.m.
On 25 May 2017, at 15:40, Barak Korren <bkorren(a)redhat.com>
wrote:
>
> But how a program running inside mock chroot can access host's /var/tmp?
By talking to libvirtd that runs outside.
Can you please add more details about the file you’ve seen?
Was it a single file? When it was created? What does it contain?
What kind of other tests are being executed on this host? Are you sure it’s from
check-patch?
Thanks,
michal
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
2:56 p.m.
On Thu, May 25, 2017 at 10:58 AM, Gil Shinar <gshinar(a)redhat.com> wrote:
I wrote that it was imageio because I have disabled deletion of
/var/tmp
on one job only (jenkins check-patch) and saw that on the same Jenkins
slave only imageio check-patch and Jenkins check-patch run. Jenkins
check-patch has nothing to do with libguestfs so I assumed that imageio did.
Here is a list of running jobs on the slave I have checked /var/tmp on.
The imageio job cleans /var/tmp and jenkins job doesn't.
I thought it was ovirt-system-tests (Lago specifically) using virt-* tools.
Y.
[image: Inline image 1]
Anyhow, I'll take your word on that and assume that the Jenkins build
history has bugs and a VDSM or some other job run on that slave.
Now lets go back to the main interest of this thread. If we'll know, that
whatever is being written to /var/tmp, can be considered as cache and can
be used by the next run of the job that uses it, it might be a good idea
not to clean /var/tmp. Jenkins is helping us with that by trying to run
jobs on the same slave as much as possible.
We will start by monitoring our disks constantly to see how fast, if at
all, they are getting full.
On Wed, May 24, 2017 at 6:28 PM, Michal Skrivanek <mskrivan(a)redhat.com>
wrote:
> To get back to the original point - I do not see a connection with
> imageio anywhere. It's libguestfs's temp dir. Now to decide what to do with
> it I think we should first understand which test uses/invokes libguestfs
> and for what purpose?
>
> On 24 May 2017, at 12:35, Gil Shinar <gshinar(a)redhat.com> wrote:
>
> On Wed, May 24, 2017 at 12:38 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
>>
>>
>> On Wed, May 24, 2017 at 11:35 AM, Barak Korren <bkorren(a)redhat.com>
>> wrote:
>>
>>> On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>>> >
>>> > /dev/shm is just as good. It's only 400MB.
>>> > Y.
>>> >
>>> Forgive my language but, hell no. This is not the gigantic Lago
>>> bare metals you are used to. We don't want GWT builds to start
>>> failing on running out of RAM.
>>>
>>
>> Buy more RAM.
>>
>
> This is the best solution as having the cache on the ram will shorten the
> time of engine jobs.
>
>> Y.
>>
>>
>>>
>>> --
>>> Barak Korren
>>> RHV DevOps team , RHCE, RHCi
>>> Red Hat EMEA
>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>
>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
>
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
3:55 a.m.
One thing I was missing and Barak found is that our cleanup script does not
remove files and folders that starts with a dot.
It is, obviously, a bug but it means that the build history screen shot I
have pasted here doesn't have anything to do with the .guestfs-0 folder.
As much as I know, system tests jobs are running only on bare metal slaves.
The slave I saw this folder on was a VM.
On Thu, May 25, 2017 at 10:56 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
On Thu, May 25, 2017 at 10:58 AM, Gil Shinar <gshinar(a)redhat.com> wrote:
> I wrote that it was imageio because I have disabled deletion of /var/tmp
> on one job only (jenkins check-patch) and saw that on the same Jenkins
> slave only imageio check-patch and Jenkins check-patch run. Jenkins
> check-patch has nothing to do with libguestfs so I assumed that imageio did.
> Here is a list of running jobs on the slave I have checked /var/tmp on.
> The imageio job cleans /var/tmp and jenkins job doesn't.
>
I thought it was ovirt-system-tests (Lago specifically) using virt-* tools.
Y.
> [image: Inline image 1]
>
> Anyhow, I'll take your word on that and assume that the Jenkins build
> history has bugs and a VDSM or some other job run on that slave.
>
> Now lets go back to the main interest of this thread. If we'll know, that
> whatever is being written to /var/tmp, can be considered as cache and can
> be used by the next run of the job that uses it, it might be a good idea
> not to clean /var/tmp. Jenkins is helping us with that by trying to run
> jobs on the same slave as much as possible.
> We will start by monitoring our disks constantly to see how fast, if at
> all, they are getting full.
>
> On Wed, May 24, 2017 at 6:28 PM, Michal Skrivanek <mskrivan(a)redhat.com>
> wrote:
>
>> To get back to the original point - I do not see a connection with
>> imageio anywhere. It's libguestfs's temp dir. Now to decide what to do
with
>> it I think we should first understand which test uses/invokes libguestfs
>> and for what purpose?
>>
>> On 24 May 2017, at 12:35, Gil Shinar <gshinar(a)redhat.com> wrote:
>>
>> On Wed, May 24, 2017 at 12:38 PM, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>>
>>>
>>>
>>> On Wed, May 24, 2017 at 11:35 AM, Barak Korren <bkorren(a)redhat.com>
>>> wrote:
>>>
>>>> On 24 May 2017 at 11:17, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>>>> >
>>>> > /dev/shm is just as good. It's only 400MB.
>>>> > Y.
>>>> >
>>>> Forgive my language but, hell no. This is not the gigantic Lago
>>>> bare metals you are used to. We don't want GWT builds to start
>>>> failing on running out of RAM.
>>>>
>>>
>>> Buy more RAM.
>>>
>>
>> This is the best solution as having the cache on the ram will shorten
>> the time of engine jobs.
>>
>>> Y.
>>>
>>>
>>>>
>>>> --
>>>> Barak Korren
>>>> RHV DevOps team , RHCE, RHCi
>>>> Red Hat EMEA
>>>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>>>>
>>>
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel(a)ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>> _______________________________________________
>> Devel mailing list
>> Devel(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/devel
>>
>>
>
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
3:34 a.m.
On 24 May 2017 at 10:30, Nir Soffer <nsoffer(a)redhat.com> wrote:
Please keep this behavior.
<snip>
Use /var/cache?
Can`t. In order to keep allowing local use of "mock_runner.sh",
the directory needs to:
1. Exist on every system by default
2. Be accessible from a user level (not root)
"/var/cache" is root owned. Stuff you see there gets 'mkdir'ed
and 'chown'ed by RPM post-install scripts.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
3:17 a.m.
On Wed, May 24, 2017 at 9:45 AM, Barak Korren <bkorren(a)redhat.com> wrote:
On 23 May 2017 at 18:34, Yaniv Kaul <ykaul(a)redhat.com> wrote:
>
> It's in /var/tmp and it's supposed to be cached. Is there an issues with
> this? It's 400MB image, AFAIR.
We currently have /var/tmp wiped out after each and every job run. We
are looking into stopping that to allow it to be used for persistent
caches, but we don't want the slaves to fill up as a result.
We we need to understand how fast may this accumulate.
It's the exact same image, I don't see it adding up.
Y.
400MB accumulation per run is a lot. Unless this is stable and also
gets recycled automatically.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
2533
days inactive
2538
days old
24 comments
6 participants
participants (6)
-
Barak Korren -
Gil Shinar -
Michal Skrivanek -
Nadav Goldin -
Nir Soffer -
Yaniv Kaul