On Tue, Apr 3, 2018 at 9:32 AM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tes... > > 10:44:32 FS info: > 10:44:32 + df -h > 10:44:32 Filesystem Size Used Avail Use% Mounted on > 10:44:32 /dev/sda3 908G 64G 844G 8% / > 10:44:32 tmpfs 24G 3.8G 20G 16% /dev/shm > 10:44:32 tmpfs 24G 34M 24G 1% /run/libvirt > 10:44:32 + echo 'RAM info:' > 10:44:32 RAM info: > 10:44:32 + free -h > 10:44:32 total used free shared > buff/cache available > 10:44:32 Mem: 46G 4.4G 333M 3.8G > 42G 38G > 10:44:32 Swap: 23G 0B 23G > 10:44:32 + [[ -d > /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default ]] > 10:44:32 + echo 'Prefix size:' > 10:44:32 Prefix size: > 10:44:32 + du -h -d 1 > /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default > 10:44:32 592M /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/internal_repo > 10:44:32 20K /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/virt > 10:44:32 16K /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/scripts > 10:44:32 3.2G /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images > 10:44:32 2.7M /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/logs > 10:44:32 3.8G /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default > 10:44:32 + [[ -d > /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images > ]] > 10:44:32 + echo 'Images Size:' > 10:44:32 Images Size: > 10:44:32 + ls -lhs > /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images > 10:44:32 total 3.2G > 10:44:32 16K -rw-rw-rw-. 1 1015 1015 101G Apr 2 10:29 > lago-upgrade-from-prevrelease-suite-4-1-engine_iscsi.raw > 10:44:32 1.8G -rw-rw-rw-. 1 1015 1015 101G Apr 2 10:30 > lago-upgrade-from-prevrelease-suite-4-1-engine_nfs.raw > 10:44:32 1.5G -rw-rw-rw-. 1 1015 1015 1.5G Apr 2 10:44 > lago-upgrade-from-prevrelease-suite-4-1-engine_root.qcow2 > 10:44:32 + echo 'suite.sh: moving artifacts' > 10:44:32 suite.sh: moving artifacts > 10:44:32 + [[ -d > /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/current/logs > ]] > 10:44:32 + mv /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/current/logs > exported-artifacts/lago_logs > 10:44:32 + [[ -d exported_images ]] > 10:44:32 + [[ -d /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1 ]] > 10:44:32 + find > /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1 -type f > -iname '*.junit.xml' -exec mv '{}' exported-artifacts/ ';' > 10:44:32 + [[ -d test_logs ]] > 10:44:32 + mv test_logs exported-artifacts/ > > This looks ok to me. Then: > > 10:44:46 Archiving artifacts > 10:46:04 ERROR: Failed to archive artifacts: exported-artifacts/** > 10:46:04 java.io.EOFException: Unexpected end of ZLIB input stream > 10:46:04 at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:186) > 10:46:04 at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106) > 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:614) > 10:46:04 at java.io.InputStream.read(InputStream.java:101) > 10:46:04 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) > 10:46:04 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) > 10:46:04 at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) > 10:46:04 at hudson.util.IOUtils.copy(IOUtils.java:40) > 10:46:04 at hudson.FilePath.readFromTar(FilePath.java:2307) > 10:46:04 Also: Also: hudson.remoting.Channel$CallSiteStackTrace: > Remote call to ovirt-srv19.phx.ovirt.org > 10:46:04 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693) > 10:46:04 at hudson.remoting.UserResponse.retrieve(UserRequest.java:310) > 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:944) > 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:941) > 10:46:04 at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) > 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2224) > 10:46:04 at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61) > 10:46:04 at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:245) > 10:46:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) > 10:46:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) > 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) > 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) > 10:46:04 at hudson.model.Build$BuildExecution.post2(Build.java:186) > 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) > 10:46:04 at hudson.model.Run.execute(Run.java:1749) > 10:46:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > 10:46:04 at hudson.model.ResourceController.execute(ResourceController.java:97) > 10:46:04 at hudson.model.Executor.run(Executor.java:429) > 10:46:04 java.io.IOException: This archives contains unclosed entries. > 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225) > 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241) > 10:46:04 at hudson.util.io.TarArchiver.close(TarArchiver.java:127) > 10:46:04 at hudson.FilePath.writeToTar(FilePath.java:2281) > 10:46:04 at hudson.FilePath.access$2100(FilePath.java:196) > 10:46:04 at hudson.FilePath$45.invoke(FilePath.java:2216) > 10:46:04 at hudson.FilePath$45.invoke(FilePath.java:2212) > 10:46:04 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2760) > 10:46:04 at hudson.remoting.UserRequest.perform(UserRequest.java:207) > 10:46:04 at hudson.remoting.UserRequest.perform(UserRequest.java:53) > 10:46:04 at hudson.remoting.Request$2.run(Request.java:358) > 10:46:04 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) > 10:46:04 at java.util.concurrent.FutureTask.run(FutureTask.java:266) > 10:46:04 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > 10:46:04 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > 10:46:04 at java.lang.Thread.run(Thread.java:748) > 10:46:04 Caused: java.util.concurrent.ExecutionException > 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:946) > 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:941) > 10:46:04 at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) > 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2224) > 10:46:04 Caused: java.io.IOException: Failed to extract > /home/jenkins/workspace/ovirt-system-tests_manual/transfer of 8678 > files I tend to guess this is the root cause. I wanted to collect all of /etc, perhaps that's too many files for jenkins. artifacts: - /etc/*ovirt*

Are there objections to a patch that will fix this to support wildcards? We have these on a typical engine machine, I'd rather not list all of them: ovirt-engine ovirt-engine-dwh ovirt-engine-metrics ovirt-engine-setup.conf.d ovirt-engine-setup.env.d ovirt-host-deploy.conf.d ovirt-imageio-proxy ovirt-provider-ovn ovirt-vmconsole ovirt-web-ui > 10:46:04 at hudson.FilePath.readFromTar(FilePath.java:2317) > 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2221) > 10:46:04 at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61) > 10:46:04 at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:245) > 10:46:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) > 10:46:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) > 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) > 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) > 10:46:04 at hudson.model.Build$BuildExecution.post2(Build.java:186) > 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) > 10:46:04 at hudson.model.Run.execute(Run.java:1749) > 10:46:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > 10:46:04 at hudson.model.ResourceController.execute(ResourceController.java:97) > 10:46:04 at hudson.model.Executor.run(Executor.java:429) > 10:46:04 Build step 'Archive the artifacts' changed build result to FAILURE > > Any idea? > > Now running it again, let's see what happens. > > http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tes... > -- > Didi -- Didi _______________________________________________ Infra mailing list Infra(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/infra

On Wed, Apr 4, 2018 at 9:16 AM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > On 3 April 2018 at 09:57, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: >> On Tue, Apr 3, 2018 at 9:32 AM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: >>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tes... >>> >>> 10:44:32 FS info: >>> 10:44:32 + df -h >>> 10:44:32 Filesystem Size Used Avail Use% Mounted on >>> 10:44:32 /dev/sda3 908G 64G 844G 8% / >>> 10:44:32 tmpfs 24G 3.8G 20G 16% /dev/shm >>> 10:44:32 tmpfs 24G 34M 24G 1% /run/libvirt >>> 10:44:32 + echo 'RAM info:' >>> 10:44:32 RAM info: >>> 10:44:32 + free -h >>> 10:44:32 total used free shared >>> buff/cache available >>> 10:44:32 Mem: 46G 4.4G 333M 3.8G >>> 42G 38G >>> 10:44:32 Swap: 23G 0B 23G >>> 10:44:32 + [[ -d >>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default ]] >>> 10:44:32 + echo 'Prefix size:' >>> 10:44:32 Prefix size: >>> 10:44:32 + du -h -d 1 >>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default >>> 10:44:32 592M /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/internal_repo >>> 10:44:32 20K /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/virt >>> 10:44:32 16K /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/scripts >>> 10:44:32 3.2G /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images >>> 10:44:32 2.7M /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/logs >>> 10:44:32 3.8G /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default >>> 10:44:32 + [[ -d >>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images >>> ]] >>> 10:44:32 + echo 'Images Size:' >>> 10:44:32 Images Size: >>> 10:44:32 + ls -lhs >>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images >>> 10:44:32 total 3.2G >>> 10:44:32 16K -rw-rw-rw-. 1 1015 1015 101G Apr 2 10:29 >>> lago-upgrade-from-prevrelease-suite-4-1-engine_iscsi.raw >>> 10:44:32 1.8G -rw-rw-rw-. 1 1015 1015 101G Apr 2 10:30 >>> lago-upgrade-from-prevrelease-suite-4-1-engine_nfs.raw >>> 10:44:32 1.5G -rw-rw-rw-. 1 1015 1015 1.5G Apr 2 10:44 >>> lago-upgrade-from-prevrelease-suite-4-1-engine_root.qcow2 >>> 10:44:32 + echo 'suite.sh: moving artifacts' >>> 10:44:32 suite.sh: moving artifacts >>> 10:44:32 + [[ -d >>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/current/logs >>> ]] >>> 10:44:32 + mv /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/current/logs >>> exported-artifacts/lago_logs >>> 10:44:32 + [[ -d exported_images ]] >>> 10:44:32 + [[ -d /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1 ]] >>> 10:44:32 + find >>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1 -type f >>> -iname '*.junit.xml' -exec mv '{}' exported-artifacts/ ';' >>> 10:44:32 + [[ -d test_logs ]] >>> 10:44:32 + mv test_logs exported-artifacts/ >>> >>> This looks ok to me. Then: >>> >>> 10:44:46 Archiving artifacts >>> 10:46:04 ERROR: Failed to archive artifacts: exported-artifacts/** >>> 10:46:04 java.io.EOFException: Unexpected end of ZLIB input stream >>> 10:46:04 at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:186) >>> 10:46:04 at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106) >>> 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:614) >>> 10:46:04 at java.io.InputStream.read(InputStream.java:101) >>> 10:46:04 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) >>> 10:46:04 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) >>> 10:46:04 at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) >>> 10:46:04 at hudson.util.IOUtils.copy(IOUtils.java:40) >>> 10:46:04 at hudson.FilePath.readFromTar(FilePath.java:2307) >>> 10:46:04 Also: Also: hudson.remoting.Channel$CallSiteStackTrace: >>> Remote call to ovirt-srv19.phx.ovirt.org >>> 10:46:04 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693) >>> 10:46:04 at hudson.remoting.UserResponse.retrieve(UserRequest.java:310) >>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:944) >>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:941) >>> 10:46:04 at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) >>> 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2224) >>> 10:46:04 at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61) >>> 10:46:04 at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:245) >>> 10:46:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) >>> 10:46:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) >>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) >>> 10:46:04 at hudson.model.Build$BuildExecution.post2(Build.java:186) >>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) >>> 10:46:04 at hudson.model.Run.execute(Run.java:1749) >>> 10:46:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>> 10:46:04 at hudson.model.ResourceController.execute(ResourceController.java:97) >>> 10:46:04 at hudson.model.Executor.run(Executor.java:429) >>> 10:46:04 java.io.IOException: This archives contains unclosed entries. >>> 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225) >>> 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241) >>> 10:46:04 at hudson.util.io.TarArchiver.close(TarArchiver.java:127) >>> 10:46:04 at hudson.FilePath.writeToTar(FilePath.java:2281) >>> 10:46:04 at hudson.FilePath.access$2100(FilePath.java:196) >>> 10:46:04 at hudson.FilePath$45.invoke(FilePath.java:2216) >>> 10:46:04 at hudson.FilePath$45.invoke(FilePath.java:2212) >>> 10:46:04 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2760) >>> 10:46:04 at hudson.remoting.UserRequest.perform(UserRequest.java:207) >>> 10:46:04 at hudson.remoting.UserRequest.perform(UserRequest.java:53) >>> 10:46:04 at hudson.remoting.Request$2.run(Request.java:358) >>> 10:46:04 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) >>> 10:46:04 at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> 10:46:04 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> 10:46:04 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> 10:46:04 at java.lang.Thread.run(Thread.java:748) >>> 10:46:04 Caused: java.util.concurrent.ExecutionException >>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:946) >>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:941) >>> 10:46:04 at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) >>> 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2224) >>> 10:46:04 Caused: java.io.IOException: Failed to extract >>> /home/jenkins/workspace/ovirt-system-tests_manual/transfer of 8678 >>> files >> >> I tend to guess this is the root cause. I wanted to collect all of /etc, >> perhaps that's too many files for jenkins. >> >> artifacts: >> - /etc/*ovirt* > > > I don't think jenkins itself will have any issue with it (given that > we have stdci - its easy to check). Not sure what you mean - it did fail. Do you suggest some other test to check if the cause is "too many files" or something else? Searching google for "IOException: Failed to extract" does find several similar reports, didn't try to investigate further.

> >> >> Are there objections to a patch that will fix this to support wildcards? >> We have these on a typical engine machine, I'd rather not list all of them: >> >> ovirt-engine >> ovirt-engine-dwh >> ovirt-engine-metrics >> ovirt-engine-setup.conf.d >> ovirt-engine-setup.env.d >> ovirt-host-deploy.conf.d >> ovirt-imageio-proxy >> ovirt-provider-ovn >> ovirt-vmconsole >> ovirt-web-ui Currently did, for now, and it's working well: https://gerrit.ovirt.org/89712 >> >>> 10:46:04 at hudson.FilePath.readFromTar(FilePath.java:2317) >>> 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2221) >>> 10:46:04 at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61) >>> 10:46:04 at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:245) >>> 10:46:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) >>> 10:46:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) >>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) >>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690) >>> 10:46:04 at hudson.model.Build$BuildExecution.post2(Build.java:186) >>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635) >>> 10:46:04 at hudson.model.Run.execute(Run.java:1749) >>> 10:46:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) >>> 10:46:04 at hudson.model.ResourceController.execute(ResourceController.java:97) >>> 10:46:04 at hudson.model.Executor.run(Executor.java:429) >>> 10:46:04 Build step 'Archive the artifacts' changed build result to FAILURE >>> >>> Any idea? >>> >>> Now running it again, let's see what happens. >>> >>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tes... >>> -- >>> Didi >> >> >> >> -- >> Didi >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra > > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted -- Didi