Collecting /etc (was: Unexpected end of ZLIB input stream)

Barak Korren bkorren at redhat.com
Wed Apr 4 06:59:02 UTC 2018 

On 4 April 2018 at 09:36, Yedidyah Bar David <didi at redhat.com> wrote:
> On Wed, Apr 4, 2018 at 9:16 AM, Barak Korren <bkorren at redhat.com> wrote:
>> On 3 April 2018 at 09:57, Yedidyah Bar David <didi at redhat.com> wrote:
>>> On Tue, Apr 3, 2018 at 9:32 AM, Yedidyah Bar David <didi at redhat.com> wrote:
>>>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/2488/consoleFull
>>>>
>>>> 10:44:32 FS info:
>>>> 10:44:32 + df -h
>>>> 10:44:32 Filesystem      Size  Used Avail Use% Mounted on
>>>> 10:44:32 /dev/sda3       908G   64G  844G   8% /
>>>> 10:44:32 tmpfs            24G  3.8G   20G  16% /dev/shm
>>>> 10:44:32 tmpfs            24G   34M   24G   1% /run/libvirt
>>>> 10:44:32 + echo 'RAM info:'
>>>> 10:44:32 RAM info:
>>>> 10:44:32 + free -h
>>>> 10:44:32               total        used        free      shared
>>>> buff/cache   available
>>>> 10:44:32 Mem:            46G        4.4G        333M        3.8G
>>>>   42G         38G
>>>> 10:44:32 Swap:           23G          0B         23G
>>>> 10:44:32 + [[ -d
>>>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default ]]
>>>> 10:44:32 + echo 'Prefix size:'
>>>> 10:44:32 Prefix size:
>>>> 10:44:32 + du -h -d 1
>>>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default
>>>> 10:44:32 592M /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/internal_repo
>>>> 10:44:32 20K /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/virt
>>>> 10:44:32 16K /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/scripts
>>>> 10:44:32 3.2G /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images
>>>> 10:44:32 2.7M /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/logs
>>>> 10:44:32 3.8G /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default
>>>> 10:44:32 + [[ -d
>>>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images
>>>> ]]
>>>> 10:44:32 + echo 'Images Size:'
>>>> 10:44:32 Images Size:
>>>> 10:44:32 + ls -lhs
>>>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/default/images
>>>> 10:44:32 total 3.2G
>>>> 10:44:32  16K -rw-rw-rw-. 1 1015 1015 101G Apr  2 10:29
>>>> lago-upgrade-from-prevrelease-suite-4-1-engine_iscsi.raw
>>>> 10:44:32 1.8G -rw-rw-rw-. 1 1015 1015 101G Apr  2 10:30
>>>> lago-upgrade-from-prevrelease-suite-4-1-engine_nfs.raw
>>>> 10:44:32 1.5G -rw-rw-rw-. 1 1015 1015 1.5G Apr  2 10:44
>>>> lago-upgrade-from-prevrelease-suite-4-1-engine_root.qcow2
>>>> 10:44:32 + echo 'suite.sh: moving artifacts'
>>>> 10:44:32 suite.sh: moving artifacts
>>>> 10:44:32 + [[ -d
>>>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/current/logs
>>>> ]]
>>>> 10:44:32 + mv /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1/current/logs
>>>> exported-artifacts/lago_logs
>>>> 10:44:32 + [[ -d exported_images ]]
>>>> 10:44:32 + [[ -d /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1 ]]
>>>> 10:44:32 + find
>>>> /dev/shm/ost/deployment-upgrade-from-prevrelease-suite-4.1 -type f
>>>> -iname '*.junit.xml' -exec mv '{}' exported-artifacts/ ';'
>>>> 10:44:32 + [[ -d test_logs ]]
>>>> 10:44:32 + mv test_logs exported-artifacts/
>>>>
>>>> This looks ok to me. Then:
>>>>
>>>> 10:44:46 Archiving artifacts
>>>> 10:46:04 ERROR: Failed to archive artifacts: exported-artifacts/**
>>>> 10:46:04 java.io.EOFException: Unexpected end of ZLIB input stream
>>>> 10:46:04 at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:186)
>>>> 10:46:04 at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
>>>> 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:614)
>>>> 10:46:04 at java.io.InputStream.read(InputStream.java:101)
>>>> 10:46:04 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
>>>> 10:46:04 at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
>>>> 10:46:04 at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
>>>> 10:46:04 at hudson.util.IOUtils.copy(IOUtils.java:40)
>>>> 10:46:04 at hudson.FilePath.readFromTar(FilePath.java:2307)
>>>> 10:46:04 Also:   Also:   hudson.remoting.Channel$CallSiteStackTrace:
>>>> Remote call to ovirt-srv19.phx.ovirt.org
>>>> 10:46:04 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
>>>> 10:46:04 at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
>>>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:944)
>>>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:941)
>>>> 10:46:04 at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
>>>> 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2224)
>>>> 10:46:04 at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61)
>>>> 10:46:04 at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:245)
>>>> 10:46:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
>>>> 10:46:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
>>>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
>>>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
>>>> 10:46:04 at hudson.model.Build$BuildExecution.post2(Build.java:186)
>>>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
>>>> 10:46:04 at hudson.model.Run.execute(Run.java:1749)
>>>> 10:46:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>>> 10:46:04 at hudson.model.ResourceController.execute(ResourceController.java:97)
>>>> 10:46:04 at hudson.model.Executor.run(Executor.java:429)
>>>> 10:46:04 java.io.IOException: This archives contains unclosed entries.
>>>> 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225)
>>>> 10:46:04 at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241)
>>>> 10:46:04 at hudson.util.io.TarArchiver.close(TarArchiver.java:127)
>>>> 10:46:04 at hudson.FilePath.writeToTar(FilePath.java:2281)
>>>> 10:46:04 at hudson.FilePath.access$2100(FilePath.java:196)
>>>> 10:46:04 at hudson.FilePath$45.invoke(FilePath.java:2216)
>>>> 10:46:04 at hudson.FilePath$45.invoke(FilePath.java:2212)
>>>> 10:46:04 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2760)
>>>> 10:46:04 at hudson.remoting.UserRequest.perform(UserRequest.java:207)
>>>> 10:46:04 at hudson.remoting.UserRequest.perform(UserRequest.java:53)
>>>> 10:46:04 at hudson.remoting.Request$2.run(Request.java:358)
>>>> 10:46:04 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
>>>> 10:46:04 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> 10:46:04 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>> 10:46:04 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>> 10:46:04 at java.lang.Thread.run(Thread.java:748)
>>>> 10:46:04 Caused: java.util.concurrent.ExecutionException
>>>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:946)
>>>> 10:46:04 at hudson.remoting.Channel$2.adapt(Channel.java:941)
>>>> 10:46:04 at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
>>>> 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2224)
>>>> 10:46:04 Caused: java.io.IOException: Failed to extract
>>>> /home/jenkins/workspace/ovirt-system-tests_manual/transfer of 8678
>>>> files
>>>
>>> I tend to guess this is the root cause. I wanted to collect all of /etc,
>>> perhaps that's too many files for jenkins.
>>>
>>> artifacts:
>>>     - /etc/*ovirt*
>>
>>
>> I don't think jenkins itself will have any issue with it (given that
>> we have stdci - its easy to check).
>
> Not sure what you mean - it did fail. Do you suggest some other test
> to check if the cause is "too many files" or something else?
>
> Searching google for "IOException: Failed to extract" does find
> several similar reports, didn't try to investigate further.
>

Blame me for replying without reading the whole thing...

Please retry that, lets see if it reproduces consistently...

>>
>>>
>>> Are there objections to a patch that will fix this to support wildcards?
>>> We have these on a typical engine machine, I'd rather not list all of them:
>>>
>>> ovirt-engine
>>> ovirt-engine-dwh
>>> ovirt-engine-metrics
>>> ovirt-engine-setup.conf.d
>>> ovirt-engine-setup.env.d
>>> ovirt-host-deploy.conf.d
>>> ovirt-imageio-proxy
>>> ovirt-provider-ovn
>>> ovirt-vmconsole
>>> ovirt-web-ui
>
> Currently did, for now, and it's working well:
>
> https://gerrit.ovirt.org/89712
>
>>>
>>>> 10:46:04 at hudson.FilePath.readFromTar(FilePath.java:2317)
>>>> 10:46:04 at hudson.FilePath.copyRecursiveTo(FilePath.java:2221)
>>>> 10:46:04 at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:61)
>>>> 10:46:04 at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:245)
>>>> 10:46:04 at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
>>>> 10:46:04 at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
>>>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
>>>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
>>>> 10:46:04 at hudson.model.Build$BuildExecution.post2(Build.java:186)
>>>> 10:46:04 at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
>>>> 10:46:04 at hudson.model.Run.execute(Run.java:1749)
>>>> 10:46:04 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
>>>> 10:46:04 at hudson.model.ResourceController.execute(ResourceController.java:97)
>>>> 10:46:04 at hudson.model.Executor.run(Executor.java:429)
>>>> 10:46:04 Build step 'Archive the artifacts' changed build result to FAILURE
>>>>
>>>> Any idea?
>>>>
>>>> Now running it again, let's see what happens.
>>>>
>>>> http://jenkins.ovirt.org/view/oVirt%20system%20tests/job/ovirt-system-tests_manual/2490/
>>>> --
>>>> Didi
>>>
>>>
>>>
>>> --
>>> Didi
>>> _______________________________________________
>>> Infra mailing list
>>> Infra at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>>
>> --
>> Barak Korren
>> RHV DevOps team , RHCE, RHCi
>> Red Hat EMEA
>> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
>
>
>
> --
> Didi



-- 
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

More information about the Infra mailing list