<div dir="ltr">Just had this failure again[1], relevant log[2].<div><br></div><div>I still feel there's an issue here:</div><div><div>{"jsonrpc": "2.0", "id": "aa241c36-d61d-4f9f-9baf-040b00638d00", "result": [{"username": "Unknown", "displayInfo": [{"tlsPort": "-1", "ipAddress": "192.168.201.4", "type": "vnc", "port": "5900"}, {"tlsPort": "5901", "ipAddress": "192.168.201.4", "type": "spice", "port": "-1"}], "hash": "964568653906073065", "acpiEnable": "true", "displayIp": "192.168.201.4", "guestFQDN": "", "vmId": "c9db9f94-3086-47e4-b997-01564ecb5319", "pid": "23127", "timeOffset": "0", "session": "Unknown", "displaySecurePort": "-1", "displayPort": "5900", "memUsage": "0", "guestIPs": "", "vcpuQuota": "-1", "kvmEnable": "true", "displayType": "vnc", "cpuUser": "0.00", "vmJobs": {}, "monitorResponse": "0", "elapsedTime": "477", "vmType": "kvm", "cpuSys": "0.00", "status": "Up", "guestCPUCount": -1, "appsList": [], "clientIp": "", "statusTime": "4295847140", "vmName": "vm0", "vcpuPeriod": 100000}]}</div><div>2017-03-16 10:36:17,558-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.Reactor] (SSL Stomp Reactor) [cb68faa] Internal server error 880 > 879: java.lang.IllegalArgumentException: 880 > 879</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at java.util.Arrays.copyOfRange(Arrays.java:3519) [rt.jar:1.8.0_121]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.vdsm.jsonrpc.client.reactors.stomp.StompCommonClient.processIncoming(StompCommonClient.java:145) [vdsm-jsonrpc-java-client.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.process(ReactorClient.java:195) [vdsm-jsonrpc-java-client.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.vdsm.jsonrpc.client.reactors.SSLClient.process(SSLClient.java:121) [vdsm-jsonrpc-java-client.jar:]</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>at org.ovirt.vdsm.jsonrpc.client.reactors.Reactor.lambda$processChannels$2(Reactor.java:80) [vdsm-jsonrpc-java-client.jar:]</div></div><div><br></div><div>Y.</div><div><br></div><div>[1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/327/">http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/327/</a></div><div>[2] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/327/artifact/exported-artifacts/basic_suite_4.1__logs/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log">http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/327/artifact/exported-artifacts/basic_suite_4.1__logs/test_logs/basic-suite-4.1/post-004_basic_sanity.py/lago-basic-suite-4-1-engine/_var_log/ovirt-engine/engine.log</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 3, 2017 at 9:53 AM, Pavel Zhukov <span dir="ltr"><<a href="mailto:pzhukov@redhat.com" target="_blank">pzhukov@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi,<br>
<br>
This is one-time failure. Non reproducible. disk_id was not inserted<br>
into DB because of jsonrpc/networking issue. But once OST runs on<br>
localhost networking (should not) be a problem here so adding Piotr.<br>
<br>
Failed jobs:<br>
<a href="http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_4.1/845/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/view/<wbr>experimental%20jobs/job/test-<wbr>repo_ovirt_experimental_4.1/<wbr>845/</a><br>
<br>
Logs:<br>
<a href="http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_4.1/845/artifact/exported-artifacts" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/view/<wbr>experimental%20jobs/job/test-<wbr>repo_ovirt_experimental_4.1/<wbr>845/artifact/exported-<wbr>artifacts</a><br>
<a href="http://jenkins.ovirt.org/view/experimental%20jobs/job/test-repo_ovirt_experimental_4.1/845/artifact/*zip*/archive.zip" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/view/<wbr>experimental%20jobs/job/test-<wbr>repo_ovirt_experimental_4.1/<wbr>845/artifact/*zip*/archive.zip</a><br>
<br>
[LOGS SNIPPETS]<br>
<br>
35897:2017-03-02 07:12:28,721-05 INFO [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.<wbr>CreateImageVDSCommand] (default task-19) [4adab878] START, CreateImageVDSCommand( CreateImageVDSCommandParameter<wbr>s:{runAsync='true', storagePoolId='9314c679-9a20-<wbr>48b4-b62a-f7b6a3703287', ignoreFailoverLimit='false', storageDomainId='49226db8-<wbr>e43f-4a15-a3a3-0bddd690d06b', imageGroupId='9cf5a90d-05fc-<wbr>45de-a3c1-adbb6822150e', imageSizeInBytes='2', volumeFormat='COW', newImageId='c5275e2f-ac80-<wbr>47c2-a43e-b5a0420a4211', newImageDescription='{"<wbr>DiskAlias":"vm0_disk0","<wbr>DiskDescription":""}', imageInitialSizeInBytes='0'}), log id: 657ef611<br>
35920:{"jsonrpc":"2.0","<wbr>method":"Volume.create","<wbr>params":{"volumeID":"c5275e2f-<wbr>ac80-47c2-a43e-b5a0420a4211","<wbr>storagepoolID":"9314c679-9a20-<wbr>48b4-b62a-f7b6a3703287","<wbr>storagedomainID":"49226db8-<wbr>e43f-4a15-a3a3-0bddd690d06b","<wbr>imageID":"9cf5a90d-05fc-45de-<wbr>a3c1-adbb6822150e","size":"2",<wbr>"volFormat":4,"preallocate":2,<wbr>"diskType":2,"desc":"{\"<wbr>DiskAlias\":\"vm0_disk0\",\"<wbr>DiskDescription\":\"\"}","<wbr>srcImgUUID":"00000000-0000-<wbr>0000-0000-000000000000","<wbr>srcVolUUID":"00000000-0000-<wbr>0000-0000-000000000000"},"id":<wbr>"0a757eae-0cc2-4f15-9bb5-<wbr>05ad3221f697"} <=== NOTE NULL character here<br>
35926:<JsonRpcRequest id: "0a757eae-0cc2-4f15-9bb5-<wbr>05ad3221f697", method: Volume.create, params: {volumeID=c5275e2f-ac80-47c2-<wbr>a43e-b5a0420a4211, storagepoolID=9314c679-9a20-<wbr>48b4-b62a-f7b6a3703287, storagedomainID=49226db8-e43f-<wbr>4a15-a3a3-0bddd690d06b, imageID=9cf5a90d-05fc-45de-<wbr>a3c1-adbb6822150e, size=2, volFormat=4, preallocate=2, diskType=2, desc={"DiskAlias":"vm0_disk0",<wbr>"DiskDescription":""}, srcImgUUID=00000000-0000-0000-<wbr>0000-000000000000, srcVolUUID=00000000-0000-0000-<wbr>0000-000000000000}><br>
35996:2017-03-02 07:12:30,468-05 ERROR [org.ovirt.engine.core.<wbr>vdsbroker.irsbroker.<wbr>IrsBrokerCommand] (default task-19) [4adab878] ERROR, CreateImageVDSCommand( CreateImageVDSCommandParameter<wbr>s:{runAsync='true', storagePoolId='9314c679-9a20-<wbr>48b4-b62a-f7b6a3703287', ignoreFailoverLimit='false', storageDomainId='49226db8-<wbr>e43f-4a15-a3a3-0bddd690d06b', imageGroupId='9cf5a90d-05fc-<wbr>45de-a3c1-adbb6822150e', imageSizeInBytes='2', volumeFormat='COW', newImageId='c5275e2f-ac80-<wbr>47c2-a43e-b5a0420a4211', newImageDescription='{"<wbr>DiskAlias":"vm0_disk0","<wbr>DiskDescription":""}', imageInitialSizeInBytes='0'}), exception: VDSGenericException: VDSNetworkException: 880 > 879, log id: 657ef611<br>
36511:2017-03-02 07:12:35,744-05 INFO [org.ovirt.engine.core.bll.<wbr>storage.disk.image.<wbr>AddImageFromScratchCommand] (default task-19) [4adab878] Command [id=35007fba-781c-4a15-8f29-<wbr>a8efe441b2f3]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.<wbr>businessentities.storage.<wbr>DiskImage; snapshot: 9cf5a90d-05fc-45de-a3c1-<wbr>adbb6822150e.<br>
2017-03-02 07:12:35,766-05 WARN [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector] (DefaultQuartzScheduler7) [6dc0e793] EVENT_ID: SYSTEM_CHANGE_STORAGE_POOL_<wbr>STATUS_PROBLEMATIC(980), Correlation ID: 6dc0e793, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center test-dc. Setting status to Non Responsive.<br>
2017-03-02 07:12:35,773-05 INFO [org.ovirt.engine.core.bll.<wbr>tasks.AsyncTaskManager] (default task-19) [4adab878] Removed task '10b3bd58-8f3e-4b23-a8b6-<wbr>27db110810d5' from DataBase<br>
2017-03-02 07:12:35,779-05 DEBUG [org.ovirt.engine.core.dal.<wbr>dbbroker.<wbr>CustomSQLErrorCodeSQLException<wbr>Translator] (default task-19) [4adab878] Translating SQLException with SQL state '23503', error code '0', message [ERROR: insert or update on table "disk_vm_element" violates foreign key constraint "fk_disk_vm_element_base_<wbr>disks"<br>
Detail: Key (disk_id)=(9cf5a90d-05fc-45de-<wbr>a3c1-adbb6822150e) is not present in table "base_disks".<br>
<br>
--<br>
Pavel<br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
</blockquote></div><br></div>