Dear Nir, None of my Hosts has any updated left to be added, they are as up to date as they can be. The imageio packages version installed is: Ovirt-imageio-common-1.5.1-0.el7.x86_64 Ovirt-imageio-daemon-1.5.1-0.el7.x86_64 Additional software info from Host OS Version: RHEL - 7 - 6.1810.2.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 957.21.3.el7.x86_64 KVM Version: 2.12.0 - 18.el7_6.5.1 LIBVIRT Version: libvirt-4.5.0-10.el7_6.12 VDSM Version: vdsm-4.30.17-1.el7 SPICE Version: 0.14.0 - 6.el7_6.1 GlusterFS Version: [N/A] CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.10.1-3.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Date: Thursday, 18 July 2019 at 08:03 To: Nir Soffer <nsoffer@redhat.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase Hi Nir, Sure, i will check. Is there a way for Adminstrator to view tickets or close them by force? Sent from my iPhone On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com<mailto:nsoffer@redhat.com>> wrote: On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Dear oVIrt, I initiated upload of qcow2 disk image for Centos 6.5: It reached finalizing phase and than started throwing following errors: 2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well. This was fixed in 4.2. Are you running the latest version on the hosts? Please update ovirt-imageio-daemon to latest version. Nir 2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module. Help! oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic ActiveVideo _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...

Hi Nir, Sure: here is the ovirt-engine/engine.log related to transaction: [root@ovirt-engine ovirt-engine]# zgrep "3452459d-aec6-430e-9509-1d9ca815b2d8" /var/log/ovirt-engine/engine.log-20190718.gz | head -n 40 2019-07-17 13:36:01,634Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (default task-376) [e935c4cd-c171-40fa-9204-db59f11ac013] START, CreateImageVDSCommand( CreateImageVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageSizeInBytes='42949672960', volumeFormat='COW', newImageId='b44659a9-607a-4eeb-a255-99532fd4fce4', imageType='Sparse', newImageDescription='{"DiskAlias":"av-07-centos-65-base","DiskDescription":""}', imageInitialSizeInBytes='0'}), log id: 41539302 2019-07-17 13:36:09,110Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [e935c4cd-c171-40fa-9204-db59f11ac013] START, GetImageInfoVDSCommand( GetImageInfoVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 3712c16a 2019-07-17 13:36:09,111Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVolumeInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [e935c4cd-c171-40fa-9204-db59f11ac013] START, GetVolumeInfoVDSCommand(HostName = ovirt-hv-03.avinity.tv, GetVolumeInfoVDSCommandParameters:{hostId='1faf7ede-6f81-41c7-8ba8-d146e6617422', storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 1b0d57f3 2019-07-17 13:36:09,420Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetQemuImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [e935c4cd-c171-40fa-9204-db59f11ac013] START, GetQemuImageInfoVDSCommand(HostName = ovirt-hv-01.avinity.tv, GetVolumeInfoVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 775faf53 2019-07-17 13:36:11,059Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[3452459d-aec6-430e-9509-1d9ca815b2d8=DISK]'}' 2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER 2019-07-17 13:36:11,142Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Successfully added Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' 2019-07-17 13:36:11,356Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, SetVolumeLegalityVDSCommand( SetVolumeLegalityVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 5e0f7654 2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 2019-07-17 13:36:11,503Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Paused by System 2019-07-17 13:36:11,563Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Finalizing Success 2019-07-17 13:36:11,610Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 13:36:11,612Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 13:36:13,658Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 13:36:13,659Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' Question regarding transfer UUID – is that one I marked green in the log lines above? Regarding Host that executed action, is in any way related to lines I marked in RED? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic From: Nir Soffer <nsoffer@redhat.com> Date: Thursday, 18 July 2019 at 12:20 To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Dear Nir, None of my Hosts has any updated left to be added, they are as up to date as they can be. The imageio packages version installed is: Ovirt-imageio-common-1.5.1-0.el7.x86_64 Ovirt-imageio-daemon-1.5.1-0.el7.x86_64 We need logs to understand the issue. Can you share logs from the time the upload was started? - engine log (/var/log/ovirt-engine/engine.log) - vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log) - daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log) To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs. Additional software info from Host OS Version: RHEL - 7 - 6.1810.2.el7.centos OS Description: CentOS Linux 7 (Core) Kernel Version: 3.10.0 - 957.21.3.el7.x86_64 KVM Version: 2.12.0 - 18.el7_6.5.1 LIBVIRT Version: libvirt-4.5.0-10.el7_6.12 VDSM Version: vdsm-4.30.17-1.el7 SPICE Version: 0.14.0 - 6.el7_6.1 GlusterFS Version: [N/A] CEPH Version: librbd1-10.2.5-4.el7 Open vSwitch Version: openvswitch-2.10.1-3.el7 Kernel Features: PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> Date: Thursday, 18 July 2019 at 08:03 To: Nir Soffer <nsoffer@redhat.com<mailto:nsoffer@redhat.com>> Cc: users <users@ovirt.org<mailto:users@ovirt.org>>, Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase Hi Nir, Sure, i will check. Is there a way for Adminstrator to view tickets or close them by force? Sent from my iPhone On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com<mailto:nsoffer@redhat.com>> wrote: On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Dear oVIrt, I initiated upload of qcow2 disk image for Centos 6.5: It reached finalizing phase and than started throwing following errors: 2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well. This was fixed in 4.2. Are you running the latest version on the hosts? Please update ovirt-imageio-daemon to latest version. Nir 2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' 2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') 2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module. Help! oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts. Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic ActiveVideo _______________________________________________ Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...

On Thu, Jul 18, 2019 at 1:44 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Hi Nir,

Sure: here is the ovirt-engine/engine.log related to transaction:

...

2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943

The transfer uuid is mentioned in this log: transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19' Please attach complete engine.log since this line. You can see that the host used for upload was ovirt-hv-01.avinity.tv. Please attach vdsm logs from this host from the time this ticket was added. Please attach imageio daemon logs from the same time on this host. — — —

Met vriendelijke groet / Kind regards,

*Marko Vrgotic*

*From: *Nir Soffer <nsoffer@redhat.com> *Date: *Thursday, 18 July 2019 at 12:20 *To: *"Vrgotic, Marko" <M.Vrgotic@activevideo.com> *Cc: *users <users@ovirt.org>, Daniel Erez <derez@redhat.com> *Subject: *Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Dear Nir,

None of my Hosts has any updated left to be added, they are as up to date as they can be.

The imageio packages version installed is:

Ovirt-imageio-common-1.5.1-0.el7.x86_64

Ovirt-imageio-daemon-1.5.1-0.el7.x86_64

We need logs to understand the issue.

Can you share logs from the time the upload was started?

- engine log (/var/log/ovirt-engine/engine.log)

- vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log)

- daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log)

To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs.

Additional software info from Host

OS Version:

RHEL - 7 - 6.1810.2.el7.centos

OS Description:

CentOS Linux 7 (Core)

Kernel Version:

3.10.0 - 957.21.3.el7.x86_64

KVM Version:

2.12.0 - 18.el7_6.5.1

LIBVIRT Version:

libvirt-4.5.0-10.el7_6.12

VDSM Version:

vdsm-4.30.17-1.el7

SPICE Version:

0.14.0 - 6.el7_6.1

GlusterFS Version:

[N/A]

CEPH Version:

librbd1-10.2.5-4.el7

Open vSwitch Version:

openvswitch-2.10.1-3.el7

Kernel Features:

PTI: 1, IBRS: 0, RETP: 1, SSBD: 3

*From: *"Vrgotic, Marko" <M.Vrgotic@activevideo.com> *Date: *Thursday, 18 July 2019 at 08:03 *To: *Nir Soffer <nsoffer@redhat.com> *Cc: *users <users@ovirt.org>, Daniel Erez <derez@redhat.com> *Subject: *Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

Hi Nir,

Sure, i will check.

Is there a way for Adminstrator to view tickets or close them by force?

Sent from my iPhone

On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com> wrote:

On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Dear oVIrt,

I initiated upload of qcow2 disk image for Centos 6.5:

It reached finalizing phase and than started throwing following errors:

2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well.

This was fixed in 4.2. Are you running the latest version on the hosts?

Please update ovirt-imageio-daemon to latest version.

Nir

2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module.

Help!

oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts.

Kindly awaiting your reply.

— — — Met vriendelijke groet / Kind regards,

*Marko Vrgotic*

ActiveVideo

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...

Hi Marko, According to the log, it seems there's an issue with a secured connection to the proxy. Can you please try the 'Test Connection' button on upload disk dialog, to ensure the certificate is properly installed? Also, is the disk currently in 'Paused by System' status? Thanks, Daniel On Thu, Jul 18, 2019 at 2:35 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Hi Nir,

Since this is already compressed file, I have added output of grep for moment this command was issue and following 300lines.

[root@ovirt-engine ovirt-engine]# zgrep "2019-07-17 13:36:11,121Z" engine.log-20190* -A300

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

gzip: -A300.gz: No such file or directory

[root@ovirt-engine ovirt-engine]# zgrep -A 300 "2019-07-17 13:36:11,121Z" engine.log-20190*

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

engine.log-20190718.gz:2019-07-17 13:36:11,142Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Successfully added Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

engine.log-20190718.gz:2019-07-17 13:36:11,188Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, PrepareImageVDSCommand(HostName = ovirt-hv-01.avinity.tv, PrepareImageVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d'}), log id: 3bd84b38

engine.log-20190718.gz:2019-07-17 13:36:11,351Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 3bd84b38

engine.log-20190718.gz:2019-07-17 13:36:11,356Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, SetVolumeLegalityVDSCommand( SetVolumeLegalityVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 5e0f7654

engine.log-20190718.gz:2019-07-17 13:36:11,390Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, SetVolumeLegalityVDSCommand, return: , log id: 5e0f7654

engine.log-20190718.gz:2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943

engine.log-20190718.gz:2019-07-17 13:36:11,410Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, AddImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 3bf6b943

engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Started transfer session with ticket id 10096b11-d10d-43aa-ad16-668b29a8c152, timeout 300 seconds

engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Adding image ticket to ovirt-imageio-proxy, id 10096b11-d10d-43aa-ad16-668b29a8c152

engine.log-20190718.gz:2019-07-17 13:36:11,469Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

engine.log-20190718.gz: at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.addImageTicketToProxy(TransferDiskImageCommand.java:988) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:911) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:554) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:393) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:388) [bll.jar:]

engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) [:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)

engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]

engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) [:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77)

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89)

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102)

engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)

engine.log-20190718.gz: at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]

engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source) [:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)

engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)

engine.log-20190718.gz: at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)

engine.log-20190718.gz: at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618)

engine.log-20190718.gz: at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)

engine.log-20190718.gz: at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)

engine.log-20190718.gz: at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)

engine.log-20190718.gz: at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)

engine.log-20190718.gz: at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) [common.jar:]

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:233)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:144)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:134)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:153)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:70)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForImage(BackendImageTransfersResource.java:39)

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:101)

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final]

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209)

engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223)

engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149)

engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98)

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72)

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102)

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)

engine.log-20190718.gz: at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)

engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

engine.log-20190718.gz: at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

engine.log-20190718.gz: at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

engine.log-20190718.gz: at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

engine.log-20190718.gz: at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)

engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)

engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)

engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

engine.log-20190718.gz: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)

engine.log-20190718.gz: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)

engine.log-20190718.gz: at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)

engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)

engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)

engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)

engine.log-20190718.gz: at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201]

engine.log-20190718.gz:Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.validator.Validator.validate(Validator.java:262) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: ... 224 more

engine.log-20190718.gz:Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) [rt.jar:1.8.0_201]

engine.log-20190718.gz: ... 230 more

engine.log-20190718.gz:

engine.log-20190718.gz:2019-07-17 13:36:11,495Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy

engine.log-20190718.gz:2019-07-17 13:36:11,502Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_ADD_TICKET_TO_PROXY(1,070), Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy.

engine.log-20190718.gz:2019-07-17 13:36:11,503Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Paused by System

engine.log-20190718.gz:2019-07-17 13:36:11,506Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Returning from proceedCommandExecution after starting transfer session for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

engine.log-20190718.gz:2019-07-17 13:36:11,520Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Upload with disk av-07-centos-65-base was initiated by mvrgotic@ictv.com@ictv.com-authz.

engine.log-20190718.gz:2019-07-17 13:36:11,562Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Running command: TransferImageStatusCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_DISK with role type USER

engine.log-20190718.gz:2019-07-17 13:36:11,563Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Finalizing Success

engine.log-20190718.gz:2019-07-17 13:36:11,610Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:11,612Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:11,613Z INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-378) [] User mvrgotic@ictv.com@ictv.com successfully logged out

engine.log-20190718.gz:2019-07-17 13:36:11,624Z INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-385) [2b577b1a] Running command: TerminateSessionsForTokenCommand internal: true.

engine.log-20190718.gz:2019-07-17 13:36:11,625Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-385) [2b577b1a] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'.

engine.log-20190718.gz:2019-07-17 13:36:13,658Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:13,659Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:17,703Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:17,703Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:25,748Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:25,748Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:35,802Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:35,802Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:45,821Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:45,821Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:53,434Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'.

engine.log-20190718.gz:2019-07-17 13:36:55,872Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:55,872Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:05,893Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:05,894Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:15,945Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:15,945Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:25,999Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:25,999Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:28,864Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access

engine.log-20190718.gz:2019-07-17 13:37:30,187Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access

engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Setting new tasks map. The map contains now 0 tasks

engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Cleared all tasks of pool '10525aa0-839d-11e9-a016-00163e4f2a6d'.

engine.log-20190718.gz:2019-07-17 13:37:35,547Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access

engine.log-20190718.gz:2019-07-17 13:37:35,599Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-376) [519046b4] Running command: CreateUserSessionCommand internal: false.

engine.log-20190718.gz:2019-07-17 13:37:35,614Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [519046b4] EVENT_ID: USER_VDC_LOGIN(30), User mvrgotic@ictv.com@ictv.com-authz connecting from '172.16.105.206' using session 'rhZQZDBy6kGp85btZDieKp2gQ5kQLWBcC7BOfXrU/fJHCVWyoyOvIwnwnaaD12f8YBK5HK3MOaQcYYptCMWRcA==' logged in.

engine.log-20190718.gz:2019-07-17 13:37:36,045Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:36,045Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:46,092Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:46,093Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:53,430Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'.

engine.log-20190718.gz:2019-07-17 13:37:56,109Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:56,110Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

From: Nir Soffer <nsoffer@redhat.com> Date: Thursday, 18 July 2019 at 13:14 To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

On Thu, Jul 18, 2019 at 1:44 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Hi Nir,

Sure: here is the ovirt-engine/engine.log related to transaction:

...

2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943

The transfer uuid is mentioned in this log:

transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

Please attach complete engine.log since this line.

You can see that the host used for upload was ovirt-hv-01.avinity.tv. Please attach vdsm logs

from this host from the time this ticket was added.

Please attach imageio daemon logs from the same time on this host.

— — — Met vriendelijke groet / Kind regards,

Marko Vrgotic

From: Nir Soffer <nsoffer@redhat.com> Date: Thursday, 18 July 2019 at 12:20 To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Dear Nir,

None of my Hosts has any updated left to be added, they are as up to date as they can be.

The imageio packages version installed is:

Ovirt-imageio-common-1.5.1-0.el7.x86_64

Ovirt-imageio-daemon-1.5.1-0.el7.x86_64

We need logs to understand the issue.

Can you share logs from the time the upload was started?

- engine log (/var/log/ovirt-engine/engine.log)

- vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log)

- daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log)

To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs.

Additional software info from Host

OS Version:

RHEL - 7 - 6.1810.2.el7.centos

OS Description:

CentOS Linux 7 (Core)

Kernel Version:

3.10.0 - 957.21.3.el7.x86_64

KVM Version:

2.12.0 - 18.el7_6.5.1

LIBVIRT Version:

libvirt-4.5.0-10.el7_6.12

VDSM Version:

vdsm-4.30.17-1.el7

SPICE Version:

0.14.0 - 6.el7_6.1

GlusterFS Version:

[N/A]

CEPH Version:

librbd1-10.2.5-4.el7

Open vSwitch Version:

openvswitch-2.10.1-3.el7

Kernel Features:

PTI: 1, IBRS: 0, RETP: 1, SSBD: 3

From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Date: Thursday, 18 July 2019 at 08:03 To: Nir Soffer <nsoffer@redhat.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

Hi Nir,

Sure, i will check.

Is there a way for Adminstrator to view tickets or close them by force?

Sent from my iPhone

On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com> wrote:

On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Dear oVIrt,

I initiated upload of qcow2 disk image for Centos 6.5:

It reached finalizing phase and than started throwing following errors:

2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well.

This was fixed in 4.2. Are you running the latest version on the hosts?

Please update ovirt-imageio-daemon to latest version.

Nir

2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module.

Help!

oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts.

Kindly awaiting your reply.

— — — Met vriendelijke groet / Kind regards,

Marko Vrgotic

ActiveVideo

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...

On Thu, Jul 18, 2019 at 7:31 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Hi Daniel,

Is it possible that upgrade from 4.3.3 to 4.3.4 affected/reset/changed the certificate for imageio?

Have you restarted the ovirt-imageio-proxy service after upgrading? It should be restarted to reload the certificate.

Reason why I am asking is that, while running 4.3.3 version, I was uploading Ubuntu 16-18, CentOS 7 cloud images, and CentOS6 no issues. Now issue is with CentOS6. I have tried to upload centos6.5 image via Ansible ovirt_disk and than one ended stuck in “Finalizing” state. Than I tried with to Upload same image from UI and that ended in “Paused” state.

I will try Test Connection and let you know.

Sent from my iPhone

...

On 18 Jul 2019, at 16:13, Daniel Erez <derez@redhat.com> wrote:

Hi Marko,

According to the log, it seems there's an issue with a secured connection to the proxy. Can you please try the 'Test Connection' button on upload disk dialog, to ensure the certificate is properly installed? Also, is the disk currently in 'Paused by System' status?

Thanks, Daniel

On Thu, Jul 18, 2019 at 2:35 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

...

Hi Nir,

Since this is already compressed file, I have added output of grep for moment this command was issue and following 300lines.

[root@ovirt-engine ovirt-engine]# zgrep "2019-07-17 13:36:11,121Z" engine.log-20190* -A300

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

gzip: -A300.gz: No such file or directory

[root@ovirt-engine ovirt-engine]# zgrep -A 300 "2019-07-17 13:36:11,121Z" engine.log-20190*

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER

engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

engine.log-20190718.gz:2019-07-17 13:36:11,142Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Successfully added Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

engine.log-20190718.gz:2019-07-17 13:36:11,188Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, PrepareImageVDSCommand(HostName = ovirt-hv-01.avinity.tv, PrepareImageVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d'}), log id: 3bd84b38

engine.log-20190718.gz:2019-07-17 13:36:11,351Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 3bd84b38

engine.log-20190718.gz:2019-07-17 13:36:11,356Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, SetVolumeLegalityVDSCommand( SetVolumeLegalityVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 5e0f7654

engine.log-20190718.gz:2019-07-17 13:36:11,390Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, SetVolumeLegalityVDSCommand, return: , log id: 5e0f7654

engine.log-20190718.gz:2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943

engine.log-20190718.gz:2019-07-17 13:36:11,410Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, AddImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 3bf6b943

engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Started transfer session with ticket id 10096b11-d10d-43aa-ad16-668b29a8c152, timeout 300 seconds

engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Adding image ticket to ovirt-imageio-proxy, id 10096b11-d10d-43aa-ad16-668b29a8c152

engine.log-20190718.gz:2019-07-17 13:36:11,469Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

engine.log-20190718.gz: at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.addImageTicketToProxy(TransferDiskImageCommand.java:988) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:911) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:554) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:393) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:]

engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:388) [bll.jar:]

engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) [:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)

engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]

engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) [:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77)

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89)

engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102)

engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)

engine.log-20190718.gz: at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]

engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source) [:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)

engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)

engine.log-20190718.gz: at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final]

engine.log-20190718.gz: at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final]

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)

engine.log-20190718.gz: at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618)

engine.log-20190718.gz: at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)

engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)

engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)

engine.log-20190718.gz: at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)

engine.log-20190718.gz: at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)

engine.log-20190718.gz: at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)

engine.log-20190718.gz: at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) [common.jar:]

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:233)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:144)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:134)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:153)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:70)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForImage(BackendImageTransfersResource.java:39)

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:101)

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final]

engine.log-20190718.gz: at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final]

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209)

engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223)

engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149)

engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98)

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72)

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111)

engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102)

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:]

engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)

engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)

engine.log-20190718.gz: at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)

engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)

engine.log-20190718.gz: at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)

engine.log-20190718.gz: at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

engine.log-20190718.gz: at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)

engine.log-20190718.gz: at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)

engine.log-20190718.gz: at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)

engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)

engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)

engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)

engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)

engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)

engine.log-20190718.gz: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)

engine.log-20190718.gz: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)

engine.log-20190718.gz: at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)

engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)

engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)

engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)

engine.log-20190718.gz: at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201]

engine.log-20190718.gz:Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.validator.Validator.validate(Validator.java:262) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) [jsse.jar:1.8.0_201]

engine.log-20190718.gz: ... 224 more

engine.log-20190718.gz:Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_201]

engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) [rt.jar:1.8.0_201]

engine.log-20190718.gz: ... 230 more

engine.log-20190718.gz:

engine.log-20190718.gz:2019-07-17 13:36:11,495Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy

engine.log-20190718.gz:2019-07-17 13:36:11,502Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_ADD_TICKET_TO_PROXY(1,070), Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy.

engine.log-20190718.gz:2019-07-17 13:36:11,503Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Paused by System

engine.log-20190718.gz:2019-07-17 13:36:11,506Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Returning from proceedCommandExecution after starting transfer session for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

engine.log-20190718.gz:2019-07-17 13:36:11,520Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Upload with disk av-07-centos-65-base was initiated by mvrgotic@ictv.com@ictv.com-authz.

engine.log-20190718.gz:2019-07-17 13:36:11,562Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Running command: TransferImageStatusCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_DISK with role type USER

engine.log-20190718.gz:2019-07-17 13:36:11,563Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Finalizing Success

engine.log-20190718.gz:2019-07-17 13:36:11,610Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:11,612Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:11,613Z INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-378) [] User mvrgotic@ictv.com@ictv.com successfully logged out

engine.log-20190718.gz:2019-07-17 13:36:11,624Z INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-385) [2b577b1a] Running command: TerminateSessionsForTokenCommand internal: true.

engine.log-20190718.gz:2019-07-17 13:36:11,625Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-385) [2b577b1a] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'.

engine.log-20190718.gz:2019-07-17 13:36:13,658Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:13,659Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:17,703Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:17,703Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:25,748Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:25,748Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:35,802Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:35,802Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:45,821Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:45,821Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:36:53,434Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'.

engine.log-20190718.gz:2019-07-17 13:36:55,872Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:36:55,872Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:05,893Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:05,894Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:15,945Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:15,945Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:25,999Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:25,999Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:28,864Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access

engine.log-20190718.gz:2019-07-17 13:37:30,187Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access

engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Setting new tasks map. The map contains now 0 tasks

engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Cleared all tasks of pool '10525aa0-839d-11e9-a016-00163e4f2a6d'.

engine.log-20190718.gz:2019-07-17 13:37:35,547Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access

engine.log-20190718.gz:2019-07-17 13:37:35,599Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-376) [519046b4] Running command: CreateUserSessionCommand internal: false.

engine.log-20190718.gz:2019-07-17 13:37:35,614Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [519046b4] EVENT_ID: USER_VDC_LOGIN(30), User mvrgotic@ictv.com@ictv.com-authz connecting from '172.16.105.206' using session 'rhZQZDBy6kGp85btZDieKp2gQ5kQLWBcC7BOfXrU/fJHCVWyoyOvIwnwnaaD12f8YBK5HK3MOaQcYYptCMWRcA==' logged in.

engine.log-20190718.gz:2019-07-17 13:37:36,045Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:36,045Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:46,092Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:46,093Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

engine.log-20190718.gz:2019-07-17 13:37:53,430Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'.

engine.log-20190718.gz:2019-07-17 13:37:56,109Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

engine.log-20190718.gz:2019-07-17 13:37:56,110Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

From: Nir Soffer <nsoffer@redhat.com> Date: Thursday, 18 July 2019 at 13:14 To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

On Thu, Jul 18, 2019 at 1:44 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Hi Nir,

Sure: here is the ovirt-engine/engine.log related to transaction:

...

2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943

The transfer uuid is mentioned in this log:

transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'

Please attach complete engine.log since this line.

You can see that the host used for upload was ovirt-hv-01.avinity.tv. Please attach vdsm logs

from this host from the time this ticket was added.

Please attach imageio daemon logs from the same time on this host.

— — — Met vriendelijke groet / Kind regards,

Marko Vrgotic

From: Nir Soffer <nsoffer@redhat.com> Date: Thursday, 18 July 2019 at 12:20 To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Dear Nir,

None of my Hosts has any updated left to be added, they are as up to date as they can be.

The imageio packages version installed is:

Ovirt-imageio-common-1.5.1-0.el7.x86_64

Ovirt-imageio-daemon-1.5.1-0.el7.x86_64

We need logs to understand the issue.

Can you share logs from the time the upload was started?

- engine log (/var/log/ovirt-engine/engine.log)

- vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log)

- daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log)

To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs.

Additional software info from Host

OS Version:

RHEL - 7 - 6.1810.2.el7.centos

OS Description:

CentOS Linux 7 (Core)

Kernel Version:

3.10.0 - 957.21.3.el7.x86_64

KVM Version:

2.12.0 - 18.el7_6.5.1

LIBVIRT Version:

libvirt-4.5.0-10.el7_6.12

VDSM Version:

vdsm-4.30.17-1.el7

SPICE Version:

0.14.0 - 6.el7_6.1

GlusterFS Version:

[N/A]

CEPH Version:

librbd1-10.2.5-4.el7

Open vSwitch Version:

openvswitch-2.10.1-3.el7

Kernel Features:

PTI: 1, IBRS: 0, RETP: 1, SSBD: 3

From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Date: Thursday, 18 July 2019 at 08:03 To: Nir Soffer <nsoffer@redhat.com> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase

Hi Nir,

Sure, i will check.

Is there a way for Adminstrator to view tickets or close them by force?

Sent from my iPhone

On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com> wrote:

On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote:

Dear oVIrt,

I initiated upload of qcow2 disk image for Centos 6.5:

It reached finalizing phase and than started throwing following errors:

2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well.

This was fixed in 4.2. Are you running the latest version on the hosts?

Please update ovirt-imageio-daemon to latest version.

Nir

2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4')

2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8'

I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module.

Help!

oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts.

Kindly awaiting your reply.

— — — Met vriendelijke groet / Kind regards,

Marko Vrgotic

ActiveVideo

_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...

Dear Daniel, I apologize up front for long email, but I just want to make sure all relevant info is provided. I have executed following actions: * Put the system into Global Maintenance * Executed "yum update" * Restarted SHE server * Once it was Up and all services Up, removed Global Maintenance mode * From UI attempted "Test Connection" several times from different Host: [cid:image001.png@01D53E1C.EB0CAAE0] Attempted two Uploads, one at a time from UI, of same centos-65.qcow2 image. * First time with alias testimport * Second time with alias av-07-centos-65-base Both Uploads ended in Paused state, which I was able to cancel and the task was removed (full engine logs of these two tasks are attached). Seem that in both case there is an issue with Certificate, even though imageio daemon was restarted (I rebooted server in steps above). Question1: Considering I did not see this issue prio to update from 4.3.3 to 4.3.4 it still makes me wonder if something happened with certificate or certificate path? I will also check in my backups Question2: [cid:image003.png@01D53E1C.EB0CAAE0] The image of 40GB with UUID of 3452459d-aec6-430e-9509-1d9ca815b2d8 pointing to -> /rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8 is still stuck in finalizing phase. This is the image that I reported issue about and the upload was executed using ovirt_disk ansible module. The files were removed afterwards manually by me (which probably was not good idea), as an attempt to cause an exception or to see if I can trigger iodaemon to break the task due to missing file. How can I remote this task or stop it? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic On 19/07/2019, 07:12, "Vrgotic, Marko" <M.Vrgotic@activevideo.com> wrote: Very good question, Now that I think of it, I might have missed this step. Most of the time/always after upgrade of oVirt packages is done, I update the rest of the system, both engine and hosts and reboot if required. I do not think I did and system update of the engine this time, so for sure did not reboot it. Will do that and report back. Thank you Daniel. Marko Vrgotic Sent from my iPhone > On 18 Jul 2019, at 18:33, Daniel Erez <derez@redhat.com> wrote: > > On Thu, Jul 18, 2019 at 7:31 PM Vrgotic, Marko > <M.Vrgotic@activevideo.com> wrote: >> >> Hi Daniel, >> >> Is it possible that upgrade from 4.3.3 to 4.3.4 affected/reset/changed the certificate for imageio? > > Have you restarted the ovirt-imageio-proxy service after upgrading? It > should be restarted to reload the certificate. > >> >> Reason why I am asking is that, while running 4.3.3 version, I was uploading Ubuntu 16-18, CentOS 7 cloud images, and CentOS6 no issues. >> Now issue is with CentOS6. >> I have tried to upload centos6.5 image via Ansible ovirt_disk and than one ended stuck in “Finalizing” state. >> Than I tried with to Upload same image from UI and that ended in “Paused” state. >> >> I will try Test Connection and let you know. >> >> Sent from my iPhone >> >>> On 18 Jul 2019, at 16:13, Daniel Erez <derez@redhat.com> wrote: >>> >>> Hi Marko, >>> >>> According to the log, it seems there's an issue with a secured >>> connection to the proxy. >>> Can you please try the 'Test Connection' button on upload disk dialog, >>> to ensure the certificate is properly installed? >>> Also, is the disk currently in 'Paused by System' status? >>> >>> Thanks, >>> Daniel >>> >>> On Thu, Jul 18, 2019 at 2:35 PM Vrgotic, Marko >>> <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Since this is already compressed file, I have added output of grep for moment this command was issue and following 300lines. >>>> >>>> >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep "2019-07-17 13:36:11,121Z" engine.log-20190* -A300 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> gzip: -A300.gz: No such file or directory >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep -A 300 "2019-07-17 13:36:11,121Z" engine.log-20190* >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,142Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Successfully added Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,188Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, PrepareImageVDSCommand(HostName = ovirt-hv-01.avinity.tv, PrepareImageVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d'}), log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,351Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,356Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, SetVolumeLegalityVDSCommand( SetVolumeLegalityVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,390Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, SetVolumeLegalityVDSCommand, return: , log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,410Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, AddImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Started transfer session with ticket id 10096b11-d10d-43aa-ad16-668b29a8c152, timeout 300 seconds >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Adding image ticket to ovirt-imageio-proxy, id 10096b11-d10d-43aa-ad16-668b29a8c152 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,469Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.addImageTicketToProxy(TransferDiskImageCommand.java:988) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:911) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:554) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:393) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:388) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) >>>> >>>> engine.log-20190718.gz: at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) [common.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:233) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:144) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:134) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:153) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:70) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForImage(BackendImageTransfersResource.java:39) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:101) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) >>>> >>>> engine.log-20190718.gz: at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.Validator.validate(Validator.java:262) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 224 more >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 230 more >>>> >>>> engine.log-20190718.gz: >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,495Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,502Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_ADD_TICKET_TO_PROXY(1,070), Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,503Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Paused by System >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,506Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Returning from proceedCommandExecution after starting transfer session for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,520Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Upload with disk av-07-centos-65-base was initiated by mvrgotic@ictv.com@ictv.com-authz. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,562Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Running command: TransferImageStatusCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_DISK with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,563Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Finalizing Success >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,610Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,612Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,613Z INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-378) [] User mvrgotic@ictv.com@ictv.com successfully logged out >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,624Z INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-385) [2b577b1a] Running command: TerminateSessionsForTokenCommand internal: true. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,625Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-385) [2b577b1a] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,658Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,659Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:53,434Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,893Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,894Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:28,864Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:30,187Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Setting new tasks map. The map contains now 0 tasks >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Cleared all tasks of pool '10525aa0-839d-11e9-a016-00163e4f2a6d'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,547Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,599Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-376) [519046b4] Running command: CreateUserSessionCommand internal: false. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,614Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [519046b4] EVENT_ID: USER_VDC_LOGIN(30), User mvrgotic@ictv.com@ictv.com-authz connecting from '172.16.105.206' using session 'rhZQZDBy6kGp85btZDieKp2gQ5kQLWBcC7BOfXrU/fJHCVWyoyOvIwnwnaaD12f8YBK5HK3MOaQcYYptCMWRcA==' logged in. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,092Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,093Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:53,430Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,109Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,110Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer@redhat.com> >>>> Date: Thursday, 18 July 2019 at 13:14 >>>> To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> >>>> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 1:44 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure: here is the ovirt-engine/engine.log related to transaction: >>>> >>>> ... >>>> >>>> 2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> >>>> >>>> The transfer uuid is mentioned in this log: >>>> >>>> >>>> >>>> transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> >>>> >>>> Please attach complete engine.log since this line. >>>> >>>> >>>> >>>> You can see that the host used for upload was ovirt-hv-01.avinity.tv. Please attach vdsm logs >>>> >>>> from this host from the time this ticket was added. >>>> >>>> >>>> >>>> Please attach imageio daemon logs from the same time on this host. >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer@redhat.com> >>>> Date: Thursday, 18 July 2019 at 12:20 >>>> To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> >>>> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Dear Nir, >>>> >>>> >>>> >>>> None of my Hosts has any updated left to be added, they are as up to date as they can be. >>>> >>>> The imageio packages version installed is: >>>> >>>> >>>> >>>> Ovirt-imageio-common-1.5.1-0.el7.x86_64 >>>> >>>> Ovirt-imageio-daemon-1.5.1-0.el7.x86_64 >>>> >>>> >>>> >>>> We need logs to understand the issue. >>>> >>>> >>>> >>>> Can you share logs from the time the upload was started? >>>> >>>> - engine log (/var/log/ovirt-engine/engine.log) >>>> >>>> - vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log) >>>> >>>> - daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log) >>>> >>>> >>>> >>>> To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs. >>>> >>>> >>>> >>>> >>>> >>>> Additional software info from Host >>>> >>>> OS Version: >>>> >>>> RHEL - 7 - 6.1810.2.el7.centos >>>> >>>> OS Description: >>>> >>>> CentOS Linux 7 (Core) >>>> >>>> Kernel Version: >>>> >>>> 3.10.0 - 957.21.3.el7.x86_64 >>>> >>>> KVM Version: >>>> >>>> 2.12.0 - 18.el7_6.5.1 >>>> >>>> LIBVIRT Version: >>>> >>>> libvirt-4.5.0-10.el7_6.12 >>>> >>>> VDSM Version: >>>> >>>> vdsm-4.30.17-1.el7 >>>> >>>> SPICE Version: >>>> >>>> 0.14.0 - 6.el7_6.1 >>>> >>>> GlusterFS Version: >>>> >>>> [N/A] >>>> >>>> CEPH Version: >>>> >>>> librbd1-10.2.5-4.el7 >>>> >>>> Open vSwitch Version: >>>> >>>> openvswitch-2.10.1-3.el7 >>>> >>>> Kernel Features: >>>> >>>> PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 >>>> >>>> >>>> >>>> From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> >>>> Date: Thursday, 18 July 2019 at 08:03 >>>> To: Nir Soffer <nsoffer@redhat.com> >>>> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure, i will check. >>>> >>>> >>>> >>>> Is there a way for Adminstrator to view tickets or close them by force? >>>> >>>> Sent from my iPhone >>>> >>>> >>>> On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com> wrote: >>>> >>>> >>>> >>>> On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Dear oVIrt, >>>> >>>> >>>> >>>> I initiated upload of qcow2 disk image for Centos 6.5: >>>> >>>> It reached finalizing phase and than started throwing following errors: >>>> >>>> >>>> >>>> 2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well. >>>> >>>> >>>> >>>> This was fixed in 4.2. Are you running the latest version on the hosts? >>>> >>>> >>>> >>>> Please update ovirt-imageio-daemon to latest version. >>>> >>>> >>>> >>>> Nir >>>> >>>> >>>> >>>> 2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module. >>>> >>>> >>>> >>>> Help! >>>> >>>> >>>> >>>> oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts. >>>> >>>> >>>> >>>> Kindly awaiting your reply. >>>> >>>> >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> ActiveVideo >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users@ovirt.org >>>> To unsubscribe send an email to users-leave@ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...

Hi Daniel, Question 1 is answered. Update overwrote the ovirt-imageio-proxy.conf to default, resulting in SSL Handshake mismatch, which is why Uploads were ending in Paused state. We will add this to standard Upgrade procedure, minor/major releases. I appreciate the help, it allows us to learn and get better. Question 2 remains open: [cid:image001.png@01D53E31.523AEA10] The image of 40GB with UUID of 3452459d-aec6-430e-9509-1d9ca815b2d8 pointing to -> /rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8 is still stuck in finalizing phase. This is the image that I reported issue about and the upload was executed using ovirt_disk ansible module. The files were removed afterwards manually by me (which probably was not good idea), as an attempt to cause an exception or to see if I can trigger iodaemon to break the task due to missing file. How can I remove/stop/kill this task? — — — Met vriendelijke groet / Kind regards, Marko Vrgotic From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Date: Friday, 19 July 2019 at 11:21 To: Daniel Erez <derez@redhat.com> Cc: "nsoffer@redhat.com" <nsoffer@redhat.com>, "users@ovirt.org" <users@ovirt.org>, "Stojchev, Darko" <D.Stojchev@activevideo.com> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase +my colleague Darko, Welcome to oVirt community — — — Met vriendelijke groet / Kind regards, Marko Vrgotic From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> Date: Friday, 19 July 2019 at 10:30 To: Daniel Erez <derez@redhat.com> Cc: "nsoffer@redhat.com" <nsoffer@redhat.com>, "users@ovirt.org" <users@ovirt.org> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase Dear Daniel, I apologize up front for long email, but I just want to make sure all relevant info is provided. I have executed following actions: * Put the system into Global Maintenance * Executed "yum update" * Restarted SHE server * Once it was Up and all services Up, removed Global Maintenance mode * From UI attempted "Test Connection" several times from different Host: [cid:image002.png@01D53E31.523AEA10] Attempted two Uploads, one at a time from UI, of same centos-65.qcow2 image. * First time with alias testimport * Second time with alias av-07-centos-65-base Both Uploads ended in Paused state, which I was able to cancel and the task was removed (full engine logs of these two tasks are attached). Seem that in both case there is an issue with Certificate, even though imageio daemon was restarted (I rebooted server in steps above). Question1: Considering I did not see this issue prio to update from 4.3.3 to 4.3.4 it still makes me wonder if something happened with certificate or certificate path? I will also check in my backups Question2: [cid:image003.png@01D53E31.523AEA10] The image of 40GB with UUID of 3452459d-aec6-430e-9509-1d9ca815b2d8 pointing to -> /rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8 is still stuck in finalizing phase. This is the image that I reported issue about and the upload was executed using ovirt_disk ansible module. The files were removed afterwards manually by me (which probably was not good idea), as an attempt to cause an exception or to see if I can trigger iodaemon to break the task due to missing file. How can I remote this task or stop it? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic On 19/07/2019, 07:12, "Vrgotic, Marko" <M.Vrgotic@activevideo.com> wrote: Very good question, Now that I think of it, I might have missed this step. Most of the time/always after upgrade of oVirt packages is done, I update the rest of the system, both engine and hosts and reboot if required. I do not think I did and system update of the engine this time, so for sure did not reboot it. Will do that and report back. Thank you Daniel. Marko Vrgotic Sent from my iPhone > On 18 Jul 2019, at 18:33, Daniel Erez <derez@redhat.com> wrote: > > On Thu, Jul 18, 2019 at 7:31 PM Vrgotic, Marko > <M.Vrgotic@activevideo.com> wrote: >> >> Hi Daniel, >> >> Is it possible that upgrade from 4.3.3 to 4.3.4 affected/reset/changed the certificate for imageio? > > Have you restarted the ovirt-imageio-proxy service after upgrading? It > should be restarted to reload the certificate. > >> >> Reason why I am asking is that, while running 4.3.3 version, I was uploading Ubuntu 16-18, CentOS 7 cloud images, and CentOS6 no issues. >> Now issue is with CentOS6. >> I have tried to upload centos6.5 image via Ansible ovirt_disk and than one ended stuck in “Finalizing” state. >> Than I tried with to Upload same image from UI and that ended in “Paused” state. >> >> I will try Test Connection and let you know. >> >> Sent from my iPhone >> >>> On 18 Jul 2019, at 16:13, Daniel Erez <derez@redhat.com> wrote: >>> >>> Hi Marko, >>> >>> According to the log, it seems there's an issue with a secured >>> connection to the proxy. >>> Can you please try the 'Test Connection' button on upload disk dialog, >>> to ensure the certificate is properly installed? >>> Also, is the disk currently in 'Paused by System' status? >>> >>> Thanks, >>> Daniel >>> >>> On Thu, Jul 18, 2019 at 2:35 PM Vrgotic, Marko >>> <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Since this is already compressed file, I have added output of grep for moment this command was issue and following 300lines. >>>> >>>> >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep "2019-07-17 13:36:11,121Z" engine.log-20190* -A300 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> gzip: -A300.gz: No such file or directory >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep -A 300 "2019-07-17 13:36:11,121Z" engine.log-20190* >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,142Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Successfully added Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,188Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, PrepareImageVDSCommand(HostName = ovirt-hv-01.avinity.tv, PrepareImageVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d'}), log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,351Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,356Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, SetVolumeLegalityVDSCommand( SetVolumeLegalityVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,390Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, SetVolumeLegalityVDSCommand, return: , log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,410Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, AddImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Started transfer session with ticket id 10096b11-d10d-43aa-ad16-668b29a8c152, timeout 300 seconds >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Adding image ticket to ovirt-imageio-proxy, id 10096b11-d10d-43aa-ad16-668b29a8c152 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,469Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.addImageTicketToProxy(TransferDiskImageCommand.java:988) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:911) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:554) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:393) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:388) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) >>>> >>>> engine.log-20190718.gz: at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) [common.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:233) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:144) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:134) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:153) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:70) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForImage(BackendImageTransfersResource.java:39) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:101) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) >>>> >>>> engine.log-20190718.gz: at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.Validator.validate(Validator.java:262) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 224 more >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 230 more >>>> >>>> engine.log-20190718.gz: >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,495Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,502Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_ADD_TICKET_TO_PROXY(1,070), Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,503Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Paused by System >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,506Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Returning from proceedCommandExecution after starting transfer session for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,520Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Upload with disk av-07-centos-65-base was initiated by mvrgotic@ictv.com@ictv.com-authz. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,562Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Running command: TransferImageStatusCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_DISK with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,563Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Finalizing Success >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,610Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,612Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,613Z INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-378) [] User mvrgotic@ictv.com@ictv.com successfully logged out >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,624Z INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-385) [2b577b1a] Running command: TerminateSessionsForTokenCommand internal: true. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,625Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-385) [2b577b1a] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,658Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,659Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:53,434Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,893Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,894Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:28,864Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:30,187Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Setting new tasks map. The map contains now 0 tasks >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Cleared all tasks of pool '10525aa0-839d-11e9-a016-00163e4f2a6d'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,547Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,599Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-376) [519046b4] Running command: CreateUserSessionCommand internal: false. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,614Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [519046b4] EVENT_ID: USER_VDC_LOGIN(30), User mvrgotic@ictv.com@ictv.com-authz connecting from '172.16.105.206' using session 'rhZQZDBy6kGp85btZDieKp2gQ5kQLWBcC7BOfXrU/fJHCVWyoyOvIwnwnaaD12f8YBK5HK3MOaQcYYptCMWRcA==' logged in. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,092Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,093Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:53,430Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,109Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,110Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer@redhat.com> >>>> Date: Thursday, 18 July 2019 at 13:14 >>>> To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> >>>> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 1:44 PM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure: here is the ovirt-engine/engine.log related to transaction: >>>> >>>> ... >>>> >>>> 2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> >>>> >>>> The transfer uuid is mentioned in this log: >>>> >>>> >>>> >>>> transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> >>>> >>>> Please attach complete engine.log since this line. >>>> >>>> >>>> >>>> You can see that the host used for upload was ovirt-hv-01.avinity.tv. Please attach vdsm logs >>>> >>>> from this host from the time this ticket was added. >>>> >>>> >>>> >>>> Please attach imageio daemon logs from the same time on this host. >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer@redhat.com> >>>> Date: Thursday, 18 July 2019 at 12:20 >>>> To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> >>>> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Dear Nir, >>>> >>>> >>>> >>>> None of my Hosts has any updated left to be added, they are as up to date as they can be. >>>> >>>> The imageio packages version installed is: >>>> >>>> >>>> >>>> Ovirt-imageio-common-1.5.1-0.el7.x86_64 >>>> >>>> Ovirt-imageio-daemon-1.5.1-0.el7.x86_64 >>>> >>>> >>>> >>>> We need logs to understand the issue. >>>> >>>> >>>> >>>> Can you share logs from the time the upload was started? >>>> >>>> - engine log (/var/log/ovirt-engine/engine.log) >>>> >>>> - vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log) >>>> >>>> - daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log) >>>> >>>> >>>> >>>> To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs. >>>> >>>> >>>> >>>> >>>> >>>> Additional software info from Host >>>> >>>> OS Version: >>>> >>>> RHEL - 7 - 6.1810.2.el7.centos >>>> >>>> OS Description: >>>> >>>> CentOS Linux 7 (Core) >>>> >>>> Kernel Version: >>>> >>>> 3.10.0 - 957.21.3.el7.x86_64 >>>> >>>> KVM Version: >>>> >>>> 2.12.0 - 18.el7_6.5.1 >>>> >>>> LIBVIRT Version: >>>> >>>> libvirt-4.5.0-10.el7_6.12 >>>> >>>> VDSM Version: >>>> >>>> vdsm-4.30.17-1.el7 >>>> >>>> SPICE Version: >>>> >>>> 0.14.0 - 6.el7_6.1 >>>> >>>> GlusterFS Version: >>>> >>>> [N/A] >>>> >>>> CEPH Version: >>>> >>>> librbd1-10.2.5-4.el7 >>>> >>>> Open vSwitch Version: >>>> >>>> openvswitch-2.10.1-3.el7 >>>> >>>> Kernel Features: >>>> >>>> PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 >>>> >>>> >>>> >>>> From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com> >>>> Date: Thursday, 18 July 2019 at 08:03 >>>> To: Nir Soffer <nsoffer@redhat.com> >>>> Cc: users <users@ovirt.org>, Daniel Erez <derez@redhat.com> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure, i will check. >>>> >>>> >>>> >>>> Is there a way for Adminstrator to view tickets or close them by force? >>>> >>>> Sent from my iPhone >>>> >>>> >>>> On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com> wrote: >>>> >>>> >>>> >>>> On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com> wrote: >>>> >>>> Dear oVIrt, >>>> >>>> >>>> >>>> I initiated upload of qcow2 disk image for Centos 6.5: >>>> >>>> It reached finalizing phase and than started throwing following errors: >>>> >>>> >>>> >>>> 2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well. >>>> >>>> >>>> >>>> This was fixed in 4.2. Are you running the latest version on the hosts? >>>> >>>> >>>> >>>> Please update ovirt-imageio-daemon to latest version. >>>> >>>> >>>> >>>> Nir >>>> >>>> >>>> >>>> 2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module. >>>> >>>> >>>> >>>> Help! >>>> >>>> >>>> >>>> oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts. >>>> >>>> >>>> >>>> Kindly awaiting your reply. >>>> >>>> >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> ActiveVideo >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users@ovirt.org >>>> To unsubscribe send an email to users-leave@ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...

Hi Daniel, This morning I tried to find the image stuck in Finalizing state, but state was completed. I guess it decided to have a weekend too. Actually it was there as an entry but there was no actual image. I was able to remove it and we have again clean situation. Thank a lot for all the help for these issues, I learned a lot. Our developers are moving from OpenStack onto oVirt platform and I am very proud how much was done to get here. Marko Vrgotic Sent from my iPhone On 21 Jul 2019, at 10:28, Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> wrote: On Fri, Jul 19, 2019 at 1:56 PM Vrgotic, Marko <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Hi Daniel, Question 1 is answered. Update overwrote the ovirt-imageio-proxy.conf to default, resulting in SSL Handshake mismatch, which is why Uploads were ending in Paused state. We will add this to standard Upgrade procedure, minor/major releases. I appreciate the help, it allows us to learn and get better. If you're using 3rd party certificates for imageio, it should be supported in next imageio release. I.e. you should be able to upgrade without encountering this issue again (see: https://gerrit.ovirt.org/#/c/97507/) Question 2 remains open: <image001.png> The image of 40GB with UUID of 3452459d-aec6-430e-9509-1d9ca815b2d8 pointing to -> /rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8 is still stuck in finalizing phase. This is the image that I reported issue about and the upload was executed using ovirt_disk ansible module. The files were removed afterwards manually by me (which probably was not good idea), as an attempt to cause an exception or to see if I can trigger iodaemon to break the task due to missing file. How can I remove/stop/kill this task? Try the following procedure: 1. Find the command_id $ psql -d engine -U engine engine=> select command_id from image_transfers where disk_id='3452459d-aec6-430e-9509-1d9ca815b2d8'; 2. Use taskcleaner.sh: https://www.ovirt.org/develop/developer-guide/db-issues/helperutilities.html... e.g. ./share/ovirt-engine/setup/dbutils/taskcleaner.sh -c {command_id} 3. Remove the disk (from the webadmin/api). — — — Met vriendelijke groet / Kind regards, Marko Vrgotic From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> Date: Friday, 19 July 2019 at 11:21 To: Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> Cc: "nsoffer@redhat.com<mailto:nsoffer@redhat.com>" <nsoffer@redhat.com<mailto:nsoffer@redhat.com>>, "users@ovirt.org<mailto:users@ovirt.org>" <users@ovirt.org<mailto:users@ovirt.org>>, "Stojchev, Darko" <D.Stojchev@activevideo.com<mailto:D.Stojchev@activevideo.com>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase +my colleague Darko, Welcome to oVirt community — — — Met vriendelijke groet / Kind regards, Marko Vrgotic From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> Date: Friday, 19 July 2019 at 10:30 To: Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> Cc: "nsoffer@redhat.com<mailto:nsoffer@redhat.com>" <nsoffer@redhat.com<mailto:nsoffer@redhat.com>>, "users@ovirt.org<mailto:users@ovirt.org>" <users@ovirt.org<mailto:users@ovirt.org>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase Dear Daniel, I apologize up front for long email, but I just want to make sure all relevant info is provided. I have executed following actions: * Put the system into Global Maintenance * Executed "yum update" * Restarted SHE server * Once it was Up and all services Up, removed Global Maintenance mode * From UI attempted "Test Connection" several times from different Host: <image002.png> Attempted two Uploads, one at a time from UI, of same centos-65.qcow2 image. * First time with alias testimport * Second time with alias av-07-centos-65-base Both Uploads ended in Paused state, which I was able to cancel and the task was removed (full engine logs of these two tasks are attached). Seem that in both case there is an issue with Certificate, even though imageio daemon was restarted (I rebooted server in steps above). Question1: Considering I did not see this issue prio to update from 4.3.3 to 4.3.4 it still makes me wonder if something happened with certificate or certificate path? I will also check in my backups Question2: <image003.png> The image of 40GB with UUID of 3452459d-aec6-430e-9509-1d9ca815b2d8 pointing to -> /rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8 is still stuck in finalizing phase. This is the image that I reported issue about and the upload was executed using ovirt_disk ansible module. The files were removed afterwards manually by me (which probably was not good idea), as an attempt to cause an exception or to see if I can trigger iodaemon to break the task due to missing file. How can I remote this task or stop it? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, Marko Vrgotic On 19/07/2019, 07:12, "Vrgotic, Marko" <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: Very good question, Now that I think of it, I might have missed this step. Most of the time/always after upgrade of oVirt packages is done, I update the rest of the system, both engine and hosts and reboot if required. I do not think I did and system update of the engine this time, so for sure did not reboot it. Will do that and report back. Thank you Daniel. Marko Vrgotic Sent from my iPhone > On 18 Jul 2019, at 18:33, Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> wrote: > > On Thu, Jul 18, 2019 at 7:31 PM Vrgotic, Marko > <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: >> >> Hi Daniel, >> >> Is it possible that upgrade from 4.3.3 to 4.3.4 affected/reset/changed the certificate for imageio? > > Have you restarted the ovirt-imageio-proxy service after upgrading? It > should be restarted to reload the certificate. > >> >> Reason why I am asking is that, while running 4.3.3 version, I was uploading Ubuntu 16-18, CentOS 7 cloud images, and CentOS6 no issues. >> Now issue is with CentOS6. >> I have tried to upload centos6.5 image via Ansible ovirt_disk and than one ended stuck in “Finalizing” state. >> Than I tried with to Upload same image from UI and that ended in “Paused” state. >> >> I will try Test Connection and let you know. >> >> Sent from my iPhone >> >>> On 18 Jul 2019, at 16:13, Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> wrote: >>> >>> Hi Marko, >>> >>> According to the log, it seems there's an issue with a secured >>> connection to the proxy. >>> Can you please try the 'Test Connection' button on upload disk dialog, >>> to ensure the certificate is properly installed? >>> Also, is the disk currently in 'Paused by System' status? >>> >>> Thanks, >>> Daniel >>> >>> On Thu, Jul 18, 2019 at 2:35 PM Vrgotic, Marko >>> <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Since this is already compressed file, I have added output of grep for moment this command was issue and following 300lines. >>>> >>>> >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep "2019-07-17 13:36:11,121Z" engine.log-20190* -A300 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> gzip: -A300.gz: No such file or directory >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep -A 300 "2019-07-17 13:36:11,121Z" engine.log-20190* >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,142Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Successfully added Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,188Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, PrepareImageVDSCommand(HostName = ovirt-hv-01.avinity.tv<http://ovirt-hv-01.avinity.tv>, PrepareImageVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d'}), log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,351Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,356Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, SetVolumeLegalityVDSCommand( SetVolumeLegalityVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,390Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, SetVolumeLegalityVDSCommand, return: , log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv<http://ovirt-hv-01.avinity.tv>, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,410Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, AddImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Started transfer session with ticket id 10096b11-d10d-43aa-ad16-668b29a8c152, timeout 300 seconds >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Adding image ticket to ovirt-imageio-proxy, id 10096b11-d10d-43aa-ad16-668b29a8c152 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,469Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.addImageTicketToProxy(TransferDiskImageCommand.java:988) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:911) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:554) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:393) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:388) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) >>>> >>>> engine.log-20190718.gz: at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) [common.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:233) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:144) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:134) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:153) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:70) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForImage(BackendImageTransfersResource.java:39) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:101) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) >>>> >>>> engine.log-20190718.gz: at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.Validator.validate(Validator.java:262) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 224 more >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 230 more >>>> >>>> engine.log-20190718.gz: >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,495Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,502Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_ADD_TICKET_TO_PROXY(1,070), Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,503Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Paused by System >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,506Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Returning from proceedCommandExecution after starting transfer session for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,520Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Upload with disk av-07-centos-65-base was initiated by mvrgotic@ictv.com<mailto:mvrgotic@ictv.com>@ictv.com-authz. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,562Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Running command: TransferImageStatusCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_DISK with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,563Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Finalizing Success >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,610Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,612Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,613Z INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-378) [] User mvrgotic@ictv.com<mailto:mvrgotic@ictv.com>@ictv.com<http://ictv.com> successfully logged out >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,624Z INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-385) [2b577b1a] Running command: TerminateSessionsForTokenCommand internal: true. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,625Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-385) [2b577b1a] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com<mailto:mvrgotic@ictv.com>@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,658Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,659Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:53,434Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com<mailto:mvrgotic@ictv.com>@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,893Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,894Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:28,864Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com<mailto:mvrgotic@ictv.com> successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:30,187Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com<mailto:mvrgotic@ictv.com> successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Setting new tasks map. The map contains now 0 tasks >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Cleared all tasks of pool '10525aa0-839d-11e9-a016-00163e4f2a6d'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,547Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic@ictv.com<mailto:mvrgotic@ictv.com> successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,599Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-376) [519046b4] Running command: CreateUserSessionCommand internal: false. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,614Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [519046b4] EVENT_ID: USER_VDC_LOGIN(30), User mvrgotic@ictv.com<mailto:mvrgotic@ictv.com>@ictv.com-authz connecting from '172.16.105.206' using session 'rhZQZDBy6kGp85btZDieKp2gQ5kQLWBcC7BOfXrU/fJHCVWyoyOvIwnwnaaD12f8YBK5HK3MOaQcYYptCMWRcA==' logged in. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,092Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,093Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:53,430Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user 'mvrgotic@ictv.com<mailto:mvrgotic@ictv.com>@ictv.com-authz'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,109Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,110Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer@redhat.com<mailto:nsoffer@redhat.com>> >>>> Date: Thursday, 18 July 2019 at 13:14 >>>> To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> >>>> Cc: users <users@ovirt.org<mailto:users@ovirt.org>>, Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 1:44 PM Vrgotic, Marko <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure: here is the ovirt-engine/engine.log related to transaction: >>>> >>>> ... >>>> >>>> 2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv<http://ovirt-hv-01.avinity.tv>, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> >>>> >>>> The transfer uuid is mentioned in this log: >>>> >>>> >>>> >>>> transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> >>>> >>>> Please attach complete engine.log since this line. >>>> >>>> >>>> >>>> You can see that the host used for upload was ovirt-hv-01.avinity.tv<http://ovirt-hv-01.avinity.tv>. Please attach vdsm logs >>>> >>>> from this host from the time this ticket was added. >>>> >>>> >>>> >>>> Please attach imageio daemon logs from the same time on this host. >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer@redhat.com<mailto:nsoffer@redhat.com>> >>>> Date: Thursday, 18 July 2019 at 12:20 >>>> To: "Vrgotic, Marko" <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> >>>> Cc: users <users@ovirt.org<mailto:users@ovirt.org>>, Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: >>>> >>>> Dear Nir, >>>> >>>> >>>> >>>> None of my Hosts has any updated left to be added, they are as up to date as they can be. >>>> >>>> The imageio packages version installed is: >>>> >>>> >>>> >>>> Ovirt-imageio-common-1.5.1-0.el7.x86_64 >>>> >>>> Ovirt-imageio-daemon-1.5.1-0.el7.x86_64 >>>> >>>> >>>> >>>> We need logs to understand the issue. >>>> >>>> >>>> >>>> Can you share logs from the time the upload was started? >>>> >>>> - engine log (/var/log/ovirt-engine/engine.log) >>>> >>>> - vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log) >>>> >>>> - daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log) >>>> >>>> >>>> >>>> To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs. >>>> >>>> >>>> >>>> >>>> >>>> Additional software info from Host >>>> >>>> OS Version: >>>> >>>> RHEL - 7 - 6.1810.2.el7.centos >>>> >>>> OS Description: >>>> >>>> CentOS Linux 7 (Core) >>>> >>>> Kernel Version: >>>> >>>> 3.10.0 - 957.21.3.el7.x86_64 >>>> >>>> KVM Version: >>>> >>>> 2.12.0 - 18.el7_6.5.1 >>>> >>>> LIBVIRT Version: >>>> >>>> libvirt-4.5.0-10.el7_6.12 >>>> >>>> VDSM Version: >>>> >>>> vdsm-4.30.17-1.el7 >>>> >>>> SPICE Version: >>>> >>>> 0.14.0 - 6.el7_6.1 >>>> >>>> GlusterFS Version: >>>> >>>> [N/A] >>>> >>>> CEPH Version: >>>> >>>> librbd1-10.2.5-4.el7 >>>> >>>> Open vSwitch Version: >>>> >>>> openvswitch-2.10.1-3.el7 >>>> >>>> Kernel Features: >>>> >>>> PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 >>>> >>>> >>>> >>>> From: "Vrgotic, Marko" <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> >>>> Date: Thursday, 18 July 2019 at 08:03 >>>> To: Nir Soffer <nsoffer@redhat.com<mailto:nsoffer@redhat.com>> >>>> Cc: users <users@ovirt.org<mailto:users@ovirt.org>>, Daniel Erez <derez@redhat.com<mailto:derez@redhat.com>> >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure, i will check. >>>> >>>> >>>> >>>> Is there a way for Adminstrator to view tickets or close them by force? >>>> >>>> Sent from my iPhone >>>> >>>> >>>> On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer@redhat.com<mailto:nsoffer@redhat.com>> wrote: >>>> >>>> >>>> >>>> On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko <M.Vrgotic@activevideo.com<mailto:M.Vrgotic@activevideo.com>> wrote: >>>> >>>> Dear oVIrt, >>>> >>>> >>>> >>>> I initiated upload of qcow2 disk image for Centos 6.5: >>>> >>>> It reached finalizing phase and than started throwing following errors: >>>> >>>> >>>> >>>> 2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well. >>>> >>>> >>>> >>>> This was fixed in 4.2. Are you running the latest version on the hosts? >>>> >>>> >>>> >>>> Please update ovirt-imageio-daemon to latest version. >>>> >>>> >>>> >>>> Nir >>>> >>>> >>>> >>>> 2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module. >>>> >>>> >>>> >>>> Help! >>>> >>>> >>>> >>>> oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts. >>>> >>>> >>>> >>>> Kindly awaiting your reply. >>>> >>>> >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> ActiveVideo >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users@ovirt.org<mailto:users@ovirt.org> >>>> To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org> >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346ZCD...