[ovirt-users] Re: Stuck in "Finalizing" disk upload phase

Hi Daniel, Question 1 is answered. Update overwrote the ovirt-imageio-proxy.conf to default, resulting in SSL Handshake mismatch, which is why Uploads were ending in Paused state. We will add this to standard Upgrade procedure, minor/major releases. I appreciate the help, it allows us to learn and get better.

Question 2 remains open: The image of 40GB with UUID of 3452459d-aec6-430e-9509-1d9ca815b2d8 pointing to -> /rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8 is still stuck in finalizing phase. This is the image that I reported issue about and the upload was executed using ovirt_disk ansible module. The files were removed afterwards manually by me (which probably was not good idea), as an attempt to cause an exception or to see if I can trigger iodaemon to break the task due to missing file. How can I remove/stop/kill this task?

— — — Met vriendelijke groet / Kind regards, *Marko Vrgotic* *From: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com&gt; *Date: *Friday, 19 July 2019 at 11:21 *To: *Daniel Erez <derez(a)redhat.com&gt; *Cc: *&quot;nsoffer(a)redhat.com&quot; <nsoffer(a)redhat.com&gt;, &quot;users(a)ovirt.org&quot; < users(a)ovirt.org&gt;, "Stojchev, Darko" <D.Stojchev(a)activevideo.com&gt; *Subject: *Re: [ovirt-users] Stuck in "Finalizing" disk upload phase +my colleague Darko, Welcome to oVirt community — — — Met vriendelijke groet / Kind regards, *Marko Vrgotic* *From: *"Vrgotic, Marko" <M.Vrgotic(a)activevideo.com&gt; *Date: *Friday, 19 July 2019 at 10:30 *To: *Daniel Erez <derez(a)redhat.com&gt; *Cc: *&quot;nsoffer(a)redhat.com&quot; <nsoffer(a)redhat.com&gt;, &quot;users(a)ovirt.org&quot; < users(a)ovirt.org&gt; *Subject: *Re: [ovirt-users] Stuck in "Finalizing" disk upload phase Dear Daniel, I apologize up front for long email, but I just want to make sure all relevant info is provided. I have executed following actions: - Put the system into Global Maintenance - Executed "yum update" - Restarted SHE server - Once it was Up and all services Up, removed Global Maintenance mode - From UI attempted "Test Connection" several times from different Host: Attempted two Uploads, one at a time from UI, of same centos-65.qcow2 image. - First time with alias testimport - Second time with alias av-07-centos-65-base Both Uploads ended in Paused state, which I was able to cancel and the task was removed (full engine logs of these two tasks are attached). Seem that in both case there is an issue with Certificate, even though imageio daemon was restarted (I rebooted server in steps above). Question1: Considering I did not see this issue prio to update from 4.3.3 to 4.3.4 it still makes me wonder if something happened with certificate or certificate path? I will also check in my backups Question2: The image of 40GB with UUID of 3452459d-aec6-430e-9509-1d9ca815b2d8 pointing to -> /rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8 is still stuck in finalizing phase. This is the image that I reported issue about and the upload was executed using ovirt_disk ansible module. The files were removed afterwards manually by me (which probably was not good idea), as an attempt to cause an exception or to see if I can trigger iodaemon to break the task due to missing file. How can I remote this task or stop it? Kindly awaiting your reply. — — — Met vriendelijke groet / Kind regards, *Marko Vrgotic* On 19/07/2019, 07:12, "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com&gt; wrote: Very good question, Now that I think of it, I might have missed this step. Most of the time/always after upgrade of oVirt packages is done, I update the rest of the system, both engine and hosts and reboot if required. I do not think I did and system update of the engine this time, so for sure did not reboot it. Will do that and report back. Thank you Daniel. Marko Vrgotic Sent from my iPhone > On 18 Jul 2019, at 18:33, Daniel Erez <derez(a)redhat.com&gt; wrote: > > On Thu, Jul 18, 2019 at 7:31 PM Vrgotic, Marko > <M.Vrgotic(a)activevideo.com&gt; wrote: >> >> Hi Daniel, >> >> Is it possible that upgrade from 4.3.3 to 4.3.4 affected/reset/changed the certificate for imageio? > > Have you restarted the ovirt-imageio-proxy service after upgrading? It > should be restarted to reload the certificate. > >> >> Reason why I am asking is that, while running 4.3.3 version, I was uploading Ubuntu 16-18, CentOS 7 cloud images, and CentOS6 no issues. >> Now issue is with CentOS6. >> I have tried to upload centos6.5 image via Ansible ovirt_disk and than one ended stuck in “Finalizing” state. >> Than I tried with to Upload same image from UI and that ended in “Paused” state. >> >> I will try Test Connection and let you know. >> >> Sent from my iPhone >> >>> On 18 Jul 2019, at 16:13, Daniel Erez <derez(a)redhat.com&gt; wrote: >>> >>> Hi Marko, >>> >>> According to the log, it seems there's an issue with a secured >>> connection to the proxy. >>> Can you please try the 'Test Connection' button on upload disk dialog, >>> to ensure the certificate is properly installed? >>> Also, is the disk currently in 'Paused by System' status? >>> >>> Thanks, >>> Daniel >>> >>> On Thu, Jul 18, 2019 at 2:35 PM Vrgotic, Marko >>> <M.Vrgotic(a)activevideo.com&gt; wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Since this is already compressed file, I have added output of grep for moment this command was issue and following 300lines. >>>> >>>> >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep "2019-07-17 13:36:11,121Z" engine.log-20190* -A300 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> gzip: -A300.gz: No such file or directory >>>> >>>> [root@ovirt-engine ovirt-engine]# zgrep -A 300 "2019-07-17 13:36:11,121Z" engine.log-20190* >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Running command: TransferDiskImageCommand internal: false. Entities affected : ID: 3452459d-aec6-430e-9509-1d9ca815b2d8 Type: DiskAction group EDIT_DISK_PROPERTIES with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,121Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Creating ImageTransfer entity for command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,142Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Successfully added Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,188Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, PrepareImageVDSCommand(HostName = ovirt-hv-01.avinity.tv, PrepareImageVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d'}), log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,351Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.PrepareImageVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, PrepareImageVDSCommand, return: PrepareImageReturn:{status='Status [code=0, message=Done]'}, log id: 3bd84b38 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,356Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, SetVolumeLegalityVDSCommand( SetVolumeLegalityVDSCommandParameters:{storagePoolId='10525aa0-839d-11e9-a016-00163e4f2a6d', ignoreFailoverLimit='false', storageDomainId='644aacaa-12e1-4fcd-b3aa-941678cf95bd', imageGroupId='3452459d-aec6-430e-9509-1d9ca815b2d8', imageId='b44659a9-607a-4eeb-a255-99532fd4fce4'}), log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,390Z INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeLegalityVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, SetVolumeLegalityVDSCommand, return: , log id: 5e0f7654 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,410Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] FINISH, AddImageTicketVDSCommand, return: StatusOnlyReturn [status=Status [code=0, message=Done]], log id: 3bf6b943 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Started transfer session with ticket id 10096b11-d10d-43aa-ad16-668b29a8c152, timeout 300 seconds >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,411Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Adding image ticket to ovirt-imageio-proxy, id 10096b11-d10d-43aa-ad16-668b29a8c152 >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,469Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.addImageTicketToProxy(TransferDiskImageCommand.java:988) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.startImageTransferSession(TransferDiskImageCommand.java:911) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.handleImageIsReadyForTransfer(TransferDiskImageCommand.java:554) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand.executeCommand(TransferDiskImageCommand.java:393) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1157) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1315) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1964) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1375) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:419) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:451) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:433) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:388) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor775.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) [weld-core-impl-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:] >>>> >>>> engine.log-20190718.gz: at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source) [:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:216) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:418) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:148) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) >>>> >>>> engine.log-20190718.gz: at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) [weld-ejb-3.0.5.Final.jar:3.0.5.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-15.0.1.Final.jar:15.0.1.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) >>>> >>>> engine.log-20190718.gz: at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:618) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) >>>> >>>> engine.log-20190718.gz: at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185) >>>> >>>> engine.log-20190718.gz: at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) [common.jar:] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendResource.doAction(BackendResource.java:250) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.doCreateEntity(AbstractBackendCollectionResource.java:233) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:144) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:134) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.AbstractBackendCollectionResource.performCreate(AbstractBackendCollectionResource.java:153) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.performCreate(BackendImageTransfersResource.java:70) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.resource.BackendImageTransfersResource.addForImage(BackendImageTransfersResource.java:39) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.resource.ImageTransfersResource.add(ImageTransfersResource.java:101) >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:509) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:399) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$0(ResourceMethodInvoker.java:363) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:365) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:337) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:137) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:100) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:439) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:229) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:135) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.interception.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:355) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:138) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:215) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) [resteasy-jaxrs-3.6.2.Final.jar:3.6.2.Final] >>>> >>>> engine.log-20190718.gz: at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) [jboss-servlet-api_4.0_spec-1.0.0.Final.jar:1.0.0.Final] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:274) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchToPath(ServletInitialHandler.java:209) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImpl(RequestDispatcherImpl.java:223) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forwardImplSetup(RequestDispatcherImpl.java:149) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.spec.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:178) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.VersionFilter.doFilter(VersionFilter.java:98) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:117) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.invocation.CurrentFilter.doFilter(CurrentFilter.java:72) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:78) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:42) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiNegotiationFilter.doFilter(SsoRestApiNegotiationFilter.java:84) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter.doFilter(SsoRestApiAuthFilter.java:47) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:59) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:35) [aaa.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:111) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.api.restapi.security.CSRFProtectionFilter.doFilter(CSRFProtectionFilter.java:102) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at org.ovirt.engine.core.utils.servlet.CORSSupportFilter.doFilter(CORSSupportFilter.java:283) [utils.jar:] >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) >>>> >>>> engine.log-20190718.gz: at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) >>>> >>>> engine.log-20190718.gz: at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360) >>>> >>>> engine.log-20190718.gz: at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) >>>> >>>> engine.log-20190718.gz: at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378) >>>> >>>> engine.log-20190718.gz: at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.Validator.validate(Validator.java:262) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) [jsse.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 224 more >>>> >>>> engine.log-20190718.gz:Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) [rt.jar:1.8.0_201] >>>> >>>> engine.log-20190718.gz: ... 230 more >>>> >>>> engine.log-20190718.gz: >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,495Z ERROR [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to add image ticket to ovirt-imageio-proxy >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,502Z ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_STOPPED_BY_SYSTEM_FAILED_TO_ADD_TICKET_TO_PROXY(1,070), Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,503Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Paused by System >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,506Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Returning from proceedCommandExecution after starting transfer session for image transfer command '28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,520Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] EVENT_ID: TRANSFER_IMAGE_INITIATED(1,031), Image Upload with disk av-07-centos-65-base was initiated by mvrgotic@ictv.com(a)ictv.com-authz. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,562Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferImageStatusCommand] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Running command: TransferImageStatusCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_DISK with role type USER >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,563Z INFO [org.ovirt.engine.core.bll.storage.disk.image.ImageTransferUpdater] (default task-376) [edda86b2-d78b-45ed-a523-18966af140f4] Updating image transfer 28eda0c2-e36b-4e70-91ea-2ecf4a030d19 (image 3452459d-aec6-430e-9509-1d9ca815b2d8) phase to Finalizing Success >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,610Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,612Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,613Z INFO [org.ovirt.engine.core.sso.servlets.OAuthRevokeServlet] (default task-378) [] User mvrgotic@ictv.com(a)ictv.com successfully logged out >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,624Z INFO [org.ovirt.engine.core.bll.aaa.TerminateSessionsForTokenCommand] (default task-385) [2b577b1a] Running command: TerminateSessionsForTokenCommand internal: true. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:11,625Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (default task-385) [2b577b1a] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user &#39;mvrgotic@ictv.com(a)ictv.com-authz&#39;. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,658Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:13,659Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-69) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:17,703Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-5) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:25,748Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-52) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:35,802Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:45,821Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-9) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:53,434Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-26) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user &#39;mvrgotic@ictv.com(a)ictv.com-authz&#39;. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:36:55,872Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,893Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:05,894Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-60) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:15,945Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-29) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:25,999Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-48) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:28,864Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic(a)ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:30,187Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic(a)ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Setting new tasks map. The map contains now 0 tasks >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:33,323Z INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [] Cleared all tasks of pool '10525aa0-839d-11e9-a016-00163e4f2a6d'. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,547Z INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-376) [] User mvrgotic(a)ictv.com successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,599Z INFO [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default task-376) [519046b4] Running command: CreateUserSessionCommand internal: false. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:35,614Z INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-376) [519046b4] EVENT_ID: USER_VDC_LOGIN(30), User mvrgotic@ictv.com(a)ictv.com-authz connecting from '172.16.105.206' using session 'rhZQZDBy6kGp85btZDieKp2gQ5kQLWBcC7BOfXrU/fJHCVWyoyOvIwnwnaaD12f8YBK5HK3MOaQcYYptCMWRcA==' logged in. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:36,045Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,092Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:46,093Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-23) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:53,430Z INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [] Not removing session 'IGb5NgFB7aK78R92aRxJYix9y4TVQJtzYtUW+EGiAanLfxkDb/CCEuU6pgN0e7EzOuki+mMK14LaJutl9Rhydw==', session has running commands for user &#39;mvrgotic@ictv.com(a)ictv.com-authz&#39;. >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,109Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> engine.log-20190718.gz:2019-07-17 13:37:56,110Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-27) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer(a)redhat.com&gt; >>>> Date: Thursday, 18 July 2019 at 13:14 >>>> To: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com&gt; >>>> Cc: users <users(a)ovirt.org&gt;, Daniel Erez <derez(a)redhat.com&gt; >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 1:44 PM Vrgotic, Marko < M.Vrgotic(a)activevideo.com&gt; wrote: >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure: here is the ovirt-engine/engine.log related to transaction: >>>> >>>> ... >>>> >>>> 2019-07-17 13:36:11,394Z INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (default task-376) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] START, AddImageTicketVDSCommand(HostName = ovirt-hv-01.avinity.tv, AddImageTicketVDSCommandParameters:{hostId='e7e3f1dc-8037-4e74-a44c-442bdb02197d', ticketId='10096b11-d10d-43aa-ad16-668b29a8c152', timeout='300', operations='[write]', size='47244640256', url='file:///rhev/data-center/mnt/172.17.28.5:_ovirt__production/644aacaa-12e1-4fcd-b3aa-941678cf95bd/images/3452459d-aec6-430e-9509-1d9ca815b2d8/b44659a9-607a-4eeb-a255-99532fd4fce4', filename='null', sparse='true', transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19'}), log id: 3bf6b943 >>>> >>>> >>>> >>>> The transfer uuid is mentioned in this log: >>>> >>>> >>>> >>>> transferId='28eda0c2-e36b-4e70-91ea-2ecf4a030d19' >>>> >>>> >>>> >>>> Please attach complete engine.log since this line. >>>> >>>> >>>> >>>> You can see that the host used for upload was ovirt-hv-01.avinity.tv. Please attach vdsm logs >>>> >>>> from this host from the time this ticket was added. >>>> >>>> >>>> >>>> Please attach imageio daemon logs from the same time on this host. >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> >>>> >>>> >>>> >>>> From: Nir Soffer <nsoffer(a)redhat.com&gt; >>>> Date: Thursday, 18 July 2019 at 12:20 >>>> To: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com&gt; >>>> Cc: users <users(a)ovirt.org&gt;, Daniel Erez <derez(a)redhat.com&gt; >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> On Thu, Jul 18, 2019 at 10:49 AM Vrgotic, Marko < M.Vrgotic(a)activevideo.com&gt; wrote: >>>> >>>> Dear Nir, >>>> >>>> >>>> >>>> None of my Hosts has any updated left to be added, they are as up to date as they can be. >>>> >>>> The imageio packages version installed is: >>>> >>>> >>>> >>>> Ovirt-imageio-common-1.5.1-0.el7.x86_64 >>>> >>>> Ovirt-imageio-daemon-1.5.1-0.el7.x86_64 >>>> >>>> >>>> >>>> We need logs to understand the issue. >>>> >>>> >>>> >>>> Can you share logs from the time the upload was started? >>>> >>>> - engine log (/var/log/ovirt-engine/engine.log) >>>> >>>> - vdsm log on the host that performed the upload (/var/log/vdsm/vdsm.log) >>>> >>>> - daemon logs on that host (/var/log/ovirt-imageio-daemon/daemon.log) >>>> >>>> >>>> >>>> To locate the right host you can grep for the transfer uuid that should be mentioned in engine logs. >>>> >>>> >>>> >>>> >>>> >>>> Additional software info from Host >>>> >>>> OS Version: >>>> >>>> RHEL - 7 - 6.1810.2.el7.centos >>>> >>>> OS Description: >>>> >>>> CentOS Linux 7 (Core) >>>> >>>> Kernel Version: >>>> >>>> 3.10.0 - 957.21.3.el7.x86_64 >>>> >>>> KVM Version: >>>> >>>> 2.12.0 - 18.el7_6.5.1 >>>> >>>> LIBVIRT Version: >>>> >>>> libvirt-4.5.0-10.el7_6.12 >>>> >>>> VDSM Version: >>>> >>>> vdsm-4.30.17-1.el7 >>>> >>>> SPICE Version: >>>> >>>> 0.14.0 - 6.el7_6.1 >>>> >>>> GlusterFS Version: >>>> >>>> [N/A] >>>> >>>> CEPH Version: >>>> >>>> librbd1-10.2.5-4.el7 >>>> >>>> Open vSwitch Version: >>>> >>>> openvswitch-2.10.1-3.el7 >>>> >>>> Kernel Features: >>>> >>>> PTI: 1, IBRS: 0, RETP: 1, SSBD: 3 >>>> >>>> >>>> >>>> From: "Vrgotic, Marko" <M.Vrgotic(a)activevideo.com&gt; >>>> Date: Thursday, 18 July 2019 at 08:03 >>>> To: Nir Soffer <nsoffer(a)redhat.com&gt; >>>> Cc: users <users(a)ovirt.org&gt;, Daniel Erez <derez(a)redhat.com&gt; >>>> Subject: Re: [ovirt-users] Stuck in "Finalizing" disk upload phase >>>> >>>> >>>> >>>> Hi Nir, >>>> >>>> >>>> >>>> Sure, i will check. >>>> >>>> >>>> >>>> Is there a way for Adminstrator to view tickets or close them by force? >>>> >>>> Sent from my iPhone >>>> >>>> >>>> On 18 Jul 2019, at 00:05, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: >>>> >>>> >>>> >>>> On Wed, Jul 17, 2019, 19:20 Vrgotic, Marko < M.Vrgotic(a)activevideo.com&gt; wrote: >>>> >>>> Dear oVIrt, >>>> >>>> >>>> >>>> I initiated upload of qcow2 disk image for Centos 6.5: >>>> >>>> It reached finalizing phase and than started throwing following errors: >>>> >>>> >>>> >>>> 2019-07-17 14:40:51,480Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:40:51,480Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-86) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> Old versions of ovirt-imageio-daemon were failing to remove a ticket if the ticket does not exist, and engine did not handle this well. >>>> >>>> >>>> >>>> This was fixed in 4.2. Are you running the latest version on the hosts? >>>> >>>> >>>> >>>> Please update ovirt-imageio-daemon to latest version. >>>> >>>> >>>> >>>> Nir >>>> >>>> >>>> >>>> 2019-07-17 14:41:01,572Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:01,574Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-19) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:11,690Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:11,690Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> 2019-07-17 14:41:21,781Z INFO [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Finalizing successful transfer for Upload disk 'av-07-centos-65-base' (disk id: '3452459d-aec6-430e-9509-1d9ca815b2d8', image id: 'b44659a9-607a-4eeb-a255-99532fd4fce4') >>>> >>>> 2019-07-17 14:41:21,782Z WARN [org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [a43180ec-afc7-429e-9f30-9e851eaf7ce7] Failed to stop image transfer session. Ticket does not exist for image '3452459d-aec6-430e-9509-1d9ca815b2d8' >>>> >>>> >>>> >>>> I can not cancel it, can not stop it, not via UI not via force option using ovirt_disk module. >>>> >>>> >>>> >>>> Help! >>>> >>>> >>>> >>>> oVIrt 4.3.4.3-1 version running with CentOS 7.6 Hosts. >>>> >>>> >>>> >>>> Kindly awaiting your reply. >>>> >>>> >>>> >>>> >>>> >>>> — — — >>>> Met vriendelijke groet / Kind regards, >>>> >>>> Marko Vrgotic >>>> >>>> ActiveVideo >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list -- users(a)ovirt.org >>>> To unsubscribe send an email to users-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LJPWK5A3346...