unable to upload disk images after upgrade to 4.3.3 -- ticket failures
Previously I had issues with the upgrades to 4.3.3 failing because of "stale" image transfer data, so I removed it from the database using the info given here on the mailing list and was able to complete the oVirt node and engine upgrades. Now I have a new problem. I can't upload a disk image anymore, which used to work. "test connection" returns success. In the dashboard Storage/Disks view, it starts and then stops with "paused by system" I tried pointing it at another node, same problem. I tried restarting services ovirt-imageio-proxy on engine and ovirt-imageio-daemon on the node, same failure. in the dashboard events when trying to upload disk I get this error Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy. or when trying to resume transfer "paused by system" giving it the local path again. Transfer was stopped by system. Reason: failure in transfer image ticket renewal. I'm not sure what logs I should be looking deeper into.. Here's what the database says engine=# select * from image_transfers; -[ RECORD 1 ]-------------+------------------------------------- command_id | 59dce2b1-f8ba-44dd-9df9-c6e39773a3f9 command_type | 1024 phase | 4 last_updated | 2019-04-23 12:22:25.098-04 message | Uploading from byte 0 vds_id | disk_id | 6693d5ac-d3eb-43d7-9abe-97c5197efc23 imaged_ticket_id | proxy_uri | signed_ticket | bytes_sent | 0 bytes_total | 1996488704 type | 2 active | f daemon_uri | client_inactivity_timeout | 60 -[ RECORD 2 ]-------------+------------------------------------- command_id | 16898219-be5b-4826-8f20-3355fa47272a command_type | 1024 phase | 4 last_updated | 2019-04-23 16:15:34.591-04 message | Uploading from byte 0 vds_id | disk_id | 3b8e3053-bfe4-49d3-abd2-5452b1674400 imaged_ticket_id | proxy_uri | signed_ticket | bytes_sent | 0 bytes_total | 1998585856 type | 2 active | f daemon_uri | client_inactivity_timeout | 60 I really need the image uploading to work. Any suggestions on what to do next?
Today, I updated engine to the latest Version 4.3.3.6-1.el7. Trying to upload an install iso to the hosted storage domain gives same disk paused by system issue. Apr 25, 2019, 5:26:18 PMTransfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy. on the node which was running ovirt-image /var/log/ovirt-imageio-daemon contains: 2019-04-25 17:08:51,757 INFO (Thread-2) [http] OPEN client=local 2019-04-25 17:08:51,758 INFO (Thread-2) [tickets] [local] ADD ticket={u'uuid': u'6971e886-5339-4df1-88df-445e51fba1c8', u'ops': [u'write'], u'url': u'fi le:///rhev/data-center/mnt/glusterSD/10.200.0.131: _engine/0ec355b1-e665-4fdf-87c4-87e1d636c76d/images/4121f1fb-c508-47bc-9acb-a10da9461358/04658323-b267-4f 66-96f1-6c5848c3d2be', u'sparse': True, u'timeout': 300, u'transfer_id': u'e85ce170-03e6-480d-bbc5-8551c266cc97', u'size': 1045426176} 2019-04-25 17:08:51,759 INFO (Thread-2) [http] CLOSE client=local [connection=0.001420/1, dispatch=0.000634/1] going to that directory above shows # ls -l total 1025 -rw-rw----. 1 vdsm kvm 1045426176 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be -rw-rw----. 1 vdsm kvm 1048576 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be.lease -rw-r--r--. 1 vdsm kvm 345 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be.meta image is showing same size as the iso file. On engine # systemctl status -l ovirt-imageio-proxy.service ● ovirt-imageio-proxy.service - oVirt ImageIO Proxy Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-04-25 16:36:24 EDT; 51min ago Main PID: 56989 (ovirt-imageio-p) CGroup: /system.slice/ovirt-imageio-proxy.service └─56989 /usr/bin/python2 /usr/bin/ovirt-imageio-proxy Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Starting oVirt ImageIO Proxy... Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Started oVirt ImageIO Proxy. Apr 25 17:08:30 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]: 128.182.168.116 - - [25/Apr/2019 17:08:30] "GET /info/ HTTP/1.1" 200 20 Apr 25 17:25:46 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]: 128.182.168.116 - - [25/Apr/2019 17:25:46] "GET /info/ HTTP/1.1" 200 20 # cat image-proxy.log (MainThread) INFO 2019-04-25 16:25:55,755 image_proxy:43:root:(main) Server shut down, exiting (MainThread) INFO 2019-04-25 16:36:24,874 server:45:server:(start) Starting (pid=56989, version=1.5.1) (MainThread) INFO 2019-04-25 16:36:24,879 image_proxy:34:root:(main) Server started, successfully notified systemd On Tue, Apr 23, 2019 at 4:51 PM Edward Berger <edwberger@gmail.com> wrote:
Previously I had issues with the upgrades to 4.3.3 failing because of "stale" image transfer data, so I removed it from the database using the info given here on the mailing list and was able to complete the oVirt node and engine upgrades.
Now I have a new problem. I can't upload a disk image anymore, which used to work. "test connection" returns success.
In the dashboard Storage/Disks view, it starts and then stops with "paused by system"
I tried pointing it at another node, same problem. I tried restarting services ovirt-imageio-proxy on engine and ovirt-imageio-daemon on the node, same failure.
in the dashboard events when trying to upload disk I get this error Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy.
or when trying to resume transfer "paused by system" giving it the local path again. Transfer was stopped by system. Reason: failure in transfer image ticket renewal.
I'm not sure what logs I should be looking deeper into..
Here's what the database says engine=# select * from image_transfers; -[ RECORD 1 ]-------------+------------------------------------- command_id | 59dce2b1-f8ba-44dd-9df9-c6e39773a3f9 command_type | 1024 phase | 4 last_updated | 2019-04-23 12:22:25.098-04 message | Uploading from byte 0 vds_id | disk_id | 6693d5ac-d3eb-43d7-9abe-97c5197efc23 imaged_ticket_id | proxy_uri | signed_ticket | bytes_sent | 0 bytes_total | 1996488704 type | 2 active | f daemon_uri | client_inactivity_timeout | 60 -[ RECORD 2 ]-------------+------------------------------------- command_id | 16898219-be5b-4826-8f20-3355fa47272a command_type | 1024 phase | 4 last_updated | 2019-04-23 16:15:34.591-04 message | Uploading from byte 0 vds_id | disk_id | 3b8e3053-bfe4-49d3-abd2-5452b1674400 imaged_ticket_id | proxy_uri | signed_ticket | bytes_sent | 0 bytes_total | 1998585856 type | 2 active | f daemon_uri | client_inactivity_timeout | 60
I really need the image uploading to work. Any suggestions on what to do next?
On Fri, Apr 26, 2019 at 12:34 AM Edward Berger <edwberger@gmail.com> wrote:
Today, I updated engine to the latest Version 4.3.3.6-1.el7. Trying to upload an install iso to the hosted storage domain gives same disk paused by system issue.
Apr 25, 2019, 5:26:18 PMTransfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy.
on the node which was running ovirt-image
/var/log/ovirt-imageio-daemon contains: 2019-04-25 17:08:51,757 INFO (Thread-2) [http] OPEN client=local 2019-04-25 17:08:51,758 INFO (Thread-2) [tickets] [local] ADD ticket={u'uuid': u'6971e886-5339-4df1-88df-445e51fba1c8', u'ops': [u'write'], u'url': u'fi le:///rhev/data-center/mnt/glusterSD/10.200.0.131:_engine/0ec355b1-e665-4fdf-87c4-87e1d636c76d/images/4121f1fb-c508-47bc-9acb-a10da9461358/04658323-b267-4f 66-96f1-6c5848c3d2be', u'sparse': True, u'timeout': 300, u'transfer_id': u'e85ce170-03e6-480d-bbc5-8551c266cc97', u'size': 1045426176} 2019-04-25 17:08:51,759 INFO (Thread-2) [http] CLOSE client=local [connection=0.001420/1, dispatch=0.000634/1]
going to that directory above shows # ls -l total 1025 -rw-rw----. 1 vdsm kvm 1045426176 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be -rw-rw----. 1 vdsm kvm 1048576 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be.lease -rw-r--r--. 1 vdsm kvm 345 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be.meta
image is showing same size as the iso file.
On engine # systemctl status -l ovirt-imageio-proxy.service ● ovirt-imageio-proxy.service - oVirt ImageIO Proxy Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-04-25 16:36:24 EDT; 51min ago Main PID: 56989 (ovirt-imageio-p) CGroup: /system.slice/ovirt-imageio-proxy.service └─56989 /usr/bin/python2 /usr/bin/ovirt-imageio-proxy
Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Starting oVirt ImageIO Proxy... Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Started oVirt ImageIO Proxy. Apr 25 17:08:30 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]: 128.182.168.116 - - [25/Apr/2019 17:08:30] "GET /info/ HTTP/1.1" 200 20 Apr 25 17:25:46 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]: 128.182.168.116 - - [25/Apr/2019 17:25:46] "GET /info/ HTTP/1.1" 200 20
# cat image-proxy.log (MainThread) INFO 2019-04-25 16:25:55,755 image_proxy:43:root:(main) Server shut down, exiting (MainThread) INFO 2019-04-25 16:36:24,874 server:45:server:(start) Starting (pid=56989, version=1.5.1) (MainThread) INFO 2019-04-25 16:36:24,879 image_proxy:34:root:(main) Server started, successfully notified systemd
On Tue, Apr 23, 2019 at 4:51 PM Edward Berger <edwberger@gmail.com> wrote:
Previously I had issues with the upgrades to 4.3.3 failing because of "stale" image transfer data, so I removed it from the database using the info given here on the mailing list and was able to complete the oVirt node and engine upgrades.
Now I have a new problem. I can't upload a disk image anymore, which used to work. "test connection" returns success.
In the dashboard Storage/Disks view, it starts and then stops with "paused by system"
Hi! Are you trying to use a 3rd-party CA for https access [4][5]? Please check this thread: [1] https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NZBGRCKW6WA4WIS... Part of this was already fixed: [2] https://bugzilla.redhat.com/show_bug.cgi?id=1687301 And another part is still pending, although you should easily "fix" manually (editing imageio-proxy.conf): [3] https://bugzilla.redhat.com/show_bug.cgi?id=1637809 If so: 1. Please try again following the entire procedure for using 3rd-party ca. 2. Please also edit /etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf as explained in above bug [3], and then restart imageio-proxy and the engine. If it still fails, please check/share all relevant differences between a vanilla setup and yours, including all the changes you did as part of following the 3rd-party-ca procedure, custom/different config files, changes you did to the system-wide truststore, etc. [4] https://www.ovirt.org/documentation/admin-guide/appe-oVirt_and_SSL.html [5] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/htm...
I tried pointing it at another node, same problem. I tried restarting services ovirt-imageio-proxy on engine and ovirt-imageio-daemon on the node, same failure.
in the dashboard events when trying to upload disk I get this error Transfer was stopped by system. Reason: failed to add image ticket to ovirt-imageio-proxy.
or when trying to resume transfer "paused by system" giving it the local path again. Transfer was stopped by system. Reason: failure in transfer image ticket renewal.
I'm not sure what logs I should be looking deeper into..
The only relevant ones are the engine's logs and imageio-proxy's, and sadly they didn't help me enough to debug this when I did.
Here's what the database says engine=# select * from image_transfers; -[ RECORD 1 ]-------------+------------------------------------- command_id | 59dce2b1-f8ba-44dd-9df9-c6e39773a3f9 command_type | 1024 phase | 4 last_updated | 2019-04-23 12:22:25.098-04 message | Uploading from byte 0 vds_id | disk_id | 6693d5ac-d3eb-43d7-9abe-97c5197efc23 imaged_ticket_id | proxy_uri | signed_ticket | bytes_sent | 0 bytes_total | 1996488704 type | 2 active | f daemon_uri | client_inactivity_timeout | 60 -[ RECORD 2 ]-------------+------------------------------------- command_id | 16898219-be5b-4826-8f20-3355fa47272a command_type | 1024 phase | 4 last_updated | 2019-04-23 16:15:34.591-04 message | Uploading from byte 0 vds_id | disk_id | 3b8e3053-bfe4-49d3-abd2-5452b1674400 imaged_ticket_id | proxy_uri | signed_ticket | bytes_sent | 0 bytes_total | 1998585856 type | 2 active | f daemon_uri | client_inactivity_timeout | 60
I really need the image uploading to work. Any suggestions on what to do next?
I am ignoring for now your debug attempts looking at logs on the hosts etc., we can go back to that if above is not enough. Good luck and best regards, -- Didi
participants (2)
-
Edward Berger -
Yedidyah Bar David