11:51 p.m.
Previously I had issues with the upgrades to 4.3.3 failing because of
"stale" image transfer data, so I removed it from the database using the
info given here on the mailing list and was able to complete the oVirt node
and engine upgrades.
Now I have a new problem. I can't upload a disk image anymore, which used
to work.
"test connection" returns success.
In the dashboard Storage/Disks view, it starts and then stops with "paused
by system"
I tried pointing it at another node, same problem. I tried restarting
services ovirt-imageio-proxy on engine and ovirt-imageio-daemon on the
node, same failure.
in the dashboard events when trying to upload disk I get this error
Transfer was stopped by system. Reason: failed to add image ticket to
ovirt-imageio-proxy.
or when trying to resume transfer "paused by system" giving it the local
path again.
Transfer was stopped by system. Reason: failure in transfer image
ticket renewal.
I'm not sure what logs I should be looking deeper into..
Here's what the database says
engine=# select * from image_transfers;
-[ RECORD 1 ]-------------+-------------------------------------
command_id | 59dce2b1-f8ba-44dd-9df9-c6e39773a3f9
command_type | 1024
phase | 4
last_updated | 2019-04-23 12:22:25.098-04
message | Uploading from byte 0
vds_id |
disk_id | 6693d5ac-d3eb-43d7-9abe-97c5197efc23
imaged_ticket_id |
proxy_uri |
signed_ticket |
bytes_sent | 0
bytes_total | 1996488704
type | 2
active | f
daemon_uri |
client_inactivity_timeout | 60
-[ RECORD 2 ]-------------+-------------------------------------
command_id | 16898219-be5b-4826-8f20-3355fa47272a
command_type | 1024
phase | 4
last_updated | 2019-04-23 16:15:34.591-04
message | Uploading from byte 0
vds_id |
disk_id | 3b8e3053-bfe4-49d3-abd2-5452b1674400
imaged_ticket_id |
proxy_uri |
signed_ticket |
bytes_sent | 0
bytes_total | 1998585856
type | 2
active | f
daemon_uri |
client_inactivity_timeout | 60
I really need the image uploading to work. Any suggestions on what to do
next?
12:32 a.m.
Today, I updated engine to the latest Version 4.3.3.6-1.el7.
Trying to upload an install iso to the hosted storage domain gives same
disk paused by system issue.
Apr 25, 2019, 5:26:18 PMTransfer was stopped by system. Reason: failed to
add image ticket to ovirt-imageio-proxy.
on the node which was running ovirt-image
/var/log/ovirt-imageio-daemon contains:
2019-04-25 17:08:51,757 INFO (Thread-2) [http] OPEN client=local
2019-04-25 17:08:51,758 INFO (Thread-2) [tickets] [local] ADD
ticket={u'uuid': u'6971e886-5339-4df1-88df-445e51fba1c8', u'ops':
[u'write'], u'url': u'fi
le:///rhev/data-center/mnt/glusterSD/10.200.0.131:
_engine/0ec355b1-e665-4fdf-87c4-87e1d636c76d/images/4121f1fb-c508-47bc-9acb-a10da9461358/04658323-b267-4f
66-96f1-6c5848c3d2be', u'sparse': True, u'timeout': 300,
u'transfer_id': u'e85ce170-03e6-480d-bbc5-8551c266cc97', u'size':
1045426176}
2019-04-25 17:08:51,759 INFO (Thread-2) [http] CLOSE client=local
[connection=0.001420/1, dispatch=0.000634/1]
going to that directory above shows
# ls -l
total 1025
-rw-rw----. 1 vdsm kvm 1045426176 Apr 25 17:08
04658323-b267-4f66-96f1-6c5848c3d2be
-rw-rw----. 1 vdsm kvm 1048576 Apr 25 17:08
04658323-b267-4f66-96f1-6c5848c3d2be.lease
-rw-r--r--. 1 vdsm kvm 345 Apr 25 17:08
04658323-b267-4f66-96f1-6c5848c3d2be.meta
image is showing same size as the iso file.
On engine
# systemctl status -l ovirt-imageio-proxy.service
● ovirt-imageio-proxy.service - oVirt ImageIO Proxy
Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service;
enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-04-25 16:36:24 EDT; 51min ago
Main PID: 56989 (ovirt-imageio-p)
CGroup: /system.slice/ovirt-imageio-proxy.service
└─56989 /usr/bin/python2 /usr/bin/ovirt-imageio-proxy
Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Starting oVirt ImageIO
Proxy...
Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Started oVirt ImageIO
Proxy.
Apr 25 17:08:30 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]:
128.182.168.116 - - [25/Apr/2019 17:08:30] "GET /info/ HTTP/1.1" 200 20
Apr 25 17:25:46 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]:
128.182.168.116 - - [25/Apr/2019 17:25:46] "GET /info/ HTTP/1.1" 200 20
# cat image-proxy.log
(MainThread) INFO 2019-04-25 16:25:55,755 image_proxy:43:root:(main) Server
shut down, exiting
(MainThread) INFO 2019-04-25 16:36:24,874 server:45:server:(start) Starting
(pid=56989, version=1.5.1)
(MainThread) INFO 2019-04-25 16:36:24,879 image_proxy:34:root:(main) Server
started, successfully notified systemd
On Tue, Apr 23, 2019 at 4:51 PM Edward Berger <edwberger(a)gmail.com> wrote:
Previously I had issues with the upgrades to 4.3.3 failing because
of
"stale" image transfer data, so I removed it from the database using the
info given here on the mailing list and was able to complete the oVirt node
and engine upgrades.
Now I have a new problem. I can't upload a disk image anymore, which used
to work.
"test connection" returns success.
In the dashboard Storage/Disks view, it starts and then stops with "paused
by system"
I tried pointing it at another node, same problem. I tried restarting
services ovirt-imageio-proxy on engine and ovirt-imageio-daemon on the
node, same failure.
in the dashboard events when trying to upload disk I get this error
Transfer was stopped by system. Reason: failed to add image ticket
to ovirt-imageio-proxy.
or when trying to resume transfer "paused by system" giving it the local
path again.
Transfer was stopped by system. Reason: failure in transfer image
ticket renewal.
I'm not sure what logs I should be looking deeper into..
Here's what the database says
engine=# select * from image_transfers;
-[ RECORD 1 ]-------------+-------------------------------------
command_id | 59dce2b1-f8ba-44dd-9df9-c6e39773a3f9
command_type | 1024
phase | 4
last_updated | 2019-04-23 12:22:25.098-04
message | Uploading from byte 0
vds_id |
disk_id | 6693d5ac-d3eb-43d7-9abe-97c5197efc23
imaged_ticket_id |
proxy_uri |
signed_ticket |
bytes_sent | 0
bytes_total | 1996488704
type | 2
active | f
daemon_uri |
client_inactivity_timeout | 60
-[ RECORD 2 ]-------------+-------------------------------------
command_id | 16898219-be5b-4826-8f20-3355fa47272a
command_type | 1024
phase | 4
last_updated | 2019-04-23 16:15:34.591-04
message | Uploading from byte 0
vds_id |
disk_id | 3b8e3053-bfe4-49d3-abd2-5452b1674400
imaged_ticket_id |
proxy_uri |
signed_ticket |
bytes_sent | 0
bytes_total | 1998585856
type | 2
active | f
daemon_uri |
client_inactivity_timeout | 60
I really need the image uploading to work. Any suggestions on what to do
next?
9 a.m.
New subject: unable to upload disk images after upgrade to 4.3.3 -- ticket failures
On Fri, Apr 26, 2019 at 12:34 AM Edward Berger <edwberger(a)gmail.com> wrote:
Today, I updated engine to the latest Version 4.3.3.6-1.el7.
Trying to upload an install iso to the hosted storage domain gives same disk paused by
system issue.
Apr 25, 2019, 5:26:18 PMTransfer was stopped by system. Reason: failed to add image
ticket to ovirt-imageio-proxy.
on the node which was running ovirt-image
/var/log/ovirt-imageio-daemon contains:
2019-04-25 17:08:51,757 INFO (Thread-2) [http] OPEN client=local
2019-04-25 17:08:51,758 INFO (Thread-2) [tickets] [local] ADD ticket={u'uuid':
u'6971e886-5339-4df1-88df-445e51fba1c8', u'ops': [u'write'],
u'url': u'fi
le:///rhev/data-center/mnt/glusterSD/10.200.0.131:_engine/0ec355b1-e665-4fdf-87c4-87e1d636c76d/images/4121f1fb-c508-47bc-9acb-a10da9461358/04658323-b267-4f
66-96f1-6c5848c3d2be', u'sparse': True, u'timeout': 300,
u'transfer_id': u'e85ce170-03e6-480d-bbc5-8551c266cc97', u'size':
1045426176}
2019-04-25 17:08:51,759 INFO (Thread-2) [http] CLOSE client=local
[connection=0.001420/1, dispatch=0.000634/1]
going to that directory above shows
# ls -l
total 1025
-rw-rw----. 1 vdsm kvm 1045426176 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be
-rw-rw----. 1 vdsm kvm 1048576 Apr 25 17:08
04658323-b267-4f66-96f1-6c5848c3d2be.lease
-rw-r--r--. 1 vdsm kvm 345 Apr 25 17:08 04658323-b267-4f66-96f1-6c5848c3d2be.meta
image is showing same size as the iso file.
On engine
# systemctl status -l ovirt-imageio-proxy.service
● ovirt-imageio-proxy.service - oVirt ImageIO Proxy
Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service; enabled; vendor
preset: disabled)
Active: active (running) since Thu 2019-04-25 16:36:24 EDT; 51min ago
Main PID: 56989 (ovirt-imageio-p)
CGroup: /system.slice/ovirt-imageio-proxy.service
└─56989 /usr/bin/python2 /usr/bin/ovirt-imageio-proxy
Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Starting oVirt ImageIO Proxy...
Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Started oVirt ImageIO Proxy.
Apr 25 17:08:30 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]: 128.182.168.116 - -
[25/Apr/2019 17:08:30] "GET /info/ HTTP/1.1" 200 20
Apr 25 17:25:46 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]: 128.182.168.116 - -
[25/Apr/2019 17:25:46] "GET /info/ HTTP/1.1" 200 20
# cat image-proxy.log
(MainThread) INFO 2019-04-25 16:25:55,755 image_proxy:43:root:(main) Server shut down,
exiting
(MainThread) INFO 2019-04-25 16:36:24,874 server:45:server:(start) Starting (pid=56989,
version=1.5.1)
(MainThread) INFO 2019-04-25 16:36:24,879 image_proxy:34:root:(main) Server started,
successfully notified systemd
On Tue, Apr 23, 2019 at 4:51 PM Edward Berger <edwberger(a)gmail.com> wrote:
>
> Previously I had issues with the upgrades to 4.3.3 failing because of
"stale" image transfer data, so I removed it from the database using the info
given here on the mailing list and was able to complete the oVirt node and engine
upgrades.
>
> Now I have a new problem. I can't upload a disk image anymore, which used to
work.
> "test connection" returns success.
>
> In the dashboard Storage/Disks view, it starts and then stops with "paused by
system"
Hi!
Are you trying to use a 3rd-party CA for https access [4][5]?
Please check this thread:
[1]
https://lists.ovirt.org/archives/list/users@ovirt.org/thread/NZBGRCKW6WA4...
Part of this was already fixed:
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1687301
And another part is still pending, although you should easily "fix"
manually (editing imageio-proxy.conf):
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1637809
If so:
1. Please try again following the entire procedure for using 3rd-party ca.
2. Please also edit
/etc/ovirt-imageio-proxy/ovirt-imageio-proxy.conf as explained in
above bug [3], and then restart imageio-proxy and the engine.
If it still fails, please check/share all relevant differences between
a vanilla setup and yours, including all the changes you did as part
of following the 3rd-party-ca procedure, custom/different config
files, changes you did to the system-wide truststore, etc.
[4] https://www.ovirt.org/documentation/admin-guide/appe-oVirt_and_SSL.html
[5]
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/...
>
> I tried pointing it at another node, same problem. I tried restarting services
ovirt-imageio-proxy on engine and ovirt-imageio-daemon on the node, same failure.
>
> in the dashboard events when trying to upload disk I get this error
> Transfer was stopped by system. Reason: failed to add image ticket to
ovirt-imageio-proxy.
>
> or when trying to resume transfer "paused by system" giving it the local
path again.
> Transfer was stopped by system. Reason: failure in transfer image ticket
renewal.
>
> I'm not sure what logs I should be looking deeper into..
The only relevant ones are the engine's logs and imageio-proxy's, and
sadly they didn't help me enough to debug this when I did.
>
> Here's what the database says
> engine=# select * from image_transfers;
> -[ RECORD 1 ]-------------+-------------------------------------
> command_id | 59dce2b1-f8ba-44dd-9df9-c6e39773a3f9
> command_type | 1024
> phase | 4
> last_updated | 2019-04-23 12:22:25.098-04
> message | Uploading from byte 0
> vds_id |
> disk_id | 6693d5ac-d3eb-43d7-9abe-97c5197efc23
> imaged_ticket_id |
> proxy_uri |
> signed_ticket |
> bytes_sent | 0
> bytes_total | 1996488704
> type | 2
> active | f
> daemon_uri |
> client_inactivity_timeout | 60
> -[ RECORD 2 ]-------------+-------------------------------------
> command_id | 16898219-be5b-4826-8f20-3355fa47272a
> command_type | 1024
> phase | 4
> last_updated | 2019-04-23 16:15:34.591-04
> message | Uploading from byte 0
> vds_id |
> disk_id | 3b8e3053-bfe4-49d3-abd2-5452b1674400
> imaged_ticket_id |
> proxy_uri |
> signed_ticket |
> bytes_sent | 0
> bytes_total | 1998585856
> type | 2
> active | f
> daemon_uri |
> client_inactivity_timeout | 60
>
> I really need the image uploading to work. Any suggestions on what to do next?
I am ignoring for now your debug attempts looking at logs on the hosts
etc., we can go back to that if above is not enough.
Good luck and best regards,
--
Didi
2035
days inactive
2041
days old
2 comments
2 participants
participants (2)
-
Edward Berger -
Yedidyah Bar David