12:32 a.m.
Today, I updated engine to the latest Version 4.3.3.6-1.el7.
Trying to upload an install iso to the hosted storage domain gives same
disk paused by system issue.
Apr 25, 2019, 5:26:18 PMTransfer was stopped by system. Reason: failed to
add image ticket to ovirt-imageio-proxy.
on the node which was running ovirt-image
/var/log/ovirt-imageio-daemon contains:
2019-04-25 17:08:51,757 INFO (Thread-2) [http] OPEN client=local
2019-04-25 17:08:51,758 INFO (Thread-2) [tickets] [local] ADD
ticket={u'uuid': u'6971e886-5339-4df1-88df-445e51fba1c8', u'ops':
[u'write'], u'url': u'fi
le:///rhev/data-center/mnt/glusterSD/10.200.0.131:
_engine/0ec355b1-e665-4fdf-87c4-87e1d636c76d/images/4121f1fb-c508-47bc-9acb-a10da9461358/04658323-b267-4f
66-96f1-6c5848c3d2be', u'sparse': True, u'timeout': 300,
u'transfer_id': u'e85ce170-03e6-480d-bbc5-8551c266cc97', u'size':
1045426176}
2019-04-25 17:08:51,759 INFO (Thread-2) [http] CLOSE client=local
[connection=0.001420/1, dispatch=0.000634/1]
going to that directory above shows
# ls -l
total 1025
-rw-rw----. 1 vdsm kvm 1045426176 Apr 25 17:08
04658323-b267-4f66-96f1-6c5848c3d2be
-rw-rw----. 1 vdsm kvm 1048576 Apr 25 17:08
04658323-b267-4f66-96f1-6c5848c3d2be.lease
-rw-r--r--. 1 vdsm kvm 345 Apr 25 17:08
04658323-b267-4f66-96f1-6c5848c3d2be.meta
image is showing same size as the iso file.
On engine
# systemctl status -l ovirt-imageio-proxy.service
● ovirt-imageio-proxy.service - oVirt ImageIO Proxy
Loaded: loaded (/usr/lib/systemd/system/ovirt-imageio-proxy.service;
enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-04-25 16:36:24 EDT; 51min ago
Main PID: 56989 (ovirt-imageio-p)
CGroup: /system.slice/ovirt-imageio-proxy.service
└─56989 /usr/bin/python2 /usr/bin/ovirt-imageio-proxy
Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Starting oVirt ImageIO
Proxy...
Apr 25 16:36:24 dispatch.yard.psc.edu systemd[1]: Started oVirt ImageIO
Proxy.
Apr 25 17:08:30 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]:
128.182.168.116 - - [25/Apr/2019 17:08:30] "GET /info/ HTTP/1.1" 200 20
Apr 25 17:25:46 dispatch.yard.psc.edu ovirt-imageio-proxy[56989]:
128.182.168.116 - - [25/Apr/2019 17:25:46] "GET /info/ HTTP/1.1" 200 20
# cat image-proxy.log
(MainThread) INFO 2019-04-25 16:25:55,755 image_proxy:43:root:(main) Server
shut down, exiting
(MainThread) INFO 2019-04-25 16:36:24,874 server:45:server:(start) Starting
(pid=56989, version=1.5.1)
(MainThread) INFO 2019-04-25 16:36:24,879 image_proxy:34:root:(main) Server
started, successfully notified systemd
On Tue, Apr 23, 2019 at 4:51 PM Edward Berger <edwberger(a)gmail.com> wrote:
Previously I had issues with the upgrades to 4.3.3 failing because
of
"stale" image transfer data, so I removed it from the database using the
info given here on the mailing list and was able to complete the oVirt node
and engine upgrades.
Now I have a new problem. I can't upload a disk image anymore, which used
to work.
"test connection" returns success.
In the dashboard Storage/Disks view, it starts and then stops with "paused
by system"
I tried pointing it at another node, same problem. I tried restarting
services ovirt-imageio-proxy on engine and ovirt-imageio-daemon on the
node, same failure.
in the dashboard events when trying to upload disk I get this error
Transfer was stopped by system. Reason: failed to add image ticket
to ovirt-imageio-proxy.
or when trying to resume transfer "paused by system" giving it the local
path again.
Transfer was stopped by system. Reason: failure in transfer image
ticket renewal.
I'm not sure what logs I should be looking deeper into..
Here's what the database says
engine=# select * from image_transfers;
-[ RECORD 1 ]-------------+-------------------------------------
command_id | 59dce2b1-f8ba-44dd-9df9-c6e39773a3f9
command_type | 1024
phase | 4
last_updated | 2019-04-23 12:22:25.098-04
message | Uploading from byte 0
vds_id |
disk_id | 6693d5ac-d3eb-43d7-9abe-97c5197efc23
imaged_ticket_id |
proxy_uri |
signed_ticket |
bytes_sent | 0
bytes_total | 1996488704
type | 2
active | f
daemon_uri |
client_inactivity_timeout | 60
-[ RECORD 2 ]-------------+-------------------------------------
command_id | 16898219-be5b-4826-8f20-3355fa47272a
command_type | 1024
phase | 4
last_updated | 2019-04-23 16:15:34.591-04
message | Uploading from byte 0
vds_id |
disk_id | 3b8e3053-bfe4-49d3-abd2-5452b1674400
imaged_ticket_id |
proxy_uri |
signed_ticket |
bytes_sent | 0
bytes_total | 1998585856
type | 2
active | f
daemon_uri |
client_inactivity_timeout | 60
I really need the image uploading to work. Any suggestions on what to do
next?