11:55 a.m.
On Tue, Aug 18, 2020 at 4:06 PM Michael Jones <mj(a)mikejonesey.co.uk> wrote:
I have image upload/download working on some older ovirt servers
where
it still has the split daemon/proxy...
on one newer one this feature is not working;
software in use:
CentOS 8
ovirt-engine-4.4.1.10-1.el8.noarch
ovirt-imageio-client-2.0.9-1.el8.x86_64
ovirt-imageio-daemon-2.0.9-1.el8.x86_64
ovirt-imageio-common-2.0.9-1.el8.x86_64
Both host and engine running these versions?
the ui test button allowed me to work out 50-vdsm.conf was setting
the
wrong remote port... (was 54322, changed to 54323)
50-vdsm.conf is cannot break the connection test, since it should be
installed only
on the hosts. On engine you have 50-engine.conf.
The correct configuration for engine is:
[tls]
enable = true
key_file = /etc/pki/ovirt-engine/keys/apache.key.nopass
cert_file = /etc/pki/ovirt-engine/certs/apache.cer
ca_file = /etc/pki/ovirt-engine/apache-ca.pem
[remote]
port = 54323
[local]
enable = false
[control]
transport = tcp
port = 54324
The correct settings for the hosts are:
[tls]
enable = true
key_file = /etc/pki/vdsm/keys/vdsmkey.pem
cert_file = /etc/pki/vdsm/certs/vdsmcert.pem
ca_file = /etc/pki/vdsm/certs/cacert.pem
[remote]
port = 54322
These files belongs to engine and vdsm and you should not change them.
Your changes will be overwirtten on the next upgrade.
The top of the file explain how to change the configuration.
updated remote with;
[remote]
host = 0.0.0.0
port = 54323
the test now passes, but on upload or download it still fails.
Do you have 50-vdsm.conf on the engine host?!
It sounds like you have all-in-one configuration when engine host is also
the single host. This configuration is not supported for 5 years or so.
Or you installed vdsm by mistake on engine host, in this case you will have
both 50-vdsm.conf and 50-engine.conf, and because "vdsm" sorts after
"engine"
its configuration will win.
Next i changed the control to be unix socket instead of tcp port
54324
(vdsm was giving an error: Image daemon is unsupported);
I looked up the error line in the vdsm code, and found it was looking
for unix socket: DAEMON_SOCK=/run/ovirt-imageio/sock
switching to sock seemed to resolve all errors in the vdsm log;
Expected, using tcp for the host is not supported.
---
content of the imageio log;
imagieo log on engine or host?
no errors as far as i can see:
2020-08-18 12:49:56,109 INFO (MainThread) [server] Starting
(pid=2696562, version=2.0.9)
2020-08-18 12:49:56,109 DEBUG (MainThread) [services] Creating
remote.service on port 54323
2020-08-18 12:49:56,111 DEBUG (MainThread) [http] Prefer IPv4: False
2020-08-18 12:49:56,111 DEBUG (MainThread) [http] Available network
interfaces: [(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>,
6, '', ('0.0.0.0', 54323))]
2020-08-18 12:49:56,111 DEBUG (MainThread) [http] Creating server
socket with family=AddressFamily.AF_INET and type=SocketKind.SOCK_STREAM
2020-08-18 12:49:56,111 DEBUG (MainThread) [services] Securing server
(cafile=/etc/pki/vdsm/certs/cacert.pem,
certfile=/etc/pki/vdsm/certs/vdsmcert.pem,
keyfile=/etc/pki/vdsm/keys/vdsmkey.pem)
So this is log from the host
2020-08-18 12:49:56,113 INFO (MainThread) [services]
remote.service
listening on ('0.0.0.0', 54323)
This port is wrong, you will not be able to tansfer anything since
engine assumes
port 54322.
2020-08-18 12:49:56,113 DEBUG (MainThread) [services] Creating
local.service on socket '\x00/org/ovirt/imageio'
2020-08-18 12:49:56,113 INFO (MainThread) [services] local.service
listening on '\x00/org/ovirt/imageio'
2020-08-18 12:49:56,113 DEBUG (MainThread) [services] Creating
control.service on socket '/run/ovirt-imageio/sock'
2020-08-18 12:49:56,113 DEBUG (MainThread) [uhttp] Removing socket
'/run/ovirt-imageio/sock'
2020-08-18 12:49:56,113 INFO (MainThread) [services] control.service
listening on '/run/ovirt-imageio/sock'
2020-08-18 12:49:56,115 DEBUG (MainThread) [server] Changing ownership
of /run/ovirt-imageio to 988:984
2020-08-18 12:49:56,115 DEBUG (MainThread) [server] Changing ownership
of /var/log/ovirt-imageio/daemon.log to 988:984
2020-08-18 12:49:56,115 DEBUG (MainThread) [server] Dropping root
privileges, running as 988:984
2020-08-18 12:49:56,116 DEBUG (MainThread) [services] Starting
remote.service
2020-08-18 12:49:56,116 DEBUG (remote.service) [services]
remote.service started
2020-08-18 12:49:56,116 DEBUG (MainThread) [services] Starting
local.service
2020-08-18 12:49:56,117 DEBUG (local.service) [services] local.service
started
2020-08-18 12:49:56,117 DEBUG (MainThread) [services] Starting
control.service
2020-08-18 12:49:56,117 DEBUG (control.service) [services]
control.service started
2020-08-18 12:49:56,117 INFO (MainThread) [server] Ready for requests
2020-08-18 12:51:34,602 INFO (Thread-1) [http] OPEN client=local
2020-08-18 12:51:34,603 INFO (Thread-1) [tickets] [local] ADD
ticket={'dirty': False, 'ops': ['read'], 'filename':
'debian-10.3.0-amd64-netinst.iso.raw', 'size': 351272960,
'sparse':
True, 'transfer_id': 'ec7b329e-3ef1-448d-ac73-1e5e1b4c0ef9',
'uuid':
'17db8f9f-3280-4e7c-8fbf-e5326e1f98be', 'timeout': 300, 'url':
'file:///rhev/data-center/mnt/_var_local_ovirt_local__data/7ee98037-126b-4c79-b921-a9ff4272fa91/images/315af697-a064-4c2c-bdf6-8144a8218526/8e709a79-4da0-4d91-803e-d6ec12ec84c3'}
Looks good, this host is using local storage.
2020-08-18 12:51:34,603 DEBUG (Thread-1) [http] Empty request
line,
client disconnected
2020-08-18 12:51:34,603 INFO (Thread-1) [http] CLOSE client=local
[connection 1 ops, 0.000842 s] [dispatch 1 ops, 0.000305 s]
---
vdsm log for transfer id: ec7b329e-3ef1-448d-ac73-1e5e1b4c0ef9
no errors as far as i can see...
2020-08-18 12:51:34,602+0000 INFO (jsonrpc/5) [vdsm.api] START
add_image_ticket(ticket={'dirty': False, 'ops': ['read'],
'filename':
'debian-10.3.0-amd64-netinst.iso.raw', 'size': 351272960,
'sparse':
True, 'transfer_id': 'ec7b329e-3ef1-448d-ac73-1e5e1b4c0ef9',
'uuid':
'17db8f9f-3280-4e7c-8fbf-e5326e1f98be', 'timeout': 300, 'url':
'file:///rhev/data-center/mnt/_var_local_ovirt_local__data/7ee98037-126b-4c79-b921-a9ff4272fa91/images/315af697-a064-4c2c-bdf6-8144a8218526/8e709a79-4da0-4d91-803e-d6ec12ec84c3'})
from=::ffff:37.187.146.167,58882,
flow_id=19204d8c-ffe6-4c45-81fe-96e07097d519,
task_id=441bf5ea-1e93-4b5b-aeae-f0e52b43cf62 (api:48)
2020-08-18 12:51:34,603+0000 INFO (jsonrpc/5) [vdsm.api] FINISH
add_image_ticket return=None from=::ffff:37.187.146.167,58882,
flow_id=19204d8c-ffe6-4c45-81fe-96e07097d519,
task_id=441bf5ea-1e93-4b5b-aeae-f0e52b43cf62 (api:54)
2020-08-18 12:51:34,603+0000 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer]
RPC call Host.add_image_ticket succeeded in 0.00 seconds (__init__:312)
---
engine log:
urn [status=Status [code=0, message=Done]], log id: 7ac2f68b
2020-08-18 12:51:34,604Z INFO
[org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand]
(default task-6) [19204d8c-ffe6-4c45-81fe-96e07097d519] Started transfer
session with ticket id 17db8f9f-328
0-4e7c-8fbf-e5326e1f98be, timeout 300 seconds
2020-08-18 12:51:34,604Z INFO
[org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand]
(default task-6) [19204d8c-ffe6-4c45-81fe-96e07097d519] Adding image
ticket to ovirt-imageio-proxy, id 17db8
f9f-3280-4e7c-8fbf-e5326e1f98be
2020-08-18 12:51:34,604Z ERROR
[org.ovirt.engine.core.bll.storage.disk.image.TransferDiskImageCommand]
(default task-6) [19204d8c-ffe6-4c45-81fe-96e07097d519] Failed to add
image ticket to ovirt-imageio-proxy: C
onnection refused (Connection refused): java.lang.RuntimeException:
Connection refused (Connection refused)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.ImageioClient.getConnection(ImageioClient.java:114)
at
deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.disk.image.ImageioClient.putTicket(ImageioClient.java:65)
is there a way for me to validate where engine thinks the proxy is? the
test button passed after changing remote port to 54323...
The test button tests that the UI can connect ot the ovirt-imageio for
transfering data,
but engine connects to localhost:54324 for (control port on engine
host) for adding
a ticket.
Sounds like you try to use unsupported configuration, it will not work.
If we want to make this work, engine should know that engine host is a
hypervisor
and disable usage of the proxy, since no proxy is needed when engine host is
the hypervisor. But we don't have such configuration.
Nir