Nir

Thanks for the info and explanation.

Don

On Mon, Apr 26, 2021 at 4:27 PM Nir Soffer <nsoffer@redhat.com> wrote:

On Mon, Apr 26, 2021 at 11:56 PM Don Dupuis <dondster@gmail.com> wrote:
>
> Nir
> After looking at the debug output of daemon.log, it looks like the code is expecting ipv6 to be enabled and I have explicitly disable ipv6. When I enable it and reboot, it now WORKS!!.

Yes, we assume that ipv6 is enabled on the engine and hypervisors.
You will have the same issue with the hypervisors, since vdsm is also
listening on the same address ("::").

> Where I work we don't use ipv6, is there another option to set so that this works on ipv4 only setups?

We use the special "::" address so we listen to both ipv4 and ipv6.
This works for
people who use ipv4 or ipv6.

If you must disable ipv6 on the engine host, you can try to listen to
an ipv4 address:

# cat /etc/ovirt-imageio/conf.d/99-local.conf
[remote]
host = 0.0.0.0

Similar change is needed for vdsm:

# cat /etc/vdsm/vdsm.conf.d/99-local.conf
[addresses]
management_ip = 0.0.0.0

I'm not sure that disabling ipv6 is tested, you may find other issues
on setup with
ipv6 disabled.

Ales may have more insight on disabling ipv6.

Nir

>
> Thanks
> Don
>
> On Mon, Apr 26, 2021 at 3:32 PM Don Dupuis <dondster@gmail.com> wrote:
>>
>> 2021-04-26 15:29:50,728 INFO    (MainThread) [server] Starting (hostname=rvsmanager2 pid=13203, version=2.1.1)
>> 2021-04-26 15:29:50,729 DEBUG   (MainThread) [services] Creating remote.service on port 54323
>> 2021-04-26 15:29:50,730 DEBUG   (MainThread) [http] Prefer IPv4: False
>> 2021-04-26 15:29:50,730 DEBUG   (MainThread) [http] Available network interfaces: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 54323, 0, 0))]
>> 2021-04-26 15:29:50,730 DEBUG   (MainThread) [http] Creating server socket with family=AddressFamily.AF_INET6 and type=SocketKind.SOCK_STREAM
>> 2021-04-26 15:29:50,732 ERROR   (MainThread) [server] Server failed
>> Traceback (most recent call last):
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 46, in main
>>     server = Server(cfg)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 110, in __init__
>>     self.remote_service = services.RemoteService(self.config, self.auth)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/services.py", line 73, in __init__
>>     self._server = http.Server((config.remote.host, port), http.Connection)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 91, in __init__
>>     self.create_socket(prefer_ipv4)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 123, in create_socket
>>     self.socket = socket.socket(self.address_family, self.socket_type)
>>   File "/usr/lib64/python3.6/socket.py", line 144, in __init__
>>     _socket.socket.__init__(self, family, type, proto, fileno)
>> OSError: [Errno 97] Address family not supported by protocol
>> 2021-04-26 15:29:50,919 INFO    (MainThread) [server] Starting (hostname=rvsmanager2 pid=13208, version=2.1.1)
>> 2021-04-26 15:29:50,919 DEBUG   (MainThread) [services] Creating remote.service on port 54323
>> 2021-04-26 15:29:50,920 DEBUG   (MainThread) [http] Prefer IPv4: False
>> 2021-04-26 15:29:50,920 DEBUG   (MainThread) [http] Available network interfaces: [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::', 54323, 0, 0))]
>> 2021-04-26 15:29:50,920 DEBUG   (MainThread) [http] Creating server socket with family=AddressFamily.AF_INET6 and type=SocketKind.SOCK_STREAM
>> 2021-04-26 15:29:50,922 ERROR   (MainThread) [server] Server failed
>> Traceback (most recent call last):
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 46, in main
>>     server = Server(cfg)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 110, in __init__
>>     self.remote_service = services.RemoteService(self.config, self.auth)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/services.py", line 73, in __init__
>>     self._server = http.Server((config.remote.host, port), http.Connection)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 91, in __init__
>>     self.create_socket(prefer_ipv4)
>>   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 123, in create_socket
>>     self.socket = socket.socket(self.address_family, self.socket_type)
>>   File "/usr/lib64/python3.6/socket.py", line 144, in __init__
>>     _socket.socket.__init__(self, family, type, proto, fileno)
>> OSError: [Errno 97] Address family not supported by protocol
>>
>> # ovirt-imageio --show-config
>> {
>>     "backend_file": {
>>         "buffer_size": 8388608
>>     },
>>     "backend_http": {
>>         "buffer_size": 8388608,
>>         "ca_file": "/etc/pki/ovirt-engine/ca.pem"
>>     },
>>     "backend_nbd": {
>>         "buffer_size": 8388608
>>     },
>>     "control": {
>>         "port": 54324,
>>         "prefer_ipv4": true,
>>         "remove_timeout": 60,
>>         "socket": "/run/ovirt-imageio/sock",
>>         "transport": "tcp"
>>     },
>>     "daemon": {
>>         "drop_privileges": true,
>>         "group_name": "ovirtimg",
>>         "max_connections": 8,
>>         "poll_interval": 1.0,
>>         "run_dir": "/run/ovirt-imageio",
>>         "user_name": "ovirtimg"
>>     },
>>     "formatter_long": {
>>         "format": "%(asctime)s %(levelname)-7s (%(threadName)s) [%(name)s] %(message)s"
>>     },
>>     "formatters": {
>>         "keys": "long"
>>     },
>>     "handler_logfile": {
>>         "args": "(\"/var/log/ovirt-imageio/daemon.log\",)",
>>         "formatter": "long",
>>         "class": "logging.handlers.RotatingFileHandler",
>>         "kwargs": "{\"maxBytes\": 20971520, \"backupCount\": 10}",
>>         "level": "DEBUG"
>>     },
>>     "handler_stderr": {
>>         "formatter": "long",
>>         "class": "logging.StreamHandler",
>>         "level": "DEBUG"
>>     },
>>     "handlers": {
>>         "keys": "logfile"
>>     },
>>     "local": {
>>         "enable": false,
>>         "socket": "\u0000/org/ovirt/imageio"
>>     },
>>     "logger_root": {
>>         "handlers": "logfile",
>>         "level": "DEBUG",
>>         "propagate": 0
>>     },
>>     "loggers": {
>>         "keys": "root"
>>     },
>>     "profile": {
>>         "filename": "/run/ovirt-imageio/profile"
>>     },
>>     "remote": {
>>         "host": "::",
>>         "port": 54323
>>     },
>>     "tls": {
>>         "ca_file": "/etc/pki/ovirt-engine/apache-ca.pem",
>>         "cert_file": "/etc/pki/ovirt-engine/certs/apache.cer",
>>         "enable": true,
>>         "enable_tls1_1": false,
>>         "key_file": "/etc/pki/ovirt-engine/keys/apache.key.nopass"
>>     }
>> }
>>
>>
>> On Mon, Apr 26, 2021 at 3:22 PM Nir Soffer <nsoffer@redhat.com> wrote:
>>>
>>> On Mon, Apr 26, 2021 at 10:58 PM Don Dupuis <dondster@gmail.com> wrote:
>>> >
>>> > Nir
>>> > It just repeats. Here you go
>>>
>>> Thanks, it seems that we need more work on logging.
>>>
>>> Can you try to change the log level to DEBUG and start the
>>> ovirt-imageio service?
>>>
>>> Add this file:
>>>
>>> $ cat /etc/ovirt-imageio/conf.d/99-local.conf
>>> [logger_root]
>>> level = DEBUG
>>>
>>> And start the service:
>>>
>>> $ systemctl start ovirt-imageio
>>>
>>> I hope we will have more info in the log explaining this issue.
>>>
>>> And to make sure we have correct configuration, please share output of:
>>>
>>> $ ovirt-imageio --show-config
>>>
>>> > 2021-04-26 13:50:49,600 INFO    (MainThread) [server] Starting (hostname=manager2 pid=60513, version=2.1.1)
>>> > 2021-04-26 13:50:49,609 ERROR   (MainThread) [server] Server failed
>>> > Traceback (most recent call last):
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 46, in main
>>> >     server = Server(cfg)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 110, in __init__
>>> >     self.remote_service = services.RemoteService(self.config, self.auth)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/services.py", line 73, in __init__
>>> >     self._server = http.Server((config.remote.host, port), http.Connection)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 91, in __init__
>>> >     self.create_socket(prefer_ipv4)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 123, in create_socket
>>> >     self.socket = socket.socket(self.address_family, self.socket_type)
>>> >   File "/usr/lib64/python3.6/socket.py", line 144, in __init__
>>> >     _socket.socket.__init__(self, family, type, proto, fileno)
>>> > OSError: [Errno 97] Address family not supported by protocol
>>> > 2021-04-26 13:50:49,954 INFO    (MainThread) [server] Starting (hostname=manager2 pid=60530, version=2.1.1)
>>> > 2021-04-26 13:50:49,967 ERROR   (MainThread) [server] Server failed
>>> > Traceback (most recent call last):
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 46, in main
>>> >     server = Server(cfg)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 110, in __init__
>>> >     self.remote_service = services.RemoteService(self.config, self.auth)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/services.py", line 73, in __init__
>>> >     self._server = http.Server((config.remote.host, port), http.Connection)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 91, in __init__
>>> >     self.create_socket(prefer_ipv4)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 123, in create_socket
>>> >     self.socket = socket.socket(self.address_family, self.socket_type)
>>> >   File "/usr/lib64/python3.6/socket.py", line 144, in __init__
>>> >     _socket.socket.__init__(self, family, type, proto, fileno)
>>> > OSError: [Errno 97] Address family not supported by protocol
>>> > 2021-04-26 13:50:50,199 INFO    (MainThread) [server] Starting (hostname=manager2 pid=60567, version=2.1.1)
>>> > 2021-04-26 13:50:50,203 ERROR   (MainThread) [server] Server failed
>>> > Traceback (most recent call last):
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 46, in main
>>> >     server = Server(cfg)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 110, in __init__
>>> >     self.remote_service = services.RemoteService(self.config, self.auth)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/services.py", line 73, in __init__
>>> >     self._server = http.Server((config.remote.host, port), http.Connection)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 91, in __init__
>>> >     self.create_socket(prefer_ipv4)
>>> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 123, in create_socket
>>> >     self.socket = socket.socket(self.address_family, self.socket_type)
>>> >   File "/usr/lib64/python3.6/socket.py", line 144, in __init__
>>> >     _socket.socket.__init__(self, family, type, proto, fileno)
>>> > OSError: [Errno 97] Address family not supported by protocol
>>> >
>>> > Don
>>> >
>>> > On Mon, Apr 26, 2021 at 2:34 PM Nir Soffer <nsoffer@redhat.com> wrote:
>>> >>
>>> >> On Mon, Apr 26, 2021 at 7:46 PM Don Dupuis <dondster@gmail.com> wrote:
>>> >> >
>>> >> > I am installing Ovirt 4.4.6.5-1 engine on CentOS Stream 8 on a dedicated vm. Towards the end of the engine-setup session I have a failure:
>>> >> >
>>> >> > [ INFO  ] Stage: Transaction setup
>>> >> > [ INFO  ] Stopping engine service
>>> >> > [ INFO  ] Stopping ovirt-fence-kdump-listener service
>>> >> > [ INFO  ] Stopping dwh service
>>> >> > [ INFO  ] Stopping vmconsole-proxy service
>>> >> > [ INFO  ] Stopping websocket-proxy service
>>> >> > [ INFO  ] Stage: Misc configuration (early)
>>> >> > [ INFO  ] Stage: Package installation
>>> >> > [ INFO  ] Stage: Misc configuration
>>> >> > [ INFO  ] Upgrading CA
>>> >> > [ INFO  ] Initializing PostgreSQL
>>> >> > [ INFO  ] Creating PostgreSQL 'engine' database
>>> >> > [ INFO  ] Configuring PostgreSQL
>>> >> > [ INFO  ] Creating PostgreSQL 'ovirt_engine_history' database
>>> >> > [ INFO  ] Configuring PostgreSQL
>>> >> > [ INFO  ] Creating CA: /etc/pki/ovirt-engine/ca.pem
>>> >> > [ INFO  ] Creating CA: /etc/pki/ovirt-engine/qemu-ca.pem
>>> >> > [ INFO  ] Updating OVN SSL configuration
>>> >> > [ INFO  ] Updating OVN timeout configuration
>>> >> > [ INFO  ] Creating/refreshing DWH database schema
>>> >> > [ INFO  ] Setting up ovirt-vmconsole proxy helper PKI artifacts
>>> >> > [ INFO  ] Setting up ovirt-vmconsole SSH PKI artifacts
>>> >> > [ INFO  ] Configuring WebSocket Proxy
>>> >> > [ INFO  ] Creating/refreshing Engine database schema
>>> >> > [ INFO  ] Creating a user for Grafana
>>> >> > [ INFO  ] Creating/refreshing Engine 'internal' domain database schema
>>> >> > [ INFO  ] Creating default mac pool range
>>> >> > [ INFO  ] Adding default OVN provider to database
>>> >> > [ INFO  ] Adding OVN provider secret to database
>>> >> > [ INFO  ] Setting a password for internal user admin
>>> >> > [ INFO  ] Install selinux module /usr/share/ovirt-engine/selinux/ansible-runner-service.cil
>>> >> > [ INFO  ] Generating post install configuration file '/etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf'
>>> >> > [ INFO  ] Stage: Transaction commit
>>> >> > [ INFO  ] Stage: Closing up
>>> >> > [ INFO  ] Starting engine service
>>> >> > [ INFO  ] Starting dwh service
>>> >> > [ INFO  ] Starting Grafana service
>>> >> > [ ERROR ] Failed to execute stage 'Closing up': Failed to start service 'ovirt-imageio'
>>> >> > [ INFO  ] Stage: Clean up
>>> >> >           Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20210426111225-g31wa5.log
>>> >> > [ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20210426111501-setup.conf'
>>> >> > [ INFO  ] Stage: Pre-termination
>>> >> > [ INFO  ] Stage: Termination
>>> >> > [ ERROR ] Execution of setup failed
>>> >> >
>>> >> > Below is output of /var/log/ovirt-imageio/daemon.log:
>>> >> >
>>> >> > 2021-04-26 11:15:01,481 ERROR   (MainThread) [server] Server failed
>>> >> > Traceback (most recent call last):
>>> >> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 46, in main
>>> >> >     server = Server(cfg)
>>> >> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/server.py", line 110, in __init__
>>> >> >     self.remote_service = services.RemoteService(self.config, self.auth)
>>> >> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/services.py", line 73, in __init__
>>> >> >     self._server = http.Server((config.remote.host, port), http.Connection)
>>> >> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 91, in __init__
>>> >> >     self.create_socket(prefer_ipv4)
>>> >> >   File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/http.py", line 123, in create_socket
>>> >> >     self.socket = socket.socket(self.address_family, self.socket_type)
>>> >> >   File "/usr/lib64/python3.6/socket.py", line 144, in __init__
>>> >> >     _socket.socket.__init__(self, family, type, proto, fileno)
>>> >> > OSError: [Errno 97] Address family not supported by protocol
>>> >> >
>>> >> > I am not seeing errors in /var/log/ovirt-engine/setup/setup.log. What other information do you need that is revelant to this issue?
>>> >>
>>> >> Can you share a complete log?
>>> >>
>>> >> /var/log/ovirt-imageio/daemon.log
>>> >>
>>> >> Nir
>>> >>
>>>