Shani,

supervdsm failing too. 

[root@ovirt1 vdsm]# systemctl status supervdsmd
● supervdsmd.service - Auxiliary vdsm service for running helper functions as root
   Loaded: loaded (/usr/lib/systemd/system/supervdsmd.service; static; vendor preset: enabled)
   Active: failed (Result: start-limit) since Tue 2019-06-11 16:18:16 MSK; 5s ago
  Process: 176025 ExecStart=/usr/share/vdsm/daemonAdapter /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock (code=exited, status=1/FAILURE)
 Main PID: 176025 (code=exited, status=1/FAILURE)

Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: Unit supervdsmd.service entered failed state.
Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: supervdsmd.service failed.
Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: supervdsmd.service holdoff time over, scheduling restart.
Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: Stopped Auxiliary vdsm service for running helper functions as root.
Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: start request repeated too quickly for supervdsmd.service
Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: Failed to start Auxiliary vdsm service for running helper functions as root.
Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: Unit supervdsmd.service entered failed state.
Jun 11 16:18:16 ovirt1.telia.ru systemd[1]: supervdsmd.service failed.


supervdsm.log is full of messages like 
logfile::DEBUG::2019-06-11 16:18:46,379::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 139896277387008)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7f3c243aa3d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:19:04,401::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 140683302770432)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7ff36298f3d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:19:06,289::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 140242149250816)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7f8cabcc73d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:19:17,535::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 139830764660480)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7f2ce35dc3d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:19:21,528::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 140355511899904)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7fa710bd33d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:19:24,541::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 139879186089728)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7f38298223d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:19:42,543::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 140400833873664)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7fb19e23a3d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:19:57,442::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 140366930253568)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7fa9b95373d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:20:18,539::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 139647603230464)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7f023e1823d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:20:32,041::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 140076440143616)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7f6616c423d0>>, args=(), kwargs={})
logfile::DEBUG::2019-06-11 16:20:41,051::concurrent::193::root::(run) START thread <Thread(logfile, started daemon 140054782310144)> (func=<bound method ThreadedHandler._run of <vdsm.common.logutils.ThreadedHandler object at 0x7f610bdbd3d0>>, args=(), kwargs={})

Regards,
Artem


On Tue, Jun 11, 2019 at 3:59 PM Shani Leviim <sleviim@redhat.com> wrote:
+Dan Kenigsberg

Hi Artem,
Thanks for the log.

It seems that this error message appears quite a lot:
2019-06-11 12:10:35,283+0300 ERROR (MainThread) [root] Panic: Connect to supervdsm service failed: [Errno 2] No such file or directory (panic:29)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 86, in _connect
    self._manager.connect, Exception, timeout=60, tries=3)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 58, in retry
    return func()
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 500, in connect
    conn = Client(self._address, authkey=self._authkey)
  File "/usr/lib64/python2.7/multiprocessing/connection.py", line 173, in Client
    c = SocketClient(address)
  File "/usr/lib64/python2.7/multiprocessing/connection.py", line 308, in SocketClient
    s.connect(address)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory

Can you please verify that the 'supervdsmd.service' is running?

Regards,
Shani Leviim


On Tue, Jun 11, 2019 at 3:04 PM Artem Tambovskiy <artem.tambovskiy@gmail.com> wrote:
Hi Shani,

yes, you are right - I can do ssh form aby to any hosts in the cluster. vdsm.log attached.
I have tried to restart vdsm manually and even done a host restart several times with no success.
Host activation fails all the time ...

Thank you in advance for your help!
Regard,
Artem

On Tue, Jun 11, 2019 at 10:51 AM Shani Leviim <sleviim@redhat.com> wrote:
Hi Artem,
According to oVirt documentation [1], hosts on the same cluster should be reachable from one to each other.

Can you please share your vdsm log?
I suppose you do manage to ssh that inactive host (correct me if I'm wrong).
While getting the vdsm log, maybe try to restart the network and vdsmd services on the host.

Another thing you can try on the UI is putting the host on maintenance and then activate it.

[1] https://www.ovirt.org/documentation/admin-guide/chap-Clusters.html#introduction-to-clusters

Regards,
Shani Leviim


On Mon, Jun 10, 2019 at 4:42 PM Artem Tambovskiy <artem.tambovskiy@gmail.com> wrote:
Hello,

May I ask you for and advise?
I'm running a small oVirt cluster and couple of months ago I decided to do an upgrade from oVirt 4.2.8 to 4.3 and having an issues since that time. I can only guess what I did wrong - probably one of the problems that I haven't switched the cluster from iptables to firewalld. But this is just my guess. 

The problem is that I have upgraded the engine and one host, and then I done an upgrade of second host I can't bring it to active state. Looks like VDSM can't detect the network and fails to start. I even tried to reinstall the hosts from UI (I have seen that the packages being installed) but again, VDSM doesn't startup at the end and reinstallation fails.

Looking at hosts process list I see  script wait_for_ipv4s  hanging forever. 

vdsm       8603      1  6 16:26 ?        00:00:00 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent
root       8630      1  0 16:26 ?        00:00:00 /bin/sh /usr/libexec/vdsm/vdsmd_init_common.sh --pre-start
root       8645   8630  6 16:26 ?        00:00:00 /usr/bin/python2 /usr/libexec/vdsm/wait_for_ipv4s
root       8688      1 30 16:27 ?        00:00:00 /usr/bin/python2 /usr/share/vdsm/supervdsmd --sockfile /var/run/vdsm/svdsm.sock
vdsm       8715      1  0 16:27 ?        00:00:00 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker

The all hosts in cluster are reachable from each other ...  That could be the issue? 

Thank you in advance!
-- 
Regards,
Artem
_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/TQX3LN2TEM4DECKKUMMRCWXTRM6BGIAB/


--
Regards,
Artem


--
Regards,
Artem