3:28 a.m.
On 1/28/20 5:11 PM, Michal Skrivanek wrote:
On 28 Jan 2020, at 16:14, Martin Perina <mperina(a)redhat.com
<mailto:mperina@redhat.com>> wrote:
>
>
> On Tue, Jan 28, 2020 at 9:58 AM Yedidyah Bar David <didi(a)redhat.com
> <mailto:didi@redhat.com>> wrote:
>
> Hi all,
>
> Tried yesterday to deploy hosted-engine on current master
> snapshot, it failed during host-deploy. host-deploy log ends with:
>
> 2020-01-27 15:25:11 IST - TASK [ovirt-host-deploy-vdsm :
> Reconfigure vdsm tool] **************************
>
> /var/log/messages on the host has:
>
> Jan 27 15:25:24 didi-centos8-host python3[12352]: detected
> unhandled Python exception in '/usr/bin/vdsm-tool'
>
> /var/log/httpd/ansible_runner_error_log on the engine has (I have
> to get used to searching there, grepped all of /var/log to find it):
>
> [Mon Jan 27 15:25:25.692397 2020] [wsgi:error] [pid 30029]
> cb_event_handler event_data={u'event': u'runner_on_failed',
> u'uuid': u'874e5fcc-12e8-419c-b010-d958adfc4aec',
'stdout':
> u'fatal: [d
> idi-centos8-host.lab.eng.tlv2.redhat.com
> <http://idi-centos8-host.lab.eng.tlv2.redhat.com>]: FAILED! =>
> {"changed": true, "cmd": "vdsm-tool configure
--force", "delta":
> "0:00:15.833532", "end": "2020-01-27
15:25:25.296212", "msg":
> "non-zero
> return code", "rc": 1, "start": "2020-01-27
15:25:09.462680",
> "stderr": "Error: Traceback (most recent call last):\\\\n File
> \\\\"/usr/bin/vdsm-tool\\\\", line 209, in main\\\\n return
> tool_command[cmd][\\\\"command\\\\"](*args)\\\\n File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py\\\\",
> line 40, in wrapper\\\\n func(*args, **kwargs)\\\\n File \\\\"/u
> sr/lib/python3.6/site-packages/vdsm/tool/configurator.py\\\\",
> line 149, in configure\\\\n service.service_start(s)\\\\n
> File \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\
> \\", line 193, in service_start\\\\n return
> _runAlts(_srvStartAlts, srvName)\\\\n File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\",
> line 172, in _runAlts\\\\n \\\\
> "%s failed\\\\" % alt.__name__, out,
> err)\\\\nvdsm.tool.service.ServiceOperationError: <exception
> str() failed>\\\\n\\\\nDuring handling of the above exception,
> another exception occurred:\\
> \\n\\\\nTraceback (most recent call last):\\\\n File
> \\\\"/usr/bin/vdsm-tool\\\\", line 224, in <module>\\\\n
> sys.exit(main())\\\\n File \\\\"/usr/bin/vdsm-tool\\\\", line
> 214, in main\
> \\\n print(\\'Error: \\', e, \\'\\\\\\\\n\\',
> file=sys.stderr)\\\\n File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\",
> line 75, in __str__\\\\n return \\'\\\\\\\\n\
> \'.join(s)\\\\nTypeError: sequence item 1: expected str instance,
> bytes found", "stderr_lines": ["Error: Traceback (most
recent
> call last):", " File \\\\"/usr/bin/vdsm-tool\\\\", line
209,
> in main", " return
> tool_command[cmd][\\\\"command\\\\"](*args)", " File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py\\\\",
> line 40, in wrapper", " func(*args, **kwargs
> )", " File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/configurator.py\\\\",
> line 149, in configure", " service.service_start(s)", "
File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/
> tool/service.py\\\\", line 193, in service_start", " return
> _runAlts(_srvStartAlts, srvName)", " File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\",
> line 172, in _runAl
> ts", " \\\\"%s failed\\\\" % alt.__name__, out,
err)",
> "vdsm.tool.service.ServiceOperationError: <exception str()
> failed>", "", "During handling of the above exception,
another
> exception
> occurred:", "", "Traceback (most recent call last):",
" File
> \\\\"/usr/bin/vdsm-tool\\\\", line 224, in <module>", "
> sys.exit(main())", " File \\\\"/usr/bin/vdsm-tool\\\\",
line
> 214, in
> main", " print(\\'Error: \\', e, \\'\\\\\\\\n\\',
> file=sys.stderr)", " File
> \\\\"/usr/lib/python3.6/site-packages/vdsm/tool/service.py\\\\",
> line 75, in __str__", " return \\'\\\\\\\
> \n\\'.join(s)", "TypeError: sequence item 1: expected str
> instance, bytes found"], "stdout": "\\\\nChecking
configuration
> status...\\\\n\\\\nCurrent revision of multipath.conf detected, pres
> erving\\\\nabrt is not configured for vdsm\\\\nlibvirtd socket
> units status: [{\\'Names\\': \\'libvirtd-tls.socket\\',
> \\'LoadState\\': \\'loaded\\'}, {\\'Names\\':
> \\'libvirtd-tcp.socket\\'
> , \\'LoadState\\': \\'masked\\'}, {\\'Names\\':
> \\'libvirtd-ro.socket\\', \\'LoadState\\':
\\'masked\\'},
> {\\'Names\\': \\'libvirtd-admin.socket\\',
\\'LoadState\\':
> \\'masked\\'}, {\\'Names
> \\': \\'libvirtd.socket\\', \\'LoadState\\':
> \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket activation
>
This is untrue now, *if* you are sure you’re using advanced virt repo
and installed libvirt/qemu from there. Can you confirm? Also, the
detection/handling is only fixed in vdsm since last Thu, not in any
build (only the tag from today has it, or nightly)
Can you confirm?
+1 with Michal here - this should never happen in real life
scenarios.
The most important thing to notice in the log is this part:
libvirtd doesn\\'t use systemd socket activation - one or more of its socket units
have been masked
As you rightly noticed in the other devel thread ("Task Start and enable
services failed to execute"),
libvirt's socket units are masked by libvirt itself in an upgrade scenario:
- you have old libvirt and vdsm installed
- vdsm configures libvirt to pass the "--listen" flag to the daemon
- you upgrade libvirt and vdsm to a version that uses socket activation
- libvirt installation scripts notice the "--listen" flag and respect it
by disabling
its own socket activation mechanism, which is done by masking sockets
- new vdsm version is lost - it sees that there's a new libvirt version that
uses socket activation (because the socket units are there), but they've
been masked in the meantime
We don't support upgrade path by design.
Current vdsm versions require libvirt >= 5.6 - versions that have socket
activation [1].
There are indeed some leftovers in 'vdsm-tool' that were trying to
handle both scenarios,
but again - no upgrade paths. 'vdsm-tool' should work for fresh
installations without any issues.
[1]
https://github.com/oVirt/vdsm/blob/c2735bec16586e1ad2d0ac0242839bfa0e35b4...
> - one or more of its socket units have been masked\\\\nlibvirt is
> not configured for vdsm yet\\\\nFAILED: conflicting vdsm and
> libvirt-qemu tls configuration.\\\\nvdsm.conf with ssl=True
> requires the following changes:\\\\nlibvirtd.conf: listen_tcp=0,
> auth_tcp=\\\\"sasl\\\\", listen_tls=1\\\\nqemu.conf:
> spice_tls=1.\\\\nlvm is configured for vdsm\\\\nManaged volume
> database is already configured\\\\n\\\\nRunning
> configure...\\\\nReconfiguration of passwd is
> done.\\\\nReconfiguration of abrt is done.\\\\nlibvirtd socket
> units status: [{\\'Names\\': \\'libvirtd-tls.socket\\',
> \\'LoadState\\': \\'loaded\\'}, {\\'Names\\':
> \\'libvirtd-tcp.socket\\', \\'LoadState\\':
\\'masked\\'},
> {\\'Names\\': \\'libvirtd-ro.socket\\', \\'LoadState\\':
> \\'masked\\'}, {\\'Names\\':
\\'libvirtd-admin.socket\\',
> \\'LoadState\\': \\'masked\\'}, {\\'Names\\':
> \\'libvirtd.socket\\', \\'LoadState\\':
> \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket
> activation - one or more of its socket units have been
> masked\\\\nlibvirtd socket units status: [{\\'Names\\':
> \\'libvirtd-tls.socket\\', \\'LoadState\\':
\\'loaded\\'},
> {\\'Names\\': \\'libvirtd-tcp.socket\\',
\\'LoadState\\':
> \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\',
> \\'LoadState\\': \\'masked\\'}, {\\'Names\\':
> \\'libvirtd-admin.socket\\', \\'LoadState\\':
\\'masked\\'},
> {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\':
> \\'masked\\'}]\\\\nlibvirtd doesn\\'t use systemd socket
> activation - one or more of its socket units have been
> masked\\\\nReconfiguration of libvirt is done.", "stdout_lines":
> ["", "Checking configuration status...", "",
"Current revision of
> multipath.conf detected, preserving", "abrt is not configured for
> vdsm", "libvirtd socket units status: [{\\'Names\\':
> \\'libvirtd-tls.socket\\', \\'LoadState\\':
\\'loaded\\'},
> {\\'Names\\': \\'libvirtd-tcp.socket\\',
\\'LoadState\\':
> \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\',
> \\'LoadState\\': \\'masked\\'}, {\\'Names\\':
> \\'libvirtd-admin.socket\\', \\'LoadState\\':
\\'masked\\'},
> {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\':
> \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket
> activation - one or more of its socket units have been masked",
> "libvirt is not configured for vdsm yet", "FAILED: conflicting
> vdsm and libvirt-qemu tls configuration.", "vdsm.conf with
> ssl=True requires the following changes:", "libvirtd.conf:
> listen_tcp=0, auth_tcp=\\\\"sasl\\\\", listen_tls=1",
"qemu.conf:
> spice_tls=1.", "lvm is configured for vdsm", "Managed volume
> database is already configured", "", "Running
configure...",
> "Reconfiguration of passwd is done.", "Reconfiguration of abrt is
> done.", "libvirtd socket units status: [{\\'Names\\':
> \\'libvirtd-tls.socket\\', \\'LoadState\\':
\\'loaded\\'},
> {\\'Names\\': \\'libvirtd-tcp.socket\\',
\\'LoadState\\':
> \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\',
> \\'LoadState\\': \\'masked\\'}, {\\'Names\\':
> \\'libvirtd-admin.socket\\', \\'LoadState\\':
\\'masked\\'},
> {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\':
> \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket
> activation - one or more of its socket units have been masked",
> "libvirtd socket units status: [{\\'Names\\':
> \\'libvirtd-tls.socket\\', \\'LoadState\\':
\\'loaded\\'},
> {\\'Names\\': \\'libvirtd-tcp.socket\\',
\\'LoadState\\':
> \\'masked\\'}, {\\'Names\\': \\'libvirtd-ro.socket\\',
> \\'LoadState\\': \\'masked\\'}, {\\'Names\\':
> \\'libvirtd-admin.socket\\', \\'LoadState\\':
\\'masked\\'},
> {\\'Names\\': \\'libvirtd.socket\\', \\'LoadState\\':
> \\'masked\\'}]", "libvirtd doesn\\'t use systemd socket
> activation - one or more of its socket units have been masked",
> "Reconfiguration of libvirt is done."]}', 'counter': 40,
u'pid':
> 31685, u'created': u'2020-01-27T13:25:25.683262',
'end_line': 35,
> 'runner_ident': '5b95ec42-4108-11ea-8afe-001a4a16027a',
> 'start_line': 34, u'event_data': {u'play_pattern':
u'all',
> u'play': u'all', u'event_loop': None,
u'task_args': u'',
> u'remote_addr': u'didi-centos8-host.lab.eng.tlv2.redhat.com
> <http://didi-centos8-host.lab.eng.tlv2.redhat.com>';, u'res':
> {u'stderr_lines': [u'Error: Traceback (most recent call
last):',
> u' File "/usr/bin/vdsm-tool", line 209, in main', u'
return
> tool_command[cmd]["command"](*args)', u' File
> "/usr/lib/python3.6/site-packages/vdsm/tool/__init__.py", line
> 40, in wrapper', u' func(*args, **kwargs)', u' File
> "/usr/lib/python3.6/site-packages/vdsm/tool/configurator.py",
> line 149, in configure', u' service.service_start(s)', u' File
> "/usr/lib/python3.6/site-
>
>
> The exception seems to be cut somehow, but anyway Marcin, could you
> please take a look?
>
> [Mon Jan 27 15:25:25.693655 2020] [wsgi:error] [pid 30029]
> cb_event_handler event_data={u'event': u'playbook_on_stats',
> u'uuid': u'2c8d843c-c1ad-41c0-940e-88354387b725',
'stdout':
> u'\\r\\nPLAY RECAP
>
*********************************************************************\\r\\ndidi-centos8-host.lab.eng.tlv2.redhat.com
> <http://ndidi-centos8-host.lab.eng.tlv2.redhat.com> : ok=11
> changed=0 unreachable=0 failed=1 skipped=1 rescued=0
> ignored=0 \\r\\n', 'counter': 41, u'pid': 31685,
u'created':
> u'2020-01-27T13:25:25.688541', 'end_line': 39,
'runner_ident':
> '5b95ec42-4108-11ea-8afe-001a4a16027a', 'start_line': 35,
> u'event_data': {u'ignored': {}, u'skipped':
> {u'didi-centos8-host.lab.eng.tlv2.redhat.com
> <http://didi-centos8-host.lab.eng.tlv2.redhat.com>';: 1},
u'ok':
> {u'didi-centos8-host.lab.eng.tlv2.redhat.com
> <http://didi-centos8-host.lab.eng.tlv2.redhat.com>';: 11},
> u'artifact_data': {}, u'rescued': {}, u'changed': {},
u'pid':
> 31685, u'dark': {}, u'playbook_uuid':
> u'ff508986-9930-47a2-b17b-61a2f2dae905', u'playbook':
> u'ovirt-host-deploy.yml', u'failures':
> {u'didi-centos8-host.lab.eng.tlv2.redhat.com
> <http://didi-centos8-host.lab.eng.tlv2.redhat.com>';: 1},
> u'processed': {u'didi-centos8-host.lab.eng.tlv2.redhat.com
> <http://didi-centos8-host.lab.eng.tlv2.redhat.com>';: 1}},
> u'parent_uuid': u'ff508986-9930-47a2-b17b-61a2f2dae905'}
>
> If I now manually run on the host 'vdsm-tool configure --force',
> it does not fail.
>
> I am now going to try deploy again after reverting to a snapshot
> I took before deploy.
>
> Best regards,
> --
> Didi
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org <mailto:devel@ovirt.org>
> To unsubscribe send an email to devel-leave(a)ovirt.org
> <mailto:devel-leave@ovirt.org>
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/Q2VEW5NXQPV...
>
>
>
> --
> Martin Perina
> Manager, Software Engineering
> Red Hat Czech s.r.o.
> _______________________________________________
> Devel mailing list -- devel(a)ovirt.org <mailto:devel@ovirt.org>
> To unsubscribe send an email to devel-leave(a)ovirt.org
> <mailto:devel-leave@ovirt.org>
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/A44YHUFU54H...