<div dir="ltr"><div><div><div><div><div><div><div>Nadav,<br><br></div>Here is the code [1] which is responsible for this check. Here is vdsm log [2] where I added logging statement to understand what is commonName value (it was 'engine').<br><br></div>Here are the steps what is done during the check:<br></div>1. We get client peer name by calling socket.getpeername()[0] which is:<br><pre>::ffff:192.168.201.3</pre>2. We get common name from the certificate. It should be engine's fqdn and as in the log we get 'engine'<br></div>3. For name use we compare common name and name lookup based on the IP. I pushed [3] a patch to normalizes the the ip (still requires my attention)<br><br></div>Based on the outcome of the logs it seems that 192.168.201.3 does not resolve to 'engine' name.<br><br></div>Thanks,<br></div>Piotr<br><div><div><div><div><div><div><div><div><br>[1] <a href="https://github.com/oVirt/vdsm/blob/master/lib/vdsm/sslutils.py#L234">https://github.com/oVirt/vdsm/blob/master/lib/vdsm/sslutils.py#L234</a><br>[2] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/331/artifact/exported-artifacts/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host0/_var_log/vdsm/vdsm.log">http://jenkins.ovirt.org/job/ovirt-system-tests_manual/331/artifact/exported-artifacts/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host0/_var_log/vdsm/vdsm.log</a><br>[3] <a href="https://gerrit.ovirt.org/#/c/76197/">https://gerrit.ovirt.org/#/c/76197/</a><br></div></div></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 30, 2017 at 12:52 PM, Nadav Goldin <span dir="ltr"><<a href="mailto:ngoldin@redhat.com" target="_blank">ngoldin@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Looking at the failure, I'm not sure what is wrong here on the setup<br>
side. The FQDN(lago-basic-suite-master-<wbr>engine) should be resolvable in<br>
the hosts - at least from what I tested that locally. On the engine<br>
setup.log I see this was the generated certificate(if we're talking<br>
about the same one here):<br>
<br>
2017-04-30 06:30:41,308-0400 DEBUG<br>
<a href="http://otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca" rel="noreferrer" target="_blank">otopi.plugins.ovirt_engine_<wbr>setup.ovirt_engine.pki.ca</a><br>
plugin.executeRaw:813 execute:<br>
('/usr/share/ovirt-engine/bin/<wbr>pki-enroll-pkcs12.sh', '--name=engine',<br>
'--password=**FILTERED**',<br>
'--subject=/C=US/O=Test/CN=<wbr>lago-basic-suite-master-<wbr>engine'),<br>
executable='None', cwd='None', env=None<br>
2017-04-30 06:30:44,542-0400 DEBUG<br>
<a href="http://otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca" rel="noreferrer" target="_blank">otopi.plugins.ovirt_engine_<wbr>setup.ovirt_engine.pki.ca</a><br>
plugin.executeRaw:863 execute-result:<br>
('/usr/share/ovirt-engine/bin/<wbr>pki-enroll-pkcs12.sh', '--name=engine',<br>
'--password=**FILTERED**',<br>
'--subject=/C=US/O=Test/CN=<wbr>lago-basic-suite-master-<wbr>engine'), rc=0<br>
2017-04-30 06:30:44,543-0400 DEBUG<br>
<a href="http://otopi.plugins.ovirt_engine_setup.ovirt_engine.pki.ca" rel="noreferrer" target="_blank">otopi.plugins.ovirt_engine_<wbr>setup.ovirt_engine.pki.ca</a><br>
plugin.execute:921 execute-output:<br>
('/usr/share/ovirt-engine/bin/<wbr>pki-enroll-pkcs12.sh', '--name=engine',<br>
'--password=**FILTERED**',<br>
'--subject=/C=US/O=Test/CN=<wbr>lago-basic-suite-master-<wbr>engine')<br>
<br>
<br>
Do we expect the '--name' parameter to be the same as the hostname? My<br>
thought was that it should use the engine FQDN, and that should match<br>
the certificate name.<br>
<br>
If that is not the problem, can you make the output more verbose in<br>
vdsm logs? so we'll know exactly what name is it looking for.<br>
<br>
<br>
Thanks<br>
<span class="HOEnZb"><font color="#888888"><br>
Nadav.<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
On Sun, Apr 30, 2017 at 1:43 PM, Piotr Kliczewski <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>> wrote:<br>
> The job failed.<br>
><br>
> Just to be clear. We need to resolve engine name on a host side or use ip<br>
> address.<br>
><br>
> Thanks,<br>
> Piotr<br>
><br>
> On Sun, Apr 30, 2017 at 12:23 PM, Piotr Kliczewski <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
> wrote:<br>
>><br>
>> Here is the link<br>
>><br>
>> <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/331/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_manual/331/</a><br>
>><br>
>> On Sun, Apr 30, 2017 at 12:17 PM, Piotr Kliczewski <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
>> wrote:<br>
>>><br>
>>> Sure, will test<br>
>>><br>
>>> 30 kwi 2017 12:14 "Nadav Goldin" <<a href="mailto:ngoldin@redhat.com">ngoldin@redhat.com</a>> napisał(a):<br>
>>>><br>
>>>> It is under-work in [1], as it requires cross-changes in all suites it<br>
>>>> takes a while to test it/cover all changes, though basic-suite-master<br>
>>>> already passed.<br>
>>>> Can you test it by running OST manual with your changes and the OST<br>
>>>> patch(i.e. put also in GERRIT_REFSPEC: refs/changes/25/76225/7 )<br>
>>>><br>
>>>><br>
>>>><br>
>>>> [1] <a href="https://gerrit.ovirt.org/76225" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/76225</a><br>
>>>><br>
>>>> On Sun, Apr 30, 2017 at 1:09 PM, Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>> wrote:<br>
>>>> ><br>
>>>> ><br>
>>>> > On Sun, Apr 30, 2017 at 1:03 PM, Piotr Kliczewski<br>
>>>> > <<a href="mailto:piotr.kliczewski@gmail.com">piotr.kliczewski@gmail.com</a>> wrote:<br>
>>>> >><br>
>>>> >> When we can have it fixed? I checked few minutes ago and the problem<br>
>>>> >> is still there.<br>
>>>> ><br>
>>>> ><br>
>>>> > <a href="https://gerrit.ovirt.org/#/c/76225/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>76225/</a> should cover this.<br>
>>>> ><br>
>>>> > What I wonder is what caused this in the first place. The SSL change?<br>
>>>> > Y.<br>
>>>> ><br>
>>>> >><br>
>>>> >><br>
>>>> >> Thanks,<br>
>>>> >> Piotr<br>
>>>> >><br>
>>>> >> On Sat, Apr 29, 2017 at 11:18 AM, Piotr Kliczewski<br>
>>>> >> <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
>>>> >> wrote:<br>
>>>> >> > Nadav,<br>
>>>> >> ><br>
>>>> >> > Yes, vdsm is not able to resolve 'engine' which is used in engine's<br>
>>>> >> > certificate.<br>
>>>> >> ><br>
>>>> >> > Thanks,<br>
>>>> >> > Piotr<br>
>>>> >> ><br>
>>>> >> > 29 kwi 2017 00:37 "Nadav Goldin" <<a href="mailto:ngoldin@redhat.com">ngoldin@redhat.com</a>> napisał(a):<br>
>>>> >> ><br>
>>>> >> > Hi Piotr,<br>
>>>> >> > Can you clarify what you noticed is not resolvable - the 'engine'<br>
>>>> >> > FQDN<br>
>>>> >> > from host0?<br>
>>>> >> ><br>
>>>> >> > Thanks,<br>
>>>> >> > Nadav.<br>
>>>> >> ><br>
>>>> >> ><br>
>>>> >> > On Fri, Apr 28, 2017 at 4:15 PM, Piotr Kliczewski<br>
>>>> >> > <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
>>>> >> > wrote:<br>
>>>> >> >> I started to investigate the issue [1] and it seems like there is<br>
>>>> >> >> an<br>
>>>> >> >> issue<br>
>>>> >> >> in Lago setup we use.<br>
>>>> >> >><br>
>>>> >> >> During handshake we have a step to verify whether client<br>
>>>> >> >> certificate<br>
>>>> >> >> was<br>
>>>> >> >> issued for a specific host (no such functionality in m2crytpo code<br>
>>>> >> >> base).<br>
>>>> >> >> It works fine when using either ip addresses or fqdns but in this<br>
>>>> >> >> particular<br>
>>>> >> >> setup we use mixed.<br>
>>>> >> >><br>
>>>> >> >> When added logging I see that in engine certificate we use<br>
>>>> >> >> 'engine'<br>
>>>> >> >> name<br>
>>>> >> >> which is not resolvable on the host side and the check fails.<br>
>>>> >> >> I posted a patch [2] which fixes IPv4 mapped addresses issue but<br>
>>>> >> >> we<br>
>>>> >> >> need<br>
>>>> >> >> to<br>
>>>> >> >> fix the setup issue.<br>
>>>> >> >><br>
>>>> >> >> Thanks,<br>
>>>> >> >> Piotr<br>
>>>> >> >><br>
>>>> >> >> [1] <a href="http://jenkins.ovirt.org/job/ovirt-system-tests_manual/326/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt-system-tests_manual/326/</a><br>
>>>> >> >> [2] <a href="https://gerrit.ovirt.org/#/c/76197/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>76197/</a><br>
>>>> >> >><br>
>>>> >> >> On Thu, Apr 27, 2017 at 3:39 PM, Piotr Kliczewski<br>
>>>> >> >> <<a href="mailto:pkliczew@redhat.com">pkliczew@redhat.com</a>><br>
>>>> >> >> wrote:<br>
>>>> >> >>><br>
>>>> >> >>><br>
>>>> >> >>><br>
>>>> >> >>> On Thu, Apr 27, 2017 at 3:13 PM, Evgheni Dereveanchin<br>
>>>> >> >>> <<a href="mailto:ederevea@redhat.com">ederevea@redhat.com</a>> wrote:<br>
>>>> >> >>>><br>
>>>> >> >>>> Test failed: 002_bootstrap/add_hosts<br>
>>>> >> >>>><br>
>>>> >> >>>> Link to suspected patches:<br>
>>>> >> >>>> <a href="https://gerrit.ovirt.org/76107" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/76107</a> - ssl: change default library<br>
>>>> >> >>>><br>
>>>> >> >>>> Link to job:<br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6491/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/6491/</a><br>
>>>> >> >>>><br>
>>>> >> >>>> VDSM log:<br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/6491/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-host0/_var_log/vdsm/vdsm.log" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/6491/artifact/exported-<wbr>artifacts/basic-suit-master-<wbr>el7/test_logs/basic-suite-<wbr>master/post-002_bootstrap.py/<wbr>lago-basic-suite-master-host0/<wbr>_var_log/vdsm/vdsm.log</a><br>
>>>> >> >>>><br>
>>>> >> >>>> Error snippet from VDSM log, this repeats on each connection<br>
>>>> >> >>>> attempt<br>
>>>> >> >>>> from<br>
>>>> >> >>>> Engine side:<br>
>>>> >> >>>><br>
>>>> >> >>>> <error><br>
>>>> >> >>>><br>
>>>> >> >>>> 2017-04-27 06:39:27,768-0400 INFO (Reactor thread)<br>
>>>> >> >>>> [ProtocolDetector.<wbr>AcceptorImpl] Accepted connection from<br>
>>>> >> >>>> ::ffff:<a href="http://192.168.201.3:49530" rel="noreferrer" target="_blank">192.168.201.3:49530</a> (protocoldetector:74)<br>
>>>> >> >>>> 2017-04-27 06:39:27,898-0400 ERROR (Reactor thread)<br>
>>>> >> >>>> [vds.dispatcher]<br>
>>>> >> >>>> uncaptured python exception, closing channel<br>
>>>> >> >>>> <yajsonrpc.betterAsyncore.<wbr>Dispatcher connected<br>
>>>> >> >>>> ('::ffff:192.168.201.3',<br>
>>>> >> >>>> 49530, 0, 0) at 0x1cc3b00> (<class 'socket.error'>:Address<br>
>>>> >> >>>> family not<br>
>>>> >> >>>> supported by protocol<br>
>>>> >> >>>> [/usr/lib64/python2.7/<wbr>asyncore.py|readwrite|110]<br>
>>>> >> >>>> [/usr/lib64/python2.7/<wbr>asyncore.py|handle_write_<wbr>event|468]<br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>> [/usr/lib/python2.7/site-<wbr>packages/yajsonrpc/<wbr>betterAsyncore.py|handle_<wbr>write|70]<br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>> [/usr/lib/python2.7/site-<wbr>packages/yajsonrpc/<wbr>betterAsyncore.py|_delegate_<wbr>call|149]<br>
>>>> >> >>>><br>
>>>> >> >>>> [/usr/lib/python2.7/site-<wbr>packages/vdsm/sslutils.py|<wbr>handle_write|213]<br>
>>>> >> >>>><br>
>>>> >> >>>> [/usr/lib/python2.7/site-<wbr>packages/vdsm/sslutils.py|_<wbr>handle_io|223]<br>
>>>> >> >>>><br>
>>>> >> >>>> [/usr/lib/python2.7/site-<wbr>packages/vdsm/sslutils.py|_<wbr>verify_host|237]<br>
>>>> >> >>>><br>
>>>> >> >>>><br>
>>>> >> >>>> [/usr/lib/python2.7/site-<wbr>packages/vdsm/sslutils.py|<wbr>compare_names|249])<br>
>>>> >> >>>> (betterAsyncore:160)<br>
>>>> >> >>>><br>
>>>> >> >>>> </error><br>
>>>> >> >>><br>
>>>> >> >>><br>
>>>> >> >>> This means that what we have in the certificate do not match the<br>
>>>> >> >>> source<br>
>>>> >> >>> address we get. I suspect that we issue the certificate for<br>
>>>> >> >>> 192.168.201.3<br>
>>>> >> >>> but when we get ::ffff:192.168.201.3.<br>
>>>> >> >>> The change was verified in the env when ipv4 is used. I pushed a<br>
>>>> >> >>> revert<br>
>>>> >> >>> [1] for now so we can work on fixing the issue.<br>
>>>> >> >>><br>
>>>> >> >>> [1] <a href="https://gerrit.ovirt.org/#/c/76160" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>76160</a><br>
>>>> >> >>><br>
>>>> >> >>>><br>
>>>> >> >>>> --<br>
>>>> >> >>>> Regards,<br>
>>>> >> >>>> Evgheni Dereveanchin<br>
>>>> >> >>><br>
>>>> >> >>><br>
>>>> >> >><br>
>>>> >> >><br>
>>>> >> >> ______________________________<wbr>_________________<br>
>>>> >> >> Devel mailing list<br>
>>>> >> >> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
>>>> >> >> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
>>>> >> ><br>
>>>> >> ><br>
>>>> >> ><br>
>>>> >> > ______________________________<wbr>_________________<br>
>>>> >> > Devel mailing list<br>
>>>> >> > <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
>>>> >> > <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
>>>> >> ______________________________<wbr>_________________<br>
>>>> >> Devel mailing list<br>
>>>> >> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
>>>> >> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
>>>> ><br>
>>>> ><br>
>>>> ><br>
>>>> > ______________________________<wbr>_________________<br>
>>>> > Devel mailing list<br>
>>>> > <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
>>>> > <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
>><br>
>><br>
><br>
</div></div></blockquote></div><br></div>