[ovirt-users] AIO 3.4 on fedora 19 initial errors before coming up

[David Sommerseth]

On 12/05/14 19:27, Dan Kenigsberg wrote:
> On Mon, May 12, 2014 at 05:34:34PM +0200, David Sommerseth wrote:
>> On 12/05/14 12:28, Dan Kenigsberg wrote:
>>> On Mon, May 12, 2014 at 02:33:52AM -0400, Francesco Romani wrote:
>>>> Hi,
>>>>
>>>> ----- Original Message -----
>>>>> From: "Gianluca Cecchi" <gianluca.cecchi at gmail.com>
>>>>> To: "Roy Golan" <rgolan at redhat.com>
>>>>> Cc: "users" <users at ovirt.org>
>>>>> Sent: Sunday, May 11, 2014 11:49:06 PM
>>>>> Subject: Re: [ovirt-users] getVdsCapabilites unexpected exception [was: Re: AIO 3.4 on fedora 19 initial errors
>>>>> before coming up]
>>>> [...]
>>>>> it seems the error in vdsm.log when I run the command above is of this type:
>>>>>
>>>>> Thread-25::ERROR::2014-05-11
>>>>> 20:18:02,202::BindingXMLRPC::1086::vds::(wrapper) unexpected error
>>>>> Traceback (most recent call last):
>>>>> File "/usr/share/vdsm/BindingXMLRPC.py", line 1070, in wrapper
>>>>> res = f(*args, **kwargs)
>>>>> File "/usr/share/vdsm/BindingXMLRPC.py", line 393, in getCapabilities
>>>>> ret = api.getCapabilities()
>>>>> File "/usr/share/vdsm/API.py", line 1185, in getCapabilities
>>>>> c = caps.get()
>>>>> File "/usr/share/vdsm/caps.py", line 369, in get
>>>>> caps.update(netinfo.get())
>>>>> File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 557, in get
>>>>> netAttr.get('qosOutbound'))
>>>>> File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 487, in
>>>>> _getNetInfo
>>>>> ipv4addr, ipv4netmask, ipv6addrs = getIpInfo(iface)
>>>>> File "/usr/lib64/python2.7/site-packages/vdsm/netinfo.py", line 317, in
>>>>> getIpInfo
>>>>> ipv6addrs = devInfo.get_ipv6_addresses()
>>>>> SystemError: error return without exception set
>>>>>
>>>>>
>>>>> Based on above errors, I think that for some reason these two python related
>>>>> packages that were updated yesterday are causing some problems with vdsm.
>>>>> Can you confirm that you can run ok the 3.4 vdsm with those?
>>>>>
>>>>> vdsm-4.14.6-0.fc19.x86_64
>>>>>
>>>>>
>>>>> May 10 21:24:23 Updated: python-ethtool-0.9-2.fc19.x86_64
>>>>> May 10 21:24:23 Updated: python-lxml-3.3.5-1.fc19.x86_64
>>>>>
>>>>> I can also try to rollback and see...
>>>>>
>>>>>
>>>>> I was right.
>>>>> Against what to bugzilla?
>>>>> This is a show stopper for fedora 19 ovirt users...
>>>>
>>>> Unfortunately, you are been hit by https://bugzilla.redhat.com/show_bug.cgi?id=1078312
>>>>
>>>> It is fixed on gerrit, but you'll need VDSM >= 4.14.8.1
>>>
>>> Too bad that we did not manage to add a "Conflicts: vdsm <= 4.18.6" to
>>> that release of python-ethtool-0.9-2.fc19.x86_64. But now that it is
>>> out, there is not much that one can do but to upgrade to a new Vdsm or
>>> roll back python-ethtool.
>>>
>>> The propblem was due to Vdsm-proper using libnl1 while that version of
>>> python-ethtool starting to use linbnl3 and solved by
>>> http://gerrit.ovirt.org/26514. Backporting this to the now-unsupported
>>> ovirt-3.3 is not really viable, I a m afraid.
>>
>> Sorry about that!  I honestly forgot about this when I had a discussion
>> with Antoni Segura Puimedon last week.  And we discovered some other
>> ugly issues too, where we need a dependency on a not yet released libnl3
>> version to fully fix some libnl connection conflicts with vdsm.
> 
> Could you (or Toni) add more information regrading the last sentence? Is
> there a known issue with ovirt-3.4.1's vdsm?

AFAIU, it's the same old issue, which caused some regression from the
tests we've done earlier, due to vdsm using libnl-1.x while
python-ethtool uses libnl3.  When vdsm in addition uses py-ethtool, the
socket py-ethtool module gets closed by itself, which it uses to query
the kernel's netlink interface.  The result is that py-ethtool gets in a
useless state is not able to re-establish a new socket and cannot query
the system for any useful information.  Thomas Haller (libnl developer)
have provided patches to libnl3 which fixes the connection handling -
which should help avoiding this issue.

I missed the detail that the libnl3 fixes hadn't been pushed out in a
release.  So when py-ethool got updated, it broke vdsm due to this
libnl1/libnl3 issue.

With an updated libnl3 + updated py-ethtool, I believe things should be
better also for the older vdsm versions.  The change in py-ethtool is
primarily improving error handling if libnl3 isn't able to establish a
netlink socket to the kernel.


--
kind regards,

David Sommerseth