[ovirt-users] 3.6 looses network on reboot

[David LeVene]

Hi Nir,

Thanks for your help. I’m going to start up another thread as to not pollute what this thread was initially about.

I’ll test your patch, and I suspect it’s the gluster version that is causing the problem as I did update the host.

When I create the thread I’ll make sure to include everyone already present (It might be a week or so, as my time Is limited atm)

Cheers
David

From: Nir Soffer [mailto:nsoffer at redhat.com]
Sent: Thursday, March 03, 2016 17:59
To: Dan Kenigsberg <danken at redhat.com>; Sahina Bose <sabose at redhat.com>; Ala Hino <ahino at redhat.com>
Cc: David LeVene <David.LeVene at blackboard.com>; users at ovirt.org
Subject: Re: [ovirt-users] 3.6 looses network on reboot

On Thu, Mar 3, 2016 at 9:06 AM, Dan Kenigsberg <danken at redhat.com<mailto:danken at redhat.com>> wrote:
>
> On Thu, Mar 03, 2016 at 12:54:25AM +0000, David LeVene wrote:
> >
> > Can you check our patches? They should resolve the problem we saw in the
> > log: https://gerrit.ovirt.org/#/c/54237  (based on oVirt-3.6.3)
> >
> > -- I've manually applied the patch to the node that I was testing on
> > and the networking comes on-line correctly - now I'm encountering a
> > gluster issue with cannot find master domain.
>
> You are most welcome to share your logs (preferably on a different
> thread, to avoid confusion)
>
> >
> > Without the fixes, as a workaround, I would suggest (if possible) to disable IPv6 on your host boot line and check if all works out for you.
> > -- Ok, but as I can manually apply the patch its good now. Do you know
> > what version are we hoping to have this put into as I won't perform an
> > ovirt/vdsm update until its part of the upstream RPM's
>
> The fix has been proposed to ovirt-3.6.4. I'll make sure it's accepted.
>
> >
> > Do you need IPv6 connectivity? If so, you'll need to use a vdsm hook or another interface that is not controlled by oVirt.
> > -- Ideally I'd prefer not to have it, but the way our network has been
> > configured some hosts are IPv6 only, so at a min the guests need it..
> > the hypervisors not so much.
>
> May I tap to what your IPv6 experience? (only if you feel confortable sharing
> this publically). What does these IPv6-only servers do? What does the guest do
> with them?
>
> >
> > -- I've now hit an issue with it not starting up the master storage
> > gluster domain - as it’s a separate issue I'll review the mailing
> > lists & create a new item if its related.. I've attached the
> > supervdsm.log incase you can save me some time and point me in the
> > right direction!
>
> All I see is this
>
> MainProcess|jsonrpc.Executor/4::ERROR::2016-03-03 11:15:04,699::supervdsmServer::118::SuperVdsm.ServerCallback::(wrapper) Error in wrapper
> Traceback (most recent call last):
>   File "/usr/share/vdsm/supervdsmServer", line 116, in wrapper
>     res = func(*args, **kwargs)
>   File "/usr/share/vdsm/supervdsmServer", line 531, in wrapper
>     return func(*args, **kwargs)
>   File "/usr/share/vdsm/gluster/cli.py", line 496, in volumeInfo
>     xmltree = _execGlusterXml(command)
>   File "/usr/share/vdsm/gluster/cli.py", line 108, in _execGlusterXml
>     raise ge.GlusterCmdExecFailedException(rc, out, err)
> GlusterCmdExecFailedException: Command execution failed
> return code: 2


We have this logs before the exception:

MainProcess|jsonrpc.Executor/3::DEBUG::2016-03-03 11:02:42,945::utils::669::root::(execCmd) /usr/bin/taskset --cpu-list 0-39 /usr/sbin/gluster --mode=script volume info --re
mote-host=ovirtmount.test.lab data --xml (cwd None)

The command looks correct

MainProcess|jsonrpc.Executor/3::DEBUG::2016-03-03 11:02:43,024::utils::687::root::(execCmd) FAILED: <err> = '\n'; <rc> = 2

gluster command line failed in an unhelpful way.

(Adding Sahina)

David, can you try to run this command manually on this host? maybe there is some
--verbose flag revealing more info?

You may also try a simpler command:

    gluster volume info --remote-host=ovirtmount.test.lab data

Another issue you should check - gluster version on the hosts and on the gluster nodes *must*
match - otherwise you should expect failures accessing gluster server.

We have this patch for handling such errors gracefully - can you test it?
https://gerrit.ovirt.org/53785

(Adding Ala)

Nir
This email and any attachments may contain confidential and proprietary information of Blackboard that is for the sole use of the intended recipient. If you are not the intended recipient, disclosure, copying, re-distribution or other use of any of this information is strictly prohibited. Please immediately notify the sender and delete this transmission if you received this email in error.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160304/edd4f915/attachment-0001.html>