[ovirt-users] vdsm ssl errors

Tue Jul 26 07:27:06 UTC 2016

Christoph,

Please apply [1] we would know exactly what verbs were called by this local
client.

Thanks,
Piotr

[1] https://gerrit.ovirt.org/#/c/61367/

On Mon, Jul 25, 2016 at 9:50 PM, Piotr Kliczewski <
piotr.kliczewski at gmail.com> wrote:

> On Mon, Jul 25, 2016 at 6:18 PM, Nir Soffer <nsoffer at redhat.com> wrote:
> > On Mon, Jul 25, 2016 at 6:22 PM, C. Handel <ovirt at macht-blau.org> wrote:
> >> two nodes (x.x.138.208, x.x.138.210), hosted-engine on x.x.139.240.
> >>
> >> the vdsm logs are from x.x.138.208 and the connection is from the node
> >> itself.
> >>
> >>> Running tcpdump it is a connect from the node to itself. I can't figure
> >>> out what is wrong. Can someone ?> give me a hint?
> >>
> >> so i know they are coming from the node itself. The process connecting
> is
> >> terminating too fast. the moment i see it in tcpdump, it is gone from
> the
> >> process table.
> >
> > I think this is ovirt hosted agent - check its logs, you will probably
> find that
> > it make some request in the same time you see the errors in your logs.
> >
> > Adding Martin, maintaining this project.
> >
> > These are the interesting events in the logs:
> >
> > $ grep 37678 vdsm.log
> > JsonRpc (StompReactor)::ERROR::2016-07-25
> > 13:48:58,074::betterAsyncore::113::vds.dispatcher::(recv) SSL error
> > during reading data from <yajsonrpc.betterAsyncore.Dispatcher
> > connected ('::ffff:140.181.138.208', 37678, 0, 0) at 0x42c9b90>:
> > unexpected eof
> >
> > $ grep 37684 vdsm.log
> > Reactor thread::INFO::2016-07-25
> >
> 13:49:00,205::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)
> > Accepting connection from ::ffff:140.181.138.208:37684
> > Reactor thread::INFO::2016-07-25
> >
> 13:49:00,211::protocoldetector::121::ProtocolDetector.Detector::(handle_read)
> > Detected protocol stomp from ::ffff:140.181.138.208:37684
> > Reactor thread::DEBUG::2016-07-25
> >
> 13:49:00,211::stompreactor::492::protocoldetector.StompDetector::(handle_socket)
> > Stomp detected from ('::ffff:140.181.138.208', 37684)
> > JsonRpc (StompReactor)::ERROR::2016-07-25
> > 13:49:01,824::betterAsyncore::113::vds.dispatcher::(recv) SSL error
> > during reading data from <yajsonrpc.betterAsyncore.Dispatcher
> > connected ('::ffff:140.181.138.208', 37684, 0, 0) at 0x42b0758>:
> > unexpected eof
> >
> > The log is too small, we see only one full request.
> >
> > Pitor, can you understand from this log what the request coming from
> > ::ffff:140.181.138.208:37684
> > is doing?
> >
>
> I stated above there are 2 verbs that were called each time:
>
>  Host.getStats and Host.getHardwareInfo
>
> >>
> >> Greetings
> >>    Christoph
> >>
> >> On Mon, Jul 25, 2016 at 4:53 PM, Piotr Kliczewski
> >> <piotr.kliczewski at gmail.com> wrote:
> >>>
> >>> Christoph,
> >>>
> >>> In log snippets you provided I can see 2 occurrences of the log entry.
> >>> There is 3 seconds between the calls.
> >>>
> >>> Each time I see calls to Host.getStats and Host.getHardwareInfo both
> >>> from x.x.138.208.
> >>> I do not see any log entries in the engine log so it not engine who
> >>> connected.
> >>>
> >>> What host is it?
> >>>
> >>> Thanks,
> >>> Piotr
> >>>
> >>>
> >>> On Mon, Jul 25, 2016 at 3:45 PM, C. Handel <ovirt at macht-blau.org>
> wrote:
> >>> > patch applied, The connection is from the node itself.
> >>> >
> >>> > logfiles with the last 100k (hope this is enough, the error happens
> >>> > every
> >>> > few seconds) of data attached.
> >>> >
> >>> > Greetings
> >>> >   Christoph
> >>> >
> >>> > On Mon, Jul 25, 2016 at 12:07 PM, Nir Soffer <nsoffer at redhat.com>
> wrote:
> >>> >>
> >>> >> On Thu, Jul 21, 2016 at 10:00 AM, C. Handel <ovirt at macht-blau.org>
> >>> >> wrote:
> >>> >> > longer logs attached, excerpts:
> >>> >> >
> >>> >> > ---+ vdsm
> >>> >> >
> >>> >> > Reactor thread::INFO::2016-07-21
> >>> >> >
> >>> >> >
> >>> >> >
> 08:01:19,544::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)
> >>> >> > Accepting connection from ::ffff:140.181.138.208:59502
> >>> >> > Reactor thread::DEBUG::2016-07-21
> >>> >> >
> >>> >> >
> >>> >> >
> 08:01:19,551::protocoldetector::85::ProtocolDetector.Detector::(__init__)
> >>> >> > Using required_size=11
> >>> >> > Reactor thread::INFO::2016-07-21
> >>> >> >
> >>> >> >
> >>> >> >
> 08:01:19,553::protocoldetector::121::ProtocolDetector.Detector::(handle_read)
> >>> >> > Detected protocol stomp from ::ffff:140.181.138.208:59502
> >>> >> > Reactor thread::INFO::2016-07-21
> >>> >> >
> 08:01:19,553::stompreactor::101::Broker.StompAdapter::(_cmd_connect)
> >>> >> > Processing CONNECT request
> >>> >> > Reactor thread::DEBUG::2016-07-21
> >>> >> >
> >>> >> >
> >>> >> >
> 08:01:19,554::stompreactor::492::protocoldetector.StompDetector::(handle_socket)
> >>> >> > Stomp detected from ('::ffff:140.181.138.208', 59502)
> >>> >> > JsonRpc (StompReactor)::INFO::2016-07-21
> >>> >> >
> >>> >> >
> 08:01:19,554::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe)
> >>> >> > Subscribe command received
> >>> >> > ...
> >>> >> >
> >>> >> > JsonRpc (StompReactor)::ERROR::2016-07-21
> >>> >> > 08:01:21,411::betterAsyncore::113::vds.dispatcher::(recv) SSL
> error
> >>> >> > during
> >>> >> > reading data: unexpected eof
> >>> >>
> >>> >> This log is not very useful as is, we must show the relevant remote
> >>> >> address.
> >>> >>
> >>> >> Should be improved in
> >>> >> https://gerrit.ovirt.org/61303
> >>> >>
> >>> >> Can you try this patch and share the log?
> >>> >>
> >>> >> >
> >>> >> >
> >>> >> > ---+ engine
> >>> >> >
> >>> >> >
> >>> >> > 2016-07-20 08:00:48,445 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler1) [1cff1b56] Fetched 7 VMs from VDS
> >>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'
> >>> >> > 2016-07-20 08:01:03,460 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler3) [78d8aa89] Fetched 0 VMs from VDS
> >>> >> > '6f7a9201-e753-4875-b89d-024120067687'
> >>> >> > 2016-07-20 08:01:03,473 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler2) [71a27a96] Fetched 7 VMs from VDS
> >>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'
> >>> >> > 2016-07-20 08:01:18,488 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler2) [71a27a96] Fetched 0 VMs from VDS
> >>> >> > '6f7a9201-e753-4875-b89d-024120067687'
> >>> >> > 2016-07-20 08:01:18,500 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler9) [47c419da] Fetched 7 VMs from VDS
> >>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'
> >>> >> > 2016-07-20 08:01:33,514 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler8) [735789] Fetched 0 VMs from VDS
> >>> >> > '6f7a9201-e753-4875-b89d-024120067687'
> >>> >> > 2016-07-20 08:01:33,527 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler3) [78d8aa89] Fetched 7 VMs from VDS
> >>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'
> >>> >> > 2016-07-20 08:01:48,543 INFO
> >>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]
> >>> >> > (DefaultQuartzScheduler8) [735789] Fetched 0 VMs from VDS
> >>> >> > '6f7a9201-e753-4875-b89d-024120067687'
> >>> >> >
> >>> >> >
> >>> >> > Greetings
> >>> >> >    Christoph
> >>> >> >
> >>> >> > On Thu, Jul 21, 2016 at 8:47 AM, Yaniv Kaul <ykaul at redhat.com>
> wrote:
> >>> >> >>
> >>> >> >> Can you share engine and VDSM logs please?
> >>> >> >> TIA,
> >>> >> >> Y.
> >>> >> >>
> >>> >> >> On Thu, Jul 21, 2016 at 9:22 AM, C. Handel <ovirt at macht-blau.org
> >
> >>> >> >> wrote:
> >>> >> >>>
> >>> >> >>> i have an ovirt 4.0.1 installation on two nodes. Hosted Engine.
> San
> >>> >> >>> storage backend.
> >>> >> >>>
> >>> >> >>> For some reason the vdsmd on the nodes is logging an error every
> >>> >> >>> few
> >>> >> >>> seconds:
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> vdsm vds.dispatcher ERROR SSL error during reading data:
> unexpected
> >>> >> >>> eof
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> Running tcpdump it is a connect from the node to itself. I can't
> >>> >> >>> figure
> >>> >> >>> out what is wrong. Can someone give me a hint?
> >>> >> >>>
> >>> >> >>> Greetings
> >>> >> >>>    Christoph
> >>> >> >>>
> >>> >> >>> _______________________________________________
> >>> >> >>> Users mailing list
> >>> >> >>> Users at ovirt.org
> >>> >> >>> http://lists.ovirt.org/mailman/listinfo/users
> >>> >> >>>
> >>> >> >>
> >>> >> >
> >>> >> >
> >>> >> > _______________________________________________
> >>> >> > Users mailing list
> >>> >> > Users at ovirt.org
> >>> >> > http://lists.ovirt.org/mailman/listinfo/users
> >>> >> >
> >>> >
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Users mailing list
> >>> > Users at ovirt.org
> >>> > http://lists.ovirt.org/mailman/listinfo/users
> >>> >
> >>
> >>
> >>
> >> _______________________________________________
> >> Users mailing list
> >> Users at ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/users
> >>
> > _______________________________________________
> > Users mailing list
> > Users at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160726/7a1410a3/attachment-0001.html>