<div dir="ltr"><div><div><div>Christoph,<br><br></div>Please apply [1] we would know exactly what verbs were called by this local client.<br><br></div>Thanks,<br></div>Piotr<br><div><div><br>[1] <a href="https://gerrit.ovirt.org/#/c/61367/">https://gerrit.ovirt.org/#/c/61367/</a><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 25, 2016 at 9:50 PM, Piotr Kliczewski <span dir="ltr"><<a href="mailto:piotr.kliczewski@gmail.com" target="_blank">piotr.kliczewski@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Mon, Jul 25, 2016 at 6:18 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br>
> On Mon, Jul 25, 2016 at 6:22 PM, C. Handel <<a href="mailto:ovirt@macht-blau.org">ovirt@macht-blau.org</a>> wrote:<br>
>> two nodes (x.x.138.208, x.x.138.210), hosted-engine on x.x.139.240.<br>
>><br>
>> the vdsm logs are from x.x.138.208 and the connection is from the node<br>
>> itself.<br>
>><br>
>>> Running tcpdump it is a connect from the node to itself. I can't figure<br>
>>> out what is wrong. Can someone ?> give me a hint?<br>
>><br>
>> so i know they are coming from the node itself. The process connecting is<br>
>> terminating too fast. the moment i see it in tcpdump, it is gone from the<br>
>> process table.<br>
><br>
> I think this is ovirt hosted agent - check its logs, you will probably find that<br>
> it make some request in the same time you see the errors in your logs.<br>
><br>
> Adding Martin, maintaining this project.<br>
><br>
> These are the interesting events in the logs:<br>
><br>
> $ grep 37678 vdsm.log<br>
> JsonRpc (StompReactor)::ERROR::2016-07-25<br>
> 13:48:58,074::betterAsyncore::113::vds.dispatcher::(recv) SSL error<br>
> during reading data from <yajsonrpc.betterAsyncore.Dispatcher<br>
> connected ('::ffff:140.181.138.208', 37678, 0, 0) at 0x42c9b90>:<br>
> unexpected eof<br>
><br>
> $ grep 37684 vdsm.log<br>
> Reactor thread::INFO::2016-07-25<br>
> 13:49:00,205::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)<br>
> Accepting connection from ::ffff:<a href="http://140.181.138.208:37684" rel="noreferrer" target="_blank">140.181.138.208:37684</a><br>
> Reactor thread::INFO::2016-07-25<br>
> 13:49:00,211::protocoldetector::121::ProtocolDetector.Detector::(handle_read)<br>
> Detected protocol stomp from ::ffff:<a href="http://140.181.138.208:37684" rel="noreferrer" target="_blank">140.181.138.208:37684</a><br>
> Reactor thread::DEBUG::2016-07-25<br>
> 13:49:00,211::stompreactor::492::protocoldetector.StompDetector::(handle_socket)<br>
> Stomp detected from ('::ffff:140.181.138.208', 37684)<br>
> JsonRpc (StompReactor)::ERROR::2016-07-25<br>
> 13:49:01,824::betterAsyncore::113::vds.dispatcher::(recv) SSL error<br>
> during reading data from <yajsonrpc.betterAsyncore.Dispatcher<br>
> connected ('::ffff:140.181.138.208', 37684, 0, 0) at 0x42b0758>:<br>
> unexpected eof<br>
><br>
> The log is too small, we see only one full request.<br>
><br>
> Pitor, can you understand from this log what the request coming from<br>
> ::ffff:<a href="http://140.181.138.208:37684" rel="noreferrer" target="_blank">140.181.138.208:37684</a><br>
> is doing?<br>
><br>
<br>
</div></div>I stated above there are 2 verbs that were called each time:<br>
<br>
Host.getStats and Host.getHardwareInfo<br>
<div class="HOEnZb"><div class="h5"><br>
>><br>
>> Greetings<br>
>> Christoph<br>
>><br>
>> On Mon, Jul 25, 2016 at 4:53 PM, Piotr Kliczewski<br>
>> <<a href="mailto:piotr.kliczewski@gmail.com">piotr.kliczewski@gmail.com</a>> wrote:<br>
>>><br>
>>> Christoph,<br>
>>><br>
>>> In log snippets you provided I can see 2 occurrences of the log entry.<br>
>>> There is 3 seconds between the calls.<br>
>>><br>
>>> Each time I see calls to Host.getStats and Host.getHardwareInfo both<br>
>>> from x.x.138.208.<br>
>>> I do not see any log entries in the engine log so it not engine who<br>
>>> connected.<br>
>>><br>
>>> What host is it?<br>
>>><br>
>>> Thanks,<br>
>>> Piotr<br>
>>><br>
>>><br>
>>> On Mon, Jul 25, 2016 at 3:45 PM, C. Handel <<a href="mailto:ovirt@macht-blau.org">ovirt@macht-blau.org</a>> wrote:<br>
>>> > patch applied, The connection is from the node itself.<br>
>>> ><br>
>>> > logfiles with the last 100k (hope this is enough, the error happens<br>
>>> > every<br>
>>> > few seconds) of data attached.<br>
>>> ><br>
>>> > Greetings<br>
>>> > Christoph<br>
>>> ><br>
>>> > On Mon, Jul 25, 2016 at 12:07 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br>
>>> >><br>
>>> >> On Thu, Jul 21, 2016 at 10:00 AM, C. Handel <<a href="mailto:ovirt@macht-blau.org">ovirt@macht-blau.org</a>><br>
>>> >> wrote:<br>
>>> >> > longer logs attached, excerpts:<br>
>>> >> ><br>
>>> >> > ---+ vdsm<br>
>>> >> ><br>
>>> >> > Reactor thread::INFO::2016-07-21<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > 08:01:19,544::protocoldetector::72::ProtocolDetector.AcceptorImpl::(handle_accept)<br>
>>> >> > Accepting connection from ::ffff:<a href="http://140.181.138.208:59502" rel="noreferrer" target="_blank">140.181.138.208:59502</a><br>
>>> >> > Reactor thread::DEBUG::2016-07-21<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > 08:01:19,551::protocoldetector::85::ProtocolDetector.Detector::(__init__)<br>
>>> >> > Using required_size=11<br>
>>> >> > Reactor thread::INFO::2016-07-21<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > 08:01:19,553::protocoldetector::121::ProtocolDetector.Detector::(handle_read)<br>
>>> >> > Detected protocol stomp from ::ffff:<a href="http://140.181.138.208:59502" rel="noreferrer" target="_blank">140.181.138.208:59502</a><br>
>>> >> > Reactor thread::INFO::2016-07-21<br>
>>> >> > 08:01:19,553::stompreactor::101::Broker.StompAdapter::(_cmd_connect)<br>
>>> >> > Processing CONNECT request<br>
>>> >> > Reactor thread::DEBUG::2016-07-21<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > 08:01:19,554::stompreactor::492::protocoldetector.StompDetector::(handle_socket)<br>
>>> >> > Stomp detected from ('::ffff:140.181.138.208', 59502)<br>
>>> >> > JsonRpc (StompReactor)::INFO::2016-07-21<br>
>>> >> ><br>
>>> >> > 08:01:19,554::stompreactor::128::Broker.StompAdapter::(_cmd_subscribe)<br>
>>> >> > Subscribe command received<br>
>>> >> > ...<br>
>>> >> ><br>
>>> >> > JsonRpc (StompReactor)::ERROR::2016-07-21<br>
>>> >> > 08:01:21,411::betterAsyncore::113::vds.dispatcher::(recv) SSL error<br>
>>> >> > during<br>
>>> >> > reading data: unexpected eof<br>
>>> >><br>
>>> >> This log is not very useful as is, we must show the relevant remote<br>
>>> >> address.<br>
>>> >><br>
>>> >> Should be improved in<br>
>>> >> <a href="https://gerrit.ovirt.org/61303" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/61303</a><br>
>>> >><br>
>>> >> Can you try this patch and share the log?<br>
>>> >><br>
>>> >> ><br>
>>> >> ><br>
>>> >> > ---+ engine<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > 2016-07-20 08:00:48,445 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler1) [1cff1b56] Fetched 7 VMs from VDS<br>
>>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'<br>
>>> >> > 2016-07-20 08:01:03,460 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler3) [78d8aa89] Fetched 0 VMs from VDS<br>
>>> >> > '6f7a9201-e753-4875-b89d-024120067687'<br>
>>> >> > 2016-07-20 08:01:03,473 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler2) [71a27a96] Fetched 7 VMs from VDS<br>
>>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'<br>
>>> >> > 2016-07-20 08:01:18,488 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler2) [71a27a96] Fetched 0 VMs from VDS<br>
>>> >> > '6f7a9201-e753-4875-b89d-024120067687'<br>
>>> >> > 2016-07-20 08:01:18,500 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler9) [47c419da] Fetched 7 VMs from VDS<br>
>>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'<br>
>>> >> > 2016-07-20 08:01:33,514 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler8) [735789] Fetched 0 VMs from VDS<br>
>>> >> > '6f7a9201-e753-4875-b89d-024120067687'<br>
>>> >> > 2016-07-20 08:01:33,527 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler3) [78d8aa89] Fetched 7 VMs from VDS<br>
>>> >> > 'a46f9de2-4b79-45d4-bed5-f4f472eb8361'<br>
>>> >> > 2016-07-20 08:01:48,543 INFO<br>
>>> >> > [org.ovirt.engine.core.vdsbroker.monitoring.VmsStatisticsFetcher]<br>
>>> >> > (DefaultQuartzScheduler8) [735789] Fetched 0 VMs from VDS<br>
>>> >> > '6f7a9201-e753-4875-b89d-024120067687'<br>
>>> >> ><br>
>>> >> ><br>
>>> >> > Greetings<br>
>>> >> > Christoph<br>
>>> >> ><br>
>>> >> > On Thu, Jul 21, 2016 at 8:47 AM, Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>> wrote:<br>
>>> >> >><br>
>>> >> >> Can you share engine and VDSM logs please?<br>
>>> >> >> TIA,<br>
>>> >> >> Y.<br>
>>> >> >><br>
>>> >> >> On Thu, Jul 21, 2016 at 9:22 AM, C. Handel <<a href="mailto:ovirt@macht-blau.org">ovirt@macht-blau.org</a>><br>
>>> >> >> wrote:<br>
>>> >> >>><br>
>>> >> >>> i have an ovirt 4.0.1 installation on two nodes. Hosted Engine. San<br>
>>> >> >>> storage backend.<br>
>>> >> >>><br>
>>> >> >>> For some reason the vdsmd on the nodes is logging an error every<br>
>>> >> >>> few<br>
>>> >> >>> seconds:<br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>> vdsm vds.dispatcher ERROR SSL error during reading data: unexpected<br>
>>> >> >>> eof<br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>> Running tcpdump it is a connect from the node to itself. I can't<br>
>>> >> >>> figure<br>
>>> >> >>> out what is wrong. Can someone give me a hint?<br>
>>> >> >>><br>
>>> >> >>> Greetings<br>
>>> >> >>> Christoph<br>
>>> >> >>><br>
>>> >> >>> _______________________________________________<br>
>>> >> >>> Users mailing list<br>
>>> >> >>> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> >> >>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>>> >> >>><br>
>>> >> >><br>
>>> >> ><br>
>>> >> ><br>
>>> >> > _______________________________________________<br>
>>> >> > Users mailing list<br>
>>> >> > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> >> > <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>>> >> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > Users mailing list<br>
>>> > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>>> > <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>>> ><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Users mailing list<br>
>> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
</div></div></blockquote></div><br></div>