This is a multi-part message in MIME format. --------------AF2BBEDF3BE980A51607346C Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit it worked for me as well - load avg. < 1 now, ovirt-ha-agent pops up in top periodically, but not on top using 100% CPU all the time anymore. thanks! sam Gianluca Cecchi wrote on 10/7/2016 10:13 AM:
On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi <stirabos@redhat.com <mailto:stirabos@redhat.com>> wrote:
On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi <gianluca.cecchi@gmail.com <mailto:gianluca.cecchi@gmail.com>> wrote:
On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer <nsoffer@redhat.com <mailto:nsoffer@redhat.com>> wrote:
wasn’t it suppose to be fixed to reuse the connection? Like all the other clients (vdsm migration code:-)
This is orthogonal issue.
Does schema validation matter then if there would be only one connection at the start up?
Loading once does not help command line tools like vdsClient, hosted-engine and vdsm-tool.
Nir
Simone, reusing the connection is good idea anyway, but what you describe is a bug in the client library. The library does *not* need to load and parse the schema at all for sending requests to vdsm.
The schema is only needed if you want to verify request parameters, or provide online help, these are not needed in a client library.
Please file an infra bug about it.
Done, https://bugzilla.redhat.com/show_bug.cgi?id=1381899 <https://bugzilla.redhat.com/show_bug.cgi?id=1381899>
Here is a patch that should eliminate most most of the problem: https://gerrit.ovirt.org/65230
Would be nice if it can be tested on the system showing this problem.
Cheers, Nir _______________________________________________
this is a video of 1 minute with the same system as the first post, but in 4.0.3 now and the same 3 VMs powered on without any particular load. It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.
https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharin... <https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharing>
Enjoy Nir ;-)
If I can apply the patch also to 4.0.3 I'm going to see if there is then a different behavior. Let me know,
I'm trying it right now. Any other tests will be really appreciated.
The patch is pretty simply, you can apply that on the fly. You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could directly edit /usr/lib/python2.7/site-packages/api/vdsmapi.py around line 97 changing from loaded_schema = yaml.load(f) to loaded_schema = yaml.load(f, Loader=yaml.CLoader) Please pay attention to keep exactly the same amount of initial spaces.
Then you can simply restart the HA agent and check.
Gianluca
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users <http://lists.ovirt.org/mailman/listinfo/users>
What I've done (I didn't read your answer in between and this is a test system not so important... )
set to global maintenance patch vdsmapi.py restart vdsmd restart ovirt-ha-agent set maintenance to none
And a bright new 3-minutes video here: https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharin...
It seems that now ovirt-ha-agent or is not present in top cpu process or at least has ranges between 5% and 12% and not more....
BTW: I see this in vdsm status now
[root@ovirt01 api]# systemctl status vdsmd ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS) Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 21023 (vdsm) CGroup: /system.slice/vdsmd.service ├─21023 /usr/bin/python /usr/share/vdsm/vdsm ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 --write-pipe-fd 40 --max-threads 10 --max-queue... ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd 46 --max-threads 10 --max-queue... ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 --write-pipe-fd 56 --max-threads 10 --max-queue... ├─21143 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd 64 --max-threads 10 --max-queue... ├─21149 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd 72 --max-threads 10 --max-queue... ├─21156 /usr/libexec/ioprocess --read-pipe-fd 80 --write-pipe-fd 78 --max-threads 10 --max-queue... ├─21177 /usr/libexec/ioprocess --read-pipe-fd 88 --write-pipe-fd 87 --max-threads 10 --max-queue... ├─21204 /usr/libexec/ioprocess --read-pipe-fd 99 --write-pipe-fd 98 --max-threads 10 --max-queue... └─21239 /usr/libexec/ioprocess --read-pipe-fd 111 --write-pipe-fd 110 --max-threads 10 --max-que...
Oct 07 16:02:52 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:02:54 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:02:56 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:02:58 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:03:11 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:03:15 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:03:15 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:03:18 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:03:20 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Oct 07 16:03:22 ovirt01.lutwyn.org <http://ovirt01.lutwyn.org> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof Hint: Some lines were ellipsized, use -l to show in full. [root@ovirt01 api]#
Now I also restarted ovirt-ha-broker just in case
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--------------AF2BBEDF3BE980A51607346C Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=UTF-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> it worked for me as well - load avg. < 1 now, ovirt-ha-agent pops up in top periodically, but not on top using 100% CPU all the time anymore.<br> thanks!<br> sam<br> <br> <div class="moz-cite-prefix">Gianluca Cecchi wrote on 10/7/2016 10:13 AM:<br> </div> <blockquote cite="mid:CAG2kNCzOviMzgm=NYArWKROQen5BJDudQ0AES8xr+qvqAHFzMg@mail.gmail.com" type="cite"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote">On Fri, Oct 7, 2016 at 3:35 PM, Simone Tiraboschi <span dir="ltr"><<a moz-do-not-send="true" href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"><br> <div class="gmail_extra"><br> <div class="gmail_quote"> <div> <div class="gmail-h5">On Fri, Oct 7, 2016 at 3:22 PM, Gianluca Cecchi <span dir="ltr"><<a moz-do-not-send="true" href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"><span class="gmail-m_2782938339651103979gmail-"> <div class="gmail_extra"> <div class="gmail_quote">On Fri, Oct 7, 2016 at 2:59 PM, Nir Soffer <span dir="ltr"><<a moz-do-not-send="true" href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"><span> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div style="word-wrap:break-word"> <div><span> <div><br> </div> </span>wasn’t it suppose to be fixed to reuse the connection? Like all the other clients (vdsm migration code:-) </div> </div> </blockquote> <div><br> </div> </span> <div>This is orthogonal issue.</div> <span> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div style="word-wrap:break-word"> <div>Does schema validation matter then if there would be only one connection at the start up?</div> </div> </blockquote> <div><br> </div> </span> <div>Loading once does not help command line tools like vdsClient, hosted-engine and</div> <div>vdsm-tool. </div> <span><font color="#888888"> <div><br> </div> <div>Nir</div> </font></span><span> <div> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div style="word-wrap:break-word"> <div><br> <blockquote type="cite"> <div><span> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"><span> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra"> <div class="gmail_quote"><span> <div><br> </div> </span> <div>Simone, reusing the connection is good idea anyway, but what you describe is </div> <div>a bug in the client library. The library does *not* need to load and parse the</div> <div>schema at all for sending requests to vdsm.</div> <div><br> </div> <div>The schema is only needed if you want to verify request parameters,</div> <div>or provide online help, these are not needed in a client library.</div> <div><br> </div> <div>Please file an infra bug about it.</div> </div> </div> </div> </blockquote> <div><br> </div> </span> <div>Done, <a moz-do-not-send="true" href="https://bugzilla.redhat.com/show_bug.cgi?id=1381899" target="_blank">https://bugzilla.redhat.com/sh<wbr>ow_bug.cgi?id=1381899</a></div> </div> </div> </div> </blockquote> <div><br> </div> <div>Here is a patch that should eliminate most most of the problem:</div> <div><a moz-do-not-send="true" href="https://gerrit.ovirt.org/65230" target="_blank">https://gerrit.ovirt.org/65230</a><br> </div> <div><br> </div> <div>Would be nice if it can be tested on the system showing this problem.</div> <div><br> </div> <div>Cheers,</div> <div>Nir</div> </div> </div> </div> </span><span> ______________________________<wbr>_________________<br> </span></div> </blockquote> </div> </div> </blockquote> </span></div> </div> </div> </blockquote> </div> <br> </div> <div class="gmail_extra"><br> </div> </span> <div class="gmail_extra">this is a video of 1 minute with the same system as the first post, but in 4.0.3 now and the same 3 VMs powered on without any particular load.</div> <div class="gmail_extra">It seems very similar to the previous 3.6.6 in cpu used by ovirt-ha-agent.</div> <div class="gmail_extra"><br> </div> <div class="gmail_extra"><a moz-do-not-send="true" href="https://drive.google.com/file/d/0BwoPbcrMv8mvSjFDUERzV1owTG8/view?usp=sharin..." target="_blank">https://drive.google.com/file/<wbr>d/0BwoPbcrMv8mvSjFDUERzV1owTG8<wbr>/view?usp=sharing</a><br> </div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">Enjoy Nir ;-)<br> </div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">If I can apply the patch also to 4.0.3 I'm going to see if there is then a different behavior. </div> <div class="gmail_extra">Let me know,</div> <div class="gmail_extra"><br> </div> </div> </blockquote> <div><br> </div> </div> </div> <div>I'm trying it right now.</div> <div>Any other tests will be really appreciated.</div> <div><br> </div> <div>The patch is pretty simply, you can apply that on the fly.</div> <div>You have to shutdown ovirt-ha-broker and ovirt-ha-agent; then you could directly edit </div> <div>/usr/lib/python2.7/site-<wbr>packages/api/vdsmapi.py<br> </div> <div>around line 97 changing from</div> <div>loaded_schema = yaml.load(f)<br> </div> <div>to</div> <div>loaded_schema = yaml.load(f, Loader=yaml.CLoader)<br> </div> <div>Please pay attention to keep exactly the same amount of initial spaces.<br> </div> <div><br> </div> <div>Then you can simply restart the HA agent and check.</div> <div><br> </div> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div dir="ltr"> <div class="gmail_extra">Gianluca</div> </div> <span class="gmail-"> <br> ______________________________<wbr>_________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> <br> </span></blockquote> </div> <br> </div> </div> </blockquote> </div> <br> </div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">What I've done (I didn't read your answer in between and this is a test system not so important... )</div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">set to global maintenance</div> <div class="gmail_extra">patch vdsmapi.py</div> <div class="gmail_extra">restart vdsmd </div> <div class="gmail_extra">restart ovirt-ha-agent</div> <div class="gmail_extra">set maintenance to none</div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">And a bright new 3-minutes video here:</div> <div class="gmail_extra"><a moz-do-not-send="true" href="https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing">https://drive.google.com/file/d/0BwoPbcrMv8mvVzBPUVRQa1pwVnc/view?usp=sharing</a><br> </div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">It seems that now ovirt-ha-agent or is not present in top cpu process or at least has ranges between 5% and 12% and not more....</div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">BTW: I see this in vdsm status now </div> <div class="gmail_extra"><br> </div> <div class="gmail_extra"> <div class="gmail_extra">[root@ovirt01 api]# systemctl status vdsmd </div> <div class="gmail_extra">● vdsmd.service - Virtual Desktop Server Manager</div> <div class="gmail_extra"> Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)</div> <div class="gmail_extra"> Active: active (running) since Fri 2016-10-07 15:30:57 CEST; 32min ago</div> <div class="gmail_extra"> Process: 20883 ExecStopPost=/usr/libexec/vdsm/vdsmd_init_common.sh --post-stop (code=exited, status=0/SUCCESS)</div> <div class="gmail_extra"> Process: 20886 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)</div> <div class="gmail_extra"> Main PID: 21023 (vdsm)</div> <div class="gmail_extra"> CGroup: /system.slice/vdsmd.service</div> <div class="gmail_extra"> ├─21023 /usr/bin/python /usr/share/vdsm/vdsm</div> <div class="gmail_extra"> ├─21117 /usr/libexec/ioprocess --read-pipe-fd 41 --write-pipe-fd 40 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> ├─21123 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd 46 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> ├─21134 /usr/libexec/ioprocess --read-pipe-fd 57 --write-pipe-fd 56 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> ├─21143 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd 64 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> ├─21149 /usr/libexec/ioprocess --read-pipe-fd 73 --write-pipe-fd 72 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> ├─21156 /usr/libexec/ioprocess --read-pipe-fd 80 --write-pipe-fd 78 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> ├─21177 /usr/libexec/ioprocess --read-pipe-fd 88 --write-pipe-fd 87 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> ├─21204 /usr/libexec/ioprocess --read-pipe-fd 99 --write-pipe-fd 98 --max-threads 10 --max-queue...</div> <div class="gmail_extra"> └─21239 /usr/libexec/ioprocess --read-pipe-fd 111 --write-pipe-fd 110 --max-threads 10 --max-que...</div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">Oct 07 16:02:52 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:02:54 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:02:56 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:02:58 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:03:11 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:03:15 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:03:15 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:03:18 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:03:20 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Oct 07 16:03:22 <a moz-do-not-send="true" href="http://ovirt01.lutwyn.org">ovirt01.lutwyn.org</a> vdsm[21023]: vdsm vds.dispatcher ERROR SSL error during reading data... eof</div> <div class="gmail_extra">Hint: Some lines were ellipsized, use -l to show in full.</div> <div class="gmail_extra">[root@ovirt01 api]# </div> <div class="gmail_extra"><br> </div> <div class="gmail_extra">Now I also restarted ovirt-ha-broker just in case</div> <div class="gmail_extra"><br> </div> </div> </div> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> </body> </html> --------------AF2BBEDF3BE980A51607346C--