vdsm using 100% CPU, rapidly filling logs with _handle_event messages
--Sig_/War1DBdDuKL/JOwM_gN0y1g Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable I'm running oVirt 3.5.x with a hosted engine. This morning I noticed that 2 of my 5 hosts were showing 99-100% cpu usage. Logging in to them, vdsmd seemed to be the culprit, and it was filling the log file with these messages: VM Channels Listener::DEBUG::2015-11-12 08:09:26,292::vmchannels::59::vds::(_handle_event) Received 00000011. On fd= removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmc= hannels::54::vds::(_handle_event) Received 00000011 on fileno 119 VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::= (_handle_event) Received 00000011. On fd removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmchannels::54::vds::(= _handle_event) Received 00000011 on fileno 75 VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::= (_handle_event) Received 00000011. On fd removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,294::vmchannels::54::vds::(= _handle_event) Received 00000011 on fileno 119 VM Channels Listener::DEBUG::2015-11-12 08:09:26,294::vmchannels::59::vds::= (_handle_event) Received 00000011. On fd removed by epoll. I googled to see how to change the debug level to turn of DEBUG messages for vdsm, which referred me to libvirtd.conf, but the debug level there was not set, which should have meant a log level of 3 (warnings and errors), so I'm not sure why the log was filling up with DEBUG/INFO messages. I restarted vdsmd, which resulted in those nodes being marked as 'disconnected', but they did eventually recover and loads went back to normal. This may or may not be related to the fact that the 3 hosts where this did not happen can't seem to keep their ha brokers up. I'll be starting a new thread on that shortly. Robert --=20 Senior Software Engineer @ Parsons --Sig_/War1DBdDuKL/JOwM_gN0y1g Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWRJgKAAoJEMHFVuy5l8Y4SGcP/jvgozmL+MsLRWFK+hWBa4+8 lP4TVCWdDqdd/3Kt3ZIO7vNBCEWNmicbtOu6o6yeG1o4l5OVh0GG+wspQOHm5D7g ScfsQ405e4EP3skKx4yauHmM8kxBuZo2adRj1CT+mbB/xXMfUd/gcVWgFYuPMMuN EkJR2Q9NqpP1ccUduccAMkiXSyRbUt6Ng7iZqx2RhDAdnRW2CYUCSqkSxI7KE/Mw NZQ4bGAl4w6eJwBxV9jB6eDW6RD2pGmncRswq7TGv8+iGvLd+XAxhQRviIJYeaFH 8GTil9/qEW8wzmbEYsd+YzbwAaPuveKx9TZCTVyhgabzeipLv0Xj4eKMO8fJ66kB s+MYZSeIMDAGnDoHdFvf8hVphb+DHoGOZXx+C76Ohfwb+WL+ZDCIAra/lLx57CUa H+JPU9pl4Bn3f0o3srX6Mkbj8qljXHGXCKI206SoYfgMyRlN9/VylpVrkryP3m2d 8CXK0X1+bxnvohelZGXlIca7t7/eTVd9Vob2aBKjvUkGLqQ920fXxVRqI78Rpmdd druse94ZN8KYk2NOMaM0YuZERlkJP8aqQAYNqGwZ6QgIPuYGyEeY4pnJWpH+Z0sF g9CUW9AblprC1E668qh4gRrKv7a3VWpkQyZNJTaIVxfJt97oXIbU0RtFQJEF5o2n 7kSRCB0xu7vyJEw32O28 =NjlM -----END PGP SIGNATURE----- --Sig_/War1DBdDuKL/JOwM_gN0y1g--
On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote:
I'm running oVirt 3.5.x with a hosted engine. This morning I noticed that 2 of my 5 hosts were showing 99-100% cpu usage. Logging in to them, vdsmd seemed to be the culprit, and it was filling the log file with these messages:
You're probably seeing Bug 1226911 - vmchannel thread consumes 100% of CPU which was closed due to missing information. Do you have any information on when this pops up? Is it reproducible? Would you be bale to test a suggested patch https://gerrit.ovirt.org/#/c/42570/ Regards, Dan.
VM Channels Listener::DEBUG::2015-11-12 08:09:26,292::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 119 VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 75 VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,294::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 119 VM Channels Listener::DEBUG::2015-11-12 08:09:26,294::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by epoll.
I googled to see how to change the debug level to turn of DEBUG messages for vdsm, which referred me to libvirtd.conf, but the debug level there was not set, which should have meant a log level of 3 (warnings and errors), so I'm not sure why the log was filling up with DEBUG/INFO messages.
I restarted vdsmd, which resulted in those nodes being marked as 'disconnected', but they did eventually recover and loads went back to normal.
This may or may not be related to the fact that the 3 hosts where this did not happen can't seem to keep their ha brokers up. I'll be starting a new thread on that shortly.
Robert
-- Senior Software Engineer @ Parsons
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
--Sig_/.YuOUBlg7p5t+Kk.wNQXWNo Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote: DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote: DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I noticed DK> > that 2 of my 5 hosts were showing 99-100% cpu usage. Logging in to DK> > them, vdsmd seemed to be the culprit, and it was filling the log file DK> > with these messages: DK>=20 DK> You're probably seeing DK> Bug 1226911 - vmchannel thread consumes 100% of CPU DK>=20 DK> which was closed due to missing information. Do you have any information DK> on when this pops up? Is it reproducible? Would you be bale to test a DK> suggested patch DK>=20 DK> https://gerrit.ovirt.org/#/c/42570/ Hi Dan, Thanks for the pointers. If it comes up again, I'll try this patch and report back on the bug... Robert --=20 Senior Software Engineer @ Parsons --Sig_/.YuOUBlg7p5t+Kk.wNQXWNo Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWRJ9IAAoJEMHFVuy5l8Y4W04QAI+1DK37aNPX9nLMcNthF6BQ 0xgNKhzlE82iwd4f0Oflp/GARW8XeKwPSdcqBlfuIkVy5jxXKyRBriz44/dU1V2/ 90qV265Sa6y/PSH6B1PwF2sWyjDJsL3KPF9uH2OO5jLN+Q6pSQRSowzodI3UtZG2 ZUZuPsqOmSD2LK0DVc+EMB2MMF002PKArd6+i8PUvAuaCNLrV8VKjymOZLRr6J81 hG9HKpu1980TWi7u9ozr6h+CCdef28REreY0Ri9LxEvL7OYuvhSvt5/amIGgdK3c PhiUB/GIS9nNEccnnVILK3+qfeli91zfjGGaewUFn4GrgyAAFcQ24gHbiUhYDfFX Z0Qu5Te8dvfgFiDTQI6Z471y0tI77y6/S2ArSvasMEyJ19B0xNVfO6M9HwfbcVTy mBPZEo4cjox17Ps+kpSG3PK1U3cF0+oK2T37agc3LxXDK+SzA8FBPRIrKipVfjRQ Oca/isGVxPQptoIGUT4tH1NuTuOpz7/2oyD5xYuURxh28RlteLeV/j2oUUj8j4uo IS6helL9V8s4Wx66wcICBi8vSF4jP/W2NLmKaVHBl5TkzgfPxBNvsOFjsXjBpCoM RwR/oCVcFm3SPUxkH1bFsMCB6oB9R/WRrlUaMpQvE/fPNpzVEvsnQr3L80JEh+Gt JzoGRG2JhWdvyUUqYJ3x =6cat -----END PGP SIGNATURE----- --Sig_/.YuOUBlg7p5t+Kk.wNQXWNo--
Hi Robert On 11/12/2015 03:16 PM, Robert Story wrote:
On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote: DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote: DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I noticed DK> > that 2 of my 5 hosts were showing 99-100% cpu usage. Logging in to DK> > them, vdsmd seemed to be the culprit, and it was filling the log file DK> > with these messages: DK> DK> You're probably seeing DK> Bug 1226911 - vmchannel thread consumes 100% of CPU DK> DK> which was closed due to missing information. Do you have any information DK> on when this pops up? Is it reproducible? Would you be bale to test a DK> suggested patch DK> DK> https://gerrit.ovirt.org/#/c/42570/
Hi Dan,
Thanks for the pointers. If it comes up again, I'll try this patch and report back on the bug... Out of curiosity, did you happen again to see that happening again?
Robert
-- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
--Sig_/GG.=3/ec7hZy8d4QIAOi9xR Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Wed, 18 Nov 2015 12:35:17 +0100 Vinzenz wrote: VF> On 11/12/2015 03:16 PM, Robert Story wrote: VF> > On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote: VF> > DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote: VF> > DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I VF> > DK> > noticed that 2 of my 5 hosts were showing 99-100% cpu usage. VF> > DK> > Logging in to them, vdsmd seemed to be the culprit, and it was VF> > DK> > filling the log file with these messages: VF> > DK> VF> > DK> You're probably seeing VF> > DK> Bug 1226911 - vmchannel thread consumes 100% of CPU VF> > DK> VF> > DK> which was closed due to missing information. Do you have any VF> > DK> information on when this pops up? Is it reproducible? Would you VF> > DK> be bale to test a suggested patch VF> > DK> VF> > DK> https://gerrit.ovirt.org/#/c/42570/ VF> > VF> > Hi Dan, VF> > VF> > Thanks for the pointers. If it comes up again, I'll try this patch and VF> > report back on the bug... VF> > VF> Out of curiosity, did you happen again to see that happening again? No, I have not. Robert --=20 Senior Software Engineer @ Parsons --Sig_/GG.=3/ec7hZy8d4QIAOi9xR Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWTG7sAAoJEMHFVuy5l8Y4R1QQAJwYgtLUXuKNMRsb3jPFtIPB W6T4T8becETOsGOetqwcADoilOmk+EnCMZ+G9USF7dqtghhIGbj6Up+h40rnd4V1 OLxn56j+oyj5Md3YoarIGVjubkFaPc7zVohMPjsF2PHwgyfbxm+EbFr0x9XP4l5K NteGICiSIHuA+oeseuObVFSv/C+7dNEtyQytE5HVgQmzgBzmHPxkWGnr1aJRqYQO fv8ImYN/fNTbMOs6q7nvpkTCUkgzwvVAPeoGK1Yq1Ab5avfgY+MIATaxxePIObyD TiEocRM+Giwqb4xj8tW3pZorssSDoGcfnKUHZIJsc0sjsXD7H+f2NokkVg1lmLKL 3Zp42llrt9hbmwnsoA6ZKlt9XR0JcBm7ff5rfx0Mx72BiBLWh2P4OmAb05rCyTvK v2mBn39VliLdtot1E7XIPUk9fcePMu8xvvfbYLO8oUGC85ZzzObnvWx9j36DthIZ dVuP3F5+2eor9ndSdVmLPFdoSKKSALPyIOTNj9tSTWk82/DHqT7Ps/6qXFeZg1sX 3pKGQ3l0T+1IcNAowARdQ4W4p6T8SIAtzDPE8WXef1nXDKP5HvWyD9Dk3KEiIekF 8ffrSvocPT8TFFc37Xg9KULNqTAYCUg1FqfNugneYb6SCXtRBs4GiJ/E35dlciyx ZAmeDUNNvjevWyOVA15X =5Lj9 -----END PGP SIGNATURE----- --Sig_/GG.=3/ec7hZy8d4QIAOi9xR--
--Sig_/xM2/5weqzNCEiCQ1Zp6THsR Content-Type: multipart/mixed; boundary="MP_/IiROFVEHqzB/cq/fJkmmawn" --MP_/IiROFVEHqzB/cq/fJkmmawn Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Wed, 18 Nov 2015 07:28:23 -0500 Robert wrote: RS> On Wed, 18 Nov 2015 12:35:17 +0100 Vinzenz wrote: RS> VF> On 11/12/2015 03:16 PM, Robert Story wrote: RS> VF> > On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote: RS> VF> > DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote: RS> VF> > DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I RS> VF> > DK> > noticed that 2 of my 5 hosts were showing 99-100% cpu usage. RS> VF> > DK> > Logging in to them, vdsmd seemed to be the culprit, and it RS> VF> > DK> > was filling the log file with these messages: RS> VF> > DK> RS> VF> > DK> You're probably seeing RS> VF> > DK> Bug 1226911 - vmchannel thread consumes 100% of CPU RS> VF> > DK> RS> VF> > DK> which was closed due to missing information. Do you have any RS> VF> > DK> information on when this pops up? Is it reproducible? Would RS> VF> > DK> you be bale to test a suggested patch RS> VF> > DK> RS> VF> > DK> https://gerrit.ovirt.org/#/c/42570/ RS> VF> > RS> VF> > Hi Dan, RS> VF> > RS> VF> > Thanks for the pointers. If it comes up again, I'll try this RS> VF> > patch and report back on the bug... RS> VF> > RS> VF> Out of curiosity, did you happen again to see that happening again? RS>=20 RS> No, I have not. So naturally it shows up again during a holiday. I came in today to find 1 of my 5 nodes (the SPM and host where hosted engine is running) with two vdsmd threads chewing up 90-100% of the CPU. I applied the patch from above and restarted vdsmd. This resulted in another node being selected as the SPM, and within about 15 minutes, that node had the same issue. Applied the patch to the new node, and restarted vdsmd, and the SPM went back to the previous (now patched) node. Hopefully things will stay stable. I've attached snippets of the logs from the SPM when the problem started, along with the server/engine log snippets from the hosted engine around the same time.. Robert --=20 Senior Software Engineer @ Parsons --MP_/IiROFVEHqzB/cq/fJkmmawn Content-Type: application/x-compressed-tar Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=vdsm-runaway.tgz H4sIAK12WFYAA+w9aVPjSJb9uX+FvrmI8aFM3dqpjQBMVbPL1RiqZ6KH8KSllK1BV+swMB38932Z KdmSD7ApKJgFV1SVlcfLd7+XKel56mZhJy0ickPuej+9zEeGj6Fp7H9sKGr9/+rzE8I61mVDwSr6 SUYKRvgnSXshfBqfIstJKkk/pVkep3frxz3W/x/6mdblzy66QTzuIvk512AC1lV1jfwNrGpqJX8E XSB/DTH5PysS6z7vXP59mlMHKJPySUqJa9v9g73Lr7aNQQgdhDrYkGTZlhVb09oIqbadpHEeO3Hg lhP5UNsG1ekeF0Hun5X9u45DE97/aZjSMJ7SoRNHEczx42hH2p99l0SvK3lpHEoIG10Z/iBblVUZ /bwdftpK/FTjEfwmJHKDOn5DttaOJBYH1Cqg0m0YPAeeum3v+ZHrR+O/HR+dn+3bNkKGYtt/C4P+ DO9PAq1B7FzTfIcvDXxKpUmeJ5Jbocax+dSa4dNqSxyjnZ8vOCYdZMmqbqkPY6SgFRgpCucbYHKT kiSh6Y7kBD6Ncun32XJXtu2QIJCm4VeaD3KSZ9KNn08AI0OVLYods6MgXe6o6mjUsQzD6HjmiKqm qY8IcVvtHenP++0wxSswVeUlTFOaF2m0hNefLbD3vMhaNnwNaZaRMYXvrX4cUWBdy4lddi3fw3c2 Mjvysxwafv+z5fpZEpC7w8iLRUMeZGdxynpbHcb3lp/sum4KQFmTzFryu4SDn0YOu0zE8KKlWbLS ur+CppCGlxUSmsYGESfxDyIyCnhbnhYcsXFBs/zLr/0TPj+e+mneodHYj2g3onlGBXzfZXMUZKgq u84AF1Bn1nYZXUfxTcRaYfyhRxyacTKK1uSGwwQeg2Iq1FaJ7YzsEYNQtCISUt5N84nMW2BJzpGi heSugSBYdDHGrauqTxedHjVBKpiD9DzboxwsQL3idOd+SE89L+OwBLNGoEhxHJUM/pOJICFFRveF TFonpwfn52wgSOI6q7jGKEhIPuFI9jiGeZyTgM3AimZAfDEEO4oWAOMMwoapa5piajpv9jI++Rb+ h0Ub8EZxnDdhalg2LBUCVAMi0kxZ0SzdXAJ4JTh+E6fXjKoCdIHmqtA/4jCF4QxYwfxWettPY1Bn sQI2RdtBmsZpTcXqg8qWc5JzjoGJijkLDfkiEKbqfEAxV5MsoeXKEKW53lSqICi4vxfqC2GRG5ig LrshyTAuZlItgMp/eUHtmo8oKp03gZ9iGA2HDnEmFbWGhgx91uOllA9H4CEUA8/aR4XnUUGIWgPv RwvrzVVCx0hVBOAEcChRQ5Y6gzkbq2AZY3neUUQzcYMGgPLcV4Z5UrFmjWHSgCQw9cLnw1qQ3mJT tljPNPyfeJRV+u4kxeCOU4O6Cvcp4M5mPqhoXdM0okFHYUYndzC2ukjtoi4NDI6kWJ1j1AHaorzj xGEYRx3EfDXqzEcuA1rfg1C5xFVF7qnQ7xVodG9NfairpZUyf3nRdIFTIHE/LiLOdcxdLo8phwlr YNcTkk24PDXZMrGCVKwZsoosTddnjnD/7LICgjkTq1Wup6Hg6iGX02ZBaIarQEKu8DyjqR8zOIhn qkfzgWcNR14O/7UAxZlHA0Bl2Y1z11X6AZdwL8Bi3xGYX+TczVYHqZMUuJL5/xYKo5uqoqmaLizx JvVzWpujGljRNa7VLT8EyR/2WbOrKyPsGVZHtVSnA/z0OhaxaMca6ZAjmCOXOiM2xQuKbFIDh2C3 ZmqcCIZczXkI7wG0VIipsgmbNV0x1Rle1XDACWtdzWRWAkHGdTamdiWN8ipM5U1wlJdQ46OE/4oj H7Ktc5olcZTNh/PYvytcFO4KzQPjT2c+8KRKGJLiEtqFb+qqRjWZpxity4RdByTLj+Ixd0pIhXBk IKxYXVlBpsEkFl8UEQ9lMwM7PBMW1oiwc/UbUKdIaS35uL9qZlEsdXsgi1J0y7ZDf5wSlu3atsJC 9TTsfgshhYLd4I7EDOjzP8FqVA1Ro0NMBFpEHb1DFBN3HNfFZGQizzXMf9o2BPEc8jFpBlFyIYaw CC/lsYQ0eav0WGXp8XIaD7x6JI0nrtvYY0BsZVjNm1al7ngr3HTd+MFboG3xM19pC/QEPF94C4QX tkDawxhZ37sFGtN8nyRk5Ad+7tNqH7S8wXkMD1iyyP0AFjUUMNMUMlBYn95SZz8EifSykR/1/ERK IcuiUjaJb5iVyRwZuSdLOYs5EsPok3PjSsxRbcUJg+2mKgy0FRgMLvf3DwYDW/orTdP/lj5D2P4v 6a+pw77K26xkmqt4vn4zt5LD2+zo/HJr0fplb/cwmoIcIXnlDf5gf3AotnWHEBF8Aj1lUtfy/4i6 yLLUjqx1IZ/qptSdkNw2FUoNxVUtxyr3cl/2uRMXGxfnGhDJMIcucqoy/gWUiEDTyJschkzWTSDA 1XKoUeEHbi6yRogcsJ9RITZ3ZZZxQNpb7u1EHsaz0SzxHdqBgDTlMamxHs/xhpBYroAMskK6ZR01 AYN2467KAbOzwUWAcg3zVdiqYD+qvgATCNa72OBA/6Bh0WEp2yJnlBJVi1vzImCImxoEpgXAGDiJ OdjAH7FEeBEqrCugrmYt1iwkL8AEiF1zjqofjp8Z1TBeJr5MxZcw1GSkmmhJRBoszHOZqAjJCWh7 H7YLJHIoBywzlUTyVbm7OIZ+vrfpFyTo7McplXaP+9JpktM0jj7l4Y4E4cEBMwJHjjDiyVDgT+kx TceNTHYSxyKTbY2oB3CG03DIzlIF2zV5OIkz8M5iKySM09VEXu+6FHZhmjbSVIQp8Uae52nUw7pG HAf4cl9thbjfF9uh1izN58cWPNG/4kccgR8Vt2KB8qAAlVowpYMI9l2TOK8jfu0WYTKofAYzpaxI 2MkMdavIKJbAXZ5zYTCY2iZe0Iw0Q/gNdpAiDgC4zEi5n+eexxPa0j/4cn56eXHA2qOY9fxyenF2 dPl13nB8cckttNxp9w+Odv8+y0VPjof7pycX56dHRwf9+Zy9U4ByfnpxKpqEtxtcnLHL2PMEnG+H +wc19C7+fsYv91LfHfMJpycMDmu7o/z4ozXifWzbcwEc48daU53RJbhSneuMIKvg5zoQsCzYafTA Y/H8FXLsG3JXMYEdoPDcQa2jCbwMSXZdDcrypIY1rKfO1uNnZHlR5w7Dp7aKbVdHawJBdjrB1YXm E/CMQDhXfr43Dsdh3pTbvLkmvXnarVrrJMnYBV2HZ7v9/vmqSUtC/rp7cfCbkOt8rKZsLPCTg4vj 3cH/ch8BQaD6K2+tDA2aS9WDLF1+WQ1BKzRkkQ+PassS4Q+oTkMiPSwWL3WpIvcRXQIVEp4HUJCr L7j6ogrNOu8fNnWKNbxhX1Ci9x58AXcBJhZy+nbcFBNcv2EpCezejZAMSwipf37RlBJreMNiKtF7 N3LSTSGn4+qspyGtohRmIxcSbtgwqsA458MXEmTbM2IB6+1jxqooIdCrooQfQT68iqaVLKpLhm/C /giaOswavluHq5D1/DpcovcfrcMbBnQWtpVZcmgqQpmRhZsCYw1vWGAlei8nMPxmBAZyKsP3HnGu i2SVuxHH83VhPZOLqXDciOwlQlc4EYHp405kLrnmdvOtRsEttpff5eu307RH+QyXcZKXR4fsaNwh WT5kt2uGIQ1HNB06jVuV7NbkkLJ+3/FzceuNtc/n/lHQ9G6YlreThlwFpuJucnUbfZzGRTL0btxh RYp8KzehZFEcJ3405tMWutgRC8wXC9XhKwjz50cmNAjiITs34qoiHhtoYugDZfWpEBkFbiG5HZbn p7hEl9MM7axN4089MLYPZ08MiNttcRIH8fhu6ExINKbD6mx+NiBJ/TgtWaZgQ19ASnA7m/hJEy+9 RIIdQQ/ZUxkgkGz+xIhoLm+DySvpXNHTZGFTwpE3ZIf5Qz/hx+jZOgpn7XNul0yb4SWexTGBgq4s oxFkFwrT7Ll7emjEIkNn4hQ3TMcUtGO2qiJX5jHDXl9Af+zMQaiWheqjSbpIrFDRWf5mg4062JbZ nyYr66ayoOtsoEs9AmOHyVSQilYZS2MeLhUxd6IFmufT+I2PdAZvGpBo6PkBtJUmI24Vx+kNSV1Q xYCU94rvl1KBR8+JPk6Ens+pozXR9TmPdx5z+Yts/n/v/pGy2i9++P/X8f/oUf+PXtr/I1m4o3fu /zc62y3PDN9q3r35IeFH2v2D027jw+2+Ibf7BtJupBrI+PC7q4/736qD3eJ4/8PDfhxsfHjYV/Ww liqU4sPDzm/UVTej3uq9jC1uPn3XGf6Hh93ew+ofHvbDw9Y9LDZk/j7ce/ewD99Gfquudovbxtvd Jf5wtd99TIutD1/7hnwtftTX4pf2tYalfHja2vMf1WO2b/WoYIvHaj+OCn60d/04jX1L3vUNZLLs rODjNLb+hDxrLwqBYv9A3zvYw0bnoG9YHZCL3DExUtm3PSahXU3mzIujwI/oflKI19XltniwxHeE P2HAH/DZlefkw5jP/q16coKVQ2Hu0Oau0Bbq0PSj67123dM/t7dd8q+Tm4q2lThXRVSY47sX+4X1 e4S988P+13qm/jCBVbq9nm24GaH44o8HpMcz+i24gBe4UDIBbcCExkMp38sKtMQK9KNZgdYohNxa ftLIWHrS6PDsm354cngxp33xQaP5nOV06PJSlASRdYOo+sjpeNhUOyrSzM4IU7BudUQsisE2dG+J UyJF2vZZo/WykFfkY1XadJBP2MvS+WqD3z2e47SxacuPm/Zmrwg0njRaL2d5Sc78HDb2cnDFFBKj qT97e5dLOCgycNtHdErLN08V8Yqy0tX4Uk5SfAnIWPhYLyna05C2XdpOMtrOM6cdZmk7IbQdOrTt 3JptkvhOO6NJO8xT6BmzHtJ2wngKw3I2TdHbTsCLqbTD8Lbt3QKEDKDBX9ye5O3sLmNBsR3dsn56 m/MhQ0gU26kLSybtIGwrbhTfsD7+pZ3SZDiOY7cdxUnQhnZ3yBDx3XYS+YAX0tsBmXhDmOmEyTCg Y+LctbNpyMZylJ3UrJrZq7wQk/NJEEflRfmO8vArWmzgPuaQvcH/pHf3hSrsx5Hnj/vgcnic5M+N c0FWLwcfnHw9PDlYKSBS5PFJEZI9AlHYEfGXvVgMOTB7E989pqHI+cTTPzGveFC+PA0XTQdQM/rn 9UcGthG1HcMmI3tEbEf4o4BM6WxwtRO4b6rswuv7PBGh4aCsr2OYmHODvbE9qxlm8nxOYFi+l17W PRLU8iJbx7x62AIMjgxwD4mUpFx7WFbeaQ1A16RjElVJzzzTYGlM4y2OKq94OMyU+7ZvR7snNVfz 2P5xTTxqJjUMo82CDKOEp14m3jIHKYMIP4V8AvXl7ZcNqa+UbkPqV6Rtj1CvbKngM+qXXsjfjPzy xYUXEX4Jexv6YcpTxW9YT6FfPED2IuQb1rbUG9ZTiYet9BOILx/ueBHqxeZ+G+p188nU65tTv0Ds 9rcltjkqW+SJvgFPHt2gzRmmP9VZGI/4isU3VtcfMy6wcyklN9Y++7+ZVm2ddS8a4SYuaHOWGw+7 p23ftOUBnp2e7IsakJU8grjMBViBGn4gw+PorKIiTzEalXFPIU2YUMI7dK2cLUpRzefTsAgIZHLH xJn4kch6ilbC6lIWLZge6F2d480aO38oWoc18jJftSFqeQX91VBfVWXvduVo3LhCjSutcdWcp8AV Y30ajQdxkZbVelspidxYVL6JE8rewIb+O9g+LJURQhB/ZKVeW4qVMmo3igeJA7AysTr/5eCo1RQI hzMrq3Z//3O/CMO7DkKmYVgPlPbS20jRbJuVTCdjOgyJH4ziW1bQDNn2QLR2j/3M6dJbVlOMVaBz JtS5/hKnxzB4R3Jdyfc+94AX055LctJhNNC0V/5eAO4s/yM+SFF7oI4wNnN7wKph6Pb8CFYHeLCH +uz6KXVgO1MEwSiInWuJH31+RtIo+4xkrAKIet20jQnG6PsIXi6qhv4iS4BrnLqZ5Ef/iOrXcZFD Q4nu6C6nmfQJdEs63tsBghKfum1WEQ7pSEFYytqSDlsV6O1l/4jWFmvTHyTQwogTGCbnlJQlGXVs 23fkX1kcpYnTHbDOAS89BvRlNGK14ijf60jV3YBNGWq2wUe8Jw0Cgi3zTWoQ1tQHNOjbsbQ/IVFE g0xidYppxKR/ePLldDWVugp2wl02L1DM6nrWao+WdSh3zw6/CS9VVSLdpJDvP237KwMswXSp9HKS uHVQFonEvB5ppfXqQ9UJrbYig0F4fkAHfRjAVH21GMY0B4tw++yEfUfqFVnaYxUa1+lfGOU9Uac5 8rKyRLQ95C3DUvo9zSWqw2r2mpo36qiuizojKrsd1yAGsVSPqKZTqWZIc8LgN7RTimFpF1aOQEuZ YgL79Jmi1rVz46KdwBBzZfHTFyrMqm4oJhU/RUzLxiL/BTWMpX7NjUXV1cpQ2NemnQAW2LKYnUCS Y0nX32knQJbGSo+GTjnHtjW1KotZFWulrIDljnROHeqz6rGl+0ISYygwJAJdR3jNyg9xlC9VW7pW BrVau4hK6VFw8hfpHZMn2FbZWFu++5T1F1yEKK9buYhyjS1rFO/mkCMl+YLeAcq9KUl7gT/qlVUj e6zSY68ivrcJ+NrJYpfZe9hl1TKfQri+QDhLKr6T8C8QHEA5asJ5SaJFTdZ1SveQuuuvp+6LS1ub Lt2VTmFtd1a/eXQn0SQOgicpvfHKRme8V6Mz36/Rma9ndNabMDrrdY1Ol9+p0S0R/n6MTkevZnRL S7+K0S1i8cONDr9Xo1sk/B0Z3evtppaWfh2je+U9nf5e93RLhL8jo3u9Pd3S0s9gdK/9W60v8Wn8 /m/5s3VBPH7WNRhf1//+L2K/jzH7/V9FUfjv/xrax+///ojPkvGoeltVTIlZtST9HqfjLj+a7pa6 4cQpZU5hlMbXNO1+c7PLxCU5PS8i9uOG7Lc7r6RPffHE+68FSfN/D9hvOhYBTYe/xSlM6uiYmZ2X 0mzyLWTmDM5P8l1pEw8l+Vn1M2MiOgEuEgFQkj+OYvbou+Tnkg9RYELBhPkiUhHlflD7dTKA4cYR /XmZeKttIvlFibd2pN8NRTdliuWrN8oGTWkr5svqgGG9VR3Q9LauKpsSP//2pQgCRgiwYz8OQwgy jzJB25EGF7vnF21pefKnX2Lxs6LS/7V3rE1tI8n7uvyKuaT2AoWkaGQ9faWqOLbZhQLMAeFuK5Vy jaQRiMiSV7IJrtT+9+uekWUbbOJAYG+vPB8SJKtn+jXdPY+e8QVtCsEX+xE82o0G2E8jBJwdppoN 21SDuOGpPLKpZ5tGSONQQab4CPJRQK8D80kRUUjpf1xHCp92FAKeAoTWJI5NA92N2HJuevZLcXNv /3j/7Ndl7FSqO8Twhq/ZBcv+SJyJfWdfiF/vzlAIi0e8qM+6l7cG+QoZJpFPDdqwFDLgg18AIZaN OI9wS6xvUUMhI3hTymstfx1f8hO8DKxq7t4ljr7l6VQh4u6uszJtJ8MrXpyNkxH3O9291odD0BDc zjIZcn+26RkABkNf/Af8CFkRVUTF8sjrcIyr9f7XiN9AvX1quFboGkxljAWgAZGnuqHH1DD2rIjZ lFleVH3qxNw2PddTDVAE+NRzVJdTgDRBZajjUNPR/YtBR3xNvorQdR2VAdIFCHy+ThPTz/1xltwC R5H89q+t4+PuoULwSO5eEfHC15FzPDwBGQxKH/dMM3kLt/81AGHBzxAkjwoI4eTHoh5UwiTHi9IS loI8UQoUQEFXQFLRlIfDdHyJj1JwuBzZy9LJ9FeJXitNWOlXoaY+5ftJgftz8Jo6gVJZXUQFpOOS rOg6SchS7OHizR/T6n60oKpPY50Hjh4aquF5VDWtAAwANcAAMEOPDTeAmMx8okzXaeKlZWr8AJnS l5HpE7m/ThM195OIT5k/3XR4uib/yzQf+fotGiwhilsdO2o+YEkmHvSpOIZhopB4nIlROfz0pO4F CP+lu1b1qeuadhTYgWrZYaSahhsDAoauBpETBvBV6Bj2E/VgnSZqPRAuB8bJL9ATGz+gJxqPVoE/ MLYRDvTzzUA4beGqqaGbwo+KjaYn4HXFlYM+xc8ltNigo5WTkkUAmCHXkCcyEvgwlLw64tl40ft+ h+jCfDg5wQ1nC0FJfSW5X59Z7Rj1axE2qICmmFrqJOXnIzbEdHn/K+OeZwQBVUHUFmhuYKie/xV3 ecotQTcRA278q3v0od+56KinvaP5X8tCpKtNL2WHWAfnYc4xmol5sYAh8EJcJ+kXo9SlDU8Z3ijk M58EOcQih2ySj0c+z9QxxK71zt0KdJiMTnmSXcspu1rikrZjedGiX+d/TnUBItPDa3bDtJRll1ov QPB/vqOeRWMaQSiGqdK9OC5BgDoq2+0FZib71J7GViLims4k+SXaLKVMsuF4VCol/FbmhSJQCFj4 WSnl9h+lrPBSyjGEpzxKMFduGnPVWPu/36Z4C+xaMbHjes86vLLmJraYJJ5URgUn80AyhGXTnky+ XPEML3jGaz9x9HRxtN44rJIKcPdzIpSvetMkX+fty6IFeUZJLO9IoJHQN6aa+HC3uqd6te8QtgOG AcOrJCwrMzf3mJZ1IP/Hcol7OumenvZOn0/i64qtssIkg2GwMMHiwuioUg9W1lOf44zfDsV6QDqR W/SqqeUmeZqPsh3dalAeqnrcgNGoZVhqEJquGoHLopHLHACufRQerMDSNM+ziu3vW4eHvd7xA15K 5IdWXmjOY80cUGWFKv9TmZ/aAVUGeM7/VAg8PgRZoRPGRifqEaHDG0YE0YquB65quqAdLIwbKkRe AW6DbQSx/SfqxJKg5DmUwlGshkv+3To9XukagjTV2mJX88VRW6aOf8cUyUcasCiIrPgTabOsk7fk olkeEyb/ulM1ieVaU5wXZAwBHTn77ey8e6SBf2FlnpUg9tZpv99qn+/3jvv9Dyed1nlXEe8wpb/f vzjq99uHHwDoVIG/qz/7+2f94955/6J1uN9ZygdbNx92kRFLtSio+gcbR8kIGCwWd8ATaS18cZhj QrfYp/y9vMmLgqdyqnC/0yReRGGQAPp6kAfihRU1ILZvmDB2wC3aOih2oFuBGpmMcVuPDCOAqLoN KkLORuDDmkQqQVuoCuni6pOoCIO4I3ltfZPUM03gywbDlEM3I9sXRwC8GIbKlKAm4RCn4SkA4NzL UZIJ0GY1X9gZF9Wz6RIAzrOo3NGW89r8Bq9R5yRu/GLwOG07xFSJuOBy3TIXwRvpijbETx/5LR6E ACELPpY++u3mesbrhqVjYMbF0Rb0uiugPqqq+LSEWk/xHOvb1PZu4g4bMWl019AeukPmlqOBwrGA BKTKt+fwGicYSwK1LpMA4uR8G6fqZnaoRKK1lxfVXveTPE/XFgtFuXBumY4LcgGHkskVdAHeJGu1 QsR5RRlLmwTtpgaSHCVo+giLY+EeSJMI/V4n34ZgdNUkc83cZZLZ1HWlYdI/iUmPU961SK+Ut3ex VxnP+0r8jOt/C+u/4rCK4qXXf6lhNu6u/xqGuVn/fYkiu5mO3czA/KymbivW4uCUDRn0Duhgk3zE NXY91FrXw2maD3Sko7M2Qc3ByE9mERGqmjvkhI3FaLItAEnr4OQt1Rq4swPqUNU620d1Hd3YqhFx Z4uQlm3MI3Id5GWpsVKrNr3khVaOg1Jkv5Ya5kCVMlMWkJJpiGcVWjKTiKABIKpKqLdDDt63znSK eUZNcJ7DNBfbnSD8VQ8679uqcL8JAx8dFdCrCxKmDEbKiMYwL0eX4GF/T7WO/G17mmzmaXRnCSWm Qm3rYUpW8bExh6mLFhKsFSJ60G6RsynxZPvgPVRI9os8O2AhgxExocBaT9uDmCBdjpLnLkcJA31s YhVGdIqRQ6GuajsOogT0v52lcIFQG0vatRTTazxZu4ydGSceq1624pnL1etbHDDX5oC9pF1HoZa+ tN3K/EZCGwcQH2pliBMkK5WjFoWlU6NSDvBQe5gMJxSjU1c17QkYxkdVSDyZbRmTCYnSlb4VCPUL XDEf8LczdMol1LiKZ7nfTc0cXvKdKmkqCXTOGVHQANnLxxluxirxiCjtC2h2kpFZpTNiNHKeQyyS wOitmP8ghIpBP5iYxyR4qhVfqE+Lcvn1Cuq856euCmH4DyJuVt1DtHmK7tLnp02e59X/0QK8X+3D tHqe+VhXssoC1XYZbMB7QatIw5WA5CNOVDffykPDcCQhh2yf7mInAltbX7MPfdMkWa5jz7mJOTbC CP/VHLteLUfEsZ+GiLEeIjMVXYWH8zQ86Hp43FejZfiY0FWeiM+afMEBzPVKJLyXEc4XDoP2QZI9 wA7vZdjxhacQiPGHEHlix1kTkSgPy9VYUPpEdqzZfSu4BzFZ01091ZDk+egBLIw1A5yn8mM6al/d Z14KFZwaxWVulq5ii6dYJp2f2a1xgap4FvEsnGhDGFCAs/smMq5lO9Nxi8Dh1Zy3nvP/CxqDex7l kIyRqiEyyJGBZPsVKweAy6CJK3GvdsiXqyS8IgM2IQGvz9CA2HG6DAA+PB6PxgWfHrRR1guLGQTP IddWsMD432XBwnBkMACatICzbDxK0vLZ+dL4S/ElhFgrfHaemH8xnkBkKtZQnltbTIiOnsYZ+r2c mffEL2xH/uwpsv/rsjD/O0uY0oZsFF79oDYenv/VDcOx6/lfx3L+plPbpMZm/vcliqqqZF7sEy2H EfhPwuronkptQvWmYYDV0epVC6LioHdrd3d3EfSn+TUbr2k5TauBd3o0XKq7JkBZAPXuHVGpq9hk F/51yLt3W+Q1ZoLAmH+Ui4yN026rc9QluMjU7p38tn/8ixj5l2LuCk9dg9hrxMAnY+CFACmYCTxa jrzeIlu7yQDjMMKLIsu3SPUkDR+YrNmbZMDrh5Kj5Ra44Woo2cXj2HSB3HwBMnlz8RUWgI5x0QTs bzC+3H5V77z6WdPd/6zMJ3yl3K/qTpE5ilu7d9+/xvRc5BewJEyCdFLVDnSCbd7ryI0Z3ZPe4eFS 4Bz3NaQABko9xFOuAF+056K98j7IqJg077+tae9LgsZZwS9xMrTYlrmbSzDnt3gsGNnvdYsCxMlK wlfU/JrsYzYPJ93jXvf4fPlHSUzAP6Goyd99KXNNfr+iViwFS0Bblv4sJbwS8o6k3+xN2T8jHdlY jgpweCDznyPyZnVtd8obITTBSrLX2sdjYBWympGrMX0ElqLp9VB9IzEUB6h1OwtIziEHgtkW2kT+ UfUuTajj/vHOnR6UB9fEr/SoNiaXfFTpkFIdVLcAA7UDWHMTGmzKpmzKpmzKpmzKpmzKpmzKpmzK pmzKpmzKpjxY/gvI+YHWAMgAAA== --MP_/IiROFVEHqzB/cq/fJkmmawn-- --Sig_/xM2/5weqzNCEiCQ1Zp6THsR Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJWWHbxAAoJEMHFVuy5l8Y4QnMP/i2Y8VyKsEdP8gDiM+GuK6T0 vUT7G67jhz4IcFKUV2/8yGIHQsy6lwzaAg3gk+PlnGvS2DL+1rJGYyrbd90g1QHy vOTLVAP1Fm9yZEcQPDDwdXlToJV8rZ0BTOVJi9+xKxg3FepxbTATful8/elTGSqg 6TrzUcaTHn6H2AI2g88kfiJ2M+9CbuNIUcZD29+3AfBuHRKcu5HbdJAHXsLHkJHs IS7ThfCX9idje1Q6FK09XQ7+9aIKl7/VFf/jpGSV6WgObiauHKDz+cYW3adcbXl0 ocKmrbhZF7/Q8z0xOqpuc7BA+wg06KGYxX8f32Re5/qtY2ab95M6+73cGSu9Ph/T JKNei0Xb6LocjAhypzv19JXXoOVjghUOZjLluPsc15fYBpgt/BnYMNOAJDPuyj4k PfPqblxsS1gw5ZC7ojQ4WQQGifDQjhOkpbaIckb+eeCfhMMMkEICA7A4OmBU6qs1 MU/bFaOaypy5rzuDb//n4cYbzA/TNku3M97wsy37mGblBCgIsorFSzv9IElZdXHZ 5g7MmBMKiRw1tP0AUUGLgAdRQq4LnlH0BANo7PxhILudZLUSKhm3d8jNP4r0Zby5 Ce/67DQiFTzt3dEy7OL4q7wlqmGspHM6BL0Jmdg+gRvNJ5hvlxXrz0Nzh/Khzrc5 G6o9xdGw3ALRmyyM851d =/Bhs -----END PGP SIGNATURE----- --Sig_/xM2/5weqzNCEiCQ1Zp6THsR--
participants (3)
-
Dan Kenigsberg -
Robert Story -
Vinzenz Feenstra