7:45 a.m.
--Sig_/War1DBdDuKL/JOwM_gN0y1g
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
I'm running oVirt 3.5.x with a hosted engine. This morning I noticed that 2
of my 5 hosts were showing 99-100% cpu usage. Logging in to them, vdsmd
seemed to be the culprit, and it was filling the log file with these
messages:
VM Channels Listener::DEBUG::2015-11-12
08:09:26,292::vmchannels::59::vds::(_handle_event) Received 00000011. On fd=
removed by epoll. VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmc=
hannels::54::vds::(_handle_event) Received 00000011 on fileno 119
VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::=
(_handle_event) Received 00000011. On fd removed by epoll.
VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmchannels::54::vds::(=
_handle_event) Received 00000011 on fileno 75
VM Channels Listener::DEBUG::2015-11-12 08:09:26,293::vmchannels::59::vds::=
(_handle_event) Received 00000011. On fd removed by epoll.
VM Channels Listener::INFO::2015-11-12 08:09:26,294::vmchannels::54::vds::(=
_handle_event) Received 00000011 on fileno 119
VM Channels Listener::DEBUG::2015-11-12 08:09:26,294::vmchannels::59::vds::=
(_handle_event) Received 00000011. On fd removed by epoll.
I googled to see how to change the debug level to turn of DEBUG messages
for vdsm, which referred me to libvirtd.conf, but the debug level there was
not set, which should have meant a log level of 3 (warnings and errors), so
I'm not sure why the log was filling up with DEBUG/INFO messages.
I restarted vdsmd, which resulted in those nodes being marked as
'disconnected', but they did eventually recover and loads went back to
normal.
This may or may not be related to the fact that the 3 hosts where this did
not happen can't seem to keep their ha brokers up. I'll be starting a new
thread on that shortly.
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/War1DBdDuKL/JOwM_gN0y1g
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWRJgKAAoJEMHFVuy5l8Y4SGcP/jvgozmL+MsLRWFK+hWBa4+8
lP4TVCWdDqdd/3Kt3ZIO7vNBCEWNmicbtOu6o6yeG1o4l5OVh0GG+wspQOHm5D7g
ScfsQ405e4EP3skKx4yauHmM8kxBuZo2adRj1CT+mbB/xXMfUd/gcVWgFYuPMMuN
EkJR2Q9NqpP1ccUduccAMkiXSyRbUt6Ng7iZqx2RhDAdnRW2CYUCSqkSxI7KE/Mw
NZQ4bGAl4w6eJwBxV9jB6eDW6RD2pGmncRswq7TGv8+iGvLd+XAxhQRviIJYeaFH
8GTil9/qEW8wzmbEYsd+YzbwAaPuveKx9TZCTVyhgabzeipLv0Xj4eKMO8fJ66kB
s+MYZSeIMDAGnDoHdFvf8hVphb+DHoGOZXx+C76Ohfwb+WL+ZDCIAra/lLx57CUa
H+JPU9pl4Bn3f0o3srX6Mkbj8qljXHGXCKI206SoYfgMyRlN9/VylpVrkryP3m2d
8CXK0X1+bxnvohelZGXlIca7t7/eTVd9Vob2aBKjvUkGLqQ920fXxVRqI78Rpmdd
druse94ZN8KYk2NOMaM0YuZERlkJP8aqQAYNqGwZ6QgIPuYGyEeY4pnJWpH+Z0sF
g9CUW9AblprC1E668qh4gRrKv7a3VWpkQyZNJTaIVxfJt97oXIbU0RtFQJEF5o2n
7kSRCB0xu7vyJEw32O28
=NjlM
-----END PGP SIGNATURE-----
--Sig_/War1DBdDuKL/JOwM_gN0y1g--
8:02 a.m.
On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote:
I'm running oVirt 3.5.x with a hosted engine. This morning I
noticed that 2
of my 5 hosts were showing 99-100% cpu usage. Logging in to them, vdsmd
seemed to be the culprit, and it was filling the log file with these
messages:
You're probably seeing
Bug 1226911 - vmchannel thread consumes 100% of CPU
which was closed due to missing information. Do you have any information
on when this pops up? Is it reproducible? Would you be bale to test a
suggested patch
https://gerrit.ovirt.org/#/c/42570/
Regards,
Dan.
VM Channels Listener::DEBUG::2015-11-12
08:09:26,292::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by
epoll. VM Channels Listener::INFO::2015-11-12
08:09:26,293::vmchannels::54::vds::(_handle_event) Received 00000011 on fileno 119
VM Channels Listener::DEBUG::2015-11-12
08:09:26,293::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by
epoll.
VM Channels Listener::INFO::2015-11-12 08:09:26,293::vmchannels::54::vds::(_handle_event)
Received 00000011 on fileno 75
VM Channels Listener::DEBUG::2015-11-12
08:09:26,293::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by
epoll.
VM Channels Listener::INFO::2015-11-12 08:09:26,294::vmchannels::54::vds::(_handle_event)
Received 00000011 on fileno 119
VM Channels Listener::DEBUG::2015-11-12
08:09:26,294::vmchannels::59::vds::(_handle_event) Received 00000011. On fd removed by
epoll.
I googled to see how to change the debug level to turn of DEBUG messages
for vdsm, which referred me to libvirtd.conf, but the debug level there was
not set, which should have meant a log level of 3 (warnings and errors), so
I'm not sure why the log was filling up with DEBUG/INFO messages.
I restarted vdsmd, which resulted in those nodes being marked as
'disconnected', but they did eventually recover and loads went back to
normal.
This may or may not be related to the fact that the 3 hosts where this did
not happen can't seem to keep their ha brokers up. I'll be starting a new
thread on that shortly.
Robert
--
Senior Software Engineer @ Parsons
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
8:16 a.m.
--Sig_/.YuOUBlg7p5t+Kk.wNQXWNo
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote:
DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote:
DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I noticed
DK> > that 2 of my 5 hosts were showing 99-100% cpu usage. Logging in to
DK> > them, vdsmd seemed to be the culprit, and it was filling the log file
DK> > with these messages:
DK>=20
DK> You're probably seeing
DK> Bug 1226911 - vmchannel thread consumes 100% of CPU
DK>=20
DK> which was closed due to missing information. Do you have any information
DK> on when this pops up? Is it reproducible? Would you be bale to test a
DK> suggested patch
DK>=20
DK> https://gerrit.ovirt.org/#/c/42570/
Hi Dan,
Thanks for the pointers. If it comes up again, I'll try this patch and
report back on the bug...
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/.YuOUBlg7p5t+Kk.wNQXWNo
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWRJ9IAAoJEMHFVuy5l8Y4W04QAI+1DK37aNPX9nLMcNthF6BQ
0xgNKhzlE82iwd4f0Oflp/GARW8XeKwPSdcqBlfuIkVy5jxXKyRBriz44/dU1V2/
90qV265Sa6y/PSH6B1PwF2sWyjDJsL3KPF9uH2OO5jLN+Q6pSQRSowzodI3UtZG2
ZUZuPsqOmSD2LK0DVc+EMB2MMF002PKArd6+i8PUvAuaCNLrV8VKjymOZLRr6J81
hG9HKpu1980TWi7u9ozr6h+CCdef28REreY0Ri9LxEvL7OYuvhSvt5/amIGgdK3c
PhiUB/GIS9nNEccnnVILK3+qfeli91zfjGGaewUFn4GrgyAAFcQ24gHbiUhYDfFX
Z0Qu5Te8dvfgFiDTQI6Z471y0tI77y6/S2ArSvasMEyJ19B0xNVfO6M9HwfbcVTy
mBPZEo4cjox17Ps+kpSG3PK1U3cF0+oK2T37agc3LxXDK+SzA8FBPRIrKipVfjRQ
Oca/isGVxPQptoIGUT4tH1NuTuOpz7/2oyD5xYuURxh28RlteLeV/j2oUUj8j4uo
IS6helL9V8s4Wx66wcICBi8vSF4jP/W2NLmKaVHBl5TkzgfPxBNvsOFjsXjBpCoM
RwR/oCVcFm3SPUxkH1bFsMCB6oB9R/WRrlUaMpQvE/fPNpzVEvsnQr3L80JEh+Gt
JzoGRG2JhWdvyUUqYJ3x
=6cat
-----END PGP SIGNATURE-----
--Sig_/.YuOUBlg7p5t+Kk.wNQXWNo--
5:35 a.m.
Hi Robert
On 11/12/2015 03:16 PM, Robert Story wrote:
On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote:
DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote:
DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I noticed
DK> > that 2 of my 5 hosts were showing 99-100% cpu usage. Logging in to
DK> > them, vdsmd seemed to be the culprit, and it was filling the log file
DK> > with these messages:
DK>
DK> You're probably seeing
DK> Bug 1226911 - vmchannel thread consumes 100% of CPU
DK>
DK> which was closed due to missing information. Do you have any information
DK> on when this pops up? Is it reproducible? Would you be bale to test a
DK> suggested patch
DK>
DK> https://gerrit.ovirt.org/#/c/42570/
Hi Dan,
Thanks for the pointers. If it comes up again, I'll try this patch and
report back on the bug...
Out of curiosity, did you happen again to see that
happening again?
Robert
--
Regards,
Vinzenz Feenstra | Senior Software Engineer
RedHat Engineering Virtualization R & D
Phone: +420 532 294 625
IRC: vfeenstr or evilissimo
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com
6:28 a.m.
--Sig_/GG.=3/ec7hZy8d4QIAOi9xR
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Wed, 18 Nov 2015 12:35:17 +0100 Vinzenz wrote:
VF> On 11/12/2015 03:16 PM, Robert Story wrote:
VF> > On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote:
VF> > DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote:
VF> > DK> > I'm running oVirt 3.5.x with a hosted engine. This morning I
VF> > DK> > noticed that 2 of my 5 hosts were showing 99-100% cpu usage.
VF> > DK> > Logging in to them, vdsmd seemed to be the culprit, and it was
VF> > DK> > filling the log file with these messages:
VF> > DK>
VF> > DK> You're probably seeing
VF> > DK> Bug 1226911 - vmchannel thread consumes 100% of CPU
VF> > DK>
VF> > DK> which was closed due to missing information. Do you have any
VF> > DK> information on when this pops up? Is it reproducible? Would you
VF> > DK> be bale to test a suggested patch
VF> > DK>
VF> > DK> https://gerrit.ovirt.org/#/c/42570/
VF> >
VF> > Hi Dan,
VF> >
VF> > Thanks for the pointers. If it comes up again, I'll try this patch and
VF> > report back on the bug...
VF> >
VF> Out of curiosity, did you happen again to see that happening again?
No, I have not.
Robert
--=20
Senior Software Engineer @ Parsons
--Sig_/GG.=3/ec7hZy8d4QIAOi9xR
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWTG7sAAoJEMHFVuy5l8Y4R1QQAJwYgtLUXuKNMRsb3jPFtIPB
W6T4T8becETOsGOetqwcADoilOmk+EnCMZ+G9USF7dqtghhIGbj6Up+h40rnd4V1
OLxn56j+oyj5Md3YoarIGVjubkFaPc7zVohMPjsF2PHwgyfbxm+EbFr0x9XP4l5K
NteGICiSIHuA+oeseuObVFSv/C+7dNEtyQytE5HVgQmzgBzmHPxkWGnr1aJRqYQO
fv8ImYN/fNTbMOs6q7nvpkTCUkgzwvVAPeoGK1Yq1Ab5avfgY+MIATaxxePIObyD
TiEocRM+Giwqb4xj8tW3pZorssSDoGcfnKUHZIJsc0sjsXD7H+f2NokkVg1lmLKL
3Zp42llrt9hbmwnsoA6ZKlt9XR0JcBm7ff5rfx0Mx72BiBLWh2P4OmAb05rCyTvK
v2mBn39VliLdtot1E7XIPUk9fcePMu8xvvfbYLO8oUGC85ZzzObnvWx9j36DthIZ
dVuP3F5+2eor9ndSdVmLPFdoSKKSALPyIOTNj9tSTWk82/DHqT7Ps/6qXFeZg1sX
3pKGQ3l0T+1IcNAowARdQ4W4p6T8SIAtzDPE8WXef1nXDKP5HvWyD9Dk3KEiIekF
8ffrSvocPT8TFFc37Xg9KULNqTAYCUg1FqfNugneYb6SCXtRBs4GiJ/E35dlciyx
ZAmeDUNNvjevWyOVA15X
=5Lj9
-----END PGP SIGNATURE-----
--Sig_/GG.=3/ec7hZy8d4QIAOi9xR--
9:29 a.m.
--Sig_/xM2/5weqzNCEiCQ1Zp6THsR
Content-Type: multipart/mixed; boundary="MP_/IiROFVEHqzB/cq/fJkmmawn"
--MP_/IiROFVEHqzB/cq/fJkmmawn
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
On Wed, 18 Nov 2015 07:28:23 -0500 Robert wrote:
RS> On Wed, 18 Nov 2015 12:35:17 +0100 Vinzenz wrote:
RS> VF> On 11/12/2015 03:16 PM, Robert Story wrote:
RS> VF> > On Thu, 12 Nov 2015 16:02:59 +0200 Dan wrote:
RS> VF> > DK> On Thu, Nov 12, 2015 at 08:45:43AM -0500, Robert Story wrote:
RS> VF> > DK> > I'm running oVirt 3.5.x with a hosted engine. This
morning I
RS> VF> > DK> > noticed that 2 of my 5 hosts were showing 99-100% cpu
usage.
RS> VF> > DK> > Logging in to them, vdsmd seemed to be the culprit, and it
RS> VF> > DK> > was filling the log file with these messages:
RS> VF> > DK>
RS> VF> > DK> You're probably seeing
RS> VF> > DK> Bug 1226911 - vmchannel thread consumes 100% of CPU
RS> VF> > DK>
RS> VF> > DK> which was closed due to missing information. Do you have any
RS> VF> > DK> information on when this pops up? Is it reproducible? Would
RS> VF> > DK> you be bale to test a suggested patch
RS> VF> > DK>
RS> VF> > DK> https://gerrit.ovirt.org/#/c/42570/
RS> VF> >
RS> VF> > Hi Dan,
RS> VF> >
RS> VF> > Thanks for the pointers. If it comes up again, I'll try this
RS> VF> > patch and report back on the bug...
RS> VF> >
RS> VF> Out of curiosity, did you happen again to see that happening again?
RS>=20
RS> No, I have not.
So naturally it shows up again during a holiday. I came in today to find 1
of my 5 nodes (the SPM and host where hosted engine is running) with two
vdsmd threads chewing up 90-100% of the CPU. I applied the patch from above
and restarted vdsmd. This resulted in another node being selected as the
SPM, and within about 15 minutes, that node had the same issue. Applied the
patch to the new node, and restarted vdsmd, and the SPM went back to the
previous (now patched) node. Hopefully things will stay stable.
I've attached snippets of the logs from the SPM when the problem started,
along with the server/engine log snippets from the hosted engine around the
same time..
Robert
--=20
Senior Software Engineer @ Parsons
--MP_/IiROFVEHqzB/cq/fJkmmawn
Content-Type: application/x-compressed-tar
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename=vdsm-runaway.tgz
H4sIAK12WFYAA+w9aVPjSJb9uX+FvrmI8aFM3dqpjQBMVbPL1RiqZ6KH8KSllK1BV+swMB38932Z
KdmSD7ApKJgFV1SVlcfLd7+XKel56mZhJy0ickPuej+9zEeGj6Fp7H9sKGr9/+rzE8I61mVDwSr6
SUYKRvgnSXshfBqfIstJKkk/pVkep3frxz3W/x/6mdblzy66QTzuIvk512AC1lV1jfwNrGpqJX8E
XSB/DTH5PysS6z7vXP59mlMHKJPySUqJa9v9g73Lr7aNQQgdhDrYkGTZlhVb09oIqbadpHEeO3Hg
lhP5UNsG1ekeF0Hun5X9u45DE97/aZjSMJ7SoRNHEczx42hH2p99l0SvK3lpHEoIG10Z/iBblVUZ
/bwdftpK/FTjEfwmJHKDOn5DttaOJBYH1Cqg0m0YPAeeum3v+ZHrR+O/HR+dn+3bNkKGYtt/C4P+
DO9PAq1B7FzTfIcvDXxKpUmeJ5Jbocax+dSa4dNqSxyjnZ8vOCYdZMmqbqkPY6SgFRgpCucbYHKT
kiSh6Y7kBD6Ncun32XJXtu2QIJCm4VeaD3KSZ9KNn08AI0OVLYods6MgXe6o6mjUsQzD6HjmiKqm
qY8IcVvtHenP++0wxSswVeUlTFOaF2m0hNefLbD3vMhaNnwNaZaRMYXvrX4cUWBdy4lddi3fw3c2
Mjvysxwafv+z5fpZEpC7w8iLRUMeZGdxynpbHcb3lp/sum4KQFmTzFryu4SDn0YOu0zE8KKlWbLS
ur+CppCGlxUSmsYGESfxDyIyCnhbnhYcsXFBs/zLr/0TPj+e+mneodHYj2g3onlGBXzfZXMUZKgq
u84AF1Bn1nYZXUfxTcRaYfyhRxyacTKK1uSGwwQeg2Iq1FaJ7YzsEYNQtCISUt5N84nMW2BJzpGi
heSugSBYdDHGrauqTxedHjVBKpiD9DzboxwsQL3idOd+SE89L+OwBLNGoEhxHJUM/pOJICFFRveF
TFonpwfn52wgSOI6q7jGKEhIPuFI9jiGeZyTgM3AimZAfDEEO4oWAOMMwoapa5piajpv9jI++Rb+
h0Ub8EZxnDdhalg2LBUCVAMi0kxZ0SzdXAJ4JTh+E6fXjKoCdIHmqtA/4jCF4QxYwfxWettPY1Bn
sQI2RdtBmsZpTcXqg8qWc5JzjoGJijkLDfkiEKbqfEAxV5MsoeXKEKW53lSqICi4vxfqC2GRG5ig
LrshyTAuZlItgMp/eUHtmo8oKp03gZ9iGA2HDnEmFbWGhgx91uOllA9H4CEUA8/aR4XnUUGIWgPv
RwvrzVVCx0hVBOAEcChRQ5Y6gzkbq2AZY3neUUQzcYMGgPLcV4Z5UrFmjWHSgCQw9cLnw1qQ3mJT
tljPNPyfeJRV+u4kxeCOU4O6Cvcp4M5mPqhoXdM0okFHYUYndzC2ukjtoi4NDI6kWJ1j1AHaorzj
xGEYRx3EfDXqzEcuA1rfg1C5xFVF7qnQ7xVodG9NfairpZUyf3nRdIFTIHE/LiLOdcxdLo8phwlr
YNcTkk24PDXZMrGCVKwZsoosTddnjnD/7LICgjkTq1Wup6Hg6iGX02ZBaIarQEKu8DyjqR8zOIhn
qkfzgWcNR14O/7UAxZlHA0Bl2Y1z11X6AZdwL8Bi3xGYX+TczVYHqZMUuJL5/xYKo5uqoqmaLizx
JvVzWpujGljRNa7VLT8EyR/2WbOrKyPsGVZHtVSnA/z0OhaxaMca6ZAjmCOXOiM2xQuKbFIDh2C3
ZmqcCIZczXkI7wG0VIipsgmbNV0x1Rle1XDACWtdzWRWAkHGdTamdiWN8ipM5U1wlJdQ46OE/4oj
H7Ktc5olcZTNh/PYvytcFO4KzQPjT2c+8KRKGJLiEtqFb+qqRjWZpxity4RdByTLj+Ixd0pIhXBk
IKxYXVlBpsEkFl8UEQ9lMwM7PBMW1oiwc/UbUKdIaS35uL9qZlEsdXsgi1J0y7ZDf5wSlu3atsJC
9TTsfgshhYLd4I7EDOjzP8FqVA1Ro0NMBFpEHb1DFBN3HNfFZGQizzXMf9o2BPEc8jFpBlFyIYaw
CC/lsYQ0eav0WGXp8XIaD7x6JI0nrtvYY0BsZVjNm1al7ngr3HTd+MFboG3xM19pC/QEPF94C4QX
tkDawxhZ37sFGtN8nyRk5Ad+7tNqH7S8wXkMD1iyyP0AFjUUMNMUMlBYn95SZz8EifSykR/1/ERK
IcuiUjaJb5iVyRwZuSdLOYs5EsPok3PjSsxRbcUJg+2mKgy0FRgMLvf3DwYDW/orTdP/lj5D2P4v
6a+pw77K26xkmqt4vn4zt5LD2+zo/HJr0fplb/cwmoIcIXnlDf5gf3AotnWHEBF8Aj1lUtfy/4i6
yLLUjqx1IZ/qptSdkNw2FUoNxVUtxyr3cl/2uRMXGxfnGhDJMIcucqoy/gWUiEDTyJschkzWTSDA
1XKoUeEHbi6yRogcsJ9RITZ3ZZZxQNpb7u1EHsaz0SzxHdqBgDTlMamxHs/xhpBYroAMskK6ZR01
AYN2467KAbOzwUWAcg3zVdiqYD+qvgATCNa72OBA/6Bh0WEp2yJnlBJVi1vzImCImxoEpgXAGDiJ
OdjAH7FEeBEqrCugrmYt1iwkL8AEiF1zjqofjp8Z1TBeJr5MxZcw1GSkmmhJRBoszHOZqAjJCWh7
H7YLJHIoBywzlUTyVbm7OIZ+vrfpFyTo7McplXaP+9JpktM0jj7l4Y4E4cEBMwJHjjDiyVDgT+kx
TceNTHYSxyKTbY2oB3CG03DIzlIF2zV5OIkz8M5iKySM09VEXu+6FHZhmjbSVIQp8Uae52nUw7pG
HAf4cl9thbjfF9uh1izN58cWPNG/4kccgR8Vt2KB8qAAlVowpYMI9l2TOK8jfu0WYTKofAYzpaxI
2MkMdavIKJbAXZ5zYTCY2iZe0Iw0Q/gNdpAiDgC4zEi5n+eexxPa0j/4cn56eXHA2qOY9fxyenF2
dPl13nB8cckttNxp9w+Odv8+y0VPjof7pycX56dHRwf9+Zy9U4ByfnpxKpqEtxtcnLHL2PMEnG+H
+wc19C7+fsYv91LfHfMJpycMDmu7o/z4ozXifWzbcwEc48daU53RJbhSneuMIKvg5zoQsCzYafTA
Y/H8FXLsG3JXMYEdoPDcQa2jCbwMSXZdDcrypIY1rKfO1uNnZHlR5w7Dp7aKbVdHawJBdjrB1YXm
E/CMQDhXfr43Dsdh3pTbvLkmvXnarVrrJMnYBV2HZ7v9/vmqSUtC/rp7cfCbkOt8rKZsLPCTg4vj
3cH/ch8BQaD6K2+tDA2aS9WDLF1+WQ1BKzRkkQ+PassS4Q+oTkMiPSwWL3WpIvcRXQIVEp4HUJCr
L7j6ogrNOu8fNnWKNbxhX1Ci9x58AXcBJhZy+nbcFBNcv2EpCezejZAMSwipf37RlBJreMNiKtF7
N3LSTSGn4+qspyGtohRmIxcSbtgwqsA458MXEmTbM2IB6+1jxqooIdCrooQfQT68iqaVLKpLhm/C
/giaOswavluHq5D1/DpcovcfrcMbBnQWtpVZcmgqQpmRhZsCYw1vWGAlei8nMPxmBAZyKsP3HnGu
i2SVuxHH83VhPZOLqXDciOwlQlc4EYHp405kLrnmdvOtRsEttpff5eu307RH+QyXcZKXR4fsaNwh
WT5kt2uGIQ1HNB06jVuV7NbkkLJ+3/FzceuNtc/n/lHQ9G6YlreThlwFpuJucnUbfZzGRTL0btxh
RYp8KzehZFEcJ3405tMWutgRC8wXC9XhKwjz50cmNAjiITs34qoiHhtoYugDZfWpEBkFbiG5HZbn
p7hEl9MM7axN4089MLYPZ08MiNttcRIH8fhu6ExINKbD6mx+NiBJ/TgtWaZgQ19ASnA7m/hJEy+9
RIIdQQ/ZUxkgkGz+xIhoLm+DySvpXNHTZGFTwpE3ZIf5Qz/hx+jZOgpn7XNul0yb4SWexTGBgq4s
oxFkFwrT7Ll7emjEIkNn4hQ3TMcUtGO2qiJX5jHDXl9Af+zMQaiWheqjSbpIrFDRWf5mg4062JbZ
nyYr66ayoOtsoEs9AmOHyVSQilYZS2MeLhUxd6IFmufT+I2PdAZvGpBo6PkBtJUmI24Vx+kNSV1Q
xYCU94rvl1KBR8+JPk6Ens+pozXR9TmPdx5z+Yts/n/v/pGy2i9++P/X8f/oUf+PXtr/I1m4o3fu
/zc62y3PDN9q3r35IeFH2v2D027jw+2+Ibf7BtJupBrI+PC7q4/736qD3eJ4/8PDfhxsfHjYV/Ww
liqU4sPDzm/UVTej3uq9jC1uPn3XGf6Hh93ew+ofHvbDw9Y9LDZk/j7ce/ewD99Gfquudovbxtvd
Jf5wtd99TIutD1/7hnwtftTX4pf2tYalfHja2vMf1WO2b/WoYIvHaj+OCn60d/04jX1L3vUNZLLs
rODjNLb+hDxrLwqBYv9A3zvYw0bnoG9YHZCL3DExUtm3PSahXU3mzIujwI/oflKI19XltniwxHeE
P2HAH/DZlefkw5jP/q16coKVQ2Hu0Oau0Bbq0PSj67123dM/t7dd8q+Tm4q2lThXRVSY47sX+4X1
e4S988P+13qm/jCBVbq9nm24GaH44o8HpMcz+i24gBe4UDIBbcCExkMp38sKtMQK9KNZgdYohNxa
ftLIWHrS6PDsm354cngxp33xQaP5nOV06PJSlASRdYOo+sjpeNhUOyrSzM4IU7BudUQsisE2dG+J
UyJF2vZZo/WykFfkY1XadJBP2MvS+WqD3z2e47SxacuPm/Zmrwg0njRaL2d5Sc78HDb2cnDFFBKj
qT97e5dLOCgycNtHdErLN08V8Yqy0tX4Uk5SfAnIWPhYLyna05C2XdpOMtrOM6cdZmk7IbQdOrTt
3JptkvhOO6NJO8xT6BmzHtJ2wngKw3I2TdHbTsCLqbTD8Lbt3QKEDKDBX9ye5O3sLmNBsR3dsn56
m/MhQ0gU26kLSybtIGwrbhTfsD7+pZ3SZDiOY7cdxUnQhnZ3yBDx3XYS+YAX0tsBmXhDmOmEyTCg
Y+LctbNpyMZylJ3UrJrZq7wQk/NJEEflRfmO8vArWmzgPuaQvcH/pHf3hSrsx5Hnj/vgcnic5M+N
c0FWLwcfnHw9PDlYKSBS5PFJEZI9AlHYEfGXvVgMOTB7E989pqHI+cTTPzGveFC+PA0XTQdQM/rn
9UcGthG1HcMmI3tEbEf4o4BM6WxwtRO4b6rswuv7PBGh4aCsr2OYmHODvbE9qxlm8nxOYFi+l17W
PRLU8iJbx7x62AIMjgxwD4mUpFx7WFbeaQ1A16RjElVJzzzTYGlM4y2OKq94OMyU+7ZvR7snNVfz
2P5xTTxqJjUMo82CDKOEp14m3jIHKYMIP4V8AvXl7ZcNqa+UbkPqV6Rtj1CvbKngM+qXXsjfjPzy
xYUXEX4Jexv6YcpTxW9YT6FfPED2IuQb1rbUG9ZTiYet9BOILx/ueBHqxeZ+G+p188nU65tTv0Ds
9rcltjkqW+SJvgFPHt2gzRmmP9VZGI/4isU3VtcfMy6wcyklN9Y++7+ZVm2ddS8a4SYuaHOWGw+7
p23ftOUBnp2e7IsakJU8grjMBViBGn4gw+PorKIiTzEalXFPIU2YUMI7dK2cLUpRzefTsAgIZHLH
xJn4kch6ilbC6lIWLZge6F2d480aO38oWoc18jJftSFqeQX91VBfVWXvduVo3LhCjSutcdWcp8AV
Y30ajQdxkZbVelspidxYVL6JE8rewIb+O9g+LJURQhB/ZKVeW4qVMmo3igeJA7AysTr/5eCo1RQI
hzMrq3Z//3O/CMO7DkKmYVgPlPbS20jRbJuVTCdjOgyJH4ziW1bQDNn2QLR2j/3M6dJbVlOMVaBz
JtS5/hKnxzB4R3Jdyfc+94AX055LctJhNNC0V/5eAO4s/yM+SFF7oI4wNnN7wKph6Pb8CFYHeLCH
+uz6KXVgO1MEwSiInWuJH31+RtIo+4xkrAKIet20jQnG6PsIXi6qhv4iS4BrnLqZ5Ef/iOrXcZFD
Q4nu6C6nmfQJdEs63tsBghKfum1WEQ7pSEFYytqSDlsV6O1l/4jWFmvTHyTQwogTGCbnlJQlGXVs
23fkX1kcpYnTHbDOAS89BvRlNGK14ijf60jV3YBNGWq2wUe8Jw0Cgi3zTWoQ1tQHNOjbsbQ/IVFE
g0xidYppxKR/ePLldDWVugp2wl02L1DM6nrWao+WdSh3zw6/CS9VVSLdpJDvP237KwMswXSp9HKS
uHVQFonEvB5ppfXqQ9UJrbYig0F4fkAHfRjAVH21GMY0B4tw++yEfUfqFVnaYxUa1+lfGOU9Uac5
8rKyRLQ95C3DUvo9zSWqw2r2mpo36qiuizojKrsd1yAGsVSPqKZTqWZIc8LgN7RTimFpF1aOQEuZ
YgL79Jmi1rVz46KdwBBzZfHTFyrMqm4oJhU/RUzLxiL/BTWMpX7NjUXV1cpQ2NemnQAW2LKYnUCS
Y0nX32knQJbGSo+GTjnHtjW1KotZFWulrIDljnROHeqz6rGl+0ISYygwJAJdR3jNyg9xlC9VW7pW
BrVau4hK6VFw8hfpHZMn2FbZWFu++5T1F1yEKK9buYhyjS1rFO/mkCMl+YLeAcq9KUl7gT/qlVUj
e6zSY68ivrcJ+NrJYpfZe9hl1TKfQri+QDhLKr6T8C8QHEA5asJ5SaJFTdZ1SveQuuuvp+6LS1ub
Lt2VTmFtd1a/eXQn0SQOgicpvfHKRme8V6Mz36/Rma9ndNabMDrrdY1Ol9+p0S0R/n6MTkevZnRL
S7+K0S1i8cONDr9Xo1sk/B0Z3evtppaWfh2je+U9nf5e93RLhL8jo3u9Pd3S0s9gdK/9W60v8Wn8
/m/5s3VBPH7WNRhf1//+L2K/jzH7/V9FUfjv/xrax+///ojPkvGoeltVTIlZtST9HqfjLj+a7pa6
4cQpZU5hlMbXNO1+c7PLxCU5PS8i9uOG7Lc7r6RPffHE+68FSfN/D9hvOhYBTYe/xSlM6uiYmZ2X
0mzyLWTmDM5P8l1pEw8l+Vn1M2MiOgEuEgFQkj+OYvbou+Tnkg9RYELBhPkiUhHlflD7dTKA4cYR
/XmZeKttIvlFibd2pN8NRTdliuWrN8oGTWkr5svqgGG9VR3Q9LauKpsSP//2pQgCRgiwYz8OQwgy
jzJB25EGF7vnF21pefKnX2Lxs6LS/7V3rE1tI8n7uvyKuaT2AoWkaGQ9faWqOLbZhQLMAeFuK5Vy
jaQRiMiSV7IJrtT+9+uekWUbbOJAYG+vPB8SJKtn+jXdPY+e8QVtCsEX+xE82o0G2E8jBJwdppoN
21SDuOGpPLKpZ5tGSONQQab4CPJRQK8D80kRUUjpf1xHCp92FAKeAoTWJI5NA92N2HJuevZLcXNv
/3j/7Ndl7FSqO8Twhq/ZBcv+SJyJfWdfiF/vzlAIi0e8qM+6l7cG+QoZJpFPDdqwFDLgg18AIZaN
OI9wS6xvUUMhI3hTymstfx1f8hO8DKxq7t4ljr7l6VQh4u6uszJtJ8MrXpyNkxH3O9291odD0BDc
zjIZcn+26RkABkNf/Af8CFkRVUTF8sjrcIyr9f7XiN9AvX1quFboGkxljAWgAZGnuqHH1DD2rIjZ
lFleVH3qxNw2PddTDVAE+NRzVJdTgDRBZajjUNPR/YtBR3xNvorQdR2VAdIFCHy+ThPTz/1xltwC
R5H89q+t4+PuoULwSO5eEfHC15FzPDwBGQxKH/dMM3kLt/81AGHBzxAkjwoI4eTHoh5UwiTHi9IS
loI8UQoUQEFXQFLRlIfDdHyJj1JwuBzZy9LJ9FeJXitNWOlXoaY+5ftJgftz8Jo6gVJZXUQFpOOS
rOg6SchS7OHizR/T6n60oKpPY50Hjh4aquF5VDWtAAwANcAAMEOPDTeAmMx8okzXaeKlZWr8AJnS
l5HpE7m/ThM195OIT5k/3XR4uib/yzQf+fotGiwhilsdO2o+YEkmHvSpOIZhopB4nIlROfz0pO4F
CP+lu1b1qeuadhTYgWrZYaSahhsDAoauBpETBvBV6Bj2E/VgnSZqPRAuB8bJL9ATGz+gJxqPVoE/
MLYRDvTzzUA4beGqqaGbwo+KjaYn4HXFlYM+xc8ltNigo5WTkkUAmCHXkCcyEvgwlLw64tl40ft+
h+jCfDg5wQ1nC0FJfSW5X59Z7Rj1axE2qICmmFrqJOXnIzbEdHn/K+OeZwQBVUHUFmhuYKie/xV3
ecotQTcRA278q3v0od+56KinvaP5X8tCpKtNL2WHWAfnYc4xmol5sYAh8EJcJ+kXo9SlDU8Z3ijk
M58EOcQih2ySj0c+z9QxxK71zt0KdJiMTnmSXcspu1rikrZjedGiX+d/TnUBItPDa3bDtJRll1ov
QPB/vqOeRWMaQSiGqdK9OC5BgDoq2+0FZib71J7GViLims4k+SXaLKVMsuF4VCol/FbmhSJQCFj4
WSnl9h+lrPBSyjGEpzxKMFduGnPVWPu/36Z4C+xaMbHjes86vLLmJraYJJ5URgUn80AyhGXTnky+
XPEML3jGaz9x9HRxtN44rJIKcPdzIpSvetMkX+fty6IFeUZJLO9IoJHQN6aa+HC3uqd6te8QtgOG
AcOrJCwrMzf3mJZ1IP/Hcol7OumenvZOn0/i64qtssIkg2GwMMHiwuioUg9W1lOf44zfDsV6QDqR
W/SqqeUmeZqPsh3dalAeqnrcgNGoZVhqEJquGoHLopHLHACufRQerMDSNM+ziu3vW4eHvd7xA15K
5IdWXmjOY80cUGWFKv9TmZ/aAVUGeM7/VAg8PgRZoRPGRifqEaHDG0YE0YquB65quqAdLIwbKkRe
AW6DbQSx/SfqxJKg5DmUwlGshkv+3To9XukagjTV2mJX88VRW6aOf8cUyUcasCiIrPgTabOsk7fk
olkeEyb/ulM1ieVaU5wXZAwBHTn77ey8e6SBf2FlnpUg9tZpv99qn+/3jvv9Dyed1nlXEe8wpb/f
vzjq99uHHwDoVIG/qz/7+2f94955/6J1uN9ZygdbNx92kRFLtSio+gcbR8kIGCwWd8ATaS18cZhj
QrfYp/y9vMmLgqdyqnC/0yReRGGQAPp6kAfihRU1ILZvmDB2wC3aOih2oFuBGpmMcVuPDCOAqLoN
KkLORuDDmkQqQVuoCuni6pOoCIO4I3ltfZPUM03gywbDlEM3I9sXRwC8GIbKlKAm4RCn4SkA4NzL
UZIJ0GY1X9gZF9Wz6RIAzrOo3NGW89r8Bq9R5yRu/GLwOG07xFSJuOBy3TIXwRvpijbETx/5LR6E
ACELPpY++u3mesbrhqVjYMbF0Rb0uiugPqqq+LSEWk/xHOvb1PZu4g4bMWl019AeukPmlqOBwrGA
BKTKt+fwGicYSwK1LpMA4uR8G6fqZnaoRKK1lxfVXveTPE/XFgtFuXBumY4LcgGHkskVdAHeJGu1
QsR5RRlLmwTtpgaSHCVo+giLY+EeSJMI/V4n34ZgdNUkc83cZZLZ1HWlYdI/iUmPU961SK+Ut3ex
VxnP+0r8jOt/C+u/4rCK4qXXf6lhNu6u/xqGuVn/fYkiu5mO3czA/KymbivW4uCUDRn0Duhgk3zE
NXY91FrXw2maD3Sko7M2Qc3ByE9mERGqmjvkhI3FaLItAEnr4OQt1Rq4swPqUNU620d1Hd3YqhFx
Z4uQlm3MI3Id5GWpsVKrNr3khVaOg1Jkv5Ya5kCVMlMWkJJpiGcVWjKTiKABIKpKqLdDDt63znSK
eUZNcJ7DNBfbnSD8VQ8679uqcL8JAx8dFdCrCxKmDEbKiMYwL0eX4GF/T7WO/G17mmzmaXRnCSWm
Qm3rYUpW8bExh6mLFhKsFSJ60G6RsynxZPvgPVRI9os8O2AhgxExocBaT9uDmCBdjpLnLkcJA31s
YhVGdIqRQ6GuajsOogT0v52lcIFQG0vatRTTazxZu4ydGSceq1624pnL1etbHDDX5oC9pF1HoZa+
tN3K/EZCGwcQH2pliBMkK5WjFoWlU6NSDvBQe5gMJxSjU1c17QkYxkdVSDyZbRmTCYnSlb4VCPUL
XDEf8LczdMol1LiKZ7nfTc0cXvKdKmkqCXTOGVHQANnLxxluxirxiCjtC2h2kpFZpTNiNHKeQyyS
wOitmP8ghIpBP5iYxyR4qhVfqE+Lcvn1Cuq856euCmH4DyJuVt1DtHmK7tLnp02e59X/0QK8X+3D
tHqe+VhXssoC1XYZbMB7QatIw5WA5CNOVDffykPDcCQhh2yf7mInAltbX7MPfdMkWa5jz7mJOTbC
CP/VHLteLUfEsZ+GiLEeIjMVXYWH8zQ86Hp43FejZfiY0FWeiM+afMEBzPVKJLyXEc4XDoP2QZI9
wA7vZdjxhacQiPGHEHlix1kTkSgPy9VYUPpEdqzZfSu4BzFZ01091ZDk+egBLIw1A5yn8mM6al/d
Z14KFZwaxWVulq5ii6dYJp2f2a1xgap4FvEsnGhDGFCAs/smMq5lO9Nxi8Dh1Zy3nvP/CxqDex7l
kIyRqiEyyJGBZPsVKweAy6CJK3GvdsiXqyS8IgM2IQGvz9CA2HG6DAA+PB6PxgWfHrRR1guLGQTP
IddWsMD432XBwnBkMACatICzbDxK0vLZ+dL4S/ElhFgrfHaemH8xnkBkKtZQnltbTIiOnsYZ+r2c
mffEL2xH/uwpsv/rsjD/O0uY0oZsFF79oDYenv/VDcOx6/lfx3L+plPbpMZm/vcliqqqZF7sEy2H
EfhPwuronkptQvWmYYDV0epVC6LioHdrd3d3EfSn+TUbr2k5TauBd3o0XKq7JkBZAPXuHVGpq9hk
F/51yLt3W+Q1ZoLAmH+Ui4yN026rc9QluMjU7p38tn/8ixj5l2LuCk9dg9hrxMAnY+CFACmYCTxa
jrzeIlu7yQDjMMKLIsu3SPUkDR+YrNmbZMDrh5Kj5Ra44Woo2cXj2HSB3HwBMnlz8RUWgI5x0QTs
bzC+3H5V77z6WdPd/6zMJ3yl3K/qTpE5ilu7d9+/xvRc5BewJEyCdFLVDnSCbd7ryI0Z3ZPe4eFS
4Bz3NaQABko9xFOuAF+056K98j7IqJg077+tae9LgsZZwS9xMrTYlrmbSzDnt3gsGNnvdYsCxMlK
wlfU/JrsYzYPJ93jXvf4fPlHSUzAP6Goyd99KXNNfr+iViwFS0Bblv4sJbwS8o6k3+xN2T8jHdlY
jgpweCDznyPyZnVtd8obITTBSrLX2sdjYBWympGrMX0ElqLp9VB9IzEUB6h1OwtIziEHgtkW2kT+
UfUuTajj/vHOnR6UB9fEr/SoNiaXfFTpkFIdVLcAA7UDWHMTGmzKpmzKpmzKpmzKpmzKpmzKpmzK
pmzKpmzKpjxY/gvI+YHWAMgAAA==
--MP_/IiROFVEHqzB/cq/fJkmmawn--
--Sig_/xM2/5weqzNCEiCQ1Zp6THsR
Content-Type: application/pgp-signature
Content-Description: OpenPGP digital signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBCAAGBQJWWHbxAAoJEMHFVuy5l8Y4QnMP/i2Y8VyKsEdP8gDiM+GuK6T0
vUT7G67jhz4IcFKUV2/8yGIHQsy6lwzaAg3gk+PlnGvS2DL+1rJGYyrbd90g1QHy
vOTLVAP1Fm9yZEcQPDDwdXlToJV8rZ0BTOVJi9+xKxg3FepxbTATful8/elTGSqg
6TrzUcaTHn6H2AI2g88kfiJ2M+9CbuNIUcZD29+3AfBuHRKcu5HbdJAHXsLHkJHs
IS7ThfCX9idje1Q6FK09XQ7+9aIKl7/VFf/jpGSV6WgObiauHKDz+cYW3adcbXl0
ocKmrbhZF7/Q8z0xOqpuc7BA+wg06KGYxX8f32Re5/qtY2ab95M6+73cGSu9Ph/T
JKNei0Xb6LocjAhypzv19JXXoOVjghUOZjLluPsc15fYBpgt/BnYMNOAJDPuyj4k
PfPqblxsS1gw5ZC7ojQ4WQQGifDQjhOkpbaIckb+eeCfhMMMkEICA7A4OmBU6qs1
MU/bFaOaypy5rzuDb//n4cYbzA/TNku3M97wsy37mGblBCgIsorFSzv9IElZdXHZ
5g7MmBMKiRw1tP0AUUGLgAdRQq4LnlH0BANo7PxhILudZLUSKhm3d8jNP4r0Zby5
Ce/67DQiFTzt3dEy7OL4q7wlqmGspHM6BL0Jmdg+gRvNJ5hvlxXrz0Nzh/Khzrc5
G6o9xdGw3ALRmyyM851d
=/Bhs
-----END PGP SIGNATURE-----
--Sig_/xM2/5weqzNCEiCQ1Zp6THsR--
3082
days inactive
3097
days old
5 comments
3 participants
participants (3)
-
Dan Kenigsberg -
Robert Story -
Vinzenz Feenstra